Wired issue that only happens on linux guest VMs in Proxmox

jjuk

I posted a similar topic last year but I hadn't collected enough info before posting, so I decided to start a new topic with more details.

This problem still exists even though both Proxmox and pfSense are updated to the latest version.

The reason I suspect is pfSense problem but not Proxmox is because Windows VMs have no such problem. Only happens on Linux VMs (MX Linux, Fedora, Ubuntu 1804, 2204, 2404).

Everything is virtual unless I mention the thing is physical

Settings in Proxmox:

pfSense VM with two NICs, (I've tried VirtIO and Intel E1000E)
One of the NICs is bridged to physical NIC of hypervisor as WAN port
Another NIC is bridged to a SDN VNet as virtual network switch LAN
All guest VMs have one NIC (again, I've tried VirtIO and Intel E1000E)
These guest VMs all connect to the SDN VNet network switch
Hardware settings between Linux VM and Windows VM are basically the same

Settings in pfSense:
Firewall rules allow all guest VMs to reach everywhere, which means they can go to the Internet, ping each other, go to and login pfSense by web portal, etc.

The problem is:
Linux VM can ping, but cannot go to the Internet and cannot reach pfSense web portal. Unless, the pfSense VM is restarted.

Example 1:

pfSense is already started and running.
I power on a Linux VM, then found that it cannot go to the Internet.
Restart pfSense. Then the Linux VM becomes normal.
Restart the Linux VM. Problem happens again.

Example 2:

pfSense is not yet started (means no network for all VMs)
I power on a Linux VM. No network access is expected as firewall isn't started.
I power on pfSense, then all VMs network connection work.
Restart the Linux VM. Problem happens again.

In a nutshell, network connection for Linux VMs only work if they start before pfSense.
However, not all network connection types have problem. Such as ping function, it always works.

Gblenn

@jjuk

This sounds super strange, and spontaneously I think it points towards something in the way the Linux machines get their IP settings... If you check your IP settings in the Linux VM when it works and when it doesn't work, what is the difference? No DNS for example?

Are you using KEA DHCP server in pfsense btw?

viragomann

@Gblenn
Interesting behavior for sure.
But as I read the last line
@jjuk said in Wired issue that only happens on linux guest VMs in Proxmox:

However, not all network connection types have problem. Such as ping function, it always works.

I went out.

Maybe the TO should double-check this.

jjuk

@Gblenn

I recently change the DHCP server types to KEA from ISC, but that was just because ISC is going to deprecated.

I couldn't find any differences in the IP settings in the Linux VMs. They are just set up like normal workstations, which means their network settings are totally controlled by the pfSense DHCP server, their IP address, DNS, etc.

jjuk

@viragomann

Maybe I should give more details about the "ping"

When the problem exists.

The pfSense VM can ping all the guest VMs on LAN subnet, it can also ping the physical devices on WAN subnet.

The Linux VMs can ping the other VMs in the same subnet, and pfSense VM. But not the internet such as www.abc.com.

As my settings has only one virtual LAN subnet, I haven't test whether pinging cross subnet work or not.

Gblenn

@jjuk So when the Linux VM's are not able to ping www.abc.com, they do show the correct default DNS in their IP settings? Like 8.8.8.8 or whatever it is that your DHCP server is handing out? What is the DNS that you have chosen in the DHCP server?

Have you tried pinging an IP on the internet directly, like ping 8.8.8.8? What result does that give you?

And did you try to switch back to ISC from KEA? Just to see if that changes anything?

jjuk

@Gblenn
Linux VMs can't ping internet IP address too.
DNS in the DHCP server is set to IP address of upstream physical router.
Both ISC and KEA tried. In fact, I change to KEA as an attempt to see if it can solve the problem.

I wonder why DNS and DHCP could be the cause. Because the Linux VMs can't even reach the pfSense web portal. They are in the same subnet.

For example, pfSense IP is 192.168.100.1, one of the Linux VMs IP is 192.168.100.2
Linux VMs get their IPs from pfSense DHCP.
pfSense and Linux can ping each other.

This shows that DHCP in pfSense is functioning.

I connect to pfSense web portal by typing IP address in web browser address bar. This doesn't need DNS resolution.

Gblenn

@jjuk Ok, so they can ping each other but nothing else is working is what you are saying. Anything in the firewall logs showing something is blocked? Do you run Suricata or Snort?