Issues with internet connectivity only for linux systems on a subnet

cde

Hello everyone,

I have searched on many forums posts for an answer that would help me but to no avail.
I have a Proxmox server with 3 network interfaces, all 3 are linux bridges. Those 3 linux bridges are attached to a virtualized pfSense router/firewall which is used to isolate my homelab from the rest of the home network.
The pfSense configurations have been modified so there is an OpenVPN service running so I can RDP to specific systems on the LAN network and I have added firewall rules to isolate the lan from my home network. The Windows systems on the LAN are able to access the internet without any issues. On the other hand, any linux systems on the same lan interface cannot access the internet unless I restart the network interface on the pfSense. Despite not being able to access the internet, the linux systems are able to ping and connect to other linux and windows systems that are connected to the same interface and they can reach the pfSense interface IP.

As part of troubleshooting I have tried:

restarted dns and dhcp services and nothing changed
Disabled dhcp server and assigned manually static IPs, nothing changed
Added static dhcp and arp mappings
Checked routes both on linux and windows and they are the same
I have disabled proxmox firewall
tried ovs bridges instead of linux bridges and the problem is the same
tried moving the a VM to my WAN interface which is operated by my home router and everything works fine, but those VMs should not be on the home network as they are part of the lab.

On all the checks windows connectivity was unaffected and linux systems continued to not have access to the internet. If i refresh the network interface from the web-console all the linux systems are able to access the internet and all is well. After this, if I power on or restart another linux system, it does not have internet connectivity and I need to refresh the router interface again.

Linux distributions that I have tried and have this issue:

Debian bookworm
Ubuntu 20 22 24
Ubuntu Server
KDEos
Kali linux
Paladin OS

stephenw10

@cde said in Issues with internet connectivity only for linux systems on a subnet:

If i refresh the network interface from the web-console

How exactly are you doing that?

After they regain connectivity what happens to make it fail again?

When they are unable to connect what error is shown if, for example, you try to ping pfsense.org?

cde

@stephenw10

How exactly are you doing that?

i go to the interface as seen in the image below, and without making any changes i click save at the bottom of the page

and this makes the interface "refresh". So basically without making any changes the linux systems are conencted to the internet.

After they regain connectivity what happens to make it fail again?

The linux hosts lose connectivity after this if they are rebooted or their networkmanager is rebooted.

When they are unable to connect what error is shown if, for example, you try to ping pfsense.org?

no error when pinging pfsense.org, it is not even starting in fact (i havent tried that in the past) which means that it cannot find a dns server i guess, while for pinging 8.8.8.8 the ping utility starts but nothing happens.

the next image is after the refresh and no other changes on the vm:

stephenw10

If it's unable to resolve it I'd expect some error there but you can confirm by just trying to resolve it:

steve@steve-NUC9i9QNX:~$ host pfsense.org
pfsense.org has address 208.123.73.78
pfsense.org has IPv6 address 2610:160:11:11::78
pfsense.org mail is handled by 10 aspmx2.googlemail.com.
pfsense.org mail is handled by 1 aspmx.l.google.com.
pfsense.org mail is handled by 10 aspmx3.googlemail.com.
pfsense.org mail is handled by 5 alt1.aspmx.l.google.com.
pfsense.org mail is handled by 5 alt2.aspmx.l.google.com.

Do you see anything blocked in the firewall log?

Check the state table in Diag > States when you are trying to ping 8.8.8.8 from a linux client. Do you see the states created? On which interfaces?

If you don't see any states then check a packet capture on the pfSense LAN.

If you see no packets then check the ARP table on the client device. Is it still showing the correct MAC for the pfSense LAN?
If it isn't then you probably have some IP conflict. Though pfSense would normally log that if it can see the arp requests.

cde

@stephenw10
connection refused before refresh:

Nothing is blocked and I do not see any packets in the packet capture.
The IP and correct mac address are on the ARP table.

Gertjan

@cde

Please confirm that you have not installed pfBlockerng(-devel) and that you are not plan to install it.

Be ware that pfBlockerng will use the same IP for its own need (can be changed) :

Unbound (or dnsmasq) is listening ?

sockstat | grep 'unbound'

Btw : "Connection refused" tells me that unbound did get the DNS request, but unbound was informed to 'refuse' it.
Nothing special unbound ACLs ?

Normally, default, ACL isn't used ...

cde

@Gertjan i am using unbound and i do not have pfBlockerng

Btw : "Connection refused" tells me that unbound did get the DNS request, but unbound was informed to 'refuse' it.
Nothing special unbound ACLs ?

nothing on ACL

stephenw10

Check the ARP table.

cde

@stephenw10 there is an entry for the system but no internet access. still i am able to ping the pfsense and other systems on the subnet

cde

@stephenw10 on the otherside, the firewall cannot ping the same system

stephenw10

No check the ARP table in Kali for the 10.10.10.1 address. It seems at least possible something else is using it and sending the wrong MAC.

cde

@stephenw10 top is before and bottom is after the refresh.

stephenw10

Aha, well there you go. Only one of those MACs is correct. Where is that other MAC coming from?

cde

@stephenw10 thank you so much! i found the culprit!

on my proxmox i had setup the switch interface to be on the 10.10.10.1/24 and so I had two interfaces with the same IP... you are a god thank you!

stephenw10

Ah, that would do it! Interesting that it only seemed to affect Linux hosts though. I suspect eventually Windows would also have hit it.

cde

@stephenw10 who knows... thanks again! I was going crazy for two weeks now until I decided to reach out to this forum! i was asking coworkers and people on mastodon but all i needed to do in the end was arp-scan and see that there was a conflict with two macs on the same IP. Saddly I cannot upvote you because of my low reputation.

For future readers please give stephenw10 an upvote for me <3