Trace why outgoing traffic doesn't traverse the gateway

lifeboy

I have a development environment set up with Proxmox that has quite a few KVM's and Linux Containers spread over 7 nodes. The whole cluster is protected by pfSense on a physical device. All these guest can access the internet and can be accessed via SSH and whatever other services are running. Except one new machine I set up on Friday last week.

I can ping the LAN addresses and the firewall WAN port too from the LXC/guest machine, but not the gateway (next hop) past the WAN address.
I cannot access web services running on the LXC via the web (just using info.php to test for now), but I can via the LAN address from another LAN host.
When I change the container ip address, I can access public internet addresses for a few minutes, ie ping 8.8.8.8), but then it starts timing out.
I use NAT, just I do for the other containers and KVM's

I can't find anything in any logs that shows what happens.

It's not a IP address conflict or a duplicate MAC address.

Even if I disable the firewall/packet filter and use pfSense only as a router, I still can get any traffice to exit the gateway.

What could be causing this and how can I trace this?

lifeboy

I had the static arp option enabled in the DHCP server some time ago as below. However, I disabled that when this cluster was only used for development.

Static ARP: Enable Static ARP entries
This option persists even if DHCP server is disabled.

The note about persistence: How can I clear this? It may be the cause of this issue.
As a test, I re-enabled it and added the MAC address and a static ARP entry, but it makes no difference, the machine still cannot reach the internet and vice-versa.

lifeboy

Just to clarify the Static ARP option: When I enable that, there are some guests that I cannot reach because they don't have static ARP entries. As soon as I disable the option again, they become reachable.

However, this has no effect on the machine in question.

lifeboy

@lifeboy I have scrapped the VM and will start over. Clearly something went wrong that is too time-consuming to troubleshoot now.

BlueSun

@lifeboy Did you ever figure this out?

lifeboy

@bluesun, no I haven't.