Network strangeness with pfSense
-
I have a Data Center (DR) that I just barely stood up a few months ago. The DR site is very simple, it has a single Netgate SG-4860 1U Firewall running 2.4.3 pfSense. It is connected internally to a 1GB switch (Ubiquity) with a single VLAN. The Network is a CIDR of /20 (255.255.240.0), so a big flat network.
I have 5 Proxmox Virtualization hosts connected using Bridges and the pfSense firewall is the default gateway for all the hosts/VMs. I initially made the pfSense the gateway by default as I didn't have a layer-3 switch installed at the time. I since now have a L3 switch but I haven't changed the routing to be on the switch yet.
What I am seeing is that if I connect to a VM that is running KVM and I try to then connect to a LXC (old school container) even if it is on the same host as the VM, about 30 seconds later all of my connection dies and I have to log back into the VM to restart the connection.
What is strange is that I did a Packet Trace on the pfSense firewall and I can see all the packets going through the firewall. What confuses me is that I thought that if the VM is local on the network that the packet wouldn't need to be sent to the firewall and then back?
My main question is anyone have thought on this? Would there be something when I mention "30 seconds" that might indicate maybe something with my pfSense configuration that may need to be adjusted or disabled? I have been fighting this for about 2 weeks now and can't make sense of it.