WAN packet loss when new LAN connection made
-
Apologies in advance this is quite a long complicated one. I am fairly experienced with networking but this is giving me nightmares. I am not sure if issues is pfSense, but tbh I am at a total loss as to where issue even lies as it is not making any sense.
I am with a UK cable provider with a Hitron CGNV4 in modem mode, I then have 2 pfSense routers (SFF PCs with pfSense community installed) connected in high availability and one server connected directly. I have 5 ips, 1 for directly connected server, 2 for each pfSense router and 2 in CARP for feeding to other servers in the LAN.
For the past few weeks I have been experiencing packet loss (40-75%) and high pings (upto 2000ms) whenever a device connects on the LAN. This don't happen with every single time which has made fixing a nightmare as everytime I think I have got the issue it crops up again later. Effectively ~75% of the time a device connects I get packet loss for around 5-20 mins on WAN. It only occurs at this time (e.g. a new LAN connection made) if no new connections made connection remains stable so I don't believe it to be an ISP issue.
It affects everything LAN based, so hard wired and WIFI devices both cause issues. Unplugging the device once packet loss starts does not stop, I have unplugged both pfSense routers leaving just the directly connected server and that was still showing packet loss. After about 5-20 mins everything returns to normal and will then stay stable until another device connects.
The LAN connection remains stable throughtout with no issues with any internal communication. Everything on pfSense appears to be working correctly (e.g. Carp / DHCP failover / etc..) when tested.
I get same results if I also try ping under web interface directly from hitron modem.
Everything been rebooted countless times and I have tried turning off each pfSense router for a day to see if it was specific to one installation but still get same issues.
Bandwidth monitoring shows nothing is coming close to saturating the connection, with utilisation very low.
I am totally at a loss and wouldn't have even thought something like this would be possible, I cannot think how a local connection would bring down WAN totally and to not come back up even after cables pulled. Issue replicates quite consistently where I can have laptop pinging an external address with no issues, switch on a PC and packet loss starts at exactly the same time (as windows login appears) give or take a second or two.
Does anyone have any idea or even an idea on what to troubleshoot?
-
@vx said in WAN packet loss when new LAN connection made:
Does anyone have any idea or even an idea on what to troubleshoot?
You can start with packet captures to see what's actually happening on the network. PfSense has Packet Capture built in and you can also use Wireshark. I expect you have a managed switch, with port mirroring available. If so, you can run Wireshark on a computer and use mirroring to watch individual ports. The switch status lights may also help. For example, a defective connection might cause a port to switch to 100 Mb mode, which can be seen in the lights on many switches. So, you have to collect some info for us to provide advise, though I doubt it's a pfSense issue.
-
Finally tracked the issue down which was with the firewall state sync.
This was setup on seperate interface and seperated from normal LAN traffic via 802.1Q VLAN on switches (the two routers were in different areas and weren't possible to run another cable through). No idea why this was causing such problems but disabled now and rather a brief connection interuption if it switches over.