Packet loss every hour during peak hours
-
We have 8 adapters on our VMWARE HCI environment for our pfSense
Only the LAN is on VMXNET and configured for multiple queues - https://docs.netgate.com/pfsense/en/latest/hardware/tune.html#vmware-vmx-4-interfaces while the rest of the interface is still on the E1000
Running version 2.7.2 with version 23.09.1 available for upgrade
Got two internet links. We notice that during peak hours from 8am to 5pm there will be packet loss seen on our multiping server to targets beyond the WAN interface almost after every hour.
This correlates to our pfSense State Table graph and the bandwidth graph as per below.
We did upgrade our RAM from 4GB to 8GB hoping it'll fix it but to no avail. I notice that state table on the dashboard goes beyond 60% with the automatic table size set (812000) according to the RAM upgrade
I'm trying to figure out why it is doing this every hour. Is there a memory setting or any other settings I should look into that is causing this every hour during our peak hours or may be the LAN on VMXNET while the rest of the interface is still on E1000 adapter type. Majority of our traffic enters LAN and exit WAN
Couldn't figure out why it's doing this
-
Have you attempted traffic shaping and tried to utilize different algorithms yet?
https://forum.netgate.com/topic/171842/queue-management-algorithms-differences
-
What does the CPU usage look like at that time?
It seems likely that something is actually opening states every hour and that should create significant load. Perhaps enough to start dropping pings.
-
@stephenw10 CPU looks fine
-
Are you seeing packet loss logged on the pfSense gateway monitoring also?
-
@stephenw10 Yes, I can see hourly packet loss logged on the GRE tunnels via our main ISP link. Don't see it on the GRE tunnels via the 2nd ISP link. Is there a way to download those logs ?
-
The traffic quality RRD data? You can download it directly from /var/db/rrd.
-
@JonathanLee Yeah just got this done today and applied on one of the WAN interfaces to see how it goes - https://docs.netgate.com/pfsense/en/latest/recipes/codel-limiters.html#configuring-codel-limiters-for-bufferbloat
Do I need to do this on the DMZ as well? Users normally upload big files from LAN to DMZ. We also have Guest on one interface averaging 30Mbps during peak hours. I also notice running speed test on our primary and it goes to around 240Mbps download around 60Mbps upload even though we got we got DIA 500Mbps Up/Down on our primary ISP
-
@usaiat I would only test the wan side first and see if that helps at all.
