Latency spikes on netgate 7100 ver 23.0.5.1
-
I have a netgate 7100 and just recently updated the firmware from 22.05 to 23.0.5.1 and now im seeing random packet loss jitter and latency spikes. We track all of our traffic with ping plotter and it started to happen directly after the update. Ive done some pings through the firewall to other devices on the same network and see latency spikes so i know its the hardware not the ISP. This is common across 2 sites we have updated to this version both 7100 hardware. I disabled gateway monitoring on one with the same result. The latency spikes can be up to 200ms.
-
How are they connected? Which interfaces is that traffic passing?
How are you testing to see those spikes?
Steve
-
@stephenw10 depends on the device some are in the switch ports some are behind the sfp+ port. As mentioned above ping plotter both internal and external also just basic ping from device to device. Anything passing through a normal switch doesn’t have the same spikes. I did a restore back to the older firmware and the issue went away.
-
So you're seeing this through both the switch ports and just via the SFP ports?
We have seen reports of this but I've never been able to replicate it to that extent.
-
@stephenw10 built in switch ports and SFP ports, however it may just be the wan throwing off the SFP being that the Wan is through a built in switch port. I can reproduce it easily and if netgate wants to take a look im more then happy to show them.
-
Ah, so you don't see it for traffic that is only using the SFP ports?
And it's consistently 200ms spikes? I spent quite a while testing this and did find an increase in latency but it was only ever in 5-10ms range. Still much higher than I expected.
-
@stephenw10
Top is inbound from wan bottom is outbound to a server about 100 feet away via wan. I rebooted it last night and reverted to 23.01 and it seems to be a little less then before. I want to say even pinging the local interface on the pfsense had latency spikes. We had some issues with some voip quality for fax and the carrier said they saw tons of jitter and latency, the said carrier is only 1ms away. The baseline average latency for the bottom ping plotter trace is .98ms. -
@stephenw10 heres a bit higher
-
@stephenw10 this is directly to the local interface
-
Hello, I am facing this issue on all ours XG7100, that are installed across Europe. Unfortunately the fix is not in 24.01 release, as internal team doesnt know what is causing the issue.
For me it is big problem, as I buyed many XG7100... Already tested that on different models this issue do not occure(6100, 3100, APU4, APU6, home build routers..).
Here you can see when the XG7100 was upgraded to 23 release:
-
@GeorgeCZ58 I ended up reverting to the last known working release. We also had both the pieces of hardware fail almost at the same time 1000 miles apart. They no longer powered on. We are phasing out pfsense.
-
@snapcom there are issues related to power supply. It is enough to replace power supply with different one, and XG7100 will work. We already had this situation.
-
@GeorgeCZ58 im not sure if it was related or not, the hardware is hundreds of miles away from me so I haven't had a chance to get it back to look at it.
-
Netgate team, please can you resolve those latency issues? It started in 23 release and continue in 24 and is related only to Netgate 7100. I cannot replace all devices just because of your unresolved bug.
-
Were you able to test the public 24.08-dev snapshot?
-
@stephenw10 Just tested, and issue is still existing here. Because I replaced one 7100 with 6100 model, I am able to help to Netgate team debug issue. Is there somebody who can work on identify this issue?
-
I will just add comment - the issue is still not resolved in 24.08 release.
-
@GeorgeCZ58
What "24.08 release"? What version exactly? -
@w0w I tested pfsense + 24.08 dev version.
-
@GeorgeCZ58
If you are talking about latest available public snapshot, it is two months old, just wait for the fresh one it should be available soon.
