ICMP spikes after 23.05 upgrade
-
@stephenw10 I do not understand. You do not see difference between PRE upgrade and after upgrade? Difference is only when ping goes thru Netgate, when doesnt, then it is normal.
-
Yes I see the difference, I'm trying to understand where the pings are between that see that difference so I can consider what might have changed in the path to cause that.
-
@GeorgeCZ58 said in ICMP spikes after 23.05 upgrade:
This is not ping to 8.8.8.8, it is ping from Zabbix monitoring tool to eg network switch on another local VLAN
We all see the image.
Only you know what is pinging from where to what trough what.
I (or we) it's a 'from pfSense or behind pfSense to somewhere in front of pfSense, but again, your actual setup could be anything.Pinging to some LAN or VLAN device : that's boring :
[23.05-RELEASE][root@pfSense.local.tld]/root: ping diskstation2 PING6(56=40+8+8 bytes) 2a01:cb19:beef:a6dc::1 --> 2a01:cb19:beef:a6dc::c2 16 bytes from 2a01:cb19:beef:a6dc::c2, icmp_seq=0 hlim=64 time=0.319 ms 16 bytes from 2a01:cb19:beef:a6dc::c2, icmp_seq=1 hlim=64 time=0.286 ms 16 bytes from 2a01:cb19:beef:a6dc::c2, icmp_seq=2 hlim=64 time=0.240 ms 16 bytes from 2a01:cb19:beef:a6dc::c2, icmp_seq=3 hlim=64 time=0.689 ms 16 bytes from 2a01:cb19:beef:a6dc::c2, icmp_seq=4 hlim=64 time=0.286 ms 16 bytes from 2a01:cb19:beef:a6dc::c2, icmp_seq=5 hlim=64 time=0.231 ms 16 bytes from 2a01:cb19:beef:a6dc::c2, icmp_seq=6 hlim=64 time=0.214 ms 16 bytes from 2a01:cb19:beef:a6dc::c2, icmp_seq=7 hlim=64 time=0.258 ms 16 bytes from 2a01:cb19:beef:a6dc::c2, icmp_seq=8 hlim=64 time=0.219 ms 16 bytes from 2a01:cb19:beef:a6dc::c2, icmp_seq=9 hlim=64 time=0.299 ms ....This shows a ping over a one Gbit link - head to head, with 32 port HP switch in between.
When I start to slam the diskstation2 device, a NAS, at full speed, I want it to react fast.
The ICMP packets, by nature, have less priority.
I will see spikes. I want to see spikes.Btw : if there is some other devices like smart switches on the path, this will add a random factor.
Still, I understand you question better now.
Your hardware (a 7100) should deal everything you throw at it. -
Yes, and I wouldn't expect to see a difference like that simply from upgrading.
Do you know if it did it in 23.01 also?
-
@stephenw10 unfortunately I didnt test 23.01. I was afraid about this release, so I go thru 23.01 to 23.05. In that time Zabbix was stopped as it will send lot on notification because of missing router meantime upgrade.
On sunday I will try to reboot all switches if production in plant will be stopped. I will come back with result. Otherwise maybe it will be better to make clean instalation.
-
@GeorgeCZ58 so I reboot core switches, but nothing changed. So issue is related to pFsense upgrade. Also it is visible on pFsense monitoring, here is graph of ping on 8.8.8.8 before and after update.
Any idea how to improve pings to get same pings as were on 22.05?
-
@GeorgeCZ58 I just check second XG7100 that I upgrade to 23.05 and it is same:
-
@GeorgeCZ58 so that is pfsense pinging 8.8.8.8 and your saying before it was like less than 1 ms (from your graph).. So your in the same DC as google?
Not sure how something in pfsense could change how long it takes something to respond to a ping that is sent by pfsense - if you were pinging through pfsense - ok maybe..
But if I send a ping from device A to device B.. How would device A have some effect on the response time? The only thing I could think of that could have effect is the payload.. But pfsense normally sends zero sized pings.. Or it use too - maybe that has changed to be 1 now vs zero
See
https://docs.netgate.com/pfsense/en/latest/routing/gateway-configure.html#advanced-gateway-settingsBut not sure how a size of 1 vs zero would cause it to go from like 1ms to 4? So the echo request is padded, so its not going to be 1.. So maybe going from 0 to 60 might be able to cause that sort of a difference?
Looks like length is 43 in my capture
What is the setting on your monitor, in routing - gateway, advanced
-
@johnpoz said in ICMP spikes after 23.05 upgrade:
So your in the same DC as google?
That's the std deviation. The actual ping times went from ~19ms to 20ms. The increase is 1ms at most from what I can see.
-
@stephenw10 so maybe the increase in packet size could maybe account for that? But my point still stands - there is really nothing that the client (pfsense) could do that could increase the time something takes to respond - other than sending a larger packet..
As to an increase in the std dev, that is a measure of the jitter really or how much the numbers fluctuate. Again your pinging across the public internet, something that takes 19ms RTT.. There is going to be fluctuation, how could the client sending the pings have anything to do with that?
Can see that in just local pings that are normally under 1ms
From 192.168.9.253: bytes=60 seq=0004 TTL=64 ID=9e18 time=0.495ms Packets: sent=4, rcvd=4, error=0, lost=0 (0.0% loss) in 1.504176 sec RTTs in ms: min/avg/max/dev: 0.402 / 0.456 / 0.498 / 0.041Now increase that size.
From 192.168.9.253: bytes=540 seq=0004 TTL=64 ID=bd0e time=0.491ms Packets: sent=4, rcvd=4, error=0, lost=0 (0.0% loss) in 1.508622 sec RTTs in ms: min/avg/max/dev: 0.357 / 0.456 / 0.530 / 0.064Notice the std dev increased.. Even though my avg is the same.. Sending 4 pings did take 4 ms more total time.. etc. .
-
@stephenw10 yes you are right. Small detail, I forgot that I am pinging 1.1.1.1, not google one. But I think it doesnt matter. Problem is, that "something" has changed. Question is, if it is related only to XG7100 on 23.05, or it is behavior on 23.05, that pings have lower priority then on 22.05 .
One XG7100 is with original expansion card and one is without - the basic version. Till now I think all services are working properly, I think (and hope) that only ICMP is affected.
-
~1ms change like that shouln't really affect anything. It's nothing like the sort of spikes you were seeing in Zabbix initially.
-
@stephenw10 what I am not understanding is how could anything in pfsense have to do with it pinging something across the public internet.. Makes no sense - other than the size of the ping, pfsense has no control over how fast something goes across the public internet and how long that something your pinging takes to respond..
Pfsense knows when it put something on the wire, and when it returned.. It has no control on how long that something might take to respond... or any possible delays across the multiple hops to get from A to B and back again...
Nor does it have any control over what the jitter or std dev in a sample of pings might be.. The only thing I could think of that could effect the std dev that might be in the control of pfsense monitoring some remote IP is the sample size it uses to determine that std dev..
-
If it's doing any sort of shaping it could.
We saw issues like that when reloading the ruleset was triggering a huge CPU load previously. But not continually.
-
@stephenw10 is he doing shaping? Traffic on the link could for sure cause fluctuations that is for sure..
-
@johnpoz said in ICMP spikes after 23.05 upgrade:
@stephenw10 is he doing shaping? Traffic on the link could for sure cause fluctuations that is for sure..
Hi, if this was question on me - no it is not related to shapeing. It seems like some sort of drivers issue on xg7100. This probably starts on 23.01 . This not happaen on APU or build from intel NUC (realtec NIC)
-
@GeorgeCZ58
You will always get a small change (less than 1ms in your case) in ping time across the internet, post-upgrade. This is normal and to be expected.This has nothing to do with the pfSense update itself, it just due to the reconnection to your ISP triggered by the reboot. The reconnection is invariably a different pathway than before (BGP, hops, load balancing, contention, IPv6 vs 4 etc). The physical routing differences in distance/time will always change the inherent latency.
Your example is particularly mild, with just a ~1ms change on the new connection path. This is excellent but it is not always the case.
My link should, theoretically, run a ping at 5.8ms to the first external hop. Wonders being what they are, on rare occasions it does exactly that. A more typical figure is around 7ms:
However, due to major fibre infrastructure changes going on, a reconnect can select a new pathway as slow as 25ms. In practice I just reconnect again until I can get a first external hop down in single digits (as that helps with precision clock sync with some external equipment).
The graph below shows an increase in ping on reconnection from 7.2ms to 9.7ms, as a working example. It also covers another significant point - where to test to.
If you are looking at your ISP connection or pfSense you should only be looking at your first external hop - in the cases above and below you should only look at or only ping to Hop 2. There is nothing you can do to influence anything beyond that point:
Your connection to connection variance looks remarkably good and any reboot or reconnection will inevitably change the pathway and therefore the ping.
In sum, everything is just fine for you (but less so for me).
️ -
@RobbieTT but this is issue taht occure only after upgrade to 23.05. It is not related only to WAN side, but also to LAN side(routing between VLANs). I tried reboot of Netgates with no resolution. And for me it is not fine, I want to have same results like in 22.05.
-
As I understand it the concern here is not the ~1ms bump but the 20-30ms spikes shown by Zabbix?
-
@GeorgeCZ58 said in ICMP spikes after 23.05 upgrade:
@RobbieTT but this is issue taht occure only after upgrade to 23.05. It is not related only to WAN side, but also to LAN side(routing between VLANs).
You posted about your pings changing to both 1.1.1.1 and 8.8.8.8 - these are WAN addresses. My reply was to the WAN issue you raised:
Also it is visible on pFsense monitoring, here is graph of ping on 8.8.8.8 before and after update.
Any idea how to improve pings to get same pings as were on 22.05?
Your graph shows as sub-1ms increase to 8.8.8.8 (a WAN target) post-upgrade & post-reconnection. That change shown is tiny; you actually had slightly greater variance before the upgrade.
The line you point at is the standard deviation, so it will be influenced by any spikes in the ICMP ping - its just maths. ICMP pings have the lowest priority across any network, so spikes are expected. If any router along the pathway is busy with packets the humble ICMP ping is going to be the first to get dropped.
In no way can ICMP be used to represent or infer what happens to regular (non-ICMP) traffic. Indeed, many sites do not allow ICMP traffic at all or limit it to the lowest ping size possible.
️
