Ping spikes one one machine, not on another
-
I've got two identical Qotom quad-ethernet fanless units, one at home and one at my church. The problem is that pings to the church firewall will periodically spike to nearly a second of latency. If I ping (from the firewall to a remote host) at 100msec interval, they all come back in a bunch, returning e.g. 9xx,8xx,7xx,etc. msec RTT. If I run mtr on the remote host, latency to all intervening hops is fine (including the terrestrial-wireless CPE), but over time the maximum ping to the final (pfsense) hop can reach 1.5sec in some cases.
Here's why this is driving my nuts:
- my home firewall is identical, with a very similar config (I've spent a fair bit of time confirming that), and I see no such spikes
- there's no "load" on the firewall, snort/ntopng have been disabled, etc.
- if I hook a cheapo Linux laptop to the WAN in place of the firewall, I see no such spikes
- the lag is in the outbound direction, if I tcpdump both ends of the ping requests coming into the destination the jitter is observable at the destination end while the responses make it back in "normal" time
- originally the outbound ping spikes seemed to correlate strongly with inbound traffic, but that seems to have been alleviated by a CPE replacement (provider is changing tower radios, both are active so maybe new was blowing out old?)
- I've turned powerd to Maximum, and that seemed to reduce frequency as well
- coincident with the CPE replacement I also found that the machine was running excessively hot, and something was bashing at the "disc" until I rebooted it, but again that only seemed to partially relieve the problem
- I've set up traffic shaping, no difference either way
- while it's not representative, a test I did just now had the spikes fairly regularly about 2.5sec apart, but that's not typical. That may actually be part of the correlation vs inbound traffic, since I'm currently seeing inbound spikes on that period, which I usually associated with e.g. somebody watching YouTube
Given that we're trying to roll out a VoIP system, I've got to find a solution to this or our call quality is going to randomly suffer. I'm not sure where else to look, so I'm hoping somebody will have an idea.
Thanks!