2.4.4 - ping to WAN getting slower after 2 minutes - dual WAN - apu2

swix

Hello, bonjour,

Since a few weeks, my pfSense router is getting a relatively high latency a few minutes after reboot or simply 2 minutes after disconnecting the WAN Ethernet plug for 10 seconds. There was no configuration change according to the history under /diag_confbak.php.

Setup: apu2 (pc engines) with WAN (connected to a Fritz!box serving as DSL/G.Fast adapter), LAN and CABLE (failover, Tier 2). I replaced all the cables, checked most of the logs, disabled the CABLE interface completely, but nothing helped yet and this is how it looks like:

WAN DSL Adapter has IP 192.168.178.1, pfSense 192.168.178.20.

PC Engines APU2 - Netgate Device ID: **************

*** Welcome to pfSense 2.4.4-RELEASE-p3 (amd64) on pf ***
 WAN (wan)       -> igb0       -> v4/DHCP4: 192.168.178.20/24
 LAN (lan)       -> igb1       -> v4: 192.168.1.100/24
 CABLE (opt1) -> igb2       -> v4/DHCP4: 178.83.123.123/21

Just after reboot:

[2.4.4-RELEASE][admin@pf]/root: ping 192.168.178.1
PING 192.168.178.1 (192.168.178.1): 56 data bytes
64 bytes from 192.168.178.1: icmp_seq=0 ttl=64 time=0.789 ms
64 bytes from 192.168.178.1: icmp_seq=1 ttl=64 time=0.689 ms
64 bytes from 192.168.178.1: icmp_seq=2 ttl=64 time=0.220 ms
64 bytes from 192.168.178.1: icmp_seq=3 ttl=64 time=0.335 ms
64 bytes from 192.168.178.1: icmp_seq=4 ttl=64 time=0.588 ms
64 bytes from 192.168.178.1: icmp_seq=5 ttl=64 time=0.553 ms
64 bytes from 192.168.178.1: icmp_seq=6 ttl=64 time=0.569 ms
64 bytes from 192.168.178.1: icmp_seq=7 ttl=64 time=0.271 ms
64 bytes from 192.168.178.1: icmp_seq=8 ttl=64 time=0.557 ms
64 bytes from 192.168.178.1: icmp_seq=9 ttl=64 time=0.589 ms

About 2 minutes later:

[2.4.4-RELEASE][admin@pf]/root: ping -c 30 192.168.178.1
PING 192.168.178.1 (192.168.178.1): 56 data bytes
64 bytes from 192.168.178.1: icmp_seq=0 ttl=64 time=37.161 ms
64 bytes from 192.168.178.1: icmp_seq=1 ttl=64 time=13.256 ms
64 bytes from 192.168.178.1: icmp_seq=2 ttl=64 time=20.262 ms
64 bytes from 192.168.178.1: icmp_seq=3 ttl=64 time=53.008 ms
64 bytes from 192.168.178.1: icmp_seq=4 ttl=64 time=29.080 ms
64 bytes from 192.168.178.1: icmp_seq=5 ttl=64 time=29.190 ms
64 bytes from 192.168.178.1: icmp_seq=6 ttl=64 time=22.295 ms
64 bytes from 192.168.178.1: icmp_seq=7 ttl=64 time=55.928 ms
64 bytes from 192.168.178.1: icmp_seq=8 ttl=64 time=69.357 ms

--- 192.168.178.1 ping statistics ---
30 packets transmitted, 30 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.235/24.537/69.357/16.741 ms

At the same time, pinging to a remote host (for example 8.8.8.8) passes from about 5-10ms to 60-100ms.

I wish I could reset the device and try from scratch, but it is not possible at the moment because of OpenVPN + IPSEC services running in production. I tried some other things like "disable hardware checksum offloading" or such, but no success yet. There is unfortunately also a double NAT since the Fritzbox cannot be run 100% as bridge mode, but the pfsense device is set as "Exposed host" there (anything coming to the WAN external IP is directly forwarded to the pfsense router).

Does this reads like something familiar to you, and if yes, what would you suggest me to try/check next ?

Thanks a log in advance & kind regards !

swix

PS: I just also noticed some of these lines when running a "dmesg" on the device:

arpresolve: can't allocate llinfo for 192.168.178.1 on igb0
arpresolve: can't allocate llinfo for 192.168.178.1 on igb0
arpresolve: can't allocate llinfo for 192.168.178.1 on igb0
arpresolve: can't allocate llinfo for 192.168.178.1 on igb0

This may also be important... I will continue my research later, focusing on this too.

swix

Update: it got much better when I replaced the Fritz!box 7582 with a Zyxel XMG3927 this morning. Tests still in progress, but in this case it seems pfSense was not the issue at all.

[2.4.4-RELEASE][admin@pf.insign]/root: ping -c 10 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: icmp_seq=0 ttl=60 time=4.159 ms
64 bytes from 1.1.1.1: icmp_seq=1 ttl=60 time=3.865 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=60 time=4.053 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=60 time=4.342 ms
64 bytes from 1.1.1.1: icmp_seq=4 ttl=60 time=3.719 ms
64 bytes from 1.1.1.1: icmp_seq=5 ttl=60 time=3.745 ms
64 bytes from 1.1.1.1: icmp_seq=6 ttl=60 time=3.957 ms
64 bytes from 1.1.1.1: icmp_seq=7 ttl=60 time=3.731 ms
64 bytes from 1.1.1.1: icmp_seq=8 ttl=60 time=3.973 ms
64 bytes from 1.1.1.1: icmp_seq=9 ttl=60 time=4.102 ms

--- 1.1.1.1 ping statistics ---
10 packets transmitted, 10 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 3.719/3.965/4.342/0.195 ms

(until now it was around 60-100ms for any external IP).