2.9.0: Gateway flapping with 500 ms latency to the directly connected WAN router – caused by Snort inline (netmap) on igb

[maex]

2.9.0: Gateway flapping with 500 ms latency to the directly connected WAN router – caused by Snort inline (netmap) on igb

This is a heads-up, not a help request – the problem is solved on my side. Posting it because the symptoms look like a WAN/ISP problem and it took me a day to find the actual cause.

TL;DR: After an in-place upgrade to pfSense CE 2.9.0, dpinger reported ~500 ms latency (stddev >600 ms) to the ISP router one cable hop away, causing the WAN gateway to flap every 30–90 s and pfSense to restart OpenVPN/IPsec/filter in a loop. The box was idle (2 % CPU). Cause: Snort in Inline IPS mode (netmap) on the igb WAN interface. Switching Snort to Legacy mode brought latency down to 0.5 ms / 0.1 ms stddev immediately. Hardware and ISP link were fine throughout.

Environment

• pfSense CE 2.9.0-RELEASE (FreeBSD 16.0-CURRENT, RELENG_2_9_0-n256132), in-place upgrade from 2.8.x
• AMD Ryzen 7 3700X, 32 GB RAM, Intel I350 quad port (igb0–igb3), ZFS
• WAN: static IPv4, single ISP, no failover
• Packages: Snort (WAN, was inline mode), pfBlockerNG, Squid+ClamAV, HAProxy, ntopng, Service Watchdog

Symptoms

System log in a loop, every 30–90 s:

php-fpm  NOTICE GW_WAN has high latency, omitting from routing group ...; 1.1.1.1|x.x.x.114|GW_WAN|1149.994ms|2022.465ms|1%|down|highdelay
check_reload_status  updating dyndns GW_WAN
check_reload_status  Restarting IPsec tunnels
check_reload_status  Restarting OpenVPN tunnels/interfaces
check_reload_status  Reloading filter
php-fpm  ERROR No default gateway found
...
php-fpm  NOTICE GW_WAN is available now, adding to routing group ...; ...|98.489ms|240.394ms|0.0%|online|none

Gateway log, monitor IP = the ISP router itself, directly connected:

dpinger  WAN_A1 x.x.x.113: Alarm latency 504977us stddev 627227us loss 0%
dpinger  WAN_A1 x.x.x.113: Clear latency 472003us stddev 630288us loss 0%

Meanwhile a manual ping -c 5 x.x.x.113 from the pfSense shell: 0.3–0.9 ms, 0 % loss. top: 95–98 % idle, load 0.4.

Kernel messages that pointed to the actual cause:

kernel  netmap_transmit igb0 full hwcur 642 hwtail 708 qlen 957
kernel  netmap_transmit igb0 full hwcur 423 hwtail 438 qlen 1008

and on every Snort (re)start:

kernel  iflib_netmap_config txr 8 rxr 8 txd 1024 rxd 1024 rbufsz 2048
kernel  igb0: link state changed to DOWN
kernel  igb0: link state changed to UP
php-fpm  Hotplug event detected for WAN (wan) ... rc.newwanip starting igb0
php-fpm  pfSense package system has detected an IP change or dynamic WAN reconnection ... Restarting packages.

What it was NOT

• Not CPU load (Ryzen 3700X at 2 %).
• Not the ISP link (router answers in <1 ms, 1.1.1.1 in 30–45 ms once the loop was broken).
• Not the monitor IP being rate-limited (same values whether monitoring 1.1.1.1 or the router).
• Not missing netmap tuning: hardware checksum offload, TSO and LRO were already disabled on igb0 (ifconfig igb0: options=48120b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,HWSTATS,MEXTPG>, no TXCSUM/RXCSUM/TSO4/LRO). No VLANs on igb0.

Root cause

Snort in Inline IPS mode uses netmap on igb0. On this 2.9.0 build the netmap TX ring fills up (netmap_transmit igb0 full, qlen close to the 1024 slots), so packets – including dpinger's ICMP to the next hop – sit in the ring for hundreds of milliseconds. dpinger measures ~500 ms, crosses the default 500 ms high-latency threshold, the gateway action fires, pfSense restarts OpenVPN/IPsec/filter, the resulting rc.newwanip restarts all packages including Snort, Snort re-attaches netmap and bounces igb0, and the cycle starts again.

Before/after with the same monitor (directly connected router), same hardware, same rules:

Snort mode	dpinger latency	stddev
Inline (netmap)	~500 ms	~630 ms
Legacy (pcap)	0.50 ms	0.12 ms

Fix

1. Services > Snort > Snort Interfaces > WAN > Edit > IPS Mode: Legacy Mode, restart Snort on WAN. (Check afterwards with ps -ax -o pid,command | grep '[s]nort -R' that only one instance is running – the mode switch left me with two.)
2. System > Routing > Gateways > WAN gateway > Advanced: Disable Gateway Monitoring Action. On a single-WAN box there is nothing to fail over to; the action can only cause restart storms.

After that: zero gateway alarms since boot, no Restarting/rc.newwanip lines in the system log, no netmap_transmit messages.

Things that made it worse (worth checking if you see the same loop)

• Service Watchdog monitoring dpinger and OpenVPN. pfSense stops/starts those itself on gateway events; the watchdog interpreted that as a crash and restarted them every minute. Remove dpinger, OpenVPN, Snort and pfBlockerNG services from the watchdog.
• Leftover gateway entries from earlier versions. I had a <gateway>dynamic</gateway> gateway on a static WAN plus a duplicate static gateway with the same IP, and a failover gateway group with a single member. Every event fired twice, and when the only member was marked "high latency" the group was empty → No default gateway found. Clean state: one static gateway for the static WAN, no group.

Environment details (per the Suricata/netmap tuning thread checklist)

• ifconfig igb0: no TXCSUM/RXCSUM/TSO4/TSO6/LRO/TXCSUM6/RXCSUM6; MTU 1500
• dev.igb.0.fc: 3 (flow control was enabled – the one tuning item I had not applied)
• dev.igb.0.iflib.override_ntxqs/override_nrxqs: 0 → 8 TX / 8 RX queues, 1024 descriptors each (iflib_netmap_config txr 8 rxr 8 txd 1024 rxd 1024 rbufsz 2048)
• iflib driver 2.5.28-fbsd; all txqN.r_stalls, r_drops, no_desc_avail, tx_map_failed = 0
• dev.netmap.buf_size: 2048, buf_num: 163840, ring_size: 36864, admode: 0, generic_rings: 1 (all defaults)
• RSS enabled, 16 CPUs, 32 buckets
• No VLANs, LAGG or bridge on igb0

Note for anyone digging deeper

The netmap_transmit igb0 full messages come from the host-stack → netmap host ring, i.e. the ring the netmap application (Snort's netmap DAQ) has to drain. That ring filled to ~1000 of 1024 slots repeatedly while the machine was at 2 % CPU and the Snort process itself showed no CPU time. So this does not look like the usual "Snort can't keep up" explanation, but like the host ring not being serviced for stretches of time. I have not tested dev.igb.0.fc=0 or inline mode on a LAN interface – Legacy mode does what I need, so I stopped there.

PS. I know WAN is not the interface to go... ;-)