Firewall locks up - root cause analysis + solutions (pfSense Plus 26.03.1 on Supermicro Xeon D-1521)

rle

Included in analysis:

Firewall locks up, possibly unbound config

Hopefully this report will help others.

With the help of AI I tracked down various issues and hard lock ups on my pfSense node (experiencing it on and off for months now).
Divided over two posts. First post: problem setting + fixes. Second post: extended report + source code of files.

The problem

My pfSense box was suffering from gradual internet speed degradation over
several days, always ending in a complete system lockup requiring a physical
reset. The daily scheduled reboot (cron) didn't always fire -- the system was
already too far gone. No crash dumps, no kernel panic messages in logs, just
a multi-hour silent gap followed by a clean boot after pressing the reset
button.

Platform: Supermicro Xeon D-1521, 32 GB RAM, pfSense Plus 26.03.1-snapshots
(FreeBSD 16.0-CURRENT #12, build date May 20 2026).

Diagnosis

WAN: FTTH 8 Gbps via PPPoE.

Three root causes were identified, one primary and two contributing.

Root cause 1 (primary): CPU stuck at half speed

The smoking gun was checking CPU frequency:

sysctl dev.cpu.0.freq
# Result: 1100 MHz (rated: 2401 MHz)

All 8 cores were running at 1.1 GHz -- less than half the rated 2.4 GHz.
During the diagnosis, the frequency bounced between 800 and 1700 MHz, dropping
within one second of being manually set back to max.

The dmesg boot log had the clue:

sysctl_register_oid: can't re-use a leaf (machdep.hwpstate_pkg_ctrl)!

Two FreeBSD drivers are conflicting on this kernel:

est (Enhanced SpeedStep) -- driver module, loads and registers first
hwpstate_intel (Hardware P-State) -- tries to register the same sysctl, fails

The EST driver aggressively downclocks for power saving and does not ramp back
up fast enough under network interrupt load. With 15 million NIC interrupts
in 50 minutes (4,843/sec) and an interrupt queue already dropping packets
(intr_queue_drops: 453), the system drowns.

The degradation cycle:

Boot > CPU at 1100 MHz > NIC interrupts pile up > 
Latency increases > Retransmits add more interrupts > 
CPU can't catch up > Interrupts overflow > 
Speed drops > Eventually unresponsive

Root cause 2: Swap overcommit

vm.swap_total:   1.0 GB
vm.swap_reserved: 3.6 GB   (335% overcommit)

The kernel was reserving 3.6x more swap than actually existed. Under the
pfBlockerNG update cycle, this could trigger page allocation failures.

Root cause 3: unbound memory baseline

unbound RSS was 1,071 MB after 50 minutes. The pfBlockerNG python module
loads its DNSBL data structures entirely inside unbound's process space
(85 MB zone file + 16 MB data file parsed into memory). Cache tuning only
constrains runtime growth, not this baseline.

Fixes applied

CPU frequency (primary fix)

Three-layer approach:

1. Continuous watchdog script (immediate, running now)

A background script resets all cores to 2401 MHz every 5 seconds:

#!/bin/sh
while :; do
  for c in 0 1 2 3 4 5 6 7; do
    sysctl dev.cpu.${c}.freq=2401 > /dev/null 2>&1
  done
  sleep 5
done

Started from /etc/rc.local with & (must background or boot hangs).

2. Boot-level disable (will take effect after next reboot)

In /boot/loader.conf.local (survives pfSense updates):

hint.est.0.disabled=1
hint.acpi_throttle.0.disabled="1"

3. Per-minute cron fallback

Added to crontab as secondary safety net. Note: EST's interval is sub-second,
so a per-minute cron alone is insufficient -- the 5-second watchdog is
essential.

Result: CPU now stable at 2400-2401 MHz.

Swap

Added an 8 GB swap file via mdconfig, bringing total swap to 9 GB (1 GB
partition + 8 GB file). swap_reserved (3.5 GB) now fits comfortably within
swap_total (9.3 GB).

Unbound tuning

Parameter	Default	Tuned
`outgoing-range`	4096	2048
`msg-cache-size`	10m	5m
`rrset-cache-size`	20m	10m
`cache-min-ttl`	0	300
`prefetch`	yes	no
`prefetch-key`	yes	no
`serve-expired`	no	yes

pfBlockerNG cron priority

Main update cron lowered to nice -n 19.

Key lessons for pfSense on FreeBSD-CURRENT

Always check sysctl dev.cpu.0.freq if you see unexplained slowdowns.
It should match the rated clock speed under load.
sysctl_register_oid: can't re-use a leaf (machdep.hwpstate_pkg_ctrl)
in dmesg = EST and hwpstate_intel are conflicting. The EST driver wins but
scales too aggressively.
intr_queue_drops is a direct indicator of CPU starvation on packet
processing. Non-zero at idle means the CPU is already overwhelmed.
loader.conf.local survives pfSense package updates and config
regeneration. Use it for boot-time tunables.
vm.swap_reserved >> vm.swap_total is an OOM precursor that shows up
long before swapinfo reports any usage. Watch this metric.
A per-minute cron is NOT fast enough to counteract EST's sub-second
frequency scaling. You need a watchdog loop with sleep 5 or shorter.

rle

pfSense Plus Stability Report

Host: pfSense Plus 26.03.1 on Supermicro Xeon D-1521
Date: July 21, 2026
Author: Diagnosis and remediation session

1. Platform Specifications

Hardware

Component	Detail
Motherboard	Supermicro Super Server
CPU	Intel Xeon D-1521, 4C/8T @ 2.4 GHz (Broadwell-DE)
RAM	32 GB (4 x 8 GB DDR4)
Storage	Samsung MZVLB256HAHQ-000H1, 256 GB NVMe (51,325 POH, 15% wear)
LAN/NIC 1	Intel X552/X557-AT 10GBASE-T (ix0)
WAN/NIC 2	Intel X552/X557-AT 10GBASE-T (ix1) - FTTH 8 Gbps PPPoE (pppoe0)
NIC 3	Intel X710 for 10GbE SFP+ (ixl0) -- VLAN trunk
NIC 4	Intel X710 for 10GbE SFP+ (ixl1) -- VLAN trunk

Software

Component	Version
pfSense	Plus 26.03.1-snapshots
Kernel	FreeBSD 16.0-CURRENT #12
ZFS	2.4.99-292-FreeBSD
Build date	Wed May 20 15:20:10 UTC 2026
BIOS	Supermicro Super Server (Supermicro)

Installed pfSense Packages

Package	Version	Status
pfBlockerNG	3.2.16_1	ACTIVE (DNSBL python module in unbound + filterlog daemon)
HAProxy	0.65.7	ACTIVE (PID 11192, 33 MB RSS)
FreeRADIUS	0.16.4	ACTIVE (PID running, 71 MB RSS)
Kea DHCP	3.0.2_1	ACTIVE (PID running)
WireGuard	0.2.13_4	ACTIVE (php_wg daemon running)
node_exporter	0.18.1_8	ACTIVE (PID 89711, 22 MB RSS)
Suricata	7.0.8_16	INSTALLED (rules update failing, process may be idle)
FRR	2.1.2_1	INSTALLED (no active routing sessions)
OpenVPN	2.6.20	INSTALLED (auth-script plugin loaded)
NUT (UPS)	2.8.2_9	INSTALLED (driver shows connection issues to UPS_White)
strongSwan/IPsec	6.0.7	INSTALLED (not running on this boot)
Zeek	3.0.4	INSTALLED (no process detected)
ACME	1.2	INSTALLED (cert renewal, cron-triggered)
Backup	0.7	INSTALLED (cron/event-triggered)
Cron	0.3.8_8	ACTIVE (system cron daemon)
Service Watchdog	1.8.7_7	INSTALLED
System Patches	2.3.3	INSTALLED
REST API	2.8_2	INSTALLED
Netgate Nexus	26.03.1	INSTALLED
Traffic Totals	2.3.5_2	INSTALLED (vnStat backend active)
UDP Broadcast Relay	1.2.8	INSTALLED
TFTP Server	0.1.3_10	INSTALLED
iperf	3.0.6	INSTALLED
Status Monitoring	1.8_9	INSTALLED

2. Symptoms

Internet speed is fast immediately after reboot, then gradually degrades over days
System eventually becomes completely unresponsive -- requires physical reset button
Daily scheduled reboot (/etc/rc.reboot at 02:03) often fails to fire
No crash dumps written, no kernel panic messages in logs -- system simply stops with a multi-hour silent gap in the log

3. Root Cause 1: CPU Throttled to Half Speed (PRIMARY)

Discovery

The CPU was running at 800-1700 MHz instead of the rated 2401 MHz -- between 33% and 71% of intended speed. After the most recent boot, observed frequencies were:

09:37: 1500 MHz (after 50 min uptime)
09:43: 1100 MHz (dropped while idle)
09:46: 2401 MHz (manually forced)
09:46+01s: 1700 MHz (EST driver dropped it back within 1 second)
09:48: 2400 MHz (watchdog active, stable)

Why

dmesg: sysctl_register_oid: can't re-use a leaf (machdep.hwpstate_pkg_ctrl)!

Two FreeBSD CPU frequency drivers conflict on this FreeBSD 16.0-CURRENT kernel:

est (Enhanced SpeedStep) -- driver module, registers first during boot
hwpstate_intel (Hardware P-State) -- tries to register the same machdep.hwpstate_pkg_ctrl sysctl second, fails

The EST driver then aggressively downclocks the CPU for power saving, dropping to 1.1 GHz at idle (46% of rated speed). Under network or interrupt load, it fails to ramp up fast enough.

The degradation cycle

Boot (fast) --> CPU at 1100-1500 MHz --> NIC interrupts pile up at 31K/sec per queue -->
Network latency increases --> Retransmits generate more interrupts -->
CPU can't catch up at half speed --> Interrupt queue overflows (intr_queue_drops > 0) -->
Packet loss --> Perceived speed degradation -->
System enters feedback loop of more retransmits/less CPU --> Eventually locks up

Evidence

Metric	Value	Significance
CPU frequency	800-1500 MHz vs 2401 MHz rated	33-62% of intended speed
NIC interrupt rate	31,250/sec per queue (locked)	Driver does not adapt to load
Total interrupts	15,325,450 in 50 min (4,843/sec)	High for idle system
`intr_queue_drops`	453 in 50 min at idle	CPU already losing packets
`vm.pfault_oom_attempts`	3 in 50 min	Kernel hitting allocation failures
CPU temperature	62C	Normal for Xeon D-1521

Fix: Three-layer approach

Layer 1: Continuous watchdog (IMMEDIATE)

A background script at /usr/local/bin/cpu_freq_watchdog.sh runs every 5 seconds
and forces all cores to 2401 MHz:

while :; do
  for c in 0 1 2 3 4 5 6 7; do
    sysctl dev.cpu.${c}.freq=2401 > /dev/null 2>&1
  done
  sleep 5
done

Started from /etc/rc.local with & (must background).

Result: CPU stable at 2400-2401 MHz with watchdog running.

Layer 2: Boot-level disable (PERMANENT AFTER REBOOT)

/boot/loader.conf.local:

hint.est.0.disabled=1
hint.acpi_throttle.0.disabled="1"

This prevents the EST driver from loading entirely. loader.conf.local survives
pfSense package updates and config regeneration -- it is NOT overwritten like
/etc/crontab or /var/unbound/unbound.conf.

Layer 3: Per-minute cron (FALLBACK)

* * * * * root for c in 0 1 2 3 4 5 6 7; do sysctl dev.cpu.${c}.freq=2401 > /dev/null 2>&1; done

Added to /etc/crontab as secondary safety net. Note: EST's scaling interval is
sub-second, so per-minute cron alone is NOT sufficient -- the 5-second watchdog
is the critical layer.

4. Root Cause 2: Swap Overcommit (SECONDARY)

Discovery

vm.swap_total:   1,073,741,824 (1.0 GB)
vm.swap_reserved: 3,594,932,224 (3.6 GB)  -->  335% overcommit

The kernel reserves swap space for every memory allocation. When reserved exceeds
total, the kernel has zero headroom to page out under pressure.

Fix

8 GB swap file created via mdconfig:

dd if=/dev/zero of=/swap bs=1m count=8192
chmod 0600 /swap
mdconfig -a -t vnode -f /swap -u 0
swapon /dev/md0

Persisted in /etc/rc.local.

Result: swap_reserved: 3.5 GB well within swap_total: 9.3 GB (38% utilization).

5. Root Cause 3: Unbound Memory Pressure (CONTRIBUTING)

Discovery

unbound RSS: 1,071 MB after only 50 minutes uptime. The pfBlockerNG
python module (pfb_unbound.py) loads entirely inside unbound's process space.
The pfb_py_zone.txt (85 MB) and pfb_py_data.txt (16 MB) DNSBL data files
are parsed into in-memory data structures. Cache tuning does not reduce this
baseline -- it only constrains runtime growth.

Fix

Parameter	Default	Tuned	Effect
`outgoing-range`	4096	2048	Reduces socket/epoll memory
`msg-cache-size`	10m	5m	Smaller message cache
`rrset-cache-size`	20m	10m	Smaller RRset cache
`cache-min-ttl`	0	300	Prevents cache thrash
`prefetch`	yes	no	Stops proactive re-querying
`prefetch-key`	yes	no	Stops proactive DNSSEC
`serve-expired`	no	yes	Serves stale data while refreshing

6. Persistence Mechanism

Layer	What	When	Reliability
`/etc/rc.local`	Swap mdconfig, watchdog start, best-effort sed patching	Early boot	Swap: reliable. Seds: unreliable (before pfSense writes configs)
`/etc/cron.d/pfsense_tweaks`	@reboot: CPU freq lock (10s delay) + unbound restart (30s)	After boot	Reliable -- runs after pfSense configs are finalized
`/boot/loader.conf.local`	Disable EST + ACPI throttle at boot loader	Pre-boot	Permanent -- survives pfSense updates
Per-minute cron	CPU max frequency	Every minute	Fallback only (too slow alone)
`loader.conf.local`	`hint.est.0.disabled=1`, `hint.acpi_throttle.0.disabled="1"`	Boot	Survives pfSense updates

7. Files Deployed

File	pfSense Path	Purpose
`rc.local`	`/etc/rc.local`	Swap, watchdog start, unbound patching
`pfsense_tweaks.cron`	`/etc/cron.d/pfsense_tweaks`	@reboot: CPU freq + unbound restart + crontab fix
`cpu_freq_watchdog.sh`	`/usr/local/bin/cpu_freq_watchdog.sh`	CPU frequency watchdog (5-second loop)
`loader.conf.local`	`/boot/loader.conf.local`	Disable EST and ACPI throttle at boot

8. Verification

# CPU frequency
sysctl dev.cpu.0.freq        # Expected: 2400-2401

# Watchdog running
ps aux | grep cpu_freq_watchdog

# Swap
swapinfo -h                  # Expected: 9 GB total, 0 used

# Unbound config
grep -E "(outgoing-range|msg-cache-size|cache-min-ttl|prefetch|serve-expired)" /var/unbound/unbound.conf

# pfBlockerNG cron priority
grep pfblockerng /etc/crontab | grep nice

# @reboot entries
cat /etc/cron.d/pfsense_tweaks

# Boot-level EST disable
cat /boot/loader.conf.local

# Link health
ifconfig ix0 | grep media
sysctl dev.ix.0.mac_stats.crc_errs  # Expected: 0

9. Source code of files deployed

rc.local

#!/bin/sh

# Swap file - added Wed Jul  1 14:40:05 CEST 2026
mdconfig -a -t vnode -f /swap -u 0 && swapon /dev/md0

# Unbound memory tunings - re-apply after pfSense regenerates config
sleep 3
sed -i "" "s/^outgoing-range: 4096/outgoing-range: 2048/" /var/unbound/unbound.conf
sed -i "" "s/^msg-cache-size: 10m/msg-cache-size: 5m/" /var/unbound/unbound.conf
sed -i "" "s/^rrset-cache-size: 20m/rrset-cache-size: 10m/" /var/unbound/unbound.conf
sed -i "" "s/^cache-min-ttl: 0/cache-min-ttl: 300/" /var/unbound/unbound.conf
sed -i "" "s/^prefetch: yes/prefetch: no/" /var/unbound/unbound.conf
sed -i "" "s/^prefetch-key: yes/prefetch-key: no/" /var/unbound/unbound.conf
sed -i "" "s/^serve-expired: no/serve-expired: yes/" /var/unbound/unbound.conf

# Restart unbound with tuned config
/bin/kill -9 $(/bin/pgrep unbound) 2>/dev/null
sleep 1
/usr/local/sbin/unbound -c /var/unbound/unbound.conf

# pfBlockerNG cron with low CPU priority
sed -i "" "s|/usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php cron|/usr/bin/nice -n 19 /usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php cron|" /etc/crontab

# Start CPU frequency watchdog (est/hwpstate conflict workaround)
/usr/local/bin/cpu_freq_watchdog.sh # Lock all CPUs

# Lock all CPUs to max frequency - est/hwpstate conflict on FreeBSD 16-CURRENT
for c in 0 1 2 3 4 5 6 7; do sysctl dev.cpu.${c}.freq=2401 > /dev/null 2>&1; done

pfsense_tweaks.cron

# pfSense post-boot tweaks
# Runs 30s after boot to re-apply settings that pfSense overwrites on boot
@reboot root sleep 10 && for c in 0 1 2 3 4 5 6 7; do /usr/sbin/sysctl dev.cpu.${c}.freq=2401 > /dev/null 2>&1; done && sleep 20 && /bin/sed -i "" "s/^outgoing-range: 4096/outgoing-range: 2048/" /var/unbound/unbound.conf && /bin/sed -i "" "s/^msg-cache-size: 10m/msg-cache-size: 5m/" /var/unbound/unbound.conf && /bin/sed -i "" "s/^rrset-cache-size: 20m/rrset-cache-size: 10m/" /var/unbound/unbound.conf && /bin/sed -i "" "s/^cache-min-ttl: 0/cache-min-ttl: 300/" /var/unbound/unbound.conf && /bin/sed -i "" "s/^prefetch: yes/prefetch: no/" /var/unbound/unbound.conf && /bin/sed -i "" "s/^prefetch-key: yes/prefetch-key: no/" /var/unbound/unbound.conf && /bin/sed -i "" "s/^serve-expired: no/serve-expired: yes/" /var/unbound/unbound.conf && /bin/kill -9 $(/bin/pgrep unbound) 2>/dev/null; sleep 1; /usr/local/sbin/unbound -c /var/unbound/unbound.conf && /bin/sed -i "" "s|/usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php cron|/usr/bin/nice -n 19 /usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php cron|" /etc/crontab

loader.conf.local

if_ixl_load="YES"
hint.est.0.disabled=1
hint.acpi_throttle.0.disabled="1"

cpu_freq_watchdog.sh

#!/bin/sh
# CPU frequency watchdog - EST/hwpstate conflict workaround
# Forces all CPUs to max frequency every 5 seconds
while :; do
  for c in 0 1 2 3 4 5 6 7; do
    /usr/sbin/sysctl dev.cpu.${c}.freq=2401 > /dev/null 2>&1
  done
  sleep 5
done

stephenw10

You shouldn't need anything custom for CPU frequency control like that.

You should only ever use one frequency control method. If your CPU supports both SpeedStep and SpeedShift you should use SpeedShift. It's far far faster reacting. It runs in the CPU directly rather than relying on a software daemon. The controls in pfSense only configure it. It varies with CPU but if you set SpeedShift towards the performance end of the scale it should have no problems. Say at 30%.

If you really want maximum performance and are not worried about heat or power then just disable both. Running multiple custom scripts is almost never a good idea.

rle

@stephenw10

I really appreciate your feedback. And you are right.

The problem was a suspected and annoying OOM and/or failing hardware or even a configuration mistake.
For the life of me, I could not find a cause or find any other solution whatsoever by myself (reinstalled, used a backup, removed packages, reinstalled packages clean, etc.).

SpeedShift (HWP) is indeed far faster than SpeedStep (EST). HWP runs in CPU microcode, EST is a software driver. The Xeon D-1521 does not support HWP.

stephenw10

Well if you enable powerd (the daemon that drives speedstep) you do have options for it's behaviour. The default setting of high-adaptive is usually pretty good.

But simply disabling that should run the CPU at full speed all the time.