[ZTE] pfnet-controller consumes 170-180% CPU on Netgate 6100 with Zero Trust Egress enabled

mr_nets

Summary
Enabling Zero Trust Egress (ZTE) causes sustained high CPU usage by pfnet-controller on a Netgate 6100 running pfSense Plus 26.07.

The issue is reproducible: disabling ZTE causes the CPU usage of the same pfnet-controller process to drop from approximately 180% to approximately 20%.

Environment
Hardware: Netgate 6100
pfSense Plus: 26.07
CPU: 4 cores
Feature: Zero Trust Egress (ZTE)
Affected process: /usr/local/bin/pfnet-controller
Configuration: /var/etc/pfnet-controller/pfnet-controller.conf

Steps to reproduce

Run pfSense Plus 26.07 on a Netgate 6100.
Enable Zero Trust Egress.
Allow the system to operate normally.
Observe pfnet-controller CPU usage with top or ps.
Disable Zero Trust Egress.
Observe pfnet-controller CPU usage again.

Actual result
With ZTE enabled, pfnet-controller consumes approximately 167–180% CPU continuously.

Example:

root 95074 180.4 31.0 4026724 2555376 - S Tue19 209:57.37 /usr/local/bin/pfnet-controller -conf /var/etc/pfnet-controller/pfnet-controller.conf

The process is also using approximately 2.55 GB RSS.

The CPU time accumulated by the process is very high relative to its uptime.

With ZTE disabled, the same process remains running but CPU usage drops substantially:

PID   CPU   MEM     RSS
95074 21.4  31.0    2555400 KB

Therefore:

ZTE enabled  -> pfnet-controller ~167–180% CPU
ZTE disabled -> pfnet-controller ~20% CPU

No reboot is required for the CPU reduction; disabling ZTE is sufficient.

Additional observations
There are two processes using the pfnet-controller executable, but only the long-running PID 95074 is consuming significant CPU.

pgrep -alf pfnet-controller:

95010 daemon: /usr/local/bin/pfnet-controller[95074]
95074 /usr/local/bin/pfnet-controller -conf /var/etc/pfnet-controller/pfnet-controller.conf
15484 /usr/local/bin/pfnet-controller -conf /var/etc/pfnet-controller/pfnet-controller.conf

PID 95074 is the process responsible for the high CPU usage.

PF state table size does not appear excessive:

current entries     1519
searches            361936606
searches/sec        1711.5
inserts             1264237
removals            1262718

There were no state memory, state-limit, or congestion errors.

The following searches for obvious ZTE/CoreDNS-specific PF tables/rules returned no results:

pfctl -vvs rules | grep -E 'coredns_(allow|deny)'

pfctl -s Tables | grep -iE 'zero|trust|dns|egress'

These commands may simply not expose the internal ZTE implementation and are included only as diagnostic information.

Expected result

Enabling Zero Trust Egress should not cause pfnet-controller to continuously consume approximately 1.7–1.8 CPU cores on a Netgate 6100 under normal operation.

Impact
The issue consumes a significant portion of the CPU capacity of the Netgate 6100.

The pfnet-controller process also reaches a very large resident memory footprint (approximately 2.5 GB in the observed process).

Regression / correlation
The CPU increase is directly correlated with the ZTE enabled state.

This can be reproduced by toggling ZTE:

ZTE ON  -> high pfnet-controller CPU
ZTE OFF -> CPU returns near normal

This makes ZTE the primary suspected trigger.

Suspected component
pfnet-controller / Zero Trust Egress

Please investigate whether ZTE is causing excessive event processing, policy synchronization, DNS-related processing, firewall interaction, or another internal processing loop in pfnet-controller.

Additional diagnostic information
I can provide:
-top -aSH output with ZTE enabled/disabled
-ps/process information
-pfctl state information
-pfnet-controller logs
-ZTE configuration
-full system configuration
-additional reproduction/testing as requested

phil_socket95

@mr_nets said:

Summary Enabling Zero Trust Egress (ZTE) causes sustained high CPU usage by pfnet-controller on a Netgate 6100 running pfSense Plus 26.07. The issue is reproducible: disabling ZTE causes the CPU usage of the same pfnet-controller process to drop from approximately 180% to approximately 20%.

Thanks for the very complete write-up, that makes it a lot easier to route. The clean ON/OFF correlation is the important bit here, especially with the same long-running pfnet-controller process dropping immediately when ZTE is disabled. The RSS value is also worth paying attention to, 2.5 GB is not something I’d hand-wave away as “normal background work”. If you can, please open a TAC case and include the top -aSH output with ZTE enabled and disabled, pfnet-controller logs around the toggle, and a sanitized ZTE config. I’d also avoid repeatedly killing the process unless support asks, since the “no reboot required” behavior is useful evidence. In the meantime, leaving ZTE disabled is the sane workaround, unless you enjoy using the 6100 as a small space heater.

mr_nets

@phil_socket95 There is another build of Nexus, I'll test again to see if it behave the same way.

Bonus, It can be tested with IPv6 activated since the bug in CoreDNS is fixed.

phil_socket95

@mr_nets said in [ZTE] pfnet-controller consumes 170-180% CPU on Netgate 6100 with Zero Trust Egress enabled:

Summary
Enabling Zero Trust Egress (ZTE) causes sustained high CPU usage by pfnet-controller on a Netgate 6100 running pfSense Plus 26.07.

The issue is reproducible: disabling ZTE causes the CPU usage of the same pfnet-controller process to drop from approximately 180% to approximately 20%.

Environment
Hardware: Netgate 6100
pfSense Plus: 26.07
CPU: 4 cores
Feature: Zero Trust Egress (ZTE)
Affected process: /usr/local/bin/pfnet-controller
Configuration: /var/etc/pfnet-controller/pfnet-controller.conf

Steps to reproduce

Run pfSense Plus 26.07 on a Netgate 6100.

Enable Zero Trust Egress.

Allow the system to operate normally.

Observe pfnet-controller CPU usage with top or ps.

Disable Zero Trust Egress.

Observe pfnet-controller CPU usage again.

Actual result
With ZTE enabled, pfnet-controller consumes approximately 167–180% CPU continuously.

Example:
root 95074 180.4 31.0 4026724 2555376 - S Tue19 209:57.37 /usr/local/bin/pfnet-controller -conf /var/etc/pfnet-controller/pfnet-controller.conf
The process is also using approximately 2.55 GB RSS.

The CPU time accumulated by the process is very high relative to its uptime.

With ZTE disabled, the same process remains running but CPU usage drops substantially:
PID   CPU   MEM     RSS
95074 21.4  31.0    2555400 KB
Therefore:
ZTE enabled  -> pfnet-controller ~167–180% CPU
ZTE disabled -> pfnet-controller ~20% CPU
No reboot is required for the CPU reduction; disabling ZTE is sufficient.

Additional observations
There are two processes using the pfnet-controller executable, but only the long-running PID 95074 is consuming significant CPU.

pgrep -alf pfnet-controller:
95010 daemon: /usr/local/bin/pfnet-controller[95074]
95074 /usr/local/bin/pfnet-controller -conf /var/etc/pfnet-controller/pfnet-controller.conf
15484 /usr/local/bin/pfnet-controller -conf /var/etc/pfnet-controller/pfnet-controller.conf
PID 95074 is the process responsible for the high CPU usage.

PF state table size does not appear excessive:
current entries     1519
searches            361936606
searches/sec        1711.5
inserts             1264237
removals            1262718
There were no state memory, state-limit, or congestion errors.

The following searches for obvious ZTE/CoreDNS-specific PF tables/rules returned no results:
pfctl -vvs rules | grep -E 'coredns_(allow|deny)'

pfctl -s Tables | grep -iE 'zero|trust|dns|egress'
These commands may simply not expose the internal ZTE implementation and are included only as diagnostic information.

Expected result

Enabling Zero Trust Egress should not cause pfnet-controller to continuously consume approximately 1.7–1.8 CPU cores on a Netgate 6100 under normal operation.

Impact
The issue consumes a significant portion of the CPU capacity of the Netgate 6100.

The pfnet-controller process also reaches a very large resident memory footprint (approximately 2.5 GB in the observed process).

Regression / correlation
The CPU increase is directly correlated with the ZTE enabled state.

This can be reproduced by toggling ZTE:
ZTE ON  -> high pfnet-controller CPU
ZTE OFF -> CPU returns near normal
This makes ZTE the primary suspected trigger.

Suspected component
pfnet-controller / Zero Trust Egress

If there is a polling interval, retry loop, or event queue involved, comparing those counters between the enabled and disabled states could narrow it down quickly. After spending enough time looking at process behavior and firewall internals, I sometimes switch to something completely unrelated like https://cocoacasinos.de/ cocoa Casino, where slots vary by themes, reel layouts, visual effects, and different game mechanics, while live games have a very different pace and presentation. The main goal here, though, is to understand exactly what ZTE changes internally that keeps pfnet-controller consuming so much CPU.

Additional diagnostic information
I can provide:
-top -aSH output with ZTE enabled/disabled
-ps/process information
-pfctl state information
-pfnet-controller logs
-ZTE configuration
-full system configuration
-additional reproduction/testing as requested

@mr_nets Sounds good, that's the right next step. Two things I'd suggest so the comparison actually tells us something:

Run it as two separate tests rather than one. First the new Nexus build with IPv4 only and ZTE toggled on/off, same top -aSH capture as before. Only once you have that baseline flip IPv6 on. Otherwise if CPU changes you won't know whether it was the build, the IPv6 path, or the extra DNS/policy work that comes with dual-stack, and TAC will ask you to split it anyway.

Also keep an eye on RSS over time, not just the snapshot. If the 2.5 GB stays flat for an hour or two it's an expensive steady state; if it keeps climbing it looks more like a leak, and that changes what Netgate will want to look at. procstat -kk <pid> a few times while it's pegged is cheap to grab too, it shows where the threads are actually spinning and is far more useful to a developer than a percentage.

Curious how the IPv6 run goes, in the CoreDNS-bug era that combination wasn't really testable so you may be the first one to report it.

mr_nets

@phil_socket95 Thanks for the advice. IPv6 doesn't change the behavior, but I get your point; I'll simply take IPv6 out of the equation for now, as testing it is a hassle—it causes outages at times when CPU usage is high. Good point regarding RSS; I'll keep a closer eye on that.

Another thing: after running further tests, I noticed the problem occurs under two different circumstances when ZTE is enabled:

Restarting the pfnet-controller service and letting ThreatGate refresh the feeds for the first time (this takes at least 3 hours, as that is the minimum allowed interval); CPU usage spikes immediately after the refresh.
Restarting the pfnet-controller service and changing the ThreatGate refresh interval to more than 24 hours to prevent a refresh; in this case, CPU usage increases gradually over time.

I'll need to write up a new summary of all this when I have the time.

mr_nets

Retest Results and Additional Findings

I performed a full retest of the issue on a cleanly restarted pfnet-controller process.

Hardware: Netgate 6100
Software: pfSense Plus 26.07
Nexus Build Version: 2aece890cbbd20849c5e30018b8d2f100046e216
Services enabled:
CoreDNS
ThreatGate
Services disabled:
DNS Resolver (Unbound)
pfBlockerNG

Prior to beginning the retest:

pfnet-controller was restarted

ThreatGate feed refresh scheduling was configured to the minimum supported interval:

3 hours

The observed trigger event was the normal scheduled ThreatGate refresh executed through the ThreatGate interface. No manual feed refresh was used.

Before execution of the scheduled ThreatGate refresh, system utilization was:

CPU: ~21%
Memory: ~48%

pfnet-controller utilization was:

CPU: 18.7% to 53.8%
RSS: 172800 KB to 309320 KB (~169 MB to ~302 MB)

Examples:

53160 18.7 172800 1664912
53160 53.8 294252 1725912
53160 37.0 309312 1742808

At this stage the system was operating normally.

Following execution of the scheduled ThreatGate refresh, system utilization increased significantly:

CPU: ~83%
Memory: ~65%

Later:

CPU: ~78%
Memory: ~79%

Later again:

CPU: ~43%
Memory: ~79%

At the same time, pfnet-controller entered a high-resource state.

Examples:

37798 205.8 1316268 2935356
37798 217.9 2235228 3921000
37798 202.4 2237916 3921000
37798 188.8 2238076 3921000

Observed range while ZTE was enabled:

CPU: 118% to 218%
RSS: 1.26 GB to 2.24 GB

Internet connectivity became effectively unusable during this state.

ZTE was then disabled while pfnet-controller was consuming approximately:

CPU: ~189% to ~203%
RSS: ~2.2 GB

Immediate observation:

37798 203.0 2247688 3938472

No immediate CPU reduction was observed.

However, subsequent measurements showed CPU gradually returning to levels comparable to those observed before the scheduled ThreatGate refresh:

17762 164.3 2708548 4461372
17762 24.1 2708748 4461372
17762 38.5 2709060 4461372
17762 19.5 2710412 4461372

Observed range after ZTE was disabled:

CPU: 19% to 39%
RSS: ~2.58 GB

Important observation:

CPU utilization eventually returned to approximately pre-event levels after ZTE was disabled.
RSS did not return to baseline.
RSS continued increasing after CPU utilization recovered.

Before the scheduled ThreatGate refresh:

CPU: ~19% to ~54%
RSS: ~173 MB to ~309 MB

After the scheduled ThreatGate refresh with ZTE enabled:

CPU: ~118% to ~218%
RSS: ~1.26 GB to ~2.24 GB

After ZTE was disabled:

CPU: ~19% to ~39%
RSS: ~2.58 GB

Operational impact was significant:

Internet became effectively unusable.
The condition persisted long enough that further testing was not practical.
Recovery ultimately required restarting pfnet-controller.

These results suggest:

The scheduled ThreatGate refresh appears to trigger the high-resource condition.
Excessive CPU utilization is strongly correlated with ZTE being enabled.
Memory growth appears to be a separate symptom because RSS remained elevated and continued increasing after CPU utilization returned to near-normal levels.
Internet usability is severely impacted while pfnet-controller is in the high-CPU state.
Recovery required restarting pfnet-controller.

Based on this retest, the most likely area for investigation remains:

pfnet-controller
Zero Trust Egress (ZTE)
ThreatGate scheduled feed processing

Particularly any policy recalculation, synchronization activity, DNS-related processing, state processing, firewall interaction, or internal processing loops that occur after the scheduled ThreatGate refresh.

I will open a TAC case