Bare metal 2.6 / 22.01 / 22.05 performance issues with high-end hardware

[q54e3w]

I attempted to upgrade my bare metal firewall from 2.5.2 to 2.6 today and suffered poor performance with both pfSense web page timeouts and high latencies. Also exhibited the same issues with 22.01 and 22.05.
Searching here I see issues related to running pfsense as a virtualised system, but nothing related to bare metal installs like mine.

Motherboard: Supermicro X11SDC-8C-TP8F
CPU: Xeon D-2146NT CPU @ 2.30GHz, 8 CPU, AES-NI
RAM: 16GB
Onboard NICs: Intel i350 (41Gb) & X772 (410Gb)
Additional NIC: Intel X710T2 (2*2.5/5/10Gb)
Disk: 400GB mirrored Intel S3710

WAN is 1Gb/1Gb fibre, typical latency of 2ms. On 2.6 this increased to somewhere in the region of 300-1000ms causing gateways to go offline due to latency & drop-outs.
pfctl and php-fpm were pegged at 75-95% causing system to report utilisation load averages of ~75% where normally its 1% or 2%.

I uninstalled all packages (pfBlocker-dev, perf, nox11, nut, openvpn-client-export, watchdog, traffic totals) which reduced the load but not significantly.

I disabled the 3 wireguard tunnels and 5 OpenVPN connections which reduced the load but still not significantly

I disabled IPv6 and removed associated gateways with didn't have an impact.

I have ~10 VLANs segmenting various traffic types

Ultimately I had to roll back to 2.5.2 which functionals correctly but am curious if there are any known issues with this hardware configuration.

thanks in a advance for any pointers

[Cool_Corona]

IMHO 2.6 is nowhere near production ready.

To avoid all these HW issues I always run PF virtualized.

I dont care about the minor performance hit compared to all the convenience of running it on ESXi.

And yes...I am still on 2.5.2.

Changing to OPNsense soon since the bugfixes are a lot faster.

[SteveITS]

@q54e3w There is a CPU usage issue with 22.05 and pfBlocker (https://forum.netgate.com/topic/173072/high-cpu-usage-after-upgrading-to-22-05/16) but that shouldn't affect 2.5.x or 2.6/22.01.

There is a patch in the System Patches package "Disable pf counter data preservation to temporarily work around latency when reloading large rulesets (Redmine #12827)"

FWIW we have one router on 2.6 on fairly old hardware and are not having issues.

[stephenw10]

@Cool_Corona Please contribute constructively. Thanks.