Web GUI crashes after upgrade from 22.05 to 23.01

[jjstecchino]

@stephenw10 @gimp

On further exploring the sysctl kern.ipc.mb_use_ext_pgs enables or disables the use of unmapped mbufs by sendfile(2) and other kernel functions.
Unmapped mbuffs can hold multiple pages of data in a single mbuff and are helpful in reducing cpu utilization https://reviews.freebsd.org/D20616.

For a NIC to use unmapped mbuffs the driver has to support this capability.
ifconfig will show NOMAP in the options flags if the driver offer unmapped mbuff support.

Checking both my firewalls, the one that faults uses the igb driver for its nic. This driver supports NOMAP.

The other firewall which doesn't fault uses the ix driver which also supports NOMAP but maybe a different implementation.

For me to fix the kernel fault it is either to disable sendfile on nginx, disable the use of unmapped mbuffs altogether or possibly disable the use of unmapped mbuffs on the nic with ifconfig -mextpg flag (did not test this).

In pfsense use scenario, the use of unmapped mbuffs may be useful to reduce load on the cpu and possibly beneficial to allow faster data move by the nic driver, so I would be reluctant to disable unmapped mbuffs altogether.

On the other hand, nginx in pfsense is not a high traffic web server as it just serves the GUI to a few users. In this scenario disabling the sendfile optimization would not make a difference and may avoid a kernel fault in few edge cases like mine.

This may be the preferred route, imo, until the issue is sorted out on freebsd 14.

Interestingly on pfsense 22.05 with freebsd 13 nginx has sendfile on, however the igb driver doesn't show the NOMAP capability flag

ifconfig igb1
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: LAN
        options=e120bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>

whereas in 23.01 the NOMAP capability is supported by the nic driver:

ifconfig igb1
igb1: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	description: LAN
	options=4e120bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>

Hope this helps reproducing the kernel fault. This is as far as my ability allows me to help. I wouldn't know how to submit bugs to the freebsd team.

I did not investigate the impact that the use of using ipsec may have but surely it is part of the problem as the fault happened only when accessing the pfsense GUI through the ipsec vpn, but didn't occur when accessing the GUI locally.

This bug may be difficult to reproduce as it may involve the interplay of a specific nic driver, ipsec and kernel optimizations which may be automatically enabled or disabled on specific hardware.

I wish I would have been able to get a full kernel core dump but even using the debug kernel package I get only the textdump in /var/crash.

[stephenw10]

Mmm, yeah, a number of moving parts there. Hard to see why ipsec would trigger it...

I'll wait to see what out developers think. At least there's relatively simple workaround with that sysctl though.

Steve

[jjstecchino]

@stephenw10
Yeah but that disables a kernel wide optimization that may be important to allow better handling of network traffic by the firewall. Turning off the sendfile optimization on nginx may be a better option as what it does is allows direct move of a file data to a tcp socket without copying to a memory buffer first. This is important for a high traffic web server but overall irrelevant for pfsense.

As more people will start to use pfsense 23.xx with freebsd 14 this bug may start to affect others as well.

Setting sysctl kern.ipc.mb_use_ext=0 would allow seamlessly updates if sendfile remains set to on on nginx config but it would turn off an important kernel optimization.

I would respectfully suggest to consider turning off sendfile in nginx config instead.

[jimp]

I do have one box here with igb and NOMAP showing (A Netgate 7551), but so far I haven't been able to make it crash.

That said, the only IPsec tunnel I have on there that is testable without some work is VTI, not tunnel mode.

I'll see if I can rig up a tunnel mode test on there.

[jimp]

Setup a tunnel and still no crash. I can reach the GUI LAN to LAN with a full browser and it appears to be working fine.

Do you have something enabled on the dashboard that might be contributing? Maybe the picture widget with a large image?

Usually the web server wouldn't be using sendfile for much on pfSense since it doesn't have many static things to serve and typically that gets kicked in for stuff like large pictures.

[jjstecchino]

@jimp No, this happened also with a bare bone default config no widgets. Clean install and ipsec tunnel vpn

[jimp]

Curious. I even tried downloading a status output and some config backups with RRD (~4MB) but it keeps chugging along.

I tried with no crypto acceleration and also with QAT enabled.

There may be something specific to that exactl igb card that is different than mine.

[jjstecchino]

@jimp my nic is <Intel(R) I211 (Copper)> port 0xd000-0xd01f mem 0xf7200000-0xf721ffff,0xf7220000-0xf7223fff at device 0.0 on pci2

[jimp]

Yeah, that's quite a bit different from this one.

igb0@pci0:0:20:0:       class=0x020000 rev=0x03 hdr=0x00 vendor=0x8086 device=0x1f41 subvendor=0x8086 subdevice=0x1f41
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Connection I354'
    class      = network
    subclass   = ethernet
    bar   [10] = type Memory, range 64, base 0xdfe60000, size 131072, enabled
    bar   [18] = type I/O Port, range 32, base 0xf0c0, size 32, enabled
    bar   [20] = type Memory, range 64, base 0xdff2c000, size 16384, enabled

I thought I had something around with an i211, but nope. I have some i210 devices but they aren't running pfSense.

[jjstecchino]

@jimp On my setup hardware acceleration (only ads-ni available) on or off doesn't make a difference.

Nginx sendfile in my case seems to be the culprit as if I set it to off, it solves the problem.

[jimp]

Right, I just tested that in case it was relevant since it seems to be sendfile in some combination with IPsec and your hardware since it works locally.

[jjstecchino]

@jimp Can it be related to this bug ?

[jimp]

It's possible, though hard to say for sure. It seems similar at least.

[Dyk Evans]

@jimp

I am having this exact same issue… just upgraded both my home and remote firewalls to the RC and now when I try to access either web gui over IPsec it immediately crashes the remote side.

[jjstecchino]

@dyk-dike
Would you mind posting the output of dmesg on the crashing firewall? I would like to compare your hardware to mine to see if there are any common threads that may help sorting out or reproducing the issue.
For the time being I patched the problem by disabling sendfile on nginx in the remote firewall

[jimp]

Also it would help to have the full textdump archive from any firewall that encounters this, will make getting the details and comparing easier.

If you are on 23.01 and can easily reproduce it, you may also want to install and boot from the debug kernel and try to trigger the crash, which will include a lot more detail in the backtrace.

And then you can use the System Patches package package to disable sendfile:

diff --git a/src/etc/inc/system.inc b/src/etc/inc/system.inc
index d36efc2fca..b7cda99366 100644
--- a/src/etc/inc/system.inc
+++ b/src/etc/inc/system.inc
@@ -1380,7 +1380,7 @@ http {
        add_header X-Frame-Options SAMEORIGIN;
        server_tokens off;
 
-       sendfile        on;
+       sendfile        off;
 
        access_log      syslog:server=unix:/var/run/log,facility=local5 combined;

After applying that patch, use the console menu option to restart the GUI (11).

[Dyk Evans]

@jjstecchino
is this how I disable sendfile
kern.ipc.mb_use_ext_pgs=0

[Dyk Evans]

This post is deleted!

[jjstecchino]

@dyk-dike
No to disable sendfile you do what Jimp just said above, either by using the system patches package with that diff or by manually editing /etc/inc/system.inc.

That sysctl disable the use of unmapped buffers (mbuffs). While setting that sysctl to 0 would solve the problem, by disabling mbuffs altogether it MAY slow down your firewall as well. Not sure about this last statement as I haven't done any speed testing with kern.ipc.mb_use_ext_pgs=0.

I would personally do what jimp suggested.

Please post the crash dump (textdump.zip) so that we can compare my hardware to yours.

Thanks

[Dyk Evans]

@jjstecchino @jimp

dumps from both servers.

server1.txt

error dump.txt