Web GUI crashes after upgrade from 22.05 to 23.01

[stephenw10]

Backtrace:

Tracing pid 3765 tid 100406 td 0xfffffe00c65a4900
kdb_enter() at kdb_enter+0x32/frame 0xfffffe00c3d6f320
vpanic() at vpanic+0x182/frame 0xfffffe00c3d6f370
panic() at panic+0x43/frame 0xfffffe00c3d6f3d0
trap_fatal() at trap_fatal+0x409/frame 0xfffffe00c3d6f430
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00c3d6f490
calltrap() at calltrap+0x8/frame 0xfffffe00c3d6f490
--- trap 0xc, rip = 0xffffffff813187ba, rsp = 0xfffffe00c3d6f560, rbp = 0xfffffe00c3d6f560 ---
memcpy_erms() at memcpy_erms+0x10a/frame 0xfffffe00c3d6f560
m_unshare() at m_unshare+0x3de/frame 0xfffffe00c3d6f5e0
esp_output() at esp_output+0x186/frame 0xfffffe00c3d6f6d0
ipsec4_perform_request() at ipsec4_perform_request+0x1d2/frame 0xfffffe00c3d6f760
ipsec4_common_output() at ipsec4_common_output+0xa2/frame 0xfffffe00c3d6f7a0
ip_output() at ip_output+0x99d/frame 0xfffffe00c3d6f8a0
tcp_default_output() at tcp_default_output+0x1d2b/frame 0xfffffe00c3d6fa70
tcp_usr_ready() at tcp_usr_ready+0x1a1/frame 0xfffffe00c3d6fad0
sendfile_iodone() at sendfile_iodone+0x11c/frame 0xfffffe00c3d6fb10
vn_sendfile() at vn_sendfile+0x1663/frame 0xfffffe00c3d6fd70
sys_sendfile() at sys_sendfile+0xf7/frame 0xfffffe00c3d6fe00
amd64_syscall() at amd64_syscall+0x10c/frame 0xfffffe00c3d6ff30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00c3d6ff30
--- syscall (393, FreeBSD ELF64, sys_sendfile), rip = 0x8254b84ba, rsp = 0x8209bed68, rbp = 0x8209bf640 ---

Panic:

[fib_algo] inet.0 (bsearch4#29) rebuild_fd_flm: switching algo to radix4_lockless


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address	= 0x0
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff813187ba
stack pointer	        = 0x28:0xfffffe00c3d6f560
frame pointer	        = 0x28:0xfffffe00c3d6f560
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 3765 (nginx)
rdi: fffff8005688c113 rsi:                0 rdx:              420
rcx:              420  r8:                1  r9:                9
rax: fffff8005688c113 rbx: fffff800273c3e00 rbp: fffffe00c3d6f560
r10:         560c7f02 r11:               66 r12: fffff80056552d00
r13:                0 r14: fffff80027599a00 r15:                1
trap number		= 12
panic: page fault
cpuid = 0
time = 1673054377
KDB: enter: panic

Hmm, interesting backtrace there. Seems familiar....

[jjstecchino]

@stephenw10 Familiar? What is fib_algo?

[jjstecchino]

@stephenw10 @gimp Ok I tried updating again to 23.01 RC and still crashes

Here is the crash dump:
textdump.tar.0

Glad to help troubleshooting this

[stephenw10]

Mmm, that's identical. Still crashes in the same place? When you try to access the GUI after upgrade?

[stephenw10]

Also do you have the IPSec widget on the dashboard of the firewall you're trying to access?

And is the local pfSense also running 23.01?

[jjstecchino]

@stephenw10 Local pfsense is running 23.01b. Upgraded the remote pfsense to 23.01r and as I try to navigate to the web guy still crashes immediately. it seems the same crash as 23.01b. I am not sure if the ipsec widget is on the dashboard. I'll try to boot to 22.05 and check.

[jjstecchino]

@stephenw10 Ok... I rebooted to 22.05, deleted all packages and all widgets from the dashboard except for traffic graphs and system informations. Re updated to 23.01, update went without error, but again as I launch the web guy the system crashes.

[stephenw10]

Hmm, same backtrace again?

[jjstecchino]

@stephenw10 Yep!
textdump.tar

[jjstecchino]

Is it nginx crashing?

[jimp]

Do you reach the GUI over an IPsec VPN? If it's over IPsec, what sort of IPsec? Mobile? Site to Site? What type of config?

When you connect to SSH, is that also across IPsec, or is it direct?

The crash appears to be during a memory operation while handling a packet from nginx across IPsec.

Though I'm not aware of anything like that happening to anyone else.

If the crashes were not nearly identical, I'd suspect hardware.

[jjstecchino]

@jimp Both ssh and GUI are through an ipsec site to site. Ipsec conf is IKEv2 with a mutual PSK, Phase 1 encryption is AES 256, SHA256, DH 2048 bit.
Phase 2 is an IPV4 tunnel, ESP, AES256-GCM 128bit PFS 14.

These are the same settings I was running on 22.05 and previous versions without a hitch. If I revert to 22.05 all is well.

[jimp]

Do you have AES-NI or some other crypto module enabled?

[jjstecchino]

@jimp aes-ni is enabled

[jimp]

Can you try disabling AES-NI to see if it makes a difference?

[jjstecchino]

@jimp any way to disable from the cli so I don't have to go back to 22.05, change and re-update?

[jimp]

Not easily, though you could use viconfig and find the aesni line and remove it.

It would look like one of the following:

<crypto_hardware>aesni</crypto_hardware>

or

<crypto_hardware>aesni_cryptodev</crypto_hardware>

If you delete that and reboot it will not load the aesni module.

[jjstecchino]

@jimp Disabled AES-NI,

AES-NI module is not loaded anymore:

/root: kldstat
Id Refs Address Size Name
1 21 0xffffffff80200000 39a4240 kernel
2 1 0xffffffff83ba6000 5b2878 zfs.ko
3 1 0xffffffff84159000 aab0 opensolaris.ko
4 1 0xffffffff84720000 2220 cpuctl.ko
5 1 0xffffffff84723000 3248 ichsmb.ko
6 1 0xffffffff84727000 2178 smbus.ko
7 1 0xffffffff8472a000 20e8 coretemp.ko

Still same crash.

[stephenw10]

Can we assume this only happens when you try to access the GUI over IPSec? Or is that the only way you can test it?

[stephenw10]

Assuming it's policy based IPSec do you have static route via LAN in place to allow that access on the remote pfSense?