Web GUI crashes after upgrade from 22.05 to 23.01
-
Backtrace:
Tracing pid 3765 tid 100406 td 0xfffffe00c65a4900 kdb_enter() at kdb_enter+0x32/frame 0xfffffe00c3d6f320 vpanic() at vpanic+0x182/frame 0xfffffe00c3d6f370 panic() at panic+0x43/frame 0xfffffe00c3d6f3d0 trap_fatal() at trap_fatal+0x409/frame 0xfffffe00c3d6f430 trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00c3d6f490 calltrap() at calltrap+0x8/frame 0xfffffe00c3d6f490 --- trap 0xc, rip = 0xffffffff813187ba, rsp = 0xfffffe00c3d6f560, rbp = 0xfffffe00c3d6f560 --- memcpy_erms() at memcpy_erms+0x10a/frame 0xfffffe00c3d6f560 m_unshare() at m_unshare+0x3de/frame 0xfffffe00c3d6f5e0 esp_output() at esp_output+0x186/frame 0xfffffe00c3d6f6d0 ipsec4_perform_request() at ipsec4_perform_request+0x1d2/frame 0xfffffe00c3d6f760 ipsec4_common_output() at ipsec4_common_output+0xa2/frame 0xfffffe00c3d6f7a0 ip_output() at ip_output+0x99d/frame 0xfffffe00c3d6f8a0 tcp_default_output() at tcp_default_output+0x1d2b/frame 0xfffffe00c3d6fa70 tcp_usr_ready() at tcp_usr_ready+0x1a1/frame 0xfffffe00c3d6fad0 sendfile_iodone() at sendfile_iodone+0x11c/frame 0xfffffe00c3d6fb10 vn_sendfile() at vn_sendfile+0x1663/frame 0xfffffe00c3d6fd70 sys_sendfile() at sys_sendfile+0xf7/frame 0xfffffe00c3d6fe00 amd64_syscall() at amd64_syscall+0x10c/frame 0xfffffe00c3d6ff30 fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00c3d6ff30 --- syscall (393, FreeBSD ELF64, sys_sendfile), rip = 0x8254b84ba, rsp = 0x8209bed68, rbp = 0x8209bf640 ---Panic:
[fib_algo] inet.0 (bsearch4#29) rebuild_fd_flm: switching algo to radix4_lockless Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x0 fault code = supervisor read data, page not present instruction pointer = 0x20:0xffffffff813187ba stack pointer = 0x28:0xfffffe00c3d6f560 frame pointer = 0x28:0xfffffe00c3d6f560 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 3765 (nginx) rdi: fffff8005688c113 rsi: 0 rdx: 420 rcx: 420 r8: 1 r9: 9 rax: fffff8005688c113 rbx: fffff800273c3e00 rbp: fffffe00c3d6f560 r10: 560c7f02 r11: 66 r12: fffff80056552d00 r13: 0 r14: fffff80027599a00 r15: 1 trap number = 12 panic: page fault cpuid = 0 time = 1673054377 KDB: enter: panicHmm, interesting backtrace there. Seems familiar....
-
@stephenw10 Familiar? What is fib_algo?
-
@stephenw10 @gimp Ok I tried updating again to 23.01 RC and still crashes
Here is the crash dump:
textdump.tar.0Glad to help troubleshooting this
-
Mmm, that's identical. Still crashes in the same place? When you try to access the GUI after upgrade?
-
Also do you have the IPSec widget on the dashboard of the firewall you're trying to access?
And is the local pfSense also running 23.01?
-
@stephenw10 Local pfsense is running 23.01b. Upgraded the remote pfsense to 23.01r and as I try to navigate to the web guy still crashes immediately. it seems the same crash as 23.01b. I am not sure if the ipsec widget is on the dashboard. I'll try to boot to 22.05 and check.
-
@stephenw10 Ok... I rebooted to 22.05, deleted all packages and all widgets from the dashboard except for traffic graphs and system informations. Re updated to 23.01, update went without error, but again as I launch the web guy the system crashes.
-
Hmm, same backtrace again?
-
@stephenw10 Yep!
textdump.tar -
Is it nginx crashing?
-
Do you reach the GUI over an IPsec VPN? If it's over IPsec, what sort of IPsec? Mobile? Site to Site? What type of config?
When you connect to SSH, is that also across IPsec, or is it direct?
The crash appears to be during a memory operation while handling a packet from nginx across IPsec.
Though I'm not aware of anything like that happening to anyone else.
If the crashes were not nearly identical, I'd suspect hardware.
-
@jimp Both ssh and GUI are through an ipsec site to site. Ipsec conf is IKEv2 with a mutual PSK, Phase 1 encryption is AES 256, SHA256, DH 2048 bit.
Phase 2 is an IPV4 tunnel, ESP, AES256-GCM 128bit PFS 14.These are the same settings I was running on 22.05 and previous versions without a hitch. If I revert to 22.05 all is well.
-
Do you have AES-NI or some other crypto module enabled?
-
@jimp aes-ni is enabled
-
Can you try disabling AES-NI to see if it makes a difference?
-
@jimp any way to disable from the cli so I don't have to go back to 22.05, change and re-update?
-
Not easily, though you could use
viconfigand find theaesniline and remove it.It would look like one of the following:
<crypto_hardware>aesni</crypto_hardware>or
<crypto_hardware>aesni_cryptodev</crypto_hardware>If you delete that and reboot it will not load the aesni module.
-
@jimp Disabled AES-NI,
AES-NI module is not loaded anymore:
/root: kldstat
Id Refs Address Size Name
1 21 0xffffffff80200000 39a4240 kernel
2 1 0xffffffff83ba6000 5b2878 zfs.ko
3 1 0xffffffff84159000 aab0 opensolaris.ko
4 1 0xffffffff84720000 2220 cpuctl.ko
5 1 0xffffffff84723000 3248 ichsmb.ko
6 1 0xffffffff84727000 2178 smbus.ko
7 1 0xffffffff8472a000 20e8 coretemp.koStill same crash.
-
Can we assume this only happens when you try to access the GUI over IPSec? Or is that the only way you can test it?
-
Assuming it's policy based IPSec do you have static route via LAN in place to allow that access on the remote pfSense?
