Web GUI crashes after upgrade from 22.05 to 23.01

[jjstecchino]

I am in a pickle! I upgrade an offsite pfSense+ at my second home. pfSense at my primary home and pfSense offsite are connected through ipsec.

After I upgraded the offsite pfSense from 22.05 to 23.01, the firewall comes up and the ipsec connection comes up. I can ssh to the remote firewall with no problem but as I try to connect to the remote web GUI the ipsec connection goes down and I can't get to the GUI.

If I ping the remote firewall and try to connect to the GUI, I got no ping responses for about 70 sec and then the connection is up again. The remote pfSense takes about 70 sec to reboot, so I suspect trying to connect to the remote pfSense GUI is causing a reboot.

I checked the logs and there is nothing strikingly wrong in dmesg.boot or system.log.

Are there any other logs other than /var/log that I can look at to figure out what is going on and a possible fix?

How do I boot to a different boot environment from the cli?

This is going to be a challenge for me. Any help appreciated!

[jimp]

What hardware is this on?

If you have local console access you can easily choose a different boot environment from the loader menu as the device boots. There are ways to set the BE via bectl at the command line as well, but we don't have any docs on that yet. There are some examples around the forum though if you search.

If you can get in over SSH as you say, you should scp off the crash dump files (if there are any) from /var/crash/ and check them out. If it is hitting a panic, they should be there, unless the device is one that doesn't have any swap space. If you have them, you can post at least the ddb.txt and msgbuf.txt from one of the crash dump archives and we can look into what caused it.

If you don't have any crash dump files, it's probably outputting the crash dump data to the console so you'd have to setup something to monitor the console and log its output.

[jjstecchino]

@jimp Thanks for the reply. Unfortunately I don't have access to the console, only ssh since the firewall is at a remote location. the hardware is a a plain i5 with 8gb of ram. bios is efi and zfs on a 256gb ssd.

I'll look into belt to boot to a previous boot environment.

Since I don't have crash logs, is there a way to enable a swap mount point from ssh?

[jjstecchino]

@jimp I found the crash files after downgrading to 22.05.

here is the latest dump:
config.txt version.txt panic.txt msgbuf.txt ddb.txt

Thanks

[stephenw10]

Backtrace:

Tracing pid 3765 tid 100406 td 0xfffffe00c65a4900
kdb_enter() at kdb_enter+0x32/frame 0xfffffe00c3d6f320
vpanic() at vpanic+0x182/frame 0xfffffe00c3d6f370
panic() at panic+0x43/frame 0xfffffe00c3d6f3d0
trap_fatal() at trap_fatal+0x409/frame 0xfffffe00c3d6f430
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00c3d6f490
calltrap() at calltrap+0x8/frame 0xfffffe00c3d6f490
--- trap 0xc, rip = 0xffffffff813187ba, rsp = 0xfffffe00c3d6f560, rbp = 0xfffffe00c3d6f560 ---
memcpy_erms() at memcpy_erms+0x10a/frame 0xfffffe00c3d6f560
m_unshare() at m_unshare+0x3de/frame 0xfffffe00c3d6f5e0
esp_output() at esp_output+0x186/frame 0xfffffe00c3d6f6d0
ipsec4_perform_request() at ipsec4_perform_request+0x1d2/frame 0xfffffe00c3d6f760
ipsec4_common_output() at ipsec4_common_output+0xa2/frame 0xfffffe00c3d6f7a0
ip_output() at ip_output+0x99d/frame 0xfffffe00c3d6f8a0
tcp_default_output() at tcp_default_output+0x1d2b/frame 0xfffffe00c3d6fa70
tcp_usr_ready() at tcp_usr_ready+0x1a1/frame 0xfffffe00c3d6fad0
sendfile_iodone() at sendfile_iodone+0x11c/frame 0xfffffe00c3d6fb10
vn_sendfile() at vn_sendfile+0x1663/frame 0xfffffe00c3d6fd70
sys_sendfile() at sys_sendfile+0xf7/frame 0xfffffe00c3d6fe00
amd64_syscall() at amd64_syscall+0x10c/frame 0xfffffe00c3d6ff30
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00c3d6ff30
--- syscall (393, FreeBSD ELF64, sys_sendfile), rip = 0x8254b84ba, rsp = 0x8209bed68, rbp = 0x8209bf640 ---

Panic:

[fib_algo] inet.0 (bsearch4#29) rebuild_fd_flm: switching algo to radix4_lockless


Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address	= 0x0
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff813187ba
stack pointer	        = 0x28:0xfffffe00c3d6f560
frame pointer	        = 0x28:0xfffffe00c3d6f560
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 3765 (nginx)
rdi: fffff8005688c113 rsi:                0 rdx:              420
rcx:              420  r8:                1  r9:                9
rax: fffff8005688c113 rbx: fffff800273c3e00 rbp: fffffe00c3d6f560
r10:         560c7f02 r11:               66 r12: fffff80056552d00
r13:                0 r14: fffff80027599a00 r15:                1
trap number		= 12
panic: page fault
cpuid = 0
time = 1673054377
KDB: enter: panic

Hmm, interesting backtrace there. Seems familiar....

[jjstecchino]

@stephenw10 Familiar? What is fib_algo?

[jjstecchino]

@stephenw10 @gimp Ok I tried updating again to 23.01 RC and still crashes

Here is the crash dump:
textdump.tar.0

Glad to help troubleshooting this

[stephenw10]

Mmm, that's identical. Still crashes in the same place? When you try to access the GUI after upgrade?

[stephenw10]

Also do you have the IPSec widget on the dashboard of the firewall you're trying to access?

And is the local pfSense also running 23.01?

[jjstecchino]

@stephenw10 Local pfsense is running 23.01b. Upgraded the remote pfsense to 23.01r and as I try to navigate to the web guy still crashes immediately. it seems the same crash as 23.01b. I am not sure if the ipsec widget is on the dashboard. I'll try to boot to 22.05 and check.

[jjstecchino]

@stephenw10 Ok... I rebooted to 22.05, deleted all packages and all widgets from the dashboard except for traffic graphs and system informations. Re updated to 23.01, update went without error, but again as I launch the web guy the system crashes.

[stephenw10]

Hmm, same backtrace again?

[jjstecchino]

@stephenw10 Yep!
textdump.tar

[jjstecchino]

Is it nginx crashing?

[jimp]

Do you reach the GUI over an IPsec VPN? If it's over IPsec, what sort of IPsec? Mobile? Site to Site? What type of config?

When you connect to SSH, is that also across IPsec, or is it direct?

The crash appears to be during a memory operation while handling a packet from nginx across IPsec.

Though I'm not aware of anything like that happening to anyone else.

If the crashes were not nearly identical, I'd suspect hardware.

[jjstecchino]

@jimp Both ssh and GUI are through an ipsec site to site. Ipsec conf is IKEv2 with a mutual PSK, Phase 1 encryption is AES 256, SHA256, DH 2048 bit.
Phase 2 is an IPV4 tunnel, ESP, AES256-GCM 128bit PFS 14.

These are the same settings I was running on 22.05 and previous versions without a hitch. If I revert to 22.05 all is well.

[jimp]

Do you have AES-NI or some other crypto module enabled?

[jjstecchino]

@jimp aes-ni is enabled

[jimp]

Can you try disabling AES-NI to see if it makes a difference?

[jjstecchino]

@jimp any way to disable from the cli so I don't have to go back to 22.05, change and re-update?