Web GUI crashes after upgrade from 22.05 to 23.01
-
Hmm, same backtrace again?
-
@stephenw10 Yep!
textdump.tar -
Is it nginx crashing?
-
Do you reach the GUI over an IPsec VPN? If it's over IPsec, what sort of IPsec? Mobile? Site to Site? What type of config?
When you connect to SSH, is that also across IPsec, or is it direct?
The crash appears to be during a memory operation while handling a packet from nginx across IPsec.
Though I'm not aware of anything like that happening to anyone else.
If the crashes were not nearly identical, I'd suspect hardware.
-
@jimp Both ssh and GUI are through an ipsec site to site. Ipsec conf is IKEv2 with a mutual PSK, Phase 1 encryption is AES 256, SHA256, DH 2048 bit.
Phase 2 is an IPV4 tunnel, ESP, AES256-GCM 128bit PFS 14.These are the same settings I was running on 22.05 and previous versions without a hitch. If I revert to 22.05 all is well.
-
Do you have AES-NI or some other crypto module enabled?
-
@jimp aes-ni is enabled
-
Can you try disabling AES-NI to see if it makes a difference?
-
@jimp any way to disable from the cli so I don't have to go back to 22.05, change and re-update?
-
Not easily, though you could use
viconfigand find theaesniline and remove it.It would look like one of the following:
<crypto_hardware>aesni</crypto_hardware>or
<crypto_hardware>aesni_cryptodev</crypto_hardware>If you delete that and reboot it will not load the aesni module.
-
@jimp Disabled AES-NI,
AES-NI module is not loaded anymore:
/root: kldstat
Id Refs Address Size Name
1 21 0xffffffff80200000 39a4240 kernel
2 1 0xffffffff83ba6000 5b2878 zfs.ko
3 1 0xffffffff84159000 aab0 opensolaris.ko
4 1 0xffffffff84720000 2220 cpuctl.ko
5 1 0xffffffff84723000 3248 ichsmb.ko
6 1 0xffffffff84727000 2178 smbus.ko
7 1 0xffffffff8472a000 20e8 coretemp.koStill same crash.
-
Can we assume this only happens when you try to access the GUI over IPSec? Or is that the only way you can test it?
-
Assuming it's policy based IPSec do you have static route via LAN in place to allow that access on the remote pfSense?
-
@stephenw10 this is the only way I can test it. At the moment I don't have local access to this firewall. Once I do have local access I want to try a default config and if it works add ipsec and then packages. It will be a few weeks before I can go to the other house.
-
Do you have something behind it you could try accessing it via? Something you could remote desktop to maybe?
-
@stephenw10 Not at the moment. maybe tomorrow I can remote to my son Macbook and try local access.
Don't know if it does matter but the problem firewall is running both ipv4 and ipv6
-
The IPSec tunnel is IPv4 only though?
-
@stephenw10 yes
-
@stephenw10 Ok, local login to the firewall works without crashes. So it is the combination of logging in to the GUI through the ipsec vpn that is causing the problem.
Weird enough I can access webcams, ssh without issues.
The configuration is the same I was using on 22.05 without issues.
-
Ok, that's good info. Our devs are looking into this I'll try to replicate it...
