Web GUI crashes after upgrade from 22.05 to 23.01

[stephenw10]

Mmm, that's identical. Still crashes in the same place? When you try to access the GUI after upgrade?

[stephenw10]

Also do you have the IPSec widget on the dashboard of the firewall you're trying to access?

And is the local pfSense also running 23.01?

[jjstecchino]

@stephenw10 Local pfsense is running 23.01b. Upgraded the remote pfsense to 23.01r and as I try to navigate to the web guy still crashes immediately. it seems the same crash as 23.01b. I am not sure if the ipsec widget is on the dashboard. I'll try to boot to 22.05 and check.

[jjstecchino]

@stephenw10 Ok... I rebooted to 22.05, deleted all packages and all widgets from the dashboard except for traffic graphs and system informations. Re updated to 23.01, update went without error, but again as I launch the web guy the system crashes.

[stephenw10]

Hmm, same backtrace again?

[jjstecchino]

@stephenw10 Yep!
textdump.tar

[jjstecchino]

Is it nginx crashing?

[jimp]

Do you reach the GUI over an IPsec VPN? If it's over IPsec, what sort of IPsec? Mobile? Site to Site? What type of config?

When you connect to SSH, is that also across IPsec, or is it direct?

The crash appears to be during a memory operation while handling a packet from nginx across IPsec.

Though I'm not aware of anything like that happening to anyone else.

If the crashes were not nearly identical, I'd suspect hardware.

[jjstecchino]

@jimp Both ssh and GUI are through an ipsec site to site. Ipsec conf is IKEv2 with a mutual PSK, Phase 1 encryption is AES 256, SHA256, DH 2048 bit.
Phase 2 is an IPV4 tunnel, ESP, AES256-GCM 128bit PFS 14.

These are the same settings I was running on 22.05 and previous versions without a hitch. If I revert to 22.05 all is well.

[jimp]

Do you have AES-NI or some other crypto module enabled?

[jjstecchino]

@jimp aes-ni is enabled

[jimp]

Can you try disabling AES-NI to see if it makes a difference?

[jjstecchino]

@jimp any way to disable from the cli so I don't have to go back to 22.05, change and re-update?

[jimp]

Not easily, though you could use viconfig and find the aesni line and remove it.

It would look like one of the following:

<crypto_hardware>aesni</crypto_hardware>

or

<crypto_hardware>aesni_cryptodev</crypto_hardware>

If you delete that and reboot it will not load the aesni module.

[jjstecchino]

@jimp Disabled AES-NI,

AES-NI module is not loaded anymore:

/root: kldstat
Id Refs Address Size Name
1 21 0xffffffff80200000 39a4240 kernel
2 1 0xffffffff83ba6000 5b2878 zfs.ko
3 1 0xffffffff84159000 aab0 opensolaris.ko
4 1 0xffffffff84720000 2220 cpuctl.ko
5 1 0xffffffff84723000 3248 ichsmb.ko
6 1 0xffffffff84727000 2178 smbus.ko
7 1 0xffffffff8472a000 20e8 coretemp.ko

Still same crash.

[stephenw10]

Can we assume this only happens when you try to access the GUI over IPSec? Or is that the only way you can test it?

[stephenw10]

Assuming it's policy based IPSec do you have static route via LAN in place to allow that access on the remote pfSense?

[jjstecchino]

@stephenw10 this is the only way I can test it. At the moment I don't have local access to this firewall. Once I do have local access I want to try a default config and if it works add ipsec and then packages. It will be a few weeks before I can go to the other house.

[stephenw10]

Do you have something behind it you could try accessing it via? Something you could remote desktop to maybe?

[jjstecchino]

@stephenw10 Not at the moment. maybe tomorrow I can remote to my son Macbook and try local access.

Don't know if it does matter but the problem firewall is running both ipv4 and ipv6