Web GUI crashes after upgrade from 22.05 to 23.01

jjstecchino · Jan 10, 2023, 8:44 PM

@jimp Both ssh and GUI are through an ipsec site to site. Ipsec conf is IKEv2 with a mutual PSK, Phase 1 encryption is AES 256, SHA256, DH 2048 bit.
Phase 2 is an IPV4 tunnel, ESP, AES256-GCM 128bit PFS 14.

These are the same settings I was running on 22.05 and previous versions without a hitch. If I revert to 22.05 all is well.

jimp · Jan 10, 2023, 8:47 PM

Do you have AES-NI or some other crypto module enabled?

jjstecchino · Jan 10, 2023, 8:48 PM

@jimp aes-ni is enabled

jimp · Jan 10, 2023, 8:49 PM

Can you try disabling AES-NI to see if it makes a difference?

jjstecchino · Jan 10, 2023, 8:50 PM

@jimp any way to disable from the cli so I don't have to go back to 22.05, change and re-update?

jimp · Jan 10, 2023, 8:52 PM

Not easily, though you could use viconfig and find the aesni line and remove it.

It would look like one of the following:

<crypto_hardware>aesni</crypto_hardware>

or

<crypto_hardware>aesni_cryptodev</crypto_hardware>

If you delete that and reboot it will not load the aesni module.

jjstecchino · Jan 10, 2023, 9:05 PM

@jimp Disabled AES-NI,

AES-NI module is not loaded anymore:

/root: kldstat
Id Refs Address Size Name
1 21 0xffffffff80200000 39a4240 kernel
2 1 0xffffffff83ba6000 5b2878 zfs.ko
3 1 0xffffffff84159000 aab0 opensolaris.ko
4 1 0xffffffff84720000 2220 cpuctl.ko
5 1 0xffffffff84723000 3248 ichsmb.ko
6 1 0xffffffff84727000 2178 smbus.ko
7 1 0xffffffff8472a000 20e8 coretemp.ko

Still same crash.

stephenw10 · Jan 10, 2023, 9:12 PM

Can we assume this only happens when you try to access the GUI over IPSec? Or is that the only way you can test it?

stephenw10 · Jan 10, 2023, 9:16 PM

Assuming it's policy based IPSec do you have static route via LAN in place to allow that access on the remote pfSense?

jjstecchino · Jan 10, 2023, 9:18 PM

@stephenw10 this is the only way I can test it. At the moment I don't have local access to this firewall. Once I do have local access I want to try a default config and if it works add ipsec and then packages. It will be a few weeks before I can go to the other house.

stephenw10 · Jan 10, 2023, 9:20 PM

Do you have something behind it you could try accessing it via? Something you could remote desktop to maybe?

jjstecchino · Jan 10, 2023, 9:29 PM

@stephenw10 Not at the moment. maybe tomorrow I can remote to my son Macbook and try local access.

Don't know if it does matter but the problem firewall is running both ipv4 and ipv6

stephenw10 · Jan 10, 2023, 10:11 PM

The IPSec tunnel is IPv4 only though?

jjstecchino · Jan 10, 2023, 11:29 PM

@stephenw10 yes

jjstecchino · Jan 11, 2023, 1:07 PM

@stephenw10 Ok, local login to the firewall works without crashes. So it is the combination of logging in to the GUI through the ipsec vpn that is causing the problem.

Weird enough I can access webcams, ssh without issues.

The configuration is the same I was using on 22.05 without issues.

stephenw10 · Jan 11, 2023, 1:49 PM

Ok, that's good info. Our devs are looking into this I'll try to replicate it...

jjstecchino · Jan 12, 2023, 12:14 PM

@stephenw10 Were you guys able to replicate the issue?
I may be able to go to the other house this weekend. Should I try a fresh reinstall?

stephenw10 · Jan 12, 2023, 2:02 PM

I haven't replicated it yet.
Do you have SWAP enabled on that device? What size if so? Getting a full core dump from that would be useful.

Steve

jjstecchino · Jan 12, 2023, 9:30 PM

@stephenw10
/root: swapinfo
Device 1K-blocks Used Avail Capacity
/dev/ada0p3 1048576 0 1048576 0%

How do I get you a full core dump?

stephenw10 · Jan 13, 2023, 1:11 AM

If you install the debug kernel with:

[23.01-RC][root@6100.stevew.lan]/root: pkg install pfSense-kernel-debug-pfSense
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        pfSense-kernel-debug-pfSense: 23.01.b.20230106.0600 [pfSense-core]

Number of packages to be installed: 1

The process will require 709 MiB more space.
145 MiB to be downloaded.

Proceed with this action? [y/N]: y
[1/1] Fetching pfSense-kernel-debug-pfSense-23.01.b.20230106.0600.pkg: 100%  145 MiB   5.2MB/s    00:29    
Checking integrity... done (0 conflicting)
[1/1] Installing pfSense-kernel-debug-pfSense-23.01.b.20230106.0600...
[1/1] Extracting pfSense-kernel-debug-pfSense-23.01.b.20230106.0600: 100%

Then when you reboot you can select that by hitting option 6 at the boot loader menu. However if you only have remote access that could be a problem.

Steve