24.03 install failed in 1 out of 3

maverickws

@stephenw10
they had 1.48GB RAM assigned, I'm changing to 2

maverickws

@stephenw10
Ok so I upped the RAM to 4GB on the secondary, proceeded with install from 23.09.1 ti 24.03.1

The same error, Fatal trap 12. Can't be RAM or 4 had to be enough

stephenw10

Ok, can you add a serial port that VM and run the upgrade from the serial console directly? That should give you the full log.

maverickws

@stephenw10 well I can connect to the regular console and do the upgrade, its one of the cli menu options or any special command you'd recommend?

stephenw10

You can't log the output from a VGA console, at least not easily. If you can do that from the serial console you can review the full process.

From the console command line though run: pfSense-upgrade -d

maverickws

Hi @stephenw10

I'm very sorry for coming back to this old topic (well, kinda still within 6 months ) but the firewalls are both in the same state - 23.09.1
Back then I had other projects that came up and completely relegated this as everything was working properly.

Today I got back to the task and upgrade our firewalls. Made all the backups and routines and when I'm going to upgrade I get this message:

Your device has not been registered for pfSense+. Please purchase a pfSense+ subscription to receive future updates.

I have also read that some changes (like removing cards) or others may cause this to happen. They are VM's. How can I fix this? I am unable to select any update branch, its just empty. Thanks

stephenw10

Yup, sounds like your NDI(s) may have changed. Send it to me in chat and I can check them.

maverickws

Hello again, coming back to provide an update on this situation:

So far I have been totally unable to upgrade these pfSense VMs to the latest 24.03.

After @stephenw10 kindly reviewed the NDI's, the update branches appeared again.

So I made two different backups: took a VM snapshot, and simply backed up the configuration.

First attempt was the upgrade from existing pfSense VM to 24.03 following the normal update process.
Failed:

The same error as before.

Then I moved on to making a clean install. I first tried netgate-installer-v1.0-RC-amd64-20240919-1435.iso, image downloaded from Netgate.

This installer first asks you to select the WAN interface - then its configuration, then the LAN interface, and its configuration.
So I configured WAN as static all settings correct upstream gateway set.
From here onwards always fails.
Since the only odd setting here is the "use local resolver", I tried different configurations:

use local resolver: true
This option shows the first issue when after selecting "local resolver: true" you are forced to add an IP address to the resolver address.
I added 127.0.0.1, failed

use local resolver: false
so I did two tests with this setting, one I configured my hosting name resolver, second I configured cloud flare's DNS.

In any of these attempts, the result is always one and the same:

However, if I exit the installer and go to the command prompt, the interface is working correctly, I am perfectly able to resolve addresses, and I am able to ping external addresses:

As you can see on the image below, I ping google.com and it resolves, and I am also able ews.netgate.com a test requested by @stephenw10

(I am thinking that the issue here is that the interface is getting 1500 MTU when it should be 1400 MTU. But installer does not provide a method to set the MTU that I am aware of? I could set the MTU manually when on command prompt, but I don't know how to go back to the installer from there)

Ok so in the meanwhile @stephenw10 kindly suggested to test with the 2.7.2 install. So I downloaded the 2.7.2 ISO and installed CE.
After install I restored the config and the upgrade from 2.7.2 to 23.09.1 is offered.
I upgrade from 2.7.2 to 23.09.1 and it goes flawlessly.
After being on 23.09.1 I am offered to upgrade to 24.03. So I do, and the result:

And I'm back to square one.

(In the meanwhile my deepest thanks to @stephenw10 for putting up with me and providing all the help he could)

stephenw10

Duplicating messages:
Are you able to get a crash report after that panic?

Are you using xn NICs?

maverickws

@stephenw10
No unfortunately from what I was able to see I am not able to get a crash report after the panic.
The NIC's show as xn yes.

# lspci | grep -E -i --color 'network|ethernet'
23:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)

maverickws

here's the backtrace:

db> bt
Tracing pid 0 tid 100000 td 0xffffffff8303de40
kdb_enter() at kdb_enter+0x33/frame 0xffffffff83f0c890
panic() at panic+0x43/frame 0xffffffff83f0c8f0
trap_fatal() at trap_fatal+0x40f/frame 0xffffffff83f0c950
trap_pfault() at trap_pfault+0x4f/frame 0xffffffff83f0c9b0
calltrap() at calltrap+0x8/frame 0xffffffff83f0c9b0
--- trap 0xc, rip = 0xffffffff8128c005, rsp = 0xffffffff83f0ca88, rbp = 0xffffffff83f0cad0 ---
xen_start32() at xen_start32+0x5/frame 0xffffffff83f0cad0
xenpci_attach() at xenpci_attach+0x207/frame 0xffffffff83f0cb10
device_attach() at device_attach+0x3b5/frame 0xffffffff83f0cb60
bus_generic_attach() at bus_generic_attach+0x4b/frame 0xffffffff83f0cb90
pci_attach() at pci_attach+0xcb/frame 0xffffffff83f0cbd0
acpi_pci_attach() at acpi_pci_attach+0x17/frame 0xffffffff83f0cc10
device_attach() at device_attach+0x3b5/frame 0xffffffff83f0cc60
bus_generic_attach() at bus_generic_attach+0x4b/frame 0xffffffff83f0cc90
acpi_pcib_acpi_attach() at acpi_pcib_acpi_attach+0x42f/frame 0xffffffff83f0ccf0
device_attach() at device_attach+0x3b5/frame 0xffffffff83f0cd40
bus_generic_attach() at bus_generic_attach+0x4b/frame 0xffffffff83f0cd70
acpi_probe_children() at acpi_probe_children+0x237/frame 0xffffffff83f0cdd0
acpi_attach() at acpi_attach+0x972/frame 0xffffffff83f0ce60
device_attach() at device_attach+0x3b5/frame 0xffffffff83f0ceb0
bus_generic_attach() at bus_generic_attach+0x4b/frame 0xffffffff83f0cee0
device_attach() at device_attach+0x3b5/frame 0xffffffff83f0cf30
bus_generic_new_pass() at bus_generic_new_pass+0x127/frame 0xffffffff83f0cf60
root_bus_configure() at root_bus_configure+0x36/frame 0xffffffff83f0cf90
configure() at configure+0x9/frame 0xffffffff83f0cfa0
mi_startup() at mi_startup+0x1c8/frame 0xffffffff83f0cff0
db>

stephenw10

Aha, this is good. So it's something Xen specific by the looks of it. Let's see...

stephenw10

Are you able to get the console out put leading up to the panic so we can see what was attaching?

I note that 24.03 is built on FreeBSD15 and 23.09.X is FreeBSD14 so there could be incompatibility there. What version of Xen (or XCP) are you using?

maverickws

@stephenw10 hi

I can try to either do a screen recording or halt the VM prior to boot, plug the console and get the output.
I'll get back to this.

Now I have a small question about this FreeBSD versioning:
FreeBSD 14.1 was released June 2024
FreeBSD 15 official release schedule points it to be released in December 2025

How exactly are we already on FreeBSD 15 here? How ready for production is it?

Xen version is latest

# cat /etc/os-release 
NAME="XCP-ng"
VERSION="8.2.1"
ID="xenenterprise"
ID_LIKE="centos rhel fedora"
VERSION_ID="8.2.1"
PRETTY_NAME="XCP-ng 8.2.1"
ANSI_COLOR="0;31"
HOME_URL="http://xcp-ng.org/"
BUG_REPORT_URL="https://github.com/xcp-ng/xcp"

stephenw10

For our purposes, ready. I've run it up to our devs. Let's see what they say.

Unfortunately, as I say, I don't think any of them are running Xen/XCP any longer.

stephenw10

You're not using any special packages or modules for Xen I assume?

Like xe-guest-utilities?

maverickws

@stephenw10 No I am not, quite plain install.

Mind if I was why were you questioning about the xn network interfaces? Do they have some known issues?

By default none of the VM's had guest utilities installed.
Yesterday on my last attempt I installed xe-guest-utilities to see if that would render some difference, but nothing.

stephenw10

I asked about xn because it's an unusual NIC type. There are default configs for some NICs types like em and igb and none for xn, hn, virtio etc. If you had the hypervisor configured to present e1000 NICs it might have behaved differently.

It appears to be an issue when trying to attach something Xen specific but it's not clear just from the backtrace what that is. It may be possible to simply disable it.

maverickws

Alright so I'll try to come back and present a better output of what happens previous to the crash. I'm completely unable to do it now but I'll try to do it today still.

stephenw10

Thanks, that should help a lot.