Kernel Panic when loading axgbe after Upgrade to FreeBSD 14 based 2.7.0

[xplChris]

Moin,

I recently upgraded a FreeBSD 12 based 2.7.0 pfSense CE running on a DEC740 hardware to the latest FreeBSD 14 with help of the instructions in https://www.netgate.com/blog/ce-snapshots-on-php-8.1-and-freebsd-main

The Upgrade worked like it should, but after the reboot the machines paniced

ax0: <AMD 10 Gigabit Ethernet Driver> mem 0xd0060000-0xd007ffff,0xd0040000-0xd005ffff,0xd0082000-0xd0083fff at device 0.1 on pci6
ax0: Using 512 TX descriptors and 512 RX descriptors
ax0: Using 3 RX queues 3 TX queues
ax0: Using MSI-X interrupts with 7 vectors
panic: Assertion nrxqs == 1 failed at /var/jenkins/workspace/pfSense-CE-snapshots-master-main/sources/FreeBSD-src-devel-main/sys/dev/axgbe/if_axgbe_pci.c:1736cpuid = 0
time = 1
KDB: enter: panic
[ thread pid 0 tid 100000 ]
Stopped at      kdb_enter+0x32: movq    $0,0x27cf353(%rip)

Since I had a up-to-date Backup of the config, I just prepared a USB stick with the latest 2.7.0 and bootet from it - boom, same error.

Recoverying the machine with a FreeBSD12 based 2.7.0 was no problem, but at the end I'd like to upgrade to the latest version.

Has anyone an idee what went wrong and if/how I could solve this?

Thanks!
Chris

[stephenw10]

Is there a complete backtrace shown?

[xplChris]

No, this is all from which I think is relevant. I attached a text file with everything the boot process printed out on the serial console.

175665-console-output.txt

[stephenw10]

Enter bt at the db> prompt there to get the backtrace. Though it might not actually help much here.

[jimp]

Where did you get that kernel module for axgbe?

It doesn't appear to be part of the build or ports tree.

If the kernel module was still from 12.x it would almost certainly fail on a 14 kernel. Or even if it was against a 14 kernel that wasn't recent enough.

[xplChris]

This is strange, I never installed a kernel module in any way.

The maschine was setup with the image pfSense-CE-memstick-serial-2.7.0-DEVELOPMENT-amd64 build from 19.01.2022, which worked out of the box.

There was an preinstalled OPNsense on the DEC740 (as it came fresh from manufactory) but I deleted it and never booted it

[stephenw10]

Maybe it's remaining from the FreeBSD 12 build. What the date shown as?
ls -ls /boot/kernel/if_ax*

[jimp]

It's possible it's in the kernel just under some other name then. I do see ax in there but not axgbe, so it's just a bit confusing. The axgbe driver code references if_axa and if_axp, and I do see if_axp.ko in the kernel modules, but trying to load it by hand says it's already in the kernel (even though kldstat -v does not show axp).

device axp is in the kernel, and a peek at the axgbe source does show it would be attaching as ax so it does appear to be in the kernel already, not a module.

That is one convoluted driver naming structure, though!

[stephenw10]

Yeah that's fun*:

The axp device driver first appeared in FreeBSD 13.0.

     Another version of	the driver is already present in FreeBSD.  This	driver
     was named as "axgbe" earlier, which is renamed as "axa" now.  This	driver
     is	for the	ACPI based Ethernet controllers	in the previous/older version
     of	the hardware.

Maybe it needs axa(4).

[jimp]

That's possible, though it doesn't look like the if_axa module is being built and it's not in the kernel. Hard to tell if they're mutually exclusive when building or if one can override the other when loaded in loader.conf.

[xplChris]

Moin,

I've no physical access to the machine (next time I think about Thursday) so I could only have a look at the running machine with FreeBSD 12. On thursday, I may boot from a FreeBSD 14 installer.

[2.7.0-DEVELOPMENT][root@boavista.emetriq.com]/root: ls -ls /boot/kernel/if_ax*
33 -r-xr-xr-x  1 root  wheel  46616 Jan 19  2022 /boot/kernel/if_axe.ko
29 -r-xr-xr-x  1 root  wheel  38688 Jan 19  2022 /boot/kernel/if_axge.ko

[stephenw10]

You could try disabling it at the boot loader prompt to allow it to boot:

set hint.ax.0.disabled=1
boot

Plus any others if there's more than one ax NIC.

Steve

[xplChris]

Thanks for this hint, I'll give it a try next time I'm in front of the machine, which is probably this thursday.

I'll also try putting the module into module_blacklist in the bootloader.conf. Right now the ax* NICs are unused, so no impact on functionality ;-)

[xplChris]

Moin,

I took some time, but today I'm on-site and gave the "NIC" disable thing a try. I added three lines in /boot/loader.conf before I upgraded the machine

[2.7.0-DEVELOPMENT][root@boavista.emetriq.com]/root: grep ax /boot/loader.conf
hint.ax.0.disabled="1"
hint.ax.1.disabled="1"
module_blacklist="axgbe"

The upgrade worked like charm and everything is working as expected

As a side node: I didn't found any clue for the 10GE AMD ax devices on the FreeBSD12 2.7.0, so the NICs weren't functioning anyway.

Thanks! for your help.

Greetings,
Chris