2100-MAX upgrade 24.03 to 24.11 failed

JonathanLee

Hello fellow Netgate community members, I cannot upgrade my 2100 I am getting the following errors when trying to upgrade, the system fails during the reboot process I pulled what was listed on the serial console session.

everything works on all the other versions I am back on my older known good config.

Any ideas?

Loading kernel...
/boot/kernel/kernel text=0x318 text=0xcbe8a8 text=0x14683e8 data=0x19cff0 data=0x0+0x2ca000 0x8+0x19d088+0x8+0x1b8d4c
Loading configured modules...
/boot/kernel/zfs.ko text=0xce1b0 text=0x220bd8 data=0x3b0f0+0xaad44 0x8+0x49380+0x8+0x302ef
/boot/kernel/opensolaris.ko text=0x195e text=0xbf0 data=0x4d0+0x1a030 0x8+0xf78+0x8+0x87b
can't find '/etc/hostid'
can't find '/boot/entropy'
Using DTB provided by EFI at 0x8000000.
---<<BOOT>>---GDB: current port: uart
KDB: debugger backends: ddb gdb
KDB: current backend: ddb
WARNING: DTB version is 6.4 while kernel expects 6.8, please update the DTB in the ESP
Copyright (c) 1992-2024 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 15.0-CURRENT #0 plus-RELENG_24_11-n256407-1bbb3194162: Fri Nov 22 05:12:13 UTC 2024
    root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-24_11-main/obj/aarch64/KRivbaXa/var/jenkins/workspace/pfSense-Plus-snapshots-24_11-main/sources/FreeBSD-src-plus-RELENG_24_11/arm64.aarch64/sys/pfSense arm64
FreeBSD clang version 18.1.6 (https://github.com/llvm/llvm-project.git llvmorg-18.1.6-0-g1118c2e05e67)
VT: init without driver.
module scmi already present!
real memory  = 3602862080 (3435 MB)
avail memory = 3461632000 (3301 MB)
Starting CPU 1 (1)
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
arc4random: WARNING: initial seeding bypassed the cryptographic random device because it was not yet seeded and the knob 'bypass_before_seeding' was enabled.
random: entropy device external interface
wlan: mac acl policy registered
kbd0 at kbdmux0
ofwbus0: <Open Firmware Device Tree>
simplebus0: <Flattened device tree simple bus> on ofwbus0
simplebus1: <Flattened device tree simple bus> on simplebus0
simple_mfd0: <Simple MFD (Multi-Functions Device)> mem 0x13800-0x138ff,0x13c00-0x13c1f on simplebus1
simple_mfd1: <Simple MFD (Multi-Functions Device)> mem 0x18800-0x188ff,0x18c00-0x18c1f on simplebus1
psci0: <ARM Power State Co-ordination Interface Driver> on ofwbus0
netgate0: <Netgate 2100>
netgate0: version: 0.1
gic0: <ARM Generic Interrupt Controller v3.0> mem 0x1d00000-0x1d0ffff,0x1d40000-0x1d7ffff,0x1d80000-0x1d81fff,0x1d90000-0x1d91fff,0x1da0000-0x1dbffff irq 27 on simplebus1
gpio0: <Armada 37x0 North Bridge GPIO Controller> mem 0x13800-0x138ff,0x13c00-0x13c1f irq 28,29,30,31,32,33,34,35,36,37,38,39 on simple_mfd0
gpiobus0: <OFW GPIO bus> on gpio0
gpio1: <Armada 37x0 South Bridge GPIO Controller> mem 0x18800-0x188ff,0x18c00-0x18c1f irq 40,41,42,43,44 on simple_mfd1
gpiobus1: <OFW GPIO bus> on gpio1
gpiomdio0: <GPIO MDIO bit-banging Bus driver> at pins 21,23 on gpiobus1
gpiomdio0: MDC pin: 21, MDIO pin: 23
a37x0-xtal0: <Marvell Armada 3700 Oscillator> mem 0x13800-0x138ff,0x13c00-0x13c1f on simple_mfd0
generic_timer0: <ARMv8 Generic Timer> irq 0,1,2,3 on ofwbus0
Timecounter "ARM MPCore Timecounter" frequency 12500000 Hz quality 1000
Event timer "ARM MPCore Eventtimer" frequency 12500000 Hz quality 1000
a37x0_tbg0: <Marvell Armada 3700 time base generators> mem 0x13200-0x132ff on simplebus1
a37x0_nb_periph_driver0: <marvell,armada-3700-nb-periph-clock> mem 0x13000-0x130ff on simplebus1
a37x0_sb_periph_driver0: <marvell,armada-3700-sb-periph-clock> mem 0x18000-0x180ff on simplebus1
cpulist0: <Open Firmware CPU Group> on ofwbus0
cpu0: <Open Firmware CPU> on cpulist0
pmu0: <Performance Monitoring Unit> irq 4 on ofwbus0
spi0: <Armada 37x0 SPI controller> mem 0x10600-0x10fff irq 6 on simplebus1
iichb0: <Marvell Armada 37x0 IIC controller> mem 0x11000-0x11023 irq 7 on simplebus1
iicbus0: <OFW I2C bus> on iichb0
iic0: <I2C generic I/O> on iicbus0
iichb1: <Marvell Armada 37x0 IIC controller> mem 0x11080-0x110a3 irq 8 on simplebus1
iicbus1: <OFW I2C bus> on iichb1
iic1: <I2C generic I/O> on iicbus1
nxprtc0: <NXP PCF8563 RTC> at addr 0xa2 on iicbus1
gpio2: <ISSI IS31FL3199 9 channel light effect LED driver> at addr 0xce on iicbus1
gpiobus2: <OFW GPIO bus> on gpio2
gpioc2: <GPIO controller> on gpio2
uart0: <Marvell Armada 3700 UART> mem 0x12000-0x121ff irq 9,10,11 on simplebus1
uart0: console (115200,n,8,1)
gpioc0: <GPIO controller> on gpio0
gpioc1: <GPIO controller> on gpio1
mvneta0: <NETA controller> mem 0x30000-0x33fff irq 14 on simplebus1
mvneta0: version is 10
miibus0: <MII bus> on gpiomdio0
mv88e151x0: <Marvell 88E1512 Gigabit PHY> PHY 1 on miibus0
mv88e151x0:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseSX, 1000baseSX-FDX, auto
mvneta0: Ethernet address: 90:ec:77:0d:da:25
mvneta1: <NETA controller> mem 0x40000-0x43fff irq 15 on simplebus1
mvneta1: version is 10
mdio0: <MDIO> on mvneta1
mdioproxy0: <MII/MDIO proxy, MDIO side> on mdio0
e6000sw0: <Marvell 88E6141> on mdio0
e6000sw0: single-chip addressing mode
e6000sw0: PHY at port 1
miibus1: <MII bus> on e6000sw0
e1000phy0: <Marvell 88E1000 Gigabit PHY> PHY 17 on miibus1
e1000phy0:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto
e6000sw0: PHY at port 2
miibus2: <MII bus> on e6000sw0
e1000phy1: <Marvell 88E1000 Gigabit PHY> PHY 18 on miibus2
e1000phy1:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto
e6000sw0: PHY at port 3
miibus3: <MII bus> on e6000sw0
e1000phy2: <Marvell 88E1000 Gigabit PHY> PHY 19 on miibus3
e1000phy2:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto
e6000sw0: PHY at port 4
miibus4: <MII bus> on e6000sw0
e1000phy3: <Marvell 88E1000 Gigabit PHY> PHY 20 on miibus4
e1000phy3:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto
e6000sw0: CPU port at 5
e6000sw0: fixed port at 5
e6000sw0: switch is ready.
etherswitch0: <Switch controller> on e6000sw0
mvneta1: Ethernet address: 90:ec:77:0d:da:26
xhci0: <Generic USB 3.0 controller> mem 0x58000-0x5bfff irq 16 on simplebus1
xhci0: 32 bytes context size, 32-bit DMA
usbus0 on xhci0
ehci0: <Marvell Integrated USB 2.0 controller> mem 0x5e000-0x5ffff irq 17 on simplebus1
usbus1: EHCI version 1.0
usbus1 on ehci0
sdhci_xenon0: <Armada Xenon SDHCI controller> mem 0xd8000-0xd82ff,0x17808-0x1780b irq 25 on simplebus1
mmc0: <MMC/SD bus> on sdhci_xenon0
ahci0: <AHCI SATA controller> mem 0xe0000-0xe1fff irq 26 on simplebus1
ahci0: AHCI v1.30 with 1 6Gbps ports, Port Multiplier supported with FBS
ahcich0: <AHCI channel> at channel 0 on ahci0
pcib0: <Marvell Armada 3700 PCIe Bus Controller> mem 0xd0070000-0xd008ffff irq 5 on simplebus0
pcib0: link up
pci0: <OFW PCI bus> on pcib0
nvme0: <Generic NVMe Device> mem 0xe8000000-0xe8003fff at device 0.0 on pci0
Fatal data abort:
  x0: 0x0000000000000000
  x1: 0x00000000e8000000
  x2: 0x0000000000000000
  x3: 0x0000000000000000
  x4: 0x0000000000020000
  x5: 0x0000000000000006
  x6: 0x0000000000000002
  x7: 0xffff0000022c9520 (initstack + 0x3520)
  x8: 0xffff000002221b00 (memmap_bus + 0x0)
  x9: 0xffff000000b6a350 (generic_bs_r_4 + 0x0)
 x10: 0xffff00000219e3b8 (bus_get_domain_desc + 0x0)
 x11: 0xffff00000219e3b8 (bus_get_domain_desc + 0x0)
 x12: 0xffff0000021b1ba8 (lock_class_mtx_spin + 0x0)
 x13: 0xfefefefefeff0f00
 x14: 0x0000000000000000
 x15: 0x0000000000000000
 x16: 0xffff0000006bc734 (bus_generic_get_domain + 0x0)
 x17: 0x0000000000000000
 x18: 0xffff0000022c95a0 (initstack + 0x35a0)
 x19: 0xffff00009cc50000 ($d.6 + 0x99ebe5b8)
 x20: 0xffffa00001c83200
 x21: 0xffff00009cc50048 ($d.6 + 0x99ebe600)
 x22: 0xffff00009cc69000 ($d.6 + 0x99ed75b8)
 x23: 0xffff00009cc50068 ($d.6 + 0x99ebe620)
 x24: 0xffffa00001c83300
 x25: 0xffff000000cf4c5b (notify.prefix + 0x326f7)
 x26: 0xffffa000019478a0
 x27: 0xffffa000015c7620
 x28: 0x0000000000000002
 x29: 0xffff0000022c9620 (initstack + 0x3620)
  sp: 0xffff0000022c95a0
  lr: 0xffff0000002f7e38 (nvme_ctrlr_construct + 0x70)
 elr: 0xffff000000b6a358 (generic_bs_r_4 + 0x8)
spsr: 0x00000000600000c5
 far: 0x00000000e8000000
 esr: 0x0000000096000004
panic: vm_fault failed: 0xffff000000b6a358 error 1
cpuid = 0
time = 1
KDB: enter: panic
[ thread pid 0 tid 100000 ]
Stopped at      kdb_enter+0x48: str     xzr, [x19, #2944]
db>

JonathanLee

It looks to hang on my NVMe drive, that works in the old versions

Gertjan

@JonathanLee

You have a backed up config, right ?
Re install from scratch, use the new installer, or TAC a firmware for the 2100.
Meanwhile, I'll keep my fingers crossed for you.

stephenw10

Mmm, it does look to be hitting an issue in the nvme driver.

You are almost certainly the only 2100 user with an NVMe drive. Including anyone at Netgate!

Does it boot past that if you disconnect the drive?

JonathanLee

@stephenw10 I would have to take it apart to disconnect it. I will try that soon. Like try to disconnect it and let the software install complete and reconnect it?

patient0

@JonathanLee can't you disable the nvme driver using a hint?

JonathanLee

@patient0

hint.nvme.0.disabled="1" ?? Add this to system tuneable and restart ?

stephenw10

Yes, that should work. Or run at the loader prompt:

set hint.nvme.0.disabled=1
boot

JonathanLee

@stephenw10 will I be able to restore the driver after the install or has the NVMe driver been removed for the 2100

stephenw10

The driver has not been removed. It looks like there is a bug/regression in the driver but since you're the only user you're only person who has hit it. It will probably fail again if you enable it. But it might not if it;s something specific to the upgrade....

JonathanLee

@stephenw10 I will have to wait until the end of the semester to test again it malformed the data on the boot environment after also. So, I had to use another one I have multiples of them in case a test failed glad I made some, I am so thankful for boot environments, it failed I was up in 3 mins after I got a term session going again.