netgate APU4 bricked after initiating update in web console
-
I have a Netgate APU4 that was working just fine since 2015 and I initiated an update to the latest version today. I forget the exact version number but it was _2 that I had and I was upgrading to the latest _3 release listed in the updater in the web administration panel of the device. Based on the version of the freebsd kernel that is listed in the boot output from the serial console (below) it was the upgrade from 11.2-RELEASE-p4 to 11.2-RELEASE-p10 that I installed. So probably pfsense 2.4.4-p2 to 2.4.4-p3.
It seemed like the update was going well but the device never came back up after restarting.
Since I had to restart my computer to get a serial port installed to be able to see the console of the device I no longer have the web console output. Below is the output of the device booting from the serial console.
I would super appreciate it for any help that could be provided to resolve this issue.
It hangs indefinitely at this point:
Trying to mount root from ufs:/dev/ufsid/5358b0d792733dcf [rw]... WARNING: / was not properly dismounted uhub1: 4 ports with 4 removable, self powered uhub0: 5 ports with 5 removable, self powered ugen6.2: <Generic Flash Card ReaderWriter> at usbus6 umass0 on uhub1 umass0: <Generic Flash Card ReaderWriter, class 0/0, rev 2.01/1.00, addr 2> on usbus6 uhub6: 5 ports with 5 removable, self powered da0 at umass-sim0 bus 0 scbus6 target 0 lun 0 da0: <Multiple Card Reader 1.00> Removable Direct Access SPC-2 SCSI device da0: Serial Number 058F63666485 da0: 40.000MB/s transfers da0: Attempt to query device size failed: NOT READY, Medium not present da0: quirks=0x2<NO_6_BYTE> random: unblocking device.This is the full output of the boot from the serial console:
PC Engines APU BIOS build date: Apr 5 2014 Reading data from file [bootorder] SeaBIOS (version ?-20140405_120742-frink) SeaBIOS (version ?-20140405_120742-frink) Found coreboot cbmem console @ df150400 Found mainboard PC Engines APU Relocating init from 0x000e8e71 to 0xdf1065e0 (size 39259) Found CBFS header at 0xfffffb90 found file "bootorder" in cbmem CPU Mhz=1000 Found 27 PCI devices (max PCI bus is 05) Copying PIR from 0xdf160400 to 0x000f27a0 Copying MPTABLE from 0xdf161400/df161410 to 0x000f25b0 with length 1ec Copying ACPI RSDP from 0xdf162400 to 0x000f2590 Copying SMBIOS entry point from 0xdf16d800 to 0x000f2570 Using pmtimer, ioport 0x808 Scan for VGA option rom EHCI init on dev 00:12.2 (regs=0xf7f08420) Found 1 lpt ports Found 2 serial ports AHCI controller at 11.0, iobase f7f08000, irq 11 EHCI init on dev 00:13.2 (regs=0xf7f08520) EHCI init on dev 00:16.2 (regs=0xf7f08620) Searching bootorder for: /rom@img/setup Searching bootorder for: /rom@img/memtest Searching bootorder for: /pci@i0cf8/*@11/drive@0/disk@0 AHCI/0: registering: "AHCI/0: INTEL SSDMCEAC030B3 ATA-9 Hard-Disk (28626 MiBytes )" OHCI init on dev 00:12.0 (regs=0xf7f04000) OHCI init on dev 00:13.0 (regs=0xf7f05000) OHCI init on dev 00:14.5 (regs=0xf7f06000) OHCI init on dev 00:16.0 (regs=0xf7f07000) Searching bootorder for: /pci@i0cf8/usb@16,2/storage@1/*@0/*@0,0 Searching bootorder for: /pci@i0cf8/usb@16,2/usb-*@1 USB MSC vendor='Multiple' product='Card Reader' rev='1.00' type=0 removable=1 Device reports MEDIUM NOT PRESENT scsi_is_ready returned -1 Unable to configure USB MSC drive. Unable to configure USB MSC device. All threads complete. Scan for option roms Running option rom at c000:0003 iPXE (http://ipxe.org) 00:00.0 C000 PCI2.10 PnP PMMpmm call arg1=1 pmm call arg1=0 +DF0F07E0pmm call arg1=1 pmm call arg1=0 +DF04C510 C000 Searching bootorder for: /rom@genroms/pxeboot.rom Build date: Apr 5 2014 System memory size: 4592 MB Press F12 for boot menu. Searching bootorder for: HALT drive 0x000f2500: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 s=58626288 Space available for UMB: c1000-ee800, f0000-f2500 Returned 253952 bytes of ZoneHigh e820 map has 7 items: 0: 0000000000000000 - 000000000009fc00 = 1 RAM 1: 000000000009fc00 - 00000000000a0000 = 2 RESERVED 2: 00000000000f0000 - 0000000000100000 = 2 RESERVED 3: 0000000000100000 - 00000000df14e000 = 1 RAM 4: 00000000df14e000 - 00000000e0000000 = 2 RESERVED 5: 00000000f8000000 - 00000000f9000000 = 2 RESERVED 6: 0000000100000000 - 000000011f000000 = 1 RAM enter handle_19: NULL Booting from Hard Disk... Booting from 0000:7c00 F1 pfSense F6 PXE Boot: F1 //bboooott//ccoonnffiigg:: --SS111155220000 --DD Consoles: internal video/keyboard serial port BIOS drive C: is disk0 BIOS 639kB/3653944kB available memory FreeBSD/x86 bootstrap loader, Revision 1.1 (Wed Nov 21 11:25:53 EST 2018 root@buildbot2.nyi.netgate.com) Error: stack underflow Error while including /boot/loader.4th, in the line: - /boot/kernel/kernel text=0x17c75d8 data=0xb949e0+0x557b28 syms=[0x8+0x1983a8+0x8+0x1990f6] Hit [Enter] to boot immediately, or any other key for command prompt. Booting [/boot/kernel/kernel]... KDB: debugger backends: ddb KDB: current backend: ddb Copyright (c) 1992-2018 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 11.2-RELEASE-p10 #9 10fea60fdde(factory-RELENG_2_4_4): Thu May 16 06:55:01 EDT 2019 root@buildbot1-nyi.netgate.com:/build/factory-crossbuild-244/obj/amd64/bwlyKhYx/build/factory-crossbuild-244/pfSense/tmp/FreeBSD-src/sys/pfSense amd64 FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0) VT(vga): resolution 640x480 CPU: AMD G-T40E Processor (1000.02-MHz K8-class CPU) Origin="AuthenticAMD" Id=0x500f20 Family=0x14 Model=0x2 Stepping=0 Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT> Features2=0x802209<SSE3,MON,SSSE3,CX16,POPCNT> AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM> AMD Features2=0x35ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,IBS,SKINIT,WDT> SVM: NP,NRIP,NAsids=8 TSC: P-state invariant, performance statistics real memory = 4815060992 (4592 MB) avail memory = 4054708224 (3866 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: <CORE COREBOOT> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 1 package(s) x 2 core(s) ioapic0 <Version 2.1> irqs 0-23 on motherboard SMP: AP CPU #1 Launched! Timecounter "TSC" frequency 1000023140 Hz quality 800 ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff80685700, 0) error 1 random: entropy device external interface ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff806857b0, 0) error 1 ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE. ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff80685860, 0) error 1 iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff806ac730, 0) error 1 iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff806ac7e0, 0) error 1 iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE. iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff806ac890, 0) error 1 wlan: mac acl policy registered kbd0 at kbdmux0 netmap: loaded module module_register_init: MOD_LOAD (vesa, 0xffffffff8120faa0, 0) error 19 nexus0 vtvga0: <VT VGA driver> on motherboard cryptosoft0: <software crypto> on motherboard padlock0: No ACE support. acpi0: <CORE COREBOOT> on motherboard acpi0: Power Button (fixed) cpu0: <ACPI CPU> on acpi0 cpu1: <ACPI CPU> on acpi0 atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0 atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 550 Event timer "HPET1" frequency 14318180 Hz quality 450 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 pcib1: <ACPI PCI-PCI bridge> irq 16 at device 4.0 on pci0 pci1: <ACPI PCI bus> on pcib1 re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0x1000-0x10ff mem 0xf7a00000-0xf7a00fff,0xf7900000-0xf7903fff irq 16 at device 0.0 on pci1 re0: Using 1 MSI-X message re0: ASPM disabled re0: Chip rev. 0x2c000000 re0: MAC rev. 0x00200000 miibus0: <MII bus> on re0 rgephy0: <RTL8169S/8110S/8211 1000BASE-T media interface> PHY 1 on miibus0 rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow re0: Using defaults for TSO: 65518/35/2048 re0: Ethernet address: 00:0d:b9:3a:4c:f8 re0: netmap queues/slots: TX 1/256, RX 1/256 pcib2: <ACPI PCI-PCI bridge> irq 17 at device 5.0 on pci0 pci2: <ACPI PCI bus> on pcib2 re1: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0x2000-0x20ff mem 0xf7c00000-0xf7c00fff,0xf7b00000-0xf7b03fff irq 17 at device 0.0 on pci2 re1: Using 1 MSI-X message re1: ASPM disabled re1: Chip rev. 0x2c000000 re1: MAC rev. 0x00200000 miibus1: <MII bus> on re1 rgephy1: <RTL8169S/8110S/8211 1000BASE-T media interface> PHY 1 on miibus1 rgephy1: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow re1: Using defaults for TSO: 65518/35/2048 re1: Ethernet address: 00:0d:b9:3a:4c:f9 re1: netmap queues/slots: TX 1/256, RX 1/256 pcib3: <ACPI PCI-PCI bridge> irq 18 at device 6.0 on pci0 pci3: <ACPI PCI bus> on pcib3 re2: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0x3000-0x30ff mem 0xf7e00000-0xf7e00fff,0xf7d00000-0xf7d03fff irq 18 at device 0.0 on pci3 re2: Using 1 MSI-X message re2: ASPM disabled re2: Chip rev. 0x2c000000 re2: MAC rev. 0x00200000 miibus2: <MII bus> on re2 rgephy2: <RTL8169S/8110S/8211 1000BASE-T media interface> PHY 1 on miibus2 rgephy2: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow re2: Using defaults for TSO: 65518/35/2048 re2: Ethernet address: 00:0d:b9:3a:4c:fa re2: netmap queues/slots: TX 1/256, RX 1/256 ahci0: <AMD SB7x0/SB8x0/SB9x0 AHCI SATA controller> port 0x4010-0x4017,0x4020-0x4023,0x4018-0x401f,0x4024-0x4027,0x4000-0x400f mem 0xf7f08000-0xf7f083ff irq 19 at device 17.0 on pci0 ahci0: AHCI v1.20 with 6 6Gbps ports, Port Multiplier supported ahci0: quirks=0x22000<ATI_PMP_BUG,1MSI> ahcich0: <AHCI channel> at channel 0 on ahci0 ahcich1: <AHCI channel> at channel 1 on ahci0 ahcich2: <AHCI channel> at channel 2 on ahci0 ahcich3: <AHCI channel> at channel 3 on ahci0 ahcich4: <AHCI channel> at channel 4 on ahci0 ahcich5: <AHCI channel> at channel 5 on ahci0 ohci0: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xf7f04000-0xf7f04fff irq 18 at device 18.0 on pci0 usbus0 on ohci0 usbus0: 12Mbps Full Speed USB v1.0 ehci0: <AMD SB7x0/SB8x0/SB9x0 USB 2.0 controller> mem 0xf7f08400-0xf7f084ff irq 17 at device 18.2 on pci0 usbus1: EHCI version 1.0 usbus1 on ehci0 usbus1: 480Mbps High Speed USB v2.0 ohci1: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xf7f05000-0xf7f05fff irq 18 at device 19.0 on pci0 usbus2 on ohci1 usbus2: 12Mbps Full Speed USB v1.0 ehci1: <AMD SB7x0/SB8x0/SB9x0 USB 2.0 controller> mem 0xf7f08500-0xf7f085ff irq 17 at device 19.2 on pci0 usbus3: EHCI version 1.0 usbus3 on ehci1 usbus3: 480Mbps High Speed USB v2.0 isab0: <PCI-ISA bridge> at device 20.3 on pci0 isa0: <ISA bus> on isab0 pcib4: <ACPI PCI-PCI bridge> at device 20.4 on pci0 pci4: <ACPI PCI bus> on pcib4 ohci2: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xf7f06000-0xf7f06fff irq 18 at device 20.5 on pci0 usbus4 on ohci2 usbus4: 12Mbps Full Speed USB v1.0 pcib5: <ACPI PCI-PCI bridge> at device 21.0 on pci0 pci5: <ACPI PCI bus> on pcib5 ohci3: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xf7f07000-0xf7f07fff at device 22.0 on pci0 usbus5 on ohci3 usbus5: 12Mbps Full Speed USB v1.0 ehci2: <AMD SB7x0/SB8x0/SB9x0 USB 2.0 controller> mem 0xf7f08600-0xf7f086ff at device 22.2 on pci0 usbus6: EHCI version 1.0 usbus6 on ehci2 usbus6: 480Mbps High Speed USB v2.0 acpi_button0: <Power Button> on acpi0 orm0: <ISA Option ROM> at iomem 0xee800-0xeffff on isa0 acpi_throttle0: <ACPI CPU Throttling> on cpu0 acpi_throttle1: <ACPI CPU Throttling> on cpu1 acpi_throttle1: failed to attach P_CNT device_attach: acpi_throttle1 attach returned 6 Timecounters tick every 1.000 msec ugen1.1: <ATI EHCI root HUB> at usbus1 ugen6.1: <ATI EHCI root HUB> at usbus6 uhub0: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1 uhub1: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus6 ugen5.1: <ATI OHCI root HUB> at usbus5 ugen2.1: <ATI OHCI root HUB> at usbus2 uhub2: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5 uhub3: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2 ugen0.1: <ATI OHCI root HUB> at usbus0 ugen4.1: <ATI OHCI root HUB> at usbus4 uhub4: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0 uhub5: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4 ugen3.1: <ATI EHCI root HUB> at usbus3 uhub6: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3 ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 uhub3: 5 ports with 5 removable, self powered uhub5: 2 ports with 2 removable, self powered uhub4: 5 ports with 5 removable, self powered ada0: <INTEL SSDMCEAC030B3 LLLi> ACS-2 ATA SATA 3.x device uhub2: ada0: Serial Number CVLI446200X7030H 4 ports with 4 removable, self powered ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 28626MB (58626288 512 byte sectors) Trying to mount root from ufs:/dev/ufsid/5358b0d792733dcf [rw]... WARNING: / was not properly dismounted uhub1: 4 ports with 4 removable, self powered uhub0: 5 ports with 5 removable, self powered ugen6.2: <Generic Flash Card ReaderWriter> at usbus6 umass0 on uhub1 umass0: <Generic Flash Card ReaderWriter, class 0/0, rev 2.01/1.00, addr 2> on usbus6 uhub6: 5 ports with 5 removable, self powered da0 at umass-sim0 bus 0 scbus6 target 0 lun 0 da0: <Multiple Card Reader 1.00> Removable Direct Access SPC-2 SCSI device da0: Serial Number 058F63666485 da0: 40.000MB/s transfers da0: Attempt to query device size failed: NOT READY, Medium not present da0: quirks=0x2<NO_6_BYTE> random: unblocking device. -
Hmm, that looks a lot like the wrong console type set as primary. Did you import the config into the APU from something else originally?
Do you have a backup config file you can check? Or use to re-install if needed.
Steve
-
@stephenw10 said in netgate APU4 bricked after initiating update in web console:
nothing was changed in the system. this came as a preinstalled pfsense image from netgate. the only thing I have ever done was press the upgrade button in the web console. no config import was performed. backup config - none unless pfsense itself does one.
-
Hmm, you could try booting into single user mode from the loader menu. But to be honest if you have a backup of the config file it's going to be quickest to reinstall. And it will guarantee you a clean filesystem.
Steve
-
I tried booting to single user mode by specifying
boot -sat the kernel loader shell, but the same problem occurred where console output stopped. I worked with some people on the freebsd IRC channel on freenode and it appears somehow multiple files were corrupted. /boot/loader.4th was corrupted totally and /etc/ttys was truncated. I ended up doing a fresh install after taking a forensic backup of the drive, which is unfortunate that I lost all my settings. -
You were not able to recover the config during the install? It must have been a significantly damaged file system if so.
Steve
