netgate APU4 bricked after initiating update in web console



  • I have a Netgate APU4 that was working just fine since 2015 and I initiated an update to the latest version today. I forget the exact version number but it was _2 that I had and I was upgrading to the latest _3 release listed in the updater in the web administration panel of the device. Based on the version of the freebsd kernel that is listed in the boot output from the serial console (below) it was the upgrade from 11.2-RELEASE-p4 to 11.2-RELEASE-p10 that I installed. So probably pfsense 2.4.4-p2 to 2.4.4-p3.

    It seemed like the update was going well but the device never came back up after restarting.

    Since I had to restart my computer to get a serial port installed to be able to see the console of the device I no longer have the web console output. Below is the output of the device booting from the serial console.

    I would super appreciate it for any help that could be provided to resolve this issue.

    It hangs indefinitely at this point:

    Trying to mount root from ufs:/dev/ufsid/5358b0d792733dcf [rw]...
    WARNING: / was not properly dismounted
    uhub1: 4 ports with 4 removable, self powered
    uhub0: 5 ports with 5 removable, self powered
    ugen6.2: <Generic Flash Card ReaderWriter> at usbus6
    umass0 on uhub1
    umass0: <Generic Flash Card ReaderWriter, class 0/0, rev 2.01/1.00, addr 2> on usbus6
    uhub6: 5 ports with 5 removable, self powered
    da0 at umass-sim0 bus 0 scbus6 target 0 lun 0
    da0: <Multiple Card  Reader 1.00> Removable Direct Access SPC-2 SCSI device
    da0: Serial Number 058F63666485
    da0: 40.000MB/s transfers
    da0: Attempt to query device size failed: NOT READY, Medium not present
    da0: quirks=0x2<NO_6_BYTE>
    random: unblocking device.
    

    This is the full output of the boot from the serial console:

    PC Engines APU BIOS build date: Apr  5 2014
    Reading data from file [bootorder]
    SeaBIOS (version ?-20140405_120742-frink)
    SeaBIOS (version ?-20140405_120742-frink)
    Found coreboot cbmem console @ df150400
    Found mainboard PC Engines APU
    Relocating init from 0x000e8e71 to 0xdf1065e0 (size 39259)
    Found CBFS header at 0xfffffb90
    found file "bootorder" in cbmem
    CPU Mhz=1000
    Found 27 PCI devices (max PCI bus is 05)
    Copying PIR from 0xdf160400 to 0x000f27a0
    Copying MPTABLE from 0xdf161400/df161410 to 0x000f25b0 with length 1ec
    Copying ACPI RSDP from 0xdf162400 to 0x000f2590
    Copying SMBIOS entry point from 0xdf16d800 to 0x000f2570
    Using pmtimer, ioport 0x808
    Scan for VGA option rom
    EHCI init on dev 00:12.2 (regs=0xf7f08420)
    Found 1 lpt ports
    Found 2 serial ports
    AHCI controller at 11.0, iobase f7f08000, irq 11
    EHCI init on dev 00:13.2 (regs=0xf7f08520)
    EHCI init on dev 00:16.2 (regs=0xf7f08620)
    Searching bootorder for: /rom@img/setup
    Searching bootorder for: /rom@img/memtest
    Searching bootorder for: /pci@i0cf8/*@11/drive@0/disk@0
    AHCI/0: registering: "AHCI/0: INTEL SSDMCEAC030B3 ATA-9 Hard-Disk (28626 MiBytes                                                                                                                                                             )"
    OHCI init on dev 00:12.0 (regs=0xf7f04000)
    OHCI init on dev 00:13.0 (regs=0xf7f05000)
    OHCI init on dev 00:14.5 (regs=0xf7f06000)
    OHCI init on dev 00:16.0 (regs=0xf7f07000)
    Searching bootorder for: /pci@i0cf8/usb@16,2/storage@1/*@0/*@0,0
    Searching bootorder for: /pci@i0cf8/usb@16,2/usb-*@1
    USB MSC vendor='Multiple' product='Card  Reader' rev='1.00' type=0 removable=1
    Device reports MEDIUM NOT PRESENT
    scsi_is_ready returned -1
    Unable to configure USB MSC drive.
    Unable to configure USB MSC device.
    All threads complete.
    Scan for option roms
    Running option rom at c000:0003
    
    
    iPXE (http://ipxe.org) 00:00.0 C000 PCI2.10 PnP PMMpmm call arg1=1
    pmm call arg1=0
    +DF0F07E0pmm call arg1=1
    pmm call arg1=0
    +DF04C510 C000
    
    
    
    Searching bootorder for: /rom@genroms/pxeboot.rom
    
    Build date: Apr  5 2014
    System memory size: 4592 MB
    
    Press F12 for boot menu.
    
    Searching bootorder for: HALT
    drive 0x000f2500: PCHS=16383/16/63 translation=lba LCHS=1024/255/63 s=58626288
    Space available for UMB: c1000-ee800, f0000-f2500
    Returned 253952 bytes of ZoneHigh
    e820 map has 7 items:
      0: 0000000000000000 - 000000000009fc00 = 1 RAM
      1: 000000000009fc00 - 00000000000a0000 = 2 RESERVED
      2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
      3: 0000000000100000 - 00000000df14e000 = 1 RAM
      4: 00000000df14e000 - 00000000e0000000 = 2 RESERVED
      5: 00000000f8000000 - 00000000f9000000 = 2 RESERVED
      6: 0000000100000000 - 000000011f000000 = 1 RAM
    enter handle_19:
      NULL
    Booting from Hard Disk...
    Booting from 0000:7c00
    
    F1  pfSense
    
    F6 PXE
    Boot:  F1
    //bboooott//ccoonnffiigg::  --SS111155220000  --DD
    
    Consoles: internal video/keyboard  serial port
    BIOS drive C: is disk0
    BIOS 639kB/3653944kB available memory
    
    FreeBSD/x86 bootstrap loader, Revision 1.1
    (Wed Nov 21 11:25:53 EST 2018 root@buildbot2.nyi.netgate.com)
    Error: stack underflow
    Error while including /boot/loader.4th, in the line:
    -
    
    /boot/kernel/kernel text=0x17c75d8 data=0xb949e0+0x557b28 syms=[0x8+0x1983a8+0x8+0x1990f6]
    Hit [Enter] to boot immediately, or any other key for command prompt.
    Booting [/boot/kernel/kernel]...
    KDB: debugger backends: ddb
    KDB: current backend: ddb
    Copyright (c) 1992-2018 The FreeBSD Project.
    Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
            The Regents of the University of California. All rights reserved.
    FreeBSD is a registered trademark of The FreeBSD Foundation.
    FreeBSD 11.2-RELEASE-p10 #9 10fea60fdde(factory-RELENG_2_4_4): Thu May 16 06:55:01 EDT 2019
        root@buildbot1-nyi.netgate.com:/build/factory-crossbuild-244/obj/amd64/bwlyKhYx/build/factory-crossbuild-244/pfSense/tmp/FreeBSD-src/sys/pfSense amd64
    FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0)
    VT(vga): resolution 640x480
    CPU: AMD G-T40E Processor (1000.02-MHz K8-class CPU)
      Origin="AuthenticAMD"  Id=0x500f20  Family=0x14  Model=0x2  Stepping=0
      Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
      Features2=0x802209<SSE3,MON,SSSE3,CX16,POPCNT>
      AMD Features=0x2e500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM>
      AMD Features2=0x35ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,IBS,SKINIT,WDT>
      SVM: NP,NRIP,NAsids=8
      TSC: P-state invariant, performance statistics
    real memory  = 4815060992 (4592 MB)
    avail memory = 4054708224 (3866 MB)
    Event timer "LAPIC" quality 600
    ACPI APIC Table: <CORE   COREBOOT>
    FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
    FreeBSD/SMP: 1 package(s) x 2 core(s)
    ioapic0 <Version 2.1> irqs 0-23 on motherboard
    SMP: AP CPU #1 Launched!
    Timecounter "TSC" frequency 1000023140 Hz quality 800
    ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
    ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff80685700, 0) error 1
    random: entropy device external interface
    ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
    ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff806857b0, 0) error 1
    ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
    ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff80685860, 0) error 1
    iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
    iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff806ac730, 0) error 1
    iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
    iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff806ac7e0, 0) error 1
    iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
    iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff806ac890, 0) error 1
    wlan: mac acl policy registered
    kbd0 at kbdmux0
    netmap: loaded module
    module_register_init: MOD_LOAD (vesa, 0xffffffff8120faa0, 0) error 19
    nexus0
    vtvga0: <VT VGA driver> on motherboard
    cryptosoft0: <software crypto> on motherboard
    padlock0: No ACE support.
    acpi0: <CORE COREBOOT> on motherboard
    acpi0: Power Button (fixed)
    cpu0: <ACPI CPU> on acpi0
    cpu1: <ACPI CPU> on acpi0
    atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
    atrtc0: registered as a time-of-day clock, resolution 1.000000s
    Event timer "RTC" frequency 32768 Hz quality 0
    attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
    Timecounter "i8254" frequency 1193182 Hz quality 0
    Event timer "i8254" frequency 1193182 Hz quality 100
    Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
    acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
    hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
    Timecounter "HPET" frequency 14318180 Hz quality 950
    Event timer "HPET" frequency 14318180 Hz quality 550
    Event timer "HPET1" frequency 14318180 Hz quality 450
    pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
    pci0: <ACPI PCI bus> on pcib0
    pcib1: <ACPI PCI-PCI bridge> irq 16 at device 4.0 on pci0
    pci1: <ACPI PCI bus> on pcib1
    re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0x1000-0x10ff mem 0xf7a00000-0xf7a00fff,0xf7900000-0xf7903fff irq 16 at device 0.0 on pci1
    re0: Using 1 MSI-X message
    re0: ASPM disabled
    re0: Chip rev. 0x2c000000
    re0: MAC rev. 0x00200000
    miibus0: <MII bus> on re0
    rgephy0: <RTL8169S/8110S/8211 1000BASE-T media interface> PHY 1 on miibus0
    rgephy0:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
    re0: Using defaults for TSO: 65518/35/2048
    re0: Ethernet address: 00:0d:b9:3a:4c:f8
    re0: netmap queues/slots: TX 1/256, RX 1/256
    pcib2: <ACPI PCI-PCI bridge> irq 17 at device 5.0 on pci0
    pci2: <ACPI PCI bus> on pcib2
    re1: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0x2000-0x20ff mem 0xf7c00000-0xf7c00fff,0xf7b00000-0xf7b03fff irq 17 at device 0.0 on pci2
    re1: Using 1 MSI-X message
    re1: ASPM disabled
    re1: Chip rev. 0x2c000000
    re1: MAC rev. 0x00200000
    miibus1: <MII bus> on re1
    rgephy1: <RTL8169S/8110S/8211 1000BASE-T media interface> PHY 1 on miibus1
    rgephy1:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
    re1: Using defaults for TSO: 65518/35/2048
    re1: Ethernet address: 00:0d:b9:3a:4c:f9
    re1: netmap queues/slots: TX 1/256, RX 1/256
    pcib3: <ACPI PCI-PCI bridge> irq 18 at device 6.0 on pci0
    pci3: <ACPI PCI bus> on pcib3
    re2: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0x3000-0x30ff mem 0xf7e00000-0xf7e00fff,0xf7d00000-0xf7d03fff irq 18 at device 0.0 on pci3
    re2: Using 1 MSI-X message
    re2: ASPM disabled
    re2: Chip rev. 0x2c000000
    re2: MAC rev. 0x00200000
    miibus2: <MII bus> on re2
    rgephy2: <RTL8169S/8110S/8211 1000BASE-T media interface> PHY 1 on miibus2
    rgephy2:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
    re2: Using defaults for TSO: 65518/35/2048
    re2: Ethernet address: 00:0d:b9:3a:4c:fa
    re2: netmap queues/slots: TX 1/256, RX 1/256
    ahci0: <AMD SB7x0/SB8x0/SB9x0 AHCI SATA controller> port 0x4010-0x4017,0x4020-0x4023,0x4018-0x401f,0x4024-0x4027,0x4000-0x400f mem 0xf7f08000-0xf7f083ff irq 19 at device 17.0 on pci0
    ahci0: AHCI v1.20 with 6 6Gbps ports, Port Multiplier supported
    ahci0: quirks=0x22000<ATI_PMP_BUG,1MSI>
    ahcich0: <AHCI channel> at channel 0 on ahci0
    ahcich1: <AHCI channel> at channel 1 on ahci0
    ahcich2: <AHCI channel> at channel 2 on ahci0
    ahcich3: <AHCI channel> at channel 3 on ahci0
    ahcich4: <AHCI channel> at channel 4 on ahci0
    ahcich5: <AHCI channel> at channel 5 on ahci0
    ohci0: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xf7f04000-0xf7f04fff irq 18 at device 18.0 on pci0
    usbus0 on ohci0
    usbus0: 12Mbps Full Speed USB v1.0
    ehci0: <AMD SB7x0/SB8x0/SB9x0 USB 2.0 controller> mem 0xf7f08400-0xf7f084ff irq 17 at device 18.2 on pci0
    usbus1: EHCI version 1.0
    usbus1 on ehci0
    usbus1: 480Mbps High Speed USB v2.0
    ohci1: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xf7f05000-0xf7f05fff irq 18 at device 19.0 on pci0
    usbus2 on ohci1
    usbus2: 12Mbps Full Speed USB v1.0
    ehci1: <AMD SB7x0/SB8x0/SB9x0 USB 2.0 controller> mem 0xf7f08500-0xf7f085ff irq 17 at device 19.2 on pci0
    usbus3: EHCI version 1.0
    usbus3 on ehci1
    usbus3: 480Mbps High Speed USB v2.0
    isab0: <PCI-ISA bridge> at device 20.3 on pci0
    isa0: <ISA bus> on isab0
    pcib4: <ACPI PCI-PCI bridge> at device 20.4 on pci0
    pci4: <ACPI PCI bus> on pcib4
    ohci2: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xf7f06000-0xf7f06fff irq 18 at device 20.5 on pci0
    usbus4 on ohci2
    usbus4: 12Mbps Full Speed USB v1.0
    pcib5: <ACPI PCI-PCI bridge> at device 21.0 on pci0
    pci5: <ACPI PCI bus> on pcib5
    ohci3: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xf7f07000-0xf7f07fff at device 22.0 on pci0
    usbus5 on ohci3
    usbus5: 12Mbps Full Speed USB v1.0
    ehci2: <AMD SB7x0/SB8x0/SB9x0 USB 2.0 controller> mem 0xf7f08600-0xf7f086ff at device 22.2 on pci0
    usbus6: EHCI version 1.0
    usbus6 on ehci2
    usbus6: 480Mbps High Speed USB v2.0
    acpi_button0: <Power Button> on acpi0
    orm0: <ISA Option ROM> at iomem 0xee800-0xeffff on isa0
    acpi_throttle0: <ACPI CPU Throttling> on cpu0
    acpi_throttle1: <ACPI CPU Throttling> on cpu1
    acpi_throttle1: failed to attach P_CNT
    device_attach: acpi_throttle1 attach returned 6
    Timecounters tick every 1.000 msec
    ugen1.1: <ATI EHCI root HUB> at usbus1
    ugen6.1: <ATI EHCI root HUB> at usbus6
    uhub0: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
    uhub1: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus6
    ugen5.1: <ATI OHCI root HUB> at usbus5
    ugen2.1: <ATI OHCI root HUB> at usbus2
    uhub2: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5
    uhub3: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
    ugen0.1: <ATI OHCI root HUB> at usbus0
    ugen4.1: <ATI OHCI root HUB> at usbus4
    uhub4: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
    uhub5: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
    ugen3.1: <ATI EHCI root HUB> at usbus3
    uhub6: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
    ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
    uhub3: 5 ports with 5 removable, self powered
    uhub5: 2 ports with 2 removable, self powered
    uhub4: 5 ports with 5 removable, self powered
    ada0: <INTEL SSDMCEAC030B3 LLLi> ACS-2 ATA SATA 3.x device
    uhub2: ada0: Serial Number CVLI446200X7030H
    4 ports with 4 removable, self powered
    ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
    ada0: Command Queueing enabled
    ada0: 28626MB (58626288 512 byte sectors)
    Trying to mount root from ufs:/dev/ufsid/5358b0d792733dcf [rw]...
    WARNING: / was not properly dismounted
    uhub1: 4 ports with 4 removable, self powered
    uhub0: 5 ports with 5 removable, self powered
    ugen6.2: <Generic Flash Card ReaderWriter> at usbus6
    umass0 on uhub1
    umass0: <Generic Flash Card ReaderWriter, class 0/0, rev 2.01/1.00, addr 2> on usbus6
    uhub6: 5 ports with 5 removable, self powered
    da0 at umass-sim0 bus 0 scbus6 target 0 lun 0
    da0: <Multiple Card  Reader 1.00> Removable Direct Access SPC-2 SCSI device
    da0: Serial Number 058F63666485
    da0: 40.000MB/s transfers
    da0: Attempt to query device size failed: NOT READY, Medium not present
    da0: quirks=0x2<NO_6_BYTE>
    random: unblocking device.
    
    

  • Netgate Administrator

    Hmm, that looks a lot like the wrong console type set as primary. Did you import the config into the APU from something else originally?

    Do you have a backup config file you can check? Or use to re-install if needed.

    Steve



  • @stephenw10 said in netgate APU4 bricked after initiating update in web console:

    nothing was changed in the system. this came as a preinstalled pfsense image from netgate. the only thing I have ever done was press the upgrade button in the web console. no config import was performed. backup config - none unless pfsense itself does one.


  • Netgate Administrator

    Hmm, you could try booting into single user mode from the loader menu. But to be honest if you have a backup of the config file it's going to be quickest to reinstall. And it will guarantee you a clean filesystem.

    Steve



  • I tried booting to single user mode by specifying boot -s at the kernel loader shell, but the same problem occurred where console output stopped. I worked with some people on the freebsd IRC channel on freenode and it appears somehow multiple files were corrupted. /boot/loader.4th was corrupted totally and /etc/ttys was truncated. I ended up doing a fresh install after taking a forensic backup of the drive, which is unfortunate that I lost all my settings.


  • Netgate Administrator

    You were not able to recover the config during the install? It must have been a significantly damaged file system if so.

    Steve


Log in to reply