SG-2220 boot issues

scottlindner

I think my SG-2220 eMMC has failed but I don't really know what I'm looking at or how to diagnose and repair. I ordered a M.2 SATA SSD (can't find one locally, but gonna keep trying) hoping that is all I need to get this working again. BUT, if there is a way to repair this today so I at least have internet I would be really happy. Below is what I grabbed from the console.

SeaBIOS (version SageBIOS-20170207_214234-localhost.localdomain)


iPXE (http://ipxe.org) 00:14.0 C100 PCI2.10 PnP PMM+7FB7D480+7FADD480 C100





iPXE (http://ipxe.org) 00:14.1 C200 PCI2.10 PnP PMM 7FB7D480 7FADD480 C200


iPXE (http://ipxe.org) 00:14.2 C300 PCI2.10 PnP PMM 7FB7D480 7FADD480 C300


iPXE (http://ipxe.org) 00:14.3 C400 PCI2.10 PnP PMM 7FB7D480 7FADD480 C400

Press F12 for boot menu.


F1  pfSense

F6 PXE
Boot:  F1
Loading /boot/loader.conf.local
/
|          __
    _ __  / _|___  ___ _ __  ___  ___      _
   | '_ \| |_/ __|/ _ \ '_ \/ __|/ _ \   _| |_
   | |_) |  _\__ \  __/ | | \__ \  __/  |_   _|
   | .__/|_| |___/\___|_| |_|___/\___|    |_|
   |_|


 +-----Welcome to Netgate pfSense Plus-----+      __________________________
 |                                         |     /                       ___\
 |  1. Boot Multi user [Enter]             |    |                      /`
 |  2. Boot Single user                    |    |                     /    :-|
 |  3. Escape to loader prompt             |    |      _________  ___/    /_ |
 |  4. Reboot                              |    |    /` ____   / /__    ___/ |
 |  5. Cons: Serial                        |    |   /  /   /  /    /   /     |
 |                                         |    |  /  /___/  /    /   /      |
 |  Options:                               |    | /   ______/    /   /  _    |
 |  6. Kernel: default/kernel (1 of 2)     |    |/   /          /   / _| |_  |
 |  7. Boot Options                        |        /          /___/ |_   _| |
 |                                         |       /                   |_|   |
 |                                         |      /_________________________/
 +-----------------------------------------+                                  \
   Autoboot in 0 seconds, hit [Enter] to boot or any other key to stop

Loading kernel...
/boot/kernel/kernel text=0x1a51484 data=0x140 data=0x1b625a0 syms=[0x8+0x1c1490+0x8+0x1c7733]
Loading configured modules...
can't find '/boot/entropy'
can't find '/etc/hostid'
KDB: debugger backends: ddb
KDB: current backend: ddb
---<<BOOT>>---
Copyright (c) 1992-2021 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.2-STABLE plus-RELENG_21_05_2-n202579-3b8ea9b365a pfSense amd64
FreeBSD clang version 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c611aa2)
VT(vga): resolution 640x480
CPU: Intel(R) Atom(TM) CPU  C2338  @ 1.74GHz (1750.04-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x406d8  Family=0x6  Model=0x4d  Stepping=8
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x43d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,AESNI,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 2143150080 (2043 MB)
avail memory = 2014986240 (1921 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <CORE   COREBOOT>
WARNING: L1 data cache covers fewer APIC IDs than a core (0 < 1)
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
arc4random: no preloaded entropy cache
ioapic0 <Version 2.0> irqs 0-23 on motherboard
Launching APs: 1
Timecounter "TSC" frequency 1750043547 Hz quality 1000
wlan: mac acl policy registered
random: entropy device external interface
ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff8073c7e0, 0) error 1
ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff8073c890, 0) error 1
ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff8073c940, 0) error 1
iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff807641d0, 0) error 1
iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff80764280, 0) error 1
iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff80764330, 0) error 1
WARNING: Device "g_ctl" is Giant locked and may be deleted before FreeBSD 14.0.
000.000018 [4344] netmap_init               netmap: loaded module
WARNING: Device "pci" is Giant locked and may be deleted before FreeBSD 14.0.
module_register_init: MOD_LOAD (vesa, 0xffffffff814050f0, 0) error 19
WARNING: Device "kbd" is Giant locked and may be deleted before FreeBSD 14.0.
kbd0 at kbdmux0
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
WARNING: Device "spkr" is Giant locked and may be deleted before FreeBSD 14.0.
[ath_hal] loaded
mlx5en: Mellanox Ethernet driver 3.6.0 (December 2020)
nexus0
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
acpi0: <CORE COREBOOT> on motherboard
acpi0: Power Button (fixed)
hpet0: <High Precision Event Timer> iomem 0xfed03000-0xfed033ff on acpi0
hpet0: invalid period
device_attach: hpet0 attach returned 6
cpu0: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed03000-0xfed033ff on acpi0
hpet0: invalid period
device_attach: hpet0 attach returned 6
atrtc0: <AT realtime clock> port 0x70-0x77 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> mem 0xdfee0000-0xdfefffff irq 16 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> mem 0xdff00000-0xdff1ffff irq 16 at device 2.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> mem 0xdff20000-0xdff3ffff irq 20 at device 3.0 on pci0
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> mem 0xdff40000-0xdff5ffff irq 23 at device 4.0 on pci0
pci4: <ACPI PCI bus> on pcib4
pci0: <base peripheral, IOMMU> at device 15.0 (no driver attached)
igb0: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0x1000-0x101f mem 0xdff60000-0xdff7ffff,0xdffe0000-0xdffe3fff irq 18 at device 20.0 on pci0
igb0: Using 1024 TX descriptors and 1024 RX descriptors
igb0: Using 2 RX queues 2 TX queues
igb0: Using MSI-X interrupts with 3 vectors
igb0: Ethernet address: 00:08:a2:0b:be:3e
igb0: netmap queues/slots: TX 2/1024, RX 2/1024
igb1: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0x1020-0x103f mem 0xdff80000-0xdff9ffff,0xdffe4000-0xdffe7fff irq 19 at device 20.1 on pci0
igb1: Using 1024 TX descriptors and 1024 RX descriptors
igb1: Using 2 RX queues 2 TX queues
igb1: Using MSI-X interrupts with 3 vectors
igb1: Ethernet address: 00:08:a2:0b:be:3f
igb1: netmap queues/slots: TX 2/1024, RX 2/1024
igb2: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0x1040-0x105f mem 0xdffa0000-0xdffbffff,0xdffe8000-0xdffebfff irq 20 at device 20.2 on pci0
igb2: Setup of Shared code failed, error -2
igb2: IFDI_ATTACH_PRE failed 6
device_attach: igb2 attach returned 6
igb2: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0x1060-0x107f mem 0xdffc0000-0xdffdffff,0xdffec000-0xdffeffff irq 21 at device 20.3 on pci0
igb2: Setup of Shared code failed, error -2
igb2: IFDI_ATTACH_PRE failed 6
device_attach: igb2 attach returned 6
ehci0: <Intel Avoton USB 2.0 controller> mem 0xdfff1400-0xdfff17ff irq 22 at device 22.0 on pci0
usbus0: EHCI version 1.0
usbus0 on ehci0
usbus0: 480Mbps High Speed USB v2.0
ahci0: <Intel Avoton AHCI SATA controller> port 0x10c0-0x10c7,0x10e0-0x10e3,0x10c8-0x10cf,0x10e4-0x10e7,0x1080-0x109f mem 0xdfff0000-0xdfff07ff irq 23 at device 23.0 on pci0
ahci0: AHCI v1.30 with 4 3Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahci1: <Intel Avoton AHCI SATA controller> port 0x10d0-0x10d7,0x10e8-0x10eb,0x10d8-0x10df,0x10ec-0x10ef,0x10a0-0x10bf mem 0xdfff0800-0xdfff0fff irq 16 at device 24.0 on pci0
ahci1: AHCI v1.30 with 2 6Gbps ports, Port Multiplier not supported
ahcich4: <AHCI channel> at channel 0 on ahci1
ahcich5: <AHCI channel> at channel 1 on ahci1
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
orm0: <ISA Option ROM> at iomem 0xc0000-0xc0fff pnpid ORM0000 on isa0
uart0: <16550 or compatible> at port 0x3f8 irq 4 on isa0
uart1: <16550 or compatible> at port 0x2f8 irq 3 flags 0x10 on isa0
uart1: console (115200,n,8,1)
est0: <Enhanced SpeedStep Frequency Control> on cpu0
Timecounters tick every 1.000 msec
igb2: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0x1040-0x105f mem 0xdffa0000-0xdffbffff,0xdffe8000-0xdffebfff irq 20 at device 20.2 on pci0
ugen0.1: <Intel EHCI root HUB> at usbus0
igb2: Setup of Shared code failed, error -2
igb2: IFDI_ATTACH_PRE failed 6
device_attach: igb2 attach returned 6
igb2: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0x1060-0x107f mem 0xdffc0000-0xdffdffff,0xdffec000-0xdffeffff irq 21 at device 20.3 on pci0
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
igb2: Setup of Shared code failed, error -2
igb2: IFDI_ATTACH_PRE failed 6
device_attach: igb2 attach returned 6
igb2: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0x1040-0x105f mem 0xdffa0000-0xdffbffff,0xdffe8000-0xdffebfff irq 20 at device 20.2 on pci0
igb2: Setup of Shared code failed, error -2
igb2: IFDI_ATTACH_PRE failed 6
device_attach: igb2 attach returned 6
igb2: <Intel(R) PRO/1000 PCI-Express Network Driver> port 0x1060-0x107f mem 0xdffc0000-0xdffdffff,0xdffec000-0xdffeffff irq 21 at device 20.3 on pci0
igb2: Setup of Shared code failed, error -2
igb2: IFDI_ATTACH_PRE failed 6
device_attach: igb2 attach returned 6
random: unblocking device.
arc4random: no preloaded entropy cache
Trying to mount root from ufs:/dev/ufsid/5911c7e65e0911e0 [rw]...
Root mount waiting for: usbus0
Root mount waiting for: usbus0
Root mount waiting for: usbus0
Root mount waiting for: usbus0
uhub0: 8 ports with 8 removable, self powered
ugen0.2: <vendor 0x8087 product 0x07db> at usbus0
uhub1 on uhub0
uhub1: <vendor 0x8087 product 0x07db, class 9/0, rev 2.00/0.02, addr 2> on usbus0
Root mount waiting for: usbus0
uhub1: 4 ports with 4 removable, self powered
Root mount waiting for: usbus0
ugen0.3: <Generic Ultra Fast Media> at usbus0
umass0 on uhub1
umass0: <Generic Ultra Fast Media, class 0/0, rev 2.00/1.98, addr 3> on usbus0
da0 at umass-sim0 bus 0 scbus6 target 0 lun 0
da0: <Generic Ultra HS-COMBO 1.98> Removable Direct Access SCSI device
da0: Serial Number 000000225001
da0: 40.000MB/s transfers
da0: 3616MB (7405568 512 byte sectors)
da0: quirks=0x2<NO_6_BYTE>
mountroot: waiting for device /dev/ufsid/5911c7e65e0911e0...
WARNING: / was not properly dismounted
WARNING: /: TRIM flag on fs but disk does not support TRIM
Configuring crash dumps...
Using /dev/label/swap0 for dump device.
** SU+J Recovering /dev/ufsid/5911c7e65e0911e0
** Reading 21200896 byte journal from inode 4.
** Building recovery table.
** Resolving unreferenced inode list.
** Processing journal entries.
(da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 00 00 01 4f 00 00 40 00
(da0:umass-sim0:0:0:0): CAM status: SCSI Status Error
(da0:umass-sim0:0:0:0): SCSI status: Check Condition
(da0:umass-sim0:0:0:0): SCSI sense: MEDIUM ERROR asc:3,0 (Peripheral device write fault)
(da0:umass-sim0:0:0:0): Retrying command (per sense data)
(da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 00 00 01 4f 00 00 40 00
(da0:umass-sim0:0:0:0): CAM status: SCSI Status Error
(da0:umass-sim0:0:0:0): SCSI status: Check Condition
(da0:umass-sim0:0:0:0): SCSI sense: NOT READY asc:4,ff (Vendor Specific ASCQ)
(da0:umass-sim0:0:0:0): Retrying command (per sense data)
(da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 00 00 01 4f 00 00 40 00
(da0:umass-sim0:0:0:0): CAM status: SCSI Status Error
(da0:umass-sim0:0:0:0): SCSI status: Check Condition
(da0:umass-sim0:0:0:0): SCSI sense: NOT READY asc:4,ff (Vendor Specific ASCQ)
(da0:umass-sim0:0:0:0): Retrying command (per sense data)
(da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 00 00 01 4f 00 00 40 00
(da0:umass-sim0:0:0:0): CAM status: SCSI Status Error
(da0:umass-sim0:0:0:0): SCSI status: Check Condition
(da0:umass-sim0:0:0:0): SCSI sense: NOT READY asc:4,ff (Vendor Specific ASCQ)
(da0:umass-sim0:0:0:0): Retrying command (per sense data)
(da0:umass-sim0:0:0:0): WRITE(10). CDB: 2a 00 00 00 01 4f 00 00 40 00
(da0:umass-sim0:0:0:0): CAM status: SCSI Status Error
(da0:umass-sim0:0:0:0): SCSI status: Check Condition
(da0:umass-sim0:0:0:0): SCSI sense: NOT READY asc:4,ff (Vendor Specific ASCQ)
(da0:umass-sim0:0:0:0): Error 5, Retries exhausted
/dev/ufsid/5911c7e65e0911e0: Unable to write cylinder group 0
/dev/ufsid/5911c7e65e0911e0: UNEXPECTED SU+J INCONSISTENCY
/dev/ufsid/5911c7e65e0911e0: INTERNAL ERROR: GOT TO reply()
/dev/ufsid/5911c7e65e0911e0: UNEXPECTED SOFT UPDATE INCONSISTENCY; RUN fsck MANUALLY.
(da0:umass-sim0:0:0:0): READ(10). CDB: 28 00 00 70 ff ff 00 00 01 00
(da0:umass-sim0:0:0:0): CAM status: SCSI Status Error
(da0:umass-sim0:0:0:0): SCSI status: Check Condition
(da0:umass-sim0:0:0:0): SCSI sense: NOT READY asc:3a,0 (Medium not present)
(da0:umass-sim0:0:0:0): Error 6, Unretryable error
(da0:umass-sim0:0:0:0): READ(10). CDB: 28 00 00 70 ff ff 00 00 01 00
(da0:umass-sim0:0:0:0): CAM status: SCSI Status Error
(da0:umass-sim0:0:0:0): SCSI status: Check Condition
(da0:umass-sim0:0:0:0): SCSI sense: NOT READY asc:3a,0 (Medium not present)
(da0:umass-sim0:0:0:0): Error 6, Unretryable error
Preen mode recommended running a check that will be performed now.
WARNING: Trying to recover filesystem from inconsistency...
Can't stat /dev/ufsid/5911c7e65e0911e0: No such file or directory
Can't stat /dev/ufsid/5911c7e65e0911e0: No such file or directory
Can't stat /dev/ufsid/5911c7e65e0911e0: No such file or directory
Can't stat /dev/ufsid/5911c7e65e0911e0: No such file or directory
Can't stat /dev/ufsid/5911c7e65e0911e0: No such file or directory
Can't stat /dev/ufsid/5911c7e65e0911e0: No such file or directory
Can't stat /dev/ufsid/5911c7e65e0911e0: No such file or directory
Can't stat /dev/ufsid/5911c7e65e0911e0: No such file or directory
Can't stat /dev/ufsid/5911c7e65e0911e0: No such file or directory
Can't stat /dev/ufsid/5911c7e65e0911e0: No such file or directory
Can't stat /dev/ufsid/5911c7e65e0911e0: No such file or directory
Can't stat /dev/ufsid/5911c7e65e0911e0: No such file or directory
Can't stat /dev/ufsid/5911c7e65e0911e0: No such file or directory
Can't stat /dev/ufsid/5911c7e65e0911e0: No such file or vm_fault: pager read error, pid 77 (sh)
pid 77 (sh), jid 0, uid 0: exited on signal 10
directory
Can't stat /dev/ufsid/5911c7e65e0911e0: No such file or directory
Can't stat /dev/ufsid/5911c7e65e0911e0: No such file or directory
Can't stat /dev/ufsid/5911c7e65e0911e0: No such file or directory
Can't stat /dev/ufsid/5911c7e65e0911e0: No such file or directory
Can't stat /dev/ufsid/5911c7e65e0911e0: No such file or directory
Can't stat /dev/ufsid/5911c7e65e0911e0: No such file or directory
Can't stat /dev/ufsid/5911c7e65e0911e0: No such file or directory
Can't stat /dev/ufsid/5911c7e65e0911e0: No such file or directory
ERROR: Impossible to mount filesystem, use interactive shell to attempt to recover it
Bus error
/etc/pfSense-rc: /sbin/reboot: Device not configured
/etc/pfSense-rc: /sbin/kldstat: Device not configured
.: cannot open /etc/rc.ramdisk_functions.sh: Device not configured
Enter full pathname of shell or RETURN for /bin/sh: /bin/sh
vm_fault: pager read error, pid 80 (sh)
pid 80 (sh), jid 0, uid 0: exited on signal 10
2017-02-06T18:43:04.678449-07:00  init 1 - - single user shell terminated, restarting
Enter full pathname of shell or RETURN for /bin/sh:
vm_fault: pager read error, pid 81 (sh)
pid 81 (sh), jid 0, uid 0: exited on signal 10
2017-02-06T18:43:06.382534-07:00  init 1 - - single user shell terminated, restarting
Enter full pathname of shell or RETURN for /bin/sh:

scottlindner

This is what I ordered for delivery tomorrow. The docs say it can't be NVME. As far as I can tell, this isn't NVME so I'm hoping this will be OK.
https://www.amazon.com/Kingdata-M-2-2260-512GB-High-Performance/dp/B08J2KQZHD/

And just for giggles, I'm posting here using a USB WiFi card in my desktop that is connected via the console to the SG-2220 using the mobile hotspot on my phone. Total suck! But it is somewhat functional. LOL

akuma1x

@scottlindner That amount of storage might be too large for the Netgate box to recognize, but I'm not sure. You could install it and see what happens.

Not sure if you are aware, but here are the instructions:

https://docs.netgate.com/pfsense/en/latest/solutions/sg-2220/m-2-sata-installation.html

This is the one I bought for my SG-2220, middle of 2020, so I could update my box while working from home during COVID:

https://www.amazon.com/gp/product/B077H276GQ?th=1

scottlindner

@akuma1x

Thank you for that. I just ordered the exact one because I just want it to work. Both are returnable so I'll try the big one first and if that doesn't work I'll try the one you already know works.

From what you know, does my console output suggest my eMMC is toast and I need this M.2 SSD? I really just want it working now and I'll still do the M.2 SSD when it arrives.

akuma1x

@scottlindner I'm a total novice when it comes to FreeBSD, I know just enough to get myself into trouble. But, some of the stuff near the end of that console dump looks like the drive has failed. Maybe one of the experts could chime in here and tell you for sure.

Do you have a backup of your config? If you do, and you can get your hands on a restore image of pfsense, it is possible that you can re-format the internal storage. It might just be messed up enough that it won't allow a successful boot of the operating system, maybe...

viragomann

@scottlindner said in SG-2220 boot issues:

Trying to mount root from ufs:/dev/ufsid/5911c7e65e0911e0 [rw]...

Do you have a backup?

If yes, reinstall pfSense using ZFS and import the backup.

If you really want to recover the UFS filesystem boot pfSense into the single user mode and enter the command prompt and try

/sbin/fsck -y -t ufs

scottlindner

@akuma1x Yes. I have always feared this situation so every time I make a config change I save the file. Restoring should be trivial once I have the pfSense installed on the new SSD.

viragomann

@scottlindner
Seems not to me that the device is damaged, but the filesystem is not clean.

ERROR: Impossible to mount filesystem, use interactive shell to attempt to recover it

scottlindner

@viragomann good catch. I have been trying to get to a shell so I can run fsck and it isn't finding a file system. When I try to boot into single user mode it prompts me with mountroot> and when I enter the device name it fails. But maybe I don't really know what I'm doing. Or maybe the device is so failed it isn't even reporting as a device anymore?

Let me get more console output for you if you don't mind looking at that.

SteveITS

@scottlindner Have you looked at https://docs.netgate.com/pfsense/en/latest/troubleshooting/filesystem-check.html?

scottlindner

@steveits said in SG-2220 boot issues:

@scottlindner Have you looked at https://docs.netgate.com/pfsense/en/latest/troubleshooting/filesystem-check.html?

I have. I get stuck at mountroot> It doesn't seem to find the eMMC device. And now I can't even get to the pfSense menu via the console anymore. I wonder if the whole thing is dying and not just the storage.

EDIT: I bet the eMMC device just completely died now that I think about it.

This is what I get now from the console and the #'s just keep tallying up forever.

SeaBIOS (version SageBIOS-20170207_214234-localhost.localdomain)


iPXE (http://ipxe.org) 00:14.0 C100 PCI2.10 PnP PMM+7FB7D480+7FADD480 C100





iPXE (http://ipxe.org) 00:14.1 C200 PCI2.10 PnP PMM 7FB7D480 7FADD480 C200


iPXE (http://ipxe.org) 00:14.2 C300 PCI2.10 PnP PMM 7FB7D480 7FADD480 C300


iPXE (http://ipxe.org) 00:14.3 C400 PCI2.10 PnP PMM 7FB7D480 7FADD480 C400

Press F12 for boot menu.

Select boot device:

1. USB MSC Drive Generic Ultra HS-COMBO 1.98
2. iPXE (PCI 00:14.0)
3. iPXE (PCI 00:14.1)
4. iPXE (PCI 00:14.2)
5. iPXE (PCI 00:14.3)
6. Payload [setup]


F1  pfSense

F6 PXE
Boot:  F1 ##

scottlindner

@viragomann said in SG-2220 boot issues:

@scottlindner said in SG-2220 boot issues:

Trying to mount root from ufs:/dev/ufsid/5911c7e65e0911e0 [rw]...

Do you have a backup?

If yes, reinstall pfSense using ZFS and import the backup.

If you really want to recover the UFS filesystem boot pfSense into the single user mode and enter the command prompt and try
/sbin/fsck -y -t ufs

I'm not able to boot into signle user mode. I put the pfSense installation on a USB stick following a guide cited in this discussion and booted into single user mode with that installer and ran the following:

/sbin/fsck -y -t ufs /dev/ufsid/5911c7e65e0911e0

And it is spewing tons of errors just like this (already filled the Putty console buffer so I can't go back to read any of the beginning stuff that was slightly different).

CANNOT READ BLK: 1311232
UNEXPECTED SOFT UPDATE INCONSISTENCY

CONTINUE? yes

THE FOLLOWING DISK SECTORS COULD NOT BE READ: 1311232, 1311233, 1311234, 1311235, 1311236, 1311237, 1311238, 1311239, 1311240, 1311241, 1311242, 1311243, 1311244, 1311245, 1311246, 1311247, 1311248, 1311249, 1311250, 1311251, 1311252, 1311253, 1311254, 1311255, 1311256, 1311257, 1311258, 1311259, 1311260, 1311261, 1311262, 1311263, 1311264, 1311265, 1311266, 1311267, 1311268, 1311269, 1311270, 1311271, 1311272, 1311273, 1311274, 1311275, 1311276, 1311277, 1311278, 1311279, 1311280, 1311281, 1311282, 1311283, 1311284, 1311285, 1311286, 1311287, 1311288, 1311289, 1311290, 1311291, 1311292, 1311293, 1311294, 1311295, 1311296, 1311297, 1311298, 1311299, 1311300, 1311301, 1311302, 1311303, 1311304, 1311305, 1311306, 1311307, 1311308, 1311309, 1311310, 1311311, 1311312, 1311313, 1311314, 1311315, 1311316, 1311317, 1311318, 1311319, 1311320, 1311321, 1311322, 1311323, 1311324, 1311325, 1311326, 1311327, 1311328, 1311329, 1311330, 1311331, 1311332, 1311333, 1311334, 1311335, 1311336, 1311337, 1311338, 1311339, 1311340, 1311341, 1311342, 1311343, 1311344, 1311345, 1311346, 1311347, 1311348, 1311349, 1311350, 1311351, 1311352, 1311353, 1311354, 1311355, 1311356, 1311357, 1311358, 1311359,

CANNOT READ BLK: 1311360
UNEXPECTED SOFT UPDATE INCONSISTENCY

CONTINUE? yes

THE FOLLOWING DISK SECTORS COULD NOT BE READ: 1311360, 1311361, 1311362, 1311363, 1311364, 1311365, 1311366, 1311367, 1311368, 1311369, 1311370, 1311371, 1311372, 1311373, 1311374, 1311375, 1311376, 1311377, 1311378, 1311379, 1311380, 1311381, 1311382, 1311383, 1311384, 1311385, 1311386, 1311387, 1311388, 1311389, 1311390, 1311391, 1311392, 1311393, 1311394, 1311395, 1311396, 1311397, 1311398, 1311399, 1311400, 1311401, 1311402, 1311403, 1311404, 1311405, 1311406, 1311407, 1311408, 1311409, 1311410, 1311411, 1311412, 1311413, 1311414, 1311415, 1311416, 1311417, 1311418, 1311419, 1311420, 1311421, 1311422, 1311423, 1311424, 1311425, 1311426, 1311427, 1311428, 1311429, 1311430, 1311431, 1311432, 1311433, 1311434, 1311435, 1311436, 1311437, 1311438, 1311439, 1311440, 1311441, 1311442, 1311443, 1311444, 1311445, 1311446, 1311447, 1311448, 1311449, 1311450, 1311451, 1311452, 1311453, 1311454, 1311455, 1311456, 1311457, 1311458, 1311459, 1311460, 1311461, 1311462, 1311463, 1311464, 1311465, 1311466, 1311467, 1311468, 1311469, 1311470, 1311471, 1311472, 1311473, 1311474, 1311475, 1311476, 1311477, 1311478, 1311479, 1311480, 1311481, 1311482, 1311483, 1311484, 1311485, 1311486, 1311487,

CANNOT READ BLK: 1311488
UNEXPECTED SOFT UPDATE INCONSISTENCY

CONTINUE? yes

I think this confirms the eMMC is dead, right?

scottlindner

I was able to reinstall pfSense+ onto a 32GB USB stick and restore the last config I had saved off from about 3 weeks ago which has all of my latest changes in it. I'm still going to add the M.2 SATA SSD when it arrives tomorrow and install again. I could not get it to install to the built in eMMC (/dev/da0). I think the eMMC is just completely toast.

I'll update again tomorrow after I get the new storage installed and see how well it all works out.

stephenw10

I agree it looks like the eMMC is damaged, those read errors do not look like just a filesystem problem.
Since you were able to install to and run from another USB drive though I would expect an m.2 drive to work fine.

Steve

scottlindner

@stephenw10 it appears every sector was marked bad. The eMMC max size is 0.

I'm happy with the fix. I didn't even know a USB install was possible until I started thinking back to the drive selection. I had to use a hub since it has one USB port which is fine. Looking forward to the M.2 to show up.

So glad I feared this situation years ago and got in the habit to save off my config after every change.

akuma1x

@scottlindner If it's possible at this point, you could do what I did with the additional storage I added to my SG-2220. Erase the internal EMMC storage, and leave it empty. That way, it won't attempt to boot from that volume. Then, simply format and install pfsense on your new internal M.2 drive.

scottlindner

@akuma1x I like the suggestion. I don't think it can attempt to boot from the eMMC because the size is now 0 after the fsck. I'm not 100% certain I know how this works but I think it was MBR and at 0 max size it can't have a boot record. Assuming I understand that is how it works. My firewall seems to be booting faster than it ever had in the past and that is off the USB. I'm hopeful it'll be even faster when the M.2 SATA SSD gets here later today.

I appreciate everyone's engagement on this. This is one of those ones that is so specific I didn't expect much interest in it.

scottlindner

@akuma1x I just installed the 512GB M.2 SATA and reinstalled pfSense using ZFS and it appears to work just fine and all of the storage is available. I don't know why I bought such a large drive. I guess price point stupidity. But this thing boots so much faster now I love it.