SG-1000 seems to have suicided on reboot

TauCeti

I'm convinced now that I own a cursed SG-1000, but it's the slow painful type.

I have had to reflash this guy four times now (owned it for a year). In the past I just put it down to beta firmware well being beta and borked updates but this time I was running 2.4.0-p1 stable since it came out. Situation was:

Webgui suddenly became unresponsive (this happens occasionally when the WAN drops out and comes back but pfsense for some reason goes to la la land).
Normally a power cycle is done to fix this, but the problem occured while I was logged in so I tried to investigate but SSH was very slow so I gave up on that idea.
Tried two restarts of webgui via the SSH menu, both failed.
Then chose restart option in SSH, and went with reroot (instead of reboot) since I assumed it would be faster.
SG-1000 never came back online. It was just stuck flashing it's LEDs to indicate it was booting.
Tried power cycles. No dice.
Had a console cable connected and it seemed to be stuck in a boot loop with the following output:


***** FILE SYSTEM MARKED CLEAN *****
Filesystems are clean, continuing...
Mounting filesystems...
random: unblocking device.

        __
 _ __  / _|___  ___ _ __  ___  ___
| '_ \| |_/ __|/ _ \ '_ \/ __|/ _ \
| |_) |  _\__ \  __/ | | \__ \  __/
| .__/|_| |___/\___|_| |_|___/\___|
|_|

Welcome to pfSense 2.4.2-RELEASE (Patch 1)...

...ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/ipsec /usr/local/lib/perl5/5.24/mach/CORE
Soft Float compatibility ldconfig path:
done.
panic: ufs_dirbad: /: bad dir ino 58890 at offset 512: mangled entry
Uptime: 17s
Automatic reboot in 15 seconds - press a key on the console to abort
--> Press a key on the console to reboot,
--> or switch off the system now.
Rebooting...

U-Boot SPL 2016.03 (Dec 28 2016 - 14:33:36)
Trying to boot from MMC
Card doesn't support part_switch
MMC partition switch failed
*** Warning - MMC partition switch failed, using default environment

reading u-boot.img
reading u-boot.img

U-Boot 2016.03 (Dec 28 2016 - 14:33:36 +0000)

       Watchdog enabled
I2C:   ready
DRAM:  512 MiB
MMC:   OMAP SD/MMC: 0, OMAP SD/MMC: 1

** Unable to use mmc 0:1 for loading the env **
Using default environment

Net:   <ethaddr>not set. Validating first E-fuse MAC
cpsw, usb_ether
starting USB...
USB0:   Port not available.
** Bad device usb 0 **
** Unrecognized filesystem type **
Hit any key to stop autoboot:  0
Booting from: mmc 1 ubldr.bin
reading ubldr.bin
227396 bytes read in 21 ms (10.3 MiB/s)
## Starting application at 0x88000000 ...
Consoles: U-Boot console
Compatible U-Boot API signature found @0x9ef3ab58

FreeBSD/armv6 U-Boot loader, Revision 1.2
(Tue Dec 12 15:07:07 CST 2017 root@buildbot2.netgate.com)

DRAM: 512MB
Number of U-Boot devices: 3
U-Boot env: loaderdev='mmc 1'
Found U-Boot device: disk
  Checking unit=1 slice= <auto>partition=<auto>... good.
Booting from disk1s2a:
console comconsole is invalid!
console vidconsole is invalid!
no valid consoles!
Available consoles:
    uboot
/boot/kernel/kernel data=0x10ff5a8+0x50a58 syms=[0x4+0xcea00+0x4+0xcac5d]

Hit [Enter] to boot immediately, or any other key for command prompt.
Booting [/boot/kernel/kernel]...
/boot/dtb/ufw.dtb size=0xb608
Loaded DTB from file 'ufw.dtb'.
Kernel entry at 0x88200100...
Kernel args: (null)
Copyright (c) 1992-2017 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.1-RELEASE-p6 #0 r313908+a5b33c9d1c4(RELENG_2_4): Tue Dec 12 15:08:51 CST 2017
    root@buildbot2.netgate.com:/xbuilder/crossbuild-242/work/obj-ufw-armv6/arm.armv6/xbuilder/crossbuild-242/pfSense/tmp/FreeBSD-src/sys/pfSense-uFW arm
FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0)
WARNING: too long kenv string, ignoring uboot.Fatboot=env exists loaderdev || env set loaderdev ${fatdev}; env exists UserFatboot && run UserFatboot; env exists trynetboot && env exists Netboot && run Netboot; echo Booting from: ${fatdev} ${bootfile}; fatload ${fatdev} ${loadaddr} ${bootfile} && go ${loadaddr};
CPU: ARM Cortex-A8 r3p2 (ECO: 0x00000000)
CPU Features:
  Thumb2, Security, VMSAv7
Optional instructions:
  UMULL, SMULL, SIMD(ext)
LoUU:2 LoC:3 LoUIS:1
Cache level 1:
 32KB/64B 4-way data cache WT WB Read-Alloc
 32KB/64B 4-way instruction cache Read-Alloc
Cache level 2:
 256KB/64B 8-way unified cache WT WB Read-Alloc Write-Alloc
real memory  = 536870912 (512 MB)
avail memory = 503279616 (479 MB)
Texas Instruments AM335x Processor, Revision ES1.2
ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_bss_fw, 0xc031550c, 0) error 1
ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_ibss_fw, 0xc03155d4, 0) error 1
ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_monitor_fw, 0xc03156a0, 0) error 1
iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_bss_fw, 0xc031f6b8, 0) error 1
iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_ibss_fw, 0xc031f780, 0) error 1
iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (iwi_monitor_fw, 0xc031f84c, 0) error 1
wlan: mac acl policy registered
random: entropy device external interface
ofwbus0: <open firmware="" device="" tree="">simplebus0: <flattened device="" tree="" simple="" bus="">on ofwbus0
simplebus1: <flattened device="" tree="" simple="" bus="">on simplebus0
simplebus2: <flattened device="" tree="" simple="" bus="">mem 0x210000-0x211fff on simplebus1
ti_scm0: <ti control="" module="">mem 0-0x7ff on simplebus2
aintc0: <ti aintc="" interrupt="" controller="">mem 0x48200000-0x48200fff on simplebus0
aintc0: Revision 5.0
cpulist0: <open firmware="" cpu="" group="">on ofwbus0
cpu0: <open firmware="" cpu="">on cpulist0
pmu0: <performance monitoring="" unit="">irq 0 on ofwbus0
am335x_prcm0: <am335x power="" and="" clock="" management="">mem 0x200000-0x203fff on simplebus1
am335x_prcm0: Clocks: System 24.0 MHz, CPU 550 MHz
ti_pinmux0: <ti pinmux="" module="">mem 0x800-0xa37 on simplebus2
am335x_scm0: <am335x control="" module="" extension="">on ti_scm0
gpio0: <ti am335x="" general="" purpose="" i="" o="" (gpio)="">mem 0x44e07000-0x44e07fff irq 7 on simplebus0
gpiobus0: <ofw gpio="" bus="">on gpio0
gpioc0: <gpio controller="">on gpio0
gpio1: <ti am335x="" general="" purpose="" i="" o="" (gpio)="">mem 0x4804c000-0x4804cfff irq 8 on simplebus0
gpiobus1: <ofw gpio="" bus="">on gpio1
gpioc1: <gpio controller="">on gpio1
gpio2: <ti am335x="" general="" purpose="" i="" o="" (gpio)="">mem 0x481ac000-0x481acfff irq 9 on simplebus0
gpiobus2: <ofw gpio="" bus="">on gpio2
gpioc2: <gpio controller="">on gpio2
gpio3: <ti am335x="" general="" purpose="" i="" o="" (gpio)="">mem 0x481ae000-0x481aefff irq 10 on simplebus0
gpiobus3: <ofw gpio="" bus="">on gpio3
gpioc3: <gpio controller="">on gpio3
uart0: console (115384,n,8,1)ible)> mem 0x44e09000-0x44e0afff irq 11 on simplebus0
iichb0: <ti i2c="" controller="">mem 0x44e0b000-0x44e0bfff irq 17 on simplebus0
iichb0: I2C revision 4.0 FIFO size: 32 bytes
iicbus0: <ofw i2c="" bus="">on iichb0
iic0: <i2c generic="" i="" o="">on iicbus0
icee0: <at24c02>at addr 0xa0 on iicbus0
iichb1: <ti i2c="" controller="">mem 0x4802a000-0x4802afff irq 18 on simplebus0
iichb1: I2C revision 4.0 FIFO size: 32 bytes
iicbus1: <ofw i2c="" bus="">on iichb1
iic1: <i2c generic="" i="" o="">on iicbus1
sdhci_ti0: <ti mmchs="" (sdhci="" 2.0)="">mem 0x48060000-0x48060fff irq 20 on simplebus0
mmc0: <mmc sd="" bus="">on sdhci_ti0
sdhci_ti1: <ti mmchs="" (sdhci="" 2.0)="">mem 0x481d8000-0x481d8fff irq 21 on simplebus0
mmc1: <mmc sd="" bus="">on sdhci_ti1
ti_wdt0: <ti watchdog="" timer="">mem 0x44e35000-0x44e35fff irq 23 on simplebus0
ti_mbox0: <ti system="" mailbox="">mem 0x480c8000-0x480c81ff irq 26 on simplebus0
ti_mbox0: revision 4.0
am335x_dmtimer0: <am335x dmtimer2="">mem 0x48040000-0x480403ff irq 28 on simplebus0
Event timer "DMTimer2" frequency 24000000 Hz quality 500
am335x_dmtimer1: <am335x dmtimer3="">mem 0x48042000-0x480423ff irq 29 on simplebus0
Timecounter "DMTimer3" frequency 24000000 Hz quality 500
usbss0: <ti am33xx="" integrated="" usb="" otg="" controller="">mem 0x47400000-0x47400fff on simplebus0
usbss0: TI AM335X USBSS v0.0.13
musbotg0: <ti am33xx="" integrated="" usb="" otg="" controller="">mem 0x47401400-0x474017ff,0x47401000-0x474011ff irq 54 on usbss0
usbus0: Dynamic FIFO sizing detected, assuming 16Kbytes of FIFO RAM
usbus0 on musbotg0
musbotg1: <ti am33xx="" integrated="" usb="" otg="" controller="">mem 0x47401c00-0x47401fff,0x47401800-0x474019ff irq 55 on usbss0
usbus1: Dynamic FIFO sizing detected, assuming 16Kbytes of FIFO RAM
usbus1 on musbotg1
cpswss0: <3-port Switch Ethernet Subsystem> mem 0x4a100000-0x4a1007ff,0x4a101200-0x4a1012ff irq 38,39,40,41 on simplebus0
cpswss0: CPSW SS Version 1.12 (0)
cpswss0: Initial queue size TX=128 RX=384
cpsw0: <ethernet switch="" port="">on cpswss0
miibus0: <mii bus="">on cpsw0
micphy0: <micrel 10="" 100="" 1000="" ksz9031="" phy="">PHY 1 on miibus0
micphy0:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto
cpsw0: Ethernet address: 68:9e:19:9c:9d:8d
cpsw1: <ethernet switch="" port="">on cpswss0
miibus1: <mii bus="">on cpsw1
micphy1: <micrel 10="" 100="" 1000="" ksz9031="" phy="">PHY 2 on miibus1
micphy1:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto
cpsw1: Ethernet address: 68:9e:19:9c:9d:8f
etherswitch0: <switch controller="">on cpswss0
ti_adc0: <ti adc="" controller="">mem 0x44e0d000-0x44e0dfff irq 44 disabled on simplebus0
ti_adc0: scheme: 0x1 func: 0x730 rtl: 0 rev: 0.1 custom rev: 0
cryptosoft0: <software crypto="">Timecounters tick every 1.000 msec
usbus0: 480Mbps High Speed USB v2.0
usbus1: 480Mbps High Speed USB v2.0
ugen0.1: <mentor graphics="" otg="" root="" hub="">at usbus0
uhub0: <mentor 1="" 9="" graphics="" otg="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr="">on usbus0
ugen1.1: <mentor graphics="" otg="" root="" hub="">at usbus1
uhub1: <mentor 1="" 9="" graphics="" otg="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr="">on usbus1
mmcsd0: 4GB <sdhc 1="" 2016="" r04gs="" 2.2="" sn="" 001cf5cb="" mfg="" 08="" by="" pa="">at mmc0 48.0MHz/4bit/65535-block
uhub1: 1 port with 1 removable, self powered
uhub0: 1 port with 1 removable, self powered
mmcsd1: 4GB <mmchc 112="" 2016="" s10004="" 0.8="" sn="" 114aca14="" mfg="" 07="" by="" 0x0000="">at mmc1 48.0MHz/8bit/65535-block
mmcsd1boot0: 4MB partion 1 at mmcsd1
mmcsd1boot1: 4MB partion 2 at mmcsd1
mmcsd1rpmb: 4MB partion 3 at mmcsd1
Trying to mount root from ufs:/dev/ufsid/58ee7f1f5968bf8a [rw,noatime]...
WARNING: / was not properly dismounted
warning: no time-of-day clock registered, system time will not be set accurately
Configuring crash dumps...
No suitable dump device was found.
** SU+J Recovering /dev/ufsid/58ee7f1f5968bf8a
** Reading 7503872 byte journal from inode 4.
** Building recovery table.
** Resolving unreferenced inode list.
** Processing journal entries.
** 2 journal records in 512 bytes for 12.50% utilization
** Freed 0 inodes (0 dirs) 0 blocks, and 2 frags.

***** FILE SYSTEM MARKED CLEAN *****
Filesystems are clean, continuing...
Mounting filesystems...
random: unblocking device.

        __
 _ __  / _|___  ___ _ __  ___  ___
| '_ \| |_/ __|/ _ \ '_ \/ __|/ _ \
| |_) |  _\__ \  __/ | | \__ \  __/
| .__/|_| |___/\___|_| |_|___/\___|
|_|

Welcome to pfSense 2.4.2-RELEASE (Patch 1)...

...ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/ipsec /usr/local/lib/perl5/5.24/mach/CORE
Soft Float compatibility ldconfig path:
done.
panic: ufs_dirbad: /: bad dir ino 58890 at offset 512: mangled entry
Uptime: 18s
Automatic reboot in 15 seconds - press a key on the console to abort
--> Press a key on the console to reboot,
--> or switch off the system now.</mmchc></sdhc></mentor></mentor></mentor></mentor></software></ti></switch></micrel></mii></ethernet></micrel></mii></ethernet></ti></ti></ti></am335x></am335x></ti></ti></mmc></ti></mmc></ti></i2c></ofw></ti></at24c02></i2c></ofw></ti></gpio></ofw></ti></gpio></ofw></ti></gpio></ofw></ti></gpio></ofw></ti></am335x></ti></am335x></performance></open></open></ti></ti></flattened></flattened></flattened></open></auto></auto></ethaddr>

This line:

panic: ufs_dirbad: /: bad dir ino 58890 at offset 512: mangled entry

suggests to me that it is indeed flash corruption and I need to reflash. I just don't understand why I keep having to do this about every couple of months. That seems too rare for some sort of manufacturing defect unless it is something really really obscure.

I have:

Just left the unit alone.
Not added any packages in a long time.
About once a week it needed rebooting via webgui when it couldn't recover from WAN dropping temporarily.
Maybe once a month you would have to power cycle it because you couldn't log in at all.
It's mounted on a metal shelf and does not get hot.
Connected it to a surge protected power strip (a good one, not that crap you get at the super market).

Why does this thing hate me ? :(

Anyone else had similar issues or suggestions on what the problem could be? Or am I just unlucky?

Cheers

ivor

Please contact our support so they can assist. https://customercare.netgate.com/

gpfsenser

I had a somewhat similar issue. In my case the filesystem appeared to have corrupted. Not clear if it was because of an update, or because I yanked power on it. I'm probably going to be setting up some external logging so I can keep tabs on it better.

I've only reflashed mine once in a year, and managed to get it back in a few hours - not too bad.

deadmalc

I had similar issues, the main thing is you need to fix the problem with the web-ui becoming unresponsive.
To prevent this I went from using the resolver to the dns forwarder.
I think the problem was caused by my netgear wifi "router" (used as an AP) flooding the SG-1000 - but moving to use the resolver fixed the issue.
I also stopped using the netgear as a secondary DNS anywhere, not sure which one fixed it - but it's been rock solid for months now.

Power cycling the sg-1000 (or any device) without a proper shutdown can cause filesystem corruption.
I once had (not a netgate device, or a firewall) the apache config in the hosts file!!!!
You maybe able to fix it by booting into single user mode (via the console) and running fsck (three times apparently).
But definitely speak to support as suggested.
Personally I'd save the config (if you can recover it) and re-flash.