SG-1000 seems to have suicided on reboot



  • I’m convinced now that I own a cursed SG-1000, but it’s the slow painful type.

    I have had to reflash this guy four times now (owned it for a year). In the past I just put it down to beta firmware well being beta and borked updates but this time I was running 2.4.0-p1 stable since it came out. Situation was:

    1. Webgui suddenly became unresponsive (this happens occasionally when the WAN drops out and comes back but pfsense for some reason goes to la la land).
    2. Normally a power cycle is done to fix this, but the problem occured while I was logged in so I tried to investigate but SSH was very slow so I gave up on that idea.
    3. Tried two restarts of webgui via the SSH menu, both failed.
    4. Then chose restart option in SSH, and went with reroot (instead of reboot) since I assumed it would be faster.
    5. SG-1000 never came back online. It was just stuck flashing it’s LEDs to indicate it was booting.
    6. Tried power cycles. No dice.
    7. Had a console cable connected and it seemed to be stuck in a boot loop with the following output:
    
    ***** FILE SYSTEM MARKED CLEAN *****
    Filesystems are clean, continuing...
    Mounting filesystems...
    random: unblocking device.
    
            __
     _ __  / _|___  ___ _ __  ___  ___
    | '_ \| |_/ __|/ _ \ '_ \/ __|/ _ \
    | |_) |  _\__ \  __/ | | \__ \  __/
    | .__/|_| |___/\___|_| |_|___/\___|
    |_|
    
    Welcome to pfSense 2.4.2-RELEASE (Patch 1)...
    
    ...ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/ipsec /usr/local/lib/perl5/5.24/mach/CORE
    Soft Float compatibility ldconfig path:
    done.
    panic: ufs_dirbad: /: bad dir ino 58890 at offset 512: mangled entry
    Uptime: 17s
    Automatic reboot in 15 seconds - press a key on the console to abort
    --> Press a key on the console to reboot,
    --> or switch off the system now.
    Rebooting...
    
    U-Boot SPL 2016.03 (Dec 28 2016 - 14:33:36)
    Trying to boot from MMC
    Card doesn't support part_switch
    MMC partition switch failed
    *** Warning - MMC partition switch failed, using default environment
    
    reading u-boot.img
    reading u-boot.img
    
    U-Boot 2016.03 (Dec 28 2016 - 14:33:36 +0000)
    
           Watchdog enabled
    I2C:   ready
    DRAM:  512 MiB
    MMC:   OMAP SD/MMC: 0, OMAP SD/MMC: 1
    
    ** Unable to use mmc 0:1 for loading the env **
    Using default environment
    
    Net:   <ethaddr>not set. Validating first E-fuse MAC
    cpsw, usb_ether
    starting USB...
    USB0:   Port not available.
    ** Bad device usb 0 **
    ** Unrecognized filesystem type **
    Hit any key to stop autoboot:  0
    Booting from: mmc 1 ubldr.bin
    reading ubldr.bin
    227396 bytes read in 21 ms (10.3 MiB/s)
    ## Starting application at 0x88000000 ...
    Consoles: U-Boot console
    Compatible U-Boot API signature found @0x9ef3ab58
    
    FreeBSD/armv6 U-Boot loader, Revision 1.2
    (Tue Dec 12 15:07:07 CST 2017 root@buildbot2.netgate.com)
    
    DRAM: 512MB
    Number of U-Boot devices: 3
    U-Boot env: loaderdev='mmc 1'
    Found U-Boot device: disk
      Checking unit=1 slice= <auto>partition=<auto>... good.
    Booting from disk1s2a:
    console comconsole is invalid!
    console vidconsole is invalid!
    no valid consoles!
    Available consoles:
        uboot
    /boot/kernel/kernel data=0x10ff5a8+0x50a58 syms=[0x4+0xcea00+0x4+0xcac5d]
    
    Hit [Enter] to boot immediately, or any other key for command prompt.
    Booting [/boot/kernel/kernel]...
    /boot/dtb/ufw.dtb size=0xb608
    Loaded DTB from file 'ufw.dtb'.
    Kernel entry at 0x88200100...
    Kernel args: (null)
    Copyright (c) 1992-2017 The FreeBSD Project.
    Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
            The Regents of the University of California. All rights reserved.
    FreeBSD is a registered trademark of The FreeBSD Foundation.
    FreeBSD 11.1-RELEASE-p6 #0 r313908+a5b33c9d1c4(RELENG_2_4): Tue Dec 12 15:08:51 CST 2017
        root@buildbot2.netgate.com:/xbuilder/crossbuild-242/work/obj-ufw-armv6/arm.armv6/xbuilder/crossbuild-242/pfSense/tmp/FreeBSD-src/sys/pfSense-uFW arm
    FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0)
    WARNING: too long kenv string, ignoring uboot.Fatboot=env exists loaderdev || env set loaderdev ${fatdev}; env exists UserFatboot && run UserFatboot; env exists trynetboot && env exists Netboot && run Netboot; echo Booting from: ${fatdev} ${bootfile}; fatload ${fatdev} ${loadaddr} ${bootfile} && go ${loadaddr};
    CPU: ARM Cortex-A8 r3p2 (ECO: 0x00000000)
    CPU Features:
      Thumb2, Security, VMSAv7
    Optional instructions:
      UMULL, SMULL, SIMD(ext)
    LoUU:2 LoC:3 LoUIS:1
    Cache level 1:
     32KB/64B 4-way data cache WT WB Read-Alloc
     32KB/64B 4-way instruction cache Read-Alloc
    Cache level 2:
     256KB/64B 8-way unified cache WT WB Read-Alloc Write-Alloc
    real memory  = 536870912 (512 MB)
    avail memory = 503279616 (479 MB)
    Texas Instruments AM335x Processor, Revision ES1.2
    ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
    ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_bss_fw, 0xc031550c, 0) error 1
    ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
    ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_ibss_fw, 0xc03155d4, 0) error 1
    ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LICENSE.
    ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_monitor_fw, 0xc03156a0, 0) error 1
    iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
    iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (iwi_bss_fw, 0xc031f6b8, 0) error 1
    iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
    iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (iwi_ibss_fw, 0xc031f780, 0) error 1
    iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LICENSE.
    iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (iwi_monitor_fw, 0xc031f84c, 0) error 1
    wlan: mac acl policy registered
    random: entropy device external interface
    ofwbus0: <open firmware="" device="" tree="">simplebus0: <flattened device="" tree="" simple="" bus="">on ofwbus0
    simplebus1: <flattened device="" tree="" simple="" bus="">on simplebus0
    simplebus2: <flattened device="" tree="" simple="" bus="">mem 0x210000-0x211fff on simplebus1
    ti_scm0: <ti control="" module="">mem 0-0x7ff on simplebus2
    aintc0: <ti aintc="" interrupt="" controller="">mem 0x48200000-0x48200fff on simplebus0
    aintc0: Revision 5.0
    cpulist0: <open firmware="" cpu="" group="">on ofwbus0
    cpu0: <open firmware="" cpu="">on cpulist0
    pmu0: <performance monitoring="" unit="">irq 0 on ofwbus0
    am335x_prcm0: <am335x power="" and="" clock="" management="">mem 0x200000-0x203fff on simplebus1
    am335x_prcm0: Clocks: System 24.0 MHz, CPU 550 MHz
    ti_pinmux0: <ti pinmux="" module="">mem 0x800-0xa37 on simplebus2
    am335x_scm0: <am335x control="" module="" extension="">on ti_scm0
    gpio0: <ti am335x="" general="" purpose="" i="" o="" (gpio)="">mem 0x44e07000-0x44e07fff irq 7 on simplebus0
    gpiobus0: <ofw gpio="" bus="">on gpio0
    gpioc0: <gpio controller="">on gpio0
    gpio1: <ti am335x="" general="" purpose="" i="" o="" (gpio)="">mem 0x4804c000-0x4804cfff irq 8 on simplebus0
    gpiobus1: <ofw gpio="" bus="">on gpio1
    gpioc1: <gpio controller="">on gpio1
    gpio2: <ti am335x="" general="" purpose="" i="" o="" (gpio)="">mem 0x481ac000-0x481acfff irq 9 on simplebus0
    gpiobus2: <ofw gpio="" bus="">on gpio2
    gpioc2: <gpio controller="">on gpio2
    gpio3: <ti am335x="" general="" purpose="" i="" o="" (gpio)="">mem 0x481ae000-0x481aefff irq 10 on simplebus0
    gpiobus3: <ofw gpio="" bus="">on gpio3
    gpioc3: <gpio controller="">on gpio3
    uart0: console (115384,n,8,1)ible)> mem 0x44e09000-0x44e0afff irq 11 on simplebus0
    iichb0: <ti i2c="" controller="">mem 0x44e0b000-0x44e0bfff irq 17 on simplebus0
    iichb0: I2C revision 4.0 FIFO size: 32 bytes
    iicbus0: <ofw i2c="" bus="">on iichb0
    iic0: <i2c generic="" i="" o="">on iicbus0
    icee0: <at24c02>at addr 0xa0 on iicbus0
    iichb1: <ti i2c="" controller="">mem 0x4802a000-0x4802afff irq 18 on simplebus0
    iichb1: I2C revision 4.0 FIFO size: 32 bytes
    iicbus1: <ofw i2c="" bus="">on iichb1
    iic1: <i2c generic="" i="" o="">on iicbus1
    sdhci_ti0: <ti mmchs="" (sdhci="" 2.0)="">mem 0x48060000-0x48060fff irq 20 on simplebus0
    mmc0: <mmc sd="" bus="">on sdhci_ti0
    sdhci_ti1: <ti mmchs="" (sdhci="" 2.0)="">mem 0x481d8000-0x481d8fff irq 21 on simplebus0
    mmc1: <mmc sd="" bus="">on sdhci_ti1
    ti_wdt0: <ti watchdog="" timer="">mem 0x44e35000-0x44e35fff irq 23 on simplebus0
    ti_mbox0: <ti system="" mailbox="">mem 0x480c8000-0x480c81ff irq 26 on simplebus0
    ti_mbox0: revision 4.0
    am335x_dmtimer0: <am335x dmtimer2="">mem 0x48040000-0x480403ff irq 28 on simplebus0
    Event timer "DMTimer2" frequency 24000000 Hz quality 500
    am335x_dmtimer1: <am335x dmtimer3="">mem 0x48042000-0x480423ff irq 29 on simplebus0
    Timecounter "DMTimer3" frequency 24000000 Hz quality 500
    usbss0: <ti am33xx="" integrated="" usb="" otg="" controller="">mem 0x47400000-0x47400fff on simplebus0
    usbss0: TI AM335X USBSS v0.0.13
    musbotg0: <ti am33xx="" integrated="" usb="" otg="" controller="">mem 0x47401400-0x474017ff,0x47401000-0x474011ff irq 54 on usbss0
    usbus0: Dynamic FIFO sizing detected, assuming 16Kbytes of FIFO RAM
    usbus0 on musbotg0
    musbotg1: <ti am33xx="" integrated="" usb="" otg="" controller="">mem 0x47401c00-0x47401fff,0x47401800-0x474019ff irq 55 on usbss0
    usbus1: Dynamic FIFO sizing detected, assuming 16Kbytes of FIFO RAM
    usbus1 on musbotg1
    cpswss0: <3-port Switch Ethernet Subsystem> mem 0x4a100000-0x4a1007ff,0x4a101200-0x4a1012ff irq 38,39,40,41 on simplebus0
    cpswss0: CPSW SS Version 1.12 (0)
    cpswss0: Initial queue size TX=128 RX=384
    cpsw0: <ethernet switch="" port="">on cpswss0
    miibus0: <mii bus="">on cpsw0
    micphy0: <micrel 10="" 100="" 1000="" ksz9031="" phy="">PHY 1 on miibus0
    micphy0:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto
    cpsw0: Ethernet address: 68:9e:19:9c:9d:8d
    cpsw1: <ethernet switch="" port="">on cpswss0
    miibus1: <mii bus="">on cpsw1
    micphy1: <micrel 10="" 100="" 1000="" ksz9031="" phy="">PHY 2 on miibus1
    micphy1:  none, 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto
    cpsw1: Ethernet address: 68:9e:19:9c:9d:8f
    etherswitch0: <switch controller="">on cpswss0
    ti_adc0: <ti adc="" controller="">mem 0x44e0d000-0x44e0dfff irq 44 disabled on simplebus0
    ti_adc0: scheme: 0x1 func: 0x730 rtl: 0 rev: 0.1 custom rev: 0
    cryptosoft0: <software crypto="">Timecounters tick every 1.000 msec
    usbus0: 480Mbps High Speed USB v2.0
    usbus1: 480Mbps High Speed USB v2.0
    ugen0.1: <mentor graphics="" otg="" root="" hub="">at usbus0
    uhub0: <mentor 1="" 9="" graphics="" otg="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr="">on usbus0
    ugen1.1: <mentor graphics="" otg="" root="" hub="">at usbus1
    uhub1: <mentor 1="" 9="" graphics="" otg="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr="">on usbus1
    mmcsd0: 4GB <sdhc 1="" 2016="" r04gs="" 2.2="" sn="" 001cf5cb="" mfg="" 08="" by="" pa="">at mmc0 48.0MHz/4bit/65535-block
    uhub1: 1 port with 1 removable, self powered
    uhub0: 1 port with 1 removable, self powered
    mmcsd1: 4GB <mmchc 112="" 2016="" s10004="" 0.8="" sn="" 114aca14="" mfg="" 07="" by="" 0x0000="">at mmc1 48.0MHz/8bit/65535-block
    mmcsd1boot0: 4MB partion 1 at mmcsd1
    mmcsd1boot1: 4MB partion 2 at mmcsd1
    mmcsd1rpmb: 4MB partion 3 at mmcsd1
    Trying to mount root from ufs:/dev/ufsid/58ee7f1f5968bf8a [rw,noatime]...
    WARNING: / was not properly dismounted
    warning: no time-of-day clock registered, system time will not be set accurately
    Configuring crash dumps...
    No suitable dump device was found.
    ** SU+J Recovering /dev/ufsid/58ee7f1f5968bf8a
    ** Reading 7503872 byte journal from inode 4.
    ** Building recovery table.
    ** Resolving unreferenced inode list.
    ** Processing journal entries.
    ** 2 journal records in 512 bytes for 12.50% utilization
    ** Freed 0 inodes (0 dirs) 0 blocks, and 2 frags.
    
    ***** FILE SYSTEM MARKED CLEAN *****
    Filesystems are clean, continuing...
    Mounting filesystems...
    random: unblocking device.
    
            __
     _ __  / _|___  ___ _ __  ___  ___
    | '_ \| |_/ __|/ _ \ '_ \/ __|/ _ \
    | |_) |  _\__ \  __/ | | \__ \  __/
    | .__/|_| |___/\___|_| |_|___/\___|
    |_|
    
    Welcome to pfSense 2.4.2-RELEASE (Patch 1)...
    
    ...ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/ipsec /usr/local/lib/perl5/5.24/mach/CORE
    Soft Float compatibility ldconfig path:
    done.
    panic: ufs_dirbad: /: bad dir ino 58890 at offset 512: mangled entry
    Uptime: 18s
    Automatic reboot in 15 seconds - press a key on the console to abort
    --> Press a key on the console to reboot,
    --> or switch off the system now.</mmchc></sdhc></mentor></mentor></mentor></mentor></software></ti></switch></micrel></mii></ethernet></micrel></mii></ethernet></ti></ti></ti></am335x></am335x></ti></ti></mmc></ti></mmc></ti></i2c></ofw></ti></at24c02></i2c></ofw></ti></gpio></ofw></ti></gpio></ofw></ti></gpio></ofw></ti></gpio></ofw></ti></am335x></ti></am335x></performance></open></open></ti></ti></flattened></flattened></flattened></open></auto></auto></ethaddr> 
    

    This line:

    panic: ufs_dirbad: /: bad dir ino 58890 at offset 512: mangled entry
    

    suggests to me that it is indeed flash corruption and I need to reflash. I just don’t understand why I keep having to do this about every couple of months. That seems too rare for some sort of manufacturing defect unless it is something really really obscure.

    I have:

    • Just left the unit alone.
    • Not added any packages in a long time.
    • About once a week it needed rebooting via webgui when it couldn’t recover from WAN dropping temporarily.
    • Maybe once a month you would have to power cycle it because you couldn’t log in at all.
    • It’s mounted on a metal shelf and does not get hot.
    • Connected it to a surge protected power strip (a good one, not that crap you get at the super market).

    Why does this thing hate me ?  😞

    Anyone else had similar issues or suggestions on what the problem could be? Or am I just unlucky?

    Cheers


  • Galactic Empire Netgate Administrator

    Please contact our support so they can assist. https://customercare.netgate.com/



  • I had a somewhat similar issue.  In my case the filesystem appeared to have corrupted.  Not clear if it was because of an update, or because I yanked power on it.  I’m probably going to be setting up some external logging so I can keep tabs on it better.

    I’ve only reflashed mine once in a year, and managed to get it back in a few hours - not too bad.



  • I had similar issues, the main thing is you need to fix the problem with the web-ui becoming unresponsive.
    To prevent this I went from using the resolver to the dns forwarder.
    I think the problem was caused by my netgear wifi “router” (used as an AP) flooding the SG-1000 - but moving to use the resolver fixed the issue.
    I also stopped using the netgear as a secondary DNS anywhere, not sure which one fixed it - but it’s been rock solid for months now.

    Power cycling the sg-1000 (or any device) without a proper shutdown can cause filesystem corruption.
    I once had (not a netgate device, or a firewall) the apache config in the hosts file!!!
    You maybe able to fix it by booting into single user mode (via the console) and running fsck (three times apparently).
    But definitely speak to support as suggested.
    Personally I’d save the config (if you can recover it) and re-flash.


 

© Copyright 2002 - 2018 Rubicon Communications, LLC | Privacy Policy