SG-1100 All Ports Dead

brownoxford

I've had a few power outages recently, and now I'm unable to connect to my SG-1100 through the LAN port or have the SG-1100 obtain IP addresses as a DHCP client via the WAN or OPT ports (previously all working well).

I've rebooted a number of times, run fsck on the filesystem through the console, and finally run a factory reset (also via the console).

At this point, I have my laptop plugged in to the LAN port using a known good cable, but am not seeing any port activity lights either on the SG-1100 or on my laptop for this connection.

At the console I can see that mvneta0.4091 has the IP 192.168.1.1 as expected following a factory reset, and ifconfig shows the interface as being up:

mvneta0.4091: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=3<RXCSUM,TXCSUM>
        ether f0:ad:4e:0e:ef:28
        inet6 fe80::f2ad:4eff:fe0e:ef28%mvneta0.4091 prefixlen 64 scopeid 0x6
        inet6 fe80::1:1%mvneta0.4091 prefixlen 64 scopeid 0x6
        inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
        groups: vlan
        vlan: 4091 vlanpcp: 0 parent interface: mvneta0
        media: Ethernet 1000baseT <full-duplex>
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Of note are the status LEDs - bright green on the black diamond, but both of the others appear to be unlit (except from the bleed over of the intensely bright black diamond LED).

Obviously, the power is actually on.

Any tips on how to get this working?

rcoleman-netgate

@brownoxford Open a ticket and get the 23.01 firmware image to verify it's not a software issue.

If you're running FSCK you're on UFS and moving to 23.01 as a clean install will default you on ZFS.

https://go.netgate.com/

brownoxford

@rcoleman-netgate Support request submitted.

brownoxford

@rcoleman-netgate I have performed the upgrade and can now see the updated version in the console after booting:

*** Welcome to Netgate pfSense Plus 23.01-RELEASE (arm64) on pfSense ***

The only change in behavior since my initial report is that now none of the LEDs are lit up after the boot process completes (the power LED was lit during the first boot after upgrading, but it shut off at the end and never came back on, even during subsequent reboots).

Also of note, during booting the Configuring VLAN interfaces... step takes quite a bit of time (30-60 seconds) before moving on, as does Configuring WAN interface.... The Configuring LAN interface... step completes immediately.

I'm stumped, and open to suggestions...

rcoleman-netgate

@brownoxford said in SG-1100 All Ports Dead:

The only change in behavior since my initial report is that now none of the LEDs are lit up after the boot process completes (the power LED was lit during the first boot after upgrading, but it shut off at the end and never came back on, even during subsequent reboots).

A known issue fixed in the Recommended Patches from the System Patches package add-on.

Also of note, during booting the Configuring VLAN interfaces... step takes quite a bit of time (30-60 seconds) before moving on, as does Configuring WAN interface.... The Configuring LAN interface... step completes immediately.

Is there something connected to the WAN? If not it will wait for a time-out if it does not get a DHCP address

brownoxford

@rcoleman-netgate I previously did not have anything connected to the WAN port, just my laptop on the LAN port. I performed another reboot with my LTE modem plugged into the WAN port with no results - never saw any lights on the ports indicating traffic, no indication of a new client on the LTE modem portal, and no IP address assigned to the SG-1100.

I confirmed the LTE modem is configured properly as a DHCP server by plugging the cable into my laptop and verifying that an IP was assigned.

Is there any way to perform an offline installation of the recommended patches, e.g., via a USB stick using the console?

rcoleman-netgate

@brownoxford said in SG-1100 All Ports Dead:

I confirmed the LTE modem is configured properly as a DHCP server by plugging the cable into my laptop and verifying that an IP was assigned.

Hmm

Is there any way to perform an offline installation of the recommended patches, e.g., via a USB stick using the console?

No, the LED issue on the top is not long-term issue.

Can you try the OPT port as a the WAN (change in Interfaces->Assignments page -- swap WAN and OPT interface name/VLAN setting) and see if the OPT port works.

Since your 1100 is OOW there's no option for TAC support beyond the software side of things.. so give the swapping of interface devices a shot.

brownoxford

@rcoleman-netgate Can that port swap be done via the console?

SteveITS

@brownoxford yes
https://docs.netgate.com/pfsense/en/latest/config/console-menu.html#assign-interfaces

brownoxford

@steveits Thanks! I've swapped mvneta0.4090 and mvneta0.4092, but that doesn't change anything - they both seem dead.

What kind of single point of failure would cause all three ports to go dead at the same time? Trying to figure out if there is any further troubleshooting I can do...

SteveITS

@brownoxford It’s a switch with VLANs so it could be the switch circuitry if they are all dead.

stephenw10

At the console/command line try running: etherswitchcfg

That should show the status of the switch ports but might return an error that should give us a clue.

Steve

brownoxford

@stephenw10 Further digging reveals the following on boot (note "Switch Device not found), along with a set of etherswitchcfg errors in the logs:

U-Boot 2018.03-devel-18.12.3-gc9aa92c-dirty (Oct 07 2021 - 18:20:55 -0300)
  
Model: Netgate 1100
       CPU     1200 [MHz]
       L2      800 [MHz]
       TClock  200 [MHz]
       DDR     750 [MHz]
DRAM:  1 GiB
Comphy chip #0:
Comphy-0: USB3          5 Gbps
Comphy-1: PEX0          2.5 Gbps
Comphy-2: SATA0         6 Gbps
SATA link 0 timeout.
AHCI 0001.0300 32 slots 1 ports 6 Gbps 0x1 impl SATA mode
flags: ncq led only pmp fbss pio slum part sxs
PCIE-0: Link down
MMC:   sdhci@d0000: 0, sdhci@d8000: 1
Loading Environment from SPI Flash... SF: Detected mx25u3235f with page size 256 Bytes, erase size 64 KiB, total 4 MiB
OK
Model: Netgate 1100
Net:   eth0: neta@30000 [PRIME]
Switch Device not found
Hit any key to stop autoboot:  0
Card did not respond to voltage select!
    17970   armada-3720-netgate-1100.dtb
    17970   armada-3720-sg1100.dtb
    12948   armada-3720-netgate-2100.dtb
    12948   armada-3720-sg2100.dtb
  
4 file(s), 0 dir(s)

system.log:Feb 11 02:23:15 pfSense php-cgi[429]: rc.bootup: The command '/sbin/etherswitchcfg -f /dev/etherswitch0 port0 forwarding' returned exit code '69', the output was 'etherswitchcfg: Can't open control file: /dev/etherswitch0: No such file or directory'
system.log:Feb 11 02:23:15 pfSense php-cgi[429]: rc.bootup: The command '/sbin/etherswitchcfg -f /dev/etherswitch0 port1 forwarding' returned exit code '69', the output was 'etherswitchcfg: Can't open control file: /dev/etherswitch0: No such file or directory'
system.log:Feb 11 02:23:15 pfSense php-cgi[429]: rc.bootup: The command '/sbin/etherswitchcfg -f /dev/etherswitch0 port2 forwarding' returned exit code '69', the output was 'etherswitchcfg: Can't open control file: /dev/etherswitch0: No such file or directory'
system.log:Feb 11 02:23:15 pfSense php-cgi[429]: rc.bootup: The command '/sbin/etherswitchcfg -f /dev/etherswitch0 port3 forwarding' returned exit code '69', the output was 'etherswitchcfg: Can't open control file: /dev/etherswitch0: No such file or directory'

Confirmed that /dev/etherswitch0 is not present, also looking into system time issue.

stephenw10

Urgh, that doesn't look good. Even uboot fails to see the switch.
If you haven't already make sure you fully power cycle the device. Actually pull the power plug.
If uboot still shows no switch device it's probably a hardware fault.

brownoxford

@stephenw10 I have done that several times, but did it once more for good measure. After 10 minutes I plugged the power back in with the same result - "Switch Device not found".

stephenw10

Hmm, well update your ticket with us with that info if you have not already.

brownoxford

@stephenw10 Thanks, I've updated my support ticket. I don't think I'm still under warranty, but maybe tac can give me advice on whether anything can be done.