SG-5100 hangs durring reboot.

hirenz

My SG-5100 has started requiring the power to be pulled during reboots.
While its rebooting it will hang at uhub0 until the power is pulled.
I'm at a loss for troubleshooting and would appreciate any advice/guidance.

stephenw10

Is that in 23.01?

What has changed when that started happening?

Steve

rcoleman-netgate

@hirenz can you drop a shutdown log in here from puTTY/Screen?

hirenz

@stephenw10 as far as I know nothing.
I first noticed this behavior when updating to the first 23.01 rc.
A while back I installed a sata m.2 drive but reboots have worked fine since then.
I encountered issues with the rc so I performed a nfs snapshot restore to 22.05 and the issue persisted.

stephenw10

Yes, I think we'll need to see a console log showing exactly where it stops responding.

You might try entering 'ctl+t' at that point to see if it's waiting on a process.

hirenz

 0) Logout (SSH only)                  9) pfTop
 1) Assign Interfaces                 10) Filter Logs
 2) Set interface(s) IP address       11) Restart webConfigurator
 3) Reset webConfigurator password    12) PHP shell + Netgate pfSense Plus tools
 4) Reset to factory defaults         13) Update from console
 5) Reboot system                     14) Disable Secure Shell (sshd)
 6) Halt system                       15) Restore recent configuration
 7) Ping host                         16) Restart PHP-FPM
 8) Shell

Enter an option: 5


Netgate pfSense Plus will reboot. This may take a few minutes, depending on your hardware.
Do you want to proceed?

    Y/y: Reboot normally
    R/r: Reroot (Stop processes, remount disks, re-run startup sequence)
    S: Reboot into Single User Mode (requires console access!)
    Enter: Abort

Enter an option: y

Netgate pfSense Plus is rebooting now.
 Stopping package nut...done.
 Stopping package suricata...done.
 Stopping package Avahi...done.
 Stopping package squid3...done.
 Stopping package Telegraf...done.
 Stopping /usr/local/etc/rc.d/haproxy.sh...done.
 Stopping /usr/local/etc/rc.d/pfb_dnsbl.sh...done.
 Stopping /usr/local/etc/rc.d/pfb_filter.sh...done.
 Stopping /usr/local/etc/rc.d/shutdown.nut.sh...done.
 Stopping /usr/local/etc/rc.d/sqp_monitor.sh...done.
 Stopping /usr/local/etc/rc.d/vnstatd.sh...done.
pflog0: promiscuous mode disabled
ovpns1: link state changed to DOWN
Waiting (max 60 seconds) for system process `vnlru' to stop... done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining... 0 0 0 0 0 done
All buffers synced.
Uptime: 4d0h9m40s
Khelp module "ertt" can't unload until its refcount drops from 26 to 0.
ix1: link state changed to DOWN
ix0: link state changed to DOWN
lagg0: link state changed to DOWN
lagg0.50: link state changed to DOWN
lagg0.2: link state changed to DOWN
lagg0.70: link state changed to DOWN
lagg0.101: link state changed to DOWN
lagg0.20: link state changed to DOWN
lagg0.201: link state changed to DOWN
lagg0.301: link state changed to DOWN
uhub0: detached

hirenz

I didnt do ctrl-t, c, or d this time. The console over usb goes non responsive.
I have tried to hard reboot using the power switch on the back of the device and it just turns off and will not power back on until the power has been disconnected.

stephenw10

Ok, so ctl+t shows nothing when entered there?

hirenz

@stephenw10 Correct, It just sits.

stephenw10

Do you have any USB devices attached to it?
We have seen one other case where here was a USB device that prevented it rebooting.

hirenz

The only usb devices that are connected would be the UPS (never had an issue with it before) and the mini-usb for console. I have tried with the the UPS instead unplugged.

Does the sata m.2 port use pcie or usb to connect?

stephenw10

It uses SATA. It's not a PCIe or USB device.

Try booting verbose to see if you get any more info. At the boot loader menu hit esc to reach the prompt then enter boot -v.

Steve

hirenz

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2023.02.22 17:53:44 =~=~=~=~=~=~=~=~=~=~=~=



Netgate pfSense Plus will reboot. This may take a few minutes, depending on your hardware.
Do you want to proceed?

    Y/y: Reboot normally
    R/r: Reroot (Stop processes, remount disks, re-run startup sequence)
    S: Reboot into Single User Mode (requires console access!)
    Enter: Abort

Enter an option: y

Netgate pfSense Plus is rebooting now.
 Stopping package nut...done.
 Stopping package suricata...done.
 Stopping package Avahi...done.
 Stopping package squid3...done.
 Stopping package Telegraf...done.
 Stopping /usr/local/etc/rc.d/haproxy.sh...done.
 Stopping /usr/local/etc/rc.d/pfb_dnsbl.sh...done.
 Stopping /usr/local/etc/rc.d/pfb_filter.sh...done.
 Stopping /usr/local/etc/rc.d/shutdown.nut.sh...done.
 Stopping /usr/local/etc/rc.d/sqp_monitor.sh...done.
 Stopping /usr/local/etc/rc.d/vnstatd.sh...done.
pflog0: promiscuous mode disabled
ovpns1: link state changed to DOWN
Waiting (max 60 seconds) for system process `vnlru' to stop... done
Waiting (max 60 seconds) for system process `syncer' to stop... 
Syncing disks, vnodes remaining... 0 0 0 0 0 done
All buffers synced.
Swap device ada0p2 removed.
Uptime: 3m56s
(ada0:ahcich0:0:0:0): spin-down
Khelp module "ertt" can't unload until its refcount drops from 37 to 0.
ix1: Link is Down
ix1: link state changed to DOWN
ix0: Link is Down
ix0: link state changed to DOWN
lagg0: link state changed to DOWN
lagg0.50: link state changed to DOWN
lagg0.2: link state changed to DOWN
lagg0.70: link state changed to DOWN
lagg0.101: link state changed to DOWN
lagg0.20: link state changed to DOWN
lagg0.201: link state changed to DOWN
lagg0.301: link state changed to DOWN
uhub0: detached
unknown: wake_prep disabled wake for \_SB_.PCI0.PEX0 (S5)
unknown: wake_prep disabled wake for \_SB_.PCI0.PEX1 (S5)
unknown: wake_prep disabled wake for \_SB_.PCI0.PEX2 (S5)
unknown: wake_prep disabled wake for \_SB_.PCI0.PEX3 (S5)
pcib2: wake_prep disabled wake for \_SB_.PCI0.PEX4 (S5)
pcib3: wake_prep disabled wake for \_SB_.PCI0.PEX5 (S5)
pcib4: wake_prep disabled wake for \_SB_.PCI0.PEX6 (S5)
pcib5: wake_prep disabled wake for \_SB_.PCI0.PEX7 (S5)
xhci0: wake_prep disabled wake for \_SB_.PCI0.XHC1 (S5)
ix0: wake_prep disabled wake for \_SB_.PCI0.VRP0.LAN0 (S5)
ix1: wake_prep disabled wake for \_SB_.PCI0.VRP0.LAN1 (S5)
ix2: wake_prep disabled wake for \_SB_.PCI0.VRP1.LAN2 (S5)
ix3: wake_prep disabled wake for \_SB_.PCI0.VRP1.LAN3 (S5)

It looks like its not related to the uhub0 as I originally though. I let it sit at this point for about 10 minutes
Pressing ctrl-t did nothing as well.

stephenw10

Hmm, that looks like it might be Wake-on-LAN related.

Have you configured anything differently in the BIOS?

hirenz

I have not, in fact I had never been in the bios until I was trying to troubleshoot this. I don’t think I changed anything but I can set it back to defaults and ensure WoL is disabled.

stephenw10

Seems unrelated in fact. My test 5100 also shows that in verbose mode but reboots as expected.

Hard to imagine it could be anything but a BIOS setting or actual hardware at that point.

hirenz

When I have time for a longer downtime I’ll get the latest factory image, reset the bios and reflash the os. Maybe even get a new boot disk. I vaguely remember this one being a cheaper one I got for a project that fell through and got tossed in a drawer.

stephenw10

It's a possibility but it's hard to see what might have changed in 23.01 that could cause it.

The 5100 can be installed UEFI or legacy (BIOS). Try running: sysctl machdep.bootmethod to find out. My test box is currently BIOS.

Steve

hirenz

Mine is currently in BIOS mode as well. However I’m not sure it’s related to 23.01.

stephenw10

Ah, you think it may have also been hanging there in 22.05?