Failure to re-start properly, possibly since 21.05
I have a really weird issue that started last night at 12:07am, although it seems to be re-boot related so the problem may have existed for some time, possibly since the last reboot which was probably when 21.05 was installed about a week ago.
The issue is that if I reboot my SG-3100, when it starts back up many services seem to not be running! Initially I can't get in to the web interface until I SSH on and re-start the web interface manually. Once I am in to the web interface I can see that DHCP and Watchdog Daemon are not running (both start OK when I manually start them). But worse still, no traffic is flowing out to the DSL line or between Lan1 and the Opt1 port! It's as if routing is also stopped, although I don't know how to check if this is the case?
In the end I restored a backup from a few months back and initially it worked, so I set about comparing the backup with the current config so I could manually re-apply changes (mainly minor firewall changes). But after a reboot, it stopped working again, and I was back to having to manually start the web interface etc.
I spent ages doing and undoing the firewall changes one at a time but in the end it seems to me that it has nothing to do with the actual config, just that it seems to 'mostly' work if I restore a backup for the first boot, but after another reboot it fails again!
I tried a factory reset, restored a backup (tried both current and the old one) and again it worked, until the next reboot.
How can I diagnose what is failing during reboots?
The other odd thing, which doesn't seem to make much sense, is that some restored settings (enabling SSH and the SSH admin password, but not the web interface one) seem to only apply after the second reboot. What I mean is I had to use a USB cable to enable SSH and then after restoring the backup the SSH remained enabled (even though it wasn't enabled in the backup) and I noticed the default SSH password persisted after the factory reset even after restoring backups. However, a reboot or so later, SSH disabled itself again and the password changed back to the one from the backup. This behaviour may be unrelated, but it isn't working logically and maybe that is why it fails only after a reboot.
The first thing to do is connect to the USB console and reboot it. Monitor the entire boot process and see what happens.
It sounds to me like some part of the boot process is getting hung up or it terminates early so it never finishes completely.