Netgate 3100 will not come up



  • My Netgate 3100 firewall has been running flawlessly for months. Today I was trying to troubleshoot my comcast cable modem speed and I disconnected the wan from the Netgate box to plug in a computer directly to the cable modem.

    When I finished the test I plugged the Netgate 3100 back in and noticed my gateway was down. I shutdown the Netgate device and the cable modem. Started them up in sequence and now the 3 lights in the front of the Netgate box flash one at a time first, them all together, turn off and start the process again.

    I cannot get it up to ping the lan, or get into the gui interface. I have not tried connecting via console but it seems to me that the device is not seeing the gateway, and restarting constently.

    Did I brick this unit?


  • LAYER 8

    you need to connect to the console,
    follow this instruction:
    https://forum.netgate.com/post/745773

    remember to repeat the fsck command 4/5 times before rebooting

    if it does not help contact netgate at https://go.netgate.com



  • OK, it seems I have a short window to get in, as it keeps rebooting.



  • So I see this in the terminal session but nothing else.

    [SERIAL/DIRECT] CONNECTED TO PORT /dev/cu.usbserial-01A898AC (115200-8N1)



  • How did you shutdown the Netgate appliance? Did you login to it and choose the "Halt System" command under the DIAGNOTICS menu, or did you just pull the power cord? If the latter, then odds are the file system is corrupted.

    You should never just "turn off" a pfSense firewall. That will usually result in corruption of the filesystem. First execute the Halt System command under the DIAGNOSTICS menu. When that completes, it is safe to pull the power.

    It is good practice to be sure your firewall is also protected by a UPS.



  • I shut it down from the menu command.



  • @amarcino said in Netgate 3100 will not come up:

    I shut it down from the menu command.

    Hmm...then should not have corrupted the disk provided enough time elapsed between choosing the Halt System command and the actual power loss. It does take it a few seconds to actually finishing flushing/writing things to the SSD/memory card before it is ready for safely powering down.

    I think you might have to hit ENTER or the spacebar key to get another prompt. If you get nothing, the Netgate support team should be able to help you out. I believe you can get help with the hardware (which sounds like what is wrong in your case) without having a paid support subscription.



  • @bmeeks I tried that also, and got nothing.

    Is it normal behavior for this unit, if it cannot see it's gateway to do this?



  • The console access should always work.
    No matter what the setup configuration is ...



  • @amarcino said in Netgate 3100 will not come up:

    [SERIAL/DIRECT] CONNECTED TO PORT /dev/cu.usbserial-01A898AC (115200-8N1)

    I was using ZOC on a Macbook pro. I will try a windows 10 laptop tonight with Putty.



  • No, changing a gateway would not cause a "no boot" condition. You likely either have a corrupted file system or some type of hardware failure that just happened to occur with the power cycle.

    As I mentioned in one of my earlier posts, even when using the Halt System command in the DIAGNOSTICS menu, you need to wait several seconds for the box to complete its shutdown sequence. Maybe you waited long enough or maybe you did not.

    If you can get into the console and then find a corrupted file system that is fixed by running fsck, then you know for the future that you likely did not wait long enough for the Halt System command to fullly complete before pulling the power cord. However, if there has been an actual hardware failure, that's just bad luck and not related to the shutdown sequence itself.



  • OK.

    I have been in contact with Netgate support as well.

    It seems I may have been using the wrong serial port discovered by ZOC on a Macbook Pro.

    I am going to try my work laptop tonight with putty installed, and make sure I am using the correct driver. Hopefully I will be able to get into the console and stop this mess.

    Wish me luck, having a dead device and no internet access @ home is no especially when you depend on smart devices for the simple things..... LOL



  • @amarcino said in Netgate 3100 will not come up:

    OK.

    I have been in contact with Netgate support as well.

    It seems I may have been using the wrong serial port discovered by ZOC on a Macbook Pro.

    I am going to try my work laptop tonight with putty installed, and make sure I am using the correct driver. Hopefully I will be able to get into the console and stop this mess.

    Wish me luck, having a dead device and no internet access @ home is no especially when you depend on smart devices for the simple things..... LOL

    If you download the correct USB/Serial port driver for Windows, it does work. Follow the instructions here: https://docs.netgate.com/pfsense/en/latest/solutions/sg-3100/connect-to-console.html. And PuTTY is a great free terminal utility.



  • No doubt on putty I used it many moons ago.



  • In that case, activate the SSH access for pfSense.
    This will enable a console look alike access over LAN (no need to cable up in front of your device).

    Also : consider shutting down the device using the real console access - that is, the cable with Putty or a like, and the micro SB serial cable.
    This way you can follow the shut down up until the bitter end ....

    And to be sure : 'smart' devices like the ones on which pfSense runs merits a UPS. This way, when the power goes off, you can set up a programmed clean shut down using the NUT package.



  • I have it on a UPS already, always did since I got it 4 months ago.

    I have SSH already enabled, and used that to get into the system. I didn't assume it would work now, since the device never comes up stable and reboots about every 40 seconds or so.

    I can try it though tonight.



  • @amarcino said in Netgate 3100 will not come up:

    I have it on a UPS already, always did since I got it 4 months ago.

    I have SSH already enabled, and used that to get into the system. I didn't assume it would work now, since the device never comes up stable and reboots about every 40 seconds or so.

    I can try it though tonight.

    If it's not booting up completely, then SSH access is not going to work. I assume that initially you were trying to access it from the LAN side via the web GUI perhaps, so if that does not work, then SSH is not likely to work either.

    Getting a PuTTY connection over the console cable is your best option to see what is actually going on.



  • I agree. Will know tonight.

    @bmeeks said in Netgate 3100 will not come up:

    @amarcino said in Netgate 3100 will not come up:

    I have it on a UPS already, always did since I got it 4 months ago.

    I have SSH already enabled, and used that to get into the system. I didn't assume it would work now, since the device never comes up stable and reboots about every 40 seconds or so.

    I can try it though tonight.

    If it's not booting up completely, then SSH access is not going to work. I assume that initially you were trying to access it from the LAN side via the web GUI perhaps, so if that does not work, then SSH is not likely to work either.

    Getting a PuTTY connection over the console cable is your best option to see what is actually going on.


  • Netgate Administrator

    I have seen boot loops in two places on the SG-3100. It can panic and reboot when it tries to mount the root filesystem, that's far more common. It happens ~30s into the boot, the complete cycle is somewhere ~50s depending on what you're booting from and what you have loaded.
    It can also boot loop directly in uboot so it fails to load the FreeBSD boot loader and resets. That is much quicker so you see the LEDs reset maybe every 15s. I never actually timed that.
    In both cases the boot cycle is fixed length and does not vary so that's useful to diagnose just by looking at the LEDs.
    The fact the three LEDs start flashing blue indicates uboot is completing it's hardware setup and running the bootcmd env.

    In all cases you should see output on the serial console though so that's the first step.
    Make sure you see a new device when you attach the cable.
    Make sure the driver is loaded.

    Steve



  • @stephenw10

    I have been seeing your first example. Will the cycle stop when I log in through the console to run fsck?


  • Netgate Administrator

    Yes, when you boot to single user mode it does not try to mount root so you reach the prompt.

    Then you can run fsck repeatedly to correct all the issues and it will mount correctly at the next boot.

    Steve


  • Rebel Alliance Developer Netgate

    @stephenw10 said in Netgate 3100 will not come up:

    Yes, when you boot to single user mode it does not try to mout root so you reach the prompt.

    It does mount root, but as read-only, which is safer and unlikely to trigger the typical sorts of filesystem panics.

    It is more rare but there are cases when even that will panic, but the filesystem has to be really, really hosed.



  • So is there a trick to boot into single user mode. The Marvel boot loader is so fast I keep missing something because it gets to the end and wants to reboot or shutdown.

    I was able to get to a loader prompt loader> but I cannot do fsck from it.



  • @amarcino said in Netgate 3100 will not come up:

    So is there a trick to boot into single user mode. The Marvel boot loader is so fast I keep missing something because it gets to the end and wants to reboot or shutdown.

    I was able to get to a loader prompt loader> but I cannot do fsck from it.

    Try this trick from an older thread:

    https://forum.netgate.com/topic/125712/netgate-sg-3100-boot-loop-ufs_dirbad.

    According to @Derelict in that thread, the ARM appliances don't present the same boot menu as the AMD/Intel boxes do.



  • @bmeeks

    Perfect, file system clean and I am watching my packages starting up!!!

    I am now looking at a PF Sense menu in console and it sure is pretty!!!



  • @amarcino said in Netgate 3100 will not come up:

    @bmeeks

    Perfect, file system clean and I am watching my packages starting up!!!

    I am now looking at a PF Sense menu in console and it sure is pretty!!!

    Good deal. Probably did not wait quite long enough before pulling power when you did the Halt System command. Whenever I've had to do that, I wait like a couple of minutes or more to be 100% sure.



  • Seeing how long it took to halt from the console, I think I rushed it yesterday.

    Anyway, got things back and that is good. I will consider myself lucky to not have to go through that again.

    :)


  • Netgate Administrator

    Backup you config file. Keep a copy of the firmware recovery image on a USB stick. You can put the config file on that USB stick for even faster recovery.
    https://docs.netgate.com/pfsense/en/latest/backup/automatically-restore-during-install.html#configuration-from-usb-during-install
    With that in place you'll probably never see any issues again. 😁

    Steve



  • I want to thank you all, and the PFSense support team in helping me with this problem I had.


Log in to reply