No Console Menu After Cold Boot - APU Board with 2.3.1/2



  • I have checked this on 6 APU.1D boards from PC Engines.  Installing v2.3.1 or v.2.3.2, everything functions normally until the first cold boot (ie. power fail).  After the first cold boot the console menu does not come up anymore after the Bootup complete message.  If I do a soft power down from the menus (console or web), then pull power, and power back up, everything is fine.  Do a soft reboot the same way and it's fine.  Cold boot, lose the menu, and it does not come back.  Everything seems to function normally.  Downgrade (fresh install) to 2.3 and it's all good.  So I feel like something broke in 2.3.1.

    Any ideas?



  • I can report a similar problem using 2.3.2-RELEASE (amd64). After an upgrade from 2.3.1-RELEASE on 28 August, I was able to perform clean restarts without incident. After a power failure, the firewall boots and operates normally, save for being "stuck" after the "Bootup Complete" message. The Web Configurator is reachable and fully functional; the console menu isn't displayed.

    Since the power failure, no combination of clean restarts and boot modes allows me to get to the console, though. Also tried reseating the RAM, with no change. I have yet to find any key combinations that drop me into the console, either.

    Though I realize this doesn't help you much, it suggests that the problem isn't limited to the hardware you're running.



  • Is your console via Serial or attached monitor?
    Embedded or hard drive install?

    If you can log in to the GUI, then your install is very likely to be good and one of your other assumptions is incorrect.

    This sounds like either you're attached via serial and the baud rate is wrong/changed or you installed a version that is serial only and you're expecting output on a monitor.



  • My particular installs, again, are on APU boards from PC Engines - embedded hardware, no video.  Same install and setup I've been doing for years.  I have the same results as @darmok.  I have logged in via the web console and confirmed that the serial image is installed and that the baud rate has not changes.  As we both said, this only occurs after a hard boot/power fail while running.  Going back to 2.3 fixes it.


  • Banned

    I am experiencing the same exact issue with pfSense 2.3.2. It was installed fresh on a fitlet-XA10-LAN (A10 Micro-6700T; 32 GB mSATA SSD; 4 GB DDR3 RAM). As I was testing the serial console on this machine, I was able to see the POST messages via serial console, then the FreeBSD messages, and finally I was able to see the pfSense menu. However, later I noticed that I could no longer see the pfSense menu via the serial console, and the last visible message is "Bootup complete". I decided to reinstall pfSense, and it fixed this issue. So, I was a happy camper (even though I didn't understand why exactly the pfSense menu stopped being displayed after a reboot in the first place).

    However, today I powered off pfSense as I was testing soft power offs, etc. I ended up rebooting and powering off (both soft and hard) pfSense several times. Once I was done, and I booted pfSense all the way up, I noticed that the problem with the pfSense menu not displayed on the console came back. This time, I no longer want to reinstall pfSense to fix this problem. So, I connected a monitor and keyboard to the fitlet-XA10-LAN, changed "Primary Console" in System > Advanced > Admin Access to VGA Console, and rebooted pfSense again. This time, all the output was to the monitor, and there was the same exact issue in that the pfSense menu is no longer displayed, and the last message visible on the console (this time the VGA console) is "Bootup complete". So, I pulled the USB keyboard out, and saw that there was a message logged to the console to the effect that the USB device was disconnected. So, the VGA console was still operational and accepting the messages from FreeBSD. The only thing that is not working is that pfSense doesn't load the pfSense menu to the console. The only way I can get to that text-driven pfSense menu is via SSH.

    I went ahead and changed the "Primary Console" setting back to "Serial Console," and rebooted pfSense again. However, pfSense menu is not being output to the console anymore. This is a big bummer! I initially thought I had some sort of corruption in my installation, or this was specific to fitlet-XA10-LAN, but now I see that others are experiencing the same problem with pfSense 2.3.2

    This pretty much renders the serial console useless for troubleshooting purposes. Yes, you can still use it to install pfSense, but once pfSense is rebooted (probably hard power off), there's no longer a way to use either the pfSense menu (for example for re-assigning interfaces) or even to drop down to the FreeBSD shell from a serial console.

    Is there a workaround that we can use to bring back this functionality until this is fixed in the new pfSense release?


  • Netgate

    What do you see if you run cat /etc/ttys in Diagnostics > Command Prompt??


  • Banned

    Strangely, the /etc/ttys file is empty.

    I tried to run the command

    
    cat /etc/ttys
    
    

    both from Diagnostics > Command Prompt and directly from the shell by connecting to the pfSense box via SSH and using Option 8 (Shell).


  • Netgate

    That's why you're not getting a console menu. No gettys spawned.

    You can try this:

    Boot single user (console boot menu option 2)
    mount -a -t ufs
    mv /etc/ttys /root/ttys.old
    tar xv -C / -f /usr/local/share/pfSense/base.txz ./etc/ttys
    /sbin/reboot

    As always, be ready to reinstall and restore when you do this sort of thing.

    Not sure why /etc/ttys falls victim to this but it also appears to be related to unclean shutdowns. I have not seen any other files that do this but I am not sure if I would trust it and might just reinstall/restore since it only takes a couple of minutes and you're rebooting anyway.


  • Banned

    Thank you for your suggestion.

    I was testing graceful shutdowns. I did unplug the pfSense box from power but only after shutting it down gracefully.

    In my opinion, the serial console is for troubleshooting purposes once pfSense has been installed. Now I'm troubleshooting the serial console (or VGA console) itself instead of having it as a troubleshooting tool. I guess as long as pfSense is working correctly,  I really don't want to reinstall it again, hoping that this will be fixed in a future release.

    So, provided that it has happened twice to me now within a month, and it apparently happens to others, do you think this should be addressed as a bug?

    Also, when the next version of pfSense comes out and I update to it from within GUI, would the update automatically fix this issue as the existing blank ttys file will be overwritten with a fresh copy from the update?

    Thank you.



  • Looks like this was quietly fixed in 2.3.2_1.

    Edit.  I was wrong.  I just factory defaulted this unit after I had tested this and the console menu didn't come up.  >:(


  • Netgate

    It doesn't really have anything to do with factory defaulting.

    Unless you have a readily-repeatable procedure, which would be great news.



  • I was actually setting up a new APU1D board for a site, and after it was ready I decided to use a NetGate unit instead. I made sure the APU board was up to date with 2.3.2_1, then pulled power to test. It booted up with the console menu. It's possible I didn't pull power long enough (it was just a few seconds). After the NetGate unit was set up and ready, I defaulted with APU board with console option 4, and it booted up with no console menu. So that was weird. I can certainly do more testing.



  • After upgrading to 2.3.2_1, I'm able to land in the console menu on restart. Something about 2.3.2_1 has to have improved matters, as nothing I tried under 2.3.2 got me past the "Bootup Complete" message. I abruptly pulled power for about 5 seconds and it booted normally.

    Again, this doesn't really help @acherman, other than to further confirm that hardware compatibility is probably an issue. I'm running pfSense on an old System76 Wilp5, which while more mainstream than an APU1D, has always had minor driver issues.



  • I'm also having the same problem. After a hard reset on pfSense 2.3.2_1 does not show up the menu. Everything is working fine except that the console.
    The detail is that I'm using a VM in ESXi and trying to access directly from the console.
    Any idea?



  • Same for me. Console lost  somewhere between 2.3.2 and 2.3.2_1 boots.
    cat /etc/ttys — empty
    board is J1900D2Y with NOTHING except lan cables and power cord connected, using IPMI.
    @Derelict:

    That's why you're not getting a console menu. No gettys spawned.

    You can try this:

    Boot single user (console boot menu option 2)
    mount -a -t ufs
    mv /etc/ttys /root/ttys.old
    tar xv -C / -f /usr/local/share/pfSense/base.txz ./etc/ttys
    /sbin/reboot

    As always, be ready to reinstall and restore when you do this sort of thing.

    Not sure why /etc/ttys falls victim to this but it also appears to be related to unclean shutdowns. I have not seen any other files that do this but I am not sure if I would trust it and might just reinstall/restore since it only takes a couple of minutes and you're rebooting anyway.

    Did the job!



  • Thank you for this post.  The solution brought my menu back to the console:

    Boot single user (console boot menu option 2)
    mount -a -t ufs
    mv /etc/ttys /root/ttys.old
    tar xv -C / -f /usr/local/share/pfSense/base.txz ./etc/ttys
    /sbin/reboot

    The reproduction steps are similar to what has already been posted.

    I pulled power on my system several times while working through a BIOS issue.  At some point during my testing, I noticed that the IPMI console showed "Bootup complete", but no menu after one of the system restarts.

    Thanks again for the solution.