PfSense serial console hangs after boot

Mooash

I've lost all access to my pfSense box after it died after a full disk. I only noticed this because I was editing a OpenVPN client (possibly I fudged this somehow?). Initial symptoms were just small things like the DNS daemon not working (fully crashed) then pfSense itself stopped all connectivity. I jumped on the serial console and removed some ntopng files (/var/db/ntopng/*) and uninstalled the package and after that didn't work rebooted the router. pfSense now hangs after its finished "booting" and doesn't respond to any serial commands nor bring up its pppoe interface or respond on any others. These are the last lines printed out in my serial session:

Disabling APM on /dev/ad4

     ___
 ___/ f \
/ p \___/ Sense
\___/   \
    \___/

Welcome to pfSense 2.3.4-RELEASE on the 'pfSense' platform...

No core dumps found.
Creating symlinks......ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/graphviz /usr/local/lib/ipsec /usr/local/lib/mysql /usr/local/lib/perl5/5.24/mach/CORE
32-bit compatibility ldconfig path: /usr/lib32
done.
External config loader 1.0 is now starting... ada0s1 ada0s1aw starting...
Launching the init system....... done.
Initializing.................. done.
Starting device manager (devd)...done.
Loading configuration......done.
Updating configuration...done.
Cleaning backup cache.................................done.
Setting up extended sysctls...done.
Setting timezone...done.
Configuring loopback interface...done.
Starting syslog...done.
Starting Secure Shell Services...done.
Setting up polling defaults...done.
Setting up interfaces microcode...done.
Configuring loopback interface...done.
Creating wireless clone interfaces...done.
Configuring LAGG interfaces...done.
Configuring VLAN interfaces...done.
Configuring QinQ interfaces...done.
Configuring CARP settings...done.
Syncing OpenVPN settings...done.
Nov  6 23:51:54 gateway php-cgi: rc.bootup: Resyncing OpenVPN instances.
Configuring firewall......done.
Starting PFLOG...done.
Setting up gateway monitors...done.
Synchronizing user settings...done.
Starting webConfigurator...done.
Configuring CRON...done.
Starting NTP time client...Nov  6 23:51:56 gateway php-cgi: rc.bootup: NTPD is starting up.
done.
Configuring firewall......done.
Generating RRD graphs...done.
Starting syslog...Nov  6 23:52:02 gateway syslogd: exiting on signal 15
done.
Starting CRON... done.
Nov  6 23:52:02 gateway php-fpm[286]: /rc.start_packages: Restarting/Starting all packages.
 Starting package arping...done.
 Starting package AutoConfigBackup...done.
 Starting package OpenVPN Client Export Utility...done.
 Starting /usr/local/etc/rc.d/ntopng.sh...done.
pfSense (pfSense) 2.3.4-RELEASE amd64 Wed May 03 15:13:29 CDT 2017
Bootup complete
Nov  6 23:52:05 gateway getty[48324]: open /dev/ttyv0: No such file or directory

And after that nothing. I'm running a pcEngines APU if that helps. I've also tried multiple baud settings to no luck.

I've tried restoring a config.xml by loading 2.3.4 on a USB but that also fails (doesn't show an error, just says it failed and reboots). I tried running this several times just in case it was tripping over a dud disk part. I've also tried following these instructions are replaced my ttys file, but that didn't help either.

Apologies if I've missed any information, been working on this for 3 hours and my brain is fried!

jimp

Are you saying that it still misbehaves after reinstalling the OS completely? Are you certain it actually reinstalled? Which installation image did you use?

It's possible that the disk is failing.

Mooash

Be reinstall I meant trying to restore the configuration. I've managed to get the system going again however through a combination of moves:

1. Moved /cf/conf/config.xml to /cf/conf/backup/config-broken.xml and moved a known working config.xml in its place.
2. Removed ntopng's rc.d file so it wouldn't start on boot

Not 100% sure which of these fixed it but I'm sure a broken ntopng trying to start on boot wasn't helping. Either way I've got a working pfSense again that I'm sure I wont try and reboot and break things anytime soon!

Time to migrate onto better hardware. :)

Just to be safe, download a config.xml from the WebUI. If the device crashes completely or the FS becomes corrupted, it's an easy way to get back to work. Something else: when reinstalling pfSense, some hardware requires a specific image to get going. You probably already used the right image: pfSense-memstick-serial-VERSION. If you didn't get a console after boot, you need to check the BIOS as well. pfSense just uses whatever the hardware has, but if it isn't configured properly it will just die as soon as it switches to the right console.

If you have spare time, I'd suggest (after backup) trying to install or at least live-boot pfSense, just to make sure your hardware isn't about to die. The APU's usually last a pretty long time.