Broken unit won't fully boot



  • I had a box "die" on me Tuesday evening. It appeared to allow traffic through existing connections in the state table but no new connections. I also couldn't ping, SSH, or use the web to get into it from either the LAN or the WAN (through specific firewall rules). I had the client reboot the unit and it would boot and play the tone but no traffic period after that. I put together a new box and restored a backup of their config and dispatched someone out to replace it. We had them up in an hour. That's great.

    So I fire up the box at my desk, connect a console cable, and watch it boot to find the filesystem is full. Reboot to single-user mode and see that the suricata log file is using 40GB of the 60GB SSD. Ran fsck, remounted in rw and deleted the file and rebooted.

    It still doesn't finish booting. Here is what I see:

    Starting Secure Shell Services...done.
    Setting up polling defaults...done.
    Setting up interfaces microcode...done.
    Configuring loopback interface...done.
    Creating wireless clone interfaces...done.
    Configuring LAGG interfaces...done.
    Configuring VLAN interfaces...done.
    Configuring QinQ interfaces...done.
    Configuring CARP settings...done.
    Syncing OpenVPN settings...done.
    Apr 19 16:45:49 Firewall php-cgi: rc.bootup: Resyncing OpenVPN instances.
    Configuring firewall..Apr 19 16:45:49 Firewall php-cgi: rc.bootup: [squid] Installed but disabled. Not installing 'nat' rules.
    .Apr 19 16:45:49 Firewall php-cgi: rc.bootup: [squid] Installed but disabled. Not installing 'pfearly' rules.
    ..Apr 19 16:45:49 Firewall php-cgi: rc.bootup: [squid] Installed but disabled. Not installing 'filter' rules.
    .done.
    Starting PFLOG...done.
    Setting up gateway monitors...done.
    Synchronizing user settings...done.
    Apr 19 16:45:50 Firewall php-cgi: rc.bootup: The command '/usr/sbin/pw groupadd -n 'LocalAdmins' -g '2000' -M '0,2000' 2>&1' returned exit code '67', the output was 'pw: user `2000' does not exist'
    Starting webConfigurator...done.
    Configuring CRON...done.
    Starting NTP time client...done.
    Apr 19 16:45:51 Firewall php-cgi: rc.bootup: NTPD is starting up.
    Configuring firewall..Apr 19 16:45:51 Firewall php-cgi: rc.bootup: [squid] Installed but disabled. Not installing 'nat' rules.
    .Apr 19 16:45:51 Firewall php-cgi: rc.bootup: [squid] Installed but disabled. Not installing 'pfearly' rules.
    ..Apr 19 16:45:52 Firewall php-cgi: rc.bootup: [squid] Installed but disabled. Not installing 'filter' rules.
    .done.
    Generating RRD graphs...done.
    Starting syslog...Apr 19 16:45:59 Firewall syslogd: exiting on signal 15
    done.
    Starting CRON... done.
    Apr 19 16:45:59 Firewall php-fpm[19122]: /rc.start_packages: Restarting/Starting all packages.
     Starting package Cron...done.
     Starting package darkstat...done.
     Starting package iftop...done.
    Apr 19 16:45:59 Firewall php-fpm[19122]: /rc.start_packages: [lightsquid] Loaded default '/usr/local/etc/lightsquid/lightsquid.cfg.sample' configuration file.
    Apr 19 16:45:59 Firewall php-fpm[19122]: /rc.start_packages: [lightsquid] Successfully created '/usr/local/etc/lightsquid/lightsquid.cfg' configuration file.
    Apr 19 16:45:59 Firewall php-fpm[19122]: /rc.start_packages: [lightsquid] Removing all cronjobs...
     Starting package Lightsquid...done.
     Starting package mailreport...done.
     Starting package nmap...done.
     Starting package Notes...done.
     Starting package OpenVPN Client Export Utility...done.
     Starting package pfBlockerNG... Export Utility...done.
     Starting package RRD Summary... Export Utility...done.
     Starting package Service Watchdog...done.
     Starting package Shellcmd...done.
    Apr 19 16:46:01 Firewall php-fpm[19122]: /rc.start_packages: [squid] - squid_resync function call pr: bp: rpc:no
    Apr 19 16:46:02 Firewall php-fpm[42688]: /rc.filter_configure_sync: [squid] Installed but disabled. Not installing 'nat' rules.
    Apr 19 16:46:02 Firewall php-fpm[42688]: /rc.filter_configure_sync: [squid] Installed but disabled. Not installing 'pfearly' rules.
    Apr 19 16:46:02 Firewall php-fpm[42688]: /rc.filter_configure_sync: [squid] Installed but disabled. Not installing 'filter' rules.
    Apr 19 16:46:04 Firewall php-fpm[19122]: /rc.start_packages: [squid] Removing cronjobs ...
    Apr 19 16:46:04 Firewall php-fpm[19122]: /rc.start_packages: [squid] 'Local Cache' not configured, disk cache will be disabled.
    Apr 19 16:46:04 Firewall php-fpm[19122]: /rc.start_packages: [squid] Please, configure and save 'Local Cache' settings before enabling Squid proxy.
    Apr 19 16:46:04 Firewall php-fpm[19122]: /rc.start_packages: [squid] Antivirus features disabled.
    Apr 19 16:46:04 Firewall php-fpm[19122]: /rc.start_packages: [squid] Removing freshclam cronjob.
    Apr 19 16:46:04 Firewall php-fpm[19122]: /rc.start_packages: [squid] 'Local Cache' not configured, disk cache will be disabled.
    Apr 19 16:46:04 Firewall php-fpm[19122]: /rc.start_packages: [squid] Please, configure and save 'Local Cache' settings before enabling Squid proxy.
     Starting package squid3...done.
    Apr 19 16:46:05 Firewall php-fpm[19122]: /rc.start_packages: [squid] - squid_resync function call pr:1 bp: rpc:no
    Apr 19 16:46:06 Firewall (squid-1): No HTTP, HTTPS, or FTP ports configured
    Apr 19 16:46:06 Firewall php-fpm[46952]: /rc.filter_configure_sync: [squid] Installed but disabled. Not installing 'nat' rules.
    Apr 19 16:46:06 Firewall php-fpm[46952]: /rc.filter_configure_sync: [squid] Installed but disabled. Not installing 'pfearly' rules.
    Apr 19 16:46:07 Firewall php-fpm[46952]: /rc.filter_configure_sync: [squid] Installed but disabled. Not installing 'filter' rules.
    Apr 19 16:46:09 Firewall php-fpm[19122]: /rc.start_packages: [squid] Removing cronjobs ...
    Apr 19 16:46:09 Firewall php-fpm[19122]: /rc.start_packages: [squid] 'Local Cache' not configured, disk cache will be disabled.
    Apr 19 16:46:09 Firewall php-fpm[19122]: /rc.start_packages: [squid] Please, configure and save 'Local Cache' settings before enabling Squid proxy.
    Apr 19 16:46:09 Firewall php-fpm[19122]: /rc.start_packages: [squid] Antivirus features disabled.
    Apr 19 16:46:09 Firewall php-fpm[19122]: /rc.start_packages: [squid] Removing freshclam cronjob.
    Apr 19 16:46:09 Firewall php-fpm[19122]: /rc.start_packages: [squid] 'Local Cache' not configured, disk cache will be disabled.
    Apr 19 16:46:09 Firewall php-fpm[19122]: /rc.start_packages: [squid] Please, configure and save 'Local Cache' settings before enabling Squid proxy.
    Apr 19 16:46:09 Firewall (squid-1): No HTTP, HTTPS, or FTP ports configured
    Apr 19 16:46:10 Firewall php-fpm[19122]: /rc.start_packages: [squid] Stopping service...
    Apr 19 16:46:13 Firewall (squid-1): No HTTP, HTTPS, or FTP ports configured
    Apr 19 16:46:15 Firewall php-fpm[19122]: /rc.start_packages: The command '/usr/local/etc/rc.d/squid.sh stop' returned exit code '1', the output was 'squid: No running copy squid: ERROR: No running copy'
     Starting package squidGuard...done.
     Starting package sudo...done.
     Starting package suricata...done.
     Starting /usr/local/etc/rc.d/dnsbl.sh...done.
     Starting /usr/local/etc/rc.d/gkstartup.sh...done.
     Starting /usr/local/etc/rc.d/lighttpd_ls.sh...done.
     Starting /usr/local/etc/rc.d/sqp_monitor.sh...done.
    amdtemp0: <AMD CPU On-Die Thermal Sensors> on hostb5
    pkg-static: Warning: Major OS version upgrade detected.  Running "pkg-static install -f pkg" recommended
    pkg-static: Warning: Major OS version upgrade detected.  Running "pkg-static install -f pkg" recommended
    Apr 19 16:46:17 Firewall php-fpm[55428]: /rc.filter_configure_sync: [squid] Installed but disabled. Not installing 'nat' rules.
    pkg-static: Warning: Major OS version upgrade detected.  Running "pkg-static install -f pkg" recommended
    pkg-static: Warning: Major OS version upgrade detected.  Running "pkg-static install -f pkg" recommended
    Apr 19 16:46:17 Firewall php-fpm[55428]: /rc.filter_configure_sync: [squid] Installed but disabled. Not installing 'pfearly' rules.
    Apr 19 16:46:17 Firewall php-fpm[55428]: /rc.filter_configure_sync: [squid] Installed but disabled. Not installing 'filter' rules.
    pfSense (pfSense) 2.3.4-RELEASE amd64 Wed May 03 15:13:29 CDT 2017
    Bootup complete
    Apr 19 16:46:19 Firewall getty[83802]: open /dev/ttyv0: No such file or directory
    Apr 19 16:46:21 Firewall (squid-1): No HTTP, HTTPS, or FTP ports configured
    Apr 19 16:46:25 Firewall (squid-1): No HTTP, HTTPS, or FTP ports configured
    Apr 19 16:46:26 Firewall php-fpm[55428]: /rc.filter_configure_sync: [squid] Installed but disabled. Not installing 'nat' rules.
    Apr 19 16:46:26 Firewall php-fpm[55428]: /rc.filter_configure_sync: [squid] Installed but disabled. Not installing 'pfearly' rules.
    Apr 19 16:46:26 Firewall php-fpm[55428]: /rc.filter_configure_sync: [squid] Installed but disabled. Not installing 'filter' rules.
    Apr 19 16:46:27 Firewall php-fpm[55428]: /rc.filter_configure_sync: [squid] Installed but disabled. Not installing 'nat' rules.
    Apr 19 16:46:27 Firewall php-fpm[55428]: /rc.filter_configure_sync: [squid] Installed but disabled. Not installing 'pfearly' rules.
    Apr 19 16:46:28 Firewall php-fpm[55428]: /rc.filter_configure_sync: [squid] Installed but disabled. Not installing 'filter' rules.
    Apr 19 16:46:28 Firewall (squid-1): No HTTP, HTTPS, or FTP ports configured
    Apr 19 16:46:31 Firewall (squid-1): No HTTP, HTTPS, or FTP ports configured
    Apr 19 16:46:35 Firewall (squid-1): No HTTP, HTTPS, or FTP ports configured
    Apr 19 16:46:35 Firewall squid[86325]: Exiting due to repeated, frequent failures
    Apr 19 16:47:22 Firewall (squid-1): No HTTP, HTTPS, or FTP ports configured
    Apr 19 16:47:25 Firewall (squid-1): No HTTP, HTTPS, or FTP ports configured
    Apr 19 16:47:27 Firewall php-fpm[93619]: /rc.filter_configure_sync: [squid] Installed but disabled. Not installing 'nat' rules.
    Apr 19 16:47:27 Firewall php-fpm[93619]: /rc.filter_configure_sync: [squid] Installed but disabled. Not installing 'pfearly' rules.
    Apr 19 16:47:27 Firewall php-fpm[93619]: /rc.filter_configure_sync: [squid] Installed but disabled. Not installing 'filter' rules.
    Apr 19 16:47:27 Firewall php-fpm[3732]: /rc.filter_configure_sync: [squid] Installed but disabled. Not installing 'nat' rules.
    Apr 19 16:47:28 Firewall php-fpm[3732]: /rc.filter_configure_sync: [squid] Installed but disabled. Not installing 'pfearly' rules.
    Apr 19 16:47:28 Firewall php-fpm[3732]: /rc.filter_configure_sync: [squid] Installed but disabled. Not installing 'filter' rules.
    Apr 19 16:47:29 Firewall (squid-1): No HTTP, HTTPS, or FTP ports configured
    Apr 19 16:47:32 Firewall (squid-1): No HTTP, HTTPS, or FTP ports configured
    Apr 19 16:47:35 Firewall (squid-1): No HTTP, HTTPS, or FTP ports configured
    Apr 19 16:47:35 Firewall squid[98501]: Exiting due to repeated, frequent failures
    Apr 19 16:48:22 Firewall (squid-1): No HTTP, HTTPS, or FTP ports configured
    Apr 19 16:48:25 Firewall (squid-1): No HTTP, HTTPS, or FTP ports configured
    Apr 19 16:48:27 Firewall php-fpm[3732]: /rc.filter_configure_sync: [squid] Installed but disabled. Not installing 'nat' rules.
    Apr 19 16:48:27 Firewall php-fpm[3732]: /rc.filter_configure_sync: [squid] Installed but disabled. Not installing 'pfearly' rules.
    Apr 19 16:48:27 Firewall php-fpm[3732]: /rc.filter_configure_sync: [squid] Installed but disabled. Not installing 'filter' rules.
    Apr 19 16:48:28 Firewall php-fpm[3732]: /rc.filter_configure_sync: [squid] Installed but disabled. Not installing 'nat' rules.
    Apr 19 16:48:28 Firewall php-fpm[3732]: /rc.filter_configure_sync: [squid] Installed but disabled. Not installing 'pfearly' rules.
    Apr 19 16:48:28 Firewall php-fpm[3732]: /rc.filter_configure_sync: [squid] Installed but disabled. Not installing 'filter' rules.
    Apr 19 16:48:29 Firewall (squid-1): No HTTP, HTTPS, or FTP ports configured
    Apr 19 16:48:32 Firewall (squid-1): No HTTP, HTTPS, or FTP ports configured
    Apr 19 16:48:36 Firewall (squid-1): No HTTP, HTTPS, or FTP ports configured
    Apr 19 16:48:36 Firewall squid[11474]: Exiting due to repeated, frequent failures®
    
    

    It never finishes booting to give me the option screen or the option to drop to CLI. I've never seen this before. Any ideas?


  • Netgate Administrator

    @Stewart said in Broken unit won't fully boot:

    pkg-static: Warning: Major OS version upgrade detected

    That implies it is either running 2.3.X and has pulled in 2.4.X packages or is set the dev channel and is trying to pull in 2.5.X packages. You can probably recover it by doing this:
    https://docs.netgate.com/pfsense/en/latest/install/upgrade-troubleshooting.html#upgrade-not-offered-library-errors

    But it will be quicker, and cleaner, to just reinstall at this point.

    The Suricata package had a bug in it at one point that meant log rotation was not working correctly. You had to go to the log management tab and save the default settings there to activate it. I imagine that's what you hit there.

    Steve


Log in to reply