Firewall (1/2?) down



  • I have been running pfSense-2.2.3 embedded for quite a while.  Today something odd happened, and my internet connection seemed to go down.  I couldn't connect to the firewall GUI, so I restarted it by removing the power.

    The firewall logs to an internal server, and I'm repeatedly seeing:

    Jul 15 11:01:25 192.168.15.1 check_reload_status: Could not connect to /var/run/php-fpm.socket
    Jul 15 11:01:25 192.168.15.1 check_reload_status: Could not connect to /var/run/php-fpm.socket
    Jul 15 11:01:25 192.168.15.1 check_reload_status: Could not connect to /var/run/php-fpm.socket
    Jul 15 11:01:25 192.168.15.1 check_reload_status: Could not connect to /var/run/php-fpm.socket

    Trying to open the http console I get 503 - Service not available.

    Trying to ssh to the firewall times out now.

    I connected my null modem cable, and am presented with a new login prompt that I don't remember ever seeing before.  I tried the firewall admin credentials and they don't work.

    The firewall is passing traffic, because I'm posting this from behind it.  However none of my VPNs are up so my office is effectively down.

    I'm not sure what to try next, does anyone have any suggestions?

    Edit: I rebooted the firewall, and it looks like my filesystem is corrupted. I watched it boot and complain about missing files in /etc and not being able to get the uid for nobody.

    When installing 2.2.3 on embedded, does pfsense setup multiple partition copies where I might be able to boot to the old partition? or is my best bet re-imaging my flash and restoring the backup I have?

    Thanks
    Tony



  • You have the broken/corrupted/gone password file problem. There can be/are issues on 2.2.3 if the power is unexpectedly removed.
    Go back to the other slice, that probably has 2.2.2 on it - power up with the console connected (probably need to be at 9600 baud for the BIOS text output). When it gives you the choice of "1" or "2" then press the opposite of what it is trying to boot from.

    After booting into the old slice, either leave yourself on 2.2.2 and wait for 2.2.4 release, or download the latest 2.2.4-DEVELOPMENT upgrade image from snapshots.pfsense.org and use that to upgrade.

    The file system issues are fixed in 2.2.4-DEVELOPMENT.



  • Thank you very much for your detailed reply.

    Unfortunately, I had already shutdown, re-imaged and restored.  Thankfully, I had a very recent backup.

    I'll get the 2.2.4-DEVELOPMENT snapshot installed tomorrow am.  Can't have any more down time today if it can be helped.

    Do you happen to know if 2.2.4 has any more fixes for Multi Tunnel IPSEC, I still have the same rekey issues since 2.2.1, which is why I tried 2.2.3.

    Thanks again.

    Tony


Log in to reply