Power loss hell. Help with fsck and corrupted drive

[tnine]

Hi all,
 We're recently had a slew of power failures at our office.  It's a small company, so we don't have a UPC yet.  Unfortunately one of the side effects of this is a corrupt file system.  Thankfully, we run pfSense off a USB drive, and I have 2 backups.  I was able to start the backup, then force mount the corrupt file system on the old USB and copy over the config.  Now that I'm back up and running, this leaves me with 2 questions.

1. Fschk doesn't appear to automatically run when the system recovers from an unclean mount.  Is it possible to turn this behavior on?  I think that if fsck had been run after every power loss, the drive wouldn't have progressed to a state that is too corrupt to recover.

2. If I set up a cron job to scp our config file nightly to a backup server, will the correctly retain all of our settings? I.E, is it functionally equivalent to downloading the config via the web interface?

Thanks,
Todd

[clarknova]

@tnine:

1. Fschk doesn't appear to automatically run when the system recovers from an unclean mount.  Is it possible to turn this behavior on?  I think that if fsck had been run after every power loss, the drive wouldn't have progressed to a state that is too corrupt to recover.

Any reason you're not running the embedded version? It mounts the filesystem read-only so a sudden poweroff won't touch the fs.

2. If I set up a cron job to scp our config file nightly to a backup server, will the correctly retain all of our settings? I.E, is it functionally equivalent to downloading the config via the web interface?

I believe this depends again on whether you're using the embedded or full version, but somebody else will know for sure.

[tnine]

1. We don't have an external syslog server (yet), so I want to keep logging on our current system.

2. I wrote a nightly backup script that uses wget to download the config file from the web interface.  I've attached it.  This works with version 1.2.3.

router.txt

[clarknova]

I don't see why an scp script on cron wouldn't work to backup your config file. It makes me wonder what that would gain you over the wget script you have now.

A cron job to make periodic scp dumps of the logs could also free you up to move to embedded without giving up a persistent log, although admittedly you would be missing a chunk of logs since the last transfer if the system went down unexpectedly.

db

[tnine]

Is it possible to change an existing installation to use the appliance install?  If not, how is the install process different than booting from CD then installing to a USB drive?  I didn't see any options that gave me a choice when installing from the CD.

[jimp]

@tnine:

Hi all,
 We're recently had a slew of power failures at our office.  It's a small company, so we don't have a UPC yet.  Unfortunately one of the side effects of this is a corrupt file system.  Thankfully, we run pfSense off a USB drive, and I have 2 backups.  I was able to start the backup, then force mount the corrupt file system on the old USB and copy over the config.  Now that I'm back up and running, this leaves me with 2 questions.

A cheap UPS can be had for ~$50. There's no excuse not to have one on a critical system, even if it only lasts for a few minutes, combined with the NUT package it can signal your system to shutdown cleanly.

1. Fschk doesn't appear to automatically run when the system recovers from an unclean mount.  Is it possible to turn this behavior on?  I think that if fsck had been run after every power loss, the drive wouldn't have progressed to a state that is too corrupt to recover.

It most definitely does run fsck on bootup, and fixes automatically. I've watched it happen several times, in testing and in live environments. Unless the drive is extremely corrupt, it always boots back up.

2. If I set up a cron job to scp our config file nightly to a backup server, will the correctly retain all of our settings? I.E, is it functionally equivalent to downloading the config via the web interface?

That should work, or you could cron a wget from a different box. There are examples on the Doc Wiki.

[clarknova]

@tnine:

Is it possible to change an existing installation to use the appliance install?

I couldn't find anything on that in the docs. If I were to try, this is what I would do. Again, I've never tried and I definitely don't guarantee it will not destroy your existing installation.

echo embedded > /etc/platform

and then use the firmware update page in the GUI to install the embedded image.

As for a new install of embedded, this is not done using the Live CD. You have to download the embedded image then use dd or physdiskwrite to write the image directly to the boot device. These instructions are what you're looking for:
http://m0n0.ch/wall/installation_generic.php

db