PFSense hardware not using embedded?

asiTechsupport · Feb 21, 2017, 5:14 PM

Okay, so I've had my FOURTH issue with corruption with one of the SG-2220 firewalls I have deployed.

The first few times were with a couple of brand new units last year, and honestly I was in a pinch and re-installed and moved on. Well, one that was servicing a satellite office went down last week, and I found that the filesystem had become corrupted.

So is there something I'm missing in regards to the image for the PFSense hardware? Is there a "supported" way to make these more tolerant of power failures, like setting the filesystem to read-only?

I need these to be extremely reliable, so any advice and direction you can provide or point me to (I'm probably missing something in my google-foo?) I'd appreciated it!

doktornotor · Feb 21, 2017, 5:32 PM

No, there are no usable workaround. The solution it to install 2.4 on ZFS.

asiTechsupport · Feb 21, 2017, 6:32 PM

@doktornotor:

No, there are no usable workaround. The solution it to install 2.4 on ZFS.

Is there a recommended way to load embedded on these systems?

Honestly, I feel a little nervous about the fact that I had no secondary slice to boot when I found them corrupted.

pfBasic · Feb 21, 2017, 9:02 PM

@asiTechsupport:

@doktornotor:

No, there are no usable workaround. The solution it to install 2.4 on ZFS.

Is there a recommended way to load embedded on these systems?

Honestly, I feel a little nervous about the fact that I had no secondary slice to boot when I found them corrupted.

I just got this installed last night, so I obviously can't personally attest to the durability but in theory it should last a very long time. You can check out my settings here:
https://forum.pfsense.org/index.php?topic=120340.msg695305#msg695305

There's no more nanoBSD in 2.4, however ZFS gives you software raid, is very resistant to power failures, and has several other features to protect your data. You can still use a RAM disk and install without swap so I'm betting that this kind of ZFS install would be more resilient than nanoBSD ever was.

Using a cheap 5 pack of USB's like I did you can get an even more durable install by installing two of the drives in a mirror, then add the other three to the pool as hot spares with autoreplace=on. This way you have a redundant disk in the event of a failure, and three more drives standing by to replace the failed disk that haven't been written to at all.

asiTechsupport · Feb 23, 2017, 6:17 PM

@pfBasic:

@asiTechsupport:

@doktornotor:

No, there are no usable workaround. The solution it to install 2.4 on ZFS.

Is there a recommended way to load embedded on these systems?

Honestly, I feel a little nervous about the fact that I had no secondary slice to boot when I found them corrupted.

There's no more nanoBSD in 2.4, however ZFS gives you software raid, is very resistant to power failures, and has several other features to protect your data. You can still use a RAM disk and install without swap so I'm betting that this kind of ZFS install would be more resilient than nanoBSD ever was.

Thank you for the write-up on the new ZFS settings that you used!

HOWEVER, while I realize that 2.4 is coming soon, and that ZFS will address my issues in the future, my concern is with my production systems right now.

So again, I ask, is there a recommended way to install embedded on the current SG-series firewalls until 2.4 comes out?

doktornotor · Feb 23, 2017, 6:25 PM

There is nothing that'd help with UFS corruption. Keep it on UPS and pray. ::)

NOYB · Feb 23, 2017, 7:53 PM

How frequently is the RAM disk being backed up? If it is real often you might increase the interval to minimize the amount of writing to the file system.

But if it is a power outage causing the corruption then redundancy, UPS, and backups for recovery are probably the answers for now.

As note previously by others 2.4 with ZFS should improve the situation in the future.

jimp · Feb 27, 2017, 6:37 PM

We have put some changes into 2.3.3 to change how fsck is handled at boot time that should help as well.

But 2.4+ZFS is the best option when it's released.

asiTechsupport · Mar 10, 2017, 10:06 PM

@jimp:

We have put some changes into 2.3.3 to change how fsck is handled at boot time that should help as well.

But 2.4+ZFS is the best option when it's released.

Okay, thanks everyone for the insights and information!

I'm looking forward to 2.4 with ZFS support, since that will definitely put my mind at ease! I'm working with the BETA at my home office and it's looking good!

I'm assuming that with 2.4 there will no longer be an embedded version at all… is that correct?

Also, if I upgrade to 2.4, is there any option to convert UFS to ZFS? I'm assuming there's probably not, and I'm fine with that, just curious that's all. ;)

pfBasic · Mar 10, 2017, 10:31 PM

I found this:

http://docs.oracle.com/cd/E19253-01/819-5461/ggpdm/index.html

Solaris is not FreeBSD so it may not translate well (or at all). I didn't read into it at all, just saw the title. You are ultimately probably better of reinstalling to ZFS and restoring your config.xml & whatever other files you may have that wouldn't be included in the config.

asiTechsupport · Mar 10, 2017, 10:40 PM

@pfBasic:

I found this:

http://docs.oracle.com/cd/E19253-01/819-5461/ggpdm/index.html

Solaris is not FreeBSD so it may not translate well (or at all). I didn't read into it at all, just saw the title. You are ultimately probably better of reinstalling to ZFS and restoring your config.xml & whatever other files you may have that wouldn't be included in the config.

I actually was reading a couple other posts in the forums and noticed a thread that said specifically that there are no tools to upgrade from UFS to ZFS. So no conversion will happen automagically, which is fine.

whosmatt · Mar 11, 2017, 4:55 AM

@asiTechsupport:

I actually was reading a couple other posts in the forums and noticed a thread that said specifically that there are no tools to upgrade from UFS to ZFS. So no conversion will happen automagically, which is fine.

A simple fresh install using ZFS and then restore of config should get you there. I might take it a step further and upgrade to 2.4 first, then back up the config, then do the fresh install and restore. That way you know your config is from the same version.

Guest · Mar 23, 2017, 4:44 PM

Okay, so I've had my FOURTH issue with corruption with one of the SG-2220 firewalls I have deployed.

A small PSU would be fine for that action to work around either which version of pfSense you are using.
On what you was installing it? (eMMC storage (internal) or mSATa or M.2 or SSD,…..)

The first few times were with a couple of brand new units last year, and honestly I was in a pinch and re-installed and moved on. Well, one that was servicing a satellite office went down last week, and I found that the filesystem had become corrupted.

What image was you taking for the re-installation? The ADI one from your pfSense-account?

So is there something I'm missing in regards to the image for the PFSense hardware? Is there a "supported" way to make these more tolerant of power failures, like setting the filesystem to read-only?

No power is no power and file corruption is often the result of this action, not only at the pfSense units to be clear at this point.

I need these to be extremely reliable, so any advice and direction you can provide or point me to (I'm probably missing
something in my google-foo?) I'd appreciated it!

As said before 2.4 & ZFS would march but for the entire problem you should be going the safe way together with a small PSU.

conor · Mar 24, 2017, 4:42 PM

I had a similar problem with SG-2220 and SG-2440 devices have seen it on over 20 of them now. Customers have a nasty habit of power cycling devices by pulling the power cable to "restart it".

We got around it by doing the following, it may help some people:

1. We rebuilt all our devices with nanobsd, we got a 8GB USB stick, popped the the FreeBSD 10.3 installer onto it and expanded the partition, then we loaded the nanobsd pfsense version 2.3 image onto the usb stick as a file. We boot from the usb stick into the freebsd installer, we exit the installer to shell and then dd the nanobsd file to the on board memory of the 2220 and 2440.

2. In the web gui we assigned the /var and /tmp to use memory instead of disk.

3. Last thing we set the device to mount in read-only after boot, we added the command:

exec("/bin/mount -o ro /");
```to the  file: /etc/rc.bootup

(item 3) - This is not advisable for the inexperienced as it will break with the following:
1\. when you upgrade you'll first have to make your root slice writable.
2\. upgrades will likely wipe out your change.
3\. After configuration changes make sure to mount read-only sometimes gets set to rw.

With this setup we connected a 2220 to a timer and power cycled it every 15 minutes, we used nagios to check and and left it cycling for 7 days (>600 power cuts) and no corruption.

The other thing we did was to purchase push button switches which are installed into the front right antenna slot and connected to the pins for power switch at the top of the board (just behind the status LED). Now when a user presses this button the OS powers off correctly. We advise customers that if they want to power cycle they have to do it that way.

Haven't tested 2.4 with zfs in regards to power but will at some point in the future.