Upgrade to ZFS

RyanM · Jun 9, 2021, 4:07 AM

Is there a way to tell if I am running ZFS? If I am not, what is the path for upgrading to ZFS? Is it straightforward?

Gertjan · Jun 9, 2021, 5:53 AM

Is it straightforward?

Maybe.
Before using it, of even think you have a choice, you should read about it.
By default, you stick - without maybe even knowing what it is or means - with what was installed when you installed a/the OS on a disk.

To use ZFS, there are some conditions.

To go from UFS to ZFS : re install pfSEnse from scratch, which is actually an easy thing to do.

@ryanm said in Upgrade to ZFS:

Is there a way to tell if I am running ZFS

For example :

zpool list

or : ask the guy who installed pfSense, as the choice was there during initial disk partitioning and formatting ;)

johnpoz · Jun 9, 2021, 10:20 AM

Or just look at the gui system widget

JeGr · Jun 9, 2021, 10:25 AM

@ryanm said in Upgrade to ZFS:

Is there a way to tell if I am running ZFS? If I am not, what is the path for upgrading to ZFS? Is it straightforward?

As @johnpoz linked above: check your dashboard widget. The disc space section tells you if your / (root) is running on either UFS or ZFS.

If I am not, what is the path for upgrading to ZFS? Is it straightforward?

Re-installation with "recover config.xml" and "Auto (ZFS)" will bring you up to speed and having ZFS in ~10min. :)

Cheers

johnpoz · Jun 9, 2021, 10:44 AM

@jegr is correct the only way move to zfs if your not is a clean install. If your on an netgate appliance just email, or open a ticket with netgate support and they will send you a + image for install. You do not need to have a support contract for this.

I moved to zfs with the release of 21.02.2 - the 21.02 had a bit of an issue with installing to zfs. There was a work around for it.. but I just waited til 21.02.2 release to make the move.

You can do a clean and then restore of your backup, or just clean and reconfigure by hand - really depends on how much stuff you have configured.. If your a pretty vanilla install - only take a few minutes to do a clean install and be up and running just out of the box.

jimp · Jun 9, 2021, 1:00 PM

If you are on CE, wait for 2.5.2 to reinstall with ZFS. It has a much better starting dataset layout.

If you are on Plus, 21.05 is good for amd64 devices (e.g. SG-5100, XG-7100, etc), but ARM devices don't yet support ZFS. We're working on bringing support for ZFS to the 64-bit ARM devices (SG-1100, SG-2100), though. If you have one of those, stay tuned.

johnpoz · Jun 9, 2021, 1:00 PM

@jimp said in Upgrade to ZFS:

It has a much better starting dataset layout.

Is there any change in the layout from 21.02.2 and 21.05? If did clean install with 21.02.2 to zfs, is there any reason that I should redo that with 21.05?

jimp · Jun 9, 2021, 1:12 PM

The layout is new to 21.05 and 2.5.2. If you have it, you'd know, since there are a lot more datasets and the pool is named pfSense instead of zroot

johnpoz · Jun 9, 2021, 3:34 PM

So should prob do a clean install then.. Since would assume the pfsense name will be what is used going forward. I do recall seeing something about recovery/restore related to the zfs name..

edit: Prob just wait til 21.09 comes out and do clean install then.. From reading the redmine about the recovery.

RyanM · Jun 9, 2021, 6:30 PM

Thank you all for the information.

I am running UFS (thanks @johnpoz for pointing out the widget).

@Gertjan I am somewhat familiar with ZFS, less so UFS. I am running a Free/TrueNAS server with ZFS and am familiar with the underlying concept of how it is a journal filesystem and such.

From what I recall reading on these forums, there are advantages of moving from UFS to ZFS regarding performance and reliability. Maybe I misread it, or the advantage is minor, idk. Even if minor, it sounds like there may be benefits of re-installing and using ZFS. I think my router originally had something in the 2.3.x or maybe early 2.4.x and I have been upgrading it pretty regularly since then.

@jimp thank you for pointing out to wait for 2.5.2. Since I am not on ZFS, I will wait until 2.5.2 before I mess with this. This only came up because I decided to upgrade to 2.5.1 yesterday and remembered that I had wanted to look into ZFS a few months ago but got distracted.

Ok, so I understand that the right path forward is a clean install. If I have been backing up my configuration to Netgate, is there anything special I need to check or ensure before I do the clean install? Would it be better to upgrade to 2.5.2, let the configuration backup, then clean install 2.5.2 and restore the config?

I assume after the clean install I will need to get connected to the internet and then use the restore from either the UI or the console menu? Is there an option to restore during installation? Does this require my Netgate login or something else?

Is there anything that may not be included in my configuration backup? Like package configuration?

AndyRH · Jun 9, 2021, 6:46 PM

You can practice with a spare PC or a VM. It does not need to replace the existing FW, just allow you to install and restore.
I install enough to get on the LAN then restore a backup through the web UI. I expect you will find it easy since you made TrueNAS work.

jimp · Jun 9, 2021, 7:09 PM

The main advantages of ZFS are stability and features such as native compression, mirroring, snapshots, etc. Some things we don't take full advantage of quite yet in pfSense.

ZFS isn't really known for its performance, but filesystem speed on a firewall isn't critical compared to stability. It also has increased memory requirements compared to UFS, which is why it's not ideal for certain platforms (e.g. 32-bit ARM, systems with low RAM).

When the time comes, take a fresh local backup (just in case), then boot the install media and choose "recover config.xml", then install, pick ZFS, select the drive(s), and go on from there. It will boot back up with your existing configuration in place, reinstall your packages, and be back up and running as it was before.

mer · Jun 15, 2021, 8:12 PM

Memory requirements. Lots of information around on that, most of the recommendations are 64bit system, at least 4GB RAM, more is way better.
From a design standpoint, ZFS was created to handle large disks.

From a usability POV, I think the best part of ZFS is Boot Environments (BEs). They are a very good way to upgrade a system safely and rollback if needed. UFS has had work recently to mimic/mirror that (think separate partitions for the root/boot filesystem, then flags used to tell which one to actually use).

pfSense with ZFS? On a big enough system, it's probably a good thing. I'd be interested in seeing how some of the things like mirrors and snapshots would be used.

johnpoz · Jun 15, 2021, 9:36 PM

The only reason I moved to zfs vs ufs is better handling of power loss. While I have never had my netgate loose power out of the blue. It is on a ups, etc. And power here is pretty stable and rarely have extended outages..

But since pfsense allows for choice.. And my box is never close to using its ram why not.. It wasn't something I was going to switch to out of the blue since it requires clean install - but with the move to new 2.5/21 versions - figured it was a good time to make the move..

Where they go with it in leveraging its other functions - can only wait and see.. I would love the ability to rollback to previous version on failed upgrade, etc. That would be slick!!

mer · Jun 15, 2021, 9:49 PM

@johnpoz said in Upgrade to ZFS:

Where they go with it in leveraging its other functions - can only wait and see.. I would love the ability to rollback to previous version on failed upgrade, etc. That would be slick!!

Yes yes it would. Having used ZFS for a while on the desktop, switching is trivial. The tricky parts are the logic of failback. If you make it at least to init you can start checking flags and such. But if the kernel crashes on the way up, manual intervention required.

JeGr · Jun 15, 2021, 10:30 PM

@mer said in Upgrade to ZFS:

pfSense with ZFS? On a big enough system, it's probably a good thing. I'd be interested in seeing how some of the things like mirrors and snapshots would be used.

I read that through the whole internet. ZFS... memory... blah. When last speaking with folks from TruNAS and iXsystems they all replied to me that most things are just BS nowadays. Yes in theory ZFS needs more RAM and more CPU. But the features they are used in are of no particular interest in a pfSense installation setting. Namely things like deduplication for large storages and other advanced features.

We are running pfSense on ZFS on hardware boxes for years now and I still have to find that "largely increased memory and CPU footprint" everyone is so quick to post when it comes to ZFS.
It all comes down to reliability and ZFS safed our customer's hides more then once when there was "a sudden power loss"... again. Since pfSense introduced ZFS in the installer we are using it and don't lost a single box since then.

Of course, I'd love to see snapshots and BEs popping up being used as that would bring back those early days when you could click the box besides the update button to "make a full backup prior to updateing the box" and you could then roll back easily. Having that with a combo of BEs and snapshots would be incredibly helpful!

But if it's just "should I run ZFS... " my answer would almost always* be a "YES".

Not counting virtual/VM setups. You can/should use & rely on your hypervisor for that.

johnpoz · Jun 15, 2021, 10:34 PM

@jegr said in Upgrade to ZFS:

Namely things like deduplication for large storages and other advanced features.

Yeah came to the same conclusion - not running it on a 100TB array ;) My pfsense is like a 24GB disk.. And it has 8GB of ram - so what if using ZFS uses a bit more than ufs..

mer · Jun 18, 2021, 1:26 PM

@johnpoz @JeGr Well, as you both point out, it's all about the use case, I was speaking in generalities, not as it would relate specifically to pfSense devices. Apologies if that was not clear.

Take the SG2100/SG3100, could you run ZFS on them? With a bit of work, probably. Would it be worth it? Up to you.

8GB RAM and 24GB disk? Of course.

The issue is perception. Remember when embedded devices had MB of RAM and KB of storage?
Heck how much does a low end phone have now?
4GB of RAM is a fairly low bar now so yes, of course, it makes sense.

JeGr · Jun 18, 2021, 1:26 PM

@mer said in Upgrade to ZFS:

@johnpoz @JeGr Well, as you both point out, it's all about the use case, I was speaking in generalities, not as it would relate specifically to pfSense devices. Apologies if that was not clear.

No problem :)

4GB of RAM is a fairly low bar now so yes, of course, it makes sense.

But that's what I was talking about. The "perception" - and in many heads that half-knowledge is still pretty current and active - is that you need veeeery much RAM to run ZFS because it is a big storage filesystem. That's simply not true nowadays according to those that deal with it on a day by day basis. Yeah sure, that was the case quite a bit ago, but today?

I run and have installed devices down to 1-2GB of RAM on a 32GB mSATA SSD - that is really small for a device today. And I didn't really see a jump in idle CPU power or RAM usage on those systems. As install size is around 1-2GB for a core installation with a few logs, that's nothing in ZFS terms. And compared with the ease of mind, that ZFS is far more robust than the default UFS installation without journaling etc against power loss - that was a really easy decision for us to roll out anything we can with ZFS. Really awaiting first contact with ARM-SGs and ZFS but also I don't expect them to be hit very hard by a switch to ZFS either.

Cheers :)

mer · Jun 18, 2021, 2:00 PM

@jegr Yep perception.

My problem is I still think of 1GB RAM and 8GB storage as huge. And yes the workload and features used are what kills it. ARC and other caching are primarily "read" operations (yes I know writes go through cache at some levels too), so if the system is primarily write (like pfSense would be) they are not going to try to use memory.
For a system like pfSense, 2GB RAM should be enough to let you use the good features of things like Boot Environments. The physical storage device I think is a bit more important than how much RAM in a small system.

Upgrading my home desktops, upgrades/updates are:
create a new BE
chroot into it
install updates/upgrades of base system and update all applications
set new BE active
reboot

If it fails, I haven't mucked up a running system, stopping in the boot loader lets you easily roll back. I can imagine a "run once" rc script that if you haven't been able to run (however that is defined) you simply activate the previous working BE and reboot and you're back up and running.

Anyway, it's Friday.