Upgrade to ZFS

johnpoz · Jun 9, 2021, 10:44 AM

@jegr is correct the only way move to zfs if your not is a clean install. If your on an netgate appliance just email, or open a ticket with netgate support and they will send you a + image for install. You do not need to have a support contract for this.

I moved to zfs with the release of 21.02.2 - the 21.02 had a bit of an issue with installing to zfs. There was a work around for it.. but I just waited til 21.02.2 release to make the move.

You can do a clean and then restore of your backup, or just clean and reconfigure by hand - really depends on how much stuff you have configured.. If your a pretty vanilla install - only take a few minutes to do a clean install and be up and running just out of the box.

jimp · Jun 9, 2021, 1:00 PM

If you are on CE, wait for 2.5.2 to reinstall with ZFS. It has a much better starting dataset layout.

If you are on Plus, 21.05 is good for amd64 devices (e.g. SG-5100, XG-7100, etc), but ARM devices don't yet support ZFS. We're working on bringing support for ZFS to the 64-bit ARM devices (SG-1100, SG-2100), though. If you have one of those, stay tuned.

johnpoz · Jun 9, 2021, 1:00 PM

@jimp said in Upgrade to ZFS:

It has a much better starting dataset layout.

Is there any change in the layout from 21.02.2 and 21.05? If did clean install with 21.02.2 to zfs, is there any reason that I should redo that with 21.05?

jimp · Jun 9, 2021, 1:12 PM

The layout is new to 21.05 and 2.5.2. If you have it, you'd know, since there are a lot more datasets and the pool is named pfSense instead of zroot

johnpoz · Jun 9, 2021, 3:34 PM

So should prob do a clean install then.. Since would assume the pfsense name will be what is used going forward. I do recall seeing something about recovery/restore related to the zfs name..

edit: Prob just wait til 21.09 comes out and do clean install then.. From reading the redmine about the recovery.

RyanM · Jun 9, 2021, 6:30 PM

Thank you all for the information.

I am running UFS (thanks @johnpoz for pointing out the widget).

@Gertjan I am somewhat familiar with ZFS, less so UFS. I am running a Free/TrueNAS server with ZFS and am familiar with the underlying concept of how it is a journal filesystem and such.

From what I recall reading on these forums, there are advantages of moving from UFS to ZFS regarding performance and reliability. Maybe I misread it, or the advantage is minor, idk. Even if minor, it sounds like there may be benefits of re-installing and using ZFS. I think my router originally had something in the 2.3.x or maybe early 2.4.x and I have been upgrading it pretty regularly since then.

@jimp thank you for pointing out to wait for 2.5.2. Since I am not on ZFS, I will wait until 2.5.2 before I mess with this. This only came up because I decided to upgrade to 2.5.1 yesterday and remembered that I had wanted to look into ZFS a few months ago but got distracted.

Ok, so I understand that the right path forward is a clean install. If I have been backing up my configuration to Netgate, is there anything special I need to check or ensure before I do the clean install? Would it be better to upgrade to 2.5.2, let the configuration backup, then clean install 2.5.2 and restore the config?

I assume after the clean install I will need to get connected to the internet and then use the restore from either the UI or the console menu? Is there an option to restore during installation? Does this require my Netgate login or something else?

Is there anything that may not be included in my configuration backup? Like package configuration?

AndyRH · Jun 9, 2021, 6:46 PM

You can practice with a spare PC or a VM. It does not need to replace the existing FW, just allow you to install and restore.
I install enough to get on the LAN then restore a backup through the web UI. I expect you will find it easy since you made TrueNAS work.

jimp · Jun 9, 2021, 7:09 PM

The main advantages of ZFS are stability and features such as native compression, mirroring, snapshots, etc. Some things we don't take full advantage of quite yet in pfSense.

ZFS isn't really known for its performance, but filesystem speed on a firewall isn't critical compared to stability. It also has increased memory requirements compared to UFS, which is why it's not ideal for certain platforms (e.g. 32-bit ARM, systems with low RAM).

When the time comes, take a fresh local backup (just in case), then boot the install media and choose "recover config.xml", then install, pick ZFS, select the drive(s), and go on from there. It will boot back up with your existing configuration in place, reinstall your packages, and be back up and running as it was before.

mer · Jun 15, 2021, 8:12 PM

Memory requirements. Lots of information around on that, most of the recommendations are 64bit system, at least 4GB RAM, more is way better.
From a design standpoint, ZFS was created to handle large disks.

From a usability POV, I think the best part of ZFS is Boot Environments (BEs). They are a very good way to upgrade a system safely and rollback if needed. UFS has had work recently to mimic/mirror that (think separate partitions for the root/boot filesystem, then flags used to tell which one to actually use).

pfSense with ZFS? On a big enough system, it's probably a good thing. I'd be interested in seeing how some of the things like mirrors and snapshots would be used.

johnpoz · Jun 15, 2021, 9:36 PM

The only reason I moved to zfs vs ufs is better handling of power loss. While I have never had my netgate loose power out of the blue. It is on a ups, etc. And power here is pretty stable and rarely have extended outages..

But since pfsense allows for choice.. And my box is never close to using its ram why not.. It wasn't something I was going to switch to out of the blue since it requires clean install - but with the move to new 2.5/21 versions - figured it was a good time to make the move..

Where they go with it in leveraging its other functions - can only wait and see.. I would love the ability to rollback to previous version on failed upgrade, etc. That would be slick!!

mer · Jun 15, 2021, 9:49 PM

@johnpoz said in Upgrade to ZFS:

Where they go with it in leveraging its other functions - can only wait and see.. I would love the ability to rollback to previous version on failed upgrade, etc. That would be slick!!

Yes yes it would. Having used ZFS for a while on the desktop, switching is trivial. The tricky parts are the logic of failback. If you make it at least to init you can start checking flags and such. But if the kernel crashes on the way up, manual intervention required.

JeGr · Jun 15, 2021, 10:30 PM

@mer said in Upgrade to ZFS:

pfSense with ZFS? On a big enough system, it's probably a good thing. I'd be interested in seeing how some of the things like mirrors and snapshots would be used.

I read that through the whole internet. ZFS... memory... blah. When last speaking with folks from TruNAS and iXsystems they all replied to me that most things are just BS nowadays. Yes in theory ZFS needs more RAM and more CPU. But the features they are used in are of no particular interest in a pfSense installation setting. Namely things like deduplication for large storages and other advanced features.

We are running pfSense on ZFS on hardware boxes for years now and I still have to find that "largely increased memory and CPU footprint" everyone is so quick to post when it comes to ZFS.
It all comes down to reliability and ZFS safed our customer's hides more then once when there was "a sudden power loss"... again. Since pfSense introduced ZFS in the installer we are using it and don't lost a single box since then.

Of course, I'd love to see snapshots and BEs popping up being used as that would bring back those early days when you could click the box besides the update button to "make a full backup prior to updateing the box" and you could then roll back easily. Having that with a combo of BEs and snapshots would be incredibly helpful!

But if it's just "should I run ZFS... " my answer would almost always* be a "YES".

Not counting virtual/VM setups. You can/should use & rely on your hypervisor for that.

johnpoz · Jun 15, 2021, 10:34 PM

@jegr said in Upgrade to ZFS:

Namely things like deduplication for large storages and other advanced features.

Yeah came to the same conclusion - not running it on a 100TB array ;) My pfsense is like a 24GB disk.. And it has 8GB of ram - so what if using ZFS uses a bit more than ufs..

mer · Jun 18, 2021, 1:26 PM

@johnpoz @JeGr Well, as you both point out, it's all about the use case, I was speaking in generalities, not as it would relate specifically to pfSense devices. Apologies if that was not clear.

Take the SG2100/SG3100, could you run ZFS on them? With a bit of work, probably. Would it be worth it? Up to you.

8GB RAM and 24GB disk? Of course.

The issue is perception. Remember when embedded devices had MB of RAM and KB of storage?
Heck how much does a low end phone have now?
4GB of RAM is a fairly low bar now so yes, of course, it makes sense.

JeGr · Jun 18, 2021, 1:26 PM

@mer said in Upgrade to ZFS:

@johnpoz @JeGr Well, as you both point out, it's all about the use case, I was speaking in generalities, not as it would relate specifically to pfSense devices. Apologies if that was not clear.

No problem :)

4GB of RAM is a fairly low bar now so yes, of course, it makes sense.

But that's what I was talking about. The "perception" - and in many heads that half-knowledge is still pretty current and active - is that you need veeeery much RAM to run ZFS because it is a big storage filesystem. That's simply not true nowadays according to those that deal with it on a day by day basis. Yeah sure, that was the case quite a bit ago, but today?

I run and have installed devices down to 1-2GB of RAM on a 32GB mSATA SSD - that is really small for a device today. And I didn't really see a jump in idle CPU power or RAM usage on those systems. As install size is around 1-2GB for a core installation with a few logs, that's nothing in ZFS terms. And compared with the ease of mind, that ZFS is far more robust than the default UFS installation without journaling etc against power loss - that was a really easy decision for us to roll out anything we can with ZFS. Really awaiting first contact with ARM-SGs and ZFS but also I don't expect them to be hit very hard by a switch to ZFS either.

Cheers :)

mer · Jun 18, 2021, 2:00 PM

@jegr Yep perception.

My problem is I still think of 1GB RAM and 8GB storage as huge. And yes the workload and features used are what kills it. ARC and other caching are primarily "read" operations (yes I know writes go through cache at some levels too), so if the system is primarily write (like pfSense would be) they are not going to try to use memory.
For a system like pfSense, 2GB RAM should be enough to let you use the good features of things like Boot Environments. The physical storage device I think is a bit more important than how much RAM in a small system.

Upgrading my home desktops, upgrades/updates are:
create a new BE
chroot into it
install updates/upgrades of base system and update all applications
set new BE active
reboot

If it fails, I haven't mucked up a running system, stopping in the boot loader lets you easily roll back. I can imagine a "run once" rc script that if you haven't been able to run (however that is defined) you simply activate the previous working BE and reboot and you're back up and running.

Anyway, it's Friday.

JeGr · Jun 18, 2021, 2:39 PM

@mer said in Upgrade to ZFS:

Upgrading my home desktops, upgrades/updates are:
create a new BE
chroot into it
install updates/upgrades of base system and update all applications
set new BE active
reboot

That's exactly what I would love to see "kinda" automated by updates of pfSense. And if there's a problem or via console you can hop back to the old BE. Or it automatically gets reselected as default after failed boot attempts etc. :)

mer · Jun 18, 2021, 2:57 PM

@jegr Absolutely would love that. I cringe everytime I'm forced to upgrade my wife's Windows machine. Cross my fingers, light a candle, pray to all the gods that it works before I say "Yes, restart now".

the ixSystems folks were doing that with TrueOS/Trident in an automatic fashion. I do it manually because I'm not far removed from "upgrading from source" (old dog, could learn new tricks, but needs better treats to want to learn :) )

bcruze · Nov 26, 2021, 10:30 AM

@jimp said in Upgrade to ZFS:

If you are on CE, wait for 2.5.2 to reinstall with ZFS. It has a much better starting dataset layout.

If you are on Plus, 21.05 is good for amd64 devices (e.g. SG-5100, XG-7100, etc), but ARM devices don't yet support ZFS. We're working on bringing support for ZFS to the 64-bit ARM devices (SG-1100, SG-2100), though. If you have one of those, stay tuned.

Good Morning,
had to reinstall Pfsense due to an issue with Pfblocker, and made the excuse to get up this morning to do it.. Plus after reading another thread. move to ZFS file system on my SG3100. after reading this I know why I couldn't
since the 3100 is EOL will this never happen I am guessing?

bingo600 · Nov 26, 2021, 12:36 PM

@johnpoz said in Upgrade to ZFS:

So should prob do a clean install then.. Since would assume the pfsense name will be what is used going forward. I do recall seeing something about recovery/restore related to the zfs name..

edit: Prob just wait til 21.09 comes out and do clean install then.. From reading the redmine about the recovery.

JP did you get a "final answer" from JimP ?

I have a system at the Office , that will soon go out to the customer.

It's a 2.4.x ZFS system upgraded to 2.5.2 with the old ZFS layout

I have another "remote system" that i recently ugraded to 2.5.2 , but with reinstall & ZFS format , using the new layout.

My question ... I think i know the answer but ...

Should i reinstall 2.5.2 on the system with the old ZFS layout, so it has the new layout ?
It's destined for Australia .. Damm long way if Sh.. hits the F..

I think i already spotted a few PR's where there was a mixup (or missing tests) for zroot vs pfsense

If "my gut feeling" ==> Yes do the reinstall is true.
Poor sod's that will have to talk a remote IT-person through that.

I'm glad i can reroute Bios output to serial (VT100) on the Qotom's , and can force USB stick boot via "Bios serial" on a TeamViewer screen. And that i sent out FTDi USB-Serial adapters & Null-Modem cables with the units.

/Bingo