HOW TO: 2.4.0 ZFS Install, RAM Disk, Hot Spare, Snapshot, Resilver Root Drive
-
I don't recommend USB Flash Drives on ZFS over SSDs unless you trying to save money and don't already have an SSD. I might recommend them over an HDD because they are silent and use less power, but the advantages over SSD are only price.
Using flash drives complicates things, so if you have an SSD definitely use that, and yes there are advantages of ZFS over UFS in a single drive configuration. In fact, single drive would be the recommended configuration for almost all use cases unless you are using USB flash drives.
ZFS does use more RAM than UFS but it's not a huge amount in a firewall implementation. 8GB is way more than enough as far as ZFS is concerned.
-
Thanks a lot - that was exactly the information I was looking for.
-
I'm assuming the answer is yes, but would the common ZFS suggestion to ensure you have ECC ram apply to PFSense as well? The FreeNAS folks definitely like to point out the possibility of entirely destroying an entire pool silently with a stuck bit in RAM.
-
No, non-ECC will be just fine. The whole FreeNAS ECC imperative is a pretty questionable argument at best. I'm pretty sure somewhere out there on the internet the developers of ZFS said in so many words that the ZFS needs ECC thing was silly.
You won't get a stuck bit that destroys your system. But for the sake of argument, even if you do, and don't have any snapshots then you just have to reinstall and restore from config on pfSense which should take about five minutes.
If you do keep snapshots regularly then you import the snapshot and mount it.Now if it's an installation for a customer that needs high availability in a production environment then you probably should use ECC. If for no other reason than to give the customer peace of mind.
In short, if you didn't already have a reason to use ECC, then ZFS on pfSense shouldn't change your mind. But if you want to be convinced otherwise just ask the same question on the FreeNAS forums and I'm sure you'll be flamed for acknowledging that such a thing as non-ECC exists.
-
If you install to a single disk, you can make zfs write two copies of everything to your drive. On flash this is probably a bad idea. The benefit is that if one copy of something you need gets corrupted, it's unlikely that the other will also
be corrupted so ZFS will likely recover from this corruption seamlessly.zfs set copies=2 yourpoolnameThanks for your earlier advice, I now have a nicely working Qotom i5 running 2.4 Beta installed on a 64GB SSD. So for an SSD would you recommend to enable this "two copies" setting? Is there any disadvantage except storage space (of which I have way more than needed)? If I do enable that should I then enable autoreplace, or is that only for if you have a 2nd drive?
You can see your zpool settings & stats with:
zpool get all yourpoolnameAre there any other settings I should change in my setup? Below is the result of a zpool getall command:
NAME PROPERTY VALUE SOURCE zroot size 57.5G - zroot capacity 1% - zroot altroot - default zroot health ONLINE - zroot guid xxxxxxxxxxxxxxxxxxx default zroot version - default zroot bootfs zroot/ROOT/default local zroot delegation on default zroot autoreplace off default zroot cachefile - default zroot failmode wait default zroot listsnapshots off default zroot autoexpand off default zroot dedupditto 0 default zroot dedupratio 1.00x - zroot free 56.6G - zroot allocated 964M - zroot readonly off - zroot comment - default zroot expandsize - - zroot freeing 0 default zroot fragmentation 5% - zroot leaked 0 default zroot feature@async_destroy enabled local zroot feature@empty_bpobj active local zroot feature@lz4_compress active local zroot feature@multi_vdev_crash_dump enabled local zroot feature@spacemap_histogram active local zroot feature@enabled_txg active local zroot feature@hole_birth active local zroot feature@extensible_dataset enabled local zroot feature@embedded_data active local zroot feature@bookmarks enabled local zroot feature@filesystem_limits enabled local zroot feature@large_blocks enabled local zroot feature@sha512 enabled local zroot feature@skein enabled local -
I would set it to 2 personally.
It isn't going to save you from everything, but it's certainly better than nothing.
Check out this article, it's far from a controlled test but I think it does a good job of showing us what multiple copies can and can't do for us.
http://www.openoid.net/testing-the-resiliency-of-zfs-set-copiesn/There is a performance impact on disk writes (you have to write everything twice). But, in pfSense an SSD is so fast that even writing twice (or three times) I don't think you will notice the difference. I also think that for a pfSense application your SSD will outlive the system even with you writing double (or even triple) copies to disk.
FWIW, setting copies=x only affects future files, not what has already been written.
Since pfSense is so easy and quick to reinstall and restore config.xml, ultimately what we are trying to achieve with copies=x is to avoid the annoyance of having to troubleshoot, reinstall, or have downtime because of a few corrupted files.
From what I've read, multiple copies offers some chance of avoiding those unpleasant situations, but is by no means a guarantee. In my mind, that's valuable enough since I believe the performance & durability costs of using it are likely negligible in pfSense. -
As far as I know multiple copies tries to spread the storage space of the copies around the medium used which is nice for spinning disks because bad blocks when they appear tend to cluster around one spot. On SSDs this is not guaranteed at all though.
-
Gents' I'm a total pfsense newbie (uses linux) , and I'm waiting for my new Qotom Q355G4 i5 box to arrive.
It will come w. 8G Ram & 64G mSata , but i'm going to install a Toshiba 240G SSD Sata disk.
Maybe i'll remove the 64G mSata , unless someone advices me to keep both disks in there.I'd like to install the 2.4.? on it straight away, and use ZFS.
If just keep the 240G SSD in there, do you have any hints for a "single disk ZFS" install.
Would there be any advantage of keeping the 64G mSata in there , besides complicating the install for a newbie.
Is the "write 2 copies" adviceable for a SSD (wear) ?Do i (ZFS) still need TRIM to be enabled ?
/Bingo
-
The guide's very good, and many people will want ZFS. I feel a lot safer with it on my NAS and data stores, and any business is likely to want it.
However it's worth noting that whether it's best for smaller and home systems is down to each person. For example, if you are happy to download or back up your config when it changes, and if a disk goes then just insert a new one and reinstall pfSense and the last config, and you're not worried about data corruption at rest (because there isn't much of it maybe, and you have backups), then ZFS adds little except a need for more hardware and an extra HDD/SSD, because a reinstall is about 15 - 40 minutes downtime while watching the TV.
After all, if data at rest that's actively used by the router for its own purposes (as opposed to files and directories it doesn't use itself) then most often it'll be caught anyway if it has a random bit flip or I/O error - the file won't make sense when read and it'll make this clear to the administrator.
If on the other hand you want to be sure that logs and RRD, tables of IPs, Squid caches, leases, or other extensive data stays 100% intact, and there isn't downtime, or your pfSense platform hosts other data and services too, then ZFS may well be very useful.
So I would add a note to any guide, of the pros and cons, because a router is a very different use case from other installations, if it isn't holding data whose integrity at rest isn't much of a concern.
-
Yeah ZFS is certainly not a must have. The majority of users would never notice a difference.
It doesn't add a requirement for more hardware though. You can install ZFS to a single disk, you just wouldn't get some of its features.
More RAM maybe - but if you don't already have enough RAM then simply do a UFS install.The major benefit for your average home user would be added protection against data corruption due to power outages in locales that are prone to them. There are quite a few threads about this on UFS.
The real solution to this is a UPS, but if you can't/don't want to afford a UPS then simply installing to ZFS is a viable stopgap that will very likely (but not certainly) solve this problem.The other home user benefit would be saving money on hardware. If you are building a budget system you can save a notable amount of $ by installing to a pair of thumb drives instead of a HDD or SSD. Doing this on ZFS allows you to mirror the drives and gives you a bit of redundancy.
But again, I agree that ZFS is by no means a must have for home users. It is a very nice option to have though.
-
Is it possible to restore a config from a UFS-based system to a ZFS-based one?
I'd like to switch to ZFS once 2.4.0 is released, which I know will require a reinstall, but I've been having a hard time finding whether restoring my old config would cause issues or whether it would be better to do a manual config from scratch. Does anybody have any information on doing that?
-
As far as I know it should work and is supported, I'd be very surprised if it didn't work because the only differences are in the storage method.
-
Is it possible to restore a config from a UFS-based system to a ZFS-based one?
I'd like to switch to ZFS once 2.4.0 is released, which I know will require a reinstall, but I've been having a hard time finding whether restoring my old config would cause issues or whether it would be better to do a manual config from scratch. Does anybody have any information on doing that?
To answer your question in the words of the almighty OP ;)-
EDIT: I don't recommend setting a second zpool as it can cause issues with booting. If you want to send snapshots on a separate device, try a UFS filesystem on it. People smarter than myself can probably get around this, if anyone has a solution please share and I'll add it here!
To use UFS:
After partitioning the drive follow the instructions here:
https://www.freebsd.org/doc/handbook/disks-adding.htmlTo send your snapshot to a UFS partition you can modify this for your mount point and copy and paste:
Code:```zfs snapshot -r yourpoolname@
date "+%d.%b.%y.%H00"&& zfs send -Rv yourpoolname@date "+%d.%b.%y.%H00"| gzip > /mnt/sshot/sshotdate "+%d.%b.%y.%H00."gz && zfs destroy -r yourpoolname@date "+%d.%b.%y.%H00"&& zfs list -r -t snapshot -o name,creation && du -hs /mnt/sshot/sshotdate "+%d.%b.%y.%H00."gzI would imagine that if you could restore a snapshot from UFS to ZFS then you could restore from the config. Config file is just an .xml file full of your system configuration settings. The underlying FS shouldn't matter.
-
If you are smarter than me I'm betting you could automate this with a script, I would think something running frequently in cron along the lines of:
check if pool is degraded if no, exit if yes, check if resilver complete if no, exit if yes, detach baddiskIf anyone does write such a script, please share! ;)
Added to feature requests, see https://redmine.pfsense.org/issues/7812
-
First of all GREAT post. Thanks pfBasic.
I've been using a 6 disk ZFS raidz2 array on my FreeNAS server for a couple of years.
I just wanted to point out, that ZFS can do more than a two disk mirror. It is technically nearly unlimited. But for pfSense I think have a ZFS three disk mirror is another option, and less setup, less disks, and still offers 2 drive failure protection.
Just wanted to throw that out there for home users looking for ZFS with only 3 disks and dual failure redundancy.
-
Appreciate this post.
I'm using 2.4RC and have a mirrored boot drive setup with ZFS.
I was wanting to partion a new SSD (ada1) with ZFS for general file system use, specifically mounting the disk in /var/squid/cache. What are the steps for partitioning the disk with ZFS so that it can be mounted into the existing file system structure?
-
I probably should have researched a bit more before asking, but man I love ZFS. Here is how I setup my new drive.
gpart create -s gpt ada1 gpart add -b 2048 -t freebsd-zfs -l gpt2 ada1 zpool create -f zdata /dev/gpt/gpt2 zfs set checksum=on zdata zfs set compression=lz4 zdata zfs set atime=off zdata zfs set recordsize=64K zdata zfs set primarycache=metadata zdata zfs set secondarycache=none zdata zfs set logbias=latency zdata zfs create -o mountpoint=/var/squid/cache zdata/cache chown -R squid:squid /var/squid/cache chmod -R 0750 /var/squid/cacheThere are specific ARC and ZIL caching features which I didn't setup which could be a benefit for squid, but as best I can tell, it wouldn't work out well in my situation. Here is a link from squid regarding ZFS:
https://wiki.squid-cache.org/SquidFaq/InstallingSquid#Is_it_okay_to_use_ZFS_on_Squid.3F -
I'm using a PC Engines APU2C4 for my pfsense box. I just upgraded to 2.4 and read about ZFS. I'm using a 16GB single SSD and I'm wanting to use ZFS. Which of the steps in the OP should I follow? I read through them and they're targetted for multiple flash drives in the system. I'm not really sure which ones are applicable in a single disk setup only.
Also, can I backup the config file that I have now, reinstall pfsense with ZFS, and just restore that same config file without any adverse effects?
-
In short, if you didn't already have a reason to use ECC, then ZFS on pfSense shouldn't change your mind. But if you want to be convinced otherwise just ask the same question on the FreeNAS forums and I'm sure you'll be flamed for acknowledging that such a thing as non-ECC exists.
The point of ECC RAM on a ZFS based fileserver is simple. ZFS provides checksumming of all files at rest (i.e. on disk) and ECC provides the same protections for data in motion. It isn't that a pool could be lost without ECC, it's actually much more sinister. Data that seems fine, data with valid checksums that passes every scrub, could have "bit rot" and, in extreme cases, be unreadable. Everything looks fine, but nothing is!
pfSense is in a different boat. A firewall absolutely shouldn't be storing any critical or irreplaceable data so 100% corruption prevention isn't necessary. 99% (or whatever the chances of bit rot in the relatively tiny memory footprint of a firewall) corruption prevention is more than sufficient and ECC isn't at all necessary (it is nice to have).
TL;DR: Just go download config.xml, enable copies=2, and setup '/sbin/zpool scrub zroot' to run periodically via cron
-
Anybody can hrmelp me with my question?
