What's the best approach to replace both disks of the ZFS mirror on which pfSense is installed

Wholelottapfsense

As I mentioned here, I have pfSense installed on 2 Supermicro SATA DOM SSDs (SSD-DM032-SMCMVN1) set up as a zfs mirror.

The SSDs are about 4 years old and their SMART status is showing as failed.

I bought 2 new SSDs (same model, same size) and, as I see it, I have 2 options:

replacing them all at once, doing a new install and putting back my backed up config
replacing them one by one and resilver each time the new disk

Which of these 2 options would be the best way to go forward?

By the way, the zpool and gpart info is as follows:

[2.7.2-RELEASE][admin@pfSense.ad.somedomain.com]/root: zpool status
  pool: zroot
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
config:

        NAME        STATE     READ WRITE CKSUM
        zroot       ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada0p3  ONLINE       0     0     0
            ada1p3  ONLINE       0     0     0

errors: No known data errors
[2.7.2-RELEASE][admin@pfSense.ad.somedomain.com]/root: gpart show ada0
=>      40  61865904  ada0  GPT  (30G)
        40      1024     1  freebsd-boot  (512K)
      1064       984        - free -  (492K)
      2048   4194304     2  freebsd-swap  (2.0G)
   4196352  57667584     3  freebsd-zfs  (27G)
  61863936      2008        - free -  (1.0M)

[2.7.2-RELEASE][admin@pfSense.ad.somedomain.com]/root: gpart show ada1
=>      40  61865904  ada1  GPT  (30G)
        40      1024     1  freebsd-boot  (512K)
      1064       984        - free -  (492K)
      2048   4194304     2  freebsd-swap  (2.0G)
   4196352  57667584     3  freebsd-zfs  (27G)
  61863936      2008        - free -  (1.0M)

SteveITS

@Wholelottapfsense reinstalling and restoring is going to take like 10-15 minutes, I would do that.

Seems odd 4yo drives are failing though. Is your router doing a high amount of logging or other disk writes?
(Ref https://www.netgate.com/supported-pfsense-plus-packages )

provels

I just replaced the 2x3TB ZFS mirror in my FreeBSD NAS with 2x10TB drives.
I did a resilver for both drives then expanded the array. Whereas mine took a couple days, yours will probably take minutes.

Old > New
New < New
Expand

(PS - DON"T upgrade the zpool features on pfSense. I did it once and toasted the install.)

Wholelottapfsense

@SteveITS Indeed, I also find it strange. I didn't change anything in the default logging settings.

The packages I have installed are:

acme
avahi
freeradius3
haproxy
iperf
ntopng
openvpn-client-export
pfBlockerNG-devel

SteveITS

@Wholelottapfsense said in What's the best approach to replace both disks of the ZFS mirror on which pfSense is installed:

ntopng

https://www.netgate.com/supported-pfsense-plus-packages

stephenw10

Yup, ntopng can log a lot.

And, yes, I would also just replace both drives and re-install.

Wholelottapfsense

@SteveITS @stephenw10 Thanks a lot for your advice. I will do a re-install and disable the ntopng package which I anyway didn't use much.

Wholelottapfsense

I reinstalled pfSense and everything went well except that none of the packages are being reinstalled

I followed this Netgate tutorial, choosing the console option with a flash drive containing \config\config.xml

My settings were applied correctly and all the interfaces are correctly configured, but I have a bunch of messages (relating to pfBlockerNG) because none of the packages are installed.

I did not tick the option 'Do not backup package information.' when backing up my config. Besides, I checked the config.xml file with Notepad++ and the package related info is included.

What could have gone wrong?

[EDIT]: I now tried restoring the backed up config via the GUI and by the time I logged in after the reboot, all the packages were there

SteveITS

@Wholelottapfsense it should reinstall all the packages after booting. Check the logs?

You can also install them manually. There’s an option I think on the backup page to reinstall all packages, that might work depending on what state it’s in.

Wholelottapfsense

@SteveITS Restoring the backed up config via the GUI did the trick, all the packages are there now

Instead of uninstalling the ntopng package, I disabled it in the settings. So with that, I'm crossing my fingers that my SSDs will last longer this time :-o