Upgrading - following the pfSense docs 'Installing and Upgrading' includes having a fall back plan

guardian

@patch said in Upgrading - following the pfSense docs 'Installing and Upgrading' includes having a fall back plan:

Given Negate policy to remove access to old binaries as soon as a new versions is released the only safe strategy I can see is for all users to individually maintain their own archive of any version they use. So

When wanting to upgrade pfSense by any method, first download and archive an iso in case you need it at a later date (note you can not reliably do this later as Netgate will pull the binary before you try to upgrade the next time).

Upgrade pfSense by whatever method you prefer

If the upgrade fails for you, use the archived version (the one you archived during a prior upgrade

Not efficient for a community but at least it ensures a reliable upgrade fall back from a known good source.

Good for the base version, but what about the plug-ins etc.?

fabrizior

@patch Any thoughts on an appropriate process to make a bootable clone of the boot disk?

Thought being: if an upgrade fails, change the boot device to the backup disk and reboot to the clone... back online almost immediately. then clone the booted backup back over the failed upgrade image to restore DR capability before attempting upgrade again at next maintenance window...

Anyone have practical experience with the partitioning/cloning a pfSense drive?

Would this need to be done offline?
If attempted online, any pid/lock/runtime state files that need to be excluded or reset on the clone?

Patch

First line upgrade strategy is for the upgrade to work as expected.

A fall back plan for me needs to be much more reliable than the upgrade path, hence the desire to return to a reference state (original install media & backed up configuration file).

Dynamic snapshots are useful if you have been using them regularly so have validated their functionality. Relying on them for a once off fall back plan is too risky for me, or if validated, too time consuming for something I intend not to use.

Dynamic snapshots do make far more sense if you run pfSense on a virtual machine (but then you are in the "using them regularly" group).

fabrizior

@patch Thanks. That standard process is probably a good fit for many. [edit] Though, one might think the most “reliable” fallback plan would be a direct clone of the (presumably stable) system immediately prior to upgrade. Having to reinstall and reapply a config reliant on specific prior versions of packages that may or may not be available...

From my point of view, we could come up with a quicker, less maintenance-intensive process from an RTO metric / SLA perspective (when relevant) without resorting to clustered hardware.

The recovery time in case of upgrade failure when the process is to re-install the base image, restore the config, validate all the packages (hopefully at same version as pre-upgrade), etc... isn’t the quickest process and somewhat manually labor intensive (in comparison.)

Executing a scripted incremental update of a cloned disk (especially if done regularly, and if could be done online (?)) just-in-time prior to the upgrade maintenance window would allow for a faulted upgrade recovery time no longer than what it takes to reboot.
Much reduced recovery effort seems valuable enough to continue investigating how this could be done.

Only downsides seems to be the cloning process discovery/analysis and the cost of a second disk.

itpp21

Reading this with interest, I'm looking at an usb stick recover method and the possibilities of adding packages from a running environment to this stick. As recovery(installation) by stick is already supported we might expand this by incorporating packages.
One question is will the usb stick auto install or do you still need console access to get it running? if so can you force auto install so it becomes a matter of insert stick and boot?
(yes you can yell at me) but under windows you can extract the wim file and modify it to your liking, repackage the wim and at boottime push out additional packages without a key pressed to add on a fresh installation.

Gertjan

@itpp21 said in Upgrading - following the pfSense docs 'Installing and Upgrading' includes having a fall back plan:

(yes you can yell at me)

Noop.
Just 'gime' 1 % of 1 % of the Microsoft develop budget and you'll have that option ready for tomorrow.
Btw : with MS conditions of course : no documentation what so ever, and a real "lets test it in the field and see what telemetry brings home".

itpp21

I don't get the reply, if the stick allows pfSense installation (which it does) then why can't it be modified to auto install and why can't you add stuff? Its just an 'image' bits and bytes. I remember the ubuntu iso's which when you don't do anything it times out of some default options and then deploys.

Gertjan

You can modify the ISO before you use it.
What's stopping you ?

( ok, you need to know how, what to place where, how things work, have plenty of time to test etc )

itpp21

@gertjan said in Upgrading - following the pfSense docs 'Installing and Upgrading' includes having a fall back plan:

( ok, you need to know how, what to place where, how things work, have plenty of time to test etc )

Exactly, I've ordered some sticks for this, will get back when I get anywhere, maybe others find this idea interesting to assist in the how, where and what.

itpp21

The easy solution:
Create a pfSense VM with virtualbox
Load packages and settings from your bare metal device
Verify the config works as bare metal.
Shut the VM down

Restore:
Use vboxmanage to convert vdi to vhd
Use rufus to load the vhd on the ram device (taken out of the bare device mounted via usb)
Place storage back, done. (and tested)

I'm gonna test the reverse (take bare metal in to a VM) but have no doubts this will work.

nb. cloning the storage device as backup also works.