Upgrading - following the pfSense docs 'Installing and Upgrading' includes having a fall back plan

Gertjan

@lohphat said in Upgrading - following the pfSense docs 'Installing and Upgrading' includes having a fall back plan:

Get a USB console connection setup before the upgrade.

Yep.
As the 3100 manual states : There are times when directly accessing the console is required.

fabrizior

@teamits No, I do not have a memstick installer archived for any releasec and I’m running 3rd-party hardware.Never occurred to me that downloads of prior versions of opensource software would be so viciously restricted.

fabrizior

Can anybody help?

DaddyGo

@fabrizior said in Upgrading - following the pfSense docs 'Installing and Upgrading' includes having a fall back plan:

Never occurred to me that downloads of prior versions of opensource software would be so viciously restricted.

Hi,

because it is meaningless...
think about it, why not,.... hmmm "say", it is not possible to get a f.e. 1.1 image today...(?!)

because it is obsolete, in the world of informatics, every minute counts

I have written several times here, pls. use Win95 and pls. be satisfied with its security features...

It's a safety tool in your hand, keep it up to date (!)

fabrizior

@daddygo That’s ridiculous. Your attempt at exaggerated humor is snide and petty.

Netgate docs say have a restore plan.
Netgate immediately removes access to the prior release download upon a new release.

Keeping a copy of the installer for the release you’re running really means: every time we do an in-place online upgrade, we must also manually go download and archive the corresponding memstick/iso installer archive.

Otherwise, you’re backup plan (or lack thereof) is screwed.

Show me any other company who removes access to the prior release downloads immediately upon launch of the new release?

Show me any other “opensource“ repo that restricts its binary packages this way?

Netgate: At least leave the prior release available for 90 days... for roll-back purposes.

One-way upgrades without continuing to providing the tools to support roll-back is a smack in the face to your users.

I realize this will change nothing other than more manual labor on my ever-evolving operations SOP, though: thanks for the opportunity to rant.

DaddyGo

@fabrizior said in Upgrading - following the pfSense docs 'Installing and Upgrading' includes having a fall back plan:

That’s ridiculous. Your attempt at exaggerated humor is snide and petty.

Just for you (ONLY!)

I say this is consistent, even if you don't agree with that...
I hope it stays that way, to avoid stupidity ....

fabrizior

@daddygo
To each their own...

Just to clarify:
Are you equating the ability to roll-back an upgrade to the last functional release & configuration to stupidity?

Or perhaps rather the approach of say a user setting up a new ofSense deployment using 2.4.5p1 instead of immediately jumping on 2.5 while so many people are having functional and configuration problems with the new release?

Or just calling those of us who didn’t keep their old memstick installer stupid?

DaddyGo

@fabrizior said in Upgrading - following the pfSense docs 'Installing and Upgrading' includes having a fall back plan:

Just to clarify:

hmmmm and hmmmm

Do you want an old installer?
I'll upload it to a DropBox account

fabrizior

@daddygo appreciate the offer, thank you. Others have already done so.

guardian

How difficult is it to make a disk image clone? (Likely drop to single user mode to keep the system from changing too much), That way all plugins can be accommodated.

Tzvia

As much as I would like to see the previous version ISO available for say a month, along side the new version, that is not a backout plan. That would be for those who didn't have a backout plan, if they find they need it after upgrading. Therefore, it should not be part of a backout plan. When there is a new release, even though I run the 'upgrade', the first thing I do is grab the ISO and make a memstick. I then upload the ISO to my NAS just in case. It's there for when the next version comes out, just as the previous version was saved. How hard it that... We all knew that 2.50 was due 'any day now', how hard would it have been to grab 2.45p1 then 'just in case' if you hadn't when it was released.

Total cost of this backout plan? A few minutes time + one memory stick. Have a router with removable drive or provision for second drive? How about adding second drive, installing PFSense on it and importing your backup. Run on it for long enough to verify it works the same as the original. When the update comes out, update it, leaving the original alone. I did this, and I did need to back out. Cake, shut it off, swap the cables back, boot back to 2.45p1. I needed that as I have to be able to work from home and can't have downtime over some bug in the new code. The next weekend, I swapped the cables back and worked through the problems, knowing that I could go back in a couple of minutes so that I could be back 'at work' come Monday.

Maybe I over thought it. But the internet has gone from a toy of tinkerers to an appliance like a refrigerator. And these days, downtime may be lost income as so many are working from home as I am. Downtime is a big deal, but it only takes a few minutes to cover your ass.

fabrizior

@tzvia That's solid advise, and a reasonable plan. The point here is that the docs could do a better job of warning that saving and personally archiving the ISO/memstick image is a necessary step at each upgrade and that their is no availability to download on-demand once a new release has become available. (Though that seems too much to ask...)

Perhaps including a simple warning note and a link to download the current installed and new images on the System Update page in the UI would help take care of some of this "archive or else" situation.

I have the memstick I did my install with back in not-so-recent history. The prior upgrades have been done online and in-place. As such, I never downloaded the memstick installer for 4.5x - which I'd need for a 4.5 upgrade back-out plan.

Is that my failure? yes.

Though Netgate is the only company that seems to have this policy that I'm aware of... accounting for either close or open-source distros.

Is it an obsurd inconvenience that I cannot rectify that directly with the ability to download the last-most-recent production release version prior to the newly-released [current] update? I say yes, and most seem to agree...

This sort of management behavior, along with the BS recently regarding WireGuard... really says that NetGate isn't treating pfSense or it's pfSense customers/potential-customers, or it's back-end supporting development community in a serious or professional manner.

One man's professional opinion...

A Former User

@fabrizior Agreed. It's an odd policy at best that on day one the "older" versions become inaccessible.

You can open a ticket with them and they will get you a 2.4.5_p1 image. No support contract required. That's a pain for both users and Netgate.

As to the other drama I'll not say anything more than I've said in other threads. I, speaking only for myself, will move on to another product when the time is convenient. I have exactly zero interest in upgrading to 2.5. I'm done...

guardian

What about all the plugins. IIUC, even if you have kept the old installer (which I have done), the installer wants to reinstall the plugins from the net. Do the old repos disappear?

It won't work for sealed appliances, but for anyone using a generic PC that can boot from a USB drive, what about a bootable Clonezilla or Foxclone disk. They are linux based, but does that matter if you image the whole disk? I am in the process of building a bootable USB hard drive which I hope will be able to store the store the image.

An ideal situation would be to have something FreeBSD based. For some reason this solution seemed to be met with hostility, but I'm not sure why.

My proposed plan:

Save the config file from the running install
Have a USB of the installer for the running version available
Download a copy of the new version and make a USB
Take pfSense offline, and boot from a USB image program. Make an image/verify it.
Restart the system and once it is up again, run the built in install
----- If successful done:
----- Else gather info for a bug report? Suggestions on this?
Run the installer made in #3 and reload the saved config.
----- If successful done:
----- Else gather info for a bug report? Suggestions on this?
Use the USB from Step #2 and reload the saved config. I'm not sure if this is an option if you have plugings - Comments?
----- If successful done:
----- Else gather info for a bug report? Suggestions on this?
Boot into USB recovery disk and restore the disk image. Reboot pfSense and you should be back to where you started.

Another possible solution would be to have a script that runs from single user mode that does a file level backup of the directories which contain all the plugins/config files. I don't know enough about how hard/easy this would be, and most importantly how reliable (would ongoing maintenance be required?).

I've been using pfSense for about 5-6 years, but the last time, I had to go to step #6, and fortunately it worked so I didn't have to worry about what to do next. It did make me think I need to be much better prepared.

I'd love to hear what the gurus here think/how they have solved this problem.

Tzvia

@guardian Good questions. I would first verify that when I do a backup of settings, that UPDATE/SYSTEM UPDATE is set to PREVIOUS STABLE. You can then also verify that the software versions of anything specific to 2.45p1 are still available in PACKAGE MANAGER. Provided they are listed, applying your backup to a fresh install of 2..45p1 should set it this way and it should be able to install all the packages when it reboots...

fabrizior

@guardian
Have any recommendations on a usb-bootable tool for step 4, preferably to make a bootable backup copy. not well-versed in FreeBSD SysAdmin.

guardian

@fabrizior said in Upgrading - following the pfSense docs 'Installing and Upgrading' includes having a fall back plan:

@guardian
Have any recommendations on a usb-bootable tool for step 4, preferably to make a bootable backup copy. not well-versed in FreeBSD SysAdmin.

I was hoping someone who is BSD experienced would comment.

IIUC as long as I am going to image the whole disk and the backup and restore are done from another running system (i.e. boot from a USB disk or second drive) almost any imaging method will do All we are doing is blindly copying bytes. If it is necessary to restore, then the whole image must be restored to the same disk, or one with identical geometry.

My plan was to use Foxclone. I installed a copy of Linux Mint and Foxclone on an old 320GB USB hard drive. I have about 300GB free to copy drive images. That should be more than enough. I was thinking that I could even do something as simple as:
Backup:

dd if=/dev/sdx bs=4096 of=backupdir/pfsense.img status=progress

Restore:

dd if=backupdir/pfsense.img bs=4096 of=/dev/sdx status=progress

Can someone confirm if my understanding is correct or not and/or a better tool to use.

fabrizior

@guardian
How’s this sound as a rough starting point?
I don’t have a dev system to try this on right now... VM infra is busy with other projects ATM.

For the purposes of switch-over / fail-back with upgrades, and to maintain RTO/RPO SLA objectives against drive faults (not catastrophic system-wide HW/electrical faults) I’m considering:

Add a second internal drive (same size)
Could try this with an approp-sized removable flash drive for off-line / off-site backup safety vs ease of switchover/fail-back, depending on SLAs, I suppose.
partitioning new drive appropriately and make it bootable (values may vary, only an pasted example)
Q: can anyone help validate details with a pfsense env?

gpart destroy -F adaX
gpart create -s GPT adaX
gpart add -b 40 -s 128 -t freebsd-boot adaX
gpart add -s 1880 -t efi adaX
gpart add -s 4G -t freebsd-swap -l SWAP adaX
gpart add -t freebsd-ufs -l RECOVER adaX

gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 adaX
gpart bootcode -p /boot/boot1.efifat -i 2 adaX
gpart set -a bootme -i 4 adaX

newfs -ntEU /dev/adaXp4
tunefs -a enable /dev/adaXp4

periodically cloning the current boot drive to the other. (with additional script logic to set source/dest parameters and manage the mount point, etc...)

use sysutils/clone to make a perfect clone of your system’s startup disk as follows:

mount -o noatime /dev/gpt/RECOVER /mnt
clone -c rwoff / /mnt

After the file system cloning has finished, you only would need to edit /mnt/etc/fstab:

# Device            Mountpoint      FStype  Options Dump    Pass#
/dev/gpt/RECOVER    /               ufs     rw      1       1
/dev/gpt/SWAP       none            swap    sw      0       0
proc                /proc           procfs  rw      0       0

clone(1) can be used to keep file systems synchronized, and you would use the very clone command together with the -s flag for keeping your recovery disk updated. In this mode only changed files would be copied and synchronization needs much less time than the initial full cloning process.

Alternatives? Is there a better way, still w/o 3rd-party software? dd doesn’t have an incremental sync mode like clone(1) does... while block-leve copies might be technically faster, dd is an all-or-nothing operation. clone seems like it would have some advantages after the first time.

Any other (perhaps pfSense-specific) partitioning/boot details to account for?

credit:
found the idea and example code from an 6-Aug-20 reply by @obsigna on the freebsd forum.

Patch

Given Negate policy to remove access to old binaries as soon as a new versions is released the only safe strategy I can see is for all users to individually maintain their own archive of any version they use. So

When wanting to upgrade pfSense by any method, first download and archive an iso in case you need it at a later date (note you can not reliably do this later as Netgate will pull the binary before you try to upgrade the next time).
Upgrade pfSense by whatever method you prefer
If the upgrade fails for you, use the archived version (the one you archived during a prior upgrade

Not efficient for a community but at least it ensures a reliable upgrade fall back from a known good source.

guardian

@patch said in Upgrading - following the pfSense docs 'Installing and Upgrading' includes having a fall back plan:

Given Negate policy to remove access to old binaries as soon as a new versions is released the only safe strategy I can see is for all users to individually maintain their own archive of any version they use. So

When wanting to upgrade pfSense by any method, first download and archive an iso in case you need it at a later date (note you can not reliably do this later as Netgate will pull the binary before you try to upgrade the next time).

Upgrade pfSense by whatever method you prefer

If the upgrade fails for you, use the archived version (the one you archived during a prior upgrade

Not efficient for a community but at least it ensures a reliable upgrade fall back from a known good source.

Good for the base version, but what about the plug-ins etc.?