How do I roll back packages
-
Thanks @bmeeks, @johnpoz, @Gertjan - I can understand the difficulty on the multiple versions. The point I'd like to add to the discussion is that rule #1 with computers is BACKUP, BACKUP, BACKUP.
I've been raising the issue of fallback for several years. I was suggesting some sort of imaging, but @jimp and others kept telling me that just reloading config.xml would solve all problems
[https://forum.netgate.com/post/727596](link url) . What I am learning here is, this is not true.Admittedly pfSense has been very reliable, but not having a quick recovery fallback is troubling and seems to go against what are established industry best practice (BACKUP!).
At very least I would think a built in image/restore would at least make sure a bad update could be fixed in minutes. This would make frequent updates almost NO RISK, so they could be done without the necessity of a lenghty "insurance maintenace window" just in case something went wrong.
Thoughts?
-
@guardian said in How do I roll back packages:
Thanks @bmeeks, @johnpoz, @Gertjan - I can understand the difficulty on the multiple versions. The point I'd like to add to the discussion is that rule #1 with computers is BACKUP, BACKUP, BACKUP.
I've been raising the issue of fallback for several years. I was suggesting some sort of imaging, but @jimp and others kept telling me that just reloading config.xml would solve all problems
[https://forum.netgate.com/post/727596](link url) . What I am learning here is, this is not true.Admittedly pfSense has been very reliable, but not having a quick recovery fallback is troubling and seems to go against what are established industry best practice (BACKUP!).
At very least I would think a built in image/restore would at least make sure a bad update could be fixed in minutes. This would make frequent updates almost NO RISK, so they could be done without the necessity of a lenghty "insurance maintenace window" just in case something went wrong.
Thoughts?
Then you need to use a virtual machine with snapshots. Nothing quicker than that to restore a "gone bad" upgrade. pfSense runs fine on ESXi. A couple of the other hypervisors have given issues from time to time, but ESXi has been pretty stable as far as I know. And snapshots require nothing extra be installed on pfSense itself. I don't mean for this response to be flippant. Using a VM is an excellent way to have image backups and restores with minimal effort via snapshots. Just take a snapshot before any firewall change. If something goes wrong, revert to the previous snapshot.
What @jimp says is 100% true for a plain-vanilla pfSense installation. With packages installed, things do get more complicated. But users need to understand two vital things about packages. First, they are optional and not required by pfSense. And second, they are created and maintained by volunteer non-paid maintainers in almost every case. So when you opt to use an add-on package, there are inherent risks in that decision as contrasted with using a plain-vanilla pfSense install. I get that some packages offer very popular or often needed features, but they are not part of the base pfSense install and are not (for the most part) maintained by the pfSense developer team. So when you install packages, you assume some risk there.
I can understand the reluctance of the pfSense team to start adding "fluff" to a security system such as a firewall. An image backup ability would entail loading additional software into the pfSense core system and that will inevitably increase potential attack surfaces. Generally speaking, it's just never a good idea to put a lot of extra software on a firewall. And yes, that can apply to many of the available packages as well. There is certainly a valid argument to be made that they in some ways decrease the firewall's security posture by adding additional attack surfaces.
I managed Checkpoint firewalls for many years, and from what I remember they had no equivalent of an image backup/restore method either. They offered essentially the same ability as pfSense. You could back up and restore your configuration, but you first had to reinstall the base OS firewall package, and then import your backup configuration. So pretty much the same way you restore pfSense. Don't know about the fancy Palo Alto firewalls. My company was just switching over to them when I retired and I never got to play with one. The closest thing I ever came across to what you would like is the old Nokia IP-series security appliances with their IPSO operating system. It could maintain a sort of dual-boot environment where you could boot into a new "upgrade", but if that failed, you could reboot back into your previous environment. However, as best I can remember, that only applied to the IPSO operating system itself. The Checkpoint packages we ran on top of IPSO still had to be reinstalled and their configuration restored as I previously described.
-
@bmeeks said in How do I roll back packages:
Then you need to use a virtual machine with snapshots. Nothing quicker than that to restore a "gone bad" upgrade. pfSense runs fine on ESXi. A couple of the other hypervisors have given issues from time to time, but ESXi has been pretty stable as far as I know. And snapshots require nothing extra be installed on pfSense itself. I don't mean for this response to be flippant. Using a VM is an excellent way to have image backups and restores with minimal effort via snapshots. Just take a snapshot before any firewall change. If something goes wrong, revert to the previous snapshot.
Good idea, but my hardware doesn't have the excess resources. I am still running a UFS install, but I'm wondering if there is a ZFS install available, and if anything has been done to support snapshot/rollback.
What @jimp says is 100% true for a plain-vanilla pfSense installation. With packages installed, things do get more complicated. But users need to understand two vital things about packages. First, they are optional and not required by pfSense. And second, they are created and maintained by volunteer non-paid maintainers in almost every case. So when you opt to use an add-on package, there are inherent risks in that decision as contrasted with using a plain-vanilla pfSense install. I get that some packages offer very popular or often needed features, but they are not part of the base pfSense install and are not (for the most part) maintained by the pfSense developer team. So when you install packages, you assume some risk there.
I can understand the reluctance of the pfSense team to start adding "fluff" to a security system such as a firewall. An image backup ability would entail loading additional software into the pfSense core system and that will inevitably increase potential attack surfaces. Generally speaking, it's just never a good idea to put a lot of extra software on a firewall. And yes, that can apply to many of the available packages as well. There is certainly a valid argument to be made that they in some ways decrease the firewall's security posture by adding additional attack surfaces.
What would it take to add something to the install media? USB drives are getting quite big... boot into the install media, select an option and a script runs and copies system and configuration data to the USB.
I managed Checkpoint firewalls for many years, and from what I remember they had no equivalent of an image backup/restore method either. They offered essentially the same ability as pfSense. You could back up and restore your configuration, but you first had to reinstall the base OS firewall package, and then import your backup configuration. So pretty much the same way you restore pfSense. Don't know about the fancy Palo Alto firewalls. My company was just switching over to them when I retired and I never got to play with one. The closest thing I ever came across to what you would like is the old Nokia IP-series security appliances with their IPSO operating system. It could maintain a sort of dual-boot environment where you could boot into a new "upgrade", but if that failed, you could reboot back into your previous environment. However, as best I can remember, that only applied to the IPSO operating system itself. The Checkpoint packages we ran on top of IPSO still had to be reinstalled and their configuration restored as I previously described.
That is the strenght/weakness of proprietary solutions. On the + side, since the manufacturer has 100% control over hardware/software they can (if they are a quality provider) do very through testing/QA so problems should be very very rare. On the - side, if the provider doesn't do things the way you want, there is nothing you can do about it.
-
@guardian said in How do I roll back packages:
Good idea, but my hardware doesn't have the excess resources. I am still running a UFS install, but I'm wondering if there is a ZFS install available, and if anything has been done to support snapshot/rollback.
There is a ZFS install option for pfSense now. Here is a link to the official install documentation. Scroll down the page a bit to see the file system options: https://docs.netgate.com/pfsense/en/latest/install/install-pfsense.html.
There is also an open Feature Request on the Redmine site for taking a snapshot on ZFS systems prior to performing an upgrade: https://redmine.pfsense.org/issues/10237.
In terms of your USB suggestion, that is already available. You can create install media on USB with a previous
config.xmlconfiguration file on the media so that during the installation pfSense will restore the configuration. However, this does not include the binary bits of any packages. It would be only the user-customized configuration parameters. As I mentioned in my first post, most packages have binary pieces and GUI configuration pieces. The binary pieces (shared libraries and executable machine code for the service itself) are pulled down from thepkgrespository. -
@johnpoz said in How do I roll back packages:
AFAIK, there is not an official supported way of doing this?
But in all my years of using pfsense, really since its been available.. I can not recall an issue with a package that would of warranted rollback to an previous version. Now I do not use all packages, so can not say its never happened on any package.
But been very active on the forums for 10 some years.. And don't really recall any threads where such a thing is an issue..
If your concerned... Prob just suggest you wait for a X amount of time after a new version of whatever package before updating it.
I can tell you running the latest version of haproxy-devel 0.60_9, and its working fine for my setup.
If your curious to what an update has changed, you can click the package number and it will take you to github and you can look to see exactly what has changed.
example for the latest haproxy-devel
https://github.com/pfsense/FreeBSD-ports/commits/devel/net/pfSense-pkg-haproxy-develNormally the updates are minor fixes for stuff with minimal code changes..
Well be amazed because right now the latest update of syslog-ng is broken. It does not like tls settings. The package before update was working fine, so I need to revert to old package.
-
See this
https://github.com/pfsense/FreeBSD-ports/commit/a5b1eda67c40592e14806a4a4bbdd946f0461045#commentsHas a patch to fix the tls problem with it.
-
@johnpoz said in How do I roll back packages:
https://github.com/pfsense/FreeBSD-ports/commit/a5b1eda67c40592e14806a4a4bbdd946f0461045#comments
It says that the patch can't apply clean and won't show me the apply button
-
@propercactus said in How do I roll back packages:
@johnpoz said in How do I roll back packages:
https://github.com/pfsense/FreeBSD-ports/commit/a5b1eda67c40592e14806a4a4bbdd946f0461045#comments
It says that the patch can't apply clean and won't show me the apply button
That posted patch link is for someone who has their own GitHub clone of the package repo and builds the package themselves. Nobody much does that -- really only package developers.
You can still manually apply the changes by making the edits to
/usr/local/pkg/syslog-ng.incshown in thedifffile. That means deleting the text in red and adding the text in green at the locations shown. Make a copy of the original file before attempting this, so you can quickly recover. You can use either the built-in option in the GUI under DIAGNOSTICS > EDIT FILE, or obtain a shell prompt on the firewall and usevi.The above instructions are given in the event you are in a rush for the fix. If not, it may be better to wait for the package to get updated in the Packages repo, and then update from the Package Manager in the GUI.
-
I think this is a really strong case for needing package roll back.
The offending commit that broke TLS is 16 days old, so even if I had of waited 14 days before upgrade i still would have upgraded and broken.
It is fortunate that the commit author very promptly supplied a patch once alerted. But if they did not, what can we do? We are stuck with a broken package!
-
@propercactus said in How do I roll back packages:
I think this is a really strong case for needing package roll back.
The offending commit that broke TLS is 16 days old, so even if I had of waited 14 days before upgrade i still would have upgraded and broken.
It is fortunate that the commit author very promptly supplied a patch once alerted. But if they did not, what can we do? We are stuck with a broken package!
@propercactus thanks for chiming in.... I need SyslogNG, and if I had updated I'd be f%ck'd. It looks like the patch doesn't work... at least it's not straight forward.
How hard would it be to include a scripts in the install disk to make it easy to save the system/packages? If things go south, at least you can roll everything back.
If the original install is available, then there should be a way to just backup key directories with the packages.
Once I can get through this upgrade, I think I'll try converting to a ZFS install, hopefully a ZFS snapshot will solve these problems - or is that just wishful thinking?
-
@guardian said in How do I roll back packages:
It looks like the patch doesn't work...
The patch is working, I was initially confusing myself and pulling the offending commit as the patch in stead of the actual patch.
But yah for the first time in a while I felt completely hopeless, my reporting from the device to SOC was down and what could I do? Not a good position to be in at all.
@guardian said in How do I roll back packages:
hopefully a ZFS snapshot will solve these problems - or is that just wishful thinking?
Good question I'll have to have a look at that myself as it's not something I know anything about yet.
-
@propercactus thanks for the reply.
What would it take to add something to rescue mode on the install so it would be easily possible to copy everything to the unused space on the installation USB drive?
-
@guardian said in How do I roll back packages:
What would it take to add something to rescue mode on the install so it would be easily possible to copy everything to the unused space on the installation USB drive?
No idea, asking wrong person. I'm brand new to pfSense and BSD
-
Package rollback is not an impossible task, but it has many nuances and some outright "hard stops" in many cases.
Packages are not standalone, because they depend on shared libraries installed on the system. Many other things on the firewall also depend on those libraries, and most times on a specific version of them. So maintaining a "rollback capability" across a pfSense operating system update would really not be possible. If package "xyz" needed a rollback, it might need to bring back now outdated versions of some shared libraries (meaning they were compiled under the older pfSense OS). Those older shared libraries wouldn't load and work on the updated pfSense. And if you forced them to install, they would then break everything else on the system.
It would be possible to offer a rollback capability of sorts so long as the underlying pfSense version is exactly the same. But right now the
pkgsystem used by FreeBSD does not offer that natively. The entire ecosystem is aimed at the "current" version of things. It would take quite a bit of coding to hack in something like a rollback feature.You can enter it as a feature request on the pfSense Redmine Site here: https://redmine.pfsense.org/projects/pfsense. Rollback is much more likely to work under ZFS using the snapshots feature. It would be similar to what you do with a virtual machine today by reverting to a previous snapshot. I think the pfSense team is heading towards making ZFS the default install method in the not too distant future. After that happens is when rollback will be more viable.
-
@bmeeks said in How do I roll back packages:
Package rollback is not an impossible task, but it has many nuances and some outright "hard stops" in many cases.
Packages are not standalone, because they depend on shared libraries installed on the system. Many other things on the firewall also depend on those libraries, and most times on a specific version of them. So maintaining a "rollback capability" across a pfSense operating system update would really not be possible. If package "xyz" needed a rollback, it might need to bring back now outdated versions of some shared libraries (meaning they were compiled under the older pfSense OS). Those older shared libraries wouldn't load and work on the updated pfSense. And if you forced them to install, they would then break everything else on the system.
It would be possible to offer a rollback capability of sorts so long as the underlying pfSense version is exactly the same. But right now the
pkgsystem used by FreeBSD does not offer that natively. The entire ecosystem is aimed at the "current" version of things. It would take quite a bit of coding to hack in something like a rollback feature.You can enter it as a feature request on the pfSense Redmine Site here: https://redmine.pfsense.org/projects/pfsense. Rollback is much more likely to work under ZFS using the snapshots feature. It would be similar to what you do with a virtual machine today by reverting to a previous snapshot. I think the pfSense team is heading towards making ZFS the default install method in the not too distant future. After that happens is when rollback will be more viable.
For now what about a stop gap of just being able to roll the system back to where it was before update? Doesn't seem too hard...
- Attempt update,
(a) everything works... good... nothing more to do...
(b) Problems that you can't live with...
(c) take some diagnostics, submit bug reports, post in forum etc.
(d) roll back,
(e) search for solutions, and then return to 1
This make a potentially urgent situation, something that can be dealt with in an non-urgent way.
Either add a rollback to the install media or even just create a well documented script/collection of scripts.
- Attempt update,
-
@bmeeks I think, if a package knows exactly what it changes on update, it can revert what it changes on rollback. It's true if another package is updated or installed after the bad one that it might rely on a dependancy of that bad update but then in that case maybe a roll back then causes a reinstall of remaining packages which will pick up any dependancies. That should work I think?
-
@propercactus said in How do I roll back packages:
@bmeeks I think, if a package knows exactly what it changes on update, it can revert what it changes on rollback. It's true if another package is updated or installed after the bad one that it might rely on a dependancy of that bad update but then in that case maybe a roll back then causes a reinstall of remaining packages which will pick up any dependancies. That should work I think?
I think the problem occurs with the rollback is required at the same time as a version upgrade. IIUC A package that works with 2.4.5 may not (or won't) work with 2.5.x, so you can't just roll back that one package, you have to roll back the whole system.
I'd be prepared to backup/restore the whole system as long as it is quick and easy. Hopefully ZFS snapshots will solve a lot of the problems. It should certainly solve a single package upgrade. Run snapshot, update package, if package update isn't good, just revert the snapshot. On an OS upgrade, the system may fail to boot. I had that happen with FreeNAS (now TrueNAS). Fortunately a clean install followed by a config update worked. Hopefully that will apply to pfSense -- except in this case I know there is a problem with SyslogNG.
Did you find out how to apply the patch?
-
@guardian said in How do I roll back packages:
Did you find out how to apply the patch?
aYep, however the fixed package has been pushed now so everyone can upgrade
