How do I roll back packages
-
AFAIK, there is not an official supported way of doing this?
But in all my years of using pfsense, really since its been available.. I can not recall an issue with a package that would of warranted rollback to an previous version. Now I do not use all packages, so can not say its never happened on any package.
But been very active on the forums for 10 some years.. And don't really recall any threads where such a thing is an issue..
If your concerned... Prob just suggest you wait for a X amount of time after a new version of whatever package before updating it.
I can tell you running the latest version of haproxy-devel 0.60_9, and its working fine for my setup.
If your curious to what an update has changed, you can click the package number and it will take you to github and you can look to see exactly what has changed.
example for the latest haproxy-devel
https://github.com/pfsense/FreeBSD-ports/commits/devel/net/pfSense-pkg-haproxy-develNormally the updates are minor fixes for stuff with minimal code changes..
-
@johnpoz said in How do I roll back packages:
AFAIK, there is not an official supported way of doing this?
But in all my years of using pfsense, really since its been available.. I can not recall an issue with a package that would of warranted rollback to an previous version. Now I do not use all packages, so can not say its never happened on any package.
But been very active on the forums for 10 some years.. And don't really recall any threads where such a thing is an issue..
If your concerned... Prob just suggest you wait for a X amount of time after a new version of whatever package before updating it.
I can tell you running the latest version of haproxy-devel 0.60_9, and its working fine for my setup.
If your curious to what an update has changed, you can click the package number and it will take you to github and you can look to see exactly what has changed.
example for the latest haproxy-devel
https://github.com/pfsense/FreeBSD-ports/commits/devel/net/pfSense-pkg-haproxy-develNormally the updates are minor fixes for stuff with minimal code changes..
Thanks for the reply @Johnpoz.... If I save the config file before doing the update, and then do a restore, does it install the old plugin that was installed at the time the backup wias mede, or do you get stuck with the new ones.
-
@guardian said in How do I roll back packages:
does it install the old plugin that was installed at the time the backup was made
It will install whatever is known by the name of the pfSense package at that moment.
The pfSense package server only has the latest version of everything. As far as I know, there are no official "old packages" available. -
To add to what others have posted here. Packages in most cases consist of both binary and PHP GUI components. The binary portion of the package is compiled against a specific version of the operating system (in the case of pfSense that means a modified and hardened version of FreeBSD). That binary version, in most cases, cannot run on an operating system version different than the one it was compiled on. Thus the only way to "roll back" a package would be to also rollback the pfSense version to the older version to match the environment where the package was compiled. That introduces security issues.
So the official packages repository is always populated with package versions compiled against the current pfSense version. If you try and force a rollback to an older package, one of two bad things can happen. First, the package may simply refuse to load and run due to issues with shared library versions. This is the most likely outcome. Or secondly, the installation of the older package may bring in older versions of critcal shared libraries and break the operating system (pfSense) to the extent it won't run. This would be the worst, albeit most rare, outcome.
So rolling back to older package versions is really just not an option. If you are understandably nervous about upgrading to the latest version of a package, you can wait for a period of time and monitor the forums here to see if there any issues. Developers are quick to fix bugs that impact a package's operation.
-
@bmeeks said in How do I roll back packages:
So rolling back to older package versions is really just not an option.
Completely agree when you update pfsense to new version. But way I read the question was more like on version 123 of pfsense, version A of package X, and version B comes out. Still on version 123 of pfsense.
And something doesn't work as expected in this new version B of the package. So how to go back to version A?
Which I really don't think is possible either..
-
@johnpoz said in How do I roll back packages:
@bmeeks said in How do I roll back packages:
So rolling back to older package versions is really just not an option.
Completely agree when you update pfsense to new version. But way I read the question was more like on version 123 of pfsense, version A of package X, and version B comes out. Still on version 123 of pfsense.
And something doesn't work as expected in this new version B of the package. So how to go back to version A?
Which I really don't think is possible either..
True, so long as the underlying pfSense version is unchanged, the binary from Package version A will work with Package version B and vice-versa. The real sticky point is the repository keeps only a single "version" of each package in the current tree. I'm not a
pkgguru so I don't know if having multiple "current" versions is possible. I'm thinking not, and even if there were, it might open users up to a new way to shoot their foot off ...
-
@bmeeks said in How do I roll back packages:
keeps only a single "version" of each package in the current tree.
exactly... Guess its something someone could put a feature request in for.
But as you mentioned on how fast package maintainers respond is something is wrong.. I think it would overly complex up everything to try and maintain different versions of packages.. And may end up causing more issues than it could possible prevent.
I honestly can not recall and issue where I would of ever wanted to roll back a package.. While I don't use all of them.. I don't recall any major threads were there was any sort of issue with a package update that broke something bad..
-
@johnpoz said in How do I roll back packages:
I think it would overly complex up everything to try and maintain different versions of packages.. And may end up causing more issues than it could possible prevent.
Plus one on that point! I had to maintain two versions of Suricata for ARM and AMD/Intel hardware and it was a huge pain and an area full of potential landmines when making changes. It was difficult for me as a developer to keep the two versions updated.
Something similar would likely occur with multiple versions of a package even on the same hardware platform.
-
Thanks @bmeeks, @johnpoz, @Gertjan - I can understand the difficulty on the multiple versions. The point I'd like to add to the discussion is that rule #1 with computers is BACKUP, BACKUP, BACKUP.
I've been raising the issue of fallback for several years. I was suggesting some sort of imaging, but @jimp and others kept telling me that just reloading config.xml would solve all problems
[https://forum.netgate.com/post/727596](link url) . What I am learning here is, this is not true.Admittedly pfSense has been very reliable, but not having a quick recovery fallback is troubling and seems to go against what are established industry best practice (BACKUP!).
At very least I would think a built in image/restore would at least make sure a bad update could be fixed in minutes. This would make frequent updates almost NO RISK, so they could be done without the necessity of a lenghty "insurance maintenace window" just in case something went wrong.
Thoughts?
-
@guardian said in How do I roll back packages:
Thanks @bmeeks, @johnpoz, @Gertjan - I can understand the difficulty on the multiple versions. The point I'd like to add to the discussion is that rule #1 with computers is BACKUP, BACKUP, BACKUP.
I've been raising the issue of fallback for several years. I was suggesting some sort of imaging, but @jimp and others kept telling me that just reloading config.xml would solve all problems
[https://forum.netgate.com/post/727596](link url) . What I am learning here is, this is not true.Admittedly pfSense has been very reliable, but not having a quick recovery fallback is troubling and seems to go against what are established industry best practice (BACKUP!).
At very least I would think a built in image/restore would at least make sure a bad update could be fixed in minutes. This would make frequent updates almost NO RISK, so they could be done without the necessity of a lenghty "insurance maintenace window" just in case something went wrong.
Thoughts?
Then you need to use a virtual machine with snapshots. Nothing quicker than that to restore a "gone bad" upgrade. pfSense runs fine on ESXi. A couple of the other hypervisors have given issues from time to time, but ESXi has been pretty stable as far as I know. And snapshots require nothing extra be installed on pfSense itself. I don't mean for this response to be flippant. Using a VM is an excellent way to have image backups and restores with minimal effort via snapshots. Just take a snapshot before any firewall change. If something goes wrong, revert to the previous snapshot.
What @jimp says is 100% true for a plain-vanilla pfSense installation. With packages installed, things do get more complicated. But users need to understand two vital things about packages. First, they are optional and not required by pfSense. And second, they are created and maintained by volunteer non-paid maintainers in almost every case. So when you opt to use an add-on package, there are inherent risks in that decision as contrasted with using a plain-vanilla pfSense install. I get that some packages offer very popular or often needed features, but they are not part of the base pfSense install and are not (for the most part) maintained by the pfSense developer team. So when you install packages, you assume some risk there.
I can understand the reluctance of the pfSense team to start adding "fluff" to a security system such as a firewall. An image backup ability would entail loading additional software into the pfSense core system and that will inevitably increase potential attack surfaces. Generally speaking, it's just never a good idea to put a lot of extra software on a firewall. And yes, that can apply to many of the available packages as well. There is certainly a valid argument to be made that they in some ways decrease the firewall's security posture by adding additional attack surfaces.
I managed Checkpoint firewalls for many years, and from what I remember they had no equivalent of an image backup/restore method either. They offered essentially the same ability as pfSense. You could back up and restore your configuration, but you first had to reinstall the base OS firewall package, and then import your backup configuration. So pretty much the same way you restore pfSense. Don't know about the fancy Palo Alto firewalls. My company was just switching over to them when I retired and I never got to play with one. The closest thing I ever came across to what you would like is the old Nokia IP-series security appliances with their IPSO operating system. It could maintain a sort of dual-boot environment where you could boot into a new "upgrade", but if that failed, you could reboot back into your previous environment. However, as best I can remember, that only applied to the IPSO operating system itself. The Checkpoint packages we ran on top of IPSO still had to be reinstalled and their configuration restored as I previously described.
-
@bmeeks said in How do I roll back packages:
Then you need to use a virtual machine with snapshots. Nothing quicker than that to restore a "gone bad" upgrade. pfSense runs fine on ESXi. A couple of the other hypervisors have given issues from time to time, but ESXi has been pretty stable as far as I know. And snapshots require nothing extra be installed on pfSense itself. I don't mean for this response to be flippant. Using a VM is an excellent way to have image backups and restores with minimal effort via snapshots. Just take a snapshot before any firewall change. If something goes wrong, revert to the previous snapshot.
Good idea, but my hardware doesn't have the excess resources. I am still running a UFS install, but I'm wondering if there is a ZFS install available, and if anything has been done to support snapshot/rollback.
What @jimp says is 100% true for a plain-vanilla pfSense installation. With packages installed, things do get more complicated. But users need to understand two vital things about packages. First, they are optional and not required by pfSense. And second, they are created and maintained by volunteer non-paid maintainers in almost every case. So when you opt to use an add-on package, there are inherent risks in that decision as contrasted with using a plain-vanilla pfSense install. I get that some packages offer very popular or often needed features, but they are not part of the base pfSense install and are not (for the most part) maintained by the pfSense developer team. So when you install packages, you assume some risk there.
I can understand the reluctance of the pfSense team to start adding "fluff" to a security system such as a firewall. An image backup ability would entail loading additional software into the pfSense core system and that will inevitably increase potential attack surfaces. Generally speaking, it's just never a good idea to put a lot of extra software on a firewall. And yes, that can apply to many of the available packages as well. There is certainly a valid argument to be made that they in some ways decrease the firewall's security posture by adding additional attack surfaces.
What would it take to add something to the install media? USB drives are getting quite big... boot into the install media, select an option and a script runs and copies system and configuration data to the USB.
I managed Checkpoint firewalls for many years, and from what I remember they had no equivalent of an image backup/restore method either. They offered essentially the same ability as pfSense. You could back up and restore your configuration, but you first had to reinstall the base OS firewall package, and then import your backup configuration. So pretty much the same way you restore pfSense. Don't know about the fancy Palo Alto firewalls. My company was just switching over to them when I retired and I never got to play with one. The closest thing I ever came across to what you would like is the old Nokia IP-series security appliances with their IPSO operating system. It could maintain a sort of dual-boot environment where you could boot into a new "upgrade", but if that failed, you could reboot back into your previous environment. However, as best I can remember, that only applied to the IPSO operating system itself. The Checkpoint packages we ran on top of IPSO still had to be reinstalled and their configuration restored as I previously described.
That is the strenght/weakness of proprietary solutions. On the + side, since the manufacturer has 100% control over hardware/software they can (if they are a quality provider) do very through testing/QA so problems should be very very rare. On the - side, if the provider doesn't do things the way you want, there is nothing you can do about it.
-
@guardian said in How do I roll back packages:
Good idea, but my hardware doesn't have the excess resources. I am still running a UFS install, but I'm wondering if there is a ZFS install available, and if anything has been done to support snapshot/rollback.
There is a ZFS install option for pfSense now. Here is a link to the official install documentation. Scroll down the page a bit to see the file system options: https://docs.netgate.com/pfsense/en/latest/install/install-pfsense.html.
There is also an open Feature Request on the Redmine site for taking a snapshot on ZFS systems prior to performing an upgrade: https://redmine.pfsense.org/issues/10237.
In terms of your USB suggestion, that is already available. You can create install media on USB with a previous
config.xmlconfiguration file on the media so that during the installation pfSense will restore the configuration. However, this does not include the binary bits of any packages. It would be only the user-customized configuration parameters. As I mentioned in my first post, most packages have binary pieces and GUI configuration pieces. The binary pieces (shared libraries and executable machine code for the service itself) are pulled down from thepkgrespository. -
@johnpoz said in How do I roll back packages:
AFAIK, there is not an official supported way of doing this?
But in all my years of using pfsense, really since its been available.. I can not recall an issue with a package that would of warranted rollback to an previous version. Now I do not use all packages, so can not say its never happened on any package.
But been very active on the forums for 10 some years.. And don't really recall any threads where such a thing is an issue..
If your concerned... Prob just suggest you wait for a X amount of time after a new version of whatever package before updating it.
I can tell you running the latest version of haproxy-devel 0.60_9, and its working fine for my setup.
If your curious to what an update has changed, you can click the package number and it will take you to github and you can look to see exactly what has changed.
example for the latest haproxy-devel
https://github.com/pfsense/FreeBSD-ports/commits/devel/net/pfSense-pkg-haproxy-develNormally the updates are minor fixes for stuff with minimal code changes..
Well be amazed because right now the latest update of syslog-ng is broken. It does not like tls settings. The package before update was working fine, so I need to revert to old package.
-
See this
https://github.com/pfsense/FreeBSD-ports/commit/a5b1eda67c40592e14806a4a4bbdd946f0461045#commentsHas a patch to fix the tls problem with it.
-
@johnpoz said in How do I roll back packages:
https://github.com/pfsense/FreeBSD-ports/commit/a5b1eda67c40592e14806a4a4bbdd946f0461045#comments
It says that the patch can't apply clean and won't show me the apply button
-
@propercactus said in How do I roll back packages:
@johnpoz said in How do I roll back packages:
https://github.com/pfsense/FreeBSD-ports/commit/a5b1eda67c40592e14806a4a4bbdd946f0461045#comments
It says that the patch can't apply clean and won't show me the apply button
That posted patch link is for someone who has their own GitHub clone of the package repo and builds the package themselves. Nobody much does that -- really only package developers.
You can still manually apply the changes by making the edits to
/usr/local/pkg/syslog-ng.incshown in thedifffile. That means deleting the text in red and adding the text in green at the locations shown. Make a copy of the original file before attempting this, so you can quickly recover. You can use either the built-in option in the GUI under DIAGNOSTICS > EDIT FILE, or obtain a shell prompt on the firewall and usevi.The above instructions are given in the event you are in a rush for the fix. If not, it may be better to wait for the package to get updated in the Packages repo, and then update from the Package Manager in the GUI.
-
I think this is a really strong case for needing package roll back.
The offending commit that broke TLS is 16 days old, so even if I had of waited 14 days before upgrade i still would have upgraded and broken.
It is fortunate that the commit author very promptly supplied a patch once alerted. But if they did not, what can we do? We are stuck with a broken package!
-
@propercactus said in How do I roll back packages:
I think this is a really strong case for needing package roll back.
The offending commit that broke TLS is 16 days old, so even if I had of waited 14 days before upgrade i still would have upgraded and broken.
It is fortunate that the commit author very promptly supplied a patch once alerted. But if they did not, what can we do? We are stuck with a broken package!
@propercactus thanks for chiming in.... I need SyslogNG, and if I had updated I'd be f%ck'd. It looks like the patch doesn't work... at least it's not straight forward.
How hard would it be to include a scripts in the install disk to make it easy to save the system/packages? If things go south, at least you can roll everything back.
If the original install is available, then there should be a way to just backup key directories with the packages.
Once I can get through this upgrade, I think I'll try converting to a ZFS install, hopefully a ZFS snapshot will solve these problems - or is that just wishful thinking?
-
@guardian said in How do I roll back packages:
It looks like the patch doesn't work...
The patch is working, I was initially confusing myself and pulling the offending commit as the patch in stead of the actual patch.
But yah for the first time in a while I felt completely hopeless, my reporting from the device to SOC was down and what could I do? Not a good position to be in at all.
@guardian said in How do I roll back packages:
hopefully a ZFS snapshot will solve these problems - or is that just wishful thinking?
Good question I'll have to have a look at that myself as it's not something I know anything about yet.
-
@propercactus thanks for the reply.
What would it take to add something to rescue mode on the install so it would be easily possible to copy everything to the unused space on the installation USB drive?
