Failed update from 2.6.0 -> 2.7.0 causing lots of issues
-
@Stewart There are the docs like https://docs.netgate.com/pfsense/en/latest/troubleshooting/upgrades.html.
We've used pfSense for many years for us and clients and have very rarely had an issue. Both I can recall were on 3100s, FWIW.
The process has been changing, for instance the latest versions I believe no longer try to install packages from the "current" branch (https://redmine.pfsense.org/issues/10464#note-23). So a process description may differ a bit now as opposed to a few months ago.
Usually people shoot themselves in the foot because the counter is still going after several minutes, especially on slower hardware, so they boot halfway through, which is why I asked.
The upgrade guide suggests removing packages before version upgrades, then reinstall after, are you doing that? I do that with "big" packages like pfBlocker and Suricata but don't bother with ones like the OpenVPN export.
Settings are in the GUI, in System>Update. Using pkg isn't supposed to be necessary.
At this point, most error messages from upgrades are posted in the forum, I'd expect.
-
@SteveITS We have roughly 65 units in the field right now. This week I've attempted to upgrade 7 of them to 2.7.2.
Two upgraded from 2.4.5 just fine (in 3 upgrade steps).
Two upgraded from 2.6.0 just fine (in 2 upgrade steps).
Two have failed the same way trying to upgrade from 2.7.0 to 2.7.2 in that they can select the 2.7.1 and 2.7.2 upgrade branches but it always says it is up to date on 2.7.0 and won't allow me to upgrade.
This one has failed completely. It ran perfectly on 2.6.0 and the upgrade just went sideways.These are the upgrade steps I take:
- Remove the pfBlocker and Suricata packages
- Navigate to System -> Update.
2a. If it doesn't show the expected available update, I go to Update Settings, choose an older branch and save, then choose the latest branch and save. - Navigate back to the System Update page and initiate the update if it is available.
- I wait for it to take me to the update screen and let the process go through on its own.
4a. If the update gets stuck at "Please wait while the update system initializes" for more than 30 minutes, a reboot usually fixes the issue. - Once it comes back up I log into the CLI and run "ps -aux | grep upgrade" periodically to ensure that all of the processes have completed.
@SteveITS said in Failed update from 2.6.0 -> 2.7.0 causing lots of issues:
There are the docs like https://docs.netgate.com/pfsense/en/latest/troubleshooting/upgrades.html.
That's where I got the pkg-static clean -ay; pkg-static install -fy pkg pfSense-repo pfSense-upgrade command from.
[2.6.0-RELEASE][root@pfSense]/root: pkg-static update -f [2.6.0-RELEASE][root@pfSense]/root: pkg-static info -x pfSense-upgrade [2.6.0-RELEASE][root@pfSense]/root: pkg-static clean -ay; pkg-static install -fy pkg pfSense-repo pfSense-upgrade [2.6.0-RELEASE][root@pfSense]/root: pkg-static -d update [2.6.0-RELEASE][root@pfSense failed to update the repository settings!!! failed to update the repository settings!!! [2.6.0-RELEASE][root@pfSense]/root:
Out of all the units I generally have issues with 5-10 of them and most of those I wind up replacing. So far 3/7 have had issues. I imagine that pkg is doing a lot of the heavy lifting in the background. It's the basis of essentially all of the repair commands on the troubleshooting page.
-
@Stewart said in Failed update from 2.6.0 -> 2.7.0 causing lots of issues:
they can select the 2.7.1 and 2.7.2 upgrade branches but it always says it is up to date on 2.7.0 and won't allow me to upgrade
For that see https://docs.netgate.com/pfsense/en/latest/releases/2-7-1.html#troubleshooting
Most of the issues posted I think have revolved around Plus, changing hardware/NDI, certificates, etc. I think "failed to update the repository settings!!!" is more general but there are often other causes. Maybe these threads will help:
(edit: it posted early)
https://forum.netgate.com/topic/181308/version-2-5-2-upgrade-2-6-0-or-2-7-0/
https://forum.netgate.com/topic/182095/please-wait-while-the-update-system-initializes/We don't have a CE anymore so most of my experience is on Plus...though it's largely the same.
-
@SteveITS Thanks for the reply. Unfortunately, I've already looked at those links and they either don't apply or don't help. The update system initializing issue isn't something I'm facing now, just the most common problem I've come across that I can remedy with a reboot. It seems like most of the issues I face are with pkg or pkg-static or in that wheelhouse since those are the commands I see in the support threads. I just don't know what's actually going on. They don't ever seem to fix my issues. Sometimes it will let me get past my roadblock at the moment but ultimately I still have to swap the unit as it's littered with random issues down the road.
-
@Stewart I'll page @stephenw10 who's really good at these issues.
Are you ever installing or updating packages without adjusting the update branch? historically that can easily break things, however, that's supposed to be fixed/better after 2.7.2.
And to be clear, the cert rehash didn't fix the two that couldn't see 2.7.2?
-
-
@SteveITS I don't usually upgrade packages by themselves. I let the upgrade update the packages or I uninstall beforehand and reinstall after the update is complete.
The certctl rehash command that is in the document only mentions for 2.7.1 so I thought it only applied to 2.7.1 but it has helped the two 2.7.0 units find the 2.7.2 update. Thanks for that. I'll put it in my post about it. It also fixed a 2.7.2 that couldn't see any packages.
Back to the original issue. Still no idea what to do about this one other than a rip and replace.
-
I just had another failed update from 2.6.0 to 2.7.0. Running the update from the GUI seems fine until the reboot. Then it's offline for a few minutes. Then it comes back as 2.6.0 with a bunch of these errors:
Filter Reload
There were error(s) loading the rules: pfctl: pfi_get_ifaces: Operation not supported by device - The line in question reads [0]: @ 2024-01-04 22:41:41 There were error(s) loading the rules: pfctl: pfi_get_ifaces: Operation not supported by device - The line in question reads [0]: @ 2024-01-04 22:41:45 There were error(s) loading the rules: pfctl: pfi_get_ifaces: Operation not supported by device - The line in question reads [0]: @ 2024-01-04 22:41:46 There were error(s) loading the rules: pfctl: pfi_get_ifaces: Operation not supported by device - The line in question reads [0]: @ 2024-01-04 22:42:22 There were error(s) loading the rules: pfctl: pfi_get_ifaces: Operation not supported by device - The line in question reads [0]: @ 2024-01-04 22:43:04 There were error(s) loading the rules: pfctl: pfi_get_ifaces: Operation not supported by device - The line in question reads [0]: @ 2024-01-04 22:43:12 There were error(s) loading the rules: pfctl: pfi_get_ifaces: Operation not supported by device - The line in question reads [0]: @ 2024-01-04 22:43:14 There were error(s) loading the rules: pfctl: pfi_get_ifaces: Operation not supported by device - The line in question reads [0]: @ 2024-01-04 22:43:55 There were error(s) loading the rules: pfctl: pfi_get_ifaces: Operation not supported by device - The line in question reads [0]: @ 2024-01-04 22:44:09 There were error(s) loading the rules: pfctl: pfi_get_ifaces: Operation not supported by device - The line in question reads [0]: @ 2024-01-04 22:44:10 There were error(s) loading the rules: pfctl: pfi_get_ifaces: Operation not supported by device - The line in question reads [0]: @ 2024-01-04 22:44:30 There were error(s) loading the rules: pfctl: pfi_get_ifaces: Operation not supported by device - The line in question reads [0]: @ 2024-01-04 22:44:38 There were error(s) loading the rules: pfctl: pfi_get_ifaces: Operation not supported by device - The line in question reads [0]: @ 2024-01-04 22:44:40 There were error(s) loading the rules: pfctl: pfi_get_ifaces: Operation not supported by device - The line in question reads [0]: @ 2024-01-04 22:44:57 There were error(s) loading the rules: pfctl: pfi_get_ifaces: Operation not supported by device - The line in question reads [0]: @ 2024-01-04 22:45:02
Traffic no longer flows. Port forwards don't work. It can't check for updates. Has anyone else reported this issue?
-
What is your underlying hardware platform?
I suspect you are having these issues after reboot because you are winding up with a mixed-version system somehow (some 2.6.0 stuff and some 2.7.0 stuff).
Is your BIOS configured to use UEFI by chance? Some hardware vendors and pfSense (really FreeBSD) don't like each other when using UEFI.
Can you login directly to the firewall console and watch the messages during the actual upgrade process AND during the follow-on reboot? Might be some clues there.
-
@bmeeks I can only find one other instance of this when I search online and that is that jail wasn't upgraded when the host was or something like that. Their answer from here was:
I was upgrading the jail and that showed `13.0-RELEASE p2`, but that wasn't enough. I created a new 13.0-RELEASE jail (after fighting with bsdinstall 12.1 see here) and that was it, configured pf just like in a physical machine.
These units are all PCEngine APU2x4 units. This unit that just had problems have the following packages installed that I didn't remove beforehand:
Cron
Darkstat
Mailreport
nmap
Notes
RRD_Summary
Status_Traffic_Totals
sudo
zabbix-agent4I've never uninstalled them before for an update but if any of them could be the culprit them I'm willing to take them out. Normally I only pull out pfBlocker and Suricata as those are the only ones that have ever given me issues (and Squid but I don't use that one anymore).
-
It certainly will not hurt to first remove all the packages, then reboot the box just to be sure it comes back fine and healthy (it should). After this, attempt the upgrade.
To be clear though, at this point the box in discussion for this thread should get a complete reinstall from an ISO or memstick image (either restore 2.6.0 or just go ahead and re-image to the 2.7.2 version). This box is currently hosed.
As I mentioned, for future updates if you can login to the console of the device during the update, I would monitor the screen to see what scrolls by. That way you can see any errors that may print. Ditto for the reboot that follows the unpacking/installation of the new pfSense kernel. If the boxes are all remote, I realize this will not be possible unless you can rig some sort of backdoor process with a laptop connected to the console and a cellphone modem or something.
Once the box is up and the basic firewall configuration appears intact, then reinstall the packages.
What you are experiencing is definitely not normal. As for "why" it's happening, the only way to begin to make a guess is to watch the messages that are printed to the console. There may some limited info in the system log, but it depends on exactly when/where the upgrade is failing to complete.
The error message you posted from the system log clearly signals to me you have maybe the 2.6.0 pfSense kernel but 2.7.0 PHP code (or some combination of the two). The reason I say this is that
pfctl
uses the relatively newlibpftcl
library and that library was updated with the recent pfSense release. The message "Operation not supported by device" is my clue here. The PHP code is askingpfctl
to do something that the corresponding library in 2.6.0 pfSense does not support (but 2.7.0 and up does).Edit: one other random thought -- is the BIOS firmware current on these PCEngine APU2x4 units? I did find some posts with a Google search where old firmware in the units can have issues with the most current FreeBSD kernels. Although in the posts I found the symptom was failure to mount the system disk and thus never getting beyond the boot prompt. Yours appear to be getting farther than that, but also do not appear to have actually completed the kernel upgrade.
-
That pfctl error is because there's a kernel/world mismatch. It's trying to use a version of pfctl against a different version of pf.
So that implies the upgrade only partially completed. Running
pkg-static info -x pfsense
should show some mismatched pkgs.You can probably just upgrade pkgs to complete the upgrade if they show as available.
Steve
-
@bmeeks I'm not sure on the firmware. I had that thought last night. Normally our units are 4.11.0.6. I know the first one was that version. The one I swapped today I'm not so sure. It's relatively old at this point but since all of our devices are remote I'm not too keen on updating that unless I need to. I can't install or uninstall packages. It says there aren't any installed and it can't check for any. If I need to monitor each unit for the upgrade I'd rather just build up a new box in the office and import the config on site and swap them. But that'll be hundreds of miles of driving, maybe more, so I'd like to not go that route.
@stephenw10 What commands would I run to complete the package upgrades?
Running pkg-static clean -ay; pkg-static install -fy pkg pfSense-repo pfSense-upgrade appears to do nothing in that it just goes to the next line.
Updating from the command line gives me:
Segmentation fault (core dumped) failed to update the repository settings!!! failed to update the repository settings!!!
I'm going to hold off on more upgrades until I can get a handle on these. I have 2 of these failed units with me now so any commands I can run to either fix or get more information on what's happening would be appreciated.
-
If it's partially upgraded you should run
pkg-static upgrade
and agree to the offered pkg upgrades. -
@stephenw10 Got a lot of projects today so I'm not sure I'll get to it. I'll reply back when I do, though. Thanks for the help on this one.
I decided to run these real quick before I head out.
[2.6.0-RELEASE][root@pfSense]/root: pkg-static info -x pfsense [2.6.0-RELEASE][root@pfSense]/root: pkg-static upgrade [2.6.0-RELEASE][root@pfSense]/root: pkg-static install [2.6.0-RELEASE][root@pfSense]/root: pkg-static inst [2.6.0-RELEASE][root@pfSense]/root: pkg-static i [2.6.0-RELEASE][root@pfSense]/root: pkg-static bootstrap -f
All just go on to the next line. Nothing is output to the screen.
[2.6.0-RELEASE][root@pfSense]/root: pkg-static ? pkg-static: No match.
This was just to see what would happen. Is pkg-static prt of pkg? Is there a way to force the reinstall of it?
-
Yes, but only if you can access the pkg servers:
pkg-static -d upgrade -f pkg
-
@stephenw10 said in Failed update from 2.6.0 -> 2.7.0 causing lots of issues:
Yes, but only if you can access the pkg servers:
pkg-static -d upgrade -f pkg
That just goes to the next line as well.
Can I pull something out of /var/cache/pkg ?
-
Yes, you can force a reinstall from there using the full path if there are any cached versions of pkg available.
-
I just wanted to follow up on this and let everyone know I could never get anywhere. While I did backup/restore last month I kept this box around to try to troubleshoot. No pkg command would yield any results, even trying to install from local.
If anyone winds up in the same boat, I'm afraid this post likely won't help.