Failed update from 2.6.0 -> 2.7.0 causing lots of issues

SteveITS

@Stewart said in Failed update from 2.6.0 -> 2.7.0 causing lots of issues:

they can select the 2.7.1 and 2.7.2 upgrade branches but it always says it is up to date on 2.7.0 and won't allow me to upgrade

For that see https://docs.netgate.com/pfsense/en/latest/releases/2-7-1.html#troubleshooting

Most of the issues posted I think have revolved around Plus, changing hardware/NDI, certificates, etc. I think "failed to update the repository settings!!!" is more general but there are often other causes. Maybe these threads will help:

(edit: it posted early)
https://forum.netgate.com/topic/181308/version-2-5-2-upgrade-2-6-0-or-2-7-0/
https://forum.netgate.com/topic/182095/please-wait-while-the-update-system-initializes/

We don't have a CE anymore so most of my experience is on Plus...though it's largely the same.

Stewart

@SteveITS Thanks for the reply. Unfortunately, I've already looked at those links and they either don't apply or don't help. The update system initializing issue isn't something I'm facing now, just the most common problem I've come across that I can remedy with a reboot. It seems like most of the issues I face are with pkg or pkg-static or in that wheelhouse since those are the commands I see in the support threads. I just don't know what's actually going on. They don't ever seem to fix my issues. Sometimes it will let me get past my roadblock at the moment but ultimately I still have to swap the unit as it's littered with random issues down the road.

SteveITS

@Stewart I'll page @stephenw10 who's really good at these issues.

Are you ever installing or updating packages without adjusting the update branch? historically that can easily break things, however, that's supposed to be fixed/better after 2.7.2.

And to be clear, the cert rehash didn't fix the two that couldn't see 2.7.2?

Stewart

@SteveITS I don't usually upgrade packages by themselves. I let the upgrade update the packages or I uninstall beforehand and reinstall after the update is complete.

The certctl rehash command that is in the document only mentions for 2.7.1 so I thought it only applied to 2.7.1 but it has helped the two 2.7.0 units find the 2.7.2 update. Thanks for that. I'll put it in my post about it. It also fixed a 2.7.2 that couldn't see any packages.

Back to the original issue. Still no idea what to do about this one other than a rip and replace.

Stewart

I just had another failed update from 2.6.0 to 2.7.0. Running the update from the GUI seems fine until the reboot. Then it's offline for a few minutes. Then it comes back as 2.6.0 with a bunch of these errors:

Filter Reload

There were error(s) loading the rules: pfctl: pfi_get_ifaces: Operation not supported by device - The line in question reads [0]: @ 2024-01-04 22:41:41
There were error(s) loading the rules: pfctl: pfi_get_ifaces: Operation not supported by device - The line in question reads [0]: @ 2024-01-04 22:41:45
There were error(s) loading the rules: pfctl: pfi_get_ifaces: Operation not supported by device - The line in question reads [0]: @ 2024-01-04 22:41:46
There were error(s) loading the rules: pfctl: pfi_get_ifaces: Operation not supported by device - The line in question reads [0]: @ 2024-01-04 22:42:22
There were error(s) loading the rules: pfctl: pfi_get_ifaces: Operation not supported by device - The line in question reads [0]: @ 2024-01-04 22:43:04
There were error(s) loading the rules: pfctl: pfi_get_ifaces: Operation not supported by device - The line in question reads [0]: @ 2024-01-04 22:43:12
There were error(s) loading the rules: pfctl: pfi_get_ifaces: Operation not supported by device - The line in question reads [0]: @ 2024-01-04 22:43:14
There were error(s) loading the rules: pfctl: pfi_get_ifaces: Operation not supported by device - The line in question reads [0]: @ 2024-01-04 22:43:55
There were error(s) loading the rules: pfctl: pfi_get_ifaces: Operation not supported by device - The line in question reads [0]: @ 2024-01-04 22:44:09
There were error(s) loading the rules: pfctl: pfi_get_ifaces: Operation not supported by device - The line in question reads [0]: @ 2024-01-04 22:44:10
There were error(s) loading the rules: pfctl: pfi_get_ifaces: Operation not supported by device - The line in question reads [0]: @ 2024-01-04 22:44:30
There were error(s) loading the rules: pfctl: pfi_get_ifaces: Operation not supported by device - The line in question reads [0]: @ 2024-01-04 22:44:38
There were error(s) loading the rules: pfctl: pfi_get_ifaces: Operation not supported by device - The line in question reads [0]: @ 2024-01-04 22:44:40
There were error(s) loading the rules: pfctl: pfi_get_ifaces: Operation not supported by device - The line in question reads [0]: @ 2024-01-04 22:44:57
There were error(s) loading the rules: pfctl: pfi_get_ifaces: Operation not supported by device - The line in question reads [0]: @ 2024-01-04 22:45:02

Traffic no longer flows. Port forwards don't work. It can't check for updates. Has anyone else reported this issue?

bmeeks

What is your underlying hardware platform?

I suspect you are having these issues after reboot because you are winding up with a mixed-version system somehow (some 2.6.0 stuff and some 2.7.0 stuff).

Is your BIOS configured to use UEFI by chance? Some hardware vendors and pfSense (really FreeBSD) don't like each other when using UEFI.

Can you login directly to the firewall console and watch the messages during the actual upgrade process AND during the follow-on reboot? Might be some clues there.

Stewart

@bmeeks I can only find one other instance of this when I search online and that is that jail wasn't upgraded when the host was or something like that. Their answer from here was:

I was upgrading the jail and that showed `13.0-RELEASE p2`, but that wasn't enough.
I created a new 13.0-RELEASE jail (after fighting with bsdinstall 12.1 see here) and that was it, configured pf just like in a physical machine.

These units are all PCEngine APU2x4 units. This unit that just had problems have the following packages installed that I didn't remove beforehand:

Cron
Darkstat
Mailreport
nmap
Notes
RRD_Summary
Status_Traffic_Totals
sudo
zabbix-agent4

I've never uninstalled them before for an update but if any of them could be the culprit them I'm willing to take them out. Normally I only pull out pfBlocker and Suricata as those are the only ones that have ever given me issues (and Squid but I don't use that one anymore).

bmeeks

It certainly will not hurt to first remove all the packages, then reboot the box just to be sure it comes back fine and healthy (it should). After this, attempt the upgrade.

To be clear though, at this point the box in discussion for this thread should get a complete reinstall from an ISO or memstick image (either restore 2.6.0 or just go ahead and re-image to the 2.7.2 version). This box is currently hosed.

As I mentioned, for future updates if you can login to the console of the device during the update, I would monitor the screen to see what scrolls by. That way you can see any errors that may print. Ditto for the reboot that follows the unpacking/installation of the new pfSense kernel. If the boxes are all remote, I realize this will not be possible unless you can rig some sort of backdoor process with a laptop connected to the console and a cellphone modem or something.

Once the box is up and the basic firewall configuration appears intact, then reinstall the packages.

What you are experiencing is definitely not normal. As for "why" it's happening, the only way to begin to make a guess is to watch the messages that are printed to the console. There may some limited info in the system log, but it depends on exactly when/where the upgrade is failing to complete.

The error message you posted from the system log clearly signals to me you have maybe the 2.6.0 pfSense kernel but 2.7.0 PHP code (or some combination of the two). The reason I say this is that pfctl uses the relatively new libpftcl library and that library was updated with the recent pfSense release. The message "Operation not supported by device" is my clue here. The PHP code is asking pfctl to do something that the corresponding library in 2.6.0 pfSense does not support (but 2.7.0 and up does).

Edit: one other random thought -- is the BIOS firmware current on these PCEngine APU2x4 units? I did find some posts with a Google search where old firmware in the units can have issues with the most current FreeBSD kernels. Although in the posts I found the symptom was failure to mount the system disk and thus never getting beyond the boot prompt. Yours appear to be getting farther than that, but also do not appear to have actually completed the kernel upgrade.

stephenw10

That pfctl error is because there's a kernel/world mismatch. It's trying to use a version of pfctl against a different version of pf.

So that implies the upgrade only partially completed. Running pkg-static info -x pfsense should show some mismatched pkgs.

You can probably just upgrade pkgs to complete the upgrade if they show as available.

Steve

Stewart

@bmeeks I'm not sure on the firmware. I had that thought last night. Normally our units are 4.11.0.6. I know the first one was that version. The one I swapped today I'm not so sure. It's relatively old at this point but since all of our devices are remote I'm not too keen on updating that unless I need to. I can't install or uninstall packages. It says there aren't any installed and it can't check for any. If I need to monitor each unit for the upgrade I'd rather just build up a new box in the office and import the config on site and swap them. But that'll be hundreds of miles of driving, maybe more, so I'd like to not go that route.

@stephenw10 What commands would I run to complete the package upgrades?

Running pkg-static clean -ay; pkg-static install -fy pkg pfSense-repo pfSense-upgrade appears to do nothing in that it just goes to the next line.

Updating from the command line gives me:

Segmentation fault (core dumped)
failed to update the repository settings!!!
failed to update the repository settings!!!

I'm going to hold off on more upgrades until I can get a handle on these. I have 2 of these failed units with me now so any commands I can run to either fix or get more information on what's happening would be appreciated.

stephenw10

If it's partially upgraded you should run pkg-static upgrade and agree to the offered pkg upgrades.

Stewart

@stephenw10 Got a lot of projects today so I'm not sure I'll get to it. I'll reply back when I do, though. Thanks for the help on this one.

I decided to run these real quick before I head out.

[2.6.0-RELEASE][root@pfSense]/root: pkg-static info -x pfsense
[2.6.0-RELEASE][root@pfSense]/root: pkg-static upgrade
[2.6.0-RELEASE][root@pfSense]/root: pkg-static install
[2.6.0-RELEASE][root@pfSense]/root: pkg-static inst
[2.6.0-RELEASE][root@pfSense]/root: pkg-static i
[2.6.0-RELEASE][root@pfSense]/root: pkg-static bootstrap -f

All just go on to the next line. Nothing is output to the screen.

[2.6.0-RELEASE][root@pfSense]/root: pkg-static ?
pkg-static: No match.

This was just to see what would happen. Is pkg-static prt of pkg? Is there a way to force the reinstall of it?

stephenw10

Yes, but only if you can access the pkg servers:
pkg-static -d upgrade -f pkg

Stewart

@stephenw10 said in Failed update from 2.6.0 -> 2.7.0 causing lots of issues:

Yes, but only if you can access the pkg servers:
pkg-static -d upgrade -f pkg

That just goes to the next line as well.

Can I pull something out of /var/cache/pkg ?

stephenw10

Yes, you can force a reinstall from there using the full path if there are any cached versions of pkg available.

Stewart

I just wanted to follow up on this and let everyone know I could never get anywhere. While I did backup/restore last month I kept this box around to try to troubleshoot. No pkg command would yield any results, even trying to install from local.

If anyone winds up in the same boat, I'm afraid this post likely won't help.