Don't upgrade to x86 CE build 2.4.0.r.20171008.0625

jimp

Erm… I have a strong feeling that the pre-kaboom snapshots should be taken, released as 2.4.0 and this insane messing should be moved to 2.4.1. WTH is anyone doing changes like this days before release, after months and months of beta testing.

It's not quite that simple, but in a vague sort of way that's what we did.

The change made was a small change to a data structure size meant to fix an interface issue. This caused older binaries for services and other programs to fail on an updated kernel, which left the system in an inconsistent state and likely without connectivity. The packages were recompiled with the new structures, but their version numbers didn't change, so pkg did not know it had to reinstall all affected packages. New installs made during this time were OK, and anyone who forced all packages to reinstall would have been OK as well, because in those cases the firewall had a fully consistent set of binaries for kernel, world, and packages.

We ultimately decided to take down those problem snapshots and back out that change, so that people who had not yet upgraded to a problem snapshot would be safe from the update issue. After reverting that change and rebuilding all of the package sets, then testing internally, we turned everything back on. We tried a few alternate solutions such as forcing pkg to update everything on upgrades between certain timestamps but we were not satisfied with any of the workarounds.

For those who did not update during the problem time, updating to a current snapshot is safe.

If you updated a firewall to a broken snapshot and fixed it manually by using "pkg upgrade -f", or if you made a fresh install from a problem image, you will need to manually update to the latest set of packages using "pkg upgrade -f" to forcefully reinstall everything. Otherwise, the firewall will end up in an inconsistent broken state again. Alternately, reinstall from a new snapshot.

Unrelated to that problem, we also had an issue with logging late last week. A problem with the clog program generated log files in an invalid format, leading to a variety of error message and a lack of logging data. The problem has been corrected on current snapshots, but if a firewall had generated new log files during the problematic period, such as on a fresh installation or after resetting/clearing logs, then those log files must be reset.

To reset the log files, navigate to Status > System Logs, Settings tab. Click "Reset Log Files", then OK to confirm.

At the moment all of the current snapshots should be OK and trouble-free.

kpa

@doktonotor The problem is the pkg consider only the version numbers when it looks for updates, if you did 'pkg upgrade' after the repos had been rebuilt you still wouldn't get the rebuilt dhcpd package because the version number had stayed the same. Only 'pkg install -f isc-dhcpd43-server' (or just 'pkg upgrade -f' to reinstall everything) gave you the updated package because that forced a refetch and reinstall. This is not really a fault of pkg because it just does what it's told, the real fault is the version numbering used in FreeBSD ports that doesn't allow the packages to have their version numbers upped if the repository has to be rebuilt because of API breakage as was the case here. Yes you could up them in the port Makefiles manually but that's a real mess and should be avoided.

doktornotor

Eh, I'll just reinstall the bunch of affected testing boxes. I don't really trust any pkg magic to make things consistent.

Cadish

Hmm. I get these errors:

Enter an option: 13

pkg: Unable to determine ABI
pkg: Cannot parse configuration file!
pkg: Unable to determine ABI
pkg: Cannot parse configuration file!
ERROR: It was not possible to identify which pfSense kernel is installed
*** Welcome to pfSense 2.4.0-RC (amd64) on firewall ***

pkg upgrade -f
Child process pid=27587 terminated abnormally: Segmentation fault

Is there anything I can do, except complete reinstall?

jimp

@Cadish:

pkg upgrade -f
Child process pid=27587 terminated abnormally: Segmentation fault

Try using "pkg-static" instead of pkg, but from the sound of it, a reinstall would be safer, faster, and less problematic.

doktornotor

While talking about pkg-static, isn't that something that should be preferably used on upgrades in general? The flood of console hints about using it on upgrading from 2.3.x would suggest it to be the case.

jimp

@doktornotor:

While talking about pkg-static, isn't that something that should be preferably used on upgrades in general? The flood of console hints about using it on upgrading from 2.3.x would suggest it to be the case.

It depends on the situation and circumstances, usually it would be used in a case when there could be a library mismatch (or missing entirely) in something pkg wants, or just to be safe.

Cadish

@jimp:

Try using "pkg-static" instead of pkg, but from the sound of it, a reinstall would be safer, faster, and less problematic.

pkg-static didn't help either. Complete reinstall succeeded. Thanks anyway!