2.5.2 to 2.6.0 upgrade fails with [pkg] Child process pid=N terminated abnormally: Segmentation fault

swix

Hi everybody,

APU1d (pc engines) board with AMD G-T40E Processor + 4GB RAM running pfSense CE 2.5.2, SSD drive with 2GB free (out of 8GB). Just a small home firewall with only openvpn, darkstat and zabbix_agent running as extra services.

Upgrade log:

>>> Updating repositories metadata... 
Updating pfSense-core repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: . done
Processing entries: . done
pfSense-core repository update completed. 7 packages processed.
Updating pfSense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: .......... done
Processing entries: 
Processing entries............. done
pfSense repository update completed. 511 packages processed.
All repositories are up to date.
>>> Removing vital flag from php74... done.
>>> Downloading upgrade packages... 
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
Checking for upgrades (211 candidates): ....
Child process pid=84482 terminated abnormally: Segmentation fault
Failed

dmesg:

pid 84482 (pkg-static), jid 0, uid 0: exited on signal 11 (core dumped)

Command running when it failed:

root    84482   9.1  0.7  28048 27156  -  D    16:38    0:01.16 pkg-static -o EVENT_PIPE=/tmp/pfSense-upgrade.sock upgrade -F -f

I did follow https://docs.netgate.com/pfsense/en/latest/troubleshooting/upgrades.html as much as possible, but also after running "pkg-static clean -ay; pkg-static install -fy pkg pfSense-repo pfSense-upgrade", switching branches to dev and back to stable, reboot/halt, no improvement. Same after stopping all essential services.

All previous upgrades were fine, so I'm wondering what could be the issue here. I'll do an install from scratch + restore if there is no other way, but I would first have to find the corresponding hardware :) I will try a few other things before, but if in the mean time you had something similar, please let me know.

Best regards & thanks for the new release in any case!

swix

PS: the directory /var/crash was empty

jimp

The few times I've seen that happen on CE it's been due to filesystem corruption.

If you are using UFS, reboot and run fsck a few times on the drive. The reboot menu has an option to do that automatically.

It's also possible that the package database itself is corrupted, which is more difficult to fix.

Another possibility, though perhaps less likely, is a hardware problem such as the disk itself being a problem.

The fastest way to resolve it if the fsck doesn't work is to try a clean install (and use ZFS)

swix

Thanks @jimp, I will try the fsck right away. I don't see why the package database should be corrupted, but it is also a possibility yes. To be continued...

jimp

It could happen for similar reasons as filesystem corruption, or as a result of previous filesystem corruption. It's rare but not unheard of.

mdomnis

@swix I had this problem when I tried going from 2.5.2 to 2.6 RC and after removing the Zabbix Agent package, I was able to get the upgrade to go through. I assumed it was because Zabbix wasn't in the repo yet or something. You could try removing that one and retrying. Maybe there's still something going on there. I haven't tried it since a couple weeks back.

swix

Thanks for the suggestion @mdomnis, it seems it did the trick!

After the fsck (which went fine as far as I can see, pfsense box went back online about 10 min after), I uninstalled the zabbix_agent packages:

pfSense-pkg-zabbix-agent52-1.0.4_11 pfSense package zabbix-agent
zabbix52-agent-5.2.6           Enterprise-class open source distributed monitoring (agent)

And the upgrade process did not segfault again, at least until now. It's still in progress, but it looks much better now.

On the System/ Package Manager / Installed Packages (pkg_mgr_installed.php) page after switching to the 2.6.0 branch (via Update Settings), the zabbix_agent52 package was marked red as "deprecated".

So in this case, it's good to know, that this kind of deprecated package will/may prevent an upgrade and generate a quite cryptical Segmentation Fault @jimp.

Upgrade still in progress, let's see how it looks like in 30-60 minutes...

swix

All good! Upgrade fine as usual, so noted for next time: make sure outdated packages are removed + reboot with fsck.

*** Welcome to pfSense 2.6.0-RELEASE (amd64) on gw ***

Thanks again @jimp and @mdomnis & best regards.

SteveITS

@swix said in 2.5.2 to 2.6.0 upgrade fails with [pkg] Child process pid=N terminated abnormally: Segmentation fault:

this kind of deprecated package will/may prevent an upgrade and generate a quite cryptical Segmentation Fault

Just FYI per https://forum.netgate.com/topic/169845/segmentation-fault-when-attempting-to-upgrade-2-5-2-2-6-0/11, "The expected behaviour is that the package would simply be removed at upgrade. But that is not the case currently."

Also worth noting, per https://docs.netgate.com/pfsense/en/latest/install/upgrade-guide-prepare.html#packages "The safest practice is to remove all packages before upgrading to a new release. The upgrade process will handle packages automatically, but packages are frequently a source of problems. To ensure a smooth upgrade, note the installed packages, remove them, perform the upgrade, and then reinstall when the upgrade is complete." Uninstalling first would also let the admin see the package has been replaced/deprecated, from the installed packages page.

swix

@steveits yes sure, you're 100% right. It's just the first time I see such an issue (in ~10 years), so I was still a bit surprised.

Maybe there could be a few simple steps during the upgrade process that would make sure this kind of error (Segmentation fault) does not happen, or that it's a least trapped and replaced by a more readable message like "something went wrong with pkg, RTFM/remove packages".

My next step would have been to remove the packages one by one, but @mdomnis suggested it even before I could think about it :)

stephenw10

For reference: https://redmine.pfsense.org/issues/12796

That should not happen. The patch linked there could resolve it shortly.

Steve

iJay-XTM5

Hello, I am also experiencing a segmentation fault with a similar message but do not have the zabbix package installed. This is a Watchguard M400 box with an upgraded i5-4590T CPU. Previous upgrades went through without a hitch so I'm a bit perplexed with the current failure.
The GUI just reports Update failed but the consoles, both ssh and local indicate the following:
Child process pid=***** terminated abnormally: Segmentation fault

I've tried following directions from this post: [https://forum.netgate.com/topic/128618/broken-updates-upgrades-packages-segmentation-faults-etc]
and get the following errors (output from each package indented for clarity):
pkg-static update -f
(...)
Newer FreeBSD version for package pfSense-pkg-node_exporter:
To ignore this error set IGNORE_OSVERSION=yes

package: 1203500
running kernel: 1202504
Ignore the mismatch and continue? [y/N]: y
Processing entries: 100%
pfSense repository update completed. 511 packages processed.
All repositories are up to date.

pkg-static upgrade -f
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
Checking for upgrades (236 candidates): 14%
(null) has no direct installation candidates, change it to python38? [Y/n]: Y
Child process pid=65740 terminated abnormally: Segmentation fault

It appears the packages are corruputed, and I'm not how to proceed at this time. Could somebody with more experience with troubleshooting please provide me with some directions and guidance?
Many Thanks for your help in advance!

iJay-XTM5

The following packages are installed on the machine:
ntopng v0.8.13_10
pfBlockerNG-devel v3.1.0_1
snort v4.1.5
System_Patches v1.2_7

stephenw10

As is always the case the safest way to upgrade is to remove all the packages, upgrade and then reinstall the packages. So I would try that first.

iJay-XTM5

Thanks for the suggestion Stephen, I uninstalled all packages but still have the segmentation error:

Child process pid=42102 terminated abnormally: Segmentation fault

Any other suggestions?
Thanks for your help!

stephenw10

Hmm, try running pkg-static -d update to get debug output.

iJay-XTM5

I was able to successfully update and upgrade pkg-static from the command line as well as upgrade to pfsense 2.6.0. Unfortunately, I forgot to capture the output from the pkg-static -d update but it completed successfully. I subsequently also tried pkg-static -d upgrade which resulted in a large number of upgrades. The -d update and -d upgrade commands appear to have done the trick.
Many thanks for your help!

stephenw10

The -d switch there only enables debugging output. It shouldn't make any difference in itself. But I'm glad you were able to get past that anyway.

Steve