2.5.2 to 2.6.0 upgrade fails with [pkg] Child process pid=N terminated abnormally: Segmentation fault

jimp

It could happen for similar reasons as filesystem corruption, or as a result of previous filesystem corruption. It's rare but not unheard of.

mdomnis

@swix I had this problem when I tried going from 2.5.2 to 2.6 RC and after removing the Zabbix Agent package, I was able to get the upgrade to go through. I assumed it was because Zabbix wasn't in the repo yet or something. You could try removing that one and retrying. Maybe there's still something going on there. I haven't tried it since a couple weeks back.

swix

Thanks for the suggestion @mdomnis, it seems it did the trick!

After the fsck (which went fine as far as I can see, pfsense box went back online about 10 min after), I uninstalled the zabbix_agent packages:

pfSense-pkg-zabbix-agent52-1.0.4_11 pfSense package zabbix-agent
zabbix52-agent-5.2.6           Enterprise-class open source distributed monitoring (agent)

And the upgrade process did not segfault again, at least until now. It's still in progress, but it looks much better now.

On the System/ Package Manager / Installed Packages (pkg_mgr_installed.php) page after switching to the 2.6.0 branch (via Update Settings), the zabbix_agent52 package was marked red as "deprecated".

So in this case, it's good to know, that this kind of deprecated package will/may prevent an upgrade and generate a quite cryptical Segmentation Fault @jimp.

Upgrade still in progress, let's see how it looks like in 30-60 minutes...

swix

All good! Upgrade fine as usual, so noted for next time: make sure outdated packages are removed + reboot with fsck.

*** Welcome to pfSense 2.6.0-RELEASE (amd64) on gw ***

Thanks again @jimp and @mdomnis & best regards.

SteveITS

@swix said in 2.5.2 to 2.6.0 upgrade fails with [pkg] Child process pid=N terminated abnormally: Segmentation fault:

this kind of deprecated package will/may prevent an upgrade and generate a quite cryptical Segmentation Fault

Just FYI per https://forum.netgate.com/topic/169845/segmentation-fault-when-attempting-to-upgrade-2-5-2-2-6-0/11, "The expected behaviour is that the package would simply be removed at upgrade. But that is not the case currently."

Also worth noting, per https://docs.netgate.com/pfsense/en/latest/install/upgrade-guide-prepare.html#packages "The safest practice is to remove all packages before upgrading to a new release. The upgrade process will handle packages automatically, but packages are frequently a source of problems. To ensure a smooth upgrade, note the installed packages, remove them, perform the upgrade, and then reinstall when the upgrade is complete." Uninstalling first would also let the admin see the package has been replaced/deprecated, from the installed packages page.

swix

@steveits yes sure, you're 100% right. It's just the first time I see such an issue (in ~10 years), so I was still a bit surprised.

Maybe there could be a few simple steps during the upgrade process that would make sure this kind of error (Segmentation fault) does not happen, or that it's a least trapped and replaced by a more readable message like "something went wrong with pkg, RTFM/remove packages".

My next step would have been to remove the packages one by one, but @mdomnis suggested it even before I could think about it :)

stephenw10

For reference: https://redmine.pfsense.org/issues/12796

That should not happen. The patch linked there could resolve it shortly.

Steve

iJay-XTM5

Hello, I am also experiencing a segmentation fault with a similar message but do not have the zabbix package installed. This is a Watchguard M400 box with an upgraded i5-4590T CPU. Previous upgrades went through without a hitch so I'm a bit perplexed with the current failure.
The GUI just reports Update failed but the consoles, both ssh and local indicate the following:
Child process pid=***** terminated abnormally: Segmentation fault

I've tried following directions from this post: [https://forum.netgate.com/topic/128618/broken-updates-upgrades-packages-segmentation-faults-etc]
and get the following errors (output from each package indented for clarity):
pkg-static update -f
(...)
Newer FreeBSD version for package pfSense-pkg-node_exporter:
To ignore this error set IGNORE_OSVERSION=yes

package: 1203500
running kernel: 1202504
Ignore the mismatch and continue? [y/N]: y
Processing entries: 100%
pfSense repository update completed. 511 packages processed.
All repositories are up to date.

pkg-static upgrade -f
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
Checking for upgrades (236 candidates): 14%
(null) has no direct installation candidates, change it to python38? [Y/n]: Y
Child process pid=65740 terminated abnormally: Segmentation fault

It appears the packages are corruputed, and I'm not how to proceed at this time. Could somebody with more experience with troubleshooting please provide me with some directions and guidance?
Many Thanks for your help in advance!

iJay-XTM5

The following packages are installed on the machine:
ntopng v0.8.13_10
pfBlockerNG-devel v3.1.0_1
snort v4.1.5
System_Patches v1.2_7

stephenw10

As is always the case the safest way to upgrade is to remove all the packages, upgrade and then reinstall the packages. So I would try that first.

iJay-XTM5

Thanks for the suggestion Stephen, I uninstalled all packages but still have the segmentation error:

Child process pid=42102 terminated abnormally: Segmentation fault

Any other suggestions?
Thanks for your help!

stephenw10

Hmm, try running pkg-static -d update to get debug output.

iJay-XTM5

I was able to successfully update and upgrade pkg-static from the command line as well as upgrade to pfsense 2.6.0. Unfortunately, I forgot to capture the output from the pkg-static -d update but it completed successfully. I subsequently also tried pkg-static -d upgrade which resulted in a large number of upgrades. The -d update and -d upgrade commands appear to have done the trick.
Many thanks for your help!

stephenw10

The -d switch there only enables debugging output. It shouldn't make any difference in itself. But I'm glad you were able to get past that anyway.

Steve