AWS Appliance upgrade from 23.01 to 23.05 failed

[NollipfSense]

@nuclearstrength said in AWS Appliance upgrade from 23.01 to 23.05 failed:

Child p Irocess pid=17164 terminated abnormally: Killed

pid-17164...have you tried rebooting after receiving that notification? I haven't looked up what process that it though.

[nuclearstrength]

@NollipfSense Yes I have, it shows I'm on 23.01 and the upgrade is needed but then If i try to upgrade I get this:

>>> Updating repositories metadata... 
Updating pfSense-core repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: . done
Processing entries: .. done
pfSense-core repository update completed. 15 packages processed.
Updating pfSense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: .......... done
Processing entries: 
Processing entries............. done
pfSense repository update completed. 549 packages processed.
All repositories are up to date.
>>> Removing vital flag from php81... done.
>>> Downloading upgrade packages... 
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
Checking for upgrades (62 candidates): .......... done
Processing candidates (62 candidates): .......... done
Checking integrity...Child process pid=63046 terminated abnormally: Killed
Failed

[NollipfSense]

@nuclearstrength said in AWS Appliance upgrade from 23.01 to 23.05 failed:

Checking integrity...Child process pid=63046 terminated abnormally: Killed
Failed

I'll link @stephenw10 in hope he may shed more light on that process for you.

[stephenw10]

You have a conflict in the php version there which implies it may have pulled in incorrect modules.

Try running: pkg-static info -x php
Check what modules you have installed now.

Steve

[nuclearstrength]

@stephenw10 said in AWS Appliance upgrade from 23.01 to 23.05 failed:

pkg-static info -x php

sorry for the late reply,

it seems to me that you are correct and somehow during the initial update process something failed and it pulled and installed some packages beloging to 23.05 but didn't finish with the upgrade itself, but I don't have much of an understanding of the bsd/pfsense package manager.

I also contacted the official support (under the basic plan that comes with the aws paid instance) and ultimately I was told that "follow the troubleshooting in the documentation and you can reinstall", that became a problem with the PM on the project.

my concern as a technician is that at the moment the system is working but there is no upgrade path and the only option in the troubleshooting documentation that seems to apply to this case is to do a forceful reinstall, but without guidance from official support or the forums I would have to take responsibility for the downtime if it doesn't work and we have to rebuild the system. which might require extensive downtime.

the system is due to move to production within weeks and integration tests are already under way, other systems have already been configured to use this infra, either we pull the plug on pfsense and go for another solution (paloalto own offering on aws has been floated around) or we solve this issue and keep going with pfsense+.

personally I would love to keep going with pfsense, I like the product and used it very successfully on other projects, but I'm no expert on the underlying bsd and the specific packages system for pfsense so I'm at a loss here as how to solve this, reinstall would mean pull the plug because it wouldn't fly with the PM given that was the only real option produced by the basic support (FYI an answer like "we can solve this under a different, more expensive, support plan" would have been accepted by the PM in question and in my experience other businesses as well).

the output of the command follows:

aws-sdk-php81-3.232.3
php81-8.1.11
php81-bcmath-8.1.11
php81-bz2-8.1.11
php81-ctype-8.1.11
php81-curl-8.1.11
php81-dom-8.1.11
php81-filter-8.1.11
php81-gettext-8.1.11
php81-intl-8.1.11_1
php81-ldap-8.1.11
php81-libbe-0.1.4.1
php81-mbstring-8.1.11
php81-opcache-8.1.11
php81-openssl_x509_crl-1.3_1
php81-pcntl-8.1.11
php81-pdo-8.1.11
php81-pdo_sqlite-8.1.11
php81-pear-1.10.13
php81-pear-Auth_RADIUS-1.1.0_4
php81-pear-Cache_Lite-1.8.3,1
php81-pear-Crypt_CHAP-1.5.0_2
php81-pear-HTTP_Request2-2.5.1,1
php81-pear-Mail-1.4.1,1
php81-pear-Net_IPv6-1.3.0.b4_2
php81-pear-Net_SMTP-1.10.1
php81-pear-Net_Socket-1.2.2
php81-pear-Net_URL2-2.2.1
php81-pear-XML_RPC2-1.1.5
php81-pecl-mcrypt-1.0.5
php81-pecl-radius-1.4.0b1_2
php81-pecl-rrd-2.0.3
php81-pfSense-module-0.91
php81-phpseclib-2.0.17
php81-posix-8.1.11
php81-readline-8.1.11
php81-session-8.1.11
php81-shmop-8.1.11
php81-simplexml-8.1.11
php81-sockets-8.1.11
php81-sqlite3-8.1.11
php81-sysvmsg-8.1.11
php81-sysvsem-8.1.11
php81-sysvshm-8.1.11
php81-tokenizer-8.1.11
php81-xml-8.1.11
php81-xmlreader-8.1.11
php81-xmlwriter-8.1.11
php81-zlib-8.1.11

[stephenw10]

Hmm, that looks OK, all php81 modules.

Try running pkg-static info -x pfsense to check you have the expected 23.01 pkgs still.

[nuclearstrength]

@stephenw10

pfSense-23.01
pfSense-Status_Monitoring-1.8
pfSense-base-23.01
pfSense-boot-23.01
pfSense-default-config-ec2-23.01
pfSense-ec2_setup-0.10
pfSense-kernel-pfSense-23.01
pfSense-openvpn_connect_profile-0.2_1
pfSense-pkg-Netgate_Firmware_Upgrade-0.54
pfSense-pkg-System_Patches-2.2.2
pfSense-pkg-aws-wizard-0.10
pfSense-pkg-haproxy-0.61_10
pfSense-pkg-ipsec-profile-wizard-1.1_1
pfSense-pkg-openvpn-client-export-1.8
pfSense-pkg-sudo-0.3_8
pfSense-rc-23.01
pfSense-repo-23.01
pfSense-repoc-20230605
pfSense-upgrade-1.0_67
php81-pfSense-module-0.91

[stephenw10]

Hmm, it's pulled in the updated repoc and upgrade pkgs. Those are the correct versions for upgrading to 23.05.

I assume its still failing the same way if you just try to upgrade?

With the branch set to 23.01 you should be able to install HAProxy if you need to though.

Steve

[nuclearstrength]

@stephenw10 said in AWS Appliance upgrade from 23.01 to 23.05 failed:

With the branch set to 23.01 you should be able to install HAProxy if you need to though.

yes, that was done and haproxy it's installed, the problem is this system is in a non-upgreadable state and this infra should stay on for the next 5 years, so upgrades might be needed.

I assume its still failing the same way if you just try to upgrade?

now I'm back to this error:
edit: it says that only 6KiB needs to be downloaded, so it already downloaded the packages it needs to upgrade, then it fails an integrity check, are those packages that have been downloaded corrupted? should it reattemp to download them instead of failing if that's the case?
how can I force it to re-download the packages?

>>> Updating repositories metadata... 
Updating pfSense-core repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: . done
Processing entries: .. done
pfSense-core repository update completed. 15 packages processed.
Updating pfSense repository catalogue...
Fetching meta.conf: . done
Fetching packagesite.pkg: .......... done
Processing entries: 
Processing entries............. done
pfSense repository update completed. 549 packages processed.
All repositories are up to date.
>>> Removing vital flag from php81... done.
>>> Downloading upgrade packages... 
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
Checking for upgrades (63 candidates): .......... done
Processing candidates (63 candidates): .......... done
The following 114 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
	aws-sdk-php82: 3.232.3 [pfSense]
	pfSense-Status_Monitoring-php82: 1.8_3 [pfSense]
	php82: 8.2.4 [pfSense]
	php82-bcmath: 8.2.4 [pfSense]
	php82-bz2: 8.2.4 [pfSense]
	php82-ctype: 8.2.4 [pfSense]
	php82-curl: 8.2.4 [pfSense]
	php82-dom: 8.2.4 [pfSense]
	php82-filter: 8.2.4 [pfSense]
	php82-gettext: 8.2.4 [pfSense]
	php82-intl: 8.2.4_1 [pfSense]
	php82-ldap: 8.2.4 [pfSense]
	php82-libbe: 0.1.4.1 [pfSense]
	php82-mbstring: 8.2.4 [pfSense]
	php82-opcache: 8.2.4 [pfSense]
	php82-openssl_x509_crl: 1.3_2 [pfSense]
	php82-pcntl: 8.2.4 [pfSense]
	php82-pdo: 8.2.4 [pfSense]
	php82-pdo_sqlite: 8.2.4 [pfSense]
	php82-pear: 1.10.13 [pfSense]
	php82-pear-Auth_RADIUS: 1.1.0_4 [pfSense]
	php82-pear-Cache_Lite: 1.8.3,1 [pfSense]
	php82-pear-Crypt_CHAP: 1.5.0_2 [pfSense]
	php82-pear-HTTP_Request2: 2.5.1,1 [pfSense]
	php82-pear-Mail: 1.4.1,1 [pfSense]
	php82-pear-Net_IPv6: 1.3.0.b4_2 [pfSense]
	php82-pear-Net_SMTP: 1.10.1 [pfSense]
	php82-pear-Net_Socket: 1.2.2 [pfSense]
	php82-pear-Net_URL2: 2.2.1 [pfSense]
	php82-pear-XML_RPC2: 1.1.5 [pfSense]
	php82-pecl-mcrypt: 1.0.6 [pfSense]
	php82-pecl-radius: 1.4.0b1_2 [pfSense]
	php82-pecl-rrd: 2.0.3 [pfSense]
	php82-pfSense-module: 0.95 [pfSense]
	php82-phpseclib: 2.0.17 [pfSense]
	php82-posix: 8.2.4 [pfSense]
	php82-readline: 8.2.4 [pfSense]
	php82-session: 8.2.4 [pfSense]
	php82-shmop: 8.2.4 [pfSense]
	php82-simplexml: 8.2.4 [pfSense]
	php82-sockets: 8.2.4 [pfSense]
	php82-sqlite3: 8.2.4 [pfSense]
	php82-sysvmsg: 8.2.4 [pfSense]
	php82-sysvsem: 8.2.4 [pfSense]
	php82-sysvshm: 8.2.4 [pfSense]
	php82-tokenizer: 8.2.4 [pfSense]
	php82-xml: 8.2.4 [pfSense]
	php82-xmlreader: 8.2.4 [pfSense]
	php82-xmlwriter: 8.2.4 [pfSense]
	php82-zlib: 8.2.4 [pfSense]
	pkcs11-helper: 1.29.0 [pfSense]

Installed packages to be UPGRADED:
	7-zip: 21.07_2 -> 22.01 [pfSense]
	bind-tools: 9.18.8 -> 9.18.13 [pfSense]
	ca_root_nss: 3.83 -> 3.89 [pfSense]
	ccid: 1.5.0 -> 1.5.1 [pfSense]
	check_reload_status: 0.0.14 -> 0.0.15 [pfSense]
	curl: 7.85.0 -> 8.0.1 [pfSense]
	dbus: 1.14.4,1 -> 1.14.6,1 [pfSense]
	devcpu-data: 20221002 -> 20230219 [pfSense]
	devcpu-data-amd: 20221002 -> 20230219 [pfSense]
	devcpu-data-intel: 20220809 -> 20230214 [pfSense]
	dmidecode: 3.4_2 -> 3.5 [pfSense]
	dnsmasq: 2.87,1 -> 2.89_1,1 [pfSense]
	dpinger: 3.2 -> 3.3 [pfSense]
	drm-510-kmod: 5.10.113_7 -> 5.10.163_5 [pfSense]
	e2fsprogs-libuuid: 1.46.5_1 -> 1.47.0 [pfSense]
	expat: 2.4.9 -> 2.5.0 [pfSense]
	flashrom: 1.2 -> 1.2.1 [pfSense]
	gettext-runtime: 0.21 -> 0.21.1 [pfSense]
	glib: 2.74.0,2 -> 2.76.1,2 [pfSense]
	haproxy22: 2.2.25 -> 2.2.29 [pfSense]
	igmpproxy: 0.3,1 -> 0.3_1,1 [pfSense]
	libedit: 3.1.20210910,1 -> 3.1.20221030,1 [pfSense]
	libffi: 3.4.3 -> 3.4.4 [pfSense]
	libgpg-error: 1.45 -> 1.46 [pfSense]
	libidn2: 2.3.3_1 -> 2.3.4 [pfSense]
	libnghttp2: 1.48.0 -> 1.52.0 [pfSense]
	libpci: 3.8.0 -> 3.9.0 [pfSense]
	libpsl: 0.21.1_6 -> 0.21.2_2 [pfSense]
	libssh2: 1.10.0,3 -> 1.10.0_1,3 [pfSense]
	libucl: 0.8.1 -> 0.8.2 [pfSense]
	lua-resty-core: 0.1.23 -> 0.1.24 [pfSense]
	luajit-openresty: 2.1.20220915 -> 2.1.20230119 [pfSense]
	mobile-broadband-provider-info: 20220725 -> 20221107 [pfSense]
	mpd5: 5.9_12 -> 5.9_14 [pfSense]
	nginx: 1.22.0_9,3 -> 1.22.1_5,3 [pfSense]
	openldap26-client: 2.6.3 -> 2.6.4 [pfSense]
	opensc: 0.22.0 -> 0.23.0_1 [pfSense]
	openvpn: 2.6.0_13 -> 2.6.2 [pfSense]
	pciids: 20220909 -> 20230223 [pfSense]
	pcre: 8.45_2 -> 8.45_3 [pfSense]
	pcre2: 10.40 -> 10.42 [pfSense]
	pcsc-lite: 1.9.5,2 -> 1.9.9,2 [pfSense]
	pfSense: 23.01 -> 23.05 [pfSense]
	pfSense-base: 23.01 -> 23.05 [pfSense-core]
	pfSense-boot: 23.01 -> 23.05 [pfSense-core]
	pfSense-default-config-ec2: 23.01 -> 23.05 [pfSense-core]
	pfSense-kernel-pfSense: 23.01 -> 23.05 [pfSense-core]
	pfSense-pkg-Netgate_Firmware_Upgrade: 0.54 -> 23.05.00 [pfSense]
	pfSense-pkg-System_Patches: 2.2.2 -> 2.2.4 [pfSense]
	pfSense-rc: 23.01 -> 23.05 [pfSense-core]
	pfSense-repo: 23.01 -> 23.05 [pfSense]
	pftop: 0.8 -> 0.8_2 [pfSense]
	py39-libzfs: 1.1.2022081600 -> 1.1.2023020700 [pfSense]
	python311: 3.11.1_1 -> 3.11.2_2 [pfSense]
	python39: 3.9.16 -> 3.9.16_2 [pfSense]
	readline: 8.1.2 -> 8.2.1 [pfSense]
	rrdtool: 1.8.0_1 -> 1.8.0_2 [pfSense]
	sqlite3: 3.39.3_1,1 -> 3.41.0_1,1 [pfSense]
	strongswan: 5.9.8 -> 5.9.10_2 [pfSense]
	sudo: 1.9.12 -> 1.9.13p3 [pfSense]
	unbound: 1.17.1_2 -> 1.17.1_3 [pfSense]
	zstd: 1.5.2_1 -> 1.5.4_1 [pfSense]

Installed packages to be REINSTALLED:
	pfSense-pkg-aws-wizard-0.10 [pfSense] (direct dependency changed: php82)

Number of packages to be installed: 51
Number of packages to be upgraded: 62
Number of packages to be reinstalled: 1

The process will require 119 MiB more space.
6 KiB to be downloaded.
[1/1] Fetching pfSense-23.05.pkg: . done
Checking integrity...Child process pid=30119 terminated abnormally: Killed
Failed

[stephenw10]

Hmm, maybe it somehow pulled a bad package or it got corrupted. Try running: pkg-static clean -ay

[stephenw10]

The error it shows:

Checking integrity...Child process pid=30119 terminated abnormally: Killed
Failed

Should have logged something that might indicate the cause.
Check the output of dmesg or the system log after triggering it.