pfSense 21.05.1 upgrade failed

pfsjap

Tried to upgrade SG-1100 from 21.05-RELEASE to 21.05.1, but it did not finish successfully.

Log from the GUI -> Gui.txt

Then with shell:

[21.05-RELEASE][admin@pfSense.localdomain]/root: pfSense-upgrade -d -c
>>> Updating repositories metadata...
Updating pfSense-core repository catalogue...
1082880000:error:141F0006:SSL routines:tls_construct_cert_verify:EVP lib:/usr/local/poudriere/jails/pfSense_plus-v21_05_aarch64/usr/src/crypto/openssl/ssl/statem/statem_lib.c:283:
Certificate verification failed for /C=US/ST=Texas/L=Austin/O=Rubicon Communications, LLC (Netgate)/CN=repo00.netgate.com
1082880000:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/local/poudriere/jails/pfSense_plus-v21_05_aarch64/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Child process pid=3229 terminated abnormally: Segmentation fault
21.05.1 version of pfSense is available
[21.05-RELEASE][admin@pfSense.localdomain]/root:

Then with shell:

[21.05-RELEASE][admin@pfSense.localdomain]/root: pkg-static clean -ay; pkg-static install -fy pkg pfSense-repo pfSense-upgrade
pkg-static: Repository pfSense missing. 'pkg update' required
pkg-static: No package database installed.  Nothing to do!
Updating pfSense-core repository catalogue...
pkg-static: Repository pfSense-core has a wrong packagesite, need to re-create database
1082880000:error:141F0006:SSL routines:tls_construct_cert_verify:EVP lib:/usr/local/poudriere/jails/pfSense_plus-v21_05_aarch64/usr/src/crypto/openssl/ssl/statem/statem_lib.c:283:
Certificate verification failed for /C=US/ST=Texas/L=Austin/O=Rubicon Communications, LLC (Netgate)/CN=repo01.netgate.com
1082880000:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:/usr/local/poudriere/jails/pfSense_plus-v21_05_aarch64/usr/src/crypto/openssl/ssl/statem/statem_clnt.c:1915:
Child process pid=12477 terminated abnormally: Segmentation fault
[21.05-RELEASE][admin@pfSense.localdomain]/root:

And finally with shell:

[21.05-RELEASE][admin@pfSense.localdomain]/root: pkg-static bootstrap -f
pkg(8) is already installed. Forcing reinstallation through pkg(7).
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from pkg+https://repo.netgate.com/pkg/pfSense_plus-v21_05_aarch64-pfSense_plus-v21_05, please wait...
1076269056:error:141F0006:SSL routines:tls_construct_cert_verify:EVP lib:/build/plus-crossbuild-2105-aarch64/sources/FreeBSD-src/crypto/openssl/ssl/statem/statem_lib.c:283:
Child process pid=38095 terminated abnormally: Segmentation fault
[21.05-RELEASE][admin@pfSense.localdomain]/root:

Now the device is unable to check updates. What happened and how can I resolve the issue?

stephenw10

The first thing to try there is a complete power cycle. That looks like the known issue with the crypto hardware on that device. It is _not_resolved by simply rebooting, a full power cycle is required.
However that should be fixed in 21.05 is it possible that has not been power cycled since it was running 2.4.5?

Steve

pfsjap

@stephenw10 Device with 21.05 installed has been power cycled earlier at least one time, but not just before trying this upgrade.

Is there a chance, the device will not start up after power cycle?

pfsjap

@stephenw10
After power cycle dashboard displays:

21.05-RELEASE (arm64)
built on Tue Jun 01 16:52:48 EDT 2021
FreeBSD 12.2-STABLE

The system is on the latest version.

Gertjan

@pfsjap said in pfSense 21.05.1 upgrade failed:

After power cycle

A real power cycle ?
If possible, use the GUI or the console access to shut down the device.
Removing the power like ripping out the power cable is the perfect way to 'break' (not booting) the device. This method should be sued as a last resort.
Apply the same behaviour as you would do with PC or a server, and you'll be fine.

Btw : a clean reboot before a pfSEnse upgrade - using the console or GUI is always advisable. It will take care of any stale issues that might influence the upgrade process.
Because it's the SG-1100 - a power down -> 30 seconds grace period -> power up is also a good thing (see earlier forum message about this 'issue').

Also : these (my) rules are not reserved of Netgate devices. I would the same thing with my dishwasher and coffee maker (and AP, and PCs and servers)

pfsjap

@stephenw10 Power cycle did it, now on 21.05.1. Had to issue command "pfSense-upgrade -d -c" in shell to get the update offred.

Thank you!

pfsjap

@gertjan A real power cycle of course, after halting the system.

stephenw10

Ah, nice.

So to be clear you were still hitting this in 21.05 after having done at least one full power cycle?

Might have to revisit that issue if so. We were unable to trigger it in 21.0X after adding the driver fix.

Steve

pfsjap

@pfsjap said in pfSense 21.05.1 upgrade failed:
I've only had SG-1100 for a few weeks and it's my first Netgate device, so my experience is very limited, but it definitely was unpowered in July while I was away from home.

mer

@pfsjap I had a related/similar issue on a SG2100. All of a sudden started reporting issues with trying to determine if there was an upgrade available.
A hard power cycle (serial console in, shutdown, wait for it to complete, then pull power cable for 30 secs) corrected it.
I'm guessing that something gets crypto hardware wedged so new operations always fail and a reboot does not reset the device state, but a good power cycle does.

After doing that and the upgrade I've not seen the issue. Often times, issues like this are "timing" issues/race conditions and are hard to reproduce.

soulc420

I tried to upgrade a sg-2100 through the GUI. It failed and when I tried to access the console this is what I got.

FreeBSD/arm64 (Amnesiac) (ttyu0)


Warning: PHP Startup: Unable to load dynamic library 'pfSense.so' (tried: /usr/local/lib/php/20190902/pfSense.so (Shared object "libvici.so.0" not found, required by "pfSense.so"), /usr/local/lib/php/20190902/pfSense.so.so (/usr/local/lib/php/20190902/pfSense.so.so: invalid file format)) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'intl.so' (tried: /usr/local/lib/php/20190902/intl.so (/usr/local/lib/libicuio.so.69: invalid file format), /usr/local/lib/php/20190902/intl.so.so (/usr/local/lib/php/20190902/intl.so.so: invalid file format)) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'pfSense.so' (tried: /usr/local/lib/php/20190902/pfSense.so (Shared object "libvici.so.0" not found, required by "pfSense.so"), /usr/local/lib/php/20190902/pfSense.so.so (/usr/local/lib/php/20190902/pfSense.so.so: invalid file format)) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'sysvmsg.so' (tried: /usr/local/lib/php/20190902/sysvmsg.so (/usr/local/lib/php/20190902/sysvmsg.so: invalid file format), /usr/local/lib/php/20190902/sysvmsg.so.so (/usr/local/lib/php/20190902/sysvmsg.so.so: invalid file format)) in Unknown on line 0

Warning: PHP Startup: Unable to load dynamic library 'xmlreader.so' (tried: /usr/local/lib/php/20190902/xmlreader.so (/usr/local/lib/php/20190902/xmlreader.so: Undefined symbol "dom_node_class_entry"), /usr/local/lib/php/20190902/xmlreader.so.so (/usr/local/lib/php/20190902/xmlreader.so.so: invalid file format)) in Unknown on line 0

Warning: Cannot load module 'pdo_sqlite' because required module 'pdo' is not loaded in Unknown on line 0

0) Logout (SSH only)                  9) pfTop
1) Assign Interfaces                 10) Filter Logs
2) Set interface(s) IP address       11) Restart webConfigurator
3) Reset webConfigurator password    12) PHP shell + Netgate pfSense Plus tools
4) Reset to factory defaults         13) Update from console
5) Reboot system                     14) Enable Secure Shell (sshd)
6) Halt system                       15) Restore recent configuration
7) Ping host                         16) Restart PHP-FPM
8) Shell

Enter an option: 1

any selection gives differing errors all missing things. So I just want to reload the damn os and pfsense But I cant find the old file. This happened in june as well. Then I opened a ticket with netgate and they emailed me instructions and a file to flash and re configuring and was back up and running.

mer

@soulc420 I'm just another user of pfSense, and I know people have had different results in trying to upgrade. That output looks like a partial upgrade to me. Why? I have no idea.
But I do know that netgate support is responsive to letting you know where to get images for reinstall.
Reinstalling is a pain, but sometimes it's the cleanest way to do things.

My configuration is relatively simple: a couple of VLAN tags on the switched ports so I can mimic LAN, OPT1, OPT2, then just normal NAT. I'm not running any extra packages, most I have is DHCP with some static mappings.

Sometimes before upgrading, the best thing to do is save your current config, reset to factory defaults, then upgrade, then restore your config.
Consumer products like this are difficult to run QA on because you never know what someone will do. There are too many different combinations of packages and config items that test cycles would be years instead of months.

stephenw10

@soulc420 That's an unrelated failure.
It looks like it didn't complete the upgrade. It's looking for libs that are mismatched to the pfSense version.
You may be able to recover by forcing a full pkg re-install but I would just re-install the firmware clean from there to be sure. It will be quicker anyway.
Open a ticket with us: https://go.netgate.com/

Steve

soulc420

@stephenw10 Thanks this was what I was looking for.