IPSEC failed to run after 2.4.4 upgrade

dtrandov · Nov 7, 2018, 5:04 PM

After upgrade to 2.4.4, IPSEC failed to run. Log file said this:

Nov 7 08:50:15	ipsec_starter	92574	Starting strongSwan 5.7.1 IPsec [starter]...
Nov 7 08:50:15	ipsec_starter	92574	no netkey IPsec stack detected
Nov 7 08:50:15	ipsec_starter	92574	no KLIPS IPsec stack detected
Nov 7 08:50:15	ipsec_starter	92574	no known IPsec stack detected, ignoring!
Nov 7 08:50:15	ipsec_starter	92821	charon has quit: integrity test of libstrongswan failed
Nov 7 08:50:15	ipsec_starter	92821	charon refused to be started
Nov 7 08:50:15	ipsec_starter	92821	ipsec starter stopped

On dashboard IPsec/Overview status shows awaiting connections.
Before upgrade everything was just fine. No configuration changes at all.

Any help will be appreciated.

jimp · Nov 7, 2018, 6:14 PM

The first four lines are normal, but that fifth line might be the issue. It's saying that the library may be corrupted. It's not related to the first three errors.

Does it work after a reboot?

Does pkg check -s strongswan show any errors?

You could force a reinstall of just that with pkg upgrade -f strongswan

dtrandov · Nov 7, 2018, 7:26 PM

I did reboot twice and result is the same. Tomorrow I will check package and reinstall if needed. Thanks.

dtrandov · Nov 8, 2018, 7:33 AM

well, no luck since now.

pkg check -s strongswan:

Checking strongswan:
strongswan-5.7.1: missing file /usr/local/man/man1/pki---acert.1.gz
strongswan-5.7.1: missing file /usr/local/man/man1/pki---dn.1.gz
strongswan-5.7.1: missing file /usr/local/man/man1/pki---gen.1.gz
strongswan-5.7.1: missing file /usr/local/man/man1/pki---issue.1.gz
strongswan-5.7.1: missing file /usr/local/man/man1/pki---keyid.1.gz
strongswan-5.7.1: missing file /usr/local/man/man1/pki---pkcs7.1.gz
strongswan-5.7.1: missing file /usr/local/man/man1/pki---print.1.gz
strongswan-5.7.1: missing file /usr/local/man/man1/pki---pub.1.gz
strongswan-5.7.1: missing file /usr/local/man/man1/pki---req.1.gz
strongswan-5.7.1: missing file /usr/local/man/man1/pki---self.1.gz
strongswan-5.7.1: missing file /usr/local/man/man1/pki---signcrl.1.gz
strongswan-5.7.1: missing file /usr/local/man/man1/pki---verify.1.gz
strongswan-5.7.1: missing file /usr/local/man/man1/pki.1.gz
strongswan-5.7.1: missing file /usr/local/man/man5/ipsec.conf.5.gz
strongswan-5.7.1: missing file /usr/local/man/man5/ipsec.secrets.5.gz
strongswan-5.7.1: missing file /usr/local/man/man5/strongswan.conf.5.gz
strongswan-5.7.1: missing file /usr/local/man/man5/swanctl.conf.5.gz
strongswan-5.7.1: missing file /usr/local/man/man8/charon-cmd.8.gz
strongswan-5.7.1: missing file /usr/local/man/man8/ipsec.8.gz
strongswan-5.7.1: missing file /usr/local/man/man8/swanctl.8.gz
Checking strongswan... done

then, reinstall and show no errors, but warn me to remove /usr/local/etc/ipsec.conf and /usr/local/etc/strongswan. conf. Reboot didn't help. Ipsec log contains the same errors. Shall I rename these files ?

Files renamed, but after reboot system created again ....

jimp · Nov 8, 2018, 1:27 PM

The man pages being missing is fine. The warnings about /usr/local/etc/ipsec.conf and strongswan.conf are expected as well, that's all normal.

Try this next:

pkg delete -f strongswan
pkg clean -ay
pkg-static install -fy strongswan

You might also try selecting the reboot option at the ssh or console menu and then choosing the option to force a disk check.

dtrandov · Nov 8, 2018, 1:51 PM

pkg went well, reboot via ssh and fsck gave this:

** /dev/ufsid/58e37092fef9beb3 (NO WRITE)

USE JOURNAL? no

** Skipping journal, falling through to full fsck

SETTING DIRTY FLAG IN READ_ONLY MODE

UNEXPECTED SOFT UPDATE INCONSISTENCY
** Last Mounted on /
** Root file system
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
UNREF FILE I=1043331  OWNER=root MODE=100666
SIZE=0 MTIME=Nov  8 15:41 2018
CLEAR? no

** Phase 5 - Check Cyl groups
SUMMARY BLK COUNT(S) WRONG IN SUPERBLK
SALVAGE? no

18226 files, 198717 used, 7413296 free (2128 frags, 926396 blocks, 0.0% fragmentation)

jimp · Nov 8, 2018, 1:56 PM

Did you run that manually? If you, you need to run fsck -y / a few more times, until it doesn't find or fix any problems.

If you ran the automatic disk check then it should have done 5 runs of it which should hopefully have been sufficient.

Any change in behavior?

dtrandov · Nov 8, 2018, 3:13 PM

No. Status is the same. Log errors are the same. Let me run fsck

jimp · Nov 8, 2018, 3:17 PM

What kind of hardware is this on?

dtrandov · Nov 8, 2018, 3:24 PM

HyperV. fsck -y / in single user mode didn't help. Hardware is IBM server brought 5-6 years ago.

jimp · Nov 8, 2018, 3:51 PM

The only references I can find to the error were from a 5 year old MIPS bug report in strongSwan about not wiping secure memory as expected.

That would seem to imply that it's having an issue with manipulating memory in some way, which doesn't sound good. Though I can't find any recent reference to say for sure.

Can you maybe try provisioning a new VM to see if the same thing happens there? Or trying on a different Hyper-V host if you have one?

Or snapshot the VM and upgrade to 2.4.5 and see if the problem persists there.

dtrandov · Nov 8, 2018, 6:18 PM

I also found this memory issue and probably this is the problem - dashboard shows 31% memory usage from 1G, without any active connection to VM.

I can provision new VM to test, also can move VM to another host.
Finally, I can't find a way to upgrade to 2.4.5. From dashboard system is up to date (2.4.4). The only available options is to switch to Latest development snapshots Experimental 2.4.X devel. Any hint how to upgrade ?

jimp · Nov 8, 2018, 6:32 PM

The development snapshots choice should get you there, 2.4.5 is under development, not released or stable.

dtrandov · Nov 9, 2018, 11:56 AM

Neither upgrade to 2.4.5 or relocation of VM resolved issue. I'm going to provision new VM, upload current configuration and see what happened. Thanks a lot for your assistance

nledoux · Nov 9, 2018, 3:44 PM

Hello.
Exact same problem here. IPSec refused to start after upgrade from 2.4.3-p1 to 2.4.4 with identical error messages. It was working fine just before. Tried the above step (reinstalling strongswan) without success. pfSense is also visualized but on a different platform (VMware 6.5 / Dell PowerEdge R630) so it might not be a virtualization specific issue. Reverted to snapshot taken just before the upgrade and everything is working fine under 2.4.3-p1. Retried the update with same result (IPSEC not starting) so this is not a upgrade glitch but a reproducible issue. Went back to 2.4.3-p1 snapshot again as it is a production firewall. Any idea what else to test/try?

jimp · Nov 9, 2018, 4:16 PM

It's working OK for me here on ESX 6.7, so it's not likely to be specific to virtualization in general, but maybe something in your environment or configuration.

Can you share the contents of /var/etc/ipsec/strongswan.conf? If there is anything private in there, you can mask/redact it.

There is a way to disable the integrity tests but I'd rather find out why it's failing first.

The way those tests are described it's a simple file checksum test, but if that was the case it should be happening for everyone consistently or flagged by pkg check -s strongswan.

jimp · Nov 9, 2018, 4:48 PM

To anyone that can still reproduce this:

Go to VPN > IPsec, Advanced tab.
Under IPsec Logging Controls set strongSwan Lib to Highest, then Save
Try to restart IPsec
look in Status > System Logs, IPsec tab for a message about why it failed. Alternately, check clog /var/log/ipsec.log from the shell.

The strongSwan source seems to imply that it could be a file/filesystem issue. The checksum is missing, the file size is wrong, or the checksum doesn't match. It could also be that somehow it can't find the library (Maybe run ldconfig in the shell and then try starting it again).

The debug logs will hopefully tell us more.

nledoux · Nov 9, 2018, 4:58 PM

Here is the strongswan.conf (0_1541782634617_strongswan.conf.txt) from our running 2.4.3 that fails after upgrade.
I cannot reproduce now but will try the above of hours if nobody was able to provide before.

dtrandov · Nov 9, 2018, 5:34 PM

One thing I did when upgrading was to refresh dashboard, because i thought that upgrade process freeze. Probably related with issue.

jimp · Nov 9, 2018, 5:42 PM

@dtrandov said in IPSEC failed to run after 2.4.4 upgrade:

One thing I did when upgrading was to refresh dashboard, because i thought that upgrade process freeze. Probably related with issue.

Probably not if the same thing happened after updating to 2.4.5.

Can you try the log changes I posted about a few replies up?