IPSec Issue After 23.01 Upgrade

[princeton255]

I have recently upgraded my pfSense device on AWS from 22.05 to 23.01. I use this device for Ipsec connections to AWS VPC.

Before:

After:

After the upgrade I noticed 2 out of 5 Ipsec tunnels i have configured on the instance were disabled.

Using the webconfigurator I tried to enable the disabled tunnel configs, the Apply the Ipsec config changes.

However, on clicking apply on the WebUI, it would crash and give below output

On investigating the line on file showing error, below is the code

Below is the crash dump

=====
Crash report begins. Anonymous machine information:

amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT #0 plus-RELENG_23_01-n256037-6e914874a5e: Fri Feb 10 20:30:29 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/obj/amd64/VDZvZksF/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/sources/FreeBS

Crash report details:

PHP Errors:
[26-Feb-2023 09:08:04 Africa/Dar_es_Salaam] PHP Fatal error: Uncaught TypeError: Cannot access offset of type string on string in /etc/inc/ipsec.inc:2546
Stack trace:
#0 /etc/inc/ipsec.inc(3267): ipsec_setup_tunnels(Array)
#1 /usr/local/www/vpn_ipsec.php(49): ipsec_configure()
#2 {main}
thrown in /etc/inc/ipsec.inc on line 2546
[26-Feb-2023 09:09:40 Africa/Dar_es_Salaam] PHP Fatal error: Uncaught TypeError: Cannot access offset of type string on string in /etc/inc/ipsec.inc:2546
Stack trace:
#0 /etc/inc/ipsec.inc(3267): ipsec_setup_tunnels(Array)
#1 /usr/local/www/vpn_ipsec.php(49): ipsec_configure()
#2 {main}
thrown in /etc/inc/ipsec.inc on line 2546
[26-Feb-2023 09:14:07 Africa/Dar_es_Salaam] PHP Fatal error: Uncaught TypeError: Cannot access offset of type string on string in /etc/inc/ipsec.inc:2546
Stack trace:
#0 /etc/inc/ipsec.inc(3267): ipsec_setup_tunnels(Array)
#1 /etc/rc.bootup(401): ipsec_configure()
#2 {main}
thrown in /etc/inc/ipsec.inc on line 2546

No FreeBSD crash data found.

=====

After failing to restart the Ipsec tunnel from the UI, i tried restarting the server from shell, after which, the Ipsec daemon failed to start completely and all Ipsec tunnels went down.

Need help with this

[princeton255]

@princeton255 I finally found the root cause of the issue

The Ipsec tunnels that got disabled and stopped working were using 3DES encryption algorithm for phase1 connection, which was deprecated and no longer supported on version 23.01.

After upgrade thus, these connections were disabled, and the encryption algorthm section of the config removed.

See below empty encryption section.

As compared to a working connection below

My thoughts on this, is that during upgrade, rather than silently drop unsupported config sections, thus causing service interruption to user, it would be better if the upgrade script simply HALTs and return error to user, indicating presence of such unsupported sections for user to fix before upgrade.

[jimp]

Halting in the way you propose isn't possible.

We can't leave those options in place because they are gone from the GUI so opening the page and re-saving would change them in perhaps unexpected ways.

Removing them was the safest option, though apparently there is a bug there in the upgrade code.

[pgb]

I upgraded to 23.01 and run into the same issue.

I understand why the tunnels got disabled, but I still need to edit them in order to change its encryption algorithm to something supported and get them back online.

Is there any way to manually edit the config file to safe values so I can have the webUI working again?

[jimp]

@pgb said in IPSec Issue After 23.01 Upgrade:

I upgraded to 23.01 and run into the same issue.

I understand why the tunnels got disabled, but I still need to edit them in order to change its encryption algorithm to something supported and get them back online.

Is there any way to manually edit the config file to safe values so I can have the webUI working again?

You can remove the empty <encryption></encryption> (or perhaps <encryption/>) tag(s) from your tunnels and then it should let you edit them again.

Alternately you can copy/paste a good <encryption>[...]</encryption> section from another tunnel if it had supported algorithms on it.

[jimp]

Looks like we already had a Redmine issue for this:

https://redmine.pfsense.org/issues/14009

It would also help me out if someone could share the pre-upgrade contents of the <encryption>[...]</encryption> section of an affected tunnel. I tried many different upgrade scenarios yet none of mine resulted in that empty tag.

[pgb]

@jimp Thank you for your answer, just fixed it by editing the config file.

I'm trying to submit my encryption settings but it gets flagged as spam by Akamai...

I'm uploading it as a screenshot (sorry!) to work around Akamai

[jimp]

I was able to reproduce this finally, given that config snippet. I fixed both the bug in the upgrade code that led to the problem tag and I fixed a few potential problem areas in the P1 edit page that didn't get along with the problem tag.

If you haven't already worked around the problem, you can install the System Patches package and then create an entry for 24c0f00ff82d78c1e5301e43df21c2ef8ebecf7f to apply the fix.

[pgb]

@jimp great news! I'm glad that this worked.
I was able to edit my config file and get the UI working based on your previous comment, so my tunnel was fixed quickly.

I'm glad the sample pointed you in the right direction. Thank you again!

[renegade]

I have an ios device with ipsec to my 4100.
After activation the tunnel works fine.
When the iphone gets in standby (no user interaction) the 4100 reboots without any error message or crash dump :-(

[jimp]

@renegade said in IPSec Issue After 23.01 Upgrade:

I have an ios device with ipsec to my 4100.
After activation the tunnel works fine.
When the iphone gets in standby (no user interaction) the 4100 reboots without any error message or crash dump :-(

That wouldn't be related to this thread, so you should start a new one just for that. And there would have to be either an error message or a crash dump somewhere, even if it's only printed to the serial console. You should attach a serial console client and log all the output while you try to make the crash happen again.