Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    IPSEC failed to run after 2.4.4 upgrade

    Scheduled Pinned Locked Moved IPsec
    33 Posts 3 Posters 3.7k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • D
      dtrandov
      last edited by

      Neither upgrade to 2.4.5 or relocation of VM resolved issue. I'm going to provision new VM, upload current configuration and see what happened. Thanks a lot for your assistance

      1 Reply Last reply Reply Quote 0
      • N
        nledoux
        last edited by

        Hello.
        Exact same problem here. IPSec refused to start after upgrade from 2.4.3-p1 to 2.4.4 with identical error messages. It was working fine just before. Tried the above step (reinstalling strongswan) without success. pfSense is also visualized but on a different platform (VMware 6.5 / Dell PowerEdge R630) so it might not be a virtualization specific issue. Reverted to snapshot taken just before the upgrade and everything is working fine under 2.4.3-p1. Retried the update with same result (IPSEC not starting) so this is not a upgrade glitch but a reproducible issue. Went back to 2.4.3-p1 snapshot again as it is a production firewall. Any idea what else to test/try?

        1 Reply Last reply Reply Quote 0
        • jimpJ
          jimp Rebel Alliance Developer Netgate
          last edited by

          It's working OK for me here on ESX 6.7, so it's not likely to be specific to virtualization in general, but maybe something in your environment or configuration.

          Can you share the contents of /var/etc/ipsec/strongswan.conf? If there is anything private in there, you can mask/redact it.

          There is a way to disable the integrity tests but I'd rather find out why it's failing first.

          The way those tests are described it's a simple file checksum test, but if that was the case it should be happening for everyone consistently or flagged by pkg check -s strongswan.

          Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

          Need help fast? Netgate Global Support!

          Do not Chat/PM for help!

          1 Reply Last reply Reply Quote 1
          • jimpJ
            jimp Rebel Alliance Developer Netgate
            last edited by jimp

            To anyone that can still reproduce this:

            • Go to VPN > IPsec, Advanced tab.
            • Under IPsec Logging Controls set strongSwan Lib to Highest, then Save
            • Try to restart IPsec
            • look in Status > System Logs, IPsec tab for a message about why it failed. Alternately, check clog /var/log/ipsec.log from the shell.

            The strongSwan source seems to imply that it could be a file/filesystem issue. The checksum is missing, the file size is wrong, or the checksum doesn't match. It could also be that somehow it can't find the library (Maybe run ldconfig in the shell and then try starting it again).

            The debug logs will hopefully tell us more.

            Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

            Need help fast? Netgate Global Support!

            Do not Chat/PM for help!

            1 Reply Last reply Reply Quote 1
            • N
              nledoux
              last edited by

              Here is the strongswan.conf (0_1541782634617_strongswan.conf.txt) from our running 2.4.3 that fails after upgrade.
              I cannot reproduce now but will try the above of hours if nobody was able to provide before.

              1 Reply Last reply Reply Quote 1
              • D
                dtrandov
                last edited by

                One thing I did when upgrading was to refresh dashboard, because i thought that upgrade process freeze. Probably related with issue.

                jimpJ 1 Reply Last reply Reply Quote 0
                • jimpJ
                  jimp Rebel Alliance Developer Netgate @dtrandov
                  last edited by

                  @dtrandov said in IPSEC failed to run after 2.4.4 upgrade:

                  One thing I did when upgrading was to refresh dashboard, because i thought that upgrade process freeze. Probably related with issue.

                  Probably not if the same thing happened after updating to 2.4.5.

                  Can you try the log changes I posted about a few replies up?

                  Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                  Need help fast? Netgate Global Support!

                  Do not Chat/PM for help!

                  1 Reply Last reply Reply Quote 0
                  • D
                    dtrandov
                    last edited by

                    Sure, but I can do it Monday and get back with result.

                    1 Reply Last reply Reply Quote 1
                    • jimpJ
                      jimp Rebel Alliance Developer Netgate
                      last edited by

                      I have opened https://redmine.pfsense.org/issues/9106 to track this, but we can't do anything until we can either reproduce it locally, or get the debug log messages stating exactly what part of the test failed.

                      After gathering the above info, I do have a hunch as it what might help. Install the system patches package and then try the following patch:

                      diff --git a/src/etc/inc/vpn.inc b/src/etc/inc/vpn.inc
                      index d12eb986c2..c055a04d66 100644
                      --- a/src/etc/inc/vpn.inc
                      +++ b/src/etc/inc/vpn.inc
                      @@ -1556,6 +1556,7 @@ EOD;
                       	}
                       
                       	/* manage process */
                      +	mwexec("/etc/rc.d/ldconfig start", false);
                       	if ($restart === true) {
                       		mwexec("/usr/local/sbin/ipsec restart", false);
                       	} else {
                      

                      If that doesn't help, revert the patch.

                      Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                      Need help fast? Netgate Global Support!

                      Do not Chat/PM for help!

                      1 Reply Last reply Reply Quote 1
                      • N
                        nledoux
                        last edited by

                        Thanks for opening the issue for tracking.
                        Retried the update with the same problem. I changed the log level to highest as requested and started IPsec but it unfortunately did not report more information:

                        Nov 11 23:56:33 	ipsec_starter 	19939 	ipsec starter stopped
                        Nov 11 23:56:33 	ipsec_starter 	19939 	charon refused to be started
                        Nov 11 23:56:33 	ipsec_starter 	19939 	charon has quit: integrity test of libstrongswan failed
                        Nov 11 23:56:33 	ipsec_starter 	19637 	no known IPsec stack detected, ignoring!
                        Nov 11 23:56:33 	ipsec_starter 	19637 	no KLIPS IPsec stack detected
                        Nov 11 23:56:33 	ipsec_starter 	19637 	no netkey IPsec stack detected
                        Nov 11 23:56:33 	ipsec_starter 	19637 	Starting strongSwan 5.7.1 IPsec [starter]...
                        

                        I also tried the patch but it did not change anything.
                        Reverted back to 2.4.3 for now.
                        Let's see if dtrandov as better results.

                        1 Reply Last reply Reply Quote 0
                        • D
                          dtrandov
                          last edited by dtrandov

                          Did required changes, but no additional info in log files and IPsec refused to start.
                          Just for test i ran again pkg check -s strongswan, and some files are missing:
                          pkg check -s strongswan

                          Checking strongswan:   0%
                          strongswan-5.7.1: missing file /usr/local/man/man1/pki---acert.1.gz
                          strongswan-5.7.1: missing file /usr/local/man/man1/pki---dn.1.gz
                          strongswan-5.7.1: missing file /usr/local/man/man1/pki---gen.1.gz
                          strongswan-5.7.1: missing file /usr/local/man/man1/pki---issue.1.gz
                          strongswan-5.7.1: missing file /usr/local/man/man1/pki---keyid.1.gz
                          strongswan-5.7.1: missing file /usr/local/man/man1/pki---pkcs7.1.gz
                          strongswan-5.7.1: missing file /usr/local/man/man1/pki---print.1.gz
                          strongswan-5.7.1: missing file /usr/local/man/man1/pki---pub.1.gz
                          strongswan-5.7.1: missing file /usr/local/man/man1/pki---req.1.gz
                          strongswan-5.7.1: missing file /usr/local/man/man1/pki---self.1.gz
                          strongswan-5.7.1: missing file /usr/local/man/man1/pki---signcrl.1.gz
                          strongswan-5.7.1: missing file /usr/local/man/man1/pki---verify.1.gz
                          strongswan-5.7.1: missing file /usr/local/man/man1/pki.1.gz
                          strongswan-5.7.1: missing file /usr/local/man/man5/ipsec.conf.5.gz
                          strongswan-5.7.1: missing file /usr/local/man/man5/ipsec.secrets.5.gz
                          strongswan-5.7.1: missing file /usr/local/man/man5/strongswan.conf.5.gz
                          strongswan-5.7.1: missing file /usr/local/man/man5/swanctl.conf.5.gz
                          strongswan-5.7.1: missing file /usr/local/man/man8/charon-cmd.8.gz
                          strongswan-5.7.1: missing file /usr/local/man/man8/ipsec.8.gz
                          strongswan-5.7.1: missing file /usr/local/man/man8/swanctl.8.gz
                          

                          I'm sure when issue arise, I did reinstall strongswan and no errors appears.

                          Also:

                          cat /var/etc/ipsec/strongswan.conf
                          
                          # Automatically generated config file - DO NOT MODIFY. Changes will be overwritten.
                          starter {
                                 load_warning = no
                                 config_file = /var/etc/ipsec/ipsec.conf
                          }
                          
                          charon {
                          # number of worker threads in charon
                                 threads = 16
                                 ikesa_table_size = 32
                                 ikesa_table_segments = 4
                                 init_limit_half_open = 1000
                                 install_routes = no
                                 load_modular = yes
                                 ignore_acquire_ts = yes
                          
                          
                                 cisco_unity = no
                          
                          
                          
                                 syslog {
                                         identifier = charon
                                         # log everything under daemon since it ends up in the same place regardless with our syslog.conf
                                         daemon {
                                                 ike_name = yes
                                                 dmn = 1
                                                 mgr = 1
                                                 ike = 1
                                                 chd = 1
                                                 job = 1
                                                 cfg = 1
                                                 knl = 1
                                                 net = 1
                                                 asn = 1
                                                 enc = 1
                                                 imc = 1
                                                 imv = 1
                                                 pts = 1
                                                 tls = 1
                                                 esp = 1
                                                 lib = 4
                          
                                         }
                                         # disable logging under auth so logs aren't duplicated
                                         auth {
                                                 default = -1
                                         }
                                 }
                          
                                 plugins {
                                         # Load defaults
                                         include /var/etc/ipsec/strongswan.d/charon/*.conf
                          
                                         stroke {
                                                 secrets_file = /var/etc/ipsec/ipsec.secrets
                                         }
                          
                                         unity {
                                                 load = no
                                         }
                                         eap-radius {
                                                 class_group = yes
                                                 eap_start = no
                                                 accounting = yes
                                                 servers {
                                                         dc1.mydomain.com-radius {
                                                                 address = 192.168.111.101
                                                                 secret = "asdasd"
                                                                 auth_port = 1812
                                                                 acct_port = 1813
                                                         }
                          
                                                 }
                                         }
                                         attr {
                                                 subnet = 172.16.15.0/24,192.168.111.0/24
                                                 split-include = 172.16.15.0/24,192.168.111.0/24
                                         }
                                         xauth-generic {
                                                 script = /etc/inc/ipsec.auth-user.php
                                                 authcfg = dc1.mydomain.com-radius
                                         }
                          
                                 }
                          }
                          
                          1 Reply Last reply Reply Quote 0
                          • jimpJ
                            jimp Rebel Alliance Developer Netgate
                            last edited by

                            The man pages being missing is expected, as we normally will strip out the man pages and other docs from the host itself to save space.

                            I was hoping that little patch would help, but at least we know it isn't the library path.

                            If you can get a host back into the failed state, try going to an ssh or console shell prompt and then run this command:

                            ipsec stop
                            ipsec start --debug-all
                            

                            And then check the console output and IPsec log

                            Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                            Need help fast? Netgate Global Support!

                            Do not Chat/PM for help!

                            1 Reply Last reply Reply Quote 1
                            • D
                              dtrandov
                              last edited by

                              Console output:

                              ipsec stop:

                              Stopping strongSwan IPsec failed: starter is not running
                              

                              ipsec start --debug-all:

                              /usr/local/etc/strongswan.conf:68: syntax error, unexpected ., expecting : or '{' or '=' [.]
                              invalid config file '/usr/local/etc/strongswan.conf'
                              abort initialization due to invalid configuration
                              Starting strongSwan 5.7.1 IPsec [starter]...
                              Loading config setup
                                uniqueids=yes
                              Loading conn 'bypasslan'
                                authby=never
                                auto=route
                                leftsubnet=192.168.111.0/24
                                rightsubnet=192.168.111.0/24
                                type=passthrough
                              Loading conn 'con-mobile'
                                auto=add
                                dpdaction=clear
                                dpddelay=10s
                                dpdtimeout=60s
                                eap_identity=%identity
                                esp=aes256-sha256,aes192-sha256,aes128-sha256,aes256-sha256,aes192-sha256,aes128-sha256!
                                forceencaps=no
                                fragmentation=yes
                                ike=aes256-sha256-modp1024!
                                ikelifetime=28800s
                                installpolicy=yes
                                keyexchange=ikev2
                                left=172.16.15.160
                                leftauth=pubkey
                                leftcert=/var/etc/ipsec/ipsec.d/certs/cert.crt
                                leftid=fqdn:vpn2.mydomain.com
                                leftsendcert=always
                                leftsubnet=172.16.15.0/24,192.168.111.0/24
                                lifetime=3600s
                                mobike=yes
                                reauth=yes
                                rekey=yes
                                right=%any
                                rightauth=eap-radius
                                rightsourceip=192.168.89.0/24
                                type=tunnel
                              kernel appears to lack the native netkey IPsec stack
                              no netkey IPsec stack detected
                              kernel appears to lack the KLIPS IPsec stack
                              no KLIPS IPsec stack detected
                              no known IPsec stack detected, ignoring!
                              

                              line 68 is where is dc1.mydomain.com-radius:

                              dc1.mydomain.com-radius {
                                                                     address = 192.168.111.101
                                                                     secret = "masked"
                                                                     auth_port = 1812
                                                                     acct_port = 1813
                                                             }
                              
                              
                              
                              1 Reply Last reply Reply Quote 0
                              • jimpJ
                                jimp Rebel Alliance Developer Netgate
                                last edited by

                                OK, that is a different condition. It doesn't like a . in the RADIUS server name. I didn't have any set that way, but now that I set one up I see the same error. Curious that the error is shown on the console but not in the logs. But now I do see the same integrity test error!

                                So we're getting closer! Let me find a fix for this, most likely it will involve removing the dots or swapping them for some other character.

                                Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                                Need help fast? Netgate Global Support!

                                Do not Chat/PM for help!

                                1 Reply Last reply Reply Quote 1
                                • D
                                  dtrandov
                                  last edited by dtrandov

                                  Great.

                                  well, I've just defined new radius server entry (without dots in name) in System/User Manager/Authentication Servers and IPSec is up now..... definitely dots in name are problematic.

                                  1 Reply Last reply Reply Quote 0
                                  • jimpJ
                                    jimp Rebel Alliance Developer Netgate
                                    last edited by

                                    Thankfully that is a very simple fix (assuming you don't fudge the regex like I did on my first commit), so if you want to keep a RADIUS server entry with . in the name, you can if you apply the following patch:

                                    diff --git a/src/etc/inc/vpn.inc b/src/etc/inc/vpn.inc
                                    index 2458a224f5..e3ccb4f3c4 100644
                                    --- a/src/etc/inc/vpn.inc
                                    +++ b/src/etc/inc/vpn.inc
                                    @@ -478,7 +478,7 @@ EOD;
                                     	$user_sources = explode(',', $config['ipsec']['client']['user_source']);
                                     	foreach ($user_sources as $user_source) {
                                     		$auth_server = auth_get_authserver($user_source);
                                    -		$nice_user_source = strtolower(preg_replace('/\s+/', '_', $user_source));
                                    +		$nice_user_source = strtolower(preg_replace('/[\s\.]+/', '_', $user_source));
                                     		if ($auth_server && $auth_server['type'] === 'radius') {
                                     			$radius_server_txt .= <<<EOD
                                     				{$nice_user_source} {
                                    

                                    Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                                    Need help fast? Netgate Global Support!

                                    Do not Chat/PM for help!

                                    1 Reply Last reply Reply Quote 1
                                    • D
                                      dtrandov
                                      last edited by

                                      Thanks a lot. Really appreciate you assistance :)

                                      1 Reply Last reply Reply Quote 0
                                      • N
                                        nledoux
                                        last edited by

                                        Thanks jimp!
                                        I'll try this also tonight and post results.
                                        Quick question: (I'm no expert with diff.) Shouldn't the path in the above patches be /etc/inc/vpn.inc? There is no /src folder on our system.
                                        Also, will this be corrected is a future release or will we need to keep the patch as long as we have names with dot?

                                        1 Reply Last reply Reply Quote 0
                                        • jimpJ
                                          jimp Rebel Alliance Developer Netgate
                                          last edited by

                                          It will be corrected in 2.4.4-p1 and 2.4.5.

                                          As for the path, the system patches package defaults to a path strip level of 2, so it will do the right thing here.

                                          The source in github has a src/ directory prefix, which is why it shows in patches made from git commits like the above.

                                          Remember: Upvote with the ๐Ÿ‘ button for any user/post you find to be helpful, informative, or deserving of recognition!

                                          Need help fast? Netgate Global Support!

                                          Do not Chat/PM for help!

                                          1 Reply Last reply Reply Quote 0
                                          • N
                                            nledoux
                                            last edited by

                                            Understood for the default strip level of the system patches package.
                                            This worked also for us and we now have a working 2.4.4 system.
                                            Thanks jimp!

                                            1 Reply Last reply Reply Quote 1
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.