Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    IPSec Down after Upgrade to 2.3

    IPsec
    30
    72
    39.7k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      moterpent
      last edited by

      After upgrading to 2.3 this evening, all site-to-site ipsec connections are no longer functioning.  Remote end points are made up of various devices (Sonicwall's, other pfsense, etc).  All tunnels were working properly prior to upgrade.

      The following is an excerpt from the IPSEC log.  I'm guessing the buffer space messages aren't a good thing.  No matter what I do with the P1 settings the NO_PROPOSAL_CHOSEN message is consistent.

      Apr 13 19:32:38 charon 15[CFG] rereading secrets
      Apr 13 19:32:38 charon 15[CFG] loading secrets from '/var/etc/ipsec/ipsec.secrets'
      Apr 13 19:32:38 charon 15[CFG] loaded IKE secret for %any a.a.a.a
      Apr 13 19:32:38 charon 15[CFG] rereading ca certificates from '/usr/local/etc/ipsec.d/cacerts'
      Apr 13 19:32:38 charon 15[CFG] rereading aa certificates from '/usr/local/etc/ipsec.d/aacerts'
      Apr 13 19:32:38 charon 15[CFG] rereading ocsp signer certificates from '/usr/local/etc/ipsec.d/ocspcerts'
      Apr 13 19:32:38 charon 15[CFG] rereading attribute certificates from '/usr/local/etc/ipsec.d/acerts'
      Apr 13 19:32:38 charon 15[CFG] rereading crls from '/usr/local/etc/ipsec.d/crls'
      Apr 13 19:32:38 charon 05[CFG] received stroke: unroute 'con1000'
      Apr 13 19:32:38 ipsec_starter 76196 configuration 'con1000' not found
      Apr 13 19:32:38 charon 06[CFG] received stroke: delete connection 'con1000'
      Apr 13 19:32:38 charon 06[CFG] deleted connection 'con1000'
      Apr 13 19:32:38 charon 15[CFG] received stroke: add connection 'con1'
      Apr 13 19:32:38 charon 15[CFG] added configuration 'con1'
      Apr 13 19:32:38 charon 07[CFG] received stroke: route 'con1'
      Apr 13 19:32:38 charon 07[KNL] error sending to PF_KEY socket: No buffer space available
      Apr 13 19:32:38 charon 07[KNL] unable to add policy x.x.x.x/23|/0 === y.y.y.y/24|/0 out
      Apr 13 19:32:38 charon 07[KNL] error sending to PF_KEY socket: No buffer space available
      Apr 13 19:32:38 charon 07[KNL] unable to add policy y.y.y.y/24|/0 === x.x.x.x/23|/0 in
      Apr 13 19:32:38 charon 07[CFG] installing trap failed
      Apr 13 19:32:38 charon 07[KNL] error sending to PF_KEY socket: No buffer space available
      Apr 13 19:32:38 charon 07[KNL] unable to delete policy x.x.x.x/23|/0 === y.y.y.y/24|/0 out
      Apr 13 19:32:38 charon 07[KNL] error sending to PF_KEY socket: No buffer space available
      Apr 13 19:32:38 charon 07[KNL] unable to delete policy y.y.y.y/24|/0 === x.x.x.x/23|/0 in
      Apr 13 19:32:38 ipsec_starter 76196 routing 'con1' failed
      Apr 13 19:32:53 charon 13[NET] <224> received packet: from remote_public_ip[500] to local_public_ip[500] (400 bytes)
      Apr 13 19:32:53 charon 13[ENC] <224> parsed AGGRESSIVE request 0 [ SA KE No ID V V V V V V V V V ]
      Apr 13 19:32:53 charon 13[IKE] <224> no IKE config found for local_public_ip…remote_public_ip, sending NO_PROPOSAL_CHOSEN
      Apr 13 19:32:53 charon 13[ENC] <224> generating INFORMATIONAL_V1 request 1486336742 [ N(NO_PROP) ]
      Apr 13 19:32:53 charon 13[NET] <224> sending packet: from local_public_ip[500] to remote_public_ip[500] (40 bytes)
      Apr 13 19:33:29 charon 11[NET] <225> received packet: from remote_public_ip[500] to local_public_ip[500] (400 bytes)
      Apr 13 19:33:29 charon 11[ENC] <225> parsed AGGRESSIVE request 0 [ SA KE No ID V V V V V V V V V ]
      Apr 13 19:33:29 charon 11[IKE] <225> no IKE config found for local_public_ip…remote_public_ip, sending NO_PROPOSAL_CHOSEN

      Any suggestions or guidance is appreciated.

      1 Reply Last reply Reply Quote 0
      • K
        kapara
        last edited by

        Hope this was not a production environment!  Have you tried completely deleting one of the tunnels and rebuilding the tunnel?

        Skype ID:  Marinhd

        1 Reply Last reply Reply Quote 0
        • F
          fattylewis
          last edited by

          Im getting the same issue. Afer a reboot the tunnels all come up and work for a few minutes, then they all fail.

          These are all pfsense to pfsense site-to-site tunnels. So just as a test i updated one of the other ends to 2.3 but that made no difference either. I have also rebuilt both sides of the tunnel but still no joy.

          
Apr 14 08:15:44	charon		14[ENC] <5384> generating INFORMATIONAL_V1 request 4284809019 [ N(NO_PROP) ]
Apr 14 08:15:44	charon		14[IKE] <5384> no proposal found
Apr 14 08:15:44	charon		14[CFG] <5384> configured proposals: IKE:AES_CBC_256/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1024
Apr 14 08:15:44	charon		14[CFG] <5384> received proposals: IKE:3DES_CBC/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1024
Apr 14 08:15:44	charon		14[IKE] <5384> ********** is initiating a Main Mode IKE_SA
Apr 14 08:15:44	charon		14[ENC] <5384> received unknown vendor ID: 82:99:03:17:57:a3:60:82:c6:a6:21:de:00:05:02:9e
Apr 14 08:15:44	charon		14[IKE] <5384> received FRAGMENTATION vendor ID
Apr 14 08:15:44	charon		14[IKE] <5384> received FRAGMENTATION vendor ID
Apr 14 08:15:44	charon		14[IKE] <5384> received DPD vendor ID
Apr 14 08:15:44	charon		14[IKE] <5384> received draft-ietf-ipsec-nat-t-ike-00 vendor ID
Apr 14 08:15:44	charon		14[ENC] <5384> received unknown vendor ID: 16:f6:ca:16:e4:a4:06:6d:83:82:1a:0f:0a:ea:a8:62
Apr 14 08:15:44	charon		14[IKE] <5384> received draft-ietf-ipsec-nat-t-ike-02\n vendor ID
Apr 14 08:15:44	charon		14[IKE] <5384> received draft-ietf-ipsec-nat-t-ike-02 vendor ID
Apr 14 08:15:44	charon		14[IKE] <5384> received draft-ietf-ipsec-nat-t-ike-03 vendor ID
Apr 14 08:15:44	charon		14[IKE] <5384> received NAT-T (RFC 3947) vendor ID
Apr 14 08:15:44	charon		14[ENC] <5384> parsed ID_PROT request 0 [ SA V V V V V V V V V V ]
Apr 14 08:15:44	charon		14[NET] <5384> received packet: from *********[500] to *********[500] (284 bytes)
Apr 14 08:15:42	charon		14[NET] <con2000|4296>sending packet: from **********[500] to ***********[500] (76 bytes)
Apr 14 08:15:42	charon		14[ENC] <con2000|4296>generating INFORMATIONAL_V1 request 2657408499 [ HASH D ]
Apr 14 08:15:42	charon		14[IKE] <con2000|4296>sending DELETE for ESP CHILD_SA with SPI cbfceb49
Apr 14 08:15:42	charon		14[KNL] <con2000|4296>unable to delete SAD entry with SPI cbfceb49
Apr 14 08:15:42	charon		14[KNL] <con2000|4296>error sending to PF_KEY socket: No buffer space available
Apr 14 08:15:42	charon		14[KNL] <con2000|4296>unable to delete SAD entry with SPI c37a576d
Apr 14 08:15:42	charon		14[KNL] <con2000|4296>error sending to PF_KEY socket: No buffer space available
Apr 14 08:15:42	charon		14[IKE] <con2000|4296>unable to install inbound and outbound IPsec SA (SAD) in kernel
Apr 14 08:15:42	charon		14[KNL] <con2000|4296>unable to add SAD entry with SPI cbfceb49
Apr 14 08:15:42	charon		14[KNL] <con2000|4296>error sending to PF_KEY socket: No buffer space available</con2000|4296></con2000|4296></con2000|4296></con2000|4296></con2000|4296></con2000|4296></con2000|4296></con2000|4296></con2000|4296></con2000|4296>

          One thing im a bit confused about is the recieved proposal is wrong. But i 100% sure the other side is correct. (actually, this might be a red herring, think its related to another tunnel)

          Luckily this isnt production, just my home.

          1 Reply Last reply Reply Quote 0
          • L
            LordKnut
            last edited by

            similar problem here. After some fiddling (IPcompression off, using other proposals, disabling crypto module) the tunnels came up in the end with the same settings they had before (!?)  but they are passing no traffic  :-[.

            Some sites are working from the start with different devices on the other side (cisco sbs, zywall, pfsense 2.2.6) but all others are up passing no traffic.
            Most of the non working tunnels are on Alix or APU Boards with GeodeLX-crypto so all of them are on AES128 proposals (any thoughts).

            Build up a fresh testdevice on an APU-Board with pfSense2.3 which showed the same behaviour first but after disabling IPcompression and rebooting it seems to work now.

            • LX Security Block is loaded
            • AES 128 - SHA256 - DH2 on P1 and P2
            • IPCompresion off

            seems to work between 2.3 and 2.3 but not between 2.3 and 2.2.6.
            I can't see any consistency in this …  ???

            If there are any ideas what to try or to check/log - I can have a look on reproducing it and report back.

            1 Reply Last reply Reply Quote 0
            • K
              kapara
              last edited by

              Are these all 2.3 systems which were upgraded from 2.2.x or fresh 2.3 installs?

              Skype ID:  Marinhd

              1 Reply Last reply Reply Quote 0
              • M
                MrMoo
                last edited by

                Same problem here with 2.2.6 upgrade to 2.3, IPsec tunnel is reported up but no traffic is flowing.

                1 Reply Last reply Reply Quote 0
                • L
                  LordKnut
                  last edited by

                  @kapara:

                  Are these all 2.3 systems which were upgraded from 2.2.x or fresh 2.3 installs?

                  only one system updated from 2.2.6 to 2.3
                  all other systems are 2.2.4 or 2.2.6

                  in the meanwhile I've got one 2.2.6 install running again with traffic throughput … but I've got NO IDEA WHY it is running again ???

                  Later today i will try to delete some of the IPsec endpoints on both sides and configure them from the scratch ... let's see what happening.

                  1 Reply Last reply Reply Quote 0
                  • C
                    cmb
                    last edited by

                    If you're getting either of these messages:

                    error sending to PF_KEY socket: No buffer space available
no socket implementation registered, sending failed

                    The root cause is here:
                    https://redmine.pfsense.org/issues/6160

                    the commit on that ticket will fix the root cause. Or if you want to manually fix quickly, run the following:

                    killall -9 charon
killall -9 starter 
ipsec stop
ipsec start

                    It'll probably still happen again at your next reboot on systems that happen to be unlucky enough to hit that race condition, but will work until then.

                    1 Reply Last reply Reply Quote 0
                    • M
                      moterpent
                      last edited by

                      I can confirm what Chris has referenced above.  Prior to seeing his post I did indeed notice that there were two instances of charon running in the process list.  The moment I killed one of the processes the tunnels popped right up.

                      I can also confirm that upon reboot there are, once again, two instances running and the problem returns.  Kill the procs and all is well again.

                      1 Reply Last reply Reply Quote 0
                      • C
                        cmb
                        last edited by

                        @moterpent:

                        I can confirm what Chris has referenced above.  Prior to seeing his post I did indeed notice that there were two instances of charon running in the process list.  The moment I killed one of the processes the tunnels popped right up.

                        I can also confirm that upon reboot there are, once again, two instances running and the problem returns.  Kill the procs and all is well again.

                        Thanks. If you apply this change either manually or using the system patches package, it'll fix.
                        https://github.com/pfsense/pfsense/commit/c520e3e322e108351f25a259f6e99d627208871c

                        If you could confirm whether that fixes it for you, it'd be appreciated.

                        1 Reply Last reply Reply Quote 0
                        • F
                          fattylewis
                          last edited by

                          @cmb:

                          @moterpent:

                          I can confirm what Chris has referenced above.  Prior to seeing his post I did indeed notice that there were two instances of charon running in the process list.  The moment I killed one of the processes the tunnels popped right up.

                          I can also confirm that upon reboot there are, once again, two instances running and the problem returns.  Kill the procs and all is well again.

                          Thanks. If you apply this change either manually or using the system patches package, it'll fix.
                          https://github.com/pfsense/pfsense/commit/c520e3e322e108351f25a259f6e99d627208871c

                          If you could confirm whether that fixes it for you, it'd be appreciated.

                          Manually made this change. Looks to have fixed it for me. Nice Job!

                          EDIT

                          Come back a few hours later, all tunnels down again. Im doing some investigating to see if this is a continuation of previous issues.

                          Yeah, still getting the exact same error.

                          Apr 15 16:58:56	charon		11[KNL] <con2000|583> unable to delete SAD entry with SPI cef0e6f0
Apr 15 16:58:56	charon		11[KNL] <con2000|583> error sending to PF_KEY socket: No buffer space available
Apr 15 16:58:56	charon		11[KNL] <con2000|583> unable to delete SAD entry with SPI cfb16bdd
Apr 15 16:58:56	charon		11[KNL] <con2000|583> error sending to PF_KEY socket: No buffer space available
Apr 15 16:58:56	charon		11[IKE] <con2000|583> unable to install inbound and outbound IPsec SA (SAD) in kernel
Apr 15 16:58:56	charon		11[KNL] <con2000|583> unable to add SAD entry with SPI cef0e6f0
Apr 15 16:58:56	charon		11[KNL] <con2000|583> error sending to PF_KEY socket: No buffer space available
Apr 15 16:58:56	charon		11[KNL] <con2000|583> unable to add SAD entry with SPI cfb16bdd
Apr 15 16:58:56	charon		11[KNL] <con2000|583> error sending to PF_KEY socket: No buffer space available
Apr 15 16:58:56	charon		11[KNL] <con2000|583> deleting SPI allocation SA failed
Apr 15 16:58:56	charon		11[KNL] <con2000|583> unable to delete SAD entry with SPI cfb16bdd
Apr 15 16:58:56	charon		11[KNL] <con2000|583> error sending to PF_KEY socket: No buffer space available</con2000|583></con2000|583></con2000|583></con2000|583></con2000|583></con2000|583></con2000|583></con2000|583></con2000|583></con2000|583></con2000|583></con2000|583>

                          I stil get it after doing:

                          
killall -9 charon
killall -9 starter 
ipsec stop
ipsec start

                          I double checked my /etc/inc/vpn.inc and it DOES have the changes still there.

                          Any suggestions?

                          1 Reply Last reply Reply Quote 0
                          • C
                            cmb
                            last edited by

                            @fattylewis:

                            Any suggestions?

                            I'd like to take a look at your system. PM me here, or /msg cmb on Freenode if you do IRC.

                            1 Reply Last reply Reply Quote 0
                            • F
                              fattylewis
                              last edited by

                              @cmb:

                              @fattylewis:

                              Any suggestions?

                              I'd like to take a look at your system. PM me here, or /msg cmb on Freenode if you do IRC.

                              I have sent you a PM.

                              1 Reply Last reply Reply Quote 0
                              • A
                                Arendtsen
                                last edited by

                                I have the same issue.

                                The fix has been applied but no success.
                                I have highlighted CMB_ on irc.

                                1 Reply Last reply Reply Quote 0
                                • C
                                  cmb
                                  last edited by

                                  fattylewis' issue looks to have been caused by a different problem, openbgpd also seems to be able to trigger that PF_KEY error (even though in his case BGP wasn't doing anything, just running).

                                  Arendtsen: /msged you on IRC.

                                  1 Reply Last reply Reply Quote 0
                                  • A
                                    Arendtsen
                                    last edited by

                                    Just noticed something new.
                                    A reboot of the server (virtual on esxi) brings up the ipsec tunnels.
                                    After about eight hours it's seems like they just appear up but there is no SADs or SPDs

                                    1 Reply Last reply Reply Quote 0
                                    • S
                                      shthead
                                      last edited by

                                      This sounds like the same issue as I am having here: https://forum.pfsense.org/index.php?topic=108706.0

                                      I am using the OpenBGPD package as well…

                                      Since I posted that I am no longer running the beta version but the issue still persists. I will give the patch linked earlier a go to see if it makes any difference for me, if anyone else has any suggested fixes happy to give them a go.

                                      1 Reply Last reply Reply Quote 0
                                      • F
                                        fattylewis
                                        last edited by

                                        @shthead:

                                        This sounds like the same issue as I am having here: https://forum.pfsense.org/index.php?topic=108706.0

                                        I am using the OpenBGPD package as well…

                                        Since I posted that I am no longer running the beta version but the issue still persists. I will give the patch linked earlier a go to see if it makes any difference for me, if anyone else has any suggested fixes happy to give them a go.

                                        After cmb removed openbgpd (i wasnt actually using it at the time) everything has been fine with my tunnels.

                                        1 Reply Last reply Reply Quote 0
                                        • S
                                          shthead
                                          last edited by

                                          Unfortunately I need OpenBGPD as I advertise prefixes that need to be routed over the IPSEC tunnels to the pfSense server. Its both too hard to keep track of which prefixes go where and much easier from a failover/traffic engineering point of view to not use it for my case. In this particular situation I also advertise the prefixes in the local office network to the pfSense server from the switches as there are quite a few different networks. I hope that there is a work around at some point, I would be happy to be a guinea pig for it.

                                          1 Reply Last reply Reply Quote 0
                                          • F
                                            fattylewis
                                            last edited by

                                            @shthead:

                                            Unfortunately I need OpenBGPD as I advertise prefixes that need to be routed over the IPSEC tunnels to the pfSense server. Its both too hard to keep track of which prefixes go where and much easier from a failover/traffic engineering point of view to not use it for my case. In this particular situation I also advertise the prefixes in the local office network to the pfSense server from the switches as there are quite a few different networks. I hope that there is a work around at some point, I would be happy to be a guinea pig for it.

                                            This is exactly why i am so glad i updated my home system before doing my office router…

                                            I guess best you can do is see what cmb says :(

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.