Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    IPSec Down after Upgrade to 2.3

    IPsec
    30
    72
    39.6k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • K
      kapara
      last edited by

      Are these all 2.3 systems which were upgraded from 2.2.x or fresh 2.3 installs?

      Skype ID:  Marinhd

      1 Reply Last reply Reply Quote 0
      • M
        MrMoo
        last edited by

        Same problem here with 2.2.6 upgrade to 2.3, IPsec tunnel is reported up but no traffic is flowing.

        1 Reply Last reply Reply Quote 0
        • L
          LordKnut
          last edited by

          @kapara:

          Are these all 2.3 systems which were upgraded from 2.2.x or fresh 2.3 installs?

          only one system updated from 2.2.6 to 2.3
          all other systems are 2.2.4 or 2.2.6

          in the meanwhile I've got one 2.2.6 install running again with traffic throughput … but I've got NO IDEA WHY it is running again ???

          Later today i will try to delete some of the IPsec endpoints on both sides and configure them from the scratch ... let's see what happening.

          1 Reply Last reply Reply Quote 0
          • C
            cmb
            last edited by

            If you're getting either of these messages:

            error sending to PF_KEY socket: No buffer space available
no socket implementation registered, sending failed

            The root cause is here:
            https://redmine.pfsense.org/issues/6160

            the commit on that ticket will fix the root cause. Or if you want to manually fix quickly, run the following:

            killall -9 charon
killall -9 starter 
ipsec stop
ipsec start

            It'll probably still happen again at your next reboot on systems that happen to be unlucky enough to hit that race condition, but will work until then.

            1 Reply Last reply Reply Quote 0
            • M
              moterpent
              last edited by

              I can confirm what Chris has referenced above.  Prior to seeing his post I did indeed notice that there were two instances of charon running in the process list.  The moment I killed one of the processes the tunnels popped right up.

              I can also confirm that upon reboot there are, once again, two instances running and the problem returns.  Kill the procs and all is well again.

              1 Reply Last reply Reply Quote 0
              • C
                cmb
                last edited by

                @moterpent:

                I can confirm what Chris has referenced above.  Prior to seeing his post I did indeed notice that there were two instances of charon running in the process list.  The moment I killed one of the processes the tunnels popped right up.

                I can also confirm that upon reboot there are, once again, two instances running and the problem returns.  Kill the procs and all is well again.

                Thanks. If you apply this change either manually or using the system patches package, it'll fix.
                https://github.com/pfsense/pfsense/commit/c520e3e322e108351f25a259f6e99d627208871c

                If you could confirm whether that fixes it for you, it'd be appreciated.

                1 Reply Last reply Reply Quote 0
                • F
                  fattylewis
                  last edited by

                  @cmb:

                  @moterpent:

                  I can confirm what Chris has referenced above.  Prior to seeing his post I did indeed notice that there were two instances of charon running in the process list.  The moment I killed one of the processes the tunnels popped right up.

                  I can also confirm that upon reboot there are, once again, two instances running and the problem returns.  Kill the procs and all is well again.

                  Thanks. If you apply this change either manually or using the system patches package, it'll fix.
                  https://github.com/pfsense/pfsense/commit/c520e3e322e108351f25a259f6e99d627208871c

                  If you could confirm whether that fixes it for you, it'd be appreciated.

                  Manually made this change. Looks to have fixed it for me. Nice Job!

                  EDIT

                  Come back a few hours later, all tunnels down again. Im doing some investigating to see if this is a continuation of previous issues.

                  Yeah, still getting the exact same error.

                  Apr 15 16:58:56	charon		11[KNL] <con2000|583> unable to delete SAD entry with SPI cef0e6f0
Apr 15 16:58:56	charon		11[KNL] <con2000|583> error sending to PF_KEY socket: No buffer space available
Apr 15 16:58:56	charon		11[KNL] <con2000|583> unable to delete SAD entry with SPI cfb16bdd
Apr 15 16:58:56	charon		11[KNL] <con2000|583> error sending to PF_KEY socket: No buffer space available
Apr 15 16:58:56	charon		11[IKE] <con2000|583> unable to install inbound and outbound IPsec SA (SAD) in kernel
Apr 15 16:58:56	charon		11[KNL] <con2000|583> unable to add SAD entry with SPI cef0e6f0
Apr 15 16:58:56	charon		11[KNL] <con2000|583> error sending to PF_KEY socket: No buffer space available
Apr 15 16:58:56	charon		11[KNL] <con2000|583> unable to add SAD entry with SPI cfb16bdd
Apr 15 16:58:56	charon		11[KNL] <con2000|583> error sending to PF_KEY socket: No buffer space available
Apr 15 16:58:56	charon		11[KNL] <con2000|583> deleting SPI allocation SA failed
Apr 15 16:58:56	charon		11[KNL] <con2000|583> unable to delete SAD entry with SPI cfb16bdd
Apr 15 16:58:56	charon		11[KNL] <con2000|583> error sending to PF_KEY socket: No buffer space available</con2000|583></con2000|583></con2000|583></con2000|583></con2000|583></con2000|583></con2000|583></con2000|583></con2000|583></con2000|583></con2000|583></con2000|583>

                  I stil get it after doing:

                  
killall -9 charon
killall -9 starter 
ipsec stop
ipsec start

                  I double checked my /etc/inc/vpn.inc and it DOES have the changes still there.

                  Any suggestions?

                  1 Reply Last reply Reply Quote 0
                  • C
                    cmb
                    last edited by

                    @fattylewis:

                    Any suggestions?

                    I'd like to take a look at your system. PM me here, or /msg cmb on Freenode if you do IRC.

                    1 Reply Last reply Reply Quote 0
                    • F
                      fattylewis
                      last edited by

                      @cmb:

                      @fattylewis:

                      Any suggestions?

                      I'd like to take a look at your system. PM me here, or /msg cmb on Freenode if you do IRC.

                      I have sent you a PM.

                      1 Reply Last reply Reply Quote 0
                      • A
                        Arendtsen
                        last edited by

                        I have the same issue.

                        The fix has been applied but no success.
                        I have highlighted CMB_ on irc.

                        1 Reply Last reply Reply Quote 0
                        • C
                          cmb
                          last edited by

                          fattylewis' issue looks to have been caused by a different problem, openbgpd also seems to be able to trigger that PF_KEY error (even though in his case BGP wasn't doing anything, just running).

                          Arendtsen: /msged you on IRC.

                          1 Reply Last reply Reply Quote 0
                          • A
                            Arendtsen
                            last edited by

                            Just noticed something new.
                            A reboot of the server (virtual on esxi) brings up the ipsec tunnels.
                            After about eight hours it's seems like they just appear up but there is no SADs or SPDs

                            1 Reply Last reply Reply Quote 0
                            • S
                              shthead
                              last edited by

                              This sounds like the same issue as I am having here: https://forum.pfsense.org/index.php?topic=108706.0

                              I am using the OpenBGPD package as well…

                              Since I posted that I am no longer running the beta version but the issue still persists. I will give the patch linked earlier a go to see if it makes any difference for me, if anyone else has any suggested fixes happy to give them a go.

                              1 Reply Last reply Reply Quote 0
                              • F
                                fattylewis
                                last edited by

                                @shthead:

                                This sounds like the same issue as I am having here: https://forum.pfsense.org/index.php?topic=108706.0

                                I am using the OpenBGPD package as well…

                                Since I posted that I am no longer running the beta version but the issue still persists. I will give the patch linked earlier a go to see if it makes any difference for me, if anyone else has any suggested fixes happy to give them a go.

                                After cmb removed openbgpd (i wasnt actually using it at the time) everything has been fine with my tunnels.

                                1 Reply Last reply Reply Quote 0
                                • S
                                  shthead
                                  last edited by

                                  Unfortunately I need OpenBGPD as I advertise prefixes that need to be routed over the IPSEC tunnels to the pfSense server. Its both too hard to keep track of which prefixes go where and much easier from a failover/traffic engineering point of view to not use it for my case. In this particular situation I also advertise the prefixes in the local office network to the pfSense server from the switches as there are quite a few different networks. I hope that there is a work around at some point, I would be happy to be a guinea pig for it.

                                  1 Reply Last reply Reply Quote 0
                                  • F
                                    fattylewis
                                    last edited by

                                    @shthead:

                                    Unfortunately I need OpenBGPD as I advertise prefixes that need to be routed over the IPSEC tunnels to the pfSense server. Its both too hard to keep track of which prefixes go where and much easier from a failover/traffic engineering point of view to not use it for my case. In this particular situation I also advertise the prefixes in the local office network to the pfSense server from the switches as there are quite a few different networks. I hope that there is a work around at some point, I would be happy to be a guinea pig for it.

                                    This is exactly why i am so glad i updated my home system before doing my office router…

                                    I guess best you can do is see what cmb says :(

                                    1 Reply Last reply Reply Quote 0
                                    • J
                                      jjb-123
                                      last edited by

                                      Hello!

                                      We have the same problem, applied the patch, however as we use OpenBGPD for AWS, ALL the IPSEC vpn's drop.

                                      The only way we fix it at the moment is to reboot the Firewall (this is not ideal).

                                      Any suggestions?

                                      Thanks

                                      1 Reply Last reply Reply Quote 0
                                      • V
                                        vsxi-13
                                        last edited by

                                        Running into what appears to be the same issue.  I've installed the patch that CMB put up and I'm testing it out.  I don't IPsec a lot, but I'll see about trying to do some more in the next couple days to see if this remains fixed.  Luckily I have an OpenVPN connection that I use for other services that I can get back in and stop/start the IPSec service.  Below is a sanitized version of the error message:

                                        
Apr 18 17:03:02	charon		07[NET] <328> received packet: from x.x.x.x[63521] to y.y.y.y[500] (300 bytes)
Apr 18 17:03:02	charon		07[ENC] <328> parsed IKE_SA_INIT request 0 [ SA KE No N(REDIR_SUP) N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) ]
Apr 18 17:03:02	charon		07[IKE] <328> x.x.x.x is initiating an IKE_SA
Apr 18 17:03:02	charon		07[IKE] <328> remote host is behind NAT
Apr 18 17:03:02	charon		07[IKE] <328> sending cert request for "C=US, ST=Wisconsin, L=Madison, O=Xinu, E=zach@xinu.io, CN=dd13-CA"
Apr 18 17:03:02	charon		07[ENC] <328> generating IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) CERTREQ N(FRAG_SUP) N(MULT_AUTH) ]
Apr 18 17:03:02	charon		07[NET] <328> sending packet: from y.y.y.y[500] to x.x.x.x[63521] (341 bytes)
Apr 18 17:03:02	charon		07[NET] <328> received packet: from x.x.x.x[4244] to y.y.y.y[4500] (332 bytes)
Apr 18 17:03:02	charon		07[ENC] <328> parsed IKE_AUTH request 1 [ IDi N(INIT_CONTACT) N(MOBIKE_SUP) IDr CPRQ(ADDR DHCP DNS MASK ADDR6 DHCP6 DNS6) N(ESP_TFC_PAD_N) N(NON_FIRST_FRAG) SA TSi TSr ]
Apr 18 17:03:02	charon		07[CFG] <328> looking for peer configs matching y.y.y.y[xxx.dyndns-web.com]...x.x.x.x[172.20.10.9]
Apr 18 17:03:02	charon		07[CFG] <con1|328>selected peer config 'con1'
Apr 18 17:03:02	charon		07[IKE] <con1|328>initiating EAP_IDENTITY method (id 0x00)
Apr 18 17:03:02	charon		07[IKE] <con1|328>received ESP_TFC_PADDING_NOT_SUPPORTED, not using ESPv3 TFC padding
Apr 18 17:03:02	charon		07[IKE] <con1|328>peer supports MOBIKE
Apr 18 17:03:02	charon		07[IKE] <con1|328>authentication of 'xxx.dyndns-web.com' (myself) with RSA signature successful
Apr 18 17:03:02	charon		07[IKE] <con1|328>sending end entity cert "C=US, ST=Wisconsin, L=Madison, O=Xinu, E=zach@xinu.io, CN=xxx.dyndns-web.com"
Apr 18 17:03:02	charon		07[ENC] <con1|328>generating IKE_AUTH response 1 [ IDr CERT AUTH EAP/REQ/ID ]
Apr 18 17:03:02	charon		07[ENC] <con1|328>splitting IKE message with length of 1596 bytes into 4 fragments
Apr 18 17:03:02	charon		07[ENC] <con1|328>generating IKE_AUTH response 1 [ EF(1/4) ]
Apr 18 17:03:02	charon		07[ENC] <con1|328>generating IKE_AUTH response 1 [ EF(2/4) ]
Apr 18 17:03:02	charon		07[ENC] <con1|328>generating IKE_AUTH response 1 [ EF(3/4) ]
Apr 18 17:03:02	charon		07[ENC] <con1|328>generating IKE_AUTH response 1 [ EF(4/4) ]
Apr 18 17:03:02	charon		07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (544 bytes)
Apr 18 17:03:02	charon		07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (544 bytes)
Apr 18 17:03:02	charon		07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (544 bytes)
Apr 18 17:03:02	charon		07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (144 bytes)
Apr 18 17:03:02	charon		07[NET] <con1|328>received packet: from x.x.x.x[4244] to y.y.y.y[4500] (84 bytes)
Apr 18 17:03:02	charon		07[ENC] <con1|328>parsed IKE_AUTH request 2 [ EAP/RES/ID ]
Apr 18 17:03:02	charon		07[IKE] <con1|328>received EAP identity 'remoteuser@domain.io'
Apr 18 17:03:02	charon		07[IKE] <con1|328>initiating EAP_MSCHAPV2 method (id 0x14)
Apr 18 17:03:02	charon		07[ENC] <con1|328>generating IKE_AUTH response 2 [ EAP/REQ/MSCHAPV2 ]
Apr 18 17:03:02	charon		07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (100 bytes)
Apr 18 17:03:02	charon		07[NET] <con1|328>received packet: from x.x.x.x[4244] to y.y.y.y[4500] (140 bytes)
Apr 18 17:03:02	charon		07[ENC] <con1|328>parsed IKE_AUTH request 3 [ EAP/RES/MSCHAPV2 ]
Apr 18 17:03:02	charon		07[ENC] <con1|328>generating IKE_AUTH response 3 [ EAP/REQ/MSCHAPV2 ]
Apr 18 17:03:02	charon		07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (132 bytes)
Apr 18 17:03:02	charon		07[NET] <con1|328>received packet: from x.x.x.x[4244] to y.y.y.y[4500] (68 bytes)
Apr 18 17:03:02	charon		07[ENC] <con1|328>parsed IKE_AUTH request 4 [ EAP/RES/MSCHAPV2 ]
Apr 18 17:03:02	charon		07[IKE] <con1|328>EAP method EAP_MSCHAPV2 succeeded, MSK established
Apr 18 17:03:02	charon		07[ENC] <con1|328>generating IKE_AUTH response 4 [ EAP/SUCC ]
Apr 18 17:03:02	charon		07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (68 bytes)
Apr 18 17:03:02	charon		07[NET] <con1|328>received packet: from x.x.x.x[4244] to y.y.y.y[4500] (84 bytes)
Apr 18 17:03:02	charon		07[ENC] <con1|328>parsed IKE_AUTH request 5 [ AUTH ]
Apr 18 17:03:02	charon		07[IKE] <con1|328>authentication of '172.20.10.9' with EAP successful
Apr 18 17:03:02	charon		07[IKE] <con1|328>authentication of 'xxx.dyndns-web.com' (myself) with EAP
Apr 18 17:03:02	charon		07[IKE] <con1|328>IKE_SA con1[328] established between y.y.y.y[xxx.dyndns-web.com]...x.x.x.x[172.20.10.9]
Apr 18 17:03:02	charon		07[IKE] <con1|328>scheduling reauthentication in 28169s
Apr 18 17:03:02	charon		07[IKE] <con1|328>maximum IKE_SA lifetime 28709s
Apr 18 17:03:02	charon		07[IKE] <con1|328>peer requested virtual IP %any
Apr 18 17:03:02	charon		07[CFG] <con1|328>reassigning offline lease to 'remoteuser@domain.io'
Apr 18 17:03:02	charon		07[IKE] <con1|328>assigning virtual IP 10.10.10.1 to peer 'remoteuser@domain.io'
Apr 18 17:03:02	charon		07[IKE] <con1|328>peer requested virtual IP %any6
Apr 18 17:03:02	charon		07[IKE] <con1|328>no virtual IP found for %any6 requested by 'remoteuser@domain.io'
Apr 18 17:03:02	charon		07[KNL] <con1|328>error sending to PF_KEY socket: No buffer space available
Apr 18 17:03:02	charon		07[KNL] <con1|328>unable to delete SAD entry with SPI cd6f355a
Apr 18 17:03:02	charon		07[KNL] <con1|328>deleting SPI allocation SA failed
Apr 18 17:03:02	charon		07[KNL] <con1|328>error sending to PF_KEY socket: No buffer space available
Apr 18 17:03:02	charon		07[KNL] <con1|328>unable to add SAD entry with SPI cd6f355a
Apr 18 17:03:02	charon		07[KNL] <con1|328>error sending to PF_KEY socket: No buffer space available
Apr 18 17:03:02	charon		07[KNL] <con1|328>unable to add SAD entry with SPI 0b91cd64
Apr 18 17:03:02	charon		07[IKE] <con1|328>unable to install inbound and outbound IPsec SA (SAD) in kernel
Apr 18 17:03:02	charon		07[IKE] <con1|328>failed to establish CHILD_SA, keeping IKE_SA
Apr 18 17:03:02	charon		07[KNL] <con1|328>error sending to PF_KEY socket: No buffer space available
Apr 18 17:03:02	charon		07[KNL] <con1|328>unable to delete SAD entry with SPI cd6f355a
Apr 18 17:03:02	charon		07[KNL] <con1|328>error sending to PF_KEY socket: No buffer space available
Apr 18 17:03:02	charon		07[KNL] <con1|328>unable to delete SAD entry with SPI 0b91cd64
Apr 18 17:03:02	charon		07[ENC] <con1|328>generating IKE_AUTH response 5 [ AUTH CPRP(ADDR DNS DNS DNS U_DEFDOM U_SPLITDNS) N(AUTH_LFT) N(MOBIKE_SUP) N(ADD_4_ADDR) N(ADD_4_ADDR) N(ADD_4_ADDR) N(NO_PROP) ]
Apr 18 17:03:02	charon		07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (212 bytes)
Apr 18 17:03:02	charon		07[NET] <con1|328>received packet: from x.x.x.x[4244] to y.y.y.y[4500] (68 bytes)
Apr 18 17:03:02	charon		07[ENC] <con1|328>parsed INFORMATIONAL request 6 [ D ]
Apr 18 17:03:02	charon		07[IKE] <con1|328>received DELETE for IKE_SA con1[328]
Apr 18 17:03:02	charon		07[IKE] <con1|328>deleting IKE_SA con1[328] between y.y.y.y[xxx.dyndns-web.com]...x.x.x.x[172.20.10.9]
Apr 18 17:03:02	charon		07[IKE] <con1|328>IKE_SA deleted
Apr 18 17:03:02	charon		07[ENC] <con1|328>generating INFORMATIONAL response 6 [ ]
Apr 18 17:03:02	charon		07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (60 bytes)
Apr 18 17:03:02	charon		07[CFG] <con1|328>lease 10.10.10.1 by 'remoteuser@domain.io' went offline</con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328>
                                        1 Reply Last reply Reply Quote 0
                                        • S
                                          shthead
                                          last edited by

                                          For me I can confirm that removing the OpenBGPD package resolves the problem with the tunnels - IPSEC works fine for me after that. Unfortunately I require OpenBGPD though, so its not a real fix for my situation. Does anyone have any other ideas of what to try?

                                          1 Reply Last reply Reply Quote 0
                                          • A
                                            Arendtsen
                                            last edited by

                                            I have now removed openbgpd and rebooted.
                                            Keeping an eye on it the next couple of days.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.