IPSec Down after Upgrade to 2.3
-
After upgrading to 2.3 this evening, all site-to-site ipsec connections are no longer functioning. Remote end points are made up of various devices (Sonicwall's, other pfsense, etc). All tunnels were working properly prior to upgrade.
The following is an excerpt from the IPSEC log. I'm guessing the buffer space messages aren't a good thing. No matter what I do with the P1 settings the NO_PROPOSAL_CHOSEN message is consistent.
Apr 13 19:32:38 charon 15[CFG] rereading secrets
Apr 13 19:32:38 charon 15[CFG] loading secrets from '/var/etc/ipsec/ipsec.secrets'
Apr 13 19:32:38 charon 15[CFG] loaded IKE secret for %any a.a.a.a
Apr 13 19:32:38 charon 15[CFG] rereading ca certificates from '/usr/local/etc/ipsec.d/cacerts'
Apr 13 19:32:38 charon 15[CFG] rereading aa certificates from '/usr/local/etc/ipsec.d/aacerts'
Apr 13 19:32:38 charon 15[CFG] rereading ocsp signer certificates from '/usr/local/etc/ipsec.d/ocspcerts'
Apr 13 19:32:38 charon 15[CFG] rereading attribute certificates from '/usr/local/etc/ipsec.d/acerts'
Apr 13 19:32:38 charon 15[CFG] rereading crls from '/usr/local/etc/ipsec.d/crls'
Apr 13 19:32:38 charon 05[CFG] received stroke: unroute 'con1000'
Apr 13 19:32:38 ipsec_starter 76196 configuration 'con1000' not found
Apr 13 19:32:38 charon 06[CFG] received stroke: delete connection 'con1000'
Apr 13 19:32:38 charon 06[CFG] deleted connection 'con1000'
Apr 13 19:32:38 charon 15[CFG] received stroke: add connection 'con1'
Apr 13 19:32:38 charon 15[CFG] added configuration 'con1'
Apr 13 19:32:38 charon 07[CFG] received stroke: route 'con1'
Apr 13 19:32:38 charon 07[KNL] error sending to PF_KEY socket: No buffer space available
Apr 13 19:32:38 charon 07[KNL] unable to add policy x.x.x.x/23|/0 === y.y.y.y/24|/0 out
Apr 13 19:32:38 charon 07[KNL] error sending to PF_KEY socket: No buffer space available
Apr 13 19:32:38 charon 07[KNL] unable to add policy y.y.y.y/24|/0 === x.x.x.x/23|/0 in
Apr 13 19:32:38 charon 07[CFG] installing trap failed
Apr 13 19:32:38 charon 07[KNL] error sending to PF_KEY socket: No buffer space available
Apr 13 19:32:38 charon 07[KNL] unable to delete policy x.x.x.x/23|/0 === y.y.y.y/24|/0 out
Apr 13 19:32:38 charon 07[KNL] error sending to PF_KEY socket: No buffer space available
Apr 13 19:32:38 charon 07[KNL] unable to delete policy y.y.y.y/24|/0 === x.x.x.x/23|/0 in
Apr 13 19:32:38 ipsec_starter 76196 routing 'con1' failed
Apr 13 19:32:53 charon 13[NET] <224> received packet: from remote_public_ip[500] to local_public_ip[500] (400 bytes)
Apr 13 19:32:53 charon 13[ENC] <224> parsed AGGRESSIVE request 0 [ SA KE No ID V V V V V V V V V ]
Apr 13 19:32:53 charon 13[IKE] <224> no IKE config found for local_public_ip…remote_public_ip, sending NO_PROPOSAL_CHOSEN
Apr 13 19:32:53 charon 13[ENC] <224> generating INFORMATIONAL_V1 request 1486336742 [ N(NO_PROP) ]
Apr 13 19:32:53 charon 13[NET] <224> sending packet: from local_public_ip[500] to remote_public_ip[500] (40 bytes)
Apr 13 19:33:29 charon 11[NET] <225> received packet: from remote_public_ip[500] to local_public_ip[500] (400 bytes)
Apr 13 19:33:29 charon 11[ENC] <225> parsed AGGRESSIVE request 0 [ SA KE No ID V V V V V V V V V ]
Apr 13 19:33:29 charon 11[IKE] <225> no IKE config found for local_public_ip…remote_public_ip, sending NO_PROPOSAL_CHOSENAny suggestions or guidance is appreciated.
-
Hope this was not a production environment! Have you tried completely deleting one of the tunnels and rebuilding the tunnel?
-
Im getting the same issue. Afer a reboot the tunnels all come up and work for a few minutes, then they all fail.
These are all pfsense to pfsense site-to-site tunnels. So just as a test i updated one of the other ends to 2.3 but that made no difference either. I have also rebuilt both sides of the tunnel but still no joy.
Apr 14 08:15:44 charon 14[ENC] <5384> generating INFORMATIONAL_V1 request 4284809019 [ N(NO_PROP) ] Apr 14 08:15:44 charon 14[IKE] <5384> no proposal found Apr 14 08:15:44 charon 14[CFG] <5384> configured proposals: IKE:AES_CBC_256/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1024 Apr 14 08:15:44 charon 14[CFG] <5384> received proposals: IKE:3DES_CBC/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1024 Apr 14 08:15:44 charon 14[IKE] <5384> ********** is initiating a Main Mode IKE_SA Apr 14 08:15:44 charon 14[ENC] <5384> received unknown vendor ID: 82:99:03:17:57:a3:60:82:c6:a6:21:de:00:05:02:9e Apr 14 08:15:44 charon 14[IKE] <5384> received FRAGMENTATION vendor ID Apr 14 08:15:44 charon 14[IKE] <5384> received FRAGMENTATION vendor ID Apr 14 08:15:44 charon 14[IKE] <5384> received DPD vendor ID Apr 14 08:15:44 charon 14[IKE] <5384> received draft-ietf-ipsec-nat-t-ike-00 vendor ID Apr 14 08:15:44 charon 14[ENC] <5384> received unknown vendor ID: 16:f6:ca:16:e4:a4:06:6d:83:82:1a:0f:0a:ea:a8:62 Apr 14 08:15:44 charon 14[IKE] <5384> received draft-ietf-ipsec-nat-t-ike-02\n vendor ID Apr 14 08:15:44 charon 14[IKE] <5384> received draft-ietf-ipsec-nat-t-ike-02 vendor ID Apr 14 08:15:44 charon 14[IKE] <5384> received draft-ietf-ipsec-nat-t-ike-03 vendor ID Apr 14 08:15:44 charon 14[IKE] <5384> received NAT-T (RFC 3947) vendor ID Apr 14 08:15:44 charon 14[ENC] <5384> parsed ID_PROT request 0 [ SA V V V V V V V V V V ] Apr 14 08:15:44 charon 14[NET] <5384> received packet: from *********[500] to *********[500] (284 bytes) Apr 14 08:15:42 charon 14[NET] <con2000|4296>sending packet: from **********[500] to ***********[500] (76 bytes) Apr 14 08:15:42 charon 14[ENC] <con2000|4296>generating INFORMATIONAL_V1 request 2657408499 [ HASH D ] Apr 14 08:15:42 charon 14[IKE] <con2000|4296>sending DELETE for ESP CHILD_SA with SPI cbfceb49 Apr 14 08:15:42 charon 14[KNL] <con2000|4296>unable to delete SAD entry with SPI cbfceb49 Apr 14 08:15:42 charon 14[KNL] <con2000|4296>error sending to PF_KEY socket: No buffer space available Apr 14 08:15:42 charon 14[KNL] <con2000|4296>unable to delete SAD entry with SPI c37a576d Apr 14 08:15:42 charon 14[KNL] <con2000|4296>error sending to PF_KEY socket: No buffer space available Apr 14 08:15:42 charon 14[IKE] <con2000|4296>unable to install inbound and outbound IPsec SA (SAD) in kernel Apr 14 08:15:42 charon 14[KNL] <con2000|4296>unable to add SAD entry with SPI cbfceb49 Apr 14 08:15:42 charon 14[KNL] <con2000|4296>error sending to PF_KEY socket: No buffer space available</con2000|4296></con2000|4296></con2000|4296></con2000|4296></con2000|4296></con2000|4296></con2000|4296></con2000|4296></con2000|4296></con2000|4296>
One thing im a bit confused about is the recieved proposal is wrong. But i 100% sure the other side is correct. (actually, this might be a red herring, think its related to another tunnel)
Luckily this isnt production, just my home.
-
similar problem here. After some fiddling (IPcompression off, using other proposals, disabling crypto module) the tunnels came up in the end with the same settings they had before (!?) but they are passing no traffic :-[.
Some sites are working from the start with different devices on the other side (cisco sbs, zywall, pfsense 2.2.6) but all others are up passing no traffic.
Most of the non working tunnels are on Alix or APU Boards with GeodeLX-crypto so all of them are on AES128 proposals (any thoughts).Build up a fresh testdevice on an APU-Board with pfSense2.3 which showed the same behaviour first but after disabling IPcompression and rebooting it seems to work now.
- LX Security Block is loaded
- AES 128 - SHA256 - DH2 on P1 and P2
- IPCompresion off
seems to work between 2.3 and 2.3 but not between 2.3 and 2.2.6.
I can't see any consistency in this … ???If there are any ideas what to try or to check/log - I can have a look on reproducing it and report back.
-
Are these all 2.3 systems which were upgraded from 2.2.x or fresh 2.3 installs?
-
Same problem here with 2.2.6 upgrade to 2.3, IPsec tunnel is reported up but no traffic is flowing.
-
Are these all 2.3 systems which were upgraded from 2.2.x or fresh 2.3 installs?
only one system updated from 2.2.6 to 2.3
all other systems are 2.2.4 or 2.2.6in the meanwhile I've got one 2.2.6 install running again with traffic throughput … but I've got NO IDEA WHY it is running again ???
Later today i will try to delete some of the IPsec endpoints on both sides and configure them from the scratch ... let's see what happening.
-
If you're getting either of these messages:
error sending to PF_KEY socket: No buffer space available no socket implementation registered, sending failed
The root cause is here:
https://redmine.pfsense.org/issues/6160the commit on that ticket will fix the root cause. Or if you want to manually fix quickly, run the following:
killall -9 charon killall -9 starter ipsec stop ipsec start
It'll probably still happen again at your next reboot on systems that happen to be unlucky enough to hit that race condition, but will work until then.
-
I can confirm what Chris has referenced above. Prior to seeing his post I did indeed notice that there were two instances of charon running in the process list. The moment I killed one of the processes the tunnels popped right up.
I can also confirm that upon reboot there are, once again, two instances running and the problem returns. Kill the procs and all is well again.
-
I can confirm what Chris has referenced above. Prior to seeing his post I did indeed notice that there were two instances of charon running in the process list. The moment I killed one of the processes the tunnels popped right up.
I can also confirm that upon reboot there are, once again, two instances running and the problem returns. Kill the procs and all is well again.
Thanks. If you apply this change either manually or using the system patches package, it'll fix.
https://github.com/pfsense/pfsense/commit/c520e3e322e108351f25a259f6e99d627208871cIf you could confirm whether that fixes it for you, it'd be appreciated.
-
@cmb:
I can confirm what Chris has referenced above. Prior to seeing his post I did indeed notice that there were two instances of charon running in the process list. The moment I killed one of the processes the tunnels popped right up.
I can also confirm that upon reboot there are, once again, two instances running and the problem returns. Kill the procs and all is well again.
Thanks. If you apply this change either manually or using the system patches package, it'll fix.
https://github.com/pfsense/pfsense/commit/c520e3e322e108351f25a259f6e99d627208871cIf you could confirm whether that fixes it for you, it'd be appreciated.
Manually made this change. Looks to have fixed it for me. Nice Job!
EDIT
Come back a few hours later, all tunnels down again. Im doing some investigating to see if this is a continuation of previous issues.
Yeah, still getting the exact same error.
Apr 15 16:58:56 charon 11[KNL] <con2000|583> unable to delete SAD entry with SPI cef0e6f0 Apr 15 16:58:56 charon 11[KNL] <con2000|583> error sending to PF_KEY socket: No buffer space available Apr 15 16:58:56 charon 11[KNL] <con2000|583> unable to delete SAD entry with SPI cfb16bdd Apr 15 16:58:56 charon 11[KNL] <con2000|583> error sending to PF_KEY socket: No buffer space available Apr 15 16:58:56 charon 11[IKE] <con2000|583> unable to install inbound and outbound IPsec SA (SAD) in kernel Apr 15 16:58:56 charon 11[KNL] <con2000|583> unable to add SAD entry with SPI cef0e6f0 Apr 15 16:58:56 charon 11[KNL] <con2000|583> error sending to PF_KEY socket: No buffer space available Apr 15 16:58:56 charon 11[KNL] <con2000|583> unable to add SAD entry with SPI cfb16bdd Apr 15 16:58:56 charon 11[KNL] <con2000|583> error sending to PF_KEY socket: No buffer space available Apr 15 16:58:56 charon 11[KNL] <con2000|583> deleting SPI allocation SA failed Apr 15 16:58:56 charon 11[KNL] <con2000|583> unable to delete SAD entry with SPI cfb16bdd Apr 15 16:58:56 charon 11[KNL] <con2000|583> error sending to PF_KEY socket: No buffer space available</con2000|583></con2000|583></con2000|583></con2000|583></con2000|583></con2000|583></con2000|583></con2000|583></con2000|583></con2000|583></con2000|583></con2000|583>
I stil get it after doing:
killall -9 charon killall -9 starter ipsec stop ipsec start
I double checked my /etc/inc/vpn.inc and it DOES have the changes still there.
Any suggestions?
-
Any suggestions?
I'd like to take a look at your system. PM me here, or /msg cmb on Freenode if you do IRC.
-
@cmb:
Any suggestions?
I'd like to take a look at your system. PM me here, or /msg cmb on Freenode if you do IRC.
I have sent you a PM.
-
I have the same issue.
The fix has been applied but no success.
I have highlighted CMB_ on irc. -
fattylewis' issue looks to have been caused by a different problem, openbgpd also seems to be able to trigger that PF_KEY error (even though in his case BGP wasn't doing anything, just running).
Arendtsen: /msged you on IRC.
-
Just noticed something new.
A reboot of the server (virtual on esxi) brings up the ipsec tunnels.
After about eight hours it's seems like they just appear up but there is no SADs or SPDs -
This sounds like the same issue as I am having here: https://forum.pfsense.org/index.php?topic=108706.0
I am using the OpenBGPD package as well…
Since I posted that I am no longer running the beta version but the issue still persists. I will give the patch linked earlier a go to see if it makes any difference for me, if anyone else has any suggested fixes happy to give them a go.
-
This sounds like the same issue as I am having here: https://forum.pfsense.org/index.php?topic=108706.0
I am using the OpenBGPD package as well…
Since I posted that I am no longer running the beta version but the issue still persists. I will give the patch linked earlier a go to see if it makes any difference for me, if anyone else has any suggested fixes happy to give them a go.
After cmb removed openbgpd (i wasnt actually using it at the time) everything has been fine with my tunnels.
-
Unfortunately I need OpenBGPD as I advertise prefixes that need to be routed over the IPSEC tunnels to the pfSense server. Its both too hard to keep track of which prefixes go where and much easier from a failover/traffic engineering point of view to not use it for my case. In this particular situation I also advertise the prefixes in the local office network to the pfSense server from the switches as there are quite a few different networks. I hope that there is a work around at some point, I would be happy to be a guinea pig for it.
-
Unfortunately I need OpenBGPD as I advertise prefixes that need to be routed over the IPSEC tunnels to the pfSense server. Its both too hard to keep track of which prefixes go where and much easier from a failover/traffic engineering point of view to not use it for my case. In this particular situation I also advertise the prefixes in the local office network to the pfSense server from the switches as there are quite a few different networks. I hope that there is a work around at some point, I would be happy to be a guinea pig for it.
This is exactly why i am so glad i updated my home system before doing my office router…
I guess best you can do is see what cmb says :(
-
Hello!
We have the same problem, applied the patch, however as we use OpenBGPD for AWS, ALL the IPSEC vpn's drop.
The only way we fix it at the moment is to reboot the Firewall (this is not ideal).
Any suggestions?
Thanks
-
Running into what appears to be the same issue. I've installed the patch that CMB put up and I'm testing it out. I don't IPsec a lot, but I'll see about trying to do some more in the next couple days to see if this remains fixed. Luckily I have an OpenVPN connection that I use for other services that I can get back in and stop/start the IPSec service. Below is a sanitized version of the error message:
Apr 18 17:03:02 charon 07[NET] <328> received packet: from x.x.x.x[63521] to y.y.y.y[500] (300 bytes) Apr 18 17:03:02 charon 07[ENC] <328> parsed IKE_SA_INIT request 0 [ SA KE No N(REDIR_SUP) N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) ] Apr 18 17:03:02 charon 07[IKE] <328> x.x.x.x is initiating an IKE_SA Apr 18 17:03:02 charon 07[IKE] <328> remote host is behind NAT Apr 18 17:03:02 charon 07[IKE] <328> sending cert request for "C=US, ST=Wisconsin, L=Madison, O=Xinu, E=zach@xinu.io, CN=dd13-CA" Apr 18 17:03:02 charon 07[ENC] <328> generating IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) CERTREQ N(FRAG_SUP) N(MULT_AUTH) ] Apr 18 17:03:02 charon 07[NET] <328> sending packet: from y.y.y.y[500] to x.x.x.x[63521] (341 bytes) Apr 18 17:03:02 charon 07[NET] <328> received packet: from x.x.x.x[4244] to y.y.y.y[4500] (332 bytes) Apr 18 17:03:02 charon 07[ENC] <328> parsed IKE_AUTH request 1 [ IDi N(INIT_CONTACT) N(MOBIKE_SUP) IDr CPRQ(ADDR DHCP DNS MASK ADDR6 DHCP6 DNS6) N(ESP_TFC_PAD_N) N(NON_FIRST_FRAG) SA TSi TSr ] Apr 18 17:03:02 charon 07[CFG] <328> looking for peer configs matching y.y.y.y[xxx.dyndns-web.com]...x.x.x.x[172.20.10.9] Apr 18 17:03:02 charon 07[CFG] <con1|328>selected peer config 'con1' Apr 18 17:03:02 charon 07[IKE] <con1|328>initiating EAP_IDENTITY method (id 0x00) Apr 18 17:03:02 charon 07[IKE] <con1|328>received ESP_TFC_PADDING_NOT_SUPPORTED, not using ESPv3 TFC padding Apr 18 17:03:02 charon 07[IKE] <con1|328>peer supports MOBIKE Apr 18 17:03:02 charon 07[IKE] <con1|328>authentication of 'xxx.dyndns-web.com' (myself) with RSA signature successful Apr 18 17:03:02 charon 07[IKE] <con1|328>sending end entity cert "C=US, ST=Wisconsin, L=Madison, O=Xinu, E=zach@xinu.io, CN=xxx.dyndns-web.com" Apr 18 17:03:02 charon 07[ENC] <con1|328>generating IKE_AUTH response 1 [ IDr CERT AUTH EAP/REQ/ID ] Apr 18 17:03:02 charon 07[ENC] <con1|328>splitting IKE message with length of 1596 bytes into 4 fragments Apr 18 17:03:02 charon 07[ENC] <con1|328>generating IKE_AUTH response 1 [ EF(1/4) ] Apr 18 17:03:02 charon 07[ENC] <con1|328>generating IKE_AUTH response 1 [ EF(2/4) ] Apr 18 17:03:02 charon 07[ENC] <con1|328>generating IKE_AUTH response 1 [ EF(3/4) ] Apr 18 17:03:02 charon 07[ENC] <con1|328>generating IKE_AUTH response 1 [ EF(4/4) ] Apr 18 17:03:02 charon 07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (544 bytes) Apr 18 17:03:02 charon 07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (544 bytes) Apr 18 17:03:02 charon 07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (544 bytes) Apr 18 17:03:02 charon 07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (144 bytes) Apr 18 17:03:02 charon 07[NET] <con1|328>received packet: from x.x.x.x[4244] to y.y.y.y[4500] (84 bytes) Apr 18 17:03:02 charon 07[ENC] <con1|328>parsed IKE_AUTH request 2 [ EAP/RES/ID ] Apr 18 17:03:02 charon 07[IKE] <con1|328>received EAP identity 'remoteuser@domain.io' Apr 18 17:03:02 charon 07[IKE] <con1|328>initiating EAP_MSCHAPV2 method (id 0x14) Apr 18 17:03:02 charon 07[ENC] <con1|328>generating IKE_AUTH response 2 [ EAP/REQ/MSCHAPV2 ] Apr 18 17:03:02 charon 07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (100 bytes) Apr 18 17:03:02 charon 07[NET] <con1|328>received packet: from x.x.x.x[4244] to y.y.y.y[4500] (140 bytes) Apr 18 17:03:02 charon 07[ENC] <con1|328>parsed IKE_AUTH request 3 [ EAP/RES/MSCHAPV2 ] Apr 18 17:03:02 charon 07[ENC] <con1|328>generating IKE_AUTH response 3 [ EAP/REQ/MSCHAPV2 ] Apr 18 17:03:02 charon 07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (132 bytes) Apr 18 17:03:02 charon 07[NET] <con1|328>received packet: from x.x.x.x[4244] to y.y.y.y[4500] (68 bytes) Apr 18 17:03:02 charon 07[ENC] <con1|328>parsed IKE_AUTH request 4 [ EAP/RES/MSCHAPV2 ] Apr 18 17:03:02 charon 07[IKE] <con1|328>EAP method EAP_MSCHAPV2 succeeded, MSK established Apr 18 17:03:02 charon 07[ENC] <con1|328>generating IKE_AUTH response 4 [ EAP/SUCC ] Apr 18 17:03:02 charon 07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (68 bytes) Apr 18 17:03:02 charon 07[NET] <con1|328>received packet: from x.x.x.x[4244] to y.y.y.y[4500] (84 bytes) Apr 18 17:03:02 charon 07[ENC] <con1|328>parsed IKE_AUTH request 5 [ AUTH ] Apr 18 17:03:02 charon 07[IKE] <con1|328>authentication of '172.20.10.9' with EAP successful Apr 18 17:03:02 charon 07[IKE] <con1|328>authentication of 'xxx.dyndns-web.com' (myself) with EAP Apr 18 17:03:02 charon 07[IKE] <con1|328>IKE_SA con1[328] established between y.y.y.y[xxx.dyndns-web.com]...x.x.x.x[172.20.10.9] Apr 18 17:03:02 charon 07[IKE] <con1|328>scheduling reauthentication in 28169s Apr 18 17:03:02 charon 07[IKE] <con1|328>maximum IKE_SA lifetime 28709s Apr 18 17:03:02 charon 07[IKE] <con1|328>peer requested virtual IP %any Apr 18 17:03:02 charon 07[CFG] <con1|328>reassigning offline lease to 'remoteuser@domain.io' Apr 18 17:03:02 charon 07[IKE] <con1|328>assigning virtual IP 10.10.10.1 to peer 'remoteuser@domain.io' Apr 18 17:03:02 charon 07[IKE] <con1|328>peer requested virtual IP %any6 Apr 18 17:03:02 charon 07[IKE] <con1|328>no virtual IP found for %any6 requested by 'remoteuser@domain.io' Apr 18 17:03:02 charon 07[KNL] <con1|328>error sending to PF_KEY socket: No buffer space available Apr 18 17:03:02 charon 07[KNL] <con1|328>unable to delete SAD entry with SPI cd6f355a Apr 18 17:03:02 charon 07[KNL] <con1|328>deleting SPI allocation SA failed Apr 18 17:03:02 charon 07[KNL] <con1|328>error sending to PF_KEY socket: No buffer space available Apr 18 17:03:02 charon 07[KNL] <con1|328>unable to add SAD entry with SPI cd6f355a Apr 18 17:03:02 charon 07[KNL] <con1|328>error sending to PF_KEY socket: No buffer space available Apr 18 17:03:02 charon 07[KNL] <con1|328>unable to add SAD entry with SPI 0b91cd64 Apr 18 17:03:02 charon 07[IKE] <con1|328>unable to install inbound and outbound IPsec SA (SAD) in kernel Apr 18 17:03:02 charon 07[IKE] <con1|328>failed to establish CHILD_SA, keeping IKE_SA Apr 18 17:03:02 charon 07[KNL] <con1|328>error sending to PF_KEY socket: No buffer space available Apr 18 17:03:02 charon 07[KNL] <con1|328>unable to delete SAD entry with SPI cd6f355a Apr 18 17:03:02 charon 07[KNL] <con1|328>error sending to PF_KEY socket: No buffer space available Apr 18 17:03:02 charon 07[KNL] <con1|328>unable to delete SAD entry with SPI 0b91cd64 Apr 18 17:03:02 charon 07[ENC] <con1|328>generating IKE_AUTH response 5 [ AUTH CPRP(ADDR DNS DNS DNS U_DEFDOM U_SPLITDNS) N(AUTH_LFT) N(MOBIKE_SUP) N(ADD_4_ADDR) N(ADD_4_ADDR) N(ADD_4_ADDR) N(NO_PROP) ] Apr 18 17:03:02 charon 07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (212 bytes) Apr 18 17:03:02 charon 07[NET] <con1|328>received packet: from x.x.x.x[4244] to y.y.y.y[4500] (68 bytes) Apr 18 17:03:02 charon 07[ENC] <con1|328>parsed INFORMATIONAL request 6 [ D ] Apr 18 17:03:02 charon 07[IKE] <con1|328>received DELETE for IKE_SA con1[328] Apr 18 17:03:02 charon 07[IKE] <con1|328>deleting IKE_SA con1[328] between y.y.y.y[xxx.dyndns-web.com]...x.x.x.x[172.20.10.9] Apr 18 17:03:02 charon 07[IKE] <con1|328>IKE_SA deleted Apr 18 17:03:02 charon 07[ENC] <con1|328>generating INFORMATIONAL response 6 [ ] Apr 18 17:03:02 charon 07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (60 bytes) Apr 18 17:03:02 charon 07[CFG] <con1|328>lease 10.10.10.1 by 'remoteuser@domain.io' went offline</con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328>
-
For me I can confirm that removing the OpenBGPD package resolves the problem with the tunnels - IPSEC works fine for me after that. Unfortunately I require OpenBGPD though, so its not a real fix for my situation. Does anyone have any other ideas of what to try?
-
I have now removed openbgpd and rebooted.
Keeping an eye on it the next couple of days. -
Running into what appears to be the same issue. I've installed the patch that CMB put up and I'm testing it out. I don't IPsec a lot, but I'll see about trying to do some more in the next couple days to see if this remains fixed. Luckily I have an OpenVPN connection that I use for other services that I can get back in and stop/start the IPSec service. Below is a sanitized version of the error message:
Apr 18 17:03:02 charon 07[NET] <328> received packet: from x.x.x.x[63521] to y.y.y.y[500] (300 bytes) Apr 18 17:03:02 charon 07[ENC] <328> parsed IKE_SA_INIT request 0 [ SA KE No N(REDIR_SUP) N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) ] Apr 18 17:03:02 charon 07[IKE] <328> x.x.x.x is initiating an IKE_SA Apr 18 17:03:02 charon 07[IKE] <328> remote host is behind NAT Apr 18 17:03:02 charon 07[IKE] <328> sending cert request for "C=US, ST=Wisconsin, L=Madison, O=Xinu, E=zach@xinu.io, CN=dd13-CA" Apr 18 17:03:02 charon 07[ENC] <328> generating IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) CERTREQ N(FRAG_SUP) N(MULT_AUTH) ] Apr 18 17:03:02 charon 07[NET] <328> sending packet: from y.y.y.y[500] to x.x.x.x[63521] (341 bytes) Apr 18 17:03:02 charon 07[NET] <328> received packet: from x.x.x.x[4244] to y.y.y.y[4500] (332 bytes) Apr 18 17:03:02 charon 07[ENC] <328> parsed IKE_AUTH request 1 [ IDi N(INIT_CONTACT) N(MOBIKE_SUP) IDr CPRQ(ADDR DHCP DNS MASK ADDR6 DHCP6 DNS6) N(ESP_TFC_PAD_N) N(NON_FIRST_FRAG) SA TSi TSr ] Apr 18 17:03:02 charon 07[CFG] <328> looking for peer configs matching y.y.y.y[xxx.dyndns-web.com]...x.x.x.x[172.20.10.9] Apr 18 17:03:02 charon 07[CFG] <con1|328>selected peer config 'con1' Apr 18 17:03:02 charon 07[IKE] <con1|328>initiating EAP_IDENTITY method (id 0x00) Apr 18 17:03:02 charon 07[IKE] <con1|328>received ESP_TFC_PADDING_NOT_SUPPORTED, not using ESPv3 TFC padding Apr 18 17:03:02 charon 07[IKE] <con1|328>peer supports MOBIKE Apr 18 17:03:02 charon 07[IKE] <con1|328>authentication of 'xxx.dyndns-web.com' (myself) with RSA signature successful Apr 18 17:03:02 charon 07[IKE] <con1|328>sending end entity cert "C=US, ST=Wisconsin, L=Madison, O=Xinu, E=zach@xinu.io, CN=xxx.dyndns-web.com" Apr 18 17:03:02 charon 07[ENC] <con1|328>generating IKE_AUTH response 1 [ IDr CERT AUTH EAP/REQ/ID ] Apr 18 17:03:02 charon 07[ENC] <con1|328>splitting IKE message with length of 1596 bytes into 4 fragments Apr 18 17:03:02 charon 07[ENC] <con1|328>generating IKE_AUTH response 1 [ EF(1/4) ] Apr 18 17:03:02 charon 07[ENC] <con1|328>generating IKE_AUTH response 1 [ EF(2/4) ] Apr 18 17:03:02 charon 07[ENC] <con1|328>generating IKE_AUTH response 1 [ EF(3/4) ] Apr 18 17:03:02 charon 07[ENC] <con1|328>generating IKE_AUTH response 1 [ EF(4/4) ] Apr 18 17:03:02 charon 07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (544 bytes) Apr 18 17:03:02 charon 07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (544 bytes) Apr 18 17:03:02 charon 07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (544 bytes) Apr 18 17:03:02 charon 07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (144 bytes) Apr 18 17:03:02 charon 07[NET] <con1|328>received packet: from x.x.x.x[4244] to y.y.y.y[4500] (84 bytes) Apr 18 17:03:02 charon 07[ENC] <con1|328>parsed IKE_AUTH request 2 [ EAP/RES/ID ] Apr 18 17:03:02 charon 07[IKE] <con1|328>received EAP identity 'remoteuser@domain.io' Apr 18 17:03:02 charon 07[IKE] <con1|328>initiating EAP_MSCHAPV2 method (id 0x14) Apr 18 17:03:02 charon 07[ENC] <con1|328>generating IKE_AUTH response 2 [ EAP/REQ/MSCHAPV2 ] Apr 18 17:03:02 charon 07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (100 bytes) Apr 18 17:03:02 charon 07[NET] <con1|328>received packet: from x.x.x.x[4244] to y.y.y.y[4500] (140 bytes) Apr 18 17:03:02 charon 07[ENC] <con1|328>parsed IKE_AUTH request 3 [ EAP/RES/MSCHAPV2 ] Apr 18 17:03:02 charon 07[ENC] <con1|328>generating IKE_AUTH response 3 [ EAP/REQ/MSCHAPV2 ] Apr 18 17:03:02 charon 07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (132 bytes) Apr 18 17:03:02 charon 07[NET] <con1|328>received packet: from x.x.x.x[4244] to y.y.y.y[4500] (68 bytes) Apr 18 17:03:02 charon 07[ENC] <con1|328>parsed IKE_AUTH request 4 [ EAP/RES/MSCHAPV2 ] Apr 18 17:03:02 charon 07[IKE] <con1|328>EAP method EAP_MSCHAPV2 succeeded, MSK established Apr 18 17:03:02 charon 07[ENC] <con1|328>generating IKE_AUTH response 4 [ EAP/SUCC ] Apr 18 17:03:02 charon 07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (68 bytes) Apr 18 17:03:02 charon 07[NET] <con1|328>received packet: from x.x.x.x[4244] to y.y.y.y[4500] (84 bytes) Apr 18 17:03:02 charon 07[ENC] <con1|328>parsed IKE_AUTH request 5 [ AUTH ] Apr 18 17:03:02 charon 07[IKE] <con1|328>authentication of '172.20.10.9' with EAP successful Apr 18 17:03:02 charon 07[IKE] <con1|328>authentication of 'xxx.dyndns-web.com' (myself) with EAP Apr 18 17:03:02 charon 07[IKE] <con1|328>IKE_SA con1[328] established between y.y.y.y[xxx.dyndns-web.com]...x.x.x.x[172.20.10.9] Apr 18 17:03:02 charon 07[IKE] <con1|328>scheduling reauthentication in 28169s Apr 18 17:03:02 charon 07[IKE] <con1|328>maximum IKE_SA lifetime 28709s Apr 18 17:03:02 charon 07[IKE] <con1|328>peer requested virtual IP %any Apr 18 17:03:02 charon 07[CFG] <con1|328>reassigning offline lease to 'remoteuser@domain.io' Apr 18 17:03:02 charon 07[IKE] <con1|328>assigning virtual IP 10.10.10.1 to peer 'remoteuser@domain.io' Apr 18 17:03:02 charon 07[IKE] <con1|328>peer requested virtual IP %any6 Apr 18 17:03:02 charon 07[IKE] <con1|328>no virtual IP found for %any6 requested by 'remoteuser@domain.io' Apr 18 17:03:02 charon 07[KNL] <con1|328>error sending to PF_KEY socket: No buffer space available Apr 18 17:03:02 charon 07[KNL] <con1|328>unable to delete SAD entry with SPI cd6f355a Apr 18 17:03:02 charon 07[KNL] <con1|328>deleting SPI allocation SA failed Apr 18 17:03:02 charon 07[KNL] <con1|328>error sending to PF_KEY socket: No buffer space available Apr 18 17:03:02 charon 07[KNL] <con1|328>unable to add SAD entry with SPI cd6f355a Apr 18 17:03:02 charon 07[KNL] <con1|328>error sending to PF_KEY socket: No buffer space available Apr 18 17:03:02 charon 07[KNL] <con1|328>unable to add SAD entry with SPI 0b91cd64 Apr 18 17:03:02 charon 07[IKE] <con1|328>unable to install inbound and outbound IPsec SA (SAD) in kernel Apr 18 17:03:02 charon 07[IKE] <con1|328>failed to establish CHILD_SA, keeping IKE_SA Apr 18 17:03:02 charon 07[KNL] <con1|328>error sending to PF_KEY socket: No buffer space available Apr 18 17:03:02 charon 07[KNL] <con1|328>unable to delete SAD entry with SPI cd6f355a Apr 18 17:03:02 charon 07[KNL] <con1|328>error sending to PF_KEY socket: No buffer space available Apr 18 17:03:02 charon 07[KNL] <con1|328>unable to delete SAD entry with SPI 0b91cd64 Apr 18 17:03:02 charon 07[ENC] <con1|328>generating IKE_AUTH response 5 [ AUTH CPRP(ADDR DNS DNS DNS U_DEFDOM U_SPLITDNS) N(AUTH_LFT) N(MOBIKE_SUP) N(ADD_4_ADDR) N(ADD_4_ADDR) N(ADD_4_ADDR) N(NO_PROP) ] Apr 18 17:03:02 charon 07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (212 bytes) Apr 18 17:03:02 charon 07[NET] <con1|328>received packet: from x.x.x.x[4244] to y.y.y.y[4500] (68 bytes) Apr 18 17:03:02 charon 07[ENC] <con1|328>parsed INFORMATIONAL request 6 [ D ] Apr 18 17:03:02 charon 07[IKE] <con1|328>received DELETE for IKE_SA con1[328] Apr 18 17:03:02 charon 07[IKE] <con1|328>deleting IKE_SA con1[328] between y.y.y.y[xxx.dyndns-web.com]...x.x.x.x[172.20.10.9] Apr 18 17:03:02 charon 07[IKE] <con1|328>IKE_SA deleted Apr 18 17:03:02 charon 07[ENC] <con1|328>generating INFORMATIONAL response 6 [ ] Apr 18 17:03:02 charon 07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (60 bytes) Apr 18 17:03:02 charon 07[CFG] <con1|328>lease 10.10.10.1 by 'remoteuser@domain.io' went offline</con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328>
Issue reoccurred for me today again. Had to connect back through OpenVPN tunnel and manually stop/start the IPSec service in order to authenticate via IPSec from my Macbook air. It doesn't appear the fix from earlier is working for me on a permanent basis.
-
There are two separate issues here with the same symptom. The starting twice problem is fixed by what I posted earlier in the thread. The issue with openbgpd causing that same PF_KEY error doesn't have a known cause or solution yet. I'm attempting to replicate that one.
-
cmb, if you would like access to my pfSense server that has the OpenBGPD issue again let me know and I will message you the details.
-
I'm not running OpenBGPD. I'll review, but I'm pretty sure I applied that patch successfully utilizing the patch package utility…
Edit: It's definitely installed. I navigated out to the file in question to check for the additions that were added and they're there.
I haven't done a full reboot, would that have any chance of affecting the application of the fix?
-
I'm not running OpenBGPD. I'll review, but I'm pretty sure I applied that patch successfully utilizing the patch package utility…
Edit: It's definitely installed. I navigated out to the file in question to check for the additions that were added and they're there.
I haven't done a full reboot, would that have any chance of affecting the application of the fix?
You have to either manually kill off the duplicate instances of strongswan (ipsec and charon processes), or reboot after applying that. That just prevents the problem circumstance from happening again.
-
@cmb:
I'm not running OpenBGPD. I'll review, but I'm pretty sure I applied that patch successfully utilizing the patch package utility…
Edit: It's definitely installed. I navigated out to the file in question to check for the additions that were added and they're there.
I haven't done a full reboot, would that have any chance of affecting the application of the fix?
You have to either manually kill off the duplicate instances of strongswan (ipsec and charon processes), or reboot after applying that. That just prevents the problem circumstance from happening again.
I'm going to keep an eye on it. I rebooted today, however I also just rebuilt my pfSense and am running it virtually now in VMWare with essentially the same configuration.
-
@cmb:
I'm not running OpenBGPD. I'll review, but I'm pretty sure I applied that patch successfully utilizing the patch package utility…
Edit: It's definitely installed. I navigated out to the file in question to check for the additions that were added and they're there.
I haven't done a full reboot, would that have any chance of affecting the application of the fix?
You have to either manually kill off the duplicate instances of strongswan (ipsec and charon processes), or reboot after applying that. That just prevents the problem circumstance from happening again.
I'm going to keep an eye on it. I rebooted today, however I also just rebuilt my pfSense and am running it virtually now in VMWare with essentially the same configuration.
This would have been a great weekend to test this, however my Macbook has decided that it doesn't want to run IKEv2 anymore…
Apr 22 07:34:01 mba nesessionmanager[427]: NESMIKEv2VPNSession[xxxx.dyndns-web.com:4B453608-183B-4727-B69F-DB98737FCCD7]: Received a start command from SystemUIServer[239] Apr 22 07:34:01 mba nesessionmanager[427]: NESMIKEv2VPNSession[xxxx.dyndns-web.com:4B453608-183B-4727-B69F-DB98737FCCD7]: status changed to connecting Apr 22 07:34:01 mba nesessionmanager[427]: Failed to find the VPN app for plugin type com.apple.neplugin.IKEv2 Apr 22 07:34:01 mba neagent[926]: IKEv2 Plugin: ikev2_dns_callback: Error -65554 Apr 22 07:34:02 mba kernel[0]: ipsec_ctl_connect: creating interface ipsec0 Apr 22 07:34:02 mba configd[51]: network changed Apr 22 07:34:04 mba neagent[926]: MSCHAPv2 Error = 691, Retry = 1, Version = 0 Apr 22 07:34:04 mba neagent[926]: Failed to process IKE Auth (EAP) packet Apr 22 07:34:04 mba neagent[926]: BUG in libdispatch client: kevent[EVFILT_READ] delete: "Bad file descriptor" - 0x9 Apr 22 07:34:04 mba nesessionmanager[427]: NESMIKEv2VPNSession[xxxx.dyndns-web.com:4B453608-183B-4727-B69F-DB98737FCCD7]: status changed to disconnecting Apr 22 07:34:04 mba kernel[0]: SIOCPROTODETACH_IN6: ipsec0 error=6 Apr 22 07:34:04 mba nesessionmanager[427]: NESMIKEv2VPNSession[xxxx.dyndns-web.com:4B453608-183B-4727-B69F-DB98737FCCD7]: status changed to disconnected, last stop reason $ Apr 22 07:34:04 mba configd[51]: network changed Apr 22 07:34:04 mba symptomsd[422]: nw_interface_get_agents SIOCGIFAGENTIDS failed for interface "ipsec0" (index 8, type other): [6] Device not configured
Of course there a good amount of posts showing this as a problem on google, but no real resolutions… My personal and work iPhone connect without a hitch, so it's definitely not pfSense side :(
-
I am in the same boat. I have two pfSense boxes in an HA pair running 2.3, with BGP and an IPSec VPN. I'm happy to help test whatever patch etc as needed.
-
Can now confirm that after removing openbgpd I havn't have had any IPSEC tunnels inactive.
-
Hi All ,
I am a newbie in pfsense , i have recently updated on pfsense 2.3 (AMD 64) ,all the things were working like a charm including IPSEC tunnels before installing OpenBgpd, as per our requirement i have installed OpenBGPD both IPsec and OpenBgpd worked for some hours ….....then all the IPSEC tunnels gone down , In Gui configuration tunnels still showing in established state but no traffic passing , for fixing the issue i have to restart the firewall , then it again starts working for some hours .
I have tried these below steps also but all goes in vain :'(
killall -9 charon
killall -9 starter
ipsec stop
ipsec startipsec start states as :-
Starting strongSwan 5.4.0 IPsec [starter]…
charon is already running (/var/run/charon.pid exists) -- skipping daemon start
no netkey IPsec stack detected
no KLIPS IPsec stack detected
no known IPsec stack detected, ignoring!
starter is already running (/var/run/starter.charon.pid exists) -- no fork donePlease suggest anybody i am using pfsense in production environment :'(
I was wondering if a downgrade to previous version can fix this issue , is any version is workable with both IPSEC and OPENBGPD.
Please if anyone have any idea on this . -
Hi All ,
I am a newbie in pfsense , i have recently updated on pfsense 2.3 (AMD 64) ,all the things were working like a charm including IPSEC tunnels before installing OpenBgpd, as per our requirement i have installed OpenBGPD both IPsec and OpenBgpd worked for some hours ….....then all the IPSEC tunnels gone down , In Gui configuration tunnels still showing in established state but no traffic passing , for fixing the issue i have to restart the firewall , then it again starts working for some hours .
I have tried these below steps also but all goes in vain :'(
killall -9 charon
killall -9 starter
ipsec stop
ipsec startipsec start states as :-
Starting strongSwan 5.4.0 IPsec [starter]…
charon is already running (/var/run/charon.pid exists) -- skipping daemon start
no netkey IPsec stack detected
no KLIPS IPsec stack detected
no known IPsec stack detected, ignoring!
starter is already running (/var/run/starter.charon.pid exists) -- no fork donePlease suggest anybody i am using pfsense in production environment :'(
I was wondering if a downgrade to previous version can fix this issue , is any version is workable with both IPSEC and OPENBGPD.
Please if anyone have any idea on this .Dude, you shouldnt have updated a prod system to 2.3 without testing!
Anyway, yes i can confirm 2.2.6 works perfectly with IPSEC and openbgpd. Im using it myself on a prod network.
There is an open bug report for this issue: https://redmine.pfsense.org/issues/6223
-
Thanks fattylewis , i have downgraded my pfsense box to 2.2.6 , now everything is working fine. :) ;D
there is one more thing to notice :- when i had edited /boot/loder.config.local >>>> net.inet.raw.maxdgram="131072"
net.inet.raw.recvspace="131072" .Pfsense worked like charm with ipsec and bgp even on pfsense 2.3. :)
for me that trick worked.
Thanks CMB and fattylewis for your replies….you guys rocksssss.. ;)
-
Thanks fattylewis , i have downgraded my pfsense box to 2.2.6 , now everything is working fine. :) ;D
there is one more thing to notice :- when i had edited /boot/loder.config.local >>>> net.inet.raw.maxdgram="131072"
net.inet.raw.recvspace="131072" .Pfsense worked like charm with ipsec and bgp even on pfsense 2.3. :)
for me that trick worked.
Thanks CMB and fattylewis for your replies….you guys rocksssss.. ;)
Oh, nice find. Ill see about knocking up another network on 2.3 and adding your change and seeing what happens.
-
We've also had this issue on 2.3, and as we required BGP for our network, we've downgraded back to 2.2.6
Looking forward to a confirmed fix (need to wait until after hours again to try the upgrade again)
-
I'm having the same problem with OpenBGP and IPSec.
Restarted the following services:
-OpenBGP
-IPSecNo luck. Only rebooting worked.
Then tried restarting:
-OpenBGP
-IPSec
-OpenVPNTunnel came back up.
Not sure if that helps some of the developers with troubleshooting.
I have stopped the OpenVPN service for now and will see if the issue returns.
UPDATE: Still having the issue even after disabling OpenVPN
-
Same issue, pair of SG-8860s with CARP failover, dual IPSec tunnels to a Verizon Private Network with OpenBGPd required for their routing. Exact same errors, even changing both tunables:
net.inet.raw.maxdgram="131072"
net.inet.raw.recvspace="131072"May extend the time, but definitely doesn't solve. Really don't want to go back to 2.2.6 :)