IPSec Down after Upgrade to 2.3



  • Hello!

    We have the same problem, applied the patch, however as we use OpenBGPD for AWS, ALL the IPSEC vpn's drop.

    The only way we fix it at the moment is to reboot the Firewall (this is not ideal).

    Any suggestions?

    Thanks



  • Running into what appears to be the same issue.  I've installed the patch that CMB put up and I'm testing it out.  I don't IPsec a lot, but I'll see about trying to do some more in the next couple days to see if this remains fixed.  Luckily I have an OpenVPN connection that I use for other services that I can get back in and stop/start the IPSec service.  Below is a sanitized version of the error message:

    
    Apr 18 17:03:02	charon		07[NET] <328> received packet: from x.x.x.x[63521] to y.y.y.y[500] (300 bytes)
    Apr 18 17:03:02	charon		07[ENC] <328> parsed IKE_SA_INIT request 0 [ SA KE No N(REDIR_SUP) N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) ]
    Apr 18 17:03:02	charon		07[IKE] <328> x.x.x.x is initiating an IKE_SA
    Apr 18 17:03:02	charon		07[IKE] <328> remote host is behind NAT
    Apr 18 17:03:02	charon		07[IKE] <328> sending cert request for "C=US, ST=Wisconsin, L=Madison, O=Xinu, E=zach@xinu.io, CN=dd13-CA"
    Apr 18 17:03:02	charon		07[ENC] <328> generating IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) CERTREQ N(FRAG_SUP) N(MULT_AUTH) ]
    Apr 18 17:03:02	charon		07[NET] <328> sending packet: from y.y.y.y[500] to x.x.x.x[63521] (341 bytes)
    Apr 18 17:03:02	charon		07[NET] <328> received packet: from x.x.x.x[4244] to y.y.y.y[4500] (332 bytes)
    Apr 18 17:03:02	charon		07[ENC] <328> parsed IKE_AUTH request 1 [ IDi N(INIT_CONTACT) N(MOBIKE_SUP) IDr CPRQ(ADDR DHCP DNS MASK ADDR6 DHCP6 DNS6) N(ESP_TFC_PAD_N) N(NON_FIRST_FRAG) SA TSi TSr ]
    Apr 18 17:03:02	charon		07[CFG] <328> looking for peer configs matching y.y.y.y[xxx.dyndns-web.com]...x.x.x.x[172.20.10.9]
    Apr 18 17:03:02	charon		07[CFG] <con1|328>selected peer config 'con1'
    Apr 18 17:03:02	charon		07[IKE] <con1|328>initiating EAP_IDENTITY method (id 0x00)
    Apr 18 17:03:02	charon		07[IKE] <con1|328>received ESP_TFC_PADDING_NOT_SUPPORTED, not using ESPv3 TFC padding
    Apr 18 17:03:02	charon		07[IKE] <con1|328>peer supports MOBIKE
    Apr 18 17:03:02	charon		07[IKE] <con1|328>authentication of 'xxx.dyndns-web.com' (myself) with RSA signature successful
    Apr 18 17:03:02	charon		07[IKE] <con1|328>sending end entity cert "C=US, ST=Wisconsin, L=Madison, O=Xinu, E=zach@xinu.io, CN=xxx.dyndns-web.com"
    Apr 18 17:03:02	charon		07[ENC] <con1|328>generating IKE_AUTH response 1 [ IDr CERT AUTH EAP/REQ/ID ]
    Apr 18 17:03:02	charon		07[ENC] <con1|328>splitting IKE message with length of 1596 bytes into 4 fragments
    Apr 18 17:03:02	charon		07[ENC] <con1|328>generating IKE_AUTH response 1 [ EF(1/4) ]
    Apr 18 17:03:02	charon		07[ENC] <con1|328>generating IKE_AUTH response 1 [ EF(2/4) ]
    Apr 18 17:03:02	charon		07[ENC] <con1|328>generating IKE_AUTH response 1 [ EF(3/4) ]
    Apr 18 17:03:02	charon		07[ENC] <con1|328>generating IKE_AUTH response 1 [ EF(4/4) ]
    Apr 18 17:03:02	charon		07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (544 bytes)
    Apr 18 17:03:02	charon		07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (544 bytes)
    Apr 18 17:03:02	charon		07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (544 bytes)
    Apr 18 17:03:02	charon		07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (144 bytes)
    Apr 18 17:03:02	charon		07[NET] <con1|328>received packet: from x.x.x.x[4244] to y.y.y.y[4500] (84 bytes)
    Apr 18 17:03:02	charon		07[ENC] <con1|328>parsed IKE_AUTH request 2 [ EAP/RES/ID ]
    Apr 18 17:03:02	charon		07[IKE] <con1|328>received EAP identity 'remoteuser@domain.io'
    Apr 18 17:03:02	charon		07[IKE] <con1|328>initiating EAP_MSCHAPV2 method (id 0x14)
    Apr 18 17:03:02	charon		07[ENC] <con1|328>generating IKE_AUTH response 2 [ EAP/REQ/MSCHAPV2 ]
    Apr 18 17:03:02	charon		07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (100 bytes)
    Apr 18 17:03:02	charon		07[NET] <con1|328>received packet: from x.x.x.x[4244] to y.y.y.y[4500] (140 bytes)
    Apr 18 17:03:02	charon		07[ENC] <con1|328>parsed IKE_AUTH request 3 [ EAP/RES/MSCHAPV2 ]
    Apr 18 17:03:02	charon		07[ENC] <con1|328>generating IKE_AUTH response 3 [ EAP/REQ/MSCHAPV2 ]
    Apr 18 17:03:02	charon		07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (132 bytes)
    Apr 18 17:03:02	charon		07[NET] <con1|328>received packet: from x.x.x.x[4244] to y.y.y.y[4500] (68 bytes)
    Apr 18 17:03:02	charon		07[ENC] <con1|328>parsed IKE_AUTH request 4 [ EAP/RES/MSCHAPV2 ]
    Apr 18 17:03:02	charon		07[IKE] <con1|328>EAP method EAP_MSCHAPV2 succeeded, MSK established
    Apr 18 17:03:02	charon		07[ENC] <con1|328>generating IKE_AUTH response 4 [ EAP/SUCC ]
    Apr 18 17:03:02	charon		07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (68 bytes)
    Apr 18 17:03:02	charon		07[NET] <con1|328>received packet: from x.x.x.x[4244] to y.y.y.y[4500] (84 bytes)
    Apr 18 17:03:02	charon		07[ENC] <con1|328>parsed IKE_AUTH request 5 [ AUTH ]
    Apr 18 17:03:02	charon		07[IKE] <con1|328>authentication of '172.20.10.9' with EAP successful
    Apr 18 17:03:02	charon		07[IKE] <con1|328>authentication of 'xxx.dyndns-web.com' (myself) with EAP
    Apr 18 17:03:02	charon		07[IKE] <con1|328>IKE_SA con1[328] established between y.y.y.y[xxx.dyndns-web.com]...x.x.x.x[172.20.10.9]
    Apr 18 17:03:02	charon		07[IKE] <con1|328>scheduling reauthentication in 28169s
    Apr 18 17:03:02	charon		07[IKE] <con1|328>maximum IKE_SA lifetime 28709s
    Apr 18 17:03:02	charon		07[IKE] <con1|328>peer requested virtual IP %any
    Apr 18 17:03:02	charon		07[CFG] <con1|328>reassigning offline lease to 'remoteuser@domain.io'
    Apr 18 17:03:02	charon		07[IKE] <con1|328>assigning virtual IP 10.10.10.1 to peer 'remoteuser@domain.io'
    Apr 18 17:03:02	charon		07[IKE] <con1|328>peer requested virtual IP %any6
    Apr 18 17:03:02	charon		07[IKE] <con1|328>no virtual IP found for %any6 requested by 'remoteuser@domain.io'
    Apr 18 17:03:02	charon		07[KNL] <con1|328>error sending to PF_KEY socket: No buffer space available
    Apr 18 17:03:02	charon		07[KNL] <con1|328>unable to delete SAD entry with SPI cd6f355a
    Apr 18 17:03:02	charon		07[KNL] <con1|328>deleting SPI allocation SA failed
    Apr 18 17:03:02	charon		07[KNL] <con1|328>error sending to PF_KEY socket: No buffer space available
    Apr 18 17:03:02	charon		07[KNL] <con1|328>unable to add SAD entry with SPI cd6f355a
    Apr 18 17:03:02	charon		07[KNL] <con1|328>error sending to PF_KEY socket: No buffer space available
    Apr 18 17:03:02	charon		07[KNL] <con1|328>unable to add SAD entry with SPI 0b91cd64
    Apr 18 17:03:02	charon		07[IKE] <con1|328>unable to install inbound and outbound IPsec SA (SAD) in kernel
    Apr 18 17:03:02	charon		07[IKE] <con1|328>failed to establish CHILD_SA, keeping IKE_SA
    Apr 18 17:03:02	charon		07[KNL] <con1|328>error sending to PF_KEY socket: No buffer space available
    Apr 18 17:03:02	charon		07[KNL] <con1|328>unable to delete SAD entry with SPI cd6f355a
    Apr 18 17:03:02	charon		07[KNL] <con1|328>error sending to PF_KEY socket: No buffer space available
    Apr 18 17:03:02	charon		07[KNL] <con1|328>unable to delete SAD entry with SPI 0b91cd64
    Apr 18 17:03:02	charon		07[ENC] <con1|328>generating IKE_AUTH response 5 [ AUTH CPRP(ADDR DNS DNS DNS U_DEFDOM U_SPLITDNS) N(AUTH_LFT) N(MOBIKE_SUP) N(ADD_4_ADDR) N(ADD_4_ADDR) N(ADD_4_ADDR) N(NO_PROP) ]
    Apr 18 17:03:02	charon		07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (212 bytes)
    Apr 18 17:03:02	charon		07[NET] <con1|328>received packet: from x.x.x.x[4244] to y.y.y.y[4500] (68 bytes)
    Apr 18 17:03:02	charon		07[ENC] <con1|328>parsed INFORMATIONAL request 6 [ D ]
    Apr 18 17:03:02	charon		07[IKE] <con1|328>received DELETE for IKE_SA con1[328]
    Apr 18 17:03:02	charon		07[IKE] <con1|328>deleting IKE_SA con1[328] between y.y.y.y[xxx.dyndns-web.com]...x.x.x.x[172.20.10.9]
    Apr 18 17:03:02	charon		07[IKE] <con1|328>IKE_SA deleted
    Apr 18 17:03:02	charon		07[ENC] <con1|328>generating INFORMATIONAL response 6 [ ]
    Apr 18 17:03:02	charon		07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (60 bytes)
    Apr 18 17:03:02	charon		07[CFG] <con1|328>lease 10.10.10.1 by 'remoteuser@domain.io' went offline</con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328> 
    


  • For me I can confirm that removing the OpenBGPD package resolves the problem with the tunnels - IPSEC works fine for me after that. Unfortunately I require OpenBGPD though, so its not a real fix for my situation. Does anyone have any other ideas of what to try?



  • I have now removed openbgpd and rebooted.
    Keeping an eye on it the next couple of days.



  • @vsxi-13:

    Running into what appears to be the same issue.  I've installed the patch that CMB put up and I'm testing it out.  I don't IPsec a lot, but I'll see about trying to do some more in the next couple days to see if this remains fixed.  Luckily I have an OpenVPN connection that I use for other services that I can get back in and stop/start the IPSec service.  Below is a sanitized version of the error message:

    
    Apr 18 17:03:02	charon		07[NET] <328> received packet: from x.x.x.x[63521] to y.y.y.y[500] (300 bytes)
    Apr 18 17:03:02	charon		07[ENC] <328> parsed IKE_SA_INIT request 0 [ SA KE No N(REDIR_SUP) N(NATD_S_IP) N(NATD_D_IP) N(FRAG_SUP) ]
    Apr 18 17:03:02	charon		07[IKE] <328> x.x.x.x is initiating an IKE_SA
    Apr 18 17:03:02	charon		07[IKE] <328> remote host is behind NAT
    Apr 18 17:03:02	charon		07[IKE] <328> sending cert request for "C=US, ST=Wisconsin, L=Madison, O=Xinu, E=zach@xinu.io, CN=dd13-CA"
    Apr 18 17:03:02	charon		07[ENC] <328> generating IKE_SA_INIT response 0 [ SA KE No N(NATD_S_IP) N(NATD_D_IP) CERTREQ N(FRAG_SUP) N(MULT_AUTH) ]
    Apr 18 17:03:02	charon		07[NET] <328> sending packet: from y.y.y.y[500] to x.x.x.x[63521] (341 bytes)
    Apr 18 17:03:02	charon		07[NET] <328> received packet: from x.x.x.x[4244] to y.y.y.y[4500] (332 bytes)
    Apr 18 17:03:02	charon		07[ENC] <328> parsed IKE_AUTH request 1 [ IDi N(INIT_CONTACT) N(MOBIKE_SUP) IDr CPRQ(ADDR DHCP DNS MASK ADDR6 DHCP6 DNS6) N(ESP_TFC_PAD_N) N(NON_FIRST_FRAG) SA TSi TSr ]
    Apr 18 17:03:02	charon		07[CFG] <328> looking for peer configs matching y.y.y.y[xxx.dyndns-web.com]...x.x.x.x[172.20.10.9]
    Apr 18 17:03:02	charon		07[CFG] <con1|328>selected peer config 'con1'
    Apr 18 17:03:02	charon		07[IKE] <con1|328>initiating EAP_IDENTITY method (id 0x00)
    Apr 18 17:03:02	charon		07[IKE] <con1|328>received ESP_TFC_PADDING_NOT_SUPPORTED, not using ESPv3 TFC padding
    Apr 18 17:03:02	charon		07[IKE] <con1|328>peer supports MOBIKE
    Apr 18 17:03:02	charon		07[IKE] <con1|328>authentication of 'xxx.dyndns-web.com' (myself) with RSA signature successful
    Apr 18 17:03:02	charon		07[IKE] <con1|328>sending end entity cert "C=US, ST=Wisconsin, L=Madison, O=Xinu, E=zach@xinu.io, CN=xxx.dyndns-web.com"
    Apr 18 17:03:02	charon		07[ENC] <con1|328>generating IKE_AUTH response 1 [ IDr CERT AUTH EAP/REQ/ID ]
    Apr 18 17:03:02	charon		07[ENC] <con1|328>splitting IKE message with length of 1596 bytes into 4 fragments
    Apr 18 17:03:02	charon		07[ENC] <con1|328>generating IKE_AUTH response 1 [ EF(1/4) ]
    Apr 18 17:03:02	charon		07[ENC] <con1|328>generating IKE_AUTH response 1 [ EF(2/4) ]
    Apr 18 17:03:02	charon		07[ENC] <con1|328>generating IKE_AUTH response 1 [ EF(3/4) ]
    Apr 18 17:03:02	charon		07[ENC] <con1|328>generating IKE_AUTH response 1 [ EF(4/4) ]
    Apr 18 17:03:02	charon		07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (544 bytes)
    Apr 18 17:03:02	charon		07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (544 bytes)
    Apr 18 17:03:02	charon		07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (544 bytes)
    Apr 18 17:03:02	charon		07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (144 bytes)
    Apr 18 17:03:02	charon		07[NET] <con1|328>received packet: from x.x.x.x[4244] to y.y.y.y[4500] (84 bytes)
    Apr 18 17:03:02	charon		07[ENC] <con1|328>parsed IKE_AUTH request 2 [ EAP/RES/ID ]
    Apr 18 17:03:02	charon		07[IKE] <con1|328>received EAP identity 'remoteuser@domain.io'
    Apr 18 17:03:02	charon		07[IKE] <con1|328>initiating EAP_MSCHAPV2 method (id 0x14)
    Apr 18 17:03:02	charon		07[ENC] <con1|328>generating IKE_AUTH response 2 [ EAP/REQ/MSCHAPV2 ]
    Apr 18 17:03:02	charon		07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (100 bytes)
    Apr 18 17:03:02	charon		07[NET] <con1|328>received packet: from x.x.x.x[4244] to y.y.y.y[4500] (140 bytes)
    Apr 18 17:03:02	charon		07[ENC] <con1|328>parsed IKE_AUTH request 3 [ EAP/RES/MSCHAPV2 ]
    Apr 18 17:03:02	charon		07[ENC] <con1|328>generating IKE_AUTH response 3 [ EAP/REQ/MSCHAPV2 ]
    Apr 18 17:03:02	charon		07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (132 bytes)
    Apr 18 17:03:02	charon		07[NET] <con1|328>received packet: from x.x.x.x[4244] to y.y.y.y[4500] (68 bytes)
    Apr 18 17:03:02	charon		07[ENC] <con1|328>parsed IKE_AUTH request 4 [ EAP/RES/MSCHAPV2 ]
    Apr 18 17:03:02	charon		07[IKE] <con1|328>EAP method EAP_MSCHAPV2 succeeded, MSK established
    Apr 18 17:03:02	charon		07[ENC] <con1|328>generating IKE_AUTH response 4 [ EAP/SUCC ]
    Apr 18 17:03:02	charon		07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (68 bytes)
    Apr 18 17:03:02	charon		07[NET] <con1|328>received packet: from x.x.x.x[4244] to y.y.y.y[4500] (84 bytes)
    Apr 18 17:03:02	charon		07[ENC] <con1|328>parsed IKE_AUTH request 5 [ AUTH ]
    Apr 18 17:03:02	charon		07[IKE] <con1|328>authentication of '172.20.10.9' with EAP successful
    Apr 18 17:03:02	charon		07[IKE] <con1|328>authentication of 'xxx.dyndns-web.com' (myself) with EAP
    Apr 18 17:03:02	charon		07[IKE] <con1|328>IKE_SA con1[328] established between y.y.y.y[xxx.dyndns-web.com]...x.x.x.x[172.20.10.9]
    Apr 18 17:03:02	charon		07[IKE] <con1|328>scheduling reauthentication in 28169s
    Apr 18 17:03:02	charon		07[IKE] <con1|328>maximum IKE_SA lifetime 28709s
    Apr 18 17:03:02	charon		07[IKE] <con1|328>peer requested virtual IP %any
    Apr 18 17:03:02	charon		07[CFG] <con1|328>reassigning offline lease to 'remoteuser@domain.io'
    Apr 18 17:03:02	charon		07[IKE] <con1|328>assigning virtual IP 10.10.10.1 to peer 'remoteuser@domain.io'
    Apr 18 17:03:02	charon		07[IKE] <con1|328>peer requested virtual IP %any6
    Apr 18 17:03:02	charon		07[IKE] <con1|328>no virtual IP found for %any6 requested by 'remoteuser@domain.io'
    Apr 18 17:03:02	charon		07[KNL] <con1|328>error sending to PF_KEY socket: No buffer space available
    Apr 18 17:03:02	charon		07[KNL] <con1|328>unable to delete SAD entry with SPI cd6f355a
    Apr 18 17:03:02	charon		07[KNL] <con1|328>deleting SPI allocation SA failed
    Apr 18 17:03:02	charon		07[KNL] <con1|328>error sending to PF_KEY socket: No buffer space available
    Apr 18 17:03:02	charon		07[KNL] <con1|328>unable to add SAD entry with SPI cd6f355a
    Apr 18 17:03:02	charon		07[KNL] <con1|328>error sending to PF_KEY socket: No buffer space available
    Apr 18 17:03:02	charon		07[KNL] <con1|328>unable to add SAD entry with SPI 0b91cd64
    Apr 18 17:03:02	charon		07[IKE] <con1|328>unable to install inbound and outbound IPsec SA (SAD) in kernel
    Apr 18 17:03:02	charon		07[IKE] <con1|328>failed to establish CHILD_SA, keeping IKE_SA
    Apr 18 17:03:02	charon		07[KNL] <con1|328>error sending to PF_KEY socket: No buffer space available
    Apr 18 17:03:02	charon		07[KNL] <con1|328>unable to delete SAD entry with SPI cd6f355a
    Apr 18 17:03:02	charon		07[KNL] <con1|328>error sending to PF_KEY socket: No buffer space available
    Apr 18 17:03:02	charon		07[KNL] <con1|328>unable to delete SAD entry with SPI 0b91cd64
    Apr 18 17:03:02	charon		07[ENC] <con1|328>generating IKE_AUTH response 5 [ AUTH CPRP(ADDR DNS DNS DNS U_DEFDOM U_SPLITDNS) N(AUTH_LFT) N(MOBIKE_SUP) N(ADD_4_ADDR) N(ADD_4_ADDR) N(ADD_4_ADDR) N(NO_PROP) ]
    Apr 18 17:03:02	charon		07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (212 bytes)
    Apr 18 17:03:02	charon		07[NET] <con1|328>received packet: from x.x.x.x[4244] to y.y.y.y[4500] (68 bytes)
    Apr 18 17:03:02	charon		07[ENC] <con1|328>parsed INFORMATIONAL request 6 [ D ]
    Apr 18 17:03:02	charon		07[IKE] <con1|328>received DELETE for IKE_SA con1[328]
    Apr 18 17:03:02	charon		07[IKE] <con1|328>deleting IKE_SA con1[328] between y.y.y.y[xxx.dyndns-web.com]...x.x.x.x[172.20.10.9]
    Apr 18 17:03:02	charon		07[IKE] <con1|328>IKE_SA deleted
    Apr 18 17:03:02	charon		07[ENC] <con1|328>generating INFORMATIONAL response 6 [ ]
    Apr 18 17:03:02	charon		07[NET] <con1|328>sending packet: from y.y.y.y[4500] to x.x.x.x[4244] (60 bytes)
    Apr 18 17:03:02	charon		07[CFG] <con1|328>lease 10.10.10.1 by 'remoteuser@domain.io' went offline</con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328></con1|328> 
    

    Issue reoccurred for me today again.  Had to connect back through OpenVPN tunnel and manually stop/start the IPSec service in order to authenticate via IPSec from my Macbook air.  It doesn't appear the fix from earlier is working for me on a permanent basis.



  • There are two separate issues here with the same symptom. The starting twice problem is fixed by what I posted earlier in the thread. The issue with openbgpd causing that same PF_KEY error doesn't have a known cause or solution yet. I'm attempting to replicate that one.



  • cmb, if you would like access to my pfSense server that has the OpenBGPD issue again let me know and I will message you the details.



  • I'm not running OpenBGPD.  I'll review, but I'm pretty sure I applied that patch successfully utilizing the patch package utility…

    Edit:  It's definitely installed.  I navigated out to the file in question to check for the additions that were added and they're there.

    I haven't done a full reboot, would that have any chance of affecting the application of the fix?



  • @vsxi-13:

    I'm not running OpenBGPD.  I'll review, but I'm pretty sure I applied that patch successfully utilizing the patch package utility…

    Edit:  It's definitely installed.  I navigated out to the file in question to check for the additions that were added and they're there.

    I haven't done a full reboot, would that have any chance of affecting the application of the fix?

    You have to either manually kill off the duplicate instances of strongswan (ipsec and charon processes), or reboot after applying that. That just prevents the problem circumstance from happening again.



  • @cmb:

    @vsxi-13:

    I'm not running OpenBGPD.  I'll review, but I'm pretty sure I applied that patch successfully utilizing the patch package utility…

    Edit:  It's definitely installed.  I navigated out to the file in question to check for the additions that were added and they're there.

    I haven't done a full reboot, would that have any chance of affecting the application of the fix?

    You have to either manually kill off the duplicate instances of strongswan (ipsec and charon processes), or reboot after applying that. That just prevents the problem circumstance from happening again.

    I'm going to keep an eye on it.  I rebooted today, however I also just rebuilt my pfSense and am running it virtually now in VMWare with essentially the same configuration.



  • @vsxi-13:

    @cmb:

    @vsxi-13:

    I'm not running OpenBGPD.  I'll review, but I'm pretty sure I applied that patch successfully utilizing the patch package utility…

    Edit:  It's definitely installed.  I navigated out to the file in question to check for the additions that were added and they're there.

    I haven't done a full reboot, would that have any chance of affecting the application of the fix?

    You have to either manually kill off the duplicate instances of strongswan (ipsec and charon processes), or reboot after applying that. That just prevents the problem circumstance from happening again.

    I'm going to keep an eye on it.  I rebooted today, however I also just rebuilt my pfSense and am running it virtually now in VMWare with essentially the same configuration.

    This would have been a great weekend to test this, however my Macbook has decided that it doesn't want to run IKEv2 anymore…

    
    Apr 22 07:34:01 mba nesessionmanager[427]: NESMIKEv2VPNSession[xxxx.dyndns-web.com:4B453608-183B-4727-B69F-DB98737FCCD7]: Received a start command from SystemUIServer[239]
    Apr 22 07:34:01 mba nesessionmanager[427]: NESMIKEv2VPNSession[xxxx.dyndns-web.com:4B453608-183B-4727-B69F-DB98737FCCD7]: status changed to connecting
    Apr 22 07:34:01 mba nesessionmanager[427]: Failed to find the VPN app for plugin type com.apple.neplugin.IKEv2
    Apr 22 07:34:01 mba neagent[926]: IKEv2 Plugin: ikev2_dns_callback: Error -65554
    Apr 22 07:34:02 mba kernel[0]: ipsec_ctl_connect: creating interface ipsec0
    Apr 22 07:34:02 mba configd[51]: network changed
    Apr 22 07:34:04 mba neagent[926]: MSCHAPv2 Error = 691, Retry = 1, Version = 0
    Apr 22 07:34:04 mba neagent[926]: Failed to process IKE Auth (EAP) packet
    Apr 22 07:34:04 mba neagent[926]: BUG in libdispatch client: kevent[EVFILT_READ] delete: "Bad file descriptor" - 0x9
    Apr 22 07:34:04 mba nesessionmanager[427]: NESMIKEv2VPNSession[xxxx.dyndns-web.com:4B453608-183B-4727-B69F-DB98737FCCD7]: status changed to disconnecting
    Apr 22 07:34:04 mba kernel[0]: SIOCPROTODETACH_IN6: ipsec0 error=6
    Apr 22 07:34:04 mba nesessionmanager[427]: NESMIKEv2VPNSession[xxxx.dyndns-web.com:4B453608-183B-4727-B69F-DB98737FCCD7]: status changed to disconnected, last stop reason $
    Apr 22 07:34:04 mba configd[51]: network changed
    Apr 22 07:34:04 mba symptomsd[422]: nw_interface_get_agents SIOCGIFAGENTIDS failed for interface "ipsec0" (index 8, type other): [6] Device not configured
    
    

    Of course there a good amount of posts showing this as a problem on google, but no real resolutions…  My personal and work iPhone connect without a hitch, so it's definitely not pfSense side :(



  • I am in the same boat.  I have two pfSense boxes in an HA pair running 2.3, with BGP and an IPSec VPN.  I'm happy to help test whatever patch etc as needed.



  • Can now confirm that after removing openbgpd I havn't have had any IPSEC tunnels inactive.



  • Hi All ,

    I am a newbie in pfsense , i have recently updated on pfsense 2.3 (AMD 64) ,all the things were working like a charm including IPSEC tunnels before installing OpenBgpd, as per our requirement i have installed OpenBGPD both IPsec and OpenBgpd worked for some hours ….....then all the IPSEC tunnels gone down , In Gui configuration tunnels still showing in established state but no traffic passing , for fixing the issue i have to restart the firewall , then it again starts working for some hours .

    I have tried these below steps also but all goes in vain  :'(

    killall -9 charon
    killall -9 starter
    ipsec stop
    ipsec start

    ipsec start states as :-

    Starting strongSwan 5.4.0 IPsec [starter]…
    charon is already running (/var/run/charon.pid exists) -- skipping daemon start
    no netkey IPsec stack detected
    no KLIPS IPsec stack detected
    no known IPsec stack detected, ignoring!
    starter is already running (/var/run/starter.charon.pid exists) -- no fork done

    Please suggest anybody i am using pfsense in production environment  :'(

    I was wondering if a downgrade to previous version can fix this issue , is any version is workable with both IPSEC and OPENBGPD.
    Please if anyone have any idea on this .



  • @choudharyprabhat:

    Hi All ,

    I am a newbie in pfsense , i have recently updated on pfsense 2.3 (AMD 64) ,all the things were working like a charm including IPSEC tunnels before installing OpenBgpd, as per our requirement i have installed OpenBGPD both IPsec and OpenBgpd worked for some hours ….....then all the IPSEC tunnels gone down , In Gui configuration tunnels still showing in established state but no traffic passing , for fixing the issue i have to restart the firewall , then it again starts working for some hours .

    I have tried these below steps also but all goes in vain  :'(

    killall -9 charon
    killall -9 starter
    ipsec stop
    ipsec start

    ipsec start states as :-

    Starting strongSwan 5.4.0 IPsec [starter]…
    charon is already running (/var/run/charon.pid exists) -- skipping daemon start
    no netkey IPsec stack detected
    no KLIPS IPsec stack detected
    no known IPsec stack detected, ignoring!
    starter is already running (/var/run/starter.charon.pid exists) -- no fork done

    Please suggest anybody i am using pfsense in production environment  :'(

    I was wondering if a downgrade to previous version can fix this issue , is any version is workable with both IPSEC and OPENBGPD.
    Please if anyone have any idea on this .

    Dude, you shouldnt have updated a prod system to 2.3 without testing!

    Anyway, yes i can confirm 2.2.6 works perfectly with IPSEC and openbgpd. Im using it myself on a prod network.

    There is an open bug report for this issue: https://redmine.pfsense.org/issues/6223



  • Thanks fattylewis , i have downgraded my pfsense box to 2.2.6 , now everything is working fine.  :) ;D

    there is one more thing to notice :- when i had edited /boot/loder.config.local >>>> net.inet.raw.maxdgram="131072"
    net.inet.raw.recvspace="131072" .

    Pfsense worked like charm with ipsec and bgp even on pfsense 2.3. :)

    for me that trick worked.

    Thanks CMB and fattylewis for your replies….you guys rocksssss..  ;)



  • @choudharyprabhat:

    Thanks fattylewis , i have downgraded my pfsense box to 2.2.6 , now everything is working fine.  :) ;D

    there is one more thing to notice :- when i had edited /boot/loder.config.local >>>> net.inet.raw.maxdgram="131072"
    net.inet.raw.recvspace="131072" .

    Pfsense worked like charm with ipsec and bgp even on pfsense 2.3. :)

    for me that trick worked.

    Thanks CMB and fattylewis for your replies….you guys rocksssss..  ;)

    Oh, nice find. Ill see about knocking up another network on 2.3 and adding your change and seeing what happens.



  • We've also had this issue on 2.3, and as we required BGP for our network, we've downgraded back to 2.2.6

    Looking forward to a confirmed fix (need to wait until after hours again to try the upgrade again)



  • I'm having the same problem with OpenBGP and IPSec.

    Restarted the following services:
    -OpenBGP
    -IPSec

    No luck.  Only rebooting worked.

    Then tried restarting:
    -OpenBGP
    -IPSec
    -OpenVPN

    Tunnel came back up.

    Not sure if that helps some of the developers with troubleshooting.

    I have stopped the OpenVPN service for now and will see if the issue returns.

    UPDATE:  Still having the issue even after disabling OpenVPN



  • Same issue, pair of SG-8860s with CARP failover, dual IPSec tunnels to a Verizon Private Network with OpenBGPd required for their routing. Exact same errors, even changing both tunables:

    net.inet.raw.maxdgram="131072"
    net.inet.raw.recvspace="131072"

    May extend the time, but definitely doesn't solve. Really don't want to go back to 2.2.6 :)



  • @obrienmd:

    Same issue, pair of SG-8860s with CARP failover, dual IPSec tunnels to a Verizon Private Network with OpenBGPd required for their routing. Exact same errors, even changing both tunables:

    net.inet.raw.maxdgram="131072"
    net.inet.raw.recvspace="131072"

    It's not just those two. Add:

    net.raw.recvspace=65535
    net.raw.sendspace=65535



  • FWIW, still seeing this problem here.  Yesterday I updated to 2.3.1 and also set these:

    @cmb:

    net.inet.raw.maxdgram="131072"
    net.inet.raw.recvspace="131072"
    net.raw.recvspace=65535
    net.raw.sendspace=65535

    I just bumped those up higher hoping it will help, but at least for us neither the 2.3.1 update nor those specific values fixed it.  Does it matter if they're set at System > Advanced > System Tunables rather than in loader.config.local?



  • We've now been up for over a week with these settings (set in System > Advanced > System Tunables):

    net.inet.raw.maxdgram 131072
    net.inet.raw.recvspace 1048576
    net.raw.recvspace 1048576
    net.raw.sendspace 1048576

    Edit:  up over 2 weeks now, still no problem



  • Hi I'm new here and have a problem with my PFSense and the IPsec connection .

    The environment :
    Location A pfsense 2.3.1_1
    Location B pfsense 2.3.1_1

    Connected via IPSec " SitetoSite "

    I tried all the tips from this thread. Unfortunately without success.

    Like
    changeing net.inet.raw.maxdgram  131072 
    net.inet.raw.recvspace  1048576 
    net.raw.recvspace  1048576 
    net.raw.sendspace  1048576

    The problem is when I try to access Site B about RMTC works without problems .
    However, if I want to print a print job from B to site A drops the connection and restarts.

    Does somebody has any idea ?

    I'm a bit desperate .

    Thank you very much

    I Forget to say that it works perfect before i updatet my pfsense …



  • Hi it´s me again, i tryed to use OPENVPN instead of IPSEC
    I have the same Problem and my PFSENSE reboot new after 2 min.

    Does anyone know this situation ?



  • Hello everybody!

    I have read that thread but unfortunately I have the same issue. We use PfSense 2.3.1 with OpenBGPD+IPsec to Amazon AWS.

    We have set that:

    
    net.inet.raw.maxdgram="131072"
    net.inet.raw.recvspace="131072"
    net.raw.recvspace=65535
    net.raw.sendspace=65535
    
    

    Our IPsec disconnect every couple hours. When I check IPsec status - looks ok, but I can not transfer any packets. I don't have to reboot Firewalls but only stop OpenBGPD and IPsec. Start again and all is working again ok for next couple of hours.

    Do you have any idea what I can check more? I didn't check that fix from GitHub. But do you think it could be it?

    Thank you for any help or answer.

    Best,
    Kamyk



  • I run a couple of pfsense boxes to link my house to a few neighbors (so hardly mission critical).
    Since the upgrade to 2.3, then 2.3.1, then 2.3.1p1, my IPsec tunnels haven't worked.
    I don't run OpenBGP (at least I don't think I do) and I tried applying the System Tuneables that jnorell suggested.
    I also tried purging all my VPN configurations, and recreating them. Still no love :(
    What's odd (at least to me), is that all the tunnels come up in the web interface, but they don't pass traffic.
    It's not the end of the world, as I moved to OpenVPN in the interim, however I'd prefer to get back to IPsec.
    Thanks in advance



  • @Kamyk:

    I have read that thread but unfortunately I have the same issue. We use PfSense 2.3.1 with OpenBGPD+IPsec to Amazon AWS.

    Known issue: https://redmine.pfsense.org/issues/6223



  • @olobley:

    Since the upgrade to 2.3, then 2.3.1, then 2.3.1p1, my IPsec tunnels haven't worked.

    What's odd (at least to me), is that all the tunnels come up in the web interface, but they don't pass traffic.

    It sounds like you have a different problem (try enabling cisco extentions in ipsec advanced settings), this one is indicated by 'No buffer space available' errors in the logs.



  • Today, after almost 29 days uptime, we're getting 'error sending to PF_KEY socket: No buffer space available' again .. I'm bumping settings up some more:

    
    net.inet.raw.maxdgram = 131072
    net.inet.raw.recvspace = 1048576
    net.raw.recvspace = 1048576
    net.raw.sendspace = 2097152
    
    


  • Hello, new to this forum. Just throwing my hat in the ring for this issue as well. Plagued by "error sending to PF_KEY socket: No buffer space available".
    I'm using three IPsec tunnels. One to AWS (with BGP), one to Azure, one to a mikrotik router at a remote office.

    Is there a way to effectively restart IPsec and flush that buffer without rebooting?
    Restarting the service via the GUI, or manually killing charon and starter and restarting ipsec via terminal does not do it.

    EDIT: Of course I should mention this problem started happening after upgrading from 2.2.(6?) to 2.3.1_1
    I have increased
    net.inet.raw.maxdgram
    net.inet.raw.recvspace
    net.raw.recvspace
    net.raw.sendspace

    to recommended values, but have not rebooted since. I will reboot late tonight.



  • Same issue with upgrade to 2.3.1_5, any idea if this will be resolved in 2.3.2 or 2.4.x (FreeBSD 11, right?)



  • Not something we're going to have time for in 2.3.2 (release next week), hopefully it's either resolved already in FreeBSD 11, so 2.4 will be fine, or someone can track down the root cause and get it fixed (my last day here is in two weeks).

    2.4 snapshots should be out soon. Help testing then would be appreciated.



  • i face the same problem when i try to establish a new ipsec site to site vpn between 2 branches with a pfsense with a firmware 2.2.6. I solved that by adding
    on the phase 1 proposal (authentication ) the real ip of my peer as it was behind the a nat

    My identifier ===== choose Ip address ======= then put your real ip address

    and on the Peer Identifier you should put the private ip of the other side if he do the same

    Peer identifier ======== ip address =========then put your private ip address



  • @cmb:

    Not something we're going to have time for in 2.3.2 (release next week), hopefully it's either resolved already in FreeBSD 11, so 2.4 will be fine, or someone can track down the root cause and get it fixed (my last day here is in two weeks).

    2.4 snapshots should be out soon. Help testing then would be appreciated.

    Sure thing, will test as soon as 2.4 snapshots are available! Good luck on your next adventure, and thanks for all the hard work on pfSense :)



  • Is anybody aware of any progress on this? Bumping the buffer sizes only extends the issue from a few hours to about two days but that is it.

    Also is there any news regarding the root cause? I am struggling to understand the interaction between IPSec (+GRE) and OpenBGPd. Surely the same would happen with any TCP-based application, or is it something that OpenBGPd specifically repeatedly calls on the sockets that causes IPSec to eventually die?

    I run a number of tunnels with IPSec + GRE + BGP (pfSense to pfSense and pfSense to Cisco) and since 2.1 they were never really stable. All the way up to 2.3 I had to monitor the GRE tunnels and bounce them after any IPSec re-key or tunnel flap because OpenBGPd was seeing them as invalid next hops. This went away in 2.3, but now IPSec is basically unusable. Doesn't matter why, it makes for an incomplete product. Nobody really runs static routing over non-trivial topologies, and with non-functional BGP, IPSec is only usable for mobile clients. I'm going to give BIRD a try - and migrating the whole network to OSPF is not really an option here, although I will consider it.

    Failing that, after many years with pfSense I am going to start looking for alternatives. pfSense is a fantastic platform, and thanks for all the hard work guys, but constant IPSec issues have just about killed it for me.



  • …OK, some progress.

    Having looked up the PF_KEY rcvbuf error got me a change and a setting introduced in StrongSWAN 5.3.0 where the event socket buffer can be tuned.

    Once all IPSec tunnels were dead, I stopped ipsec, stopped openbgpd, then I opened /etc/inc/vpn.inc, searched for the charon { plugins { section and added the following:

    ....
    
                    kernel-pfkey {
    
                            events_buffer_size = 1048576
    
                    }
    
    

    Started ipsec via GUI which re-generated the configs, started openbgpd. Guess what - tunnels came back up, I can see SADs and SPDs again, and some of the BGP sessions are up again (those to Cisco, funny enough). I have now rebooted all pfSense instances and will see how long they will last.

    Thanks,
    owczi



  • @owczi:

    …OK, some progress.

    ...

    Started ipsec via GUI which re-generated the configs, started openbgpd. Guess what - tunnels came back up, I can see SADs and SPDs again, and some of the BGP sessions are up again (those to Cisco, funny enough). I have now rebooted all pfSense instances and will see how long they will last.

    Thanks,
    owczi

    Still up?



  • @obrienmd:

    Still up?

    Nope - shat itself after about 24 hours. HOWEVER, I don't have to reboot to get the tunnels and BGP sessions back up. The setting I added to charon config may not have anything to do with it. I will keep trying various combinations to get a sensible answer: on some of the pfSense instances I did not have to restart IPSec at all, only bgpd, but it could have been that they had BGP down because of the other peers, and an ipsec restart is still required. I have no time to investigate right now.

    Basically:```
    /usr/local/etc/rc.d/bgpd.sh stop; ipsec stop; sleep 1; ipsec start; sleep 2; /usr/local/etc/rc.d/bgpd.sh start

    
    I need to write a monitoring script that will do this when all tunnels go down. For now I will just make it a cron job every few hours, maybe even every hour. offset so it doesn't happen on all instances at the same time. This will at least keep me going.


  • EDIT: full path for ipsec - required when invoked from cron; do not reset ipsec / bgpd if there are no connections.
    EDIT2: fixed to correctly pick up connections when nothing is up and check for buffer errors
    Crude as can be, but will do the job… I run this every 5 minutes via a cron job:

    
    #!/bin/sh
    estabcount=0
    p2count=0
    totalcount=0
    buferr=0
    
    bounceall() {
    /usr/local/etc/rc.d/bgpd.sh stop
    sleep 1
    $ipsecpath stop
    sleep 1
    $ipsecpath start
    sleep 3
    /usr/local/etc/rc.d/bgpd.sh start
    }
    
    ipsecpath=/usr/local/sbin/ipsec
    
    echo "=== started at `date` ==="
    
    for con in `$ipsecpath status | grep "[" | sed 's/[.*//g' | sort | uniq` ; do 
    echo $con
    estab=0
    p2=0
    
    $ipsecpath status $con | grep ESTAB >/dev/null 2>&1 && estab=1
    $ipsecpath status $con | grep INSTALLED >/dev/null 2>&1 && p2=1
    
    [ $estab -eq 1 ] && { 
    	echo $con p1 up
    	estabcount=$(( $estabcount + 1 ))
    	[ $p2 -eq 0 ] && {
    	 	echo $con p2 down, restarting
                    echo stopping $con...
    		$ipsecpath down $con >/dev/null 2>&1
    		sleep 1
                    echo starting $con...
    		$ipsecpath up $con | grep error | grep "buffer space" >/dev/null 2>&1  && { echo "PF_KEY buffer error while starting $con"; buferr=$(( $buferr + 1 )); }
    	}
    
    }
    [ $estab -eq 0 ] && { echo $con p1 down; }
    [ $p2 -eq 1 ] && { echo $con p2 up; p2count=$(( $p2count + 1 )); }
    totalcount=$(( $totalcount + 1 ))
    done
    
    echo
    echo ===
    echo estab $estabcount / $totalcount
    echo p2 $p2count / $totalcount
    echo buf_err $buferr / $totalcount
    echo ===
    echo
    
    [ $totalcount -gt 0 ] && [ $buferr -gt 0 ] && {
    echo $buferr connections show buffer space errors - bouncing openbgpd and ipsec
    bounceall
    exit
    }
    
    [ $totalcount -gt 0 ] && [ $estabcount -eq 0 ] && {
    echo no connections have p1 up - bouncing openbgpd and ipsec
    bounceall
    exit
    }
    
    [ $totalcount -gt 0 ] && [ $estabcount -eq $totalcount ] && [ $p2count -eq 0 ] && {
    echo all connections have p1 up but no connections have p2 up - bouncing openbgpd and ipsec
    bounceall
    exit
    }
    
    

    It will bounce all tunnels which have phase 2 down, and if no tunnels have p1 it will bounce ipsec and bgpd. We'll see how long this will last.


Log in to reply