After 2.2.3 upgrade IPsec tunnels wont come back up



  • I am using site-to-site IPsec VPNs between pfSense hosts where one end of the tunnel has a static IP, the other a dynamic IP. Both ends have WAN interfaces with publicly resolvable dynamic DNS hostnames.

    On 2.2.2 the tunnels were working fine with certificates but since upgrading to 2.2.3 the tunnels wont come back up. My tunnels are configured to use Mutual RSA authentication and ASN.1 distinguished names for both My Identifier and Peer Identifier.

    I am seeing this new message in the logs.

    charon: 09[IKE] <con1|10>received 3 cert requests for an unknown ca

    I haven't changed any certificates, CA or other. The CA and Intermediate CA certificates that I imported in 2.2.2 are still visible in System:Certificate Authority Manager: CAs. All I have done is apply the 2.2.3 update.</con1|10>



  • I have tried a lot of different things today but I have been unable to fix this.

    I am going to enable all diags on IPsec and save/print the IPsec logs for 2.2.3 before reinstalling 2.2.2 and restoring from backup config.

    When I have a working IPsec tunnel again I will enable all diags on 2.2.2 and save/print IPsec logs for comparison.



  • I am relieved that reinstalling 2.2.2 and restoring my backup config has got my IPsec tunnels working again.

    pfBlockerNG appears to be adding another 30 minutes or so to startup cycle following a config restore. Forcing a list update in pfBlockerNG looks like it fixes it for subsequent boots.


  • Banned

    @vbentley:

    pfBlockerNG appears to be adding another 30 minutes or so to startup cycle following a config restore.

    Yeah. So use 2.2.3 and you won't have such issue.



  • Hi,

    Bug created for further investigation. I suggest watching the bug, rather than posting that it doesn't work, since there seems to be multiple confirmations from people that 2.2.3 IPSec isn't healthly, so more "me toos" doesn't further the cause (unless there is important information to add) :-)

    https://redmine.pfsense.org/issues/4791

    -=david=-



  • Thanks for reporting the bug.

    I am still investigating the problem myself. I don't have the same problem probably because I am using Broadcom hardware crypto cards. My Phase 1 fails on 2.2.3 and most people are reporting just Phase 2 problems.

    I have a few more things to try before I pull the crypto cards to check if I get the same P2 problem.



  • I created a new P1 that uses PSK instead of certs, and distinguished names (DNS hostnames) instead of ASN.1 distinguished names for the local and peer identifiers. Hardware crypto cards still installed at both ends.

    Success! Of sorts. P1 and P2 both up on 3DES SHA1 ESP however. I can ping in both directions. But I can only ssh into the remote host from the central host but not the other way round.

    Remote is on 2.2.3
    Central is on 2.2.2

    No recent firewall rule changes have been made. The logs show SSH being accepted in both directions, but it only works in one.



  • Something really weird qoing on with the 'Enable bypass for LAN interface IP'. I read the bug reports and manually verified both ends ipsec.conf are similar. One had bypass the other didn't. Bypass is now enabled at both ends and I can SSH in both directions with PSK authentication.

    I'm not sure if this would affect my Phase 1 with certificate authentication but I will try again tomorrow.


  • Netgate Administrator

    It shouldn't do that's a Phase2/routing type option.

    So it's working PSK and still using the hardware crypto?

    Steve



  • Yes. PSK is working.
    P1 established on 3DES SHA1
    P2 established on 3DES SHA1
    Full packet flow in both directions.
    I haven't tested throughput benchmark to see if crypto cards are working but it seems normal.

    I cannot get RSA certs working at all with 2.2.3.
    I have tried:-

    2.2.2 initiating connection to 2.2.3 - no luck
    2.2.3 initiating connection to 2.2.2 - no luck
    2.2.3 initiating connection to 2.2.3 - no luck

    2.2.2 initiating connection to 2.2.2 - works every time!

    I have been reading the changelog for StrongSwan and I'm wondering if this has something to do with reqid.
    Perhaps there is something missing in the existing RSA certs that reqid is looking for?

    My RSA certs are setup similar to the process described in the HowTo except that instead of IP:value in the certs I am using DNS:value instead. This works well with dynamic IP and dynamic DNS on 2.2.2 IPsec VPNs.
    https://doc.pfsense.org/index.php/IPsec_RSA_Authentication_Quick_Start


  • Netgate Administrator

    Ok, I setup my test tunnel here using certs as described in the quick start doc and it came right up.
    That's running 2.2.3 on one side and 2.2.4-dev on the other but at this point the only difference in 2.2.4 is removing the offending AES-NI patch.

    That's running: P1 AES 256/SHA1  P2 AES-GCM 256/SHA1

    If you tell me exactly what encryption settings you were using I can verify that. It does look likely to be an issue with your encryption hardware though.

    Steve



  • Thanks for trying to replicate my issue.

    Did you use IP addresses for your P1 config or FQDN?
    Did you use IP: references in your RSA certs or DNS: references?
    Did you use ASN.1 distinguished names for 'My Identifier' and 'Peer Identifier' Eg. "C=GB, ST=E… "

    My P1 config that works well on 2.2.2 does not use IP addresses only FQDN.
    I am using only hostnames because one end of my tunnel is on dynamic IP.
    My P2 config does use IP addresses.

    I realise that I am probably using an off-the-beaten track configuration and I will be putting time in next week to try to find the root cause of the problem. Thanks in advance for confirming a known good starting point for my tests.


  • Netgate Administrator

    I went for a basic setup just to confirm that certs were working so exactly as described in the quick_start doc.
    I'll try to test some more variations.

    Steve



  • Thanks Steve.
    My guess is that it's a data matching problem with the references in the certs.


  • Netgate Administrator

    There is a patch for 2.2.3 you can try that adds the various fixes from 2.2.4. Let me know if you'd like to try that.

    Steve



  • Hi!

    How to apply this patch?

    BR



  • Thank you to the team for the hard work that went into fixing IPsec in 2.2.4 , my RSA certificate authenticated tunnels are running again.

    I read the release notes for 2.2.4 first.
    I installed 2.2.4 from the console.
    I created new certificates and revoked the originals and installed imported certs where necessary.
    I had to remove a set of "" from the ASN.1 config in the GUI but after that the links came up.

    I haven't tried any hardware crypto tests yet but it looks like it is still working.

    Thanks again.


  • Netgate Administrator

    Great, thanks for confirming that.  :)

    Steve