IPSec packet loss/routing issue with 21.02-RELEASE

Majik

I recently upgraded to 21.02-RELEASE and I seem to be having issues with one of my IPSec VPNs which was previously working.

The symptom is that the tunnel seems to (relatively silently) stop routing traffic. The VPN Phase 1 and 2 remain up and Traffic Capture shows that pfSense is trying to pass data across the tunnel, but I don't get a response. At this point I don't know if the traffic is reaching the far end or not.

In the logs, the only thing I see is DPD requests:

Feb 24 09:30:49 firewall charon[69212]: 11[IKE] <con400000|5> sending DPD request
Feb 24 09:30:49 firewall charon[69212]: 11[ENC] <con400000|5> generating INFORMATIONAL_V1 request 3965837724 [ HASH N(DPD) ]
Feb 24 09:30:49 firewall charon[69212]: 11[NET] <con400000|5> sending packet: from x.x.x.x[500] to y.y.y.y[500] (108 bytes)
Feb 24 09:30:49 firewall charon[69212]: 11[NET] <con400000|5> received packet: from y.y.y.y[500] to x.x.x.x[500] (108 bytes)
Feb 24 09:30:49 firewall charon[69212]: 11[ENC] <con400000|5> parsed INFORMATIONAL_V1 request 3397238830 [ HASH N(DPD_ACK) ]

When connectivity drops, I get these about once every 5 seconds.

I realise this is probably not nearly enough information to diagnose an issue, but I'm starting a thread here so I can gather and add data to it as I can. If there's anything specific I should be looking for, please let me know.

Cheers,

Keith

Majik

I should add that, at this point, the time between the tunnel connecting and the packet loss seems to be fairly random. Sometimes it seems to happen within a minute, on other occasions it can run for hours without any issues.

Restarting the Child SA resolves the connectivity, until it fails again.

Cheers,

Keith

Majik

I should also add that the far-end is a Juniper SRX device.

The VPN is IKEv1 with
Algo: AES_CBC (256), HMAC_SHA2_256_128,PRF_HMAC_SHA2_256,MODP_2048

Phase 2s:
Algo: AES_CBC (256), HMAC_SHA2_256_128, MODP_2048, IPComp: none

This is running on a Netgate XG-7100.

Cheers,

Keith

Majik

Additional info:

I am running the following patches:

ead6515637a34ce6e170e2d2b0802e4fa1e63a00
57beb9ad8ca11703778fc483c7cba0f6770657ac
10eb04259fd139c62e08df8de877b71fdd0eedc8
ded7970ba57a99767e08243103e55d8a58edfc35
afffe759c4fd19fe6b8311196f4b6d5e288ea4fb
2fe5cc52bd881ed26723a81e0eed848fd505fba6
95a4e1a0e42392fe4523bf769589f74864446f8c
4e5857b656c7bfd59efadbb9a124876a5516c7df

Cheers,

Keith

Majik

Interestingly, I have switched the Phase 2 tunnels from AES 256 with SHA256 to AES256-GCM and they seem a lot more stable.

Cheers,

Keith

jimp

What hardware are you using?

There are two potential issues which may be in play here:

An issue with SHA-256 and AES-NI which could affect some implementations: https://redmine.pfsense.org/issues/11524
A potential issue with CESA on SG-3100 and AES-CBC which we are still actively investigating.

Majik

@jimp Netgate XG-7100

It does very much sound like https://redmine.pfsense.org/issues/11524

Regards,

Keith

jimp

Try switching from AES-NI to QAT, reboot, and see if things stabilize for you. That worked for the user in the other thread.

Majik

@jimp Yes, I have done that. TBF it was pretty stable with AES-256-GCP, but I would rather have the option to use AES 256 if I have to.

Thanks.

Keith

jimp

So is it working OK with that setup now?

Majik

The VPN seems to be OK now. I have switched to QAT as a precaution, but AES-256-GCM seemed to work fine without that.

I have not tried AES 256 with QAT.

I have some issue with Unbound randomly dying, but that's something else, and I see there's already a thread or two about that.

Cheers,

Keith

e37921

@jimp I had this issue with one Ipsec vpn on a SG-5100. I am using AES-CBC SHA256. I set the hardware acceleration to none, now the tunnel is staying up. Before, the vpn would show as up but stop sending traffic. Took a few hours to find this topic but glad I did. Thanks

jimp

@e37921 said in IPSec packet loss/routing issue with 21.02-RELEASE:

@jimp I had this issue with one Ipsec vpn on a SG-5100. I am using AES-CBC SHA256. I set the hardware acceleration to none, now the tunnel is staying up. Before, the vpn would show as up but stop sending traffic. Took a few hours to find this topic but glad I did. Thanks

If you find that you need the acceleration you could enable QAT on that SG-5100, or you could switch from SHA256 to another hash

mbr

@jimp I'd like to add that I updated an SG-5100 to 21.02 and am having what appears to be the same problem.

Site-to-site IPsec with Cisco ASA on remote end with AES256 and SHA1 for P2. Tunnel connects but after ~30 seconds my traffic stops and remote side reports ESP failed authentication. Using a different hash works but I'll try switching to QAT instead of AES-NI for now since that seems to work for others.

Issue seems related to this post as well:
https://forum.netgate.com/topic/161268/ipsec-tunnels-using-sha256-may-not-connect/5

Thanks to everyone for the troubleshooting tips!

viniciusmerlim

@majik The number of operations required to brute force a 256-bit cipher is 3.31 x 10^56. This is roughly equal to the number of atoms in the universe. And 1.02 x 10^18 - around one billion billion (one quintillion) - years to crack a 128-bit AES key by force. This is older than the age of the universe (13.75 billion years).

The most powerful supercomputer in the world in 2017 was the Sunway TaihuLight in China. This beast is capable of a peak speed of 93.02 petaflops. This means that the most powerful computer in the world would still take some 885 quadrillion years to brute force a 128-bit AES key.

I think you can use 128bits key with no worries.

adeparker

Just to add in that I had the same issue with an XG-7100. Turned off the Crypto acceleration totally and now all ok. Using AES256, SHA1 and AES128, SHA1.

Tried switching hash methods but no real change at this end (other than mobile clients not being able to connect)

Hoping this get's fixed in the next production release so can turn back on.

sgnoc

I found this topic after having similar issues. I have an IPSec tunnel to a remote endpoint with a file server. I was able to ping the file server from my end with no problems but as soon as a computer (didn't matter which one) tried a TCP CIFS connection to the server, I was only able to usually get the directory structure loaded before the connection became unresponsive. Oddly enough, the P1 and P2 connections were still functioning and keeping the connection open.

I just had no response from the server with CIFS and the ping replies stopped.

I first tried to change the AES-NI hardware encryption to QAT. That got better results, but after transversing through another sub-directory or two, the CIFS became unresponsive again. I then did as @adeparker suggested and turned off hardware encryption completely, did a reboot to unload the modules and now the CIFS is completely responsive and there are no problems with the file server connection anymore.

Also hoping this gets fixed in the next release. I'm running pfsense on the XG-7100 and one of the reasons I selected this one was specifically for the hardware encryption with OpenVPN and IPSec.

viniciusmerlim

@adeparker IST has banned the use of SHA-1 by U.S. federal agencies since 2010. This offers no acceptable security at all. I'm using AES-128GCM with SHA256. I have an SG5100 with QAT enabled and it is working wonderfully. Is it possible that this problem is related to the XG7100?

viniciusmerlim

@sgnoc How was your IPSec configuration before you disabled qat?

sgnoc

@viniciusmerlim I had Cryptographic Hardware set to AES-NI and BSD Crypto Device. I ran that way for the last year and a half with no issues until recently. QAT didn't seem to resolve the problem, but improved it just slightly. Although I did not reboot after changing to QAT, so maybe that would have been a difference?

I then switched to None, rebooted and it is working the way it used to. No other settings were modified.

P1 is set AES / 256 bits / SHA256
P2 is set ESP / AES (256 bits) Transform, SHA256