PfSense 2.4.1 - ikev2 IPSEC tunnel under load crashes whole firewall VM

timthetortoise

Can confirm this is occurring for me on two different systems.
Both are running on ESXi 6.5, one on DL380 G8, the other on DL380 G9.
NIC type is vmxnet3, open-vm-tools installed on both.
Phase 1: AES128-GCM / 128 / SHA1 / DH2
Phase 2: AES128-GCM / AES-XCBC / no PFS

Hard crash with a reboot within 5 minutes of initiating continuous iperf run, sometimes one side, sometimes both.

Switching to any non-AES-NI algorithms kills throughput, but doesn't hard crash.

My```
dmesg | grep -i aes

Features2=0xffba2203 <sse3,pclmulqdq,ssse3,cx16,pcid,sse4.1,sse4.2,x2apic,popcnt,tscdlt,aesni,xsave,osxsave,avx,f16c,rdrand,hv>aesni0: <aes-cbc,aes-xts,aes-gcm,aes-icm>on motherboard</aes-cbc,aes-xts,aes-gcm,aes-icm></sse3,pclmulqdq,ssse3,cx16,pcid,sse4.1,sse4.2,x2apic,popcnt,tscdlt,aesni,xsave,osxsave,avx,f16c,rdrand,hv>


I'll do some more testing this weekend when there's not as much production traffic flowing but for right now I'm knocked back down to plain AES.

It does indeed make pfSense unusable for installations requiring decent IPSec interconnect speeds. Considering this issue I'll likely move to VyOS for my concentrators.

Has anyone attempted to use the patch from the previous FreeBSD thread posted?

Edit: both running 2.4.3-Release

lkolbe

@jimp we're experiencing the same problem. One client, using AES256-gcm, reliably crashes the SG-8860 w/pfsense 2.4.3 when using e.g. speedtest.net (during the upload phase), another can't bring it down at all. Switching back to plain AES with SHA512 seems to fix it for now. All clients are Macbook Pros.
Kind regards,
Lukas

dave.opc

is AES256-GCM better than AES?

Derelict

AES-GCM is an authenticated cipher so you can eliminate the hashing step. If max IPsec performance is what you seek, AES-GCM is the way to go.

dave.opc

I got 2 sites connected via IPSEC using AES. Both have 100Mb connection to internet. And IPSEC uses the whole 100Mb bandwidth on file transfers. So what is the limitation for AES, compared to AES-GCM?

Derelict

AES-GCM will consume fewer CPU cycles to accomplish the same task.

dave.opc

So basically, if you got powerful enough CPU/PC it dosn't matter which algorithm to use?

Derelict

AES-GCM will use fewer CPU cycles to accomplish the same task.

Fewer cycles means more cycles available for other tasks.

You can waste them if you so desire.

lkolbe

... as long as you can live with the occasional hard crash (at least on an SG-8860/Atom C7258 using AES-NI). I haven't yet found out what kind of traffic pattern causes this crash; switching to AES-CBC w/SHA512 removed the crashes reliably for us.

msrachelchen

@jimp said in PfSense 2.4.1 - ikev2 IPSEC tunnel under load crashes whole firewall VM:

To claim it's unusable in general is untrue. The crash must be specific to a certain combination of hardware, traffic load, and/or pattern of traffic.

Loads of people are using AES-NI and AES-GCM without crashing, including just about every Netgate employee from our home firewalls.

If that's the case, what are their hardware configurations? It seems to me that this issue is pretty common among users, and there isn't a pattern in hardware I can see.

msrachelchen

Same issue with AES-GCM+AES-NI crashing the system:

Version	2.4.3-RELEASE-p1 (amd64) 
built on Thu May 10 15:02:52 CDT 2018 
FreeBSD 11.1-RELEASE-p10 
	 
CPU Type	Intel(R) Xeon(R) CPU E3-1271 v3 @ 3.60GHz
Current: 3600 MHz, Max: 3601 MHz
4 CPUs: 1 package(s) x 4 core(s)
AES-NI CPU Crypto: Yes (active)

Derelict

All kinds, but mostly Netgate devices such as the SG-2440, SG-4860, SG-3100.

Intel(R) Atom(TM) CPU C2558 @ 2.40GHz
4 CPUs: 1 package(s) x 4 core(s)
AES-NI CPU Crypto: Yes (active)

Uptime 122 Days 21 Hours 45 Minutes 01 Seconds

AES_GCM_16
MODP_2048
IPComp: none

I used TRex a few weeks ago to run terabytes and terabytes through AES-GCM IPsec trying to make it crash. Hundreds of megabits per second for days on end. Could not duplicate.

msrachelchen

@derelict a few things were mentioned in the thread

2.3 was unaffected yet 2.4 was crashing
issues appeared on VM and bare metal
there was a bug in the kernel
this "issue" first appeared 10 months ago
usr @RMB is using Netgate product yet he was experiencing the issue

Maybe I suggest:

Let us know if that particular patch was merged
Can you try running AES-GCM with any EC (say nist ecp384)
What do you suggest for us the users to get to the bottom of this?

msrachelchen

This is fixed in 2.4.4. If you are having issues try again on 2.4.4.

lkolbe

Just wanted to confirm here that the AES-GCM-crashes with AES-NI on our SG-8860 are indeed gone now on pfSense 2.4.4. No crashes since I restartet testing AES-GCM a few weeks ago.