Aes-ni not working?

miloman

I've built myself a lab so i can test the performance of pfsense in different scenarios. The setup looks like this:

It consists if 3 shuttle PC's with dual nics.

Top and bottom is used for firewall installations. The middle one is used solely as a router.

Top and bottom is a Shuttle XH61V with an I5-3470 cpu (aes-ni capable)
The Shuttle in the middle is a XG41 with a Core 2 duo 8400. Maximum thoughput of this box is ~820mbit

I'm using Iperf to test the throughput between the Knoppix installations.

I've limited the I5-3470 CPU's to 2 cores (easier to see performance impact when enabling/disabling features in pfsense)

When testing with a NAT on port 5001 on the WAN interface on one of the firewalls i get 815mbit throughput. This tells me the setup is working.

When testing with an VPN with Phase 2 AES-256bit encryption i get ~190mbit throughput.

If i then enable AES-NI in "advanced - miscellaneous" and test again, i ~get 190mbit.

I don't see a difference in CPU usage with aes-ni enabled/disabled.

Am I doing it wrong?

setup.jpg_thumb

diagram.png_thumb

jimp

Sounds like it should be right. You're enabling AES-NI on both firewalls at the same time?

What does the output of "cryptostat" look like without it enabled, and with it enabled?

I added the support in for AES-NI but I don't have access to any hardware that is capable of using it, so I couldn't test it.

dhatz

What is the output of

kldstat

kldload aesni

(run from CLI)

miloman

@jimp:

Sounds like it should be right. You're enabling AES-NI on both firewalls at the same time?

What does the output of "cryptostat" look like without it enabled, and with it enabled?

I added the support in for AES-NI but I don't have access to any hardware that is capable of using it, so I couldn't test it.

cryptostat = command not found

I assume you wanted me to run the command in a cli.

miloman

@dhatz:

What is the output of

kldstat

kldload aesni

(run from CLI)

kldstat:
id refs address size name
1 4 0xc0400000 13a57e8 kernel
2 1 0xc858a000 4000 aesni.ko

kldstat aesni:
can't load aesni: File exists

jimp

Sorry that should be cryptostats. I left off the s.

miloman

@jimp:

Sorry that should be cryptostats. I left off the s.

same output whether aes-ni enabled or disabled:

[2.1-BETA0][admin@pfSense.localdomain]/root(12): cryptostats
0 symmetric crypto ops (0 errors, 0 times driver blocked)
0 key ops (0 errors, 0 times driver blocked)
0 crypto dispatch thread activations
0 crypto return thread activations

dhatz

@miloman:

kldstat:
id refs address size name
1 4 0xc0400000 13a57e8 kernel
2 1 0xc858a000 4000 aesni.ko

kldstat aesni:
can't load aesni: File exists

So it's already loaded. Try

kldunload aesni
kldload aesni

the output of the latter command indicates if the aesni driver thinks AES-NI is supported by your hw.

jimp

also:

dmesg | grep -i aes

It may be that the driver isn't attaching to your chip. Your chip may not support AES-NI or it may be a newer chip than the AES-NI driver knows about.

miloman

@dhatz:

@miloman:

kldstat:
id refs address size name
1 4 0xc0400000 13a57e8 kernel
2 1 0xc858a000 4000 aesni.ko

kldstat aesni:
can't load aesni: File exists

So it's already loaded. Try

kldunload aesni
kldload aesni

the output of the latter command indicates if the aesni driver thinks AES-NI is supported by your hw.

[2.1-BETA0][admin@pfSense.localdomain]/root(9): kldunload aesni
[2.1-BETA0][admin@pfSense.localdomain]/root(10): kldstat
Id Refs Address Size Name
1 1 0xc0400000 13a57e8 kernel

[2.1-BETA0][admin@pfSense.localdomain]/root(11): kldload aesni
[2.1-BETA0][admin@pfSense.localdomain]/root(12): kldstat
Id Refs Address Size Name
1 4 0xc0400000 13a57e8 kernel
2 1 0xc813c000 4000 aesni.ko

miloman

@jimp:

also:

dmesg | grep -i aes

It may be that the driver isn't attaching to your chip. Your chip may not support AES-NI or it may be a newer chip than the AES-NI driver knows about.

[2.1-BETA0][admin@pfSense.localdomain]/root(13): dmesg | grep -i aes
Features2=0x77bae3ff<sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,sse4.1,sse4.2,x2apic,popcnt,tscdlt,aesni,xsave,avx,f16c,<b30>>
aesni0: <aes-cbc,aes-xts>on motherboard
aesni0: detached
aesni0: <aes-cbc,aes-xts>on motherboard
aesni0: detached
aesni0: <aes-cbc,aes-xts>on motherboard
aesni0: detached
aesni0: <aes-cbc,aes-xts>on motherboard</aes-cbc,aes-xts></aes-cbc,aes-xts></aes-cbc,aes-xts></aes-cbc,aes-xts></sse3,pclmulqdq,dtes64,mon,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,sse4.1,sse4.2,x2apic,popcnt,tscdlt,aesni,xsave,avx,f16c,<b30>

miloman

any updates regarding aes-ni not working?

i have a test environment with aes-ni capabilities that i'd be more than happy to let you use for testing.

tbo2k

I just registered to leave back my high interest in this topic.
We built our latest internal-test appliance based on xeon e3-1220Lv2 (Dual-Core 2,3 GHz Low-Voltage) which has AES-NI build in.
With the latest 2.1 pfsense we expected ipsec VPN performance with aes-256 reaching 1 Gbit/s.
But in fact we only get around 230 MBit/s, or as use case 27 MB/s with File-Transfers between sites.
Without ipsec we can transfer with almost 1 GBit/s.
With Quad-Core (E3-1260L) we get almost the same.
Cryptostats tells:
79369 symmetric crypto ops (0 errors, 0 times driver blocked)
0 key ops (0 errors, 0 times driver blocked)
0 crypto dispatch thread activations
0 crypto return thread activations

–> it's not beeing used. Besides that we have the same output as the previous posters, dmesg reports AES-NI, device/driver is loaded and activated.
Also the performance is exactly the same with aes enabled or not.

Which leaves two big "downsides" right now with pfsense and high-performance hardware:

First: ipsec is not multithreaded. It is only using one core, so only Gigahertz matters not core-count. For mixed usage that is ok, e.g. 500.000 Sessions hitting the packet-filter and besides that some 100 Mbit/s VPN Tunnels you get overall good performance. But as site-to-site link we only care for ipsec netto transfer rates.
With two cores we could get 400 Mbit/s with that E3-1220L (CPU Load is 55 percent with 230 MBit/s and 2 Cores).

Second: aes-ni not working. With one core and aes-ni I was thinking the calculated performance should be 2 GBit/s.

This can also be verified with vmware and aes-ni capable CPUs, as vmware passes that feature though.

I think this should be focused on, as aes capable cpus will be standard on all systems and this is supported since 2 generations of intel cpus (westmere & sandy bridge).
All other things of pfsense are already more than minimum needed. With ipsec-nat reaching the latest 2.1 this is becoming one big thing to consider at companyside, only missing central managing.

jimp

As far as I know, we still don't have any routers capable of AES-NI in the hands of any developers for testing.

In absence of that, it's going to take some debugging from those that have the hardware.

First step would be to try configuring/using AES-NI on a stock FreeBSD 8.3 image to see if it works for them there.

We are loading the module, which is supposed to be sufficient for actually using it. So the first big question is whether or not we're doing something else in the OS that breaks it, or perhaps it is broken or not configured correctly in the stock OS without our changes.

It's possible that the backporting of AES-NI to FreeBSD 8.3 from 9.x missed something, if that is the case, this probably won't work 100% until we move to a FreeBSD 9.x base. Checking that means comparing the results of the stock FreeBSD 8.3 test with a stock FreeBSD 9.1 test.

miloman

As i've previously stated, if you want to borrow my test-setup for testing please just pm me.

I can set it up with the snapshot of your choice, and provide a jumphost from which you can reach the physical servers.

jimp

@miloman:

As i've previously stated, if you want to borrow my test-setup for testing please just pm me.

I can set it up with the snapshot of your choice, and provide a jumphost from which you can reach the physical servers.

Having remote access in this case isn't really all that helpful, it would take a ton of coordination and such to make the tests happen, since it would involve multiple reinstalls of a few different operating systems (pfSense, FreeBSD 8.3, FreeBSD 9.1) and various tests.

Ideally either someone can run the tests directly on their own hardware, or eventually we'll get hardware on hand that supports it.

miloman

i can install vmware esxi on the hardware… with a jumphost you can do snapshots and reinstall as much as you like. :)

jimp

Ran across something today that might narrow something down.

Can you run this on your board?

# /usr/bin/openssl engine -t -c
# /usr/local/bin/openssl engine -t -c

Also the next round of 1.1 images should have OpenSSL 1.0.1, and from what I've read, that contains better support for AES-NI.

miloman

sorry for the late reply… i've been very busy.

image: pfSense-memstick-2.1-BETA1-i386-20130130-0420.img

/usr/bin/openssl engine -t -c
(cryptodev) BSD cryptodev engine
[RSA, RSA, DH]
[available]
(padlock) VIA PadLock (no-RNG, no-ACE)
[unavailable]
(dynamic) Dynamic engine loading support
[unavailable]

/usr/local/bin/openssl engine -t -c
(cryptodev) BSD cryptodev engine
[RSA, RSA, DH]
[available]
(rdrand) Intel RDRAND engine
[RAND]
[available]
(dynamic) Dynamic engine loading support
[unavailable]
(padlock) VIA PadLock: not supported
[unavailable]

jimp

Is aesni.ko loaded during those tests? (check the output of kldstat)

I would expect to see at least AES-128-CBC in the cryptodev list if it attached, but then again, some others have reported that OpenSSL 1.0.1 did use AES-NI but didn't ever report it as being present, so it may take some more speed tests to tell for sure…