AES-NI / Cryptodev / OpenVPN – help a n00b understand

c0lp4nik

Greetings-

Re-posting here as this is an akin topic:

Greetings!

Long-time listener, first-time caller.

I have been running pfSense in Azure (not the Netgate addition, sorry Netgate on a tight budget right now…) for sometime and and just upgraded to pfSense 2.4 and noticed that speeds from the appliance itself get 250-300 Mbps download tested with iperf (client) against he.net and scottlinux.com (public iperf servers), but my openvpn 2.4 (not to be confused with pfSense 2.4) clients are only getting a symmetric MAX 6 Mbps download and upload "capped".

I have no limiters in place:

ipfw show pipe - blank.
XML - none.

My /temp/rules.limits:

set limit table-entries 2000000
set optimization conservative
set timeout { udp.first 300, udp.single 150, udp.multiple 900 }
set limit states 1429000
set limit src-nodes 1429000

(which I am assuming is default, as I have no limits pushed to XML via the GUI).

Note: AES-NI Accel is noted:
CPU Type Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
4 CPUs: 1 package(s) x 4 core(s)
AES-NI CPU Crypto: Yes (active) -----------> CHECK!
Hardware crypto AES-CBC,AES-XTS,AES-GCM,AES-ICM

Openvpn Crypto used: AES-256-CBC (CHECK!)

OpenVPN config (Screen in GUI): Hardware Crypto: BSD Cryptodev......

Checked kernel mods loaded:

kldstat
Id Refs Address Size Name
1 8 0xffffffff80200000 2c3e9a0 kernel
2 1 0xffffffff83019000 46c6 cryptodev.ko
3 1 0xffffffff8301e000 7f92 aesni.ko

On-board speed test:

openssl speed -evp aes-256-cbc

Doing aes-256-cbc for 3s on 16 size blocks: 1240941 aes-256-cbc's in 0.11s
Doing aes-256-cbc for 3s on 64 size blocks: 1143048 aes-256-cbc's in 0.13s
Doing aes-256-cbc for 3s on 256 size blocks: 877391 aes-256-cbc's in 0.07s
Doing aes-256-cbc for 3s on 1024 size blocks: 500204 aes-256-cbc's in 0.07s
Doing aes-256-cbc for 3s on 8192 size blocks: 95778 aes-256-cbc's in 0.02s
OpenSSL 1.0.2k-freebsd 26 Jan 2017
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-cbc 181531.94k 550814.66k 3194483.14k 7284748.74k 33476837.38k

Baffled. <shrugs shoulders="">.... :-\

This thread proved extremely insightful, however I am still not breaking the 6 Mbps barrier <sheds tear...=""> :'(

Any insight or corrections appreciated!

Thanks much!
C0l. P.</sheds></shrugs>

VAMike

@c0lp4nik:

On-board speed test:

openssl speed -evp aes-256-cbc

Doing aes-256-cbc for 3s on 16 size blocks: 1240941 aes-256-cbc's in 0.11s
Doing aes-256-cbc for 3s on 64 size blocks: 1143048 aes-256-cbc's in 0.13s
Doing aes-256-cbc for 3s on 256 size blocks: 877391 aes-256-cbc's in 0.07s
Doing aes-256-cbc for 3s on 1024 size blocks: 500204 aes-256-cbc's in 0.07s
Doing aes-256-cbc for 3s on 8192 size blocks: 95778 aes-256-cbc's in 0.02s
OpenSSL 1.0.2k-freebsd 26 Jan 2017
built on: date not available
options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes
aes-256-cbc 181531.94k 550814.66k 3194483.14k 7284748.74k 33476837.38k

Bogus numbers, you have cryptodev enabled and aren't using -elapsed. You're not getting 7GByte/s with 1k blocks, you're getting ~170MByte/s.

Turn off cryptodev.

You may still not get great speeds, because you may be sharing a CPU with other VMs, but it shouldn't be that bad.

c0lp4nik

Thanks for the feedback VAMike!

I could be posting as MDCP (from the other side of the river…)..

So have now tested with "NO Hardware Crypto Accel" set in the VPN config (GUI), and with AES-NI enabled.

Same result :( on pfSense 2.3.4-P1, OpenVPN 2.3.17, and on pfSense 2.4, OpenVPN 2.4.4, respectively...

<shurg>....caveat it's on Azure, but it's a Quad-core with 14GB RAM....you'd think that should handle it...

No LB or other shaping devices in between....

Anything I can offer that might trigger an idear?

Thanks so much in advance!
CP</shurg>

VAMike

@c0lp4nik:

Thanks for the feedback VAMike!

I could be posting as MDCP (from the other side of the river…)..

So have now tested with "NO Hardware Crypto Accel" set in the VPN config (GUI), and with AES-NI enabled.

Same result :( on pfSense 2.3.4-P1, OpenVPN 2.3.17, and on pfSense 2.4, OpenVPN 2.4.4, respectively...

<shurg>....caveat it's on Azure, but it's a Quad-core with 14GB RAM....you'd think that should handle it...

No LB or other shaping devices in between....

Anything I can offer that might trigger an idear?

Thanks so much in advance!
CP</shurg>

If you run the openssl speed test without -elapsed again and cryptodev not loaded, you should be getting about 500MByte/s if my back of the envelope math is right. If you're getting significantly less than that you're losing cycles on the VM. If the crypto rate looks about right, then check the logs for stuff like MTU warnings or other problems. That data rate is low enough that either something is broken or something on the network is intentionally or unintentionally throttling you. That's a tough thing for an armchair diagnosis, unfortunately. Also, with openvpn 2.4 you can configure AES-128-GCM, which should perform better than AES-256-CBC, but you're still so far below the expected limit of AES-CBC on that hardware that I wouldn't expect a miracle.

sensemann

Hello!
what are the best settings for OpenVPN HIDEME_VPN with

APU4B4 and pfsense 2.4.3-RELEASE-p1 ?
(AMD Embedded G series GX-412TC, 1 GHz quad Jaguar core with 64 bit and AES-NI support)

I ask because I get only around 8-10MBit down and 18MBit up.
Without OpenVPN it is 220Mbit down and 20Mbit up.

On a System with an Atom C2358, SoC,
(Rangeley), 7W 2-Core, 1.7-2.0GHz (Board: A1SRM-LN7F-2358 ) I get similar low rates.

Thanks!

sensemann

any news?

bcruze

@sensemann said in AES-NI / Cryptodev / OpenVPN – help a n00b understand:

any news?

Have you increase you fast IO buffers to at least 256?

sensemann

the red marked values I set now..

bcruze

You have one set for 256 one for 512 not sure if that will cause an issue

It’s in custom options and the drop down to 256

beefer

Sorry to necrobump, but this should be pinned in official pfsense OpenVPN tutorials. Two years I've been using ~30-40Mps VPN being sure it's speed is limited by the provider. I just tested snd/rcvbuffer and fast-io and immediately landed on stable 60Mbps. Holy smokes! Thanks for making my life better :)