IPSEC with QAT - low performance (Netgate CPIC-8955)

[alexandru.racovita]

Hello everybody,

I'm having trouble getting normal speed performance using IPSEC with QAT (CPIC-8955) in pfSense+ (22.05-RELEASE / FreeBSD 12.3-STABLE).

This is the maximum speed I achieved with 10GbE network adapters: (does not exceed 1Gb/s even with 4 clients or parallel connections)

iperf3 -c 10.10.100.1 -p 5101
Connecting to host 10.10.100.1, port 5101
[  5] local 10.0.0.1 port 51254 connected to 10.10.100.1 port 5101
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.01   sec   100 MBytes   837 Mbits/sec    0    126 KBytes
[  5]   1.01-2.01   sec   100 MBytes   835 Mbits/sec    0    126 KBytes

Server configuration (Pfsense+): Intel Xeon CPU E5-2680 v4 @ 2.40GHz (14 cores / 28 threads), 64 GB RAM, OS installed on NVMe, 10GbE NIC (Intel X520-DA2), X99-F8 Motherboard

Client configuration (Ubuntu 22.04.1 LTS): HP Z420, Intel Xeon E5-1620 v2 @ 3.7Ghz 8 cores, 32 GB RAM, OS installed on Samsung 850 SSD, 10GbE NIC (Intel 82599)

It's not a network problem, as you can see, it's close to 10Gb/s without IPSEC tunnel:

iperf3 -c 10.10.100.1 -p 5101
Connecting to host 10.10.100.1, port 5101
[  5] local 10.10.200.2 port 47648 connected to 10.10.100.1 port 5101
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.41 MBytes
[  5]   1.00-2.00   sec  1.15 GBytes  9.90 Gbits/sec    0   1.41 MBytes

I activated Intel Quick Assist (QAT) from System - Advanced - Cryptographic Hardware and it is started in the dashboard - QAT Crypto: Yes (active)

The board is recognized:

/root: dmesg | grep qat
qat0: <Intel 8950 QuickAssist PCIe Adapter PF> mem 0xf2200000-0xf227ffff,0xfb640000-0xfb67ffff,0xfb600000-0xfb63ffff irq 32 at device 0.0 numa-domain 0 on pci4

The drivers are loaded:

/root: kldstat | grep qat
 4    1 0xffffffff840aa000    e3510 qat_dh895xccfw.ko
 5    1 0xffffffff8418e000    1f2a0 qat.ko

It looks like the QAT adapter is used (according to this):

vmstat -i | grep qat
irq278: qat0                       15260          6
irq279: qat0                       11767          5
irq280: qat0                       17667          7
irq281: qat0                       15827          6
......

Tunnel config:

Clients successfully connect to IPSEC VPN using AES_GCM_16 (128), PRF_HMAC_SHA1

I would like to add that I also tried different tunables as shown here but without success.

Can you help me, please, to reach the normal performance of the QAT adapter?

In the past I tested with 100GbE adapters, I am aware that without VPP and TNSR I cannot reach the maximum performance, but 800Mb/s for 1 stream is a very poor speed in IPSEC tunnel. (CPIC-895 supports 50 Gbps according to specifications)

Attached you will find the backup of the pfsense+ configuration in different stages of changes, tests as well as other related outputs (loader.conf, iperf, network config, sysctl). -> pfsense+ qat tests.zip

I also want to state that I tested this QAT adapter on a different server with the same results.

Thank you,
Alex

[alexandru.racovita]

Hi, I didn't solve it even after updating to 23.01-RELEASE (amd64), FreeBSD 14.0-CURRENT. Can someone help me please?
I am additionally attaching the openssl rdrand and devcrypto tests, between which there is no difference, I get the same result without the QAT card on AES-NI.

WITH HARDWARE ACCELERATION (rdrand + devcrypto):

/root: openssl engine
(devcrypto) /dev/crypto engine
(rdrand) Intel RDRAND engine
(dynamic) Dynamic engine loading support


/root: openssl speed -engine rdrand -evp aes-128-gcm
engine "rdrand" set.
Doing aes-128-gcm for 3s on 16 size blocks: 109473266 aes-128-gcm's in 3.15s
Doing aes-128-gcm for 3s on 64 size blocks: 59620644 aes-128-gcm's in 3.06s
Doing aes-128-gcm for 3s on 256 size blocks: 37145965 aes-128-gcm's in 3.05s
Doing aes-128-gcm for 3s on 1024 size blocks: 12758891 aes-128-gcm's in 3.07s
Doing aes-128-gcm for 3s on 8192 size blocks: 1961291 aes-128-gcm's in 3.06s
Doing aes-128-gcm for 3s on 16384 size blocks: 1004601 aes-128-gcm's in 3.09s
OpenSSL 1.1.1t-freebsd  7 Feb 2023
built on: reproducible build, date unspecified
options:bn(64,64) rc4(8x,int) des(int) aes(partial) idea(int) blowfish(ptr)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-128-gcm     556330.64k  1245949.78k  3121023.03k  4255301.17k  5246333.35k  5320204.54k


/root: openssl speed -engine devcrypto -evp aes-128-gcm
engine "devcrypto" set.
Doing aes-128-gcm for 3s on 16 size blocks: 109588628 aes-128-gcm's in 3.09s
Doing aes-128-gcm for 3s on 64 size blocks: 58764133 aes-128-gcm's in 3.08s
Doing aes-128-gcm for 3s on 256 size blocks: 36989212 aes-128-gcm's in 3.08s
Doing aes-128-gcm for 3s on 1024 size blocks: 12517930 aes-128-gcm's in 3.03s
Doing aes-128-gcm for 3s on 8192 size blocks: 1892616 aes-128-gcm's in 3.01s
Doing aes-128-gcm for 3s on 16384 size blocks: 962895 aes-128-gcm's in 3.02s
OpenSSL 1.1.1t-freebsd  7 Feb 2023
built on: reproducible build, date unspecified
options:bn(64,64) rc4(8x,int) des(int) aes(partial) idea(int) blowfish(ptr)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-128-gcm     566761.39k  1221816.69k  3076300.76k  4228737.43k  5154679.78k  5217925.52k

NO HARDWARE ACCELERATION:

/root: openssl engine
(rdrand) Intel RDRAND engine
(dynamic) Dynamic engine loading support


/root: openssl speed -engine rdrand -evp aes-128-gcm
engine "rdrand" set.
Doing aes-128-gcm for 3s on 16 size blocks: 102136466 aes-128-gcm's in 3.01s
Doing aes-128-gcm for 3s on 64 size blocks: 60435126 aes-128-gcm's in 3.16s
Doing aes-128-gcm for 3s on 256 size blocks: 36288986 aes-128-gcm's in 3.04s
Doing aes-128-gcm for 3s on 1024 size blocks: 12394560 aes-128-gcm's in 3.03s
Doing aes-128-gcm for 3s on 8192 size blocks: 1920849 aes-128-gcm's in 3.00s
Doing aes-128-gcm for 3s on 16384 size blocks: 1021912 aes-128-gcm's in 3.18s
OpenSSL 1.1.1t-freebsd  7 Feb 2023
built on: reproducible build, date unspecified
options:bn(64,64) rc4(8x,int) des(int) aes(partial) idea(int) blowfish(ptr)
compiler: clang
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes  16384 bytes
aes-128-gcm     543312.94k  1225456.81k  3056857.31k  4187061.26k  5245198.34k  5265613.75k


/root: openssl speed -engine devcrypto -evp aes-128-gcm
invalid engine "devcrypto"