IPSec throughput with pfsense



  • Hey,

    I've 2 appliances here (Nexcom NSA31500, i3 CPU AES-NI, 8GB Ram), both installed with Debian Linux and Libreswan. Throughput with Netkey was 903Mbit/s (TCP, IPerf).
    Now I installed PFSense on one appliance and made a VPN with AES256/SHA256/DH14 (P1 and P2). First I had many fragments and set lower MSS.
    Also AES-NI is activated: aesni0: <aes-cbc,aes-xts,aes-gcm>on motherboard

    But whatever I do, I only geht around 435Mbit/s over the VPN.

    Is this normal behavior? Any ideas how to tune this up?

    Thanks!</aes-cbc,aes-xts,aes-gcm>



  • You need AES-GCM cipher to get max performance.



  • Hi ermal,

    just switched to AES-GCM, possibly found a bug.
    The VPN gets established and when I ping from PF LAN, the Ping gets replied at the other side, also I can see the ping in ESP packets, but on PFsense LAN interface there's no outgoing packet. When I switch back to AES256/SHA2 on Phase2 the replies are fine!

    PFsense release is from friday (will update now). P1 is AES256/SHA2, P2 is AES128GCM, no Hash, DH14.

    Here's the successful log from the linux side:

    002 "vpn" #14: initiating Main Mode
    104 "vpn" #14: STATE_MAIN_I1: initiate
    003 "vpn" #14: received Vendor ID payload [XAUTH]
    003 "vpn" #14: received Vendor ID payload [Dead Peer Detection]
    003 "vpn" #14: received Vendor ID payload [Cisco-Unity]
    003 "vpn" #14: received Vendor ID payload [FRAGMENTATION 80000000]
    003 "vpn" #14: received Vendor ID payload [RFC 3947]
    002 "vpn" #14: enabling possible NAT-traversal with method RFC 3947 (NAT-Traversal)
    002 "vpn" #14: transition from state STATE_MAIN_I1 to state STATE_MAIN_I2
    106 "vpn" #14: STATE_MAIN_I2: sent MI2, expecting MR2
    003 "vpn" #14: NAT-Traversal: Result using RFC 3947 (NAT-Traversal) sender port 500: no NAT detected
    002 "vpn" #14: transition from state STATE_MAIN_I2 to state STATE_MAIN_I3
    108 "vpn" #14: STATE_MAIN_I3: sent MI3, expecting MR3
    002 "vpn" #14: Main mode peer ID is ID_IPV4_ADDR: '10.10.11.100'
    002 "vpn" #14: transition from state STATE_MAIN_I3 to state STATE_MAIN_I4
    004 "vpn" #14: STATE_MAIN_I4: ISAKMP SA established {auth=PRESHARED_KEY cipher=aes_256 integ=OAKLEY_SHA2_256 group=MODP2048}
    002 "vpn" #15: initiating Quick Mode PSK+ENCRYPT+TUNNEL+PFS+UP+IKEV2_ALLOW+SAREF_TRACK+IKE_FRAG_ALLOW {using isakmp#14 msgid:dced4920 proposal=AES_GCM_C(20)_000-NONE(0)_000 pfsgroup=OAKLEY_GROUP_MODP2048}
    117 "vpn" #15: STATE_QUICK_I1: initiate
    002 "vpn" #15: transition from state STATE_QUICK_I1 to state STATE_QUICK_I2
    004 "vpn" #15: STATE_QUICK_I2: sent QI2, IPsec SA established tunnel mode {ESP=>0xc6f6cf20 <0xb1d3f298 xfrm=AES_GCM_C_128-NONE NATOA=none NATD=none DPD=passive}

    Any ideas?



  • I installed on both sides a webserver with DocRoot in RAM Disk.
    Create a 2GB file and with AES256 I got 55MB/s out and 45MB/s in. Switching to AES-GCM I can start downloading the file but the transfer stops after some seconds (MTU?). Uploading is not possible, no successful 3way handshake.



  • I have tested AES-GCM with linux and it surely works.
    You need to enable AESNI module to get performance proper.

    Though what you say means some configuration or MTU issue of sort though if you have it working with AES there should be no change in that regard with AES-GCM.

    Can you put net.inet.ipsec.debug=0xffffff and see what comes out on dmesg -a after that with AES-GCM traffic going?



  • @ermal:

    I have tested AES-GCM with linux and it surely works.
    You need to enable AESNI module to get performance proper.

    I disabled AESNI via WebUI and tested again, same speed with AES256/SHA2, turned on again (dmesg: aesni0: <aes-cbc,aes-xts,aes-gcm>on motherboard), same speed.

    @ermal:

    Though what you say means some configuration or MTU issue of sort though if you have it working with AES there should be no change in that regard with AES-GCM.

    Doublechecked everything, MTU is 1500. I lower the NIC MTU on the clients to 1300, but I can't even Ping hosts with AES-GCM :(

    @ermal:

    Can you put net.inet.ipsec.debug=0xffffff and see what comes out on dmesg -a after that with AES-GCM traffic going?

    There you go:
    esp_input: payload of 56 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
    esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
    esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
    esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
    esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
    esp_input: payload of 56 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
    esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
    esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
    esp_input: payload of 56 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
    esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
    esp_input: payload of 56 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
    esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
    esp_input: payload of 56 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
    esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
    esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
    esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
    esp_input: payload of 56 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
    esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
    esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087

    On the client behind pfsense I did a ping and on pfsense a tcpdump on WAN:
    08:24:13.405650 IP 10.10.11.100 > 10.10.10.100: ESP(spi=0x81bb30b5,seq=0x190), length 128
    08:24:13.405808 IP 10.10.10.100 > 10.10.11.100: ESP(spi=0xc8c2a087,seq=0x1d7), length 120

    As you can see, the ping will be responded, but echo reply does not leave LAN interface

    I'll now install pfsense on the other box too.</aes-cbc,aes-xts,aes-gcm>



  • This is with the latest of pfSense snapshots?

    The padding should be done by the host on this and i expect linux to not send such frames without multiple of block size!



  • Ok, tests are finished successfully!
    I installed pfSense on the other box too. Latest RC on both now.
    With AES256 I get in and out 45MB/s, with AES-GCM everything works now and I get 106MB/s in and out.

    I'll go to libreswan mailing list to check if they have a error in the code.

    Thanks for your patience and the fast help!

    Will put all the result to www.routerperformance.net



  • That performance is from software test only i guess.
    AES-GCM can go up to 90% link speed if you have the CPU power with the AESNI module loaded.

    This was seen in tests performed internally with pfSense.



  • Sorry for coming back, but now I'm testing compability between pfSense and ASA5515 (9.3.2).
    Same situation, with AES-GCM there's no ping going through, switching back to AES256/SHA2 everything works fine.

    It seems to me theres a workaround or better unterstanding between 2 pfSense boxes :(



  • Can you let me know if you get the same error on dmesg -a when enabling that sysctl as before?

    FYII https://redmine.pfsense.org/issues/4248



  • Yes, same error with ASA:

    esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.10.100/c95c270c
    esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.10.100/c95c270c
    esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.10.100/c95c270c
    esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.10.100/c95c270c
    esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.10.100/c95c270c
    esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.10.100/c95c270c



  • A fix will go in for 2.2 that will correct the issue.


Log in to reply