Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    IPSec throughput with pfsense

    Scheduled Pinned Locked Moved General pfSense Questions
    13 Posts 2 Posters 6.7k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • W Offline
      Wordo
      last edited by

      Hey,

      I've 2 appliances here (Nexcom NSA31500, i3 CPU AES-NI, 8GB Ram), both installed with Debian Linux and Libreswan. Throughput with Netkey was 903Mbit/s (TCP, IPerf).
      Now I installed PFSense on one appliance and made a VPN with AES256/SHA256/DH14 (P1 and P2). First I had many fragments and set lower MSS.
      Also AES-NI is activated: aesni0: <aes-cbc,aes-xts,aes-gcm>on motherboard

      But whatever I do, I only geht around 435Mbit/s over the VPN.

      Is this normal behavior? Any ideas how to tune this up?

      Thanks!</aes-cbc,aes-xts,aes-gcm>

      1 Reply Last reply Reply Quote 0
      • E Offline
        eri--
        last edited by

        You need AES-GCM cipher to get max performance.

        1 Reply Last reply Reply Quote 0
        • W Offline
          Wordo
          last edited by

          Hi ermal,

          just switched to AES-GCM, possibly found a bug.
          The VPN gets established and when I ping from PF LAN, the Ping gets replied at the other side, also I can see the ping in ESP packets, but on PFsense LAN interface there's no outgoing packet. When I switch back to AES256/SHA2 on Phase2 the replies are fine!

          PFsense release is from friday (will update now). P1 is AES256/SHA2, P2 is AES128GCM, no Hash, DH14.

          Here's the successful log from the linux side:

          002 "vpn" #14: initiating Main Mode
          104 "vpn" #14: STATE_MAIN_I1: initiate
          003 "vpn" #14: received Vendor ID payload [XAUTH]
          003 "vpn" #14: received Vendor ID payload [Dead Peer Detection]
          003 "vpn" #14: received Vendor ID payload [Cisco-Unity]
          003 "vpn" #14: received Vendor ID payload [FRAGMENTATION 80000000]
          003 "vpn" #14: received Vendor ID payload [RFC 3947]
          002 "vpn" #14: enabling possible NAT-traversal with method RFC 3947 (NAT-Traversal)
          002 "vpn" #14: transition from state STATE_MAIN_I1 to state STATE_MAIN_I2
          106 "vpn" #14: STATE_MAIN_I2: sent MI2, expecting MR2
          003 "vpn" #14: NAT-Traversal: Result using RFC 3947 (NAT-Traversal) sender port 500: no NAT detected
          002 "vpn" #14: transition from state STATE_MAIN_I2 to state STATE_MAIN_I3
          108 "vpn" #14: STATE_MAIN_I3: sent MI3, expecting MR3
          002 "vpn" #14: Main mode peer ID is ID_IPV4_ADDR: '10.10.11.100'
          002 "vpn" #14: transition from state STATE_MAIN_I3 to state STATE_MAIN_I4
          004 "vpn" #14: STATE_MAIN_I4: ISAKMP SA established {auth=PRESHARED_KEY cipher=aes_256 integ=OAKLEY_SHA2_256 group=MODP2048}
          002 "vpn" #15: initiating Quick Mode PSK+ENCRYPT+TUNNEL+PFS+UP+IKEV2_ALLOW+SAREF_TRACK+IKE_FRAG_ALLOW {using isakmp#14 msgid:dced4920 proposal=AES_GCM_C(20)_000-NONE(0)_000 pfsgroup=OAKLEY_GROUP_MODP2048}
          117 "vpn" #15: STATE_QUICK_I1: initiate
          002 "vpn" #15: transition from state STATE_QUICK_I1 to state STATE_QUICK_I2
          004 "vpn" #15: STATE_QUICK_I2: sent QI2, IPsec SA established tunnel mode {ESP=>0xc6f6cf20 <0xb1d3f298 xfrm=AES_GCM_C_128-NONE NATOA=none NATD=none DPD=passive}

          Any ideas?

          1 Reply Last reply Reply Quote 0
          • W Offline
            Wordo
            last edited by

            I installed on both sides a webserver with DocRoot in RAM Disk.
            Create a 2GB file and with AES256 I got 55MB/s out and 45MB/s in. Switching to AES-GCM I can start downloading the file but the transfer stops after some seconds (MTU?). Uploading is not possible, no successful 3way handshake.

            1 Reply Last reply Reply Quote 0
            • E Offline
              eri--
              last edited by

              I have tested AES-GCM with linux and it surely works.
              You need to enable AESNI module to get performance proper.

              Though what you say means some configuration or MTU issue of sort though if you have it working with AES there should be no change in that regard with AES-GCM.

              Can you put net.inet.ipsec.debug=0xffffff and see what comes out on dmesg -a after that with AES-GCM traffic going?

              1 Reply Last reply Reply Quote 0
              • W Offline
                Wordo
                last edited by

                @ermal:

                I have tested AES-GCM with linux and it surely works.
                You need to enable AESNI module to get performance proper.

                I disabled AESNI via WebUI and tested again, same speed with AES256/SHA2, turned on again (dmesg: aesni0: <aes-cbc,aes-xts,aes-gcm>on motherboard), same speed.

                @ermal:

                Though what you say means some configuration or MTU issue of sort though if you have it working with AES there should be no change in that regard with AES-GCM.

                Doublechecked everything, MTU is 1500. I lower the NIC MTU on the clients to 1300, but I can't even Ping hosts with AES-GCM :(

                @ermal:

                Can you put net.inet.ipsec.debug=0xffffff and see what comes out on dmesg -a after that with AES-GCM traffic going?

                There you go:
                esp_input: payload of 56 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
                esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
                esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
                esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
                esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
                esp_input: payload of 56 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
                esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
                esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
                esp_input: payload of 56 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
                esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
                esp_input: payload of 56 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
                esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
                esp_input: payload of 56 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
                esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
                esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
                esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
                esp_input: payload of 56 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
                esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087
                esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.11.100/c8c2a087

                On the client behind pfsense I did a ping and on pfsense a tcpdump on WAN:
                08:24:13.405650 IP 10.10.11.100 > 10.10.10.100: ESP(spi=0x81bb30b5,seq=0x190), length 128
                08:24:13.405808 IP 10.10.10.100 > 10.10.11.100: ESP(spi=0xc8c2a087,seq=0x1d7), length 120

                As you can see, the ping will be responded, but echo reply does not leave LAN interface

                I'll now install pfsense on the other box too.</aes-cbc,aes-xts,aes-gcm>

                1 Reply Last reply Reply Quote 0
                • E Offline
                  eri--
                  last edited by

                  This is with the latest of pfSense snapshots?

                  The padding should be done by the host on this and i expect linux to not send such frames without multiple of block size!

                  1 Reply Last reply Reply Quote 0
                  • W Offline
                    Wordo
                    last edited by

                    Ok, tests are finished successfully!
                    I installed pfSense on the other box too. Latest RC on both now.
                    With AES256 I get in and out 45MB/s, with AES-GCM everything works now and I get 106MB/s in and out.

                    I'll go to libreswan mailing list to check if they have a error in the code.

                    Thanks for your patience and the fast help!

                    Will put all the result to www.routerperformance.net

                    1 Reply Last reply Reply Quote 0
                    • E Offline
                      eri--
                      last edited by

                      That performance is from software test only i guess.
                      AES-GCM can go up to 90% link speed if you have the CPU power with the AESNI module loaded.

                      This was seen in tests performed internally with pfSense.

                      1 Reply Last reply Reply Quote 0
                      • W Offline
                        Wordo
                        last edited by

                        Sorry for coming back, but now I'm testing compability between pfSense and ASA5515 (9.3.2).
                        Same situation, with AES-GCM there's no ping going through, switching back to AES256/SHA2 everything works fine.

                        It seems to me theres a workaround or better unterstanding between 2 pfSense boxes :(

                        1 Reply Last reply Reply Quote 0
                        • E Offline
                          eri--
                          last edited by

                          Can you let me know if you get the same error on dmesg -a when enabling that sysctl as before?

                          FYII https://redmine.pfsense.org/issues/4248

                          1 Reply Last reply Reply Quote 0
                          • W Offline
                            Wordo
                            last edited by

                            Yes, same error with ASA:

                            esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.10.100/c95c270c
                            esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.10.100/c95c270c
                            esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.10.100/c95c270c
                            esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.10.100/c95c270c
                            esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.10.100/c95c270c
                            esp_input: payload of 88 octets not a multiple of 16 octets,  SA 10.10.10.100/c95c270c

                            1 Reply Last reply Reply Quote 0
                            • E Offline
                              eri--
                              last edited by

                              A fix will go in for 2.2 that will correct the issue.

                              1 Reply Last reply Reply Quote 0
                              • First post
                                Last post
                              Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.