Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    AES-NI / Cryptodev / OpenVPN – help a n00b understand

    Scheduled Pinned Locked Moved OpenVPN
    39 Posts 16 Posters 39.7k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S
      steve28
      last edited by

      All,

      I'm new to pfSense, but not to OpenVPN and "pro" routers/firewalls (coming from a Ubiquiti EdgeRouterX)…

      I just built an APU2C4 and install pfsense 2.3.3 (and upgraded to 2.3.3_1).  The APU2 has AES-NI support, so in System->Advanced-> Misc. I set Cryptographic Hardware to "AES-NI CPU-based Acceleration"

      I then created an OpenVPN client to PrivateInternetAccess and in the setup of the OpenVPN client under Hardware Crypto, I have selected "BSD Cryptodev Engine" (which is what it says to do in the text under System-Advanced->Misc-> Cryptographic Hardware:

      OpenVPN should be set for AES-128-CBC and have cryptodev enabled for hardware acceleration.

      However, many posts on this board say that using cryptodev is actually counter-productive.  I.e., that OpenVPN uses the AES-NI instructions by default, and forcing cryptodev has degraded performance.

      I have tried my OpenVPN client with "No Hardware Crypto Accel" and "BSD cryptodev" and cannot see any real performance difference between the two.

      My non-controlled/non-scientific tests (3 tests each):

      "No Hardware Crypto" selected in OpenVPN:  37.36 Mbit/s, 36.89 Mbit/s, 41.05 Mbit/s
      "BSD cryptodev" selected in OpenVPN: 32.69 Mbit/s, 34.66 Mbit/s, 39.35 Mbit/s

      Can someone definitively answer what the proper settings are (and why)?

      Thank you so much!

      1 Reply Last reply Reply Quote 0
      • H
        heper
        last edited by

        aes-ni on openvpn only gives a performance increase when using pfSense 2.4 beta & using the aes-gcm algorithms

        1 Reply Last reply Reply Quote 0
        • V
          VAMike
          last edited by

          @heper:

          aes-ni on openvpn only gives a performance increase when using pfSense 2.4 beta & using the aes-gcm algorithms

          That is simply not true, I don't understand why it keeps getting repeated.

          1 Reply Last reply Reply Quote 0
          • H
            heper
            last edited by

            What's not true about it?

            1 Reply Last reply Reply Quote 0
            • V
              VAMike
              last edited by

              @steve28:

              However, many posts on this board say that using cryptodev is actually counter-productive.  I.e., that OpenVPN uses the AES-NI instructions by default, and forcing cryptodev has degraded performance.

              I have tried my OpenVPN client with "No Hardware Crypto Accel" and "BSD cryptodev" and cannot see any real performance difference between the two.

              My non-controlled/non-scientific tests (3 tests each):

              "No Hardware Crypto" selected in OpenVPN:  37.36 Mbit/s, 36.89 Mbit/s, 41.05 Mbit/s
              "BSD cryptodev" selected in OpenVPN: 32.69 Mbit/s, 34.66 Mbit/s, 39.35 Mbit/s

              Can someone definitively answer what the proper settings are (and why)?

              It seems that you've already answered that question. AES-NI is always on in OpenSSL unless you go out of your way to set environment variables to cause it to ignore the presence of AES-NI on the system. Turning on the BSD cryptodev in 2.3 makes OpenVPN use the same AES-NI through a kernel interface (/dev/crypto), which adds context switching overhead to every crypto block. Unfortunately, it's just as hard to to make OpenVPN ignore /dev/crypto as it is to make it ignore AES-NI–it assumes that if the device is present you really meant for it to be used. Your test results show this. The amount of overhead varies depending on the platform, but it's always there. In current and prior versions of pfsense there was a gotcha: the buttons in gui basically made it so that you couldn't get AES-NI for IPSEC (a desirable thing) without also getting AES-NI+/dev/crypto for OpenVPN (an undesirable thing). In 2.4 they've made changes so the two things aren't coupled and you can get AES-NI for IPSEC without screwing up OpenVPN. (It's been this way in upstream FreeBSD--by default you get cryptodev [the kernel interface that IPSEC and other kernel modules use] without /dev/crypto [the userspace interface that hurts performance on modern platforms].)

              Now, something seems wrong with your numbers unless you're either on a 40Mbps link or your VPN provider is having trouble keeping up–an APU2 can do more than 40Mbps.

              Bottom line: on older platforms like the CPUs with VIA Padlock or HiFn add-in cards /dev/crypto really was useful, but it's been functionally obsolete since AES-NI was integrated directly into every significant crypto library.

              1 Reply Last reply Reply Quote 0
              • V
                VAMike
                last edited by

                @heper:

                What's not true about it?

                Everything. OpenVPN+OpenSSL has used AES-NI for years.

                1 Reply Last reply Reply Quote 0
                • PippinP
                  Pippin
                  last edited by

                  @VAMike:

                  @heper:

                  aes-ni on openvpn only gives a performance increase when using pfSense 2.4 beta & using the aes-gcm algorithms

                  That is simply not true, I don't understand why it keeps getting repeated.

                  I gave up arguing …:)

                  I gloomily came to the ironic conclusion that if you take a highly intelligent person and give them the best possible, elite education, then you will most likely wind up with an academic who is completely impervious to reality.
                  Halton Arp

                  1 Reply Last reply Reply Quote 0
                  • P
                    pfBasic Banned
                    last edited by

                    @heper:

                    aes-ni on openvpn only gives a performance increase when using pfSense 2.4 beta & using the aes-gcm algorithms

                    :o
                    Where in the world did this silly idea originate?

                    1 Reply Last reply Reply Quote 0
                    • S
                      steve28
                      last edited by

                      @VAMike:

                      …
                      In current and prior versions of pfsense there was a gotcha: the buttons in gui basically made it so that you couldn't get AES-NI for IPSEC (a desirable thing) without also getting AES-NI+/dev/crypto for OpenVPN (an undesirable thing). In 2.4 they've made changes so the two things aren't coupled and you can get AES-NI for IPSEC without screwing up OpenVPN. (It's been this way in upstream FreeBSD--by default you get cryptodev [the kernel interface that IPSEC and other kernel modules use] without /dev/crypto [the userspace interface that hurts performance on modern platforms].)

                      VAMike - thank you for taking the time to explain this!

                      Just to be explicit, you recommend running with AES-NI selected in System->Advanced->Misc and "No Hardware Crypto Acceleration" in my OpenVPN settings? (pfSense 2.3.3_1)

                      1 Reply Last reply Reply Quote 1
                      • S
                        steve28
                        last edited by

                        @VAMike:

                        Now, something seems wrong with your numbers unless you're either on a 40Mbps link or your VPN provider is having trouble keeping up–an APU2 can do more than 40Mbps.

                        Interesting you should should say that.  I have a 100/10 Mbps connection, which at speedtest.net gives me 115/12.

                        Another user in this thread shows a method of benchmarking openvpn at the command line (not through an actual internet connection) and estimating the "max" throughput achievable.  He gets 41 Mbps for the APU2.  I repeated his test on my APU2C4 and got the same result.  His method:

                        @lra:

                        openvpn --genkey --secret /tmp/secret
                        time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-256-cbc
                        

                        then to estimate the max throughput:

                        ( 3200 / execution_time_seconds ) = Projected Maximum OpenVPN Performance in Mbps
                        

                        For example (tested using Linux 3.2.x)…

                        PC Engines APU2 Quad Core AMD GX-412TC:
                        Execution time: 77.3 secs.
                        Maximum OpenVPN: 41 Mbps

                        Jetway NF9HG-2930 Quad Core Celeron N2930:
                        Execution time: 42.4 secs.
                        Maximum OpenVPN: 75 Mbps

                        So far, in my testing, this benchmark comes close to actual Maximum OpenVPN Performance measurements under optimum conditions.  The projected speed should be an upper limit.

                        Note: The magic number of 3200 comes from summing 1 to 20000, multiply by 2 for encrypt and decrypt and by 8 bits/byte and divide by 1,000,000 for a result of Mbps

                        Do you have experience with the APU2 that shows this should be higher?  I would LOVE to get better performance with this box.

                        1 Reply Last reply Reply Quote 0
                        • V
                          VAMike
                          last edited by

                          @steve28:

                          @VAMike:

                          Now, something seems wrong with your numbers unless you're either on a 40Mbps link or your VPN provider is having trouble keeping up–an APU2 can do more than 40Mbps.

                          Interesting you should should say that.  I have a 100/10 Mbps connection, which at speedtest.net gives me 115/12.

                          Another user in this thread shows a method of benchmarking openvpn at the command line (not through an actual internet connection) and estimating the "max" throughput achievable.  He gets 41 Mbps for the APU2.  I repeated his test on my APU2C4 and got the same result.  His method:

                          His method is a heuristic, good for rough estimates of performance. People are putting too much into it. That said, while I'm sure I saw at least a bit more than 40Mbps on an APU2, I don't currently have one on the end of a link fast enough to confirm that, and it was almost certainly on Linux so it might not be comparable. Also, try AES-128 instead of AES-256.

                          1 Reply Last reply Reply Quote 0
                          • S
                            steve28
                            last edited by

                            Just wanted to follow up on APU2 VPN performance for future searchers:

                            I turned OFF every mention of crypto hardware acceleration, which did show an increase in performance.

                            To PIA, I was able to get as high as 72 Mbps and can get reliably above 60 Mbps.  I haven't had time to set up a test to a server of my own where I can eliminate the internet/PIA servers as a factor, but I am happy with this performance.  If/when I get around to setting up a controlled test I will report back.

                            1 Reply Last reply Reply Quote 0
                            • P
                              pfBasic Banned
                              last edited by

                              A controlled test of what?

                              1 Reply Last reply Reply Quote 0
                              • S
                                steve28
                                last edited by

                                @pfBasic:

                                A controlled test of what?

                                "Controlled test" as in I can control all of the variables.

                                My test was done to Private Internet Access servers.  I haven't had time to set up a test where I put my own VPN server on the WAN interface using a high-powered desktop so that I'm truely testing the APU's OpenVPN throughput.  As it was my speeds could have been limited by the load on the PIA servers, slow internet to my house, or some other bottleneck along the way.

                                1 Reply Last reply Reply Quote 0
                                • P
                                  pfBasic Banned
                                  last edited by

                                  Don't waste your time. If your CPU supports AES-NI, use it, it's used by default without you selecting BSD cryptodev.

                                  It looks like I found the source of at least one of this forums users incorrect claim that "AES-NI didn't work well prior to 2.4 and only works on AEAD ciphers" , an unsupported reddit post…  ::). That line is a bunch of crap. AES-NI works great on pfSense prior to 2.4.
                                  https://forum.pfsense.org/index.php?topic=129246.msg713031#msg713031

                                  You can't believe everything you read on the internet. If something goes against the grain, you are probably better off not taking it seriously unless that something can provide a solid reference, or at least a strong argument.

                                  Reddit/Forum posts ≠ Solid Reference

                                  This is true of my posts also, but you'll find a whitepaper linked in my post.

                                  1 Reply Last reply Reply Quote 0
                                  • jimpJ
                                    jimp Rebel Alliance Developer Netgate
                                    last edited by

                                    The origin actually is much older than that.

                                    With the BSD Cryptodev engine loaded along with the AES-NI module, OpenVPN would latch onto that instead of using AES-NI, resulting in lower speeds because the BSD Cryptodev hooks for AES-NI only supported AES-GCM, while claiming to support more. Before 2.4, you could not run without the BSD cryptodev engine active, and on 2.4 you can.

                                    Now if you didn't have the AES-NI module loaded, it wouldn't matter, OpenVPN would latch onto it and use it to accelerate anything it could. But you couldn't accelerate AES-GCM with IPsec without the AES-NI module loaded.

                                    For a proper set of tests on 2.4 you'd need to run with a variety of settings in OpenVPN while also testing with the modules in their various states (aesni.ko loaded vs unloaded, cryptodev.ko loaded vs unloaded, both loaded, neither loaded).

                                    Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                                    Need help fast? Netgate Global Support!

                                    Do not Chat/PM for help!

                                    1 Reply Last reply Reply Quote 0
                                    • S
                                      steve28
                                      last edited by

                                      @pfBasic:

                                      Don't waste your time. If your CPU supports AES-NI, use it, it's used by default without you selecting BSD cryptodev.

                                      I know it's being used… what I'm interested in is what the max openVPN capability is using the PC Engines APU2 board with the AES-128-CBC cipher.

                                      For that I need a a controlled test where I remove the internet from the equation.

                                      1 Reply Last reply Reply Quote 0
                                      • BBcan177B
                                        BBcan177 Moderator
                                        last edited by

                                        https://blog.cloudflare.com/aes-cbc-going-the-way-of-the-dodo/

                                        "Experience is something you don't get until just after you need it."

                                        Website: http://pfBlockerNG.com
                                        Twitter: @BBcan177  #pfBlockerNG
                                        Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                                        1 Reply Last reply Reply Quote 0
                                        • R
                                          rdrcrmatt
                                          last edited by

                                          @steve28:

                                          @pfBasic:

                                          Don't waste your time. If your CPU supports AES-NI, use it, it's used by default without you selecting BSD cryptodev.

                                          I know it's being used… what I'm interested in is what the max openVPN capability is using the PC Engines APU2 board with the AES-128-CBC cipher.

                                          For that I need a a controlled test where I remove the internet from the equation.

                                          I'm beating on this myself right now and I'm a little disappointed, or perhaps I have it setup poorly.

                                          I tried a single connection to PIA w/ OpenVPN, APU2d, AES-NI is not turned on anyway (because this thread says not to).  I have multiple IPsec connections to remote clients, I use OpenVPN for outbound Internet activity.

                                          I've tried every mix of cryptographic hardware settings.

                                          System/Advanced/Miscellaneous - Cryptographic Hardware - AES-NI  (haven't tried AMD Gecode LX yet but I don't think it applies)
                                          In OpenVPN I've turned it on and off.

                                          I direct clients out the OpenVPN by way of an alias, that is referenced in a FW rule that points them to the PIA / OpenVPN Interface gateway.  I even tried 4 OpenVPN connections, 4 interfaces, 4 gateways all in a gateway group to try to get the OpenVPN processes to run on different cores in the APU2d.  I still moved around the same amount of traffic.  No matter what I get 10-20mbit/sec.  On top of that, I only see CPU utilizations in the 20% range and usually only on one core.  The others are around 5%.

                                          I'd love to find a way to max out my Internet connection (300/200), I'm thinking I just need to install an OpenVPN client on the few server that need to move a ton of traffic but I'd really like my router/fw to do all the networking tasks.

                                          1 Reply Last reply Reply Quote 0
                                          • P
                                            pfBasic Banned
                                            last edited by

                                            @rdrcrmatt:

                                            in the APU2d…

                                            I'd love to find a way to max out my Internet connection (300/200)

                                            You aren't going to max out 500Mbps of OpenVPN throughput on 4 x 1GHz cores from 2014.

                                            You won't get close. Not even with gateway groups.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.