Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    AES-NI / Cryptodev / OpenVPN – help a n00b understand

    Scheduled Pinned Locked Moved OpenVPN
    39 Posts 16 Posters 39.8k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • R
      rdrcrmatt
      last edited by

      @steve28:

      @pfBasic:

      Don't waste your time. If your CPU supports AES-NI, use it, it's used by default without you selecting BSD cryptodev.

      I know it's being used… what I'm interested in is what the max openVPN capability is using the PC Engines APU2 board with the AES-128-CBC cipher.

      For that I need a a controlled test where I remove the internet from the equation.

      I'm beating on this myself right now and I'm a little disappointed, or perhaps I have it setup poorly.

      I tried a single connection to PIA w/ OpenVPN, APU2d, AES-NI is not turned on anyway (because this thread says not to).  I have multiple IPsec connections to remote clients, I use OpenVPN for outbound Internet activity.

      I've tried every mix of cryptographic hardware settings.

      System/Advanced/Miscellaneous - Cryptographic Hardware - AES-NI  (haven't tried AMD Gecode LX yet but I don't think it applies)
      In OpenVPN I've turned it on and off.

      I direct clients out the OpenVPN by way of an alias, that is referenced in a FW rule that points them to the PIA / OpenVPN Interface gateway.  I even tried 4 OpenVPN connections, 4 interfaces, 4 gateways all in a gateway group to try to get the OpenVPN processes to run on different cores in the APU2d.  I still moved around the same amount of traffic.  No matter what I get 10-20mbit/sec.  On top of that, I only see CPU utilizations in the 20% range and usually only on one core.  The others are around 5%.

      I'd love to find a way to max out my Internet connection (300/200), I'm thinking I just need to install an OpenVPN client on the few server that need to move a ton of traffic but I'd really like my router/fw to do all the networking tasks.

      1 Reply Last reply Reply Quote 0
      • P
        pfBasic Banned
        last edited by

        @rdrcrmatt:

        in the APU2d…

        I'd love to find a way to max out my Internet connection (300/200)

        You aren't going to max out 500Mbps of OpenVPN throughput on 4 x 1GHz cores from 2014.

        You won't get close. Not even with gateway groups.

        1 Reply Last reply Reply Quote 0
        • R
          rdrcrmatt
          last edited by

          @pfBasic:

          @rdrcrmatt:

          in the APU2d…

          I'd love to find a way to max out my Internet connection (300/200)

          You aren't going to max out 500Mbps of OpenVPN throughput on 4 x 1GHz cores from 2014.

          You won't get close. Not even with gateway groups.

          Of course.  But I'd like to get more than 5-7%

          1 Reply Last reply Reply Quote 0
          • P
            pfBasic Banned
            last edited by

            OpenVPN is single threaded which is why you only see it on one core. Even running for clients in a gateway group not all traffic can utilize this so you will be stuck to one cores performance sometimes (often).

            ~40Mbps seems to be about the max for an APU2 with OpenVPN.

            1 Reply Last reply Reply Quote 0
            • S
              steve28
              last edited by

              @pfBasic:

              ~40Mbps seems to be about the max for an APU2 with OpenVPN.

              I seem to be able to do better than that on my APU2C4.  I'm using AES-128-CBC / SHA1 with a 100/10 connection (112/12 actual), I have all mention of hardware crypto turned off (i.e., in both System/Advanced/Misc and In the OpenVPN Client Settings).

              I was getting ~40 Mbit with this setup until i added the following three lines my OpenVPN Custom Options:

              
              fast-io
              sndbuf 524288
              rcvbuf 524288
              
              

              At that point I am able to get in the 90's of Mbps.  I systematically add/removed them and it's the sndbuf/rcvbuf settings that are making the difference.

              1 Reply Last reply Reply Quote 1
              • R
                rdrcrmatt
                last edited by

                @steve28:

                @pfBasic:

                ~40Mbps seems to be about the max for an APU2 with OpenVPN.

                I seem to be able to do better than that on my APU2C4.  I'm using AES-128-CBC / SHA1 with a 100/10 connection (112/12 actual), I have all mention of hardware crypto turned off (i.e., in both System/Advanced/Misc and In the OpenVPN Client Settings).

                I was getting ~40 Mbit with this setup until i added the following three lines my OpenVPN Custom Options:

                
                fast-io
                sndbuf 524288
                rcvbuf 524288
                
                

                At that point I am able to get in the 90's of Mbps.  I systematically add/removed them and it's the sndbuf/rcvbuf settings that are making the difference.

                Holy crap thank you!!!!

                I just got 67/21 on a single PIA OpenVPN connection using these settings on my APU2D.  Much better!

                1 Reply Last reply Reply Quote 0
                • M
                  mifronte
                  last edited by

                  @steve28:

                  @VAMike:

                  …
                  In current and prior versions of pfsense there was a gotcha: the buttons in gui basically made it so that you couldn't get AES-NI for IPSEC (a desirable thing) without also getting AES-NI+/dev/crypto for OpenVPN (an undesirable thing). In 2.4 they've made changes so the two things aren't coupled and you can get AES-NI for IPSEC without screwing up OpenVPN. (It's been this way in upstream FreeBSD--by default you get cryptodev [the kernel interface that IPSEC and other kernel modules use] without /dev/crypto [the userspace interface that hurts performance on modern platforms].)

                  VAMike - thank you for taking the time to explain this!

                  Just to be explicit, you recommend running with AES-NI selected in System->Advanced->Misc and "No Hardware Crypto Acceleration" in my OpenVPN settings? (pfSense 2.3.3_1)

                  Is this the recommended settings for pfSense prior to 2.4?  I don't think the question has been explicitly answered.

                  SuperMicro Atom C2758 A1SRI-2758F 16GB
                  2.7.2 (amd64)

                  1 Reply Last reply Reply Quote 0
                  • W
                    whosmatt
                    last edited by

                    @steve28:

                    @pfBasic:

                    ~40Mbps seems to be about the max for an APU2 with OpenVPN.

                    I seem to be able to do better than that on my APU2C4.  I'm using AES-128-CBC / SHA1 with a 100/10 connection (112/12 actual), I have all mention of hardware crypto turned off (i.e., in both System/Advanced/Misc and In the OpenVPN Client Settings).

                    I was getting ~40 Mbit with this setup until i added the following three lines my OpenVPN Custom Options:

                    
                    fast-io
                    sndbuf 524288
                    rcvbuf 524288
                    
                    

                    At that point I am able to get in the 90's of Mbps.  I systematically add/removed them and it's the sndbuf/rcvbuf settings that are making the difference.

                    I was about to chime in with my similar experience but it looks like you've already found what I was going to offer.

                    Those three settings made a huge difference for me as well.  My setup (bare metal to ESXi on the same hardware) and WAN connection have changed in the meantime, but i've settled on leaving my PIA clients set with "no hardware acceleration."

                    Here's my initial thread on the subject; the replies led me down a rabbit hole of multiple PIA tunnels and config tweaks.  https://forum.pfsense.org/index.php?topic=115992.msg643637#msg643637

                    1 Reply Last reply Reply Quote 0
                    • T
                      TheNarc
                      last edited by

                      Just a heads up to anyone adding the "fast-io" option to their OpenVPN client config.  I'm pretty sure from reading the documentation that this option only applies to UDP.  I'm not sure whether it would just be ignored for a client config using TCP (which is my guess) or mess it up in some way.  But I don't believe there's any point to adding it to a TCP client config.  Also, I realize that if you only have one client connection, you probably want to use UDP anyway.  But in my situation, for example, I maintain two client connections to the same provider, and need to have one UDP and one TCP.

                      And if I can tack on a bump to mifronte's question . . . is there a definitive answer to whether hardware crypto acceleration should be selected in System->Advanced->Misc?  It seems clear that we do not want it enabled for OpenVPN client configs, but is the system-wide setting useful in any way?

                      1 Reply Last reply Reply Quote 0
                      • jimpJ
                        jimp Rebel Alliance Developer Netgate
                        last edited by

                        FYI- I added GUI knobs for fast-io and sndbuf/rcvbuf to 2.4, will be in snaps soon. See https://forum.pfsense.org/index.php?topic=130350.0 and https://redmine.pfsense.org/issues/7507

                        Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                        Need help fast? Netgate Global Support!

                        Do not Chat/PM for help!

                        1 Reply Last reply Reply Quote 1
                        • bingo600B
                          bingo600
                          last edited by

                          @mifronte:

                          @steve28:

                          @VAMike:

                          …
                          In current and prior versions of pfsense there was a gotcha: the buttons in gui basically made it so that you couldn't get AES-NI for IPSEC (a desirable thing) without also getting AES-NI+/dev/crypto for OpenVPN (an undesirable thing). In 2.4 they've made changes so the two things aren't coupled and you can get AES-NI for IPSEC without screwing up OpenVPN. (It's been this way in upstream FreeBSD--by default you get cryptodev [the kernel interface that IPSEC and other kernel modules use] without /dev/crypto [the userspace interface that hurts performance on modern platforms].)

                          VAMike - thank you for taking the time to explain this!

                          Just to be explicit, you recommend running with AES-NI selected in System->Advanced->Misc and "No Hardware Crypto Acceleration" in my OpenVPN settings? (pfSense 2.3.3_1)

                          Is this the recommended settings for pfSense prior to 2.4?  I don't think the question has been explicitly answered.

                          I just got my Core i5 Box , and am using pfSense 2.4.0 Beta , and would like to use/enable AES-NI

                          After install it defaulted to : cryptodev

                          In System->Advanced->Misc->Cryptographic Hardware :  Should i select None or AES-NI ?

                          If i use "None"      the AES-NI shows up on the Main page as : AES-NI CPU Crypto: Yes (inactive)
                          If i use "AES-NI"    the AES-NI shows up on the Main page as : AES-NI CPU Crypto: Yes (active)

                          And i get an extra line :
                          Hardware crypto AES-CBC,AES-XTS,AES-GCM,AES-ICM

                          Could someone capable , give a definitive ansver ??

                          It seems like the performance is the same with all of the 3 selections , using : openssl speed -evp aes-128-cbc

                          
                          openssl speed -evp aes-128-cbc
                          Doing aes-128-cbc for 3s on 16 size blocks: 94684757 aes-128-cbc's in 2.99s
                          Doing aes-128-cbc for 3s on 64 size blocks: 25963476 aes-128-cbc's in 3.05s
                          Doing aes-128-cbc for 3s on 256 size blocks: 6553759 aes-128-cbc's in 3.01s
                          Doing aes-128-cbc for 3s on 1024 size blocks: 1642176 aes-128-cbc's in 3.00s
                          Doing aes-128-cbc for 3s on 8192 size blocks: 206252 aes-128-cbc's in 3.01s
                          OpenSSL 1.0.2k-freebsd  26 Jan 2017
                          built on: date not available
                          options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
                          compiler: clang
                          The 'numbers' are in 1000s of bytes per second processed.
                          type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
                          aes-128-cbc     506303.87k   543971.34k   557801.49k   560529.41k   561742.59k
                          
                          

                          TIA

                          /Bingo

                          If you find my answer useful - Please give the post a 👍 - "thumbs up"

                          pfSense+ 23.05.1 (ZFS)

                          QOTOM-Q355G4 Quad Lan.
                          CPU  : Core i5 5250U, Ram : 8GB Kingston DDR3LV 1600
                          LAN  : 4 x Intel 211, Disk  : 240G SAMSUNG MZ7L3240HCHQ SSD

                          1 Reply Last reply Reply Quote 0
                          • C
                            c0lp4nik
                            last edited by

                            Greetings-

                            Re-posting here as this is an akin topic:

                            Greetings!

                            Long-time listener, first-time caller.

                            I have been running pfSense in Azure (not the Netgate addition, sorry Netgate on a tight budget right now…) for sometime and and just upgraded to pfSense 2.4 and noticed that speeds from the appliance itself get 250-300 Mbps download tested with iperf (client) against he.net and scottlinux.com (public iperf servers), but my openvpn 2.4 (not to be confused with pfSense 2.4) clients are only getting a symmetric MAX 6 Mbps download and upload "capped".

                            I have no limiters in place:

                            ipfw show pipe - blank.
                            XML - none.

                            My /temp/rules.limits:

                            set limit table-entries 2000000
                            set optimization conservative
                            set timeout { udp.first 300, udp.single 150, udp.multiple 900 }
                            set limit states 1429000
                            set limit src-nodes 1429000

                            (which I am assuming is default, as I have no limits pushed to XML via the GUI).

                            Note: AES-NI Accel is noted:
                            CPU Type  Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz
                            4 CPUs: 1 package(s) x 4 core(s)
                            AES-NI CPU Crypto: Yes (active) -----------> CHECK!
                            Hardware crypto  AES-CBC,AES-XTS,AES-GCM,AES-ICM

                            Openvpn Crypto used: AES-256-CBC (CHECK!)

                            OpenVPN config (Screen in GUI): Hardware Crypto:  BSD Cryptodev......

                            Checked kernel mods loaded:

                            kldstat
                            Id Refs Address            Size    Name
                            1    8 0xffffffff80200000 2c3e9a0  kernel
                            2    1 0xffffffff83019000 46c6    cryptodev.ko
                            3    1 0xffffffff8301e000 7f92    aesni.ko

                            On-board speed test:

                            openssl speed -evp aes-256-cbc

                            Doing aes-256-cbc for 3s on 16 size blocks: 1240941 aes-256-cbc's in 0.11s
                            Doing aes-256-cbc for 3s on 64 size blocks: 1143048 aes-256-cbc's in 0.13s
                            Doing aes-256-cbc for 3s on 256 size blocks: 877391 aes-256-cbc's in 0.07s
                            Doing aes-256-cbc for 3s on 1024 size blocks: 500204 aes-256-cbc's in 0.07s
                            Doing aes-256-cbc for 3s on 8192 size blocks: 95778 aes-256-cbc's in 0.02s
                            OpenSSL 1.0.2k-freebsd  26 Jan 2017
                            built on: date not available
                            options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
                            compiler: clang
                            The 'numbers' are in 1000s of bytes per second processed.
                            type            16 bytes    64 bytes    256 bytes  1024 bytes  8192 bytes
                            aes-256-cbc    181531.94k  550814.66k  3194483.14k  7284748.74k 33476837.38k

                            Baffled. <shrugs shoulders="">....  :-\

                            This thread proved extremely insightful, however I am still not breaking the 6 Mbps barrier <sheds tear...="">  :'(

                            Any insight or corrections appreciated!

                            Thanks much!
                            C0l. P.</sheds></shrugs>

                            1 Reply Last reply Reply Quote 0
                            • V
                              VAMike
                              last edited by

                              @c0lp4nik:

                              On-board speed test:

                              openssl speed -evp aes-256-cbc

                              Doing aes-256-cbc for 3s on 16 size blocks: 1240941 aes-256-cbc's in 0.11s
                              Doing aes-256-cbc for 3s on 64 size blocks: 1143048 aes-256-cbc's in 0.13s
                              Doing aes-256-cbc for 3s on 256 size blocks: 877391 aes-256-cbc's in 0.07s
                              Doing aes-256-cbc for 3s on 1024 size blocks: 500204 aes-256-cbc's in 0.07s
                              Doing aes-256-cbc for 3s on 8192 size blocks: 95778 aes-256-cbc's in 0.02s
                              OpenSSL 1.0.2k-freebsd  26 Jan 2017
                              built on: date not available
                              options:bn(64,64) rc4(16x,int) des(idx,cisc,16,int) aes(partial) idea(int) blowfish(idx)
                              compiler: clang
                              The 'numbers' are in 1000s of bytes per second processed.
                              type            16 bytes    64 bytes    256 bytes  1024 bytes  8192 bytes
                              aes-256-cbc    181531.94k  550814.66k  3194483.14k  7284748.74k 33476837.38k

                              Bogus numbers, you have cryptodev enabled and aren't using -elapsed. You're not getting 7GByte/s with 1k blocks, you're getting ~170MByte/s.

                              Turn off cryptodev.

                              You may still not get great speeds, because you may be sharing a CPU with other VMs, but it shouldn't be that bad.

                              1 Reply Last reply Reply Quote 0
                              • C
                                c0lp4nik
                                last edited by

                                Thanks for the feedback VAMike!

                                I could be posting as MDCP (from the other side of the river…)..

                                So have now tested with "NO Hardware Crypto Accel" set in the VPN config (GUI), and with AES-NI enabled.

                                Same result  :(  on pfSense 2.3.4-P1, OpenVPN 2.3.17, and on pfSense 2.4, OpenVPN 2.4.4, respectively...

                                <shurg>....caveat it's on Azure, but it's a Quad-core with 14GB RAM....you'd think that should handle it...

                                No LB or other shaping devices in between....

                                Anything I can offer that might trigger an idear?

                                Thanks so much in advance!
                                CP</shurg>

                                1 Reply Last reply Reply Quote 0
                                • V
                                  VAMike
                                  last edited by

                                  @c0lp4nik:

                                  Thanks for the feedback VAMike!

                                  I could be posting as MDCP (from the other side of the river…)..

                                  So have now tested with "NO Hardware Crypto Accel" set in the VPN config (GUI), and with AES-NI enabled.

                                  Same result  :(  on pfSense 2.3.4-P1, OpenVPN 2.3.17, and on pfSense 2.4, OpenVPN 2.4.4, respectively...

                                  <shurg>....caveat it's on Azure, but it's a Quad-core with 14GB RAM....you'd think that should handle it...

                                  No LB or other shaping devices in between....

                                  Anything I can offer that might trigger an idear?

                                  Thanks so much in advance!
                                  CP</shurg>

                                  If you run the openssl speed test without -elapsed again and cryptodev not loaded, you should be getting about 500MByte/s if my back of the envelope math is right. If you're getting significantly less than that you're losing cycles on the VM. If the crypto rate looks about right, then check the logs for stuff like MTU warnings or other problems. That data rate is low enough that either something is broken or something on the network is intentionally or unintentionally throttling you. That's a tough thing for an armchair diagnosis, unfortunately. Also, with openvpn 2.4 you can configure AES-128-GCM, which should perform better than AES-256-CBC, but you're still so far below the expected limit of AES-CBC on that hardware that I wouldn't expect a miracle.

                                  1 Reply Last reply Reply Quote 0
                                  • S
                                    sensemann
                                    last edited by sensemann

                                    Hello!
                                    what are the best settings for OpenVPN HIDEME_VPN with

                                    APU4B4 and pfsense 2.4.3-RELEASE-p1 ?
                                    (AMD Embedded G series GX-412TC, 1 GHz quad Jaguar core with 64 bit and AES-NI support)

                                    I ask because I get only around 8-10MBit down and 18MBit up.
                                    Without OpenVPN it is 220Mbit down and 20Mbit up.

                                    On a System with an Atom C2358, SoC,
                                    (Rangeley), 7W 2-Core, 1.7-2.0GHz (Board: A1SRM-LN7F-2358 ) I get similar low rates.

                                    Thanks!

                                    S 1 Reply Last reply Reply Quote 0
                                    • S
                                      sensemann @sensemann
                                      last edited by

                                      any news?

                                      B 1 Reply Last reply Reply Quote 0
                                      • B
                                        bcruze @sensemann
                                        last edited by

                                        @sensemann said in AES-NI / Cryptodev / OpenVPN – help a n00b understand:

                                        any news?

                                        Have you increase you fast IO buffers to at least 256?

                                        S 1 Reply Last reply Reply Quote 0
                                        • S
                                          sensemann @bcruze
                                          last edited by

                                          0_1537480204839_OpenVPNBuffer.png

                                          the red marked values I set now..

                                          1 Reply Last reply Reply Quote 0
                                          • B
                                            bcruze
                                            last edited by

                                            You have one set for 256 one for 512 not sure if that will cause an issue

                                            It’s in custom options and the drop down to 256

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.