OpenVPN Slow - local network test



  • Spent a long time reading so hopefully I can get this detailed and the group can help me.
    Just testing a OpenVPN setup on a local 1Gb Lan.
    Server is running the latest version.
    This is on a Dell R610 with 48Gb Ram
    Doing an IPERF to the Pfsense (natted to a Iperf server on the other side..so no vpn) shows 939 - 978 Mbits/s. This is expected as they are all in the same building.
    Same test, same machines, however, through the vpn tunnel (specifying the Iperf server on the other side of the wall) shows 118-135 Mbits/s
    I know from reading that there is overhead and OpenVPN is slower then IPSEC. However, it cant be that much slower. I have to be doing something wrong.

    Any help would be appreciated. Love the product...so I must be doing something silly.
    I have tried mucking around with buffers MTU....ect...but just no luck.

    Final installation will be using a full 1GB Fiber line so that is why I am trying to get as much speed as possible.
    Thanks all.

    My specs:
    24 CPU X5680 Xeons
    48 GB Ram
    AES-N1
    UDP tunel

    Server Config:
    dev ovpns3
    verb 1
    dev-type tun
    dev-node /dev/tun3
    writepid /var/run/openvpn_server3.pid
    #user nobody
    #group nobody
    script-security 3
    daemon
    keepalive 10 60
    ping-timer-rem
    persist-tun
    persist-key
    proto udp4
    cipher AES-128-CBC
    auth SHA256
    up /usr/local/sbin/ovpn-linkup
    down /usr/local/sbin/ovpn-linkdown
    client-connect /usr/local/sbin/openvpn.attributes.sh
    client-disconnect /usr/local/sbin/openvpn.attributes.sh
    local 172.18.19.24
    engine cryptodev
    tls-server
    server 192.168.27.0 255.255.255.0
    client-config-dir /var/etc/openvpn-csc/server3
    username-as-common-name
    plugin /usr/local/lib/openvpn/plugins/openvpn-plugin-auth-script.so /usr/local/sbin/ovpn_auth_verify_async user UkFESVVT false server3 1194
    tls-verify "/usr/local/sbin/ovpn_auth_verify tls 'test.com' 1"
    lport 1194
    management /var/etc/openvpn/server3.sock unix
    push "route 172.25.0.0 255.255.0.0"
    push "dhcp-option DNS 172.25.21.10"
    push "dhcp-option DNS 8.8.8.8"
    client-to-client
    ca /var/etc/openvpn/server3.ca
    cert /var/etc/openvpn/server3.cert
    key /var/etc/openvpn/server3.key
    dh /etc/dh-parameters.2048
    tls-auth /var/etc/openvpn/server3.tls-auth 0
    ncp-ciphers AES-128-GCM
    persist-remote-ip
    float
    topology subnet
    fast-io

    Client Config:
    dev tun
    persist-tun
    persist-key
    cipher AES-128-CBC
    ncp-ciphers AES-128-GCM
    auth SHA256
    tls-client
    client
    resolv-retry infinite
    remote 172.18.19.24 1194 udp4
    lport 0
    verify-x509-name "test.com" name
    auth-user-pass
    ca pfSense-UDP4-1194-VPNUsers-ca.crt
    cryptoapicert "Myprivatestuffgoeshere.LOL."
    tls-auth pfSense-UDP4-1194-VPNUsers-tls.key 1
    remote-cert-tls server


  • LAYER 8 Rebel Alliance

    Try with AES-128-GCM
    AES-NI is selected in System > Advanced > Miscellaneous > Cryptographic Hardware ? And does it show active (widget)?
    Leave the Hardware Crypto setting in the OpenVPN instance set to No Hardware Crypto.

    -Rico



  • @Rico
    Yes, Hardwareware Crypto shows AES-CBC,AES-XTS,AES-GCM,AES-ICM in the dashboard widget.
    OpenVPN was showing BSD Cryptodev engine.....Turning that to none and testing again.

    BRB.



  • Rico,
    So that gave me an ever so slight increase. I think I am up to 165 Mb/s.

    So still has to be something weird. You can see the first set is going through the VPN while the other is just hitting the wan and nat to the iperf server

    No VPN.png


  • LAYER 8 Rebel Alliance

    Did you try with GCM or still CBC?
    Also try to set

    sndbuf 524288
    rcvbuf 524288
    

    -Rico



  • @Rico
    Buffers go in the server config, client config or both?


  • LAYER 8 Rebel Alliance

    Set the Send/Receive Buffer on both.

    -Rico



  • Thank you for trying to help btw.
    Changed to use GCM and set the buffers on both sides...actually made it a little worse.

    I am totally baffled by this. I know it can work and is not the network. Just something in OpenVpn is throttling it all down.



  • @Rico
    Also, wondering if I need to set the tunnel size.
    Pinging at 1500 shows fragmentation. Setting it to 1472 shows no frag. (hitting a box on the remote network of course).

    What do you think?


  • LAYER 8 Global Moderator



  • @johnpoz
    Thanks, I did read this but going to go over again to see if I missed something.
    I know it talks about jumbo frames....will try that and let everyone know what i find.


  • LAYER 8 Rebel Alliance

    Pls run those two via the command line or SSH and post the output here

    openvpn --genkey --secret /tmp/secret
    time openvpn --test-crypto --secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-128-gcm
    

    -Rico



  • @Rico

    [2.4.5-RELEASE][admin@pfSense.seradex.local]/root: openvpn --genkey --secret /tm p/secret
    [2.4.5-RELEASE][admin@pfSense.seradex.local]/root: time openvpn --test-crypto -- secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-128-gcm
    Fri Jun 19 14:47:16 2020 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode
    10.007u 0.000s 0:10.00 100.0% 843+177k 0+0io 0pf+0w
    [2.4.5-RELEASE][admin@pfSense.seradex.local]/root:



  • @spyder0552
    Also just did a test one using UDP and setting the tunnel size to 9000....but still same results


  • LAYER 8 Rebel Alliance

    Hmm with this output I'd expect more than ~165 Mbps but you will never get to 1 Gbps with this CPU and OpenVPN, not even close.
    Run again your iperf via the tunnel and check top -aSH - does the OpenVPN process run close to 100%?
    Also for testing try with Encryption Algorithm set to None and try iperf via the tunnel again.

    -Rico



  • @Rico
    From what I see, no where near any cpu usage
    Top.png



  • @spyder0552
    Back on this again now. Anyone have suggestions?
    This is all on the same network using Cat6 cables. Just testing from a laptop to the wall to a device on the other side.
    No VPN - Speed is fantastic.
    OpenVPN tunnel - speed is pathetic.

    Any help would be appreciated before I have to dump this and look to another solution. I just can't believe that OpenVPN can't handle at least 400-600 Mbs.

    Thanks all.


  • LAYER 8 Rebel Alliance

    Did you try with Encryption Algorithm set to None for testing?
    If you really aim for highspeed VPN traffic I think you need to try with IPsec. 😐

    -Rico


  • LAYER 8 Global Moderator

    I'm about ready to head out - but I I want to try and see what my 4860 can do via this local sort of testing.. If find some time will do a local only test..



  • @johnpoz
    Hi all.
    So, still having the same issue. I have now setup a home lab and get the exact same results.
    Test Lab:
    Intel i5 with 32Gb Ram.
    Using Windows HyperV as the host.
    Pfsense and Windows10 as guest machines.

    Test 1 = laptop -> Firewall -> Windows 10
    Get ~900 Mb/s

    Test 2 = Laptop with OpenVP -> Firewall -> Windows 10
    Get ~ 170 Mb/s

    This is the same results I am seeing at my office using Dell R610 with 24 CPU Xenon's.

    Things I have tried:

    1. Enable/disable the network options Hardware TCP Seg and Large Receive Offload (check box set to disable them).
    2. Enable fast I/O (with it turned off, I go down to ~155-160)...so currently enabled.
    3. Snd/rcv buffers. Tried default and 2Mb. No significant change.
    4. Encryption at AES-128-CBC and GCM. No significant change.

    So for anyone that wants to setup my test enviroment:

    1. Just setup a HyperV box.
    2. Add 2 virtual switches (one external and one internal).
    3. Create a VM for Pfsense and give it both virtual switches (WAN = External, LAN = Internal switch).
    4. Create a VM for a test box
    5. Put Iperf on the test box
    6. Create your PFsense with default OpenVPN (using TLS). Just run through the wizard.

    I am sure if someone did this they would get to the same spot I am where the speed is just no where at what it should be capable of.
    You can rule the hardware out to some extent as testing without the VPN shows near full speed (meaning the nics are talking to each other properly).

    Anyone willing to try my test?



  • @Rico
    Hey, just replied below. Yes, I did try that on Friday and it did not make a difference.

    Is anyone out there trying this with Windows 10 clients? Perhaps this is a windows issue?



  • @spyder0552 said in OpenVPN Slow - local network test:

    [2.4.5-RELEASE][admin@pfSense.seradex.local]/root: openvpn --genkey --secret /tm p/secret
    [2.4.5-RELEASE][admin@pfSense.seradex.local]/root: time openvpn --test-crypto -- secret /tmp/secret --verb 0 --tun-mtu 20000 --cipher aes-128-gcm
    Fri Jun 19 14:47:16 2020 disabling NCP mode (--ncp-disable) because not in P2MP client or server mode
    10.007u 0.000s 0:10.00 100.0% 843+177k 0+0io 0pf+0w
    [2.4.5-RELEASE][admin@pfSense.seradex.local]/root:

    By the looks of that ^^^ it theoretically could do ~320Mbps

    How about testing both ways? (client>server - server>client)
    Which I5 is that?

    Your top -aSH screenshot is timed wrong, ..... I guess.



  • @Pippin
    Home test lab is i5-4690K (yeah..little dated...best I can get atm).

    I can't really get it to test from the inside out. The wall seems to be blocking it out. I guess the IP I would connect to would be the end point of the tunel...but not working it seems.



  • You can reverse test mode by using iperf3 -R
    "-R, Reverse test mode – Server sends, client receives"



  • @Pippin
    WOW...actually slower in reverse. Averaged only 100 Mb/s

    But if I drop the VPN and just go through the firewall, it goes up to 900



  • @spyder0552
    Just to show the config on the client/server side
    Client:
    dev tun
    persist-tun
    persist-key
    cipher AES-128-GCM
    ncp-ciphers AES-128-GCM
    auth SHA256
    tls-client
    client
    resolv-retry infinite
    remote 192.168.2.98 1194 udp4
    verify-x509-name "OpenVPN-Server-Cert" name
    auth-user-pass
    pkcs12 pfSense-UDP4-1194-vpn.p12
    tls-crypt pfSense-UDP4-1194-vpn-tls.key
    remote-cert-tls server

    Server:
    dev ovpns1
    verb 1
    dev-type tun
    dev-node /dev/tun1
    writepid /var/run/openvpn_server1.pid
    #user nobody
    #group nobody
    script-security 3
    daemon
    keepalive 10 60
    ping-timer-rem
    persist-tun
    persist-key
    proto udp4
    cipher AES-128-GCM
    auth SHA256
    up /usr/local/sbin/ovpn-linkup
    down /usr/local/sbin/ovpn-linkdown
    client-connect /usr/local/sbin/openvpn.attributes.sh
    client-disconnect /usr/local/sbin/openvpn.attributes.sh
    local 192.168.2.98
    tls-server
    server 192.168.99.0 255.255.255.0
    client-config-dir /var/etc/openvpn-csc/server1
    username-as-common-name
    plugin /usr/local/lib/openvpn/plugins/openvpn-plugin-auth-script.so /usr/local/sbin/ovpn_auth_verify_async user TG9jYWwgRGF0YWJhc2U= false server1 1194
    tls-verify "/usr/local/sbin/ovpn_auth_verify tls 'OpenVPN-Server-Cert' 1"
    lport 1194
    management /var/etc/openvpn/server1.sock unix
    push "route 192.168.1.0 255.255.255.0"
    push "dhcp-option DOMAIN test.local"
    push "dhcp-option DNS 8.8.8.8"
    client-to-client
    ca /var/etc/openvpn/server1.ca
    cert /var/etc/openvpn/server1.cert
    key /var/etc/openvpn/server1.key
    dh /etc/dh-parameters.2048
    tls-crypt /var/etc/openvpn/server1.tls-crypt
    ncp-ciphers AES-128-GCM
    persist-remote-ip
    float
    topology subnet
    fast-io


  • LAYER 8 Global Moderator

    @spyder0552 said in OpenVPN Slow - local network test:

    Just setup a HyperV box.

    That doesn't seem like a legit test.. My test will be this

    windows 10 PC --- sg4860--- NAS

    Will set this up later.. But you don't show any of the mtu settings or txqueuelen..


  • LAYER 8 Global Moderator

    Ok I just set this up...

    With default everything on open seeing..

    $ iperf3 -c 192.168.9.10
    warning: Ignoring nonsense TCP MSS 334848
    Connecting to host 192.168.9.10, port 5201
    [  5] local 10.0.100.2 port 52251 connected to 192.168.9.10 port 5201
    [ ID] Interval           Transfer     Bitrate
    [  5]   0.00-1.00   sec  25.5 MBytes   214 Mbits/sec
    [  5]   1.00-2.00   sec  29.5 MBytes   248 Mbits/sec
    [  5]   2.00-3.00   sec  32.4 MBytes   272 Mbits/sec
    [  5]   3.00-4.00   sec  31.1 MBytes   261 Mbits/sec
    [  5]   4.00-5.00   sec  29.5 MBytes   247 Mbits/sec
    [  5]   5.00-6.00   sec  30.8 MBytes   258 Mbits/sec
    [  5]   6.00-7.00   sec  30.6 MBytes   257 Mbits/sec
    [  5]   7.00-8.00   sec  31.6 MBytes   265 Mbits/sec
    [  5]   8.00-9.00   sec  31.0 MBytes   260 Mbits/sec
    [  5]   9.00-10.00  sec  32.0 MBytes   268 Mbits/sec
    - - - - - - - - - - - - - - - - - - - - - - - - -
    [ ID] Interval           Transfer     Bitrate
    [  5]   0.00-10.00  sec   304 MBytes   255 Mbits/sec                  sender
    [  5]   0.00-10.01  sec   304 MBytes   255 Mbits/sec                  receiver
    

    Let me see if can tweak that a bit..

    win10 pc (192.168.200.10) --- switch - 192.168.200.1 (test igb4) pfsense (sg4860) ( igb0 lan) 192.168.9.253 -- switch -- 192.168.9.10 (NAS)

    Tweaking didn't make much difference to be honest, but using aes-128-gcm vs cbc was huge difference.. When changed to cbc vs gcm when to 130 vs mid 200's



  • @johnpoz
    Thanks for doing the testing.
    I am surprised to see that the best we can get on a full 1Gb/s link is ~250 Mb/s
    So I guess my ~150Mb/s on the first crack is not horrible?

    I know there is overhead to deal with...just surprised it is so much. This is looking like a 70% loss of speed using OpenVpn.
    Is this what others are seeing as well?


  • LAYER 8 Global Moderator

    openvpn is single threaded.. its easy to use - its never been "speedy" ;)

    Keep in mind my sg4860, not a rocketship vpn endpoint concentrater either..

    It has enough umph to get the job done with lower power requirements.. But prob not what I would use for my vpn endpoint if what I wanted to as much throughput as possible.. Nor would openvpn be my first choice in that area - ipsec is better geared for throughput..

    Openvpn advantage is ease of use, and deployment, etc.

    But overall your sort of test with everything on the same vm host is not really a valid sort of testing.. It works for poc, etc. But its not going to be a good indicator of what sort of bandwidth you could expect when using in the real world..



  • But overall your sort of test with everything on the same vm host is not really a valid sort of testing

    Yes I think so too.
    Looking at the hardware it should be capable of more.
    .

    Nor would openvpn be my first choice in that area

    Wait a bit, who knows ;)


  • LAYER 8 Global Moderator

    @Pippin said in OpenVPN Slow - local network test:

    Wait a bit, who knows ;)

    If the goal was pure throughput, openvpn would not be on the top of the list of choices.. It has many other attributes it shines at.. But if what I am looking for is closest to line speed using least amount of horsepower.. Then no its not on the top of really any list ;)



  • @johnpoz
    That is what I am concluding.
    OpenVPN = easy to deploy and flexible....just not that speedy.

    I am in the process of setting up another test to use IPSEC and compare. I will post my results for those that are interested.

    For this initial issue, I am calling it closed as it seems we have beaten the heck out of it. Thank you all for the fantastic support on this. So very much appreciated.



  • If the goal was pure throughput, openvpn would not be on the top of the list of choices

    Sure, but maybe it will get to that top some time in the future.


  • LAYER 8 Global Moderator

    ^ we can hope yeah ;)


Log in to reply