Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Can't break 15mbps OpenVPN throughput

    OpenVPN
    5
    34
    9.0k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C
      coachmark2
      last edited by

      I have OpenVPN running on a 2.2.1 PFSense box and I can't seem to get more than ~15mbit/s throughput out of it. I will try to concisely summarize my hardware and configuration below:

      ISP for Server: Comcast
      Arris CPE in bridge mode
      105mbit/s downstream
      20mbit/s upstream

      ISP for Client1 and Client2: Other business grade ISP
      100mbit/s downstream
      100mbit/s upstream

      Server:
      Pfsense 2.2.1
      Dell R210
      2 x Broadcom Gigabit NICs
      Intel Xeon x3450 (Nehalem quad at 2.6GHz)
      4GB RAM

      Client 1:
      Windows 8.1 Pro
      i7-2600k @ 3.4GHz
      16GB RAM

      Client 2:
      Dell Latitude D630
      Fedora 20 "Heisenbug"
      2.8GHz Core2Duo
      4GB RAM

      OpenVPN config file (IP addresses and hostnames modified)

      dev tun
      persist-tun
      persist-key
      cipher AES-256-CBC
      auth SHA256
      tls-client
      client
      resolv-retry infinite
      remote 11.22.33.44 1194 udp
      lport 0
      auth-user-pass
      ca MyPFSenseBox-udp-1194-ca.crt
      tls-auth MyPFSenseBox-udp-1194-tls.key 1
      comp-lzo adaptive

      This is all working very well for things which do not require substantial bandwidth. Copying to/from the site is fine and latency is actually quite good. My only problem is bandwidth. Whether via FTP, SMB, or iperf, I cannot seem to get above 15mbit/s. This is either to or from the network; that is, whether I'm pushing or pulling from a server behind the VPN, it caps out from 12-15mbit/s.

      CPU usage on the client is usually 5% or less and on the server is 4-6% during an iperf test or FTP transfer.

      What might I be doing wrong?

      Thanks,
      coach

      1 Reply Last reply Reply Quote 0
      • K
        kejianshi
        last edited by

        Is this the first version of pfsense you have tried?

        Also, what it the max up/down bandwidth on both ends of the connection?

        1 Reply Last reply Reply Quote 0
        • C
          coachmark2
          last edited by

          It's been this way since 2.1.4, but I haven't made a thread because it hasn't been a tremendous problem.

          From the OP:

          ISP for Server: Comcast
          Arris CPE in bridge mode
          105mbit/s downstream
          20mbit/s upstream

          ISP for Client1 and Client2: Other business grade ISP
          100mbit/s downstream
          100mbit/s upstream

          Client1 and Client2 are on a network with an ISP that provides 100megabits per second synchronous. They are the only devices on said network.
          The OpenVPN server running on pfSense are on Comcast behind their CPE which is in bridge mode. That connection is rated at 105 down and 20 up, but real world sees 125 down and 22-25 up.

          Whether pushing files TO a server behind the pfSense box or when retrieving files FROM the server behind the pfSense box, speeds never exceed 12-15mbit/s

          I have also used iperf from both sides to test. iperf varies from 11-14mbit/s.

          1 Reply Last reply Reply Quote 0
          • DerelictD
            Derelict LAYER 8 Netgate
            last edited by

            12-15mbit/s

            What do you expect on a 20M upstream?

            Chattanooga, Tennessee, USA
            A comprehensive network diagram is worth 10,000 words and 15 conference calls.
            DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
            Do Not Chat For Help! NO_WAN_EGRESS(TM)

            1 Reply Last reply Reply Quote 0
            • C
              coachmark2
              last edited by

              @derelict

              I understand that pulling files FROM the site would be limited to 20 megabits minus some overhead.

              However, when pushing files the other direction, the speeds are the same. Since my client has 100mbit/s upload and the network behind the pfSense box has 100mbit/s download, I would expect transfers to be higher than 15mbit/s…

              1 Reply Last reply Reply Quote 0
              • DerelictD
                Derelict LAYER 8 Netgate
                last edited by

                Sorry.  I read fast and only saw pulling files FROM server.

                So what are the various CPU loads while you're running these uploads?

                Chattanooga, Tennessee, USA
                A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                Do Not Chat For Help! NO_WAN_EGRESS(TM)

                1 Reply Last reply Reply Quote 0
                • C
                  coachmark2
                  last edited by

                  :)

                  No worries.

                  2012R2 VM behind the pfSense box: 3% CPU
                  Dell R210 running OpenVPN and pfSense: Average 5-6% with occasional spikes to 9%
                  Client: i7-2600k showing 8-10% usage.

                  I have also checked carefully here to make sure that nothing is pegging a single thread, etc. Throughput remains in the 15mbit/s range as I'm transferring a .mkv.

                  I can add pics if you'd like. :)

                  1 Reply Last reply Reply Quote 0
                  • C
                    coachmark2
                    last edited by

                    Bump. Any other thoughts? Other tests I could run to see what's happening?

                    1 Reply Last reply Reply Quote 0
                    • M
                      marvosa
                      last edited by

                      Over the years, I've read many posts on other forums that state using software-based NIC's contribute to that 20 Mb/sec cap.  The posts always mention upgrading to high quality, hardware-based Intel NIC's.

                      Here's an interesting article on network tuning and performance:
                      https://calomel.org/network_performance.html

                      It touches on getting the most out of your firewall by looking at hardware, bus speed, os tweaks, MTU, etc

                      1 Reply Last reply Reply Quote 0
                      • C
                        coachmark2
                        last edited by

                        Thanks for the reply but… I'm not using a software NIC.... It's a well-supported Broadcom unit.

                        1 Reply Last reply Reply Quote 0
                        • M
                          marvosa
                          last edited by

                          Supported doesn't necessarily = max performance.  What NIC's are you using?

                          1 Reply Last reply Reply Quote 0
                          • K
                            kejianshi
                            last edited by

                            Have you considered trying a well supported Intel unit?

                            1 Reply Last reply Reply Quote 0
                            • C
                              coachmark2
                              last edited by

                              I don't have an Intel unit to test on. :(

                              But when not using VPN, I can pull 120+mbit/s through that interface all day long. It's just over VPN that it chokes.

                              1 Reply Last reply Reply Quote 0
                              • DerelictD
                                Derelict LAYER 8 Netgate
                                last edited by

                                Wasn't the pfSense store recently selling Dell R210s?  I would think that pretty much clears his hardware.

                                Chattanooga, Tennessee, USA
                                A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                1 Reply Last reply Reply Quote 0
                                • C
                                  coachmark2
                                  last edited by

                                  @__Derelict__

                                  That was my thought. :( They were actually R200s, but the R210 uses a very similar NIC setup.

                                  I have a performance update for inquiring minds. I re-ran my iperf testing with a few different parameters. When I use 8 simultaneous TCP streams, I see at or around 50mbit/s :D That's more like it and very tolerable. UDP looks like about the same.

                                  So… what could possibly be limiting a single TCP stream to 15 mbit/s?

                                  1 Reply Last reply Reply Quote 0
                                  • K
                                    kejianshi
                                    last edited by

                                    I also get throttled reliably at certain times of day.

                                    Example.  I can always download at my max rate from the web (like hulu or netflix) but a vpn is throttled to death after say 5pm here and not as bad at say 9am.

                                    It could be an ISP deal and traffic shaping.

                                    1 Reply Last reply Reply Quote 0
                                    • C
                                      coachmark2
                                      last edited by

                                      Sure, I'm certain that happens to me too. But this IS a VPN. To the ISP, it appears as a bunch of UDP gibberish, so they wouldn't be able to even see the fact that I'm running 1 vs. 8 TCP streams inside the tunnel.

                                      Consistently reproducible is the fact that I get 15mbit/s for a TCP data stream and 50mbit/s aggregate when number of streams >4. No matter the time of the day.

                                      Any idea what could be causing that?

                                      1 Reply Last reply Reply Quote 0
                                      • K
                                        kejianshi
                                        last edited by

                                        Yeah - sounds like they are throttling you per connection.  That or "long fat pipe" issues.  How far away are these vpns?

                                        1 Reply Last reply Reply Quote 0
                                        • C
                                          coachmark2
                                          last edited by

                                          Approx. 400 miles by road. Since the East Coast routing tables are all kinds of screwed over right now (thanks Comcast!) it might be going through anywhere from 10 to 17 hops depending on what BGP feels like doing this particular time of day.

                                          How would the ISP realize whether or not I'm pushing multiple TCP connections over this VPN though if it's encrypted UDP?

                                          1 Reply Last reply Reply Quote 0
                                          • K
                                            kejianshi
                                            last edited by

                                            Are you telling me that all your traffic is going out over this 1 udp vpn but that if you are doing 1 TCP downlaod with this vpn you are limited to 15 but when doing many tcp downloads over this same vpn you can hit 50?

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.