Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    iperf3 testing 500/40 connection: 33mpbs

    Scheduled Pinned Locked Moved General pfSense Questions
    39 Posts 5 Posters 5.0k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • stephenw10S
      stephenw10 Netgate Administrator @Cabledude
      last edited by

      @cabledude said in iperf3 testing 500/40 connection: 33mpbs:

      Something of this sort?

      I would set the protocol to ICMP and the destination to WAN address. But since that's a single source IP and one you trust the risk is minimal.

      Steve

      johnpozJ 1 Reply Last reply Reply Quote 0
      • johnpozJ
        johnpoz LAYER 8 Global Moderator @stephenw10
        last edited by

        Just taking his 18ms I see there as lowest one..

        With only a 64k window your looking at 29.13 Mbit/sec due to the BDP..

        Also lets not forget the overhead, are you trying to send 1500, those will get fragmented, again also lowering the performance. What is the box on his end doing the vpn connection? His router, what router is that - if some soho type of router, while it might be able to do a 200mbps connection, what can it push through a openvpn connection?

        You prob want to adjust your txqueuelen and your tcp-queue-limit as well if your trying to see what is the max you could achieve for performance.

        An intelligent man is sometimes forced to be drunk to spend time with his fools
        If you get confused: Listen to the Music Play
        Please don't Chat/PM me for help, unless mod related
        SG-4860 24.11 | Lab VMs 2.8, 24.11

        C 1 Reply Last reply Reply Quote 0
        • C
          Cabledude @johnpoz
          last edited by

          @johnpoz said in iperf3 testing 500/40 connection: 33mpbs:

          Just taking his 18ms I see there as lowest one..

          With only a 64k window your looking at 29.13 Mbit/sec due to the BDP..

          That sounds about right! Could you share your calculation, I am very interested in that.

          His router is a UniFi USG 3P. Not the fastest box but he can do about 80 mbps with his VPNs.

          Pete
          Home: SG-2100 + UniFi + Synology. SG-1100 retired
          Parents: SG-1100 + UniFi + Synology
          Testing: SG-1100 w/ 120GB SSD via ext USB (eMMC dead). Works great

          johnpozJ 1 Reply Last reply Reply Quote 0
          • johnpozJ
            johnpoz LAYER 8 Global Moderator @Cabledude
            last edited by

            @cabledude

            https://www.cisco.com/c/en/us/support/docs/ip/transmission-control-protocol-tcp/200943-Why-Your-Application-only-Uses-10Mbps-Ev.html#anc3

            There are few calculators around.. Here is one I use
            https://www.switch.ch/network/tools/tcp_throughput/

            An intelligent man is sometimes forced to be drunk to spend time with his fools
            If you get confused: Listen to the Music Play
            Please don't Chat/PM me for help, unless mod related
            SG-4860 24.11 | Lab VMs 2.8, 24.11

            C 1 Reply Last reply Reply Quote 0
            • C
              Cabledude @johnpoz
              last edited by Cabledude

              @johnpoz Hi John, thank you, good read.
              I was just thinking what protocol am I using for the pfSense OpenVPN setup and it is UDP. Is this relevant for the BDP? If i read the article correctly it assumes TCP.
              9253518f-5e9c-4789-90df-8617f991cbe8-image.jpeg

              Thank you for helping out! I am learning every day :)

              Pete

              Pete
              Home: SG-2100 + UniFi + Synology. SG-1100 retired
              Parents: SG-1100 + UniFi + Synology
              Testing: SG-1100 w/ 120GB SSD via ext USB (eMMC dead). Works great

              johnpozJ 1 Reply Last reply Reply Quote 0
              • johnpozJ
                johnpoz LAYER 8 Global Moderator @Cabledude
                last edited by

                @cabledude what your tunnel is doesn't really matter. Your testing with TCP from your iperf outputs.

                An intelligent man is sometimes forced to be drunk to spend time with his fools
                If you get confused: Listen to the Music Play
                Please don't Chat/PM me for help, unless mod related
                SG-4860 24.11 | Lab VMs 2.8, 24.11

                C 1 Reply Last reply Reply Quote 0
                • C
                  Cabledude @johnpoz
                  last edited by

                  @johnpoz Ah okay I see, thank you.

                  So apparently building a VPN tunnel for high speed backups isn't as easy as simply connecting two gigabit nodes.

                  In order to troubleshoot further, would it be of any use to repeat the iperf3 test using UDP? Or using any other flags?

                  My friend is not available for further testing today but I might get a chance next week. He is willing to ping to my network tonight (UTC +1) though.

                  Pete

                  Pete
                  Home: SG-2100 + UniFi + Synology. SG-1100 retired
                  Parents: SG-1100 + UniFi + Synology
                  Testing: SG-1100 w/ 120GB SSD via ext USB (eMMC dead). Works great

                  johnpozJ 1 Reply Last reply Reply Quote 0
                  • johnpozJ
                    johnpoz LAYER 8 Global Moderator @Cabledude
                    last edited by

                    @cabledude said in iperf3 testing 500/40 connection: 33mpbs:

                    would it be of any use to repeat the iperf3 test using UDP?

                    Depends on what your wanting accomplish - if just to see if a difference sure, but does your backup actually leverage udp ;)

                    An intelligent man is sometimes forced to be drunk to spend time with his fools
                    If you get confused: Listen to the Music Play
                    Please don't Chat/PM me for help, unless mod related
                    SG-4860 24.11 | Lab VMs 2.8, 24.11

                    1 Reply Last reply Reply Quote 0
                    • stephenw10S
                      stephenw10 Netgate Administrator
                      last edited by

                      Testing using UDP eliminates the TCP windowing/ack packets issue so can be useful as long as you interpret the result correctly.

                      You should definitely use UDP for the VPN tunnel itself. You should only ever use TCP there if you are forced to, like is UDP is blocked. A TCP connection inside a TCP tunnel can produce horrible results.

                      Steve

                      1 Reply Last reply Reply Quote 0
                      • C
                        Cabledude @stephenw10
                        last edited by

                        @stephenw10 said in iperf3 testing 500/40 connection: 33mpbs:

                        Not really, you'd have to ping between the public IPs directly outside the VPN.

                        18 packets transmitted, 18 packets received, 0.0% packet loss
                        round-trip min/avg/max/stddev = 15.493/18.116/24.227/2.289 ms
                        

                        This is a ping outside the tunnel from my friend's UniFi USG 200/200 fibre to my SG-1100 500/40 cable.

                        Pete
                        Home: SG-2100 + UniFi + Synology. SG-1100 retired
                        Parents: SG-1100 + UniFi + Synology
                        Testing: SG-1100 w/ 120GB SSD via ext USB (eMMC dead). Works great

                        johnpozJ 1 Reply Last reply Reply Quote 0
                        • johnpozJ
                          johnpoz LAYER 8 Global Moderator @Cabledude
                          last edited by

                          @cabledude said in iperf3 testing 500/40 connection: 33mpbs:

                          15.493/18.116/24.227

                          Yeah with those ping results, without updating your window size your seeing what would be prob normal for a 64k window size..

                          I recently got pulled into an issue in real life work ;) Where a customer was complaining they were seeing terrible speeds in an app from their location into the DC, well its like 30ms between the locations.

                          Their local firewall (palo) was preventing the window size from scaling - because they were using syn cookies..

                          You would be surprised how many time issues with bandwidth comes down to window size ;) We have a gig pipe and only seeing fraction of that - well yeah your X ms apart and using window size Y.. So yeah your not going to be able to come anywhere close to filling the pipe ;)

                          An intelligent man is sometimes forced to be drunk to spend time with his fools
                          If you get confused: Listen to the Music Play
                          Please don't Chat/PM me for help, unless mod related
                          SG-4860 24.11 | Lab VMs 2.8, 24.11

                          C 1 Reply Last reply Reply Quote 1
                          • C
                            Cabledude @johnpoz
                            last edited by

                            @johnpoz said in iperf3 testing 500/40 connection: 33mpbs:

                            Yeah with those ping results, without updating your window size your seeing what would be prob normal for a 64k window size..

                            You would be surprised how many time issues with bandwidth comes down to window size ;) We have a gig pipe and only seeing fraction of that - well yeah your X ms apart and using window size Y.. So yeah your not going to be able to come anywhere close to filling the pipe ;)

                            @johnpoz
                            Dear John,
                            Thank you for laying that out. As from tomorrow I will be having a second WAN to play with, a fiber gig line.

                            I had a good look at the BDP calculator you linked to. Looking at the variables, the only factor that is within our control is the window size. Thus, there is no other way to benefit from the gigabit potential than to increase the window size. I may do some testing using varying numbers here:
                            13bee181-3489-4c78-ad77-c101bc1c2efd-image.png

                            Ultimately, after reading VPN scaling,I may need to revert to IPsec instead of OpenVPN.

                            Question 1
                            Do you have any advice for me there? This connection is for a handful of users only (my family and me) and only I will be stressing it for speed.

                            Also (Question 2), after doing the math above, getting a much more powerful negate appliance such as the 3100 or even 6100 won't speed up the OpenVPN connection if the window size is too small. Is openVPN even a viable solution at all if I would like to achieve several 100 mbps throughput?

                            Thanks
                            Pete

                            Pete
                            Home: SG-2100 + UniFi + Synology. SG-1100 retired
                            Parents: SG-1100 + UniFi + Synology
                            Testing: SG-1100 w/ 120GB SSD via ext USB (eMMC dead). Works great

                            johnpozJ 1 Reply Last reply Reply Quote 0
                            • johnpozJ
                              johnpoz LAYER 8 Global Moderator @Cabledude
                              last edited by

                              @cabledude said in iperf3 testing 500/40 connection: 33mpbs:

                              Is openVPN even a viable solution at all if I would like to achieve several 100 mbps throughput?
                              Thanks

                              Sure it is.. Keep in mind that the outside tunnel is normally UDP, where there is not acks, etc. so the tcp traffic inside window size can be adjusted.

                              There is also the MTU of the tunnel that can be adjust, etc.

                              the only factor that is within our control is the window size

                              This is true when your talking about BDP sure..

                              only I will be stressing it for speed.

                              Not sure exactly what that means - what speed are you trying to achive, and why?

                              Is openVPN even a viable solution at all if I would like to achieve several 100 mbps throughput?

                              Yeah it is.. This is becoming dated but you might want to look at
                              https://community.openvpn.net/openvpn/wiki/Gigabit_Networks_Linux

                              Here is is really a pretty much stock openvpn connection running on a very low cost VPS I have out in LV.. Look at the latency.. And still can achieve almost 200mbps through it.. My upload is limited to 50, and looking I currently have 2 clients streaming off my plex server.. So

                              speedtest.jpg

                              here is plex running

                              plex.jpg

                              Which could put a real stress on my upload. So I wouldn't look to much at that, I only have 50 up max.. And this time of night its rare to see that..

                              I wasn't try to push you away from openvpn, was just trying to point out when doing such tests that quite often there can be variables at play that typical users might not be aware of that can effect results, etc.

                              An intelligent man is sometimes forced to be drunk to spend time with his fools
                              If you get confused: Listen to the Music Play
                              Please don't Chat/PM me for help, unless mod related
                              SG-4860 24.11 | Lab VMs 2.8, 24.11

                              C 1 Reply Last reply Reply Quote 0
                              • C
                                Cabledude @johnpoz
                                last edited by Cabledude

                                @johnpoz Thank you, that helps somewhat. And I mean that in a very positive way as I consider myself a newbie here. Only a year ago I was quite unaware of networking, except that I knew how to set up a router in bridge mode and configure my own firewall (started with UniFi USG, now SG-1100).
                                I am really struggling with this VPN stuff as I feel I need to learn so much in order to be able to do even some simple tricks. It took me quite some hours of learning how to set up OpenVPN on pfSense (with FreeRADIUS to authenticate) and connect to my network from my Macbook while off site. I managed to do this, with mostly default settings. It's like learning how to sail: one can be up and going in a few days but learning the fine details can take years.

                                Anyway sorry for the long post this has become.

                                I absolutely don't understand how you manage 200mbps with 70ms latency:

                                b48d20a4-c89f-4fbb-a8a0-f061032faae1-image.jpeg
                                So you would need a massive window size to accomplish this. Or am I seeing this the wrong way? Could you shed some light or share your secret?

                                Thanks,
                                Pete

                                Pete
                                Home: SG-2100 + UniFi + Synology. SG-1100 retired
                                Parents: SG-1100 + UniFi + Synology
                                Testing: SG-1100 w/ 120GB SSD via ext USB (eMMC dead). Works great

                                johnpozJ 1 Reply Last reply Reply Quote 0
                                • johnpozJ
                                  johnpoz LAYER 8 Global Moderator @Cabledude
                                  last edited by johnpoz

                                  @cabledude keep in mind doing multiple streams - the other option to doing large window sizes ;) for overcoming latency..

                                  Game is on - so don't have a chance currently... But I will repeat the test in morning and check the window size being used.

                                  Just wanted to show that openvpn can quite easy handle over 100mbps..

                                  An intelligent man is sometimes forced to be drunk to spend time with his fools
                                  If you get confused: Listen to the Music Play
                                  Please don't Chat/PM me for help, unless mod related
                                  SG-4860 24.11 | Lab VMs 2.8, 24.11

                                  C 1 Reply Last reply Reply Quote 1
                                  • stephenw10S
                                    stephenw10 Netgate Administrator
                                    last edited by

                                    As a side note I enable fastIO and set send/recv buffers to 512K on pretty much any server I configure these days. Unless it's a TCP tunnel but if you're doing that speed will be low anyway.

                                    Steve

                                    1 Reply Last reply Reply Quote 1
                                    • C
                                      Cabledude @johnpoz
                                      last edited by Cabledude

                                      @johnpoz @stephenw10 : If I put up a VPN connection on two SG-1100 units, one at each node, I can probably max out that hardware.

                                      I just had an insight: replacing the hardware at only one end node (e.g. swapping one of the SG-1100's for a beefy SG-6100) will probably make zero difference as the other end will be the bottleneck. Correct?

                                      Thanks,
                                      Pete

                                      Pete
                                      Home: SG-2100 + UniFi + Synology. SG-1100 retired
                                      Parents: SG-1100 + UniFi + Synology
                                      Testing: SG-1100 w/ 120GB SSD via ext USB (eMMC dead). Works great

                                      1 Reply Last reply Reply Quote 0
                                      • stephenw10S
                                        stephenw10 Netgate Administrator
                                        last edited by

                                        Probably, though there is a difference between encode and decode loading you may find it helps in one direction.

                                        Steve

                                        C 1 Reply Last reply Reply Quote 0
                                        • B
                                          Boethius @Cabledude
                                          last edited by

                                          This post is deleted!
                                          1 Reply Last reply Reply Quote 0
                                          • C
                                            Cabledude @stephenw10
                                            last edited by

                                            @stephenw10 The key assumption is that the firewall CPU is the bottleneck, correct? So you are suggesting there might be an asynchronous CPU load for server side vs client side? That sounds interesting.
                                            The next question then is: at which end would I place the most powerful device?

                                            No wait - I should be able to figure this out myself by testing the same connection again and then watch the CPU load at both ends and look for differences. Would this be a good approach?

                                            Pete
                                            Home: SG-2100 + UniFi + Synology. SG-1100 retired
                                            Parents: SG-1100 + UniFi + Synology
                                            Testing: SG-1100 w/ 120GB SSD via ext USB (eMMC dead). Works great

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.