iperf3 testing 500/40 connection: 33mpbs

[Cabledude]

@stephenw10 good question. This my home connection (the pfSense SG-1100):

Does this give you any clue?

The other node is the fibre 200/200 connection of my friend. He has a UniFi USG. I will ask him about his normal WAN latency.

Pete

[stephenw10]

Not really, you'd have to ping between the public IPs directly outside the VPN.

[Cabledude]

@stephenw10
This is the iperf3 output, default options + -t 40

Connecting to host 192.168.60.104, port 5201
[  4] local 172.16.120.2 port 51753 connected to 192.168.60.104 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   872 KBytes  7.14 Mbits/sec                  
[  4]   1.00-2.00   sec  1.56 MBytes  13.1 Mbits/sec                  
[  4]   2.00-3.00   sec  3.10 MBytes  26.0 Mbits/sec                  
[  4]   3.00-4.00   sec  4.11 MBytes  34.5 Mbits/sec                  
[  4]   4.00-5.00   sec  5.04 MBytes  42.3 Mbits/sec                  
[  4]   5.00-6.00   sec  3.98 MBytes  33.4 Mbits/sec                  
[  4]   6.00-7.00   sec  4.76 MBytes  39.9 Mbits/sec                  
[  4]   7.00-8.00   sec  3.58 MBytes  30.0 Mbits/sec                  
[  4]   8.00-9.00   sec  4.25 MBytes  35.7 Mbits/sec                  
[  4]   9.00-10.00  sec  4.71 MBytes  39.5 Mbits/sec                  
[  4]  10.00-11.00  sec  4.27 MBytes  35.8 Mbits/sec                  
[  4]  11.00-12.00  sec  4.93 MBytes  41.4 Mbits/sec                  
[  4]  12.00-13.00  sec  4.39 MBytes  36.8 Mbits/sec                  
[  4]  13.00-14.00  sec  3.74 MBytes  31.4 Mbits/sec                  
[  4]  14.00-15.00  sec  4.72 MBytes  39.6 Mbits/sec                  
[  4]  15.00-16.00  sec  3.94 MBytes  33.1 Mbits/sec                  
[  4]  16.00-17.00  sec  4.21 MBytes  35.3 Mbits/sec                  
[  4]  17.00-18.00  sec  4.08 MBytes  34.3 Mbits/sec                  
[  4]  18.00-19.00  sec  4.05 MBytes  34.0 Mbits/sec                  
[  4]  19.00-20.00  sec  3.68 MBytes  30.9 Mbits/sec                  
[  4]  20.00-21.00  sec  3.30 MBytes  27.7 Mbits/sec                  
[  4]  21.00-22.00  sec  4.85 MBytes  40.6 Mbits/sec                  
[  4]  22.00-23.00  sec  3.86 MBytes  32.4 Mbits/sec                  
[  4]  23.00-24.00  sec  4.45 MBytes  37.3 Mbits/sec                  
[  4]  24.00-25.00  sec  4.20 MBytes  35.2 Mbits/sec                  
[  4]  25.00-26.00  sec  3.68 MBytes  30.9 Mbits/sec                  
[  4]  26.00-27.00  sec  4.56 MBytes  38.3 Mbits/sec                  
[  4]  27.00-28.00  sec  4.17 MBytes  35.0 Mbits/sec                  
[  4]  28.00-29.00  sec  3.78 MBytes  31.7 Mbits/sec                  
[  4]  29.00-30.00  sec  4.62 MBytes  38.7 Mbits/sec                  
[  4]  30.00-31.00  sec  2.51 MBytes  21.0 Mbits/sec                  
[  4]  31.00-32.00  sec  4.07 MBytes  34.1 Mbits/sec                  
[  4]  32.00-33.00  sec  3.76 MBytes  31.5 Mbits/sec                  
[  4]  33.00-34.00  sec  4.40 MBytes  36.9 Mbits/sec                  
[  4]  34.00-35.00  sec  4.52 MBytes  38.0 Mbits/sec                  
[  4]  35.00-36.00  sec  4.11 MBytes  34.4 Mbits/sec                  
[  4]  36.00-37.00  sec  4.25 MBytes  35.7 Mbits/sec                  
[  4]  37.00-38.00  sec  4.39 MBytes  36.9 Mbits/sec                  
[  4]  38.00-39.00  sec  4.79 MBytes  40.2 Mbits/sec                  
[  4]  39.00-40.00  sec  3.62 MBytes  30.4 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-40.00  sec   160 MBytes  33.5 Mbits/sec                  sender
[  4]   0.00-40.00  sec   160 MBytes  33.5 Mbits/sec                  receiver

iperf Done.

[Cabledude]

@stephenw10 My friend and I have no direct ports open / nodes to ping so I'm not sure how I could ping between the public IP's.

[stephenw10]

Add a specific rule to allow only that ping from the remote public IP.

[Patch]

@cabledude said in iperf3 testing 500/40 connection: 33mpbs:

Actually I did test all individual components

What about the hardware at the other end? Running a vpn tunnel is reasonalbly processor intensive.

[Cabledude]

@stephenw10 said in iperf3 testing 500/40 connection: 33mpbs:

Add a specific rule to allow only that ping from the remote public IP.

Something of this sort?

[Cabledude]

@patch said in iperf3 testing 500/40 connection: 33mpbs:

What about the hardware at the other end? Running a vpn tunnel is reasonalbly processor intensive.

this is my end (speedtest is over WiFi but FTR VPN test was wired):

My friend's firewall is a USG 3P but the VPN wasn't S2S but remote access so I think his firewall isn't doing any encryption is it?
This is his line:

[stephenw10]

@cabledude said in iperf3 testing 500/40 connection: 33mpbs:

Something of this sort?

I would set the protocol to ICMP and the destination to WAN address. But since that's a single source IP and one you trust the risk is minimal.

Steve

[johnpoz]

Just taking his 18ms I see there as lowest one..

With only a 64k window your looking at 29.13 Mbit/sec due to the BDP..

Also lets not forget the overhead, are you trying to send 1500, those will get fragmented, again also lowering the performance. What is the box on his end doing the vpn connection? His router, what router is that - if some soho type of router, while it might be able to do a 200mbps connection, what can it push through a openvpn connection?

You prob want to adjust your txqueuelen and your tcp-queue-limit as well if your trying to see what is the max you could achieve for performance.

[Cabledude]

@johnpoz said in iperf3 testing 500/40 connection: 33mpbs:

Just taking his 18ms I see there as lowest one..

With only a 64k window your looking at 29.13 Mbit/sec due to the BDP..

That sounds about right! Could you share your calculation, I am very interested in that.

His router is a UniFi USG 3P. Not the fastest box but he can do about 80 mbps with his VPNs.

[johnpoz]

@cabledude

https://www.cisco.com/c/en/us/support/docs/ip/transmission-control-protocol-tcp/200943-Why-Your-Application-only-Uses-10Mbps-Ev.html#anc3

There are few calculators around.. Here is one I use
https://www.switch.ch/network/tools/tcp_throughput/

[Cabledude]

@johnpoz Hi John, thank you, good read.
I was just thinking what protocol am I using for the pfSense OpenVPN setup and it is UDP. Is this relevant for the BDP? If i read the article correctly it assumes TCP.

Thank you for helping out! I am learning every day :)

Pete

[johnpoz]

@cabledude what your tunnel is doesn't really matter. Your testing with TCP from your iperf outputs.

[Cabledude]

@johnpoz Ah okay I see, thank you.

So apparently building a VPN tunnel for high speed backups isn't as easy as simply connecting two gigabit nodes.

In order to troubleshoot further, would it be of any use to repeat the iperf3 test using UDP? Or using any other flags?

My friend is not available for further testing today but I might get a chance next week. He is willing to ping to my network tonight (UTC +1) though.

Pete

[johnpoz]

@cabledude said in iperf3 testing 500/40 connection: 33mpbs:

would it be of any use to repeat the iperf3 test using UDP?

Depends on what your wanting accomplish - if just to see if a difference sure, but does your backup actually leverage udp ;)

[stephenw10]

Testing using UDP eliminates the TCP windowing/ack packets issue so can be useful as long as you interpret the result correctly.

You should definitely use UDP for the VPN tunnel itself. You should only ever use TCP there if you are forced to, like is UDP is blocked. A TCP connection inside a TCP tunnel can produce horrible results.

Steve

[Cabledude]

@stephenw10 said in iperf3 testing 500/40 connection: 33mpbs:

Not really, you'd have to ping between the public IPs directly outside the VPN.

18 packets transmitted, 18 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 15.493/18.116/24.227/2.289 ms

This is a ping outside the tunnel from my friend's UniFi USG 200/200 fibre to my SG-1100 500/40 cable.

[johnpoz]

@cabledude said in iperf3 testing 500/40 connection: 33mpbs:

15.493/18.116/24.227

Yeah with those ping results, without updating your window size your seeing what would be prob normal for a 64k window size..

I recently got pulled into an issue in real life work ;) Where a customer was complaining they were seeing terrible speeds in an app from their location into the DC, well its like 30ms between the locations.

Their local firewall (palo) was preventing the window size from scaling - because they were using syn cookies..

You would be surprised how many time issues with bandwidth comes down to window size ;) We have a gig pipe and only seeing fraction of that - well yeah your X ms apart and using window size Y.. So yeah your not going to be able to come anywhere close to filling the pipe ;)

[Cabledude]

@johnpoz said in iperf3 testing 500/40 connection: 33mpbs:

Yeah with those ping results, without updating your window size your seeing what would be prob normal for a 64k window size..

You would be surprised how many time issues with bandwidth comes down to window size ;) We have a gig pipe and only seeing fraction of that - well yeah your X ms apart and using window size Y.. So yeah your not going to be able to come anywhere close to filling the pipe ;)

@johnpoz
Dear John,
Thank you for laying that out. As from tomorrow I will be having a second WAN to play with, a fiber gig line.

I had a good look at the BDP calculator you linked to. Looking at the variables, the only factor that is within our control is the window size. Thus, there is no other way to benefit from the gigabit potential than to increase the window size. I may do some testing using varying numbers here:

Ultimately, after reading VPN scaling,I may need to revert to IPsec instead of OpenVPN.

Question 1
Do you have any advice for me there? This connection is for a handful of users only (my family and me) and only I will be stressing it for speed.

Also (Question 2), after doing the math above, getting a much more powerful negate appliance such as the 3100 or even 6100 won't speed up the OpenVPN connection if the window size is too small. Is openVPN even a viable solution at all if I would like to achieve several 100 mbps throughput?

Thanks
Pete