WAN optimization/acceleration
-
@rtw915 Can you possibly try iperf on pf boxes and outside the tunnel ?
Or port forward one end on windows and run iperf again over public ip and compare.
This will give an idea about tunnel overheads. -
I setup a port forward NAT rule. I don't want to have my public IP on here so it will be officiated. It does not seem like the tunnel is really adding that much overhead, right?
C:\Temp\iperf-3.1.3-win64>iperf3.exe -c public_IP Connecting to host public_IP, port 5201 [ 4] local 10.10.10.42 port 65164 connected to public_IP port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.00 sec 768 KBytes 6.29 Mbits/sec [ 4] 1.00-2.00 sec 2.00 MBytes 16.8 Mbits/sec [ 4] 2.00-3.00 sec 1.88 MBytes 15.7 Mbits/sec [ 4] 3.00-4.00 sec 1.88 MBytes 15.7 Mbits/sec [ 4] 4.00-5.00 sec 1.88 MBytes 15.7 Mbits/sec [ 4] 5.00-6.00 sec 2.00 MBytes 16.8 Mbits/sec [ 4] 6.00-7.00 sec 1.88 MBytes 15.7 Mbits/sec [ 4] 7.00-8.00 sec 1.88 MBytes 15.7 Mbits/sec [ 4] 8.00-9.00 sec 2.00 MBytes 16.8 Mbits/sec [ 4] 9.00-10.00 sec 1.88 MBytes 15.7 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.00 sec 18.0 MBytes 15.1 Mbits/sec sender [ 4] 0.00-10.00 sec 18.0 MBytes 15.1 Mbits/sec receiver iperf Done.Optimal TCP window size:
C:\Temp\iperf-3.1.3-win64>iperf3.exe -c public_IP -w 2250000 Connecting to host public_IP, port 5201 [ 4] local 10.10.10.42 port 60842 connected to public_IP port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.00 sec 2.88 MBytes 24.1 Mbits/sec [ 4] 1.00-2.00 sec 1.00 MBytes 8.39 Mbits/sec [ 4] 2.00-3.00 sec 2.25 MBytes 18.9 Mbits/sec [ 4] 3.00-4.00 sec 4.12 MBytes 34.6 Mbits/sec [ 4] 4.00-5.00 sec 5.12 MBytes 43.0 Mbits/sec [ 4] 5.00-6.00 sec 6.50 MBytes 54.6 Mbits/sec [ 4] 6.00-7.00 sec 8.25 MBytes 69.2 Mbits/sec [ 4] 7.00-8.00 sec 9.25 MBytes 77.6 Mbits/sec [ 4] 8.00-9.00 sec 6.00 MBytes 50.3 Mbits/sec [ 4] 9.00-10.00 sec 3.00 MBytes 25.2 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.00 sec 48.4 MBytes 40.6 Mbits/sec sender [ 4] 0.00-10.00 sec 46.5 MBytes 39.0 Mbits/sec receiver iperf Done.10 Parallel streams:
C:\Temp\iperf-3.1.3-win64>iperf3.exe -c public_IP -P 10 Connecting to host public_IP, port 5201 [ 4] local 10.10.10.42 port 59407 connected to public_IP port 5201 [ 6] local 10.10.10.42 port 59417 connected to public_IP port 5201 [ 8] local 10.10.10.42 port 59421 connected to public_IP port 5201 [ 10] local 10.10.10.42 port 59431 connected to public_IP port 5201 [ 12] local 10.10.10.42 port 59441 connected to public_IP port 5201 [ 14] local 10.10.10.42 port 59451 connected to public_IP port 5201 [ 16] local 10.10.10.42 port 59464 connected to public_IP port 5201 [ 18] local 10.10.10.42 port 59465 connected to public_IP port 5201 [ 20] local 10.10.10.42 port 59475 connected to public_IP port 5201 [ 22] local 10.10.10.42 port 59488 connected to public_IP port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.00 sec 256 KBytes 2.10 Mbits/sec [ 6] 0.00-1.00 sec 896 KBytes 7.33 Mbits/sec [ 8] 0.00-1.00 sec 384 KBytes 3.14 Mbits/sec [ 10] 0.00-1.00 sec 384 KBytes 3.14 Mbits/sec [ 12] 0.00-1.00 sec 384 KBytes 3.14 Mbits/sec [ 14] 0.00-1.00 sec 256 KBytes 2.10 Mbits/sec [ 16] 0.00-1.00 sec 896 KBytes 7.33 Mbits/sec [ 18] 0.00-1.00 sec 640 KBytes 5.24 Mbits/sec [ 20] 0.00-1.00 sec 384 KBytes 3.14 Mbits/sec [ 22] 0.00-1.00 sec 384 KBytes 3.14 Mbits/sec [SUM] 0.00-1.00 sec 4.75 MBytes 39.8 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 1.00-2.00 sec 512 KBytes 4.20 Mbits/sec [ 6] 1.00-2.00 sec 2.00 MBytes 16.8 Mbits/sec [ 8] 1.00-2.00 sec 512 KBytes 4.20 Mbits/sec [ 10] 1.00-2.00 sec 128 KBytes 1.05 Mbits/sec [ 12] 1.00-2.00 sec 512 KBytes 4.20 Mbits/sec [ 14] 1.00-2.00 sec 512 KBytes 4.20 Mbits/sec [ 16] 1.00-2.00 sec 1.88 MBytes 15.7 Mbits/sec [ 18] 1.00-2.00 sec 896 KBytes 7.34 Mbits/sec [ 20] 1.00-2.00 sec 0.00 Bytes 0.00 bits/sec [ 22] 1.00-2.00 sec 256 KBytes 2.10 Mbits/sec [SUM] 1.00-2.00 sec 7.12 MBytes 59.8 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 2.00-3.00 sec 512 KBytes 4.20 Mbits/sec [ 6] 2.00-3.00 sec 2.00 MBytes 16.8 Mbits/sec [ 8] 2.00-3.00 sec 768 KBytes 6.29 Mbits/sec [ 10] 2.00-3.00 sec 128 KBytes 1.05 Mbits/sec [ 12] 2.00-3.00 sec 512 KBytes 4.20 Mbits/sec [ 14] 2.00-3.00 sec 512 KBytes 4.20 Mbits/sec [ 16] 2.00-3.00 sec 1.88 MBytes 15.7 Mbits/sec [ 18] 2.00-3.00 sec 1.25 MBytes 10.5 Mbits/sec [ 20] 2.00-3.00 sec 256 KBytes 2.10 Mbits/sec [ 22] 2.00-3.00 sec 512 KBytes 4.20 Mbits/sec [SUM] 2.00-3.00 sec 8.25 MBytes 69.2 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 3.00-4.00 sec 640 KBytes 5.24 Mbits/sec [ 6] 3.00-4.00 sec 1.88 MBytes 15.7 Mbits/sec [ 8] 3.00-4.00 sec 896 KBytes 7.34 Mbits/sec [ 10] 3.00-4.00 sec 384 KBytes 3.14 Mbits/sec [ 12] 3.00-4.00 sec 768 KBytes 6.29 Mbits/sec [ 14] 3.00-4.00 sec 640 KBytes 5.24 Mbits/sec [ 16] 3.00-4.00 sec 2.00 MBytes 16.8 Mbits/sec [ 18] 3.00-4.00 sec 896 KBytes 7.34 Mbits/sec [ 20] 3.00-4.00 sec 256 KBytes 2.10 Mbits/sec [ 22] 3.00-4.00 sec 640 KBytes 5.24 Mbits/sec [SUM] 3.00-4.00 sec 8.88 MBytes 74.4 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 4.00-5.00 sec 896 KBytes 7.34 Mbits/sec [ 6] 4.00-5.00 sec 1.88 MBytes 15.7 Mbits/sec [ 8] 4.00-5.00 sec 1.12 MBytes 9.43 Mbits/sec [ 10] 4.00-5.00 sec 384 KBytes 3.14 Mbits/sec [ 12] 4.00-5.00 sec 1.00 MBytes 8.38 Mbits/sec [ 14] 4.00-5.00 sec 896 KBytes 7.34 Mbits/sec [ 16] 4.00-5.00 sec 1.88 MBytes 15.7 Mbits/sec [ 18] 4.00-5.00 sec 1.00 MBytes 8.38 Mbits/sec [ 20] 4.00-5.00 sec 384 KBytes 3.14 Mbits/sec [ 22] 4.00-5.00 sec 768 KBytes 6.29 Mbits/sec [SUM] 4.00-5.00 sec 10.1 MBytes 84.9 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 5.00-6.00 sec 1.25 MBytes 10.5 Mbits/sec [ 6] 5.00-6.00 sec 1.88 MBytes 15.7 Mbits/sec [ 8] 5.00-6.00 sec 1.62 MBytes 13.6 Mbits/sec [ 10] 5.00-6.00 sec 640 KBytes 5.25 Mbits/sec [ 12] 5.00-6.00 sec 1.38 MBytes 11.5 Mbits/sec [ 14] 5.00-6.00 sec 1.12 MBytes 9.44 Mbits/sec [ 16] 5.00-6.00 sec 2.00 MBytes 16.8 Mbits/sec [ 18] 5.00-6.00 sec 1.25 MBytes 10.5 Mbits/sec [ 20] 5.00-6.00 sec 640 KBytes 5.25 Mbits/sec [ 22] 5.00-6.00 sec 1.00 MBytes 8.39 Mbits/sec [SUM] 5.00-6.00 sec 12.8 MBytes 107 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 6.00-7.00 sec 1.62 MBytes 13.6 Mbits/sec [ 6] 6.00-7.00 sec 2.00 MBytes 16.8 Mbits/sec [ 8] 6.00-7.00 sec 1.88 MBytes 15.7 Mbits/sec [ 10] 6.00-7.00 sec 896 KBytes 7.34 Mbits/sec [ 12] 6.00-7.00 sec 1.75 MBytes 14.7 Mbits/sec [ 14] 6.00-7.00 sec 1.62 MBytes 13.6 Mbits/sec [ 16] 6.00-7.00 sec 1.88 MBytes 15.7 Mbits/sec [ 18] 6.00-7.00 sec 1.88 MBytes 15.7 Mbits/sec [ 20] 6.00-7.00 sec 768 KBytes 6.29 Mbits/sec [ 22] 6.00-7.00 sec 1.38 MBytes 11.5 Mbits/sec [SUM] 6.00-7.00 sec 15.6 MBytes 131 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 7.00-8.00 sec 1.00 MBytes 8.39 Mbits/sec [ 6] 7.00-8.00 sec 1.88 MBytes 15.7 Mbits/sec [ 8] 7.00-8.00 sec 2.00 MBytes 16.8 Mbits/sec [ 10] 7.00-8.00 sec 512 KBytes 4.20 Mbits/sec [ 12] 7.00-8.00 sec 1.88 MBytes 15.7 Mbits/sec [ 14] 7.00-8.00 sec 128 KBytes 1.05 Mbits/sec [ 16] 7.00-8.00 sec 2.00 MBytes 16.8 Mbits/sec [ 18] 7.00-8.00 sec 1.88 MBytes 15.7 Mbits/sec [ 20] 7.00-8.00 sec 512 KBytes 4.20 Mbits/sec [ 22] 7.00-8.00 sec 128 KBytes 1.05 Mbits/sec [SUM] 7.00-8.00 sec 11.9 MBytes 99.6 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 8.00-9.00 sec 1.38 MBytes 11.5 Mbits/sec [ 6] 8.00-9.00 sec 1.88 MBytes 15.7 Mbits/sec [ 8] 8.00-9.00 sec 1.88 MBytes 15.7 Mbits/sec [ 10] 8.00-9.00 sec 768 KBytes 6.29 Mbits/sec [ 12] 8.00-9.00 sec 1.88 MBytes 15.7 Mbits/sec [ 14] 8.00-9.00 sec 256 KBytes 2.10 Mbits/sec [ 16] 8.00-9.00 sec 1.88 MBytes 15.7 Mbits/sec [ 18] 8.00-9.00 sec 1.88 MBytes 15.7 Mbits/sec [ 20] 8.00-9.00 sec 640 KBytes 5.24 Mbits/sec [ 22] 8.00-9.00 sec 384 KBytes 3.14 Mbits/sec [SUM] 8.00-9.00 sec 12.8 MBytes 107 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 4] 9.00-10.00 sec 1.75 MBytes 14.7 Mbits/sec [ 6] 9.00-10.00 sec 1.88 MBytes 15.7 Mbits/sec [ 8] 9.00-10.00 sec 1.88 MBytes 15.7 Mbits/sec [ 10] 9.00-10.00 sec 1.00 MBytes 8.39 Mbits/sec [ 12] 9.00-10.00 sec 1.88 MBytes 15.7 Mbits/sec [ 14] 9.00-10.00 sec 384 KBytes 3.15 Mbits/sec [ 16] 9.00-10.00 sec 1.88 MBytes 15.7 Mbits/sec [ 18] 9.00-10.00 sec 1.88 MBytes 15.7 Mbits/sec [ 20] 9.00-10.00 sec 1.00 MBytes 8.39 Mbits/sec [ 22] 9.00-10.00 sec 384 KBytes 3.15 Mbits/sec [SUM] 9.00-10.00 sec 13.9 MBytes 116 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.00 sec 9.75 MBytes 8.18 Mbits/sec sender [ 4] 0.00-10.00 sec 9.70 MBytes 8.14 Mbits/sec receiver [ 6] 0.00-10.00 sec 18.1 MBytes 15.2 Mbits/sec sender [ 6] 0.00-10.00 sec 18.1 MBytes 15.2 Mbits/sec receiver [ 8] 0.00-10.00 sec 12.9 MBytes 10.8 Mbits/sec sender [ 8] 0.00-10.00 sec 12.9 MBytes 10.8 Mbits/sec receiver [ 10] 0.00-10.00 sec 5.12 MBytes 4.30 Mbits/sec sender [ 10] 0.00-10.00 sec 5.04 MBytes 4.23 Mbits/sec receiver [ 12] 0.00-10.00 sec 11.9 MBytes 9.96 Mbits/sec sender [ 12] 0.00-10.00 sec 11.8 MBytes 9.92 Mbits/sec receiver [ 14] 0.00-10.00 sec 6.25 MBytes 5.24 Mbits/sec sender [ 14] 0.00-10.00 sec 6.11 MBytes 5.12 Mbits/sec receiver [ 16] 0.00-10.00 sec 18.1 MBytes 15.2 Mbits/sec sender [ 16] 0.00-10.00 sec 18.1 MBytes 15.2 Mbits/sec receiver [ 18] 0.00-10.00 sec 13.4 MBytes 11.2 Mbits/sec sender [ 18] 0.00-10.00 sec 13.4 MBytes 11.2 Mbits/sec receiver [ 20] 0.00-10.00 sec 4.75 MBytes 3.98 Mbits/sec sender [ 20] 0.00-10.00 sec 4.65 MBytes 3.90 Mbits/sec receiver [ 22] 0.00-10.00 sec 5.75 MBytes 4.82 Mbits/sec sender [ 22] 0.00-10.00 sec 5.55 MBytes 4.66 Mbits/sec receiver [SUM] 0.00-10.00 sec 106 MBytes 88.9 Mbits/sec sender [SUM] 0.00-10.00 sec 105 MBytes 88.4 Mbits/sec receiver iperf Done.Running the UDP switch publicly resulted in hundreds of datagrams being received out-of-order.
-
@rtw915 Well, yes, it looks like the tunnel isn't adding anything significant. However speed does fluctuate a lot and this is probably an indication for congestion somewhere.
Not much you can do I guess too.
With such fluctuations, anything tcp will suffer, especially anything smb based
7zip supports a multihread option in bzip2 format.
Check it out. (also be careful with multithreading. It can easily saturate all your resources.
Also robocopy supports parallel copies, (that is if you break up your file in volumes)
Linux is better at compressing, splitting, rsyncing and decompressing, but probably introducing a new server just for that might be an overkill too. -
Mmm, 38Mbps doesn't seem too bad given those results though. That's close to what you see with a single stream in iperf using the optimised window size.
Given that the tunnel itself (or even the route) doesn't support more that 80Mbps it's about what I might expect there. I'm not sure if there's any improvement to be found without changing something significant.Steve
-
@netblues I remembered why I got stuck when trying to split a single file into smaller volumes. The source does not have enough space to duplicate the file into the smaller volumes to then transfer in parallel. Is it possible to start moving the created volumes in parallel as they are created?
@stephenw10 I agree, but it is just too slow.
-
Higher WAN bandwidth? Faster hardware? What's it running on now?
-
Using Mathis equation: Rate < (MSS/RTT)*(1 / sqrt(p))
Where:
MSS: maximum segment size - 1300 for IPSec overhead
RTT: round trip time - 90ms
P: Loss rate in % - .0002 seems about right I hope10213764.62 bytes < (1300/.09) * (1/SQRT(.0002%))
10213764.62 bytes * 8 /10^6 = ~81Mbps
This calculator https://wintelguy.com/wanperf.pl does the Mathis equation too, and increasing the WAN bandwidth does not increase the throughput.
@stephenw10 We are running SYS-1019D-FRN8TP with 64GB of mem. Based on the minimal throughput difference between public and IPSec iperf tests and Mathis equation output, I don’t think it is a hardware issue. I think TCP connections need to be parallelized and/or a proxy to perform TCP ACK trickery with an optimal TCP window size. I don't know, maybe there is a third solution.
-Ryan
-
Ok, so after a few more hours of googling I found a file transfer tool for high latency connections that seems to work great! The throughput that UFTP http://uftp-multicast.sourceforge.net/ was able to achieve almost saturated the bandwidth of the connection through the tunnel (186 Mbps)! I tested with an 8GB file.
The remote side (Client) I ran this:
uftpd -B 2250000 -L C:\temp\uftpd_log.txt -F @LOGOn the local side (Server) this:
uftp -f -B 2250000 -b 1200 -R 200000 -M 10.30.20.20 "C:\temp\2-25-2021.pcapng"Transfer summary:
Transfer status: Host: 0x8811096B Status: Completed time: 359.405 seconds Total elapsed time: 359.405 seconds Overall throughput: 23350.92 KB/s ----------------------------- Finishing group Sending DONE 1.1 Got COMPLETE from client 0x8811096B Late completions: Sending DONE_CONF 2.1I think this solves the batch file transfer speed issue. Now the SQL team needs me to find a way to improve SQL linked server transfer rates to synchronize transactions.
-
Nice find!
-
@rtw915 said in WAN optimization/acceleration:
Now the SQL team needs me to find a way to improve SQL linked server transfer rates to synchronize transactions.
This will bring you back to the initial wan accelerator solution.
The only other possible solution is to redesing the db subsystem, utilizing some way of sql replication, taking into consideration propagation delays
