Poor performace with Openvpn
-
@stephenw10 I did both tests as you suggested.
This is a test of iperf3 outside the tunnel. It shows line speeds.
Site A to public IP SiteB
$ iperf3 -c SiteB Connecting to host SiteB, port 5201 [ 5] local 172.16.9.21 port 47024 connected to SiteB port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 15.0 MBytes 126 Mbits/sec 0 2.02 MBytes [ 5] 1.00-2.00 sec 17.5 MBytes 147 Mbits/sec 0 2.90 MBytes [ 5] 2.00-3.00 sec 17.5 MBytes 147 Mbits/sec 661 1.52 MBytes [ 5] 3.00-4.00 sec 17.5 MBytes 147 Mbits/sec 0 1.61 MBytes [ 5] 4.00-5.00 sec 17.5 MBytes 147 Mbits/sec 0 1.68 MBytes [ 5] 5.00-6.00 sec 17.5 MBytes 147 Mbits/sec 0 1.72 MBytes [ 5] 6.00-7.00 sec 17.5 MBytes 147 Mbits/sec 0 1.76 MBytes [ 5] 7.00-8.00 sec 17.5 MBytes 147 Mbits/sec 1 1.26 MBytes [ 5] 8.00-9.00 sec 17.5 MBytes 147 Mbits/sec 0 1.35 MBytes [ 5] 9.00-10.00 sec 17.5 MBytes 147 Mbits/sec 2 1017 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 172 MBytes 145 Mbits/sec 664 sender [ 5] 0.00-10.06 sec 171 MBytes 142 Mbits/sec receiver iperf Done.Site A to public IP SiteB in reverse
$ iperf3 -c SiteB -R Connecting to host SiteB, port 5201 Reverse mode, remote host SiteB is sending [ 5] local 172.16.9.21 port 47042 connected to SiteB port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 13.3 MBytes 111 Mbits/sec [ 5] 1.00-2.00 sec 17.7 MBytes 148 Mbits/sec [ 5] 2.00-3.00 sec 17.7 MBytes 148 Mbits/sec [ 5] 3.00-4.00 sec 17.7 MBytes 149 Mbits/sec [ 5] 4.00-5.00 sec 17.7 MBytes 148 Mbits/sec [ 5] 5.00-6.00 sec 17.7 MBytes 148 Mbits/sec [ 5] 6.00-7.00 sec 17.7 MBytes 148 Mbits/sec [ 5] 7.00-8.00 sec 17.7 MBytes 148 Mbits/sec [ 5] 8.00-9.00 sec 17.7 MBytes 148 Mbits/sec [ 5] 9.00-10.00 sec 17.7 MBytes 148 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.05 sec 175 MBytes 146 Mbits/sec 0 sender [ 5] 0.00-10.00 sec 172 MBytes 145 Mbits/sec receiver iperf Done.Next is speedtest inside the tunnel. Here was seed again that the Download speed (which is equivalent to Reverse in iperf3) is poor compared to upload.
Also ran iperf3 -P4, but the results were the same as before.
Is the bottleneck in SiteA router or SiteB router? -
Hmm, much faster with the speedtest result over VPN though. Using 4 streams really made no difference?
Is there a much smaller window size in one direction?
Steve
-
This is the speed i get with 4 streams in the reverse direction. There is some improvement in speed (14 mbps to 23 mbps).
[ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.05 sec 9.12 MBytes 7.61 Mbits/sec 26 sender [ 5] 0.00-10.00 sec 8.91 MBytes 7.47 Mbits/sec receiver [ 7] 0.00-10.05 sec 6.31 MBytes 5.27 Mbits/sec 39 sender [ 7] 0.00-10.00 sec 6.20 MBytes 5.20 Mbits/sec receiver [ 9] 0.00-10.05 sec 6.31 MBytes 5.27 Mbits/sec 58 sender [ 9] 0.00-10.00 sec 6.20 MBytes 5.20 Mbits/sec receiver [ 11] 0.00-10.05 sec 6.55 MBytes 5.47 Mbits/sec 29 sender [ 11] 0.00-10.00 sec 6.38 MBytes 5.35 Mbits/sec receiver [SUM] 0.00-10.05 sec 28.3 MBytes 23.6 Mbits/sec 152 sender [SUM] 0.00-10.00 sec 27.7 MBytes 23.2 Mbits/sec receiverWhat do you mean by window size and how do i check?
-
The 'Cwnd' column in iperf. It only shows it at the end sending so you need to check both.
But you can see it's much larger outside the tunnel. You might need mss-fix to prevent fragmentation.
Steve
-
I followed this article to get the value for mssfix.
$ping -M do -s 1470 SiteB PING SiteB 1470(1498) bytes of data. ping: local error: message too long, mtu=1492 ping: local error: message too long, mtu=1492 ping: local error: message too long, mtu=1492 $ ping -M do -s 1464 -c 1 SiteB PING SiteB 1464(1492) bytes of data. 1468 bytes from SiteB: icmp_seq=1 ttl=55 time=51.5 msAccording to the article mssfix = mtu-40, so i used mssfix 1424 in the client config of SiteB. Further following this article, i subtracted 28 from the MTU and set link-mtu to 1436.
So finally the config looks so,
$cat /var/etc/openvpn/client2.conf dev ovpnc2 verb 1 dev-type tun dev-node /dev/tun2 writepid /var/run/openvpn_client2.pid #user nobody #group nobody script-security 3 daemon keepalive 10 60 ping-timer-rem persist-tun persist-key proto udp4 cipher AES-256-CBC auth SHA256 up /usr/local/sbin/ovpn-linkup down /usr/local/sbin/ovpn-linkdown local myip engine cryptodev tls-client client lport 0 management /var/etc/openvpn/client2.sock unix remote SiteA 1194 ifconfig 10.8.9.2 10.8.9.1 auth-user-pass /var/etc/openvpn/client2.up auth-retry nointeract route 172.16.1.0 255.255.255.0 route 172.16.9.0 255.255.255.0 ca /var/etc/openvpn/client2.ca cert /var/etc/openvpn/client2.cert key /var/etc/openvpn/client2.key tls-auth /var/etc/openvpn/client2.tls-auth 1 ncp-ciphers AES-128-GCM:AES-256-GCM compress lz4-v2 resolv-retry infinite topology subnet mssfix 1424 link-mtu 1436With the above config i get following result in the reverse.
$iperf3 -s Accepted connection from 172.16.9.21, port 35516 [ 5] local 192.168.1.111 port 5201 connected to 172.16.9.21 port 35518 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 416 KBytes 3.41 Mbits/sec 0 30.9 KBytes [ 5] 1.00-2.00 sec 1.09 MBytes 9.16 Mbits/sec 0 78.6 KBytes [ 5] 2.00-3.00 sec 2.71 MBytes 22.8 Mbits/sec 0 201 KBytes [ 5] 3.00-4.00 sec 2.26 MBytes 19.0 Mbits/sec 20 112 KBytes [ 5] 4.00-5.00 sec 2.22 MBytes 18.6 Mbits/sec 0 121 KBytes [ 5] 5.00-6.00 sec 2.47 MBytes 20.7 Mbits/sec 0 135 KBytes [ 5] 6.00-7.00 sec 2.71 MBytes 22.7 Mbits/sec 0 149 KBytes [ 5] 7.00-8.00 sec 2.96 MBytes 24.8 Mbits/sec 0 162 KBytes [ 5] 8.00-9.00 sec 3.21 MBytes 26.9 Mbits/sec 0 175 KBytes [ 5] 9.00-10.00 sec 3.45 MBytes 29.0 Mbits/sec 0 189 KBytes [ 5] 10.00-10.05 sec 252 KBytes 41.2 Mbits/sec 0 189 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.05 sec 23.7 MBytes 19.8 Mbits/sec 20 sender ----------------------------------------------------------- Server listening on 5201 -----------------------------------------------------------So it hasnt improved the speed.
-
Hmm the window size is tiny though.
I would run a packet capture of the iperf traffic over the tunnel and see what's happening there, is it still fragmenting.
You can test it by setting the window and mss size in the iperf client.
Steve
-
Hmmm, even the performance isnt symetrical, it is way to low.
What are the crypto settings of this tunnel ? Is AESNI used? Did you check the tunnel IPv4 settings? What version of Pfsense is that on both sites? Are this the standard Nic of the boards? With the newer OVPN versions, there are some additional buffer Settings , did you use that? Anything other on that connection?
-
What is the value of cryptographic settings in Advanced- Miscellaneous? It should be "aes-ni" on both sites... and inside the tunnel configuration ... "none" ...
-
Even without AES-NI it should be faster with that hardware.
It is possible to incorrectly use the crypto framework which can actually reduce throughput. OpenSSL will use AES-NI if the CPU has it.
But even with that 30Mbps is far lower than expected.
Steve
-
@pete35 said in Poor performace with Openvpn:
What is the value of cryptographic settings in Advanced- Miscellaneous? It should be "aes-ni" on both sites... and inside the tunnel configuration ... "none" ...
These are the crypto settings on both sides, https://imgur.com/a/Qzar59q
-
pls remove all configurations, where "cryptodev" is included and set it to aesni only.
-
@pete35 said in Poor performace with Openvpn:
pls remove all configurations, where "cryptodev" is included and set it to aesni only.
I have enabled AESNI in Advanced-Miscellaneous. In the tunnel configuration, should the 'Hardware Crypto' be set to 'No Hardware Crypto Acceleration'?
-
Yes.
-Rico
-
Set it to no-hardware crypto there.
It will be interesting to see if that makes any measurable difference. The speeds you're seeing seem to be less than anything I would expect to be affected by that.
Steve
-
I set it to 'No Hardware Crypto'. It did not make a difference.
[ 5] local 192.168.1.111 port 5201 connected to 192.16.9.21 port 33160 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.74 MBytes 14.6 Mbits/sec 2 92.3 KBytes [ 5] 1.00-2.00 sec 1.87 MBytes 15.7 Mbits/sec 0 109 KBytes [ 5] 2.00-3.00 sec 2.05 MBytes 17.2 Mbits/sec 0 117 KBytes [ 5] 3.00-4.00 sec 2.24 MBytes 18.8 Mbits/sec 0 125 KBytes [ 5] 4.00-5.00 sec 2.43 MBytes 20.3 Mbits/sec 0 138 KBytes [ 5] 5.00-6.00 sec 2.30 MBytes 19.3 Mbits/sec 3 110 KBytes [ 5] 6.00-7.00 sec 2.24 MBytes 18.8 Mbits/sec 0 131 KBytes [ 5] 7.00-8.00 sec 1.99 MBytes 16.7 Mbits/sec 12 71.5 KBytes [ 5] 8.00-9.00 sec 1.49 MBytes 12.5 Mbits/sec 0 81.9 KBytes [ 5] 9.00-10.00 sec 1.49 MBytes 12.5 Mbits/sec 0 94.9 KBytes [ 5] 10.00-10.05 sec 191 KBytes 29.6 Mbits/sec 0 96.2 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.05 sec 20.0 MBytes 16.7 Mbits/sec 17 sender ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- -
Please share all your OpenVPN settings.
What is your Encryption Algorithm?
With GCM I have seen OpenVPN traffic beyond 400 MBit/s
My SG-5100 can easy do ~250 MBit/s-Rico
-
For testing...could you set the Encryption Algorithm to None? Just to rule this out...
-Rico
-
@Rico I am using 'cipher AES-256-CBC', 'auth SHA256' and ncp-ciphers 'AES-256-GCM:AES-128-GCM'. The server side VPN config is the following and the client side config is posted above.
$less /var/etc/openvpn/server1.conf dev ovpns1 verb 1 dev-type tun dev-node /dev/tun1 writepid /var/run/openvpn_server1.pid #user nobody #group nobody script-security 3 daemon keepalive 10 60 ping-timer-rem persist-tun persist-key proto udp4 cipher AES-256-CBC auth SHA256 up /usr/local/sbin/ovpn-linkup down /usr/local/sbin/ovpn-linkdown local 127.0.0.1 tls-server server 10.8.9.0 255.255.255.0 client-config-dir /var/etc/openvpn-csc/server1 ifconfig 10.8.9.1 10.8.9.2 tls-verify "/usr/local/sbin/ovpn_auth_verify tls 'VoipVPNServer' 1" lport 1194 management /var/etc/openvpn/server1.sock unix route 192.168.0.0 255.255.255.0 route 192.168.1.0 255.255.255.0 route 192.168.2.0 255.255.255.0 route 192.168.5.0 255.255.255.0 route 192.168.6.0 255.255.255.0 route 192.168.10.0 255.255.255.0 route 192.168.18.0 255.255.255.0 route 192.168.40.0 255.255.255.0 route 192.168.50.0 255.255.255.0 ca /var/etc/openvpn/server1.ca cert /var/etc/openvpn/server1.cert key /var/etc/openvpn/server1.key dh /etc/dh-parameters.2048 crl-verify /var/etc/openvpn/server1.crl-verify tls-auth /var/etc/openvpn/server1.tls-auth 0 ncp-ciphers AES-256-GCM:AES-128-GCM compress lz4-v2 persist-remote-ip float topology subnet -
@Rico said in Poor performace with Openvpn:
For testing...could you set the Encryption Algorithm to None? Just to rule this out...
-Rico
There is no change to the result,
$ iperf3 -c 192.168.1.111 -R Connecting to host 192.168.1.111, port 5201 Reverse mode, remote host 192.168.1.111 is sending [ 5] local 172.16.9.21 port 33962 connected to 192.168.1.111 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 1.21 MBytes 10.2 Mbits/sec [ 5] 1.00-2.00 sec 1.61 MBytes 13.5 Mbits/sec [ 5] 2.00-3.00 sec 905 KBytes 7.41 Mbits/sec [ 5] 3.00-4.00 sec 1.01 MBytes 8.48 Mbits/sec [ 5] 4.00-5.00 sec 538 KBytes 4.41 Mbits/sec [ 5] 5.00-6.00 sec 753 KBytes 6.17 Mbits/sec [ 5] 6.00-7.00 sec 987 KBytes 8.09 Mbits/sec [ 5] 7.00-8.00 sec 1.18 MBytes 9.88 Mbits/sec [ 5] 8.00-9.00 sec 1.43 MBytes 12.0 Mbits/sec [ 5] 9.00-10.00 sec 1.65 MBytes 13.9 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.05 sec 11.6 MBytes 9.67 Mbits/sec 150 sender [ 5] 0.00-10.00 sec 11.2 MBytes 9.40 Mbits/sec receiver iperf Done. -
Did you disable NCP? Otherwise it will still be negotiating those.
But as I said this looks like some sort of mtu/fragmenting issue. Run a packet capture and see what's happening.
Steve
