Poor performace with Openvpn
-
I followed this article to get the value for mssfix.
$ping -M do -s 1470 SiteB PING SiteB 1470(1498) bytes of data. ping: local error: message too long, mtu=1492 ping: local error: message too long, mtu=1492 ping: local error: message too long, mtu=1492 $ ping -M do -s 1464 -c 1 SiteB PING SiteB 1464(1492) bytes of data. 1468 bytes from SiteB: icmp_seq=1 ttl=55 time=51.5 msAccording to the article mssfix = mtu-40, so i used mssfix 1424 in the client config of SiteB. Further following this article, i subtracted 28 from the MTU and set link-mtu to 1436.
So finally the config looks so,
$cat /var/etc/openvpn/client2.conf dev ovpnc2 verb 1 dev-type tun dev-node /dev/tun2 writepid /var/run/openvpn_client2.pid #user nobody #group nobody script-security 3 daemon keepalive 10 60 ping-timer-rem persist-tun persist-key proto udp4 cipher AES-256-CBC auth SHA256 up /usr/local/sbin/ovpn-linkup down /usr/local/sbin/ovpn-linkdown local myip engine cryptodev tls-client client lport 0 management /var/etc/openvpn/client2.sock unix remote SiteA 1194 ifconfig 10.8.9.2 10.8.9.1 auth-user-pass /var/etc/openvpn/client2.up auth-retry nointeract route 172.16.1.0 255.255.255.0 route 172.16.9.0 255.255.255.0 ca /var/etc/openvpn/client2.ca cert /var/etc/openvpn/client2.cert key /var/etc/openvpn/client2.key tls-auth /var/etc/openvpn/client2.tls-auth 1 ncp-ciphers AES-128-GCM:AES-256-GCM compress lz4-v2 resolv-retry infinite topology subnet mssfix 1424 link-mtu 1436With the above config i get following result in the reverse.
$iperf3 -s Accepted connection from 172.16.9.21, port 35516 [ 5] local 192.168.1.111 port 5201 connected to 172.16.9.21 port 35518 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 416 KBytes 3.41 Mbits/sec 0 30.9 KBytes [ 5] 1.00-2.00 sec 1.09 MBytes 9.16 Mbits/sec 0 78.6 KBytes [ 5] 2.00-3.00 sec 2.71 MBytes 22.8 Mbits/sec 0 201 KBytes [ 5] 3.00-4.00 sec 2.26 MBytes 19.0 Mbits/sec 20 112 KBytes [ 5] 4.00-5.00 sec 2.22 MBytes 18.6 Mbits/sec 0 121 KBytes [ 5] 5.00-6.00 sec 2.47 MBytes 20.7 Mbits/sec 0 135 KBytes [ 5] 6.00-7.00 sec 2.71 MBytes 22.7 Mbits/sec 0 149 KBytes [ 5] 7.00-8.00 sec 2.96 MBytes 24.8 Mbits/sec 0 162 KBytes [ 5] 8.00-9.00 sec 3.21 MBytes 26.9 Mbits/sec 0 175 KBytes [ 5] 9.00-10.00 sec 3.45 MBytes 29.0 Mbits/sec 0 189 KBytes [ 5] 10.00-10.05 sec 252 KBytes 41.2 Mbits/sec 0 189 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.05 sec 23.7 MBytes 19.8 Mbits/sec 20 sender ----------------------------------------------------------- Server listening on 5201 -----------------------------------------------------------So it hasnt improved the speed.
-
Hmm the window size is tiny though.
I would run a packet capture of the iperf traffic over the tunnel and see what's happening there, is it still fragmenting.
You can test it by setting the window and mss size in the iperf client.
Steve
-
Hmmm, even the performance isnt symetrical, it is way to low.
What are the crypto settings of this tunnel ? Is AESNI used? Did you check the tunnel IPv4 settings? What version of Pfsense is that on both sites? Are this the standard Nic of the boards? With the newer OVPN versions, there are some additional buffer Settings , did you use that? Anything other on that connection?
-
What is the value of cryptographic settings in Advanced- Miscellaneous? It should be "aes-ni" on both sites... and inside the tunnel configuration ... "none" ...
-
Even without AES-NI it should be faster with that hardware.
It is possible to incorrectly use the crypto framework which can actually reduce throughput. OpenSSL will use AES-NI if the CPU has it.
But even with that 30Mbps is far lower than expected.
Steve
-
@pete35 said in Poor performace with Openvpn:
What is the value of cryptographic settings in Advanced- Miscellaneous? It should be "aes-ni" on both sites... and inside the tunnel configuration ... "none" ...
These are the crypto settings on both sides, https://imgur.com/a/Qzar59q
-
pls remove all configurations, where "cryptodev" is included and set it to aesni only.
-
@pete35 said in Poor performace with Openvpn:
pls remove all configurations, where "cryptodev" is included and set it to aesni only.
I have enabled AESNI in Advanced-Miscellaneous. In the tunnel configuration, should the 'Hardware Crypto' be set to 'No Hardware Crypto Acceleration'?
-
Yes.
-Rico
-
Set it to no-hardware crypto there.
It will be interesting to see if that makes any measurable difference. The speeds you're seeing seem to be less than anything I would expect to be affected by that.
Steve
-
I set it to 'No Hardware Crypto'. It did not make a difference.
[ 5] local 192.168.1.111 port 5201 connected to 192.16.9.21 port 33160 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.74 MBytes 14.6 Mbits/sec 2 92.3 KBytes [ 5] 1.00-2.00 sec 1.87 MBytes 15.7 Mbits/sec 0 109 KBytes [ 5] 2.00-3.00 sec 2.05 MBytes 17.2 Mbits/sec 0 117 KBytes [ 5] 3.00-4.00 sec 2.24 MBytes 18.8 Mbits/sec 0 125 KBytes [ 5] 4.00-5.00 sec 2.43 MBytes 20.3 Mbits/sec 0 138 KBytes [ 5] 5.00-6.00 sec 2.30 MBytes 19.3 Mbits/sec 3 110 KBytes [ 5] 6.00-7.00 sec 2.24 MBytes 18.8 Mbits/sec 0 131 KBytes [ 5] 7.00-8.00 sec 1.99 MBytes 16.7 Mbits/sec 12 71.5 KBytes [ 5] 8.00-9.00 sec 1.49 MBytes 12.5 Mbits/sec 0 81.9 KBytes [ 5] 9.00-10.00 sec 1.49 MBytes 12.5 Mbits/sec 0 94.9 KBytes [ 5] 10.00-10.05 sec 191 KBytes 29.6 Mbits/sec 0 96.2 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.05 sec 20.0 MBytes 16.7 Mbits/sec 17 sender ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- -
Please share all your OpenVPN settings.
What is your Encryption Algorithm?
With GCM I have seen OpenVPN traffic beyond 400 MBit/s
My SG-5100 can easy do ~250 MBit/s-Rico
-
For testing...could you set the Encryption Algorithm to None? Just to rule this out...
-Rico
-
@Rico I am using 'cipher AES-256-CBC', 'auth SHA256' and ncp-ciphers 'AES-256-GCM:AES-128-GCM'. The server side VPN config is the following and the client side config is posted above.
$less /var/etc/openvpn/server1.conf dev ovpns1 verb 1 dev-type tun dev-node /dev/tun1 writepid /var/run/openvpn_server1.pid #user nobody #group nobody script-security 3 daemon keepalive 10 60 ping-timer-rem persist-tun persist-key proto udp4 cipher AES-256-CBC auth SHA256 up /usr/local/sbin/ovpn-linkup down /usr/local/sbin/ovpn-linkdown local 127.0.0.1 tls-server server 10.8.9.0 255.255.255.0 client-config-dir /var/etc/openvpn-csc/server1 ifconfig 10.8.9.1 10.8.9.2 tls-verify "/usr/local/sbin/ovpn_auth_verify tls 'VoipVPNServer' 1" lport 1194 management /var/etc/openvpn/server1.sock unix route 192.168.0.0 255.255.255.0 route 192.168.1.0 255.255.255.0 route 192.168.2.0 255.255.255.0 route 192.168.5.0 255.255.255.0 route 192.168.6.0 255.255.255.0 route 192.168.10.0 255.255.255.0 route 192.168.18.0 255.255.255.0 route 192.168.40.0 255.255.255.0 route 192.168.50.0 255.255.255.0 ca /var/etc/openvpn/server1.ca cert /var/etc/openvpn/server1.cert key /var/etc/openvpn/server1.key dh /etc/dh-parameters.2048 crl-verify /var/etc/openvpn/server1.crl-verify tls-auth /var/etc/openvpn/server1.tls-auth 0 ncp-ciphers AES-256-GCM:AES-128-GCM compress lz4-v2 persist-remote-ip float topology subnet -
@Rico said in Poor performace with Openvpn:
For testing...could you set the Encryption Algorithm to None? Just to rule this out...
-Rico
There is no change to the result,
$ iperf3 -c 192.168.1.111 -R Connecting to host 192.168.1.111, port 5201 Reverse mode, remote host 192.168.1.111 is sending [ 5] local 172.16.9.21 port 33962 connected to 192.168.1.111 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 1.21 MBytes 10.2 Mbits/sec [ 5] 1.00-2.00 sec 1.61 MBytes 13.5 Mbits/sec [ 5] 2.00-3.00 sec 905 KBytes 7.41 Mbits/sec [ 5] 3.00-4.00 sec 1.01 MBytes 8.48 Mbits/sec [ 5] 4.00-5.00 sec 538 KBytes 4.41 Mbits/sec [ 5] 5.00-6.00 sec 753 KBytes 6.17 Mbits/sec [ 5] 6.00-7.00 sec 987 KBytes 8.09 Mbits/sec [ 5] 7.00-8.00 sec 1.18 MBytes 9.88 Mbits/sec [ 5] 8.00-9.00 sec 1.43 MBytes 12.0 Mbits/sec [ 5] 9.00-10.00 sec 1.65 MBytes 13.9 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.05 sec 11.6 MBytes 9.67 Mbits/sec 150 sender [ 5] 0.00-10.00 sec 11.2 MBytes 9.40 Mbits/sec receiver iperf Done. -
Did you disable NCP? Otherwise it will still be negotiating those.
But as I said this looks like some sort of mtu/fragmenting issue. Run a packet capture and see what's happening.
Steve
-
@stephenw10 said in Poor performace with Openvpn:
Did you disable NCP? Otherwise it will still be negotiating those.
But as I said this looks like some sort of mtu/fragmenting issue. Run a packet capture and see what's happening.
Steve
I have done packet capture. What should i be looking for in wireshark?
-
Fragmented packets is the first thing I would be looking for. Then what size the large fragments are.
Otherwise check the initial TCP transactions for the window size etc. Do you see retranmissions or missing packet errors.
Steve
-
@stephenw10 said in Poor performace with Openvpn:
Fragmented packets is the first thing I would be looking for. Then what size the large fragments are.
Otherwise check the initial TCP transactions for the window size etc. Do you see retranmissions or missing packet errors.
Steve
I captured VPN interface and opened the capture in wireshark. I dont see any fragmented packet in the Info column.
- Should I capture the WAN interface instead of the VPN interface?
- Is there any wireshark tutorial which shows how to identify whether fragmentation is occuring?
-
I'm not aware of any specific tutorial for that, there are many though.
Packet fragmentation is not difficult to spot though even just from the pfSense interface.
11:54:54.373954 IP 172.21.16.35 > 172.21.16.5: ICMP echo request, id 39018, seq 0, length 1480 11:54:54.373963 IP 172.21.16.35 > 172.21.16.5: ip-proto-1 11:54:55.374734 IP 172.21.16.35 > 172.21.16.5: ICMP echo request, id 39018, seq 1, length 1480 11:54:55.374743 IP 172.21.16.35 > 172.21.16.5: ip-proto-1 11:54:56.375739 IP 172.21.16.35 > 172.21.16.5: ICMP echo request, id 39018, seq 2, length 1480 11:54:56.375749 IP 172.21.16.35 > 172.21.16.5: ip-proto-1Pinging size 2000 byte packets, each full size packet is followed by a fragment to make up the full packet.
In that particular case there is a switch I have that doesn't pass packet fragments.
Try looking at the OpenVPN traffic on the WAN also. If that is fragmented it will kill performance.
Steve
