Poor performace with Openvpn
-
Hmm the window size is tiny though.
I would run a packet capture of the iperf traffic over the tunnel and see what's happening there, is it still fragmenting.
You can test it by setting the window and mss size in the iperf client.
Steve
-
Hmmm, even the performance isnt symetrical, it is way to low.
What are the crypto settings of this tunnel ? Is AESNI used? Did you check the tunnel IPv4 settings? What version of Pfsense is that on both sites? Are this the standard Nic of the boards? With the newer OVPN versions, there are some additional buffer Settings , did you use that? Anything other on that connection?
-
What is the value of cryptographic settings in Advanced- Miscellaneous? It should be "aes-ni" on both sites... and inside the tunnel configuration ... "none" ...
-
Even without AES-NI it should be faster with that hardware.
It is possible to incorrectly use the crypto framework which can actually reduce throughput. OpenSSL will use AES-NI if the CPU has it.
But even with that 30Mbps is far lower than expected.
Steve
-
@pete35 said in Poor performace with Openvpn:
What is the value of cryptographic settings in Advanced- Miscellaneous? It should be "aes-ni" on both sites... and inside the tunnel configuration ... "none" ...
These are the crypto settings on both sides, https://imgur.com/a/Qzar59q
-
pls remove all configurations, where "cryptodev" is included and set it to aesni only.
-
@pete35 said in Poor performace with Openvpn:
pls remove all configurations, where "cryptodev" is included and set it to aesni only.
I have enabled AESNI in Advanced-Miscellaneous. In the tunnel configuration, should the 'Hardware Crypto' be set to 'No Hardware Crypto Acceleration'?
-
Yes.
-Rico
-
Set it to no-hardware crypto there.
It will be interesting to see if that makes any measurable difference. The speeds you're seeing seem to be less than anything I would expect to be affected by that.
Steve
-
I set it to 'No Hardware Crypto'. It did not make a difference.
[ 5] local 192.168.1.111 port 5201 connected to 192.16.9.21 port 33160 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 1.74 MBytes 14.6 Mbits/sec 2 92.3 KBytes [ 5] 1.00-2.00 sec 1.87 MBytes 15.7 Mbits/sec 0 109 KBytes [ 5] 2.00-3.00 sec 2.05 MBytes 17.2 Mbits/sec 0 117 KBytes [ 5] 3.00-4.00 sec 2.24 MBytes 18.8 Mbits/sec 0 125 KBytes [ 5] 4.00-5.00 sec 2.43 MBytes 20.3 Mbits/sec 0 138 KBytes [ 5] 5.00-6.00 sec 2.30 MBytes 19.3 Mbits/sec 3 110 KBytes [ 5] 6.00-7.00 sec 2.24 MBytes 18.8 Mbits/sec 0 131 KBytes [ 5] 7.00-8.00 sec 1.99 MBytes 16.7 Mbits/sec 12 71.5 KBytes [ 5] 8.00-9.00 sec 1.49 MBytes 12.5 Mbits/sec 0 81.9 KBytes [ 5] 9.00-10.00 sec 1.49 MBytes 12.5 Mbits/sec 0 94.9 KBytes [ 5] 10.00-10.05 sec 191 KBytes 29.6 Mbits/sec 0 96.2 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.05 sec 20.0 MBytes 16.7 Mbits/sec 17 sender ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- -
Please share all your OpenVPN settings.
What is your Encryption Algorithm?
With GCM I have seen OpenVPN traffic beyond 400 MBit/s
My SG-5100 can easy do ~250 MBit/s-Rico
-
For testing...could you set the Encryption Algorithm to None? Just to rule this out...
-Rico
-
@Rico I am using 'cipher AES-256-CBC', 'auth SHA256' and ncp-ciphers 'AES-256-GCM:AES-128-GCM'. The server side VPN config is the following and the client side config is posted above.
$less /var/etc/openvpn/server1.conf dev ovpns1 verb 1 dev-type tun dev-node /dev/tun1 writepid /var/run/openvpn_server1.pid #user nobody #group nobody script-security 3 daemon keepalive 10 60 ping-timer-rem persist-tun persist-key proto udp4 cipher AES-256-CBC auth SHA256 up /usr/local/sbin/ovpn-linkup down /usr/local/sbin/ovpn-linkdown local 127.0.0.1 tls-server server 10.8.9.0 255.255.255.0 client-config-dir /var/etc/openvpn-csc/server1 ifconfig 10.8.9.1 10.8.9.2 tls-verify "/usr/local/sbin/ovpn_auth_verify tls 'VoipVPNServer' 1" lport 1194 management /var/etc/openvpn/server1.sock unix route 192.168.0.0 255.255.255.0 route 192.168.1.0 255.255.255.0 route 192.168.2.0 255.255.255.0 route 192.168.5.0 255.255.255.0 route 192.168.6.0 255.255.255.0 route 192.168.10.0 255.255.255.0 route 192.168.18.0 255.255.255.0 route 192.168.40.0 255.255.255.0 route 192.168.50.0 255.255.255.0 ca /var/etc/openvpn/server1.ca cert /var/etc/openvpn/server1.cert key /var/etc/openvpn/server1.key dh /etc/dh-parameters.2048 crl-verify /var/etc/openvpn/server1.crl-verify tls-auth /var/etc/openvpn/server1.tls-auth 0 ncp-ciphers AES-256-GCM:AES-128-GCM compress lz4-v2 persist-remote-ip float topology subnet -
@Rico said in Poor performace with Openvpn:
For testing...could you set the Encryption Algorithm to None? Just to rule this out...
-Rico
There is no change to the result,
$ iperf3 -c 192.168.1.111 -R Connecting to host 192.168.1.111, port 5201 Reverse mode, remote host 192.168.1.111 is sending [ 5] local 172.16.9.21 port 33962 connected to 192.168.1.111 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 1.21 MBytes 10.2 Mbits/sec [ 5] 1.00-2.00 sec 1.61 MBytes 13.5 Mbits/sec [ 5] 2.00-3.00 sec 905 KBytes 7.41 Mbits/sec [ 5] 3.00-4.00 sec 1.01 MBytes 8.48 Mbits/sec [ 5] 4.00-5.00 sec 538 KBytes 4.41 Mbits/sec [ 5] 5.00-6.00 sec 753 KBytes 6.17 Mbits/sec [ 5] 6.00-7.00 sec 987 KBytes 8.09 Mbits/sec [ 5] 7.00-8.00 sec 1.18 MBytes 9.88 Mbits/sec [ 5] 8.00-9.00 sec 1.43 MBytes 12.0 Mbits/sec [ 5] 9.00-10.00 sec 1.65 MBytes 13.9 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.05 sec 11.6 MBytes 9.67 Mbits/sec 150 sender [ 5] 0.00-10.00 sec 11.2 MBytes 9.40 Mbits/sec receiver iperf Done. -
Did you disable NCP? Otherwise it will still be negotiating those.
But as I said this looks like some sort of mtu/fragmenting issue. Run a packet capture and see what's happening.
Steve
-
@stephenw10 said in Poor performace with Openvpn:
Did you disable NCP? Otherwise it will still be negotiating those.
But as I said this looks like some sort of mtu/fragmenting issue. Run a packet capture and see what's happening.
Steve
I have done packet capture. What should i be looking for in wireshark?
-
Fragmented packets is the first thing I would be looking for. Then what size the large fragments are.
Otherwise check the initial TCP transactions for the window size etc. Do you see retranmissions or missing packet errors.
Steve
-
@stephenw10 said in Poor performace with Openvpn:
Fragmented packets is the first thing I would be looking for. Then what size the large fragments are.
Otherwise check the initial TCP transactions for the window size etc. Do you see retranmissions or missing packet errors.
Steve
I captured VPN interface and opened the capture in wireshark. I dont see any fragmented packet in the Info column.
- Should I capture the WAN interface instead of the VPN interface?
- Is there any wireshark tutorial which shows how to identify whether fragmentation is occuring?
-
I'm not aware of any specific tutorial for that, there are many though.
Packet fragmentation is not difficult to spot though even just from the pfSense interface.
11:54:54.373954 IP 172.21.16.35 > 172.21.16.5: ICMP echo request, id 39018, seq 0, length 1480 11:54:54.373963 IP 172.21.16.35 > 172.21.16.5: ip-proto-1 11:54:55.374734 IP 172.21.16.35 > 172.21.16.5: ICMP echo request, id 39018, seq 1, length 1480 11:54:55.374743 IP 172.21.16.35 > 172.21.16.5: ip-proto-1 11:54:56.375739 IP 172.21.16.35 > 172.21.16.5: ICMP echo request, id 39018, seq 2, length 1480 11:54:56.375749 IP 172.21.16.35 > 172.21.16.5: ip-proto-1Pinging size 2000 byte packets, each full size packet is followed by a fragment to make up the full packet.
In that particular case there is a switch I have that doesn't pass packet fragments.
Try looking at the OpenVPN traffic on the WAN also. If that is fragmented it will kill performance.
Steve
-
For comparison sake i setup wireguard between the two networks. Wireguard starts with an MTU of 1420, and it reaches almost the line-speed in both forward and reverse directions.
# wg-quick up wg0 [#] ip link add wg0 type wireguard [#] wg setconf wg0 /dev/fd/63 [#] ip -4 address add 10.0.0.2/32 dev wg0 [#] ip link set mtu 1420 up dev wg0 [#] ip -4 route add 10.0.0.0/24 dev wg0root@wireguard:/etc/wireguard# iperf3 -c 10.0.0.1 Connecting to host 10.0.0.1, port 5201 [ 4] local 10.0.0.2 port 34842 connected to 10.0.0.1 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.00 sec 14.8 MBytes 124 Mbits/sec 0 1.74 MBytes [ 4] 1.00-2.00 sec 16.8 MBytes 141 Mbits/sec 0 2.58 MBytes [ 4] 2.00-3.00 sec 17.0 MBytes 142 Mbits/sec 3 2.96 MBytes [ 4] 3.00-4.00 sec 16.0 MBytes 134 Mbits/sec 1 2.07 MBytes [ 4] 4.00-5.00 sec 16.7 MBytes 140 Mbits/sec 1 1.52 MBytes [ 4] 5.00-6.00 sec 17.0 MBytes 142 Mbits/sec 0 1.62 MBytes [ 4] 6.00-7.00 sec 16.9 MBytes 142 Mbits/sec 0 1.70 MBytes [ 4] 7.00-8.00 sec 16.8 MBytes 141 Mbits/sec 0 1.75 MBytes [ 4] 8.00-9.00 sec 16.9 MBytes 142 Mbits/sec 0 1.79 MBytes [ 4] 9.00-10.00 sec 17.0 MBytes 142 Mbits/sec 2 920 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 166 MBytes 139 Mbits/sec 7 sender [ 4] 0.00-10.00 sec 163 MBytes 137 Mbits/sec receiver iperf Done. root@wireguard:/etc/wireguard# iperf3 -c 10.0.0.1 -R Connecting to host 10.0.0.1, port 5201 Reverse mode, remote host 10.0.0.1 is sending [ 4] local 10.0.0.2 port 35042 connected to 10.0.0.1 port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.00 sec 13.0 MBytes 109 Mbits/sec [ 4] 1.00-2.00 sec 16.9 MBytes 141 Mbits/sec [ 4] 2.00-3.00 sec 16.9 MBytes 142 Mbits/sec [ 4] 3.00-4.00 sec 16.9 MBytes 142 Mbits/sec [ 4] 4.00-5.00 sec 16.9 MBytes 142 Mbits/sec [ 4] 5.00-6.00 sec 16.9 MBytes 142 Mbits/sec [ 4] 6.00-7.00 sec 16.9 MBytes 142 Mbits/sec [ 4] 7.00-8.00 sec 16.9 MBytes 141 Mbits/sec [ 4] 8.00-9.00 sec 16.9 MBytes 142 Mbits/sec [ 4] 9.00-10.00 sec 16.9 MBytes 142 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 168 MBytes 141 Mbits/sec 3 sender [ 4] 0.00-10.00 sec 168 MBytes 141 Mbits/sec receiver
