Delay related performance issues with OpenVPN

[ebo]

Hello,

I have a Pfsense 2.2.4 running over an ESXI 6.0.0 located in France.
I have a Pfsense 2.2.4 running over an ESXI 6.0.0 located in Montreal.

I have 2 types of OpenVPN Client.

• One which is another Pfsense on an APU (2.2.4).
• One which is my Windows 10 laptop.

Which I can connect alternatively to either servers.

My VPN is configured in UDP, works fine and is very stable but I experience performance issue (delay related) when I try to upload a large file from a server behind either pfsense through the OpenVPN tunnel.

I have done Upload and Download testing without the VPN and I have 16 Mbps throughput in both direction (limited by my internet connection)
Client <===Internet===>Pfsense (VPN termination)<==>Server

But using OpenVPN, I have still nearly 16Mbps Download, But the Upload drop drasticallly. 
Client <===Internet===>Pfsense (NAT PortForwarding)<===>Server

File Upload to Montreal (30ms RTD) : 3,2 Mbps
File Upload to France (140ms RTD) : 560 KBps (!)
I have asked someone to make a test from China to the Montreal's server (450ms RTD) : 150 KBps (!)

I have made packet capture to try spotting the problem.
When I capture packets on the OpenVPN interface, I can see that there is some packet drop and retransmission (which explains why it's delay related).
However the packet drop profile is very strange, very even and regular. I loose one packet very regularly (every 40-50 packets to France).
You can see below a list of the missing packets in my trace :

Packet ID         Time         Packet count Time difference
64 0.96
112 1.78         48 0.82
160 2.61         48 0.83
226 3.71         66 1.1
286 4.67         60 0.96
334 5.5         48 0.83
384 6.32         50 0.82
424 7.02         40 0.7
472 7.84         48 0.82
511 8.53         39 0.69
558 9.35         47 0.82
615 10.18        57 0.83
666 11.14 51 0.96
716 11.97 50 0.83
770 12.93 54 0.96
810 13.62 40 0.69
858 14.44 48 0.82
887 14.99 29 0.55
936 15.82 49 0.83

This and the fact that I have good performance in all other configuration let me think that the packet loss is due to a problem in my servers configuration.

I have tryed different things to clear this up, with no result at all.

• Changing the network interface from vmx3 to E1000.
• Trying to set net.inet.ip.fastforwarding = 1 (https://forum.pfsense.org/index.php/topic,47567.0.html)
• Disable hardware checksum offload (https://doc.pfsense.org/index.php/Lost_Traffic_/_Packets_Disappear).

Any idea ?

Thank you very much,

Ebo

[ebo]

Hello again,

It seems that my problem doesn't generate a lot of ideas… :P

For information, I have solved my problem by putting traffic limiter in place (applied to my openvpn rule).
By limiting my upload traffic to 5 Mbps (Optimal setting for me according to my tests), it works much better.

With no traffic limiter :
Upload to montreal : 3,2 Mbps
Upload to France : 560 Kbps

With a traffic limiter at 5 Mbps.
Upload to Montreal : 4,7 Mbps (stable)
Upload to France : 4,4 Mbps (peak)

Hope it can help,

Ebo

[bitboy0]

do you use outbound-NAT ?
do you have more than one openVPN-Server running on that pfsense?

if yes, look here:
https://forum.pfsense.org/index.php?topic=101115.0