Large data transfers stalling over VPN
-
Hi,
I hope someone can shed some light on this situation for me.
In summary: pfSense seems to break TCP connections over IPsec. The problem happens BOTH when using ipsec site-to-site configured in pfSense AND ALSO when using a VPN client on my Mac, with pfSense only doing NAT (VPN disabled in pfSense)
Details:
I want to connect to my off-site data centre. The data centre has a Cisco ASA 5505 in front of it, and I have used the wizard in that to configure both "Remote Access IPsec" and "Site-to-site" IPsec access.
The general problem appears to be when VPN traffic goes through pfSense, transfers stall. I will get an initial transfer of about 5.2MB, then it will pause for about 15 seconds, then another burst of several MB, then another pause, and that will repeat till the transfer finishes.
This happens with both HTTP and SCP transfers; I haven't tried other TCP protocols.
I end up getting a transfer rate of maybe 300KB/sec, whereas if I transfer not using VPN, I get 6 to 8 MB/sec (so ~ 10x as fast, with none of the stalling)
1. Remote access
–-----------------I can connect via remote access IPsec using my Mac, having created a new Cisco VPN service in the Mac's Network Control Panel. This appears to work fine if I connect from home, or from my office network port. However, if I plug the pfsense box into the office network port, and connect my laptop to pfSense, then I see the problem.
2. Site-to-site
I have also configured pfSense to connect to the Cisco box at the data centre as a site-to-site IPSec VPN. The connection looks to work fine at first, and I can ssh into machines in the data centre. However, if I scp a file across the VPN or try a wget via HTTP, (from the 192.168.. IP) I see the stalling. However, if I scp etc to the public IP of a server in the datacentre, I don't get the problem (this transfer is going via pfsense, but is being routed straight to the next hop, not going over the VPN)
So, it seems obvious to me that pfSense is doing something to VPN traffic.
I don't know if it's a buffer problem, ACKs or MTU.
I have been googling this like mad, and the standard answer internet answer looks like MTU, but I have tried fiddling with all of that and no luck.
Unfortunately, I have inconsistent results, which I'll try to relate as best I can here
I'm fairly sure when I started to mess about with this, the max data size I could ping from my mac to a ubuntu server in the data centre was ~ 1394. I'm pretty sure that when I turned on MSS Clamping (pfSense->IPsec->Advanced->Enable MSS clamping on VPN traffic), this limit disappeared. However, now I can always send up to 1472 even with clamping off, which I guess is hitting the ethernet interface MTU of 1500.
At no time did I fix the stalling issue.
If I ping the other way, from ubuntu at the data centre to my mac, it gives me something like this:
$ ping -D -s 1418 10.20.1.1
PING 10.20.1.1 (10.20.1.1) 1418(1446) bytes of data.
[1471876716.200621] From 192.168.10.254 icmp_seq=1 Frag needed and DF set (mtu = 1418)
[1471876717.208597] 1426 bytes from 10.20.1.1: icmp_seq=2 ttl=63 time=7.76 ms
[1471876718.210391] 1426 bytes from 10.20.1.1: icmp_seq=3 ttl=63 time=7.57 msDoes anyone have a clue where I should be looking to fix this?
-
MSS clamping is what you want, but ICMP can't test MSS clamping – MSS clamping only affects TCP traffic.
Set it to ~1300 and see if it works better. If it does you can start to increase it a bit until it breaks, then dial it back down.
-
Thanks for your suggestion, but dialing it down as low as 1280 doesn't help.
I set MSS to 1280 on the IPsec advanced settings, and tried restarting the service.
I also tried setting MSS on the LAN and WAN interfaces, with no luck.
I also tried changing the "Force maximum segment size for TCP connections" option on the Cisco ASA 5505. This defaults to 1380. I tried turning it off as well as dialing it down, but no luck.
What else could it be, and why would pfSense introduce the problem, when it's not evident if I connect directly to the network socket in the wall (which also goes on to some sort of NAT device)?
-
So,
I have just managed to try another experiment:
I booted from a clean embedded USB image of pfSense, set the WAN and LAN interfaces, and connected my Mac's IPsec client to the remote VPN endpoint.
As I understand it, pfSense is only doing NAT in this scenario.
I still get the same problem. I went on to try setting a low MSS on both interfaces, with no luck.
I then went on to try setting a 1:1 NAT from the WAN IP to the client IP. Still same problem.
-
Is there somewhere I can download an older version? Which version would be recommended, eg, when were there last any kind of significant changes which might affect this.
-
There were no changes that would have affected this as far back as I remember. We have a lot of people using the current version in both of those scenarios with success, so perhaps the problem lies elsewhere.
If you want to try older versions, you can find them here: https://atxfiles.pfsense.org/mirror/downloads/old/
-
Okay, thanks.
I have just tried with another machine and another extra network card: Previously I was using AMD64 pfSense on an older Core 2 Duo Dell machine with an onboard intel NIC + a double intel PCIe NIC (interfaces showing up as em0, em1, em2); and just now I tried a Dell Core i7 with onboard intel NIC + a realtek NIC and i386 pfsense.
Exact same problem. So I guess it's not the hardware.
I have managed to at least affect the symptoms of the problem: If I lower the Max Segment Size on the Cisco box I can lower the amount of data that is transferred before the stall. The Cisco box has it's MSS set to 1380 and this will transfer about 5.14MB before the stall. If I lower the MSS to 1000 then it stalls after about 3.7MB.
I can see in the packet capture that playing with the MSS on both the Cisco end and the pfSense end affects the size of the packets traversing it.
I do have an older version of ASA on the Cisco box: version 8.4.1. Perhaps this bug is relevant https://bst.cloudapps.cisco.com/bugsearch/bug/CSCtj50580
I can upgrade the Cisco box if needed, but I am reluctant to go over the the data centre and have site downtime on the off chance this is the problem, considering that I never have a problem when pfSense isn't in the mix.
-
I just tried running ipfire instead of pfSense and I get the same problem.
-
Now I just tried at home, over a cheap ADSL wireless router, and it's 100% fine.
-
I have just installed pfSense onto a virtualbox instance on my machine and am using it to NAT traffic to the internet. I can connect my VPN client to the remote end everything is fine, I don't get any stalling.
The traffic is going through pfSense at about 30Mbps, I can see it on the graphs.
Tomorrow I will try to connect my laptop to the office internet, and see if I can still route VPN traffic through the virtual box pfSense without the stalling.
-
I think you can forget about me, it looks like it's a problem with our network and not pfSense.
Sorry for wasting your time, I feel embarrassed for not working this out before manically posting here.
