Routed (VTI) IPsec Tunnel troubleshooting, no or slow traffic
-
I hope someone can help me out here, I don’t know what else to check.
I have set up a routed (VTI) IPsec tunnel between to offices and it seems to work fine. Both ends can ping devices on the other network, login to hosts with ssh, but any actual traffic is super slow and often it is impossible to establish a connection at all. And I often see a couple of % package loss on the VTI gateways. So I cannot effectively use any services on the remote networks (HTTP, HTTPS, AFP, SMB, SCP). Copying even a file of just a few kb to a fileserver takes forever or does not work at all.
Firewall rules on both sides are set to allow all traffic between both LANs and the IPsec tunnel for now (IPv4 * from * to *) while I am still testing things.
I get a ping payload cutoff at 1380 bytes through the tunnel, as I have enabled „MMS clamping on VPN traffic“ (default 1400) in IPsec - Advanced Settings.
As suggested by Jim in https://forum.netgate.com/topic/135977/after-a-reboot-i-no-longer-have-the-road-to-ipsec-vti I have deleted the static routes and the VTI gateways, restarted the router, set up the static routes again but that did not help.
I still get the packet losses and an scp transfer only worked briefly after the restarts and never finished (broken pipe). Sometimes that rate jumps above 10% out of nowhere.
I would appreciate any idea on where else to look.
Cheers
pfSenpai -
@pfsenpai
Hey- Set the value of the Maximum MSS =1360 (/vpn/ipsec/advanced settings)
If it doesn't help - Show the output of the command ifconfig -m
- Upload pcap file here when there are problems
/diagnostics/packet capture /
interface VTI
protocol any
start/stop -> Download Capture
- Set the value of the Maximum MSS =1360 (/vpn/ipsec/advanced settings)
-
Hey,
first of all thank you for your reply.
I was suspecting an MTU issue and have been playing with MTU and MMS setting for a while now, only to realise that some of my changes did not seem to stick without a reboot of my routers.
So... I think I am getting there now that I have set Maximum MMS = 1360 for IPsec as you suggested (I had that at 1380) and I am finally able to copy files.
I also turned off Asynchronous Cryptography as this was suggested elsewhere, and it turns out that enabling it again on my Netgate SG-3100 (that I use at my smaller site No 2) is a bad idea as it breaks transfers almost entirely, again. Site 1 is a Netgate SG-8860 btw.
An ifconfig ipsec1000 shows an MTU of 1400 for the VTIs now (although I had set the MTU to 1480 in the VTI settings before, but I reverted this back to the default 1500 <left blank> now and rebooted).
But…
a) I only get 100 Mbps speeds on a 1000 Mbps line now (in both directions),
b) during a transfer from site 2 to a server on site 1 the packet loss rate goes up to above 50% on both routers, when transferring back from site 1 to site 2 packet loss basically stays at 0% (with the same pattern when accessing a server on site 2 from site 1, e.g. high packet loss only in the direction site 2 to site 1).
Attached are pcaps on the VTI interfaces on both sites during the transfer from and to an afp fileserver, filtered by the hosts address on site 2.
Again, thank you for looking into this.
.
No packet loss:
0_1550943499923_PCAP-site1--afp-copy-from-site1-to-site2.cap.zip
0_1550943507119_PCAP-site2--afp-copy-from-site1-to-site2.cap.zipNasty packet loss:
0_1550943467158_PCAP-site1--afp-copy-from-site2-to-site1.cap.zip
0_1550943492439_PCAP-site2--afp-copy-from-site2-to-site1.cap.zip -
@pfsenpai
Hmm.
Are you sure that PFSEnse is the problem?
I don't know much about the AFP Protocol , but I see a lot of requests and responses that the object was not found . -
@pfsenpai
There may be a problem with the device 24.13 -
Not perfectly sure, but I get the same when copying the file with scp to a Linux host at site 1.
0_1550946093228_PCAP-site2--scp-to-linux-site1.cap.zip 0_1550946089002_PCAP-site1--scp-to-linux-site1.cap.zip
Edit: I will try another devices next week (24.13. is the only one up right now).
What exactly is the difference in the PCAPs with the two transfer directions?Edit 2: I get transfer rates 2-3 times as fast when I connect to site 2 via OpenVPN, mount an afp server volume shared on the 24.13 machine on my desktop and copy files to an from it.
-
Much information. Need time
For example, in PCAP-site2--afp-copy-from-site2-to-site1.pcap I don't see any tcp anomalies
File BigOleTestFile-created and transferred from 24.13 to 10.11
In PCAP-site2--scp-to-linux-site1.cap already there are anomalies, but you've got to watch carefully what happens.
http://blog.manton.im/2016/08/tcp-retransmission-tcp-dup-ack.htmlHost 24.13-what operating system ?
-
Both 10.11 and 24.13 are Mac OS X 10.11.6 servers, Linux machine is Ubuntu 16.04.6 LTS (GNU/Linux 4.4.0-142-generic x86_64).
pfSense routers are these
site 1:
Netgate SG-8860
pfSense 2.4.4-RELEASE-p2 (amd64)
Hardware Crypto: AES-CBC,AES-XTS,AES-GCM,AES-ICM
Cryptographic Hardware: AES-NIsite 2:
Netgate SG-3100
pfSense 2.4.4-RELEASE-p2 (arm)
Crypto: Marvell Cryptographic Engine and Security Accelerator
Cryptographic Hardware: AES-NI and BSD Crypto DeviceI was considering setting up a site-to-site tunnel with OpenVPN for comparison (I have OpenVPN set up on both sites for client machines to connect while on the road and it works just fine), but I will not be able to get to this before next week.
Again, thank you!
-
FYI / update: I was able to run same test by now and copied a 200 MB file from a different machine on the 192.168.24.x side (site 2) to the FreeNAS box on the 192.168.10.x side (site 1) and got the same results, so I think I can rule out that the issue is with the 24.13 machine.
-
So... I replaced the SG-3100 with an XG-7100 today, setting up that side (site2) from scratch, and it now works as expected. IPsec tunnel speed is decent and no more packet loss. I can't see that I did anything differently, but I don't really have the time to look into it right now, if ever.
Anyhow, thanks, again, for your time and input.