vti dropped packets (mtu?)
-
IPSEC + vti Local: EdgeRouter X v1.10.11 Remote: pfSense 2.4.5-RELEASE-p1After some trial and error I have set the mtu on the vti device on both endpoints to 1400 and I can pass pings with the DF bit set and payload up to 1372 bytes in both directions across the tunnel.
The problem is that for reasons I can't yet explain, I'm seeing packets leave pfSense's vti interface that I don't see arrive on the ERX's vti interface:
pfsense:
: tcpdump -niipsec1000 host x.y.z.86 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ipsec1000, link-type NULL (BSD loopback), capture size 262144 bytes 10:17:35.629287 IP 172.30.2.106.57572 > x.y.z.86.443: Flags [S], seq 1048637436, win 64240, options [mss 1460,sackOK,TS val 3679769371 ecr 0,nop,wscale 7], length 0 10:17:35.629693 IP x.y.z.86.443 > 172.30.2.106.57572: Flags [S.], seq 3989331873, ack 1048637437, win 16384, options [mss 1460,nop,nop,sackOK,nop,wscale 6,nop,nop,TS val 572298750 ecr 3679769371], length 0 10:17:35.637946 IP 172.30.2.106.57572 > x.y.z.86.443: Flags [.], ack 1, win 502, options [nop,nop,TS val 3679769380 ecr 572298750], length 0 10:17:35.641598 IP 172.30.2.106.57572 > x.y.z.86.443: Flags [P.], seq 1:518, ack 1, win 502, options [nop,nop,TS val 3679769380 ecr 572298750], length 517 10:17:35.644753 IP x.y.z.86.443 > 172.30.2.106.57572: Flags [.], seq 1:1345, ack 518, win 271, options [nop,nop,TS val 572298751 ecr 3679769380], length 1344 10:17:35.644776 IP x.y.z.86.443 > 172.30.2.106.57572: Flags [P.], seq 1449:2793, ack 518, win 271, options [nop,nop,TS val 572298751 ecr 3679769380], length 1344 10:17:35.850883 IP 172.30.2.106.57572 > x.y.z.86.443: Flags [P.], seq 1:518, ack 1, win 502, options [nop,nop,TS val 3679769590 ecr 572298750], length 517 10:17:35.851646 IP x.y.z.86.443 > 172.30.2.106.57572: Flags [.], ack 518, win 271, options [nop,nop,TS val 572298751 ecr 3679769590], length 0 10:17:38.667111 IP x.y.z.86.443 > 172.30.2.106.57572: Flags [.], seq 1:1345, ack 518, win 271, options [nop,nop,TS val 572298756 ecr 3679769590], length 1344 10:17:44.725704 IP x.y.z.86.443 > 172.30.2.106.57572: Flags [.], seq 1:1345, ack 518, win 271, options [nop,nop,TS val 572298768 ecr 3679769590], length 1344 ^C 10 packets captured 36 packets received by filter 0 packets dropped by kernelERX:
# tcpdump -nivti0 host x.y.z.86 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on vti0, link-type RAW (Raw IP), capture size 262144 bytes 10:17:35.625493 IP 172.30.2.106.57572 > x.y.z.86.443: Flags [S], seq 1048637436, win 64240, options [mss 1460,sackOK,TS val 3679769371 ecr 0,nop,wscale 7], length 0 10:17:35.633538 IP x.y.z.86.443 > 172.30.2.106.57572: Flags [S.], seq 3989331873, ack 1048637437, win 16384, options [mss 1460,nop,nop,sackOK,nop,wscale 6,nop,nop,TS val 572298750 ecr 3679769371], length 0 10:17:35.634310 IP 172.30.2.106.57572 > x.y.z.86.443: Flags [.], ack 1, win 502, options [nop,nop,TS val 3679769380 ecr 572298750], length 0 10:17:35.637753 IP 172.30.2.106.57572 > x.y.z.86.443: Flags [P.], seq 1:518, ack 1, win 502, options [nop,nop,TS val 3679769380 ecr 572298750], length 517 10:17:35.846780 IP 172.30.2.106.57572 > x.y.z.86.443: Flags [P.], seq 1:518, ack 1, win 502, options [nop,nop,TS val 3679769590 ecr 572298750], length 517 10:17:35.855055 IP x.y.z.86.443 > 172.30.2.106.57572: Flags [.], ack 518, win 271, options [nop,nop,TS val 572298751 ecr 3679769590], length 0 ^C 6 packets captured 6 packets received by filter 0 packets dropped by kernelIt looks to me like larger ACK packets are disappearing in the tunnel, but as I've already verified that I can pass larger ICMP packets through the tunnel, I don't know how to explain this.
-
Here's a similar dump showing larger echo requests passing fine.
pfSense:
: tcpdump -c2 -eniipsec1000 host x.y.z.86 and proto ICMP tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ipsec1000, link-type NULL (BSD loopback), capture size 262144 bytes 11:29:22.336860 AF IPv4 (2), length 1404: 172.30.2.106 > x.y.z.86: ICMP echo request, id 12126, seq 1, length 1380 11:29:22.337497 AF IPv4 (2), length 1404: x.y.z.86 > 172.30.2.106: ICMP echo reply, id 12126, seq 1, length 1380 2 packets captured 6 packets received by filter 0 packets dropped by kernelERX:
# tcpdump -c21 -enivti0 host x.y.z.86 and proto ICMP tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on vti0, link-type RAW (Raw IP), capture size 262144 bytes 11:29:22.332098 ip: 172.30.2.106 > x.y.z.86: ICMP echo request, id 12126, seq 1, length 1380 11:29:22.341427 ip: x.y.z.86 > 172.30.2.106: ICMP echo reply, id 12126, seq 1, length 1380 ^C 2 packets captured 2 packets received by filter 0 packets dropped by kernel -
I had a second vti interface on pfSense that was inactive, so I deleted that vti and disabled the Phase 1 and 2 entries that were associated to it, and this resulted in my web site working for a while. tcpdump showed packets of length 1380 on both pfSense and the ERX's vti interfaces.
Now some time has passed and I'm again seeing packets of length 1344 leaving the pfSense vti and not arriving on the ERX vti.
-
I ran a packet capture on pfSense's WAN, filtering for ESP protocol. I'm not seeing any large packets here while trying to load the web site in question, and the WAN interface is not seeing any packets that the ERX is not seeing.This appears to be a case of pfSense's vti sending the packets, but they never get encapsulated and forwarded out the WAN. How can I troubleshoot this further?
pfSense (showing only outbound ESP):
13:08:50.536463 IP p.f.s.e.w > e.r.x.w: ESP(spi=0xc39680aa,seq=0x404), length 120 13:08:50.756663 IP p.f.s.e.w > e.r.x.w: ESP(spi=0xc39680aa,seq=0x405), length 104 13:08:58.743825 IP p.f.s.e.w > e.r.x.w: ESP(spi=0xc39680aa,seq=0x406), length 88ERX (showing only inbound ESP during the same period):
13:08:50.539161 00:23:3e:6b:1f:ab > 18:e8:29:28:c8:84, ethertype IPv4 (0x0800), length 154: p.f.s.e.w > e.r.x.w: ESP(spi=0xc39680aa,seq=0x404), length 120 13:08:50.759107 00:23:3e:6b:1f:ab > 18:e8:29:28:c8:84, ethertype IPv4 (0x0800), length 138: p.f.s.e.w > e.r.x.w: ESP(spi=0xc39680aa,seq=0x405), length 104 13:08:58.748670 00:23:3e:6b:1f:ab > 18:e8:29:28:c8:84, ethertype IPv4 (0x0800), length 122: p.f.s.e.w > e.r.x.w: ESP(spi=0xc39680aa,seq=0x406), length 88 -
I tried the "Clear invalid DF bits" and "Disable scrub" options, but it made no difference.
-
I set the mtu on pfsense's vti as low as it would let me, 1280, and likewise on the ERX. This resulted in smaller packets egressing on pfsense's vti, but those largest packets were still dropped by the parent WAN interface.
Then I set both vti interfaces back to 1438 (I wrongly stated 1400 in my original post, which is why you see ICMP packets >1400 on some packet dumps) and reduced the mtu on the web server (nginx reverse proxy, to be precise) to 1438. This change made it possible to load web pages across the tunnel.
I believe there is a problem within pfsense that it is fragmenting packets improperly on the vti interface such that they are not properly encapsulated and passed to the parent interface. I would appreciate if somebody could suggest ways that I could demonstrate this more specifically and convincingly before I open a bug report.
-
@clarknova https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242744 ?
-
@metisit said in vti dropped packets (mtu?):
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242744
Thanks for this. The description of symptoms appears to be a perfect match. I'll have to educate myself on transport mode, as the bug reporter seems to be saying that his bug wouldn't affect a vti with the mtu set, and in my case I have a vti and I didn't find any mtu value that would prevent the problem.
I will have to look into it further though, as the similarities are too much to ignore. I've also started testing wireguard, so it will be interesting to see how it deals with this situation.
-
@clarknova we saw this in opnsense as after switching to VTI. SMB stopped working over ipsec. hard to track down. it worked like half a year, then problems started and it was kind of reproducible. pings works, SMB browsing too. SMB access timeout. reverting to classic ipsec solved the issue. i assume we did tunnel VTI back then, although the freebsd bug mentions transport mode. maybe it hurts both modes. why i wonder how pfsense is affected by this open freebsd bug as well. i would like to use VTI soon...
https://github.com/opnsense/core/issues/3674