Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    vti dropped packets (mtu?)

    Scheduled Pinned Locked Moved IPsec
    9 Posts 2 Posters 655 Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C
      clarknova
      last edited by clarknova 

      IPSEC + vti
Local: EdgeRouter X v1.10.11
Remote: pfSense 2.4.5-RELEASE-p1

      After some trial and error I have set the mtu on the vti device on both endpoints to 1400 and I can pass pings with the DF bit set and payload up to 1372 bytes in both directions across the tunnel.

      The problem is that for reasons I can't yet explain, I'm seeing packets leave pfSense's vti interface that I don't see arrive on the ERX's vti interface:

      pfsense:

      : tcpdump -niipsec1000 host x.y.z.86
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ipsec1000, link-type NULL (BSD loopback), capture size 262144 bytes
10:17:35.629287 IP 172.30.2.106.57572 > x.y.z.86.443: Flags [S], seq 1048637436, win 64240, options [mss 1460,sackOK,TS val 3679769371 ecr 0,nop,wscale 7], length 0
10:17:35.629693 IP x.y.z.86.443 > 172.30.2.106.57572: Flags [S.], seq 3989331873, ack 1048637437, win 16384, options [mss 1460,nop,nop,sackOK,nop,wscale 6,nop,nop,TS val 572298750 ecr 3679769371], length 0
10:17:35.637946 IP 172.30.2.106.57572 > x.y.z.86.443: Flags [.], ack 1, win 502, options [nop,nop,TS val 3679769380 ecr 572298750], length 0
10:17:35.641598 IP 172.30.2.106.57572 > x.y.z.86.443: Flags [P.], seq 1:518, ack 1, win 502, options [nop,nop,TS val 3679769380 ecr 572298750], length 517
10:17:35.644753 IP x.y.z.86.443 > 172.30.2.106.57572: Flags [.], seq 1:1345, ack 518, win 271, options [nop,nop,TS val 572298751 ecr 3679769380], length 1344
10:17:35.644776 IP x.y.z.86.443 > 172.30.2.106.57572: Flags [P.], seq 1449:2793, ack 518, win 271, options [nop,nop,TS val 572298751 ecr 3679769380], length 1344
10:17:35.850883 IP 172.30.2.106.57572 > x.y.z.86.443: Flags [P.], seq 1:518, ack 1, win 502, options [nop,nop,TS val 3679769590 ecr 572298750], length 517
10:17:35.851646 IP x.y.z.86.443 > 172.30.2.106.57572: Flags [.], ack 518, win 271, options [nop,nop,TS val 572298751 ecr 3679769590], length 0
10:17:38.667111 IP x.y.z.86.443 > 172.30.2.106.57572: Flags [.], seq 1:1345, ack 518, win 271, options [nop,nop,TS val 572298756 ecr 3679769590], length 1344
10:17:44.725704 IP x.y.z.86.443 > 172.30.2.106.57572: Flags [.], seq 1:1345, ack 518, win 271, options [nop,nop,TS val 572298768 ecr 3679769590], length 1344
^C
10 packets captured
36 packets received by filter
0 packets dropped by kernel

      ERX:

      # tcpdump -nivti0 host x.y.z.86    
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vti0, link-type RAW (Raw IP), capture size 262144 bytes
10:17:35.625493 IP 172.30.2.106.57572 > x.y.z.86.443: Flags [S], seq 1048637436, win 64240, options [mss 1460,sackOK,TS val 3679769371 ecr 0,nop,wscale 7], length 0
10:17:35.633538 IP x.y.z.86.443 > 172.30.2.106.57572: Flags [S.], seq 3989331873, ack 1048637437, win 16384, options [mss 1460,nop,nop,sackOK,nop,wscale 6,nop,nop,TS val 572298750 ecr 3679769371], length 0
10:17:35.634310 IP 172.30.2.106.57572 > x.y.z.86.443: Flags [.], ack 1, win 502, options [nop,nop,TS val 3679769380 ecr 572298750], length 0
10:17:35.637753 IP 172.30.2.106.57572 > x.y.z.86.443: Flags [P.], seq 1:518, ack 1, win 502, options [nop,nop,TS val 3679769380 ecr 572298750], length 517
10:17:35.846780 IP 172.30.2.106.57572 > x.y.z.86.443: Flags [P.], seq 1:518, ack 1, win 502, options [nop,nop,TS val 3679769590 ecr 572298750], length 517
10:17:35.855055 IP x.y.z.86.443 > 172.30.2.106.57572: Flags [.], ack 518, win 271, options [nop,nop,TS val 572298751 ecr 3679769590], length 0
^C
6 packets captured
6 packets received by filter
0 packets dropped by kernel

      It looks to me like larger ACK packets are disappearing in the tunnel, but as I've already verified that I can pass larger ICMP packets through the tunnel, I don't know how to explain this.

      db

      C 1 Reply Last reply Reply Quote 0
      • C
        clarknova @clarknova
        last edited by clarknova

        Here's a similar dump showing larger echo requests passing fine.

        pfSense:

        : tcpdump -c2 -eniipsec1000 host x.y.z.86 and proto ICMP
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ipsec1000, link-type NULL (BSD loopback), capture size 262144 bytes
11:29:22.336860 AF IPv4 (2), length 1404: 172.30.2.106 > x.y.z.86: ICMP echo request, id 12126, seq 1, length 1380
11:29:22.337497 AF IPv4 (2), length 1404: x.y.z.86 > 172.30.2.106: ICMP echo reply, id 12126, seq 1, length 1380
2 packets captured
6 packets received by filter
0 packets dropped by kernel

        ERX:

        # tcpdump -c21 -enivti0 host x.y.z.86 and proto ICMP
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vti0, link-type RAW (Raw IP), capture size 262144 bytes
11:29:22.332098 ip: 172.30.2.106 > x.y.z.86: ICMP echo request, id 12126, seq 1, length 1380
11:29:22.341427 ip: x.y.z.86 > 172.30.2.106: ICMP echo reply, id 12126, seq 1, length 1380
^C
2 packets captured
2 packets received by filter
0 packets dropped by kernel

        db

        1 Reply Last reply Reply Quote 0
        • C
          clarknova
          last edited by clarknova

          I had a second vti interface on pfSense that was inactive, so I deleted that vti and disabled the Phase 1 and 2 entries that were associated to it, and this resulted in my web site working for a while. tcpdump showed packets of length 1380 on both pfSense and the ERX's vti interfaces.

          Now some time has passed and I'm again seeing packets of length 1344 leaving the pfSense vti and not arriving on the ERX vti.

          db

          C 1 Reply Last reply Reply Quote 0
          • C
            clarknova @clarknova
            last edited by

            I ran a packet capture on pfSense's WAN, filtering for ESP protocol. I'm not seeing any large packets here while trying to load the web site in question, and the WAN interface is not seeing any packets that the ERX is not seeing.This appears to be a case of pfSense's vti sending the packets, but they never get encapsulated and forwarded out the WAN. How can I troubleshoot this further?

            pfSense (showing only outbound ESP):

            13:08:50.536463 IP p.f.s.e.w > e.r.x.w: ESP(spi=0xc39680aa,seq=0x404), length 120
13:08:50.756663 IP p.f.s.e.w > e.r.x.w: ESP(spi=0xc39680aa,seq=0x405), length 104
13:08:58.743825 IP p.f.s.e.w > e.r.x.w: ESP(spi=0xc39680aa,seq=0x406), length 88

            ERX (showing only inbound ESP during the same period):

            13:08:50.539161 00:23:3e:6b:1f:ab > 18:e8:29:28:c8:84, ethertype IPv4 (0x0800), length 154: p.f.s.e.w > e.r.x.w: ESP(spi=0xc39680aa,seq=0x404), length 120
13:08:50.759107 00:23:3e:6b:1f:ab > 18:e8:29:28:c8:84, ethertype IPv4 (0x0800), length 138: p.f.s.e.w > e.r.x.w: ESP(spi=0xc39680aa,seq=0x405), length 104
13:08:58.748670 00:23:3e:6b:1f:ab > 18:e8:29:28:c8:84, ethertype IPv4 (0x0800), length 122: p.f.s.e.w > e.r.x.w: ESP(spi=0xc39680aa,seq=0x406), length 88

            db

            1 Reply Last reply Reply Quote 0
            • C
              clarknova
              last edited by

              I tried the "Clear invalid DF bits" and "Disable scrub" options, but it made no difference.

              db

              C 1 Reply Last reply Reply Quote 0
              • C
                clarknova @clarknova
                last edited by

                I set the mtu on pfsense's vti as low as it would let me, 1280, and likewise on the ERX. This resulted in smaller packets egressing on pfsense's vti, but those largest packets were still dropped by the parent WAN interface.

                Then I set both vti interfaces back to 1438 (I wrongly stated 1400 in my original post, which is why you see ICMP packets >1400 on some packet dumps) and reduced the mtu on the web server (nginx reverse proxy, to be precise) to 1438. This change made it possible to load web pages across the tunnel.

                I believe there is a problem within pfsense that it is fragmenting packets improperly on the vti interface such that they are not properly encapsulated and passed to the parent interface. I would appreciate if somebody could suggest ways that I could demonstrate this more specifically and convincingly before I open a bug report.

                db

                M 1 Reply Last reply Reply Quote 0
                • M
                  metisit @clarknova
                  last edited by

                  @clarknova https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242744 ?

                  C 1 Reply Last reply Reply Quote 0
                  • C
                    clarknova @metisit
                    last edited by

                    @metisit said in vti dropped packets (mtu?):

                    https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242744

                    Thanks for this. The description of symptoms appears to be a perfect match. I'll have to educate myself on transport mode, as the bug reporter seems to be saying that his bug wouldn't affect a vti with the mtu set, and in my case I have a vti and I didn't find any mtu value that would prevent the problem.

                    I will have to look into it further though, as the similarities are too much to ignore. I've also started testing wireguard, so it will be interesting to see how it deals with this situation.

                    db

                    M 1 Reply Last reply Reply Quote 0
                    • M
                      metisit @clarknova
                      last edited by

                      @clarknova we saw this in opnsense as after switching to VTI. SMB stopped working over ipsec. hard to track down. it worked like half a year, then problems started and it was kind of reproducible. pings works, SMB browsing too. SMB access timeout. reverting to classic ipsec solved the issue. i assume we did tunnel VTI back then, although the freebsd bug mentions transport mode. maybe it hurts both modes. why i wonder how pfsense is affected by this open freebsd bug as well. i would like to use VTI soon...

                      https://github.com/opnsense/core/issues/3674

                      1 Reply Last reply Reply Quote 0
                      • First post
                        Last post
                      Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.