Navigation

    Netgate Discussion Forum
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search

    vti dropped packets (mtu?)

    IPsec
    2
    9
    148
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C
      clarknova last edited by clarknova

      IPSEC + vti
      Local: EdgeRouter X v1.10.11
      Remote: pfSense 2.4.5-RELEASE-p1
      

      After some trial and error I have set the mtu on the vti device on both endpoints to 1400 and I can pass pings with the DF bit set and payload up to 1372 bytes in both directions across the tunnel.

      The problem is that for reasons I can't yet explain, I'm seeing packets leave pfSense's vti interface that I don't see arrive on the ERX's vti interface:

      pfsense:

      : tcpdump -niipsec1000 host x.y.z.86
      tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
      listening on ipsec1000, link-type NULL (BSD loopback), capture size 262144 bytes
      10:17:35.629287 IP 172.30.2.106.57572 > x.y.z.86.443: Flags [S], seq 1048637436, win 64240, options [mss 1460,sackOK,TS val 3679769371 ecr 0,nop,wscale 7], length 0
      10:17:35.629693 IP x.y.z.86.443 > 172.30.2.106.57572: Flags [S.], seq 3989331873, ack 1048637437, win 16384, options [mss 1460,nop,nop,sackOK,nop,wscale 6,nop,nop,TS val 572298750 ecr 3679769371], length 0
      10:17:35.637946 IP 172.30.2.106.57572 > x.y.z.86.443: Flags [.], ack 1, win 502, options [nop,nop,TS val 3679769380 ecr 572298750], length 0
      10:17:35.641598 IP 172.30.2.106.57572 > x.y.z.86.443: Flags [P.], seq 1:518, ack 1, win 502, options [nop,nop,TS val 3679769380 ecr 572298750], length 517
      10:17:35.644753 IP x.y.z.86.443 > 172.30.2.106.57572: Flags [.], seq 1:1345, ack 518, win 271, options [nop,nop,TS val 572298751 ecr 3679769380], length 1344
      10:17:35.644776 IP x.y.z.86.443 > 172.30.2.106.57572: Flags [P.], seq 1449:2793, ack 518, win 271, options [nop,nop,TS val 572298751 ecr 3679769380], length 1344
      10:17:35.850883 IP 172.30.2.106.57572 > x.y.z.86.443: Flags [P.], seq 1:518, ack 1, win 502, options [nop,nop,TS val 3679769590 ecr 572298750], length 517
      10:17:35.851646 IP x.y.z.86.443 > 172.30.2.106.57572: Flags [.], ack 518, win 271, options [nop,nop,TS val 572298751 ecr 3679769590], length 0
      10:17:38.667111 IP x.y.z.86.443 > 172.30.2.106.57572: Flags [.], seq 1:1345, ack 518, win 271, options [nop,nop,TS val 572298756 ecr 3679769590], length 1344
      10:17:44.725704 IP x.y.z.86.443 > 172.30.2.106.57572: Flags [.], seq 1:1345, ack 518, win 271, options [nop,nop,TS val 572298768 ecr 3679769590], length 1344
      ^C
      10 packets captured
      36 packets received by filter
      0 packets dropped by kernel
      

      ERX:

      # tcpdump -nivti0 host x.y.z.86    
      tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
      listening on vti0, link-type RAW (Raw IP), capture size 262144 bytes
      10:17:35.625493 IP 172.30.2.106.57572 > x.y.z.86.443: Flags [S], seq 1048637436, win 64240, options [mss 1460,sackOK,TS val 3679769371 ecr 0,nop,wscale 7], length 0
      10:17:35.633538 IP x.y.z.86.443 > 172.30.2.106.57572: Flags [S.], seq 3989331873, ack 1048637437, win 16384, options [mss 1460,nop,nop,sackOK,nop,wscale 6,nop,nop,TS val 572298750 ecr 3679769371], length 0
      10:17:35.634310 IP 172.30.2.106.57572 > x.y.z.86.443: Flags [.], ack 1, win 502, options [nop,nop,TS val 3679769380 ecr 572298750], length 0
      10:17:35.637753 IP 172.30.2.106.57572 > x.y.z.86.443: Flags [P.], seq 1:518, ack 1, win 502, options [nop,nop,TS val 3679769380 ecr 572298750], length 517
      10:17:35.846780 IP 172.30.2.106.57572 > x.y.z.86.443: Flags [P.], seq 1:518, ack 1, win 502, options [nop,nop,TS val 3679769590 ecr 572298750], length 517
      10:17:35.855055 IP x.y.z.86.443 > 172.30.2.106.57572: Flags [.], ack 518, win 271, options [nop,nop,TS val 572298751 ecr 3679769590], length 0
      ^C
      6 packets captured
      6 packets received by filter
      0 packets dropped by kernel
      

      It looks to me like larger ACK packets are disappearing in the tunnel, but as I've already verified that I can pass larger ICMP packets through the tunnel, I don't know how to explain this.

      C 1 Reply Last reply Reply Quote 0
      • C
        clarknova @clarknova last edited by clarknova

        Here's a similar dump showing larger echo requests passing fine.

        pfSense:

        : tcpdump -c2 -eniipsec1000 host x.y.z.86 and proto ICMP
        tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
        listening on ipsec1000, link-type NULL (BSD loopback), capture size 262144 bytes
        11:29:22.336860 AF IPv4 (2), length 1404: 172.30.2.106 > x.y.z.86: ICMP echo request, id 12126, seq 1, length 1380
        11:29:22.337497 AF IPv4 (2), length 1404: x.y.z.86 > 172.30.2.106: ICMP echo reply, id 12126, seq 1, length 1380
        2 packets captured
        6 packets received by filter
        0 packets dropped by kernel
        

        ERX:

        # tcpdump -c21 -enivti0 host x.y.z.86 and proto ICMP
        tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
        listening on vti0, link-type RAW (Raw IP), capture size 262144 bytes
        11:29:22.332098 ip: 172.30.2.106 > x.y.z.86: ICMP echo request, id 12126, seq 1, length 1380
        11:29:22.341427 ip: x.y.z.86 > 172.30.2.106: ICMP echo reply, id 12126, seq 1, length 1380
        ^C
        2 packets captured
        2 packets received by filter
        0 packets dropped by kernel
        
        1 Reply Last reply Reply Quote 0
        • C
          clarknova last edited by clarknova

          I had a second vti interface on pfSense that was inactive, so I deleted that vti and disabled the Phase 1 and 2 entries that were associated to it, and this resulted in my web site working for a while. tcpdump showed packets of length 1380 on both pfSense and the ERX's vti interfaces.

          Now some time has passed and I'm again seeing packets of length 1344 leaving the pfSense vti and not arriving on the ERX vti.

          C 1 Reply Last reply Reply Quote 0
          • C
            clarknova @clarknova last edited by

            I ran a packet capture on pfSense's WAN, filtering for ESP protocol. I'm not seeing any large packets here while trying to load the web site in question, and the WAN interface is not seeing any packets that the ERX is not seeing.This appears to be a case of pfSense's vti sending the packets, but they never get encapsulated and forwarded out the WAN. How can I troubleshoot this further?

            pfSense (showing only outbound ESP):

            13:08:50.536463 IP p.f.s.e.w > e.r.x.w: ESP(spi=0xc39680aa,seq=0x404), length 120
            13:08:50.756663 IP p.f.s.e.w > e.r.x.w: ESP(spi=0xc39680aa,seq=0x405), length 104
            13:08:58.743825 IP p.f.s.e.w > e.r.x.w: ESP(spi=0xc39680aa,seq=0x406), length 88
            

            ERX (showing only inbound ESP during the same period):

            13:08:50.539161 00:23:3e:6b:1f:ab > 18:e8:29:28:c8:84, ethertype IPv4 (0x0800), length 154: p.f.s.e.w > e.r.x.w: ESP(spi=0xc39680aa,seq=0x404), length 120
            13:08:50.759107 00:23:3e:6b:1f:ab > 18:e8:29:28:c8:84, ethertype IPv4 (0x0800), length 138: p.f.s.e.w > e.r.x.w: ESP(spi=0xc39680aa,seq=0x405), length 104
            13:08:58.748670 00:23:3e:6b:1f:ab > 18:e8:29:28:c8:84, ethertype IPv4 (0x0800), length 122: p.f.s.e.w > e.r.x.w: ESP(spi=0xc39680aa,seq=0x406), length 88
            
            1 Reply Last reply Reply Quote 0
            • C
              clarknova last edited by

              I tried the "Clear invalid DF bits" and "Disable scrub" options, but it made no difference.

              C 1 Reply Last reply Reply Quote 0
              • C
                clarknova @clarknova last edited by

                I set the mtu on pfsense's vti as low as it would let me, 1280, and likewise on the ERX. This resulted in smaller packets egressing on pfsense's vti, but those largest packets were still dropped by the parent WAN interface.

                Then I set both vti interfaces back to 1438 (I wrongly stated 1400 in my original post, which is why you see ICMP packets >1400 on some packet dumps) and reduced the mtu on the web server (nginx reverse proxy, to be precise) to 1438. This change made it possible to load web pages across the tunnel.

                I believe there is a problem within pfsense that it is fragmenting packets improperly on the vti interface such that they are not properly encapsulated and passed to the parent interface. I would appreciate if somebody could suggest ways that I could demonstrate this more specifically and convincingly before I open a bug report.

                M 1 Reply Last reply Reply Quote 0
                • M
                  metisit @clarknova last edited by

                  @clarknova https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242744 ?

                  C 1 Reply Last reply Reply Quote 0
                  • C
                    clarknova @metisit last edited by

                    @metisit said in vti dropped packets (mtu?):

                    https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=242744

                    Thanks for this. The description of symptoms appears to be a perfect match. I'll have to educate myself on transport mode, as the bug reporter seems to be saying that his bug wouldn't affect a vti with the mtu set, and in my case I have a vti and I didn't find any mtu value that would prevent the problem.

                    I will have to look into it further though, as the similarities are too much to ignore. I've also started testing wireguard, so it will be interesting to see how it deals with this situation.

                    M 1 Reply Last reply Reply Quote 0
                    • M
                      metisit @clarknova last edited by

                      @clarknova we saw this in opnsense as after switching to VTI. SMB stopped working over ipsec. hard to track down. it worked like half a year, then problems started and it was kind of reproducible. pings works, SMB browsing too. SMB access timeout. reverting to classic ipsec solved the issue. i assume we did tunnel VTI back then, although the freebsd bug mentions transport mode. maybe it hurts both modes. why i wonder how pfsense is affected by this open freebsd bug as well. i would like to use VTI soon...

                      https://github.com/opnsense/core/issues/3674

                      1 Reply Last reply Reply Quote 0
                      • First post
                        Last post

                      Products

                      • Platform Overview
                      • TNSR
                      • pfSense
                      • Appliances

                      Services

                      • Training
                      • Professional Services

                      Support

                      • Subscription Plans
                      • Contact Support
                      • Product Lifecycle
                      • Documentation

                      News

                      • Media Coverage
                      • Press
                      • Events

                      Resources

                      • Blog
                      • FAQ
                      • Find a Partner
                      • Resource Library
                      • Security Information

                      Company

                      • About Us
                      • Careers
                      • Partners
                      • Contact Us
                      • Legal
                      Our Mission

                      We provide leading-edge network security at a fair price - regardless of organizational size or network sophistication. We believe that an open-source security model offers disruptive pricing along with the agility required to quickly address emerging threats.

                      Subscribe to our Newsletter

                      Product information, software announcements, and special offers. See our newsletter archive to sign up for future newsletters and to read past announcements.

                      © 2021 Rubicon Communications, LLC | Privacy Policy