Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login
    Introducing Netgate Nexus: Multi-Instance Management at Your Fingertips.

    TCP traffic over IPSec stalls with some packets not appearing on enc0

    Scheduled Pinned Locked Moved IPsec
    21 Posts 2 Posters 361 Views 2 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P Offline
      Pentium100
      last edited by

      Hello,

      I have this very weird problem.

      Configuration:
      IPSec from pfsense to some other device (Cisco or Fortinet probably). Only pfsense is in my control.
      IPSec Phase2 mode "routed (VTI)".
      firewall rules for ipsec and LAN tabs are "IPv4 any allow".
      There is no traffic shaping configured on pfsense.

      pfsense version 2.8.1.
      A lot of connections work OK, but I have found a way to reproduce my problem with iperf3. There are other IPSec tunnels on this router, all of them seem to work fine.

      The setup is like this:
      server1 (192.168.10.x) - pfsense - ipsec tunnel - server2 (192.168.0.x).
      On server2 I launch iperf3 -s
      On server1 I launch iperf3 -c 192.168.0.x

      This is what I get.

      Connecting to host 192.168.0.x, port 5201
[  5] local 192.168.10.x port 57498 connected to 192.168.0.132 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   253 KBytes  2.07 Mbits/sec    3   1.32 KBytes       
[  5]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec    1   1.32 KBytes       
[  5]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec    0   1.32 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-3.00   sec   253 KBytes   690 Kbits/sec    4             sender
[  5]   0.00-3.01   sec  0.00 Bytes  0.00 bits/sec                  receiver

      If I test the reverse (using -R option), it works fine.

      I tried capturing packets on interface vtnet2 (connected to server1) and enc0. It seems that packets get dropped by pfsense for some reason.

      This is how it looks on vtnet2 (sorry for the long paste):

      15:53:22.656999 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [P.], seq 63394:64742, ack 1, win 510, options [nop,nop,TS val 1313200811 ecr 2971974], length 1348
15:53:22.659235 IP 192.168.0.x.5201 > 192.168.10.x.43696: Flags [.], ack 31042, win 258, options [nop,nop,TS val 2971974 ecr 1313200806], length 0
15:53:22.659516 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 64742:66090, ack 1, win 510, options [nop,nop,TS val 1313200814 ecr 2971974], length 1348
15:53:22.659582 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 66090:67438, ack 1, win 510, options [nop,nop,TS val 1313200814 ecr 2971974], length 1348
15:53:22.659587 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 67438:68786, ack 1, win 510, options [nop,nop,TS val 1313200814 ecr 2971974], length 1348
15:53:22.659591 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 68786:70134, ack 1, win 510, options [nop,nop,TS val 1313200814 ecr 2971974], length 1348
15:53:22.659594 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 70134:71482, ack 1, win 510, options [nop,nop,TS val 1313200814 ecr 2971974], length 1348
15:53:22.659598 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 71482:72830, ack 1, win 510, options [nop,nop,TS val 1313200814 ecr 2971974], length 1348
15:53:22.659623 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 72830:74178, ack 1, win 510, options [nop,nop,TS val 1313200814 ecr 2971974], length 1348
15:53:22.659627 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 74178:75526, ack 1, win 510, options [nop,nop,TS val 1313200814 ecr 2971974], length 1348
15:53:22.659630 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [P.], seq 75526:76874, ack 1, win 510, options [nop,nop,TS val 1313200814 ecr 2971974], length 1348
15:53:22.659860 IP 192.168.0.x.5201 > 192.168.10.x.43696: Flags [.], ack 35086, win 258, options [nop,nop,TS val 2971974 ecr 1313200810], length 0
15:53:22.660210 IP 192.168.0.x.5201 > 192.168.10.x.43696: Flags [.], ack 39130, win 258, options [nop,nop,TS val 2971974 ecr 1313200810], length 0
15:53:22.660413 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 76874:78222, ack 1, win 510, options [nop,nop,TS val 1313200815 ecr 2971974], length 1348
15:53:22.660495 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 78222:79570, ack 1, win 510, options [nop,nop,TS val 1313200815 ecr 2971974], length 1348
15:53:22.660521 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 79570:80918, ack 1, win 510, options [nop,nop,TS val 1313200815 ecr 2971974], length 1348
15:53:22.660528 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 80918:82266, ack 1, win 510, options [nop,nop,TS val 1313200815 ecr 2971974], length 1348
15:53:22.660536 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 82266:83614, ack 1, win 510, options [nop,nop,TS val 1313200815 ecr 2971974], length 1348
15:53:22.660541 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 83614:84962, ack 1, win 510, options [nop,nop,TS val 1313200815 ecr 2971974], length 1348
15:53:22.660547 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 84962:86310, ack 1, win 510, options [nop,nop,TS val 1313200815 ecr 2971974], length 1348
15:53:22.660552 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 86310:87658, ack 1, win 510, options [nop,nop,TS val 1313200815 ecr 2971974], length 1348
15:53:22.660558 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 87658:89006, ack 1, win 510, options [nop,nop,TS val 1313200815 ecr 2971974], length 1348
15:53:22.660564 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 89006:90354, ack 1, win 510, options [nop,nop,TS val 1313200815 ecr 2971974], length 1348
15:53:22.660570 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 90354:91702, ack 1, win 510, options [nop,nop,TS val 1313200815 ecr 2971974], length 1348
15:53:22.660575 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [P.], seq 91702:93050, ack 1, win 510, options [nop,nop,TS val 1313200815 ecr 2971974], length 1348
15:53:22.660609 IP 192.168.0.x.5201 > 192.168.10.x.43696: Flags [.], ack 43174, win 258, options [nop,nop,TS val 2971974 ecr 1313200810], length 0
15:53:22.660760 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 93050:94398, ack 1, win 510, options [nop,nop,TS val 1313200816 ecr 2971974], length 1348
15:53:22.660805 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 94398:95746, ack 1, win 510, options [nop,nop,TS val 1313200816 ecr 2971974], length 1348
15:53:22.660809 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [P.], seq 95746:97094, ack 1, win 510, options [nop,nop,TS val 1313200816 ecr 2971974], length 1348
15:53:22.660850 IP 192.168.0.x.5201 > 192.168.10.x.43696: Flags [.], ack 47218, win 258, options [nop,nop,TS val 2971974 ecr 1313200810], length 0
15:53:22.661182 IP 192.168.0.x.5201 > 192.168.10.x.43696: Flags [.], ack 51262, win 258, options [nop,nop,TS val 2971974 ecr 1313200810], length 0
15:53:22.661577 IP 192.168.0.x.5201 > 192.168.10.x.43696: Flags [.], ack 55306, win 258, options [nop,nop,TS val 2971975 ecr 1313200811], length 0
15:53:22.661789 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 97094:98442, ack 1, win 510, options [nop,nop,TS val 1313200817 ecr 2971975], length 1348
15:53:22.661842 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 98442:99790, ack 1, win 510, options [nop,nop,TS val 1313200817 ecr 2971975], length 1348
15:53:22.661846 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 99790:101138, ack 1, win 510, options [nop,nop,TS val 1313200817 ecr 2971975], length 1348
15:53:22.661850 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 101138:102486, ack 1, win 510, options [nop,nop,TS val 1313200817 ecr 2971975], length 1348
15:53:22.661853 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 102486:103834, ack 1, win 510, options [nop,nop,TS val 1313200817 ecr 2971975], length 1348
15:53:22.661869 IP 192.168.0.x.5201 > 192.168.10.x.43696: Flags [.], ack 59350, win 258, options [nop,nop,TS val 2971975 ecr 1313200811], length 0
15:53:22.661883 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 103834:105182, ack 1, win 510, options [nop,nop,TS val 1313200817 ecr 2971975], length 1348
15:53:22.661886 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 105182:106530, ack 1, win 510, options [nop,nop,TS val 1313200817 ecr 2971975], length 1348
15:53:22.661890 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 106530:107878, ack 1, win 510, options [nop,nop,TS val 1313200817 ecr 2971975], length 1348
15:53:22.661893 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 107878:109226, ack 1, win 510, options [nop,nop,TS val 1313200817 ecr 2971975], length 1348
15:53:22.661937 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 109226:110574, ack 1, win 510, options [nop,nop,TS val 1313200817 ecr 2971975], length 1348
15:53:22.661940 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 110574:111922, ack 1, win 510, options [nop,nop,TS val 1313200817 ecr 2971975], length 1348
15:53:22.661943 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 111922:113270, ack 1, win 510, options [nop,nop,TS val 1313200817 ecr 2971975], length 1348
15:53:22.661961 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 113270:114618, ack 1, win 510, options [nop,nop,TS val 1313200817 ecr 2971975], length 1348
15:53:22.661964 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 114618:115966, ack 1, win 510, options [nop,nop,TS val 1313200817 ecr 2971975], length 1348
15:53:22.661968 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 115966:117314, ack 1, win 510, options [nop,nop,TS val 1313200817 ecr 2971975], length 1348
15:53:22.661982 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 117314:118662, ack 1, win 510, options [nop,nop,TS val 1313200817 ecr 2971975], length 1348
15:53:22.661986 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [P.], seq 118662:120010, ack 1, win 510, options [nop,nop,TS val 1313200817 ecr 2971975], length 1348
15:53:22.662183 IP 192.168.0.x.5201 > 192.168.10.x.43696: Flags [.], ack 62046, win 258, options [nop,nop,TS val 2971975 ecr 1313200811], length 0
15:53:22.662584 IP 192.168.0.x.5201 > 192.168.10.x.43696: Flags [.], ack 64742, win 258, options [nop,nop,TS val 2971975 ecr 1313200811], length 0
15:53:22.662782 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 120010:121358, ack 1, win 510, options [nop,nop,TS val 1313200818 ecr 2971975], length 1348
15:53:22.662852 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 121358:122706, ack 1, win 510, options [nop,nop,TS val 1313200818 ecr 2971975], length 1348
15:53:22.662867 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 122706:124054, ack 1, win 510, options [nop,nop,TS val 1313200818 ecr 2971975], length 1348
15:53:22.662873 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 124054:125402, ack 1, win 510, options [nop,nop,TS val 1313200818 ecr 2971975], length 1348
15:53:22.662880 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 125402:126750, ack 1, win 510, options [nop,nop,TS val 1313200818 ecr 2971975], length 1348
15:53:22.662886 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 126750:128098, ack 1, win 510, options [nop,nop,TS val 1313200818 ecr 2971975], length 1348
15:53:22.662893 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [P.], seq 128098:129446, ack 1, win 510, options [nop,nop,TS val 1313200818 ecr 2971975], length 1348
15:53:22.680760 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 128098:129446, ack 1, win 510, options [nop,nop,TS val 1313200836 ecr 2971975], length 1348
15:53:22.688752 IP 192.168.10.x.43680 > 192.168.0.x.5201: Flags [.], ack 5, win 510, options [nop,nop,TS val 1313200844 ecr 2971973], length 0
15:53:22.862262 IP 192.168.0.x.5201 > 192.168.10.x.43696: Flags [.], ack 66090, win 258, options [nop,nop,TS val 2971995 ecr 1313200814], length 0
15:53:22.862776 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 129446:130794, ack 1, win 510, options [nop,nop,TS val 1313201017 ecr 2971995], length 1348
15:53:22.862783 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [P.], seq 130794:132138, ack 1, win 510, options [nop,nop,TS val 1313201017 ecr 2971995], length 1344
15:53:22.904941 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 66090:67438, ack 1, win 510, options [nop,nop,TS val 1313201060 ecr 2971995], length 1348
15:53:23.376759 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 66090:67438, ack 1, win 510, options [nop,nop,TS val 1313201532 ecr 2971995], length 1348
15:53:24.340813 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 66090:67438, ack 1, win 510, options [nop,nop,TS val 1313202496 ecr 2971995], length 1348
15:53:26.224860 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 66090:67438, ack 1, win 510, options [nop,nop,TS val 1313204380 ecr 2971995], length 1348
15:53:30.128844 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 66090:67438, ack 1, win 510, options [nop,nop,TS val 1313208284 ecr 2971995], length 1348
15:53:32.646444 IP 192.168.10.x.43680 > 192.168.0.x.5201: Flags [P.], seq 166:167, ack 5, win 510, options [nop,nop,TS val 1313210801 ecr 2971973], length 1
15:53:32.651453 IP 192.168.0.x.5201 > 192.168.10.x.43696: Flags [F.], seq 1, ack 66090, win 258, options [nop,nop,TS val 2972973 ecr 1313200814], length 0
15:53:32.652741 IP 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], ack 2, win 510, options [nop,nop,TS val 1313210808 ecr 2972973], length 0

      And this is how it looks on enc0

      15:53:22.657002 (authentic,confidential): SPI 0x9bbda156: 192.168.10.x.43696 > 192.168.0.x.5201: Flags [P.], seq 63394:64742, ack 1, win 510, options [nop,nop,TS val 1313200811 ecr 2971974], length 1348
15:53:22.659218 (authentic,confidential): SPI 0xc3b00dd6: 192.168.0.x.5201 > 192.168.10.x.43696: Flags [.], ack 31042, win 258, options [nop,nop,TS val 2971974 ecr 1313200806], length 0
15:53:22.659528 (authentic,confidential): SPI 0x9bbda156: 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], seq 64742:66090, ack 1, win 510, options [nop,nop,TS val 1313200814 ecr 2971974], length 1348
15:53:22.659851 (authentic,confidential): SPI 0xc3b00dd6: 192.168.0.x.5201 > 192.168.10.x.43696: Flags [.], ack 35086, win 258, options [nop,nop,TS val 2971974 ecr 1313200810], length 0
15:53:22.660204 (authentic,confidential): SPI 0xc3b00dd6: 192.168.0.x.5201 > 192.168.10.x.43696: Flags [.], ack 39130, win 258, options [nop,nop,TS val 2971974 ecr 1313200810], length 0
15:53:22.660596 (authentic,confidential): SPI 0xc3b00dd6: 192.168.0.x.5201 > 192.168.10.x.43696: Flags [.], ack 43174, win 258, options [nop,nop,TS val 2971974 ecr 1313200810], length 0
15:53:22.660844 (authentic,confidential): SPI 0xc3b00dd6: 192.168.0.x.5201 > 192.168.10.x.43696: Flags [.], ack 47218, win 258, options [nop,nop,TS val 2971974 ecr 1313200810], length 0
15:53:22.661177 (authentic,confidential): SPI 0xc3b00dd6: 192.168.0.x.5201 > 192.168.10.x.43696: Flags [.], ack 51262, win 258, options [nop,nop,TS val 2971974 ecr 1313200810], length 0
15:53:22.661562 (authentic,confidential): SPI 0xc3b00dd6: 192.168.0.x.5201 > 192.168.10.x.43696: Flags [.], ack 55306, win 258, options [nop,nop,TS val 2971975 ecr 1313200811], length 0
15:53:22.661863 (authentic,confidential): SPI 0xc3b00dd6: 192.168.0.x.5201 > 192.168.10.x.43696: Flags [.], ack 59350, win 258, options [nop,nop,TS val 2971975 ecr 1313200811], length 0
15:53:22.662177 (authentic,confidential): SPI 0xc3b00dd6: 192.168.0.x.5201 > 192.168.10.x.43696: Flags [.], ack 62046, win 258, options [nop,nop,TS val 2971975 ecr 1313200811], length 0
15:53:22.662569 (authentic,confidential): SPI 0xc3b00dd6: 192.168.0.x.5201 > 192.168.10.x.43696: Flags [.], ack 64742, win 258, options [nop,nop,TS val 2971975 ecr 1313200811], length 0
15:53:22.688775 (authentic,confidential): SPI 0x9bbda156: 192.168.10.x.43680 > 192.168.0.x.5201: Flags [.], ack 5, win 510, options [nop,nop,TS val 1313200844 ecr 2971973], length 0
15:53:22.862240 (authentic,confidential): SPI 0xc3b00dd6: 192.168.0.x.5201 > 192.168.10.x.43696: Flags [.], ack 66090, win 258, options [nop,nop,TS val 2971995 ecr 1313200814], length 0
15:53:32.646590 (authentic,confidential): SPI 0x9bbda156: 192.168.10.x.43680 > 192.168.0.x.5201: Flags [P.], seq 166:167, ack 5, win 510, options [nop,nop,TS val 1313210801 ecr 2971973], length 1
15:53:32.651440 (authentic,confidential): SPI 0xc3b00dd6: 192.168.0.x.5201 > 192.168.10.x.43696: Flags [F.], seq 1, ack 66090, win 258, options [nop,nop,TS val 2972973 ecr 1313200814], length 0
15:53:32.652750 (authentic,confidential): SPI 0x9bbda156: 192.168.10.x.43696 > 192.168.0.x.5201: Flags [.], ack 2, win 510, options [nop,nop,TS val 1313210808 ecr 2972973], length 0

      For some reason packets received between 15:53:22.862240 and 15:53:32.646590 are blocked by pfsense and are not sent to enc0.

      If I start pinging server2 the ping goes through, but the TCP connection stalls anyway.
      I have tried reducing MSS (iperf3 -M ), but it did not help,

      If I use UDP packets it works OK, as does sending data in the opposite direction - from server2 to server1.

      Have you encountered anything similar? How can I fix this problem? I tried rebooting pfsense, it did not help.

      1 Reply Last reply Reply Quote 0
      • tinfoilmattT Offline
        tinfoilmatt LAYER 8
        last edited by

        Could be an MTU issue somewhere between "vtnet2" and "enc0". What's MTU set to for 1.) "vtnet2"; 2.) whichever pfSense LAN interface "vtnet2" attaches to; and 3.) "enc0"?

        P 1 Reply Last reply Reply Quote 0
        • P Offline
          Pentium100 @tinfoilmatt
          last edited by

          @tinfoilmatt vtnet2 is 1500 MTU, the same goes for the pfsense interface attached to it. enc0 (also ipsec30) MTU is 1400.

          I first suspected this too, but as you can see from the capture, all packets are the same length 1348 and they pass through initially, but then stop.

          Also, running iperf3 -c 192.168.0.x -M 100 (so it uses MSS of 100 bytes, making for very small packets) gets the same result, so does running it with MSS of 1000.

          tinfoilmattT 2 Replies Last reply Reply Quote 0
          • tinfoilmattT Offline
            tinfoilmatt LAYER 8 @Pentium100
            last edited by

            I'd test the theory by setting "enc0" back to the default setting.

            P 1 Reply Last reply Reply Quote 0
            • tinfoilmattT Offline
              tinfoilmatt LAYER 8 @Pentium100
              last edited by

              56 plus 1,348 equals 1,404.

              1 Reply Last reply Reply Quote 0
              • P Offline
                Pentium100 @tinfoilmatt
                last edited by

                @tinfoilmatt Sorry, I got mixed up
                enc0 (as I understand all IPSec traffic goes through it) mtu is 1536
                ipsec30 (the actual VTI interface for this VPN) mtu is 1400

                In any case, those are the defaults.

                I cannot run new tests right now and I did not save the captures from the previous tests with small MSS, but it behaved the same. I will rerun them and post the capture when I can.

                tinfoilmattT 1 Reply Last reply Reply Quote 0
                • tinfoilmattT Offline
                  tinfoilmatt LAYER 8 @Pentium100
                  last edited by tinfoilmatt

                  enc0 (as I understand all IPSec traffic goes through it) mtu is 1536

                  That's what I see on the system I checked, too.

                  ipsec30 (the actual VTI interface for this VPN) mtu is 1400

                  Bump this up to at least 1,420 bytes. You may be able to go as high as 1,480-1,500 bytes. If MTU ends up being the actual issue, try to find the 'break point'.

                  P 1 Reply Last reply Reply Quote 0
                  • P Offline
                    Pentium100 @tinfoilmatt
                    last edited by

                    @tinfoilmatt
                    Changed MTU of the VTI interface to 1420, the problem remains

                    I tried running iperf with -M 200 parameter, which limits MSS to 200 bytes. This way it should avoid any MTU issues.

                    This is how it looks in vtnet2

                    08:27:57.873521 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 66590:66778, ack 1, win 511, options [nop,nop,TS val 1372876030 ecr 215863], length 188
08:27:57.873542 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 66778:66966, ack 1, win 511, options [nop,nop,TS val 1372876030 ecr 215863], length 188
08:27:57.873564 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 66966:67154, ack 1, win 511, options [nop,nop,TS val 1372876030 ecr 215863], length 188
08:27:57.873569 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 67154:67342, ack 1, win 511, options [nop,nop,TS val 1372876030 ecr 215863], length 188
...snip due to character count limit...
08:27:57.874753 IP 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 53994, win 257, options [nop,nop,TS val 215864 ecr 1372876026], length 0
08:27:57.874786 IP 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 54934, win 257, options [nop,nop,TS val 215864 ecr 1372876026], length 0
08:27:57.874795 IP 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 55874, win 257, options [nop,nop,TS val 215864 ecr 1372876026], length 0
08:27:57.874857 IP 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 57754, win 257, options [nop,nop,TS val 215864 ecr 1372876026], length 0
08:27:57.874878 IP 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 58130, win 256, options [nop,nop,TS val 215864 ecr 1372876027], length 0
08:27:57.875051 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 102498:102686, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875070 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 102686:102874, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875074 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 102874:103062, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875077 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 103062:103250, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875081 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 103250:103438, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875084 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 103438:103626, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875087 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 103626:103814, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875091 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 103814:104002, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875094 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 104002:104190, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875097 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 104190:104378, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875100 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 104378:104566, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875104 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 104566:104754, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875107 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 104754:104942, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875110 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 104942:105130, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875113 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 105130:105318, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875116 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 105318:105506, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875119 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 105506:105694, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875122 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 105694:105882, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875125 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 105882:106070, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875128 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 106070:106258, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875131 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 106258:106446, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875134 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 106446:106634, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875137 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 106634:106822, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875140 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 106822:107010, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875143 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 107010:107198, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875146 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 107198:107386, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875149 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 107386:107574, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875152 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 107574:107762, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875155 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 107762:107950, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875158 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 107950:108138, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875161 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 108138:108326, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875164 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 108326:108514, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875167 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 108514:108702, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875170 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 108702:108890, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875173 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 108890:109078, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875176 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 109078:109266, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875179 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 109266:109454, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875182 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 109454:109642, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875185 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 109642:109830, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875188 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 109830:110018, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875190 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 110018:110206, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875193 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 110206:110394, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875196 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 110394:110582, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875199 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 110582:110770, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875202 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 110770:110958, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875205 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 110958:111146, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875208 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 111146:111334, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875211 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 111334:111522, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875213 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [P.], seq 111522:111710, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.876173 IP 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 60010, win 257, options [nop,nop,TS val 215864 ecr 1372876029], length 0
08:27:57.876253 IP 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 61890, win 257, options [nop,nop,TS val 215864 ecr 1372876029], length 0
08:27:57.876349 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 111710:111898, ack 1, win 511, options [nop,nop,TS val 1372876034 ecr 215864], length 188
08:27:57.876355 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 111898:112086, ack 1, win 511, options [nop,nop,TS val 1372876034 ecr 215864], length 188
08:27:57.876358 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 112086:112274, ack 1, win 511, options [nop,nop,TS val 1372876034 ecr 215864], length 188
08:27:57.876361 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 112274:112462, ack 1, win 511, options [nop,nop,TS val 1372876034 ecr 215864], length 188
08:27:57.876364 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 112462:112650, ack 1, win 511, options [nop,nop,TS val 1372876034 ecr 215864], length 188
08:27:57.876395 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 112650:112838, ack 1, win 511, options [nop,nop,TS val 1372876034 ecr 215864], length 188
08:27:57.876398 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 112838:113026, ack 1, win 511, options [nop,nop,TS val 1372876034 ecr 215864], length 188
08:27:57.876401 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 113026:113214, ack 1, win 511, options [nop,nop,TS val 1372876034 ecr 215864], length 188
08:27:57.876404 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 113214:113402, ack 1, win 511, options [nop,nop,TS val 1372876034 ecr 215864], length 188
08:27:57.876407 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 113402:113590, ack 1, win 511, options [nop,nop,TS val 1372876034 ecr 215864], length 188
08:27:57.876410 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 113590:113778, ack 1, win 511, options [nop,nop,TS val 1372876034 ecr 215864], length 188
08:27:57.876413 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 113778:113966, ack 1, win 511, options [nop,nop,TS val 1372876034 ecr 215864], length 188
08:27:57.876416 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 113966:114154, ack 1, win 511, options [nop,nop,TS val 1372876034 ecr 215864], length 188
08:27:57.876446 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 114154:114342, ack 1, win 511, options [nop,nop,TS val 1372876034 ecr 215864], length 188
08:27:57.876450 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [P.], seq 114342:114530, ack 1, win 511, options [nop,nop,TS val 1372876034 ecr 215864], length 188
08:27:57.876905 IP 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 63582, win 257, options [nop,nop,TS val 215864 ecr 1372876030], length 0
08:27:57.877179 IP 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 65274, win 257, options [nop,nop,TS val 215864 ecr 1372876030], length 0
08:27:57.877617 IP 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 66966, win 257, options [nop,nop,TS val 215864 ecr 1372876030], length 0
08:27:57.877802 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 114530:114718, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877875 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 114718:114906, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877882 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 114906:115094, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877887 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 115094:115282, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877894 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 115282:115470, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877900 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 115470:115658, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877907 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 115658:115846, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877913 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 115846:116034, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877918 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 116034:116222, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877923 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 116222:116410, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877928 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 116410:116598, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877933 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 116598:116786, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877938 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 116786:116974, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877943 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 116974:117162, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877949 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 117162:117350, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877954 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 117350:117538, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877959 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 117538:117726, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877964 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 117726:117914, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877969 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 117914:118102, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877974 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 118102:118290, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877979 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 118290:118478, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877984 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 118478:118666, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877990 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 118666:118854, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877995 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 118854:119042, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.878000 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 119042:119230, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.878004 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 119230:119418, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.878009 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 119418:119606, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.878014 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 119606:119794, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.878019 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 119794:119982, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.878024 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 119982:120170, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.878029 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 120170:120358, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.878035 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 120358:120546, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.878041 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 120546:120734, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.878047 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 120734:120922, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.878052 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 120922:121110, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.878057 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 121110:121298, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.878063 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 121298:121486, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.878068 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [P.], seq 121486:121674, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.894202 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 121674:121862, ack 1, win 511, options [nop,nop,TS val 1372876052 ecr 215864], length 188
08:27:58.106175 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 66966:67154, ack 1, win 511, options [nop,nop,TS val 1372876264 ecr 215864], length 188
08:27:58.542199 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 66966:67154, ack 1, win 511, options [nop,nop,TS val 1372876700 ecr 215864], length 188
08:27:59.406359 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 66966:67154, ack 1, win 511, options [nop,nop,TS val 1372877564 ecr 215864], length 188
08:28:01.102305 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 66966:67154, ack 1, win 511, options [nop,nop,TS val 1372879260 ecr 215864], length 188
08:28:04.494214 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 66966:67154, ack 1, win 511, options [nop,nop,TS val 1372882652 ecr 215864], length 188
08:28:07.850260 IP 192.168.10.x.37134 > 192.168.0.x.5201: Flags [P.], seq 176:177, ack 5, win 510, options [nop,nop,TS val 1372886008 ecr 215859], length 1
08:28:07.855152 IP 192.168.0.x.5201 > 192.168.10.x.37150: Flags [F.], seq 1, ack 66966, win 257, options [nop,nop,TS val 216862 ecr 1372876030], length 0
08:28:07.857710 IP 192.168.0.x.5201 > 192.168.10.x.37134: Flags [P.], seq 5:6, ack 177, win 257, options [nop,nop,TS val 216862 ecr 1372886008], length 1
08:28:07.858054 IP 192.168.10.x.37134 > 192.168.0.x.5201: Flags [P.], seq 177:181, ack 6, win 510, options [nop,nop,TS val 1372886015 ecr 216862], length 4
08:28:07.858097 IP 192.168.10.x.37134 > 192.168.0.x.5201: Flags [P.], seq 181:448, ack 6, win 510, options [nop,nop,TS val 1372886015 ecr 216862], length 267
08:28:07.858218 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], ack 2, win 511, options [nop,nop,TS val 1372886016 ecr 216862], length 0
08:28:07.862217 IP 192.168.0.x.5201 > 192.168.10.x.37134: Flags [.], ack 448, win 256, options [nop,nop,TS val 216862 ecr 1372886015], length 0
08:28:07.862719 IP 192.168.0.x.5201 > 192.168.10.x.37134: Flags [P.], seq 6:10, ack 448, win 256, options [nop,nop,TS val 216862 ecr 1372886015], length 4
08:28:07.862786 IP 192.168.0.x.5201 > 192.168.10.x.37134: Flags [P.], seq 10:253, ack 448, win 256, options [nop,nop,TS val 216862 ecr 1372886015], length 243
08:28:07.862805 IP 192.168.0.x.5201 > 192.168.10.x.37134: Flags [P.], seq 253:254, ack 448, win 256, options [nop,nop,TS val 216862 ecr 1372886015], length 1
08:28:07.863034 IP 192.168.10.x.37134 > 192.168.0.x.5201: Flags [.], ack 254, win 509, options [nop,nop,TS val 1372886020 ecr 216862], length 0
08:28:07.863098 IP 192.168.10.x.37134 > 192.168.0.x.5201: Flags [P.], seq 448:449, ack 254, win 509, options [nop,nop,TS val 1372886020 ecr 216862], length 1
08:28:07.863120 IP 192.168.10.x.37134 > 192.168.0.x.5201: Flags [F.], seq 449, ack 254, win 509, options [nop,nop,TS val 1372886021 ecr 216862], length 0
08:28:07.867403 IP 192.168.0.x.5201 > 192.168.10.x.37134: Flags [.], ack 450, win 256, options [nop,nop,TS val 216863 ecr 1372886020], length 0
08:28:07.868341 IP 192.168.0.x.5201 > 192.168.10.x.37134: Flags [F.], seq 254, ack 450, win 256, options [nop,nop,TS val 216863 ecr 1372886020], length 0
08:28:07.868372 IP 192.168.0.x.5201 > 192.168.10.x.37134: Flags [R.], seq 255, ack 450, win 0, length 0
08:28:07.868670 IP 192.168.10.x.37134 > 192.168.0.x.5201: Flags [.], ack 255, win 509, options [nop,nop,TS val 1372886026 ecr 216863], length 0
08:28:11.406218 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 66966:67154, ack 2, win 511, options [nop,nop,TS val 1372889564 ecr 216862], length 188
08:28:16.782327 IP 192.168.10.x.51892 > 192.168.0.x.5201: Flags [.], seq 0:1348, ack 1, win 510, options [nop,nop,TS val 1372894940 ecr 213348], length 1348
08:28:16.786874 IP 192.168.0.x.5201 > 192.168.10.x.51892: Flags [R.], seq 1, ack 1348, win 0, length 0

                    And this is how it loks on enc0

                    08:27:57.873524 (authentic,confidential): SPI 0x9bbda17c: 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 66590:66778, ack 1, win 511, options [nop,nop,TS val 1372876030 ecr 215863], length 188
08:27:57.873546 (authentic,confidential): SPI 0x9bbda17c: 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 66778:66966, ack 1, win 511, options [nop,nop,TS val 1372876030 ecr 215863], length 188
08:27:57.873853 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 40834, win 257, options [nop,nop,TS val 215863 ecr 1372876026], length 0
08:27:57.873897 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 42714, win 257, options [nop,nop,TS val 215864 ecr 1372876026], length 0
08:27:57.873905 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 43842, win 257, options [nop,nop,TS val 215864 ecr 1372876026], length 0
08:27:57.873911 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 44594, win 257, options [nop,nop,TS val 215864 ecr 1372876026], length 0
08:27:57.873918 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 46474, win 257, options [nop,nop,TS val 215864 ecr 1372876026], length 0
08:27:57.873928 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 48354, win 257, options [nop,nop,TS val 215864 ecr 1372876026], length 0
08:27:57.873939 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 49294, win 257, options [nop,nop,TS val 215864 ecr 1372876026], length 0
08:27:57.873949 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 50234, win 257, options [nop,nop,TS val 215864 ecr 1372876026], length 0
08:27:57.874110 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 52114, win 257, options [nop,nop,TS val 215864 ecr 1372876026], length 0
08:27:57.874739 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 53994, win 257, options [nop,nop,TS val 215864 ecr 1372876026], length 0
08:27:57.874769 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 54934, win 257, options [nop,nop,TS val 215864 ecr 1372876026], length 0
08:27:57.874778 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 55874, win 257, options [nop,nop,TS val 215864 ecr 1372876026], length 0
08:27:57.874847 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 57754, win 257, options [nop,nop,TS val 215864 ecr 1372876026], length 0
08:27:57.874871 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 58130, win 256, options [nop,nop,TS val 215864 ecr 1372876027], length 0
08:27:57.876157 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 60010, win 257, options [nop,nop,TS val 215864 ecr 1372876029], length 0
08:27:57.876242 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 61890, win 257, options [nop,nop,TS val 215864 ecr 1372876029], length 0
08:27:57.876888 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 63582, win 257, options [nop,nop,TS val 215864 ecr 1372876030], length 0
08:27:57.877173 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 65274, win 257, options [nop,nop,TS val 215864 ecr 1372876030], length 0
08:27:57.877611 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 66966, win 257, options [nop,nop,TS val 215864 ecr 1372876030], length 0
08:28:07.850292 (authentic,confidential): SPI 0x9bbda17c: 192.168.10.x.37134 > 192.168.0.x.5201: Flags [P.], seq 176:177, ack 5, win 510, options [nop,nop,TS val 1372886008 ecr 215859], length 1
08:28:07.855136 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [F.], seq 1, ack 66966, win 257, options [nop,nop,TS val 216862 ecr 1372876030], length 0
08:28:07.857694 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37134: Flags [P.], seq 5:6, ack 177, win 257, options [nop,nop,TS val 216862 ecr 1372886008], length 1
08:28:07.858068 (authentic,confidential): SPI 0x9bbda17c: 192.168.10.x.37134 > 192.168.0.x.5201: Flags [P.], seq 177:181, ack 6, win 510, options [nop,nop,TS val 1372886015 ecr 216862], length 4
08:28:07.858102 (authentic,confidential): SPI 0x9bbda17c: 192.168.10.x.37134 > 192.168.0.x.5201: Flags [P.], seq 181:448, ack 6, win 510, options [nop,nop,TS val 1372886015 ecr 216862], length 267
08:28:07.858222 (authentic,confidential): SPI 0x9bbda17c: 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], ack 2, win 511, options [nop,nop,TS val 1372886016 ecr 216862], length 0
08:28:07.862208 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37134: Flags [.], ack 448, win 256, options [nop,nop,TS val 216862 ecr 1372886015], length 0
08:28:07.862713 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37134: Flags [P.], seq 6:10, ack 448, win 256, options [nop,nop,TS val 216862 ecr 1372886015], length 4
08:28:07.862780 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37134: Flags [P.], seq 10:253, ack 448, win 256, options [nop,nop,TS val 216862 ecr 1372886015], length 243
08:28:07.862800 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37134: Flags [P.], seq 253:254, ack 448, win 256, options [nop,nop,TS val 216862 ecr 1372886015], length 1
08:28:07.863040 (authentic,confidential): SPI 0x9bbda17c: 192.168.10.x.37134 > 192.168.0.x.5201: Flags [.], ack 254, win 509, options [nop,nop,TS val 1372886020 ecr 216862], length 0
08:28:07.863103 (authentic,confidential): SPI 0x9bbda17c: 192.168.10.x.37134 > 192.168.0.x.5201: Flags [P.], seq 448:449, ack 254, win 509, options [nop,nop,TS val 1372886020 ecr 216862], length 1
08:28:07.863123 (authentic,confidential): SPI 0x9bbda17c: 192.168.10.x.37134 > 192.168.0.x.5201: Flags [F.], seq 449, ack 254, win 509, options [nop,nop,TS val 1372886021 ecr 216862], length 0
08:28:07.867391 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37134: Flags [.], ack 450, win 256, options [nop,nop,TS val 216863 ecr 1372886020], length 0
08:28:07.868334 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37134: Flags [F.], seq 254, ack 450, win 256, options [nop,nop,TS val 216863 ecr 1372886020], length 0
08:28:07.868354 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37134: Flags [R.], seq 255, ack 450, win 0, length 0
08:28:07.868676 (authentic,confidential): SPI 0x9bbda17c: 192.168.10.x.37134 > 192.168.0.x.5201: Flags [.], ack 255, win 509, options [nop,nop,TS val 1372886026 ecr 216863], length 0
08:28:16.782353 (authentic,confidential): SPI 0x9bbda17c: 192.168.10.x.51892 > 192.168.0.x.5201: Flags [.], seq 3382332850:3382334198, ack 1144945350, win 510, options [nop,nop,TS val 1372894940 ecr 213348], length 1348
08:28:16.786867 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.51892: Flags [R.], seq 1, ack 1348, win 0, length 0

                    The packets sent from 192.168.10.x just disappear. And since these are small packets there should be no issues with MTU.

                    It seems that once some data is transferred, pfsense stops forwarding the packets, but in one direction only.

                    If I do a reverse test (iperf3 -R) packets are not dropped, everything works as it should.
                    If I do a bidirectional test (iperf3 --bidir) upload from 192.168.10.x stalls, download to 192.168.10.x works correctly.

                    tinfoilmattT P 2 Replies Last reply Reply Quote 0
                    • tinfoilmattT Offline
                      tinfoilmatt LAYER 8 @Pentium100
                      last edited by

                      Would be better to upload PCAPs of entire traces and/or text files containing more complete logging.

                      1 Reply Last reply Reply Quote 0
                      • P Offline
                        Pentium100 @Pentium100
                        last edited by

                        Sorry for double post, but I found something else interesting.
                        I am not too familiar with BSD, I am more familiar with Linux, but this looks weird to me:

                        If I start the iperf3 test, then while it is running (stalled transfer) I run
                        pfctl -s state | grep 192.168.0.x | grep 192.168.10.x | grep 5201
                        I get this:

                        vtnet2 tcp 192.168.0.x:5201 <- 192.168.10.x:42202       ESTABLISHED:ESTABLISHED
ipsec30 tcp 192.168.10.x:42202 -> 192.168.0.x:5201       SYN_SENT:CLOSED
all tcp 192.168.10.x:42202 -> 192.168.0.x:5201       ESTABLISHED:ESTABLISHED

                        So, on "all" and vtnet2 it says ESTABLISHED, but on ipsec30 it says CLOSED.
                        Are the states supposed to look like this?

                        I checked this on another pfsense router (older version 2.6.0) and all states there are on "all" and not separate interfaces. What is the difference here?

                        tinfoilmattT 1 Reply Last reply Reply Quote 0
                        • tinfoilmattT Offline
                          tinfoilmatt LAYER 8 @Pentium100
                          last edited by

                          What does the firewall ruleset look like on both IPsec and IPsec30 tabs?

                          I checked this on another pfsense router (older version 2.6.0) and all states there are on "all" and not separate interfaces.

                          Obvious question, but is that system also using Routed IPsec or no?

                          P 1 Reply Last reply Reply Quote 0
                          • P Offline
                            Pentium100 @tinfoilmatt
                            last edited by

                            @tinfoilmatt said in TCP traffic over IPSec stalls with some packets not appearing on enc0:

                            Obvious question, but is that system also using Routed IPsec or no?

                            Yes. It's the only other pfsense with routed IPsec..

                            @tinfoilmatt said in TCP traffic over IPSec stalls with some packets not appearing on enc0:

                            What does the firewall ruleset look like on both IPsec and IPsec30 tabs?

                            IPSec tab has only one rule
                            c9283e3a-bac9-4278-9137-0f3d69f5f119-image.png

                            ipsec30 interface does not have its own firewall tab. That's probably because I have this setting on:
                            afbef87c-da7d-46e8-a926-8bafd9a8c0bc-image.png

                            This is because there are policy-based IPsec tunnels too, so I cannot change it.
                            This setting is the same on both routers, as both have a combination of routed IPsec and policy-based.

                            tinfoilmattT 1 Reply Last reply Reply Quote 0
                            • tinfoilmattT Offline
                              tinfoilmatt LAYER 8 @Pentium100
                              last edited by

                              ipsec30 interface does not have its own firewall tab. That's probably because I have this setting on:
                              afbef87c-da7d-46e8-a926-8bafd9a8c0bc-image.png

                              So then you're aware that that's literally the setting preventing you from using "per-interface rules, NAT, or reply-to" reliably.

                              Maybe there's a way to make this behavior more reliable/consistent using static routing?

                              P 1 Reply Last reply Reply Quote 0
                              • P Offline
                                Pentium100 @tinfoilmatt
                                last edited by

                                @tinfoilmatt Changing the setting will break the policy-based VPNs.

                                I do not really need per-interface rules, as I just have allowed everything in the IPsec tab and it works correctly on the other router with an older pfsense version.

                                As I only have two such routers I cannot say if the other one works because of the version of something else.

                                tinfoilmattT 1 Reply Last reply Reply Quote 0
                                • tinfoilmattT Offline
                                  tinfoilmatt LAYER 8 @Pentium100
                                  last edited by

                                  The issue is not the firewall ruleset.

                                  1 Reply Last reply Reply Quote 0
                                  • tinfoilmattT Offline
                                    tinfoilmatt LAYER 8
                                    last edited by

                                    Why does this P2 require "Routed (VTI)" mode?

                                    tinfoilmattT P 2 Replies Last reply Reply Quote 0
                                    • tinfoilmattT Offline
                                      tinfoilmatt LAYER 8 @tinfoilmatt
                                      last edited by

                                      And can we assume you've read this section of the documentation, IPsec Interface Assignment?

                                      1 Reply Last reply Reply Quote 0
                                      • P Offline
                                        Pentium100 @tinfoilmatt
                                        last edited by

                                        @tinfoilmatt
                                        Because using policy based it would be something like 30 phase2 entries and we had a problem where some of them would stop working at random.

                                        Anyway, I found a solution to this.
                                        Firewall State Policy - set to "Floating States" (default was "Interface Bound States" and apparently this default was different on the older pfsense version)
                                        Now it seems to work

                                        tinfoilmattT 1 Reply Last reply Reply Quote 0
                                        • tinfoilmattT Offline
                                          tinfoilmatt LAYER 8 @Pentium100
                                          last edited by

                                          If it works and it's messy, it still works.

                                          P 1 Reply Last reply Reply Quote 0
                                          • P Offline
                                            Pentium100 @tinfoilmatt
                                            last edited by

                                            @tinfoilmatt Yeah. I have to use routed ipsec for this and also have to use policy-based tunnels. If there is no other way to make this work, then using floating states (like it is in the older version) seems like a solution.

                                            tinfoilmattT 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2026 Rubicon Communications LLC (Netgate). All rights reserved.