Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login
    Introducing Netgate Nexus: Multi-Instance Management at Your Fingertips.

    TCP traffic over IPSec stalls with some packets not appearing on enc0

    Scheduled Pinned Locked Moved IPsec
    21 Posts 2 Posters 531 Views 2 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P Offline
      Pentium100 @tinfoilmatt
      last edited by

      @tinfoilmatt
      Changed MTU of the VTI interface to 1420, the problem remains

      I tried running iperf with -M 200 parameter, which limits MSS to 200 bytes. This way it should avoid any MTU issues.

      This is how it looks in vtnet2

      08:27:57.873521 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 66590:66778, ack 1, win 511, options [nop,nop,TS val 1372876030 ecr 215863], length 188
08:27:57.873542 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 66778:66966, ack 1, win 511, options [nop,nop,TS val 1372876030 ecr 215863], length 188
08:27:57.873564 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 66966:67154, ack 1, win 511, options [nop,nop,TS val 1372876030 ecr 215863], length 188
08:27:57.873569 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 67154:67342, ack 1, win 511, options [nop,nop,TS val 1372876030 ecr 215863], length 188
...snip due to character count limit...
08:27:57.874753 IP 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 53994, win 257, options [nop,nop,TS val 215864 ecr 1372876026], length 0
08:27:57.874786 IP 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 54934, win 257, options [nop,nop,TS val 215864 ecr 1372876026], length 0
08:27:57.874795 IP 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 55874, win 257, options [nop,nop,TS val 215864 ecr 1372876026], length 0
08:27:57.874857 IP 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 57754, win 257, options [nop,nop,TS val 215864 ecr 1372876026], length 0
08:27:57.874878 IP 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 58130, win 256, options [nop,nop,TS val 215864 ecr 1372876027], length 0
08:27:57.875051 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 102498:102686, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875070 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 102686:102874, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875074 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 102874:103062, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875077 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 103062:103250, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875081 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 103250:103438, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875084 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 103438:103626, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875087 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 103626:103814, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875091 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 103814:104002, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875094 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 104002:104190, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875097 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 104190:104378, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875100 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 104378:104566, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875104 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 104566:104754, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875107 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 104754:104942, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875110 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 104942:105130, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875113 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 105130:105318, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875116 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 105318:105506, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875119 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 105506:105694, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875122 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 105694:105882, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875125 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 105882:106070, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875128 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 106070:106258, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875131 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 106258:106446, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875134 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 106446:106634, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875137 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 106634:106822, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875140 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 106822:107010, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875143 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 107010:107198, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875146 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 107198:107386, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875149 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 107386:107574, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875152 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 107574:107762, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875155 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 107762:107950, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875158 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 107950:108138, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875161 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 108138:108326, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875164 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 108326:108514, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875167 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 108514:108702, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875170 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 108702:108890, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875173 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 108890:109078, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875176 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 109078:109266, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875179 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 109266:109454, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875182 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 109454:109642, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875185 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 109642:109830, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875188 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 109830:110018, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875190 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 110018:110206, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875193 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 110206:110394, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875196 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 110394:110582, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875199 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 110582:110770, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875202 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 110770:110958, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875205 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 110958:111146, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875208 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 111146:111334, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875211 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 111334:111522, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.875213 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [P.], seq 111522:111710, ack 1, win 511, options [nop,nop,TS val 1372876032 ecr 215864], length 188
08:27:57.876173 IP 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 60010, win 257, options [nop,nop,TS val 215864 ecr 1372876029], length 0
08:27:57.876253 IP 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 61890, win 257, options [nop,nop,TS val 215864 ecr 1372876029], length 0
08:27:57.876349 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 111710:111898, ack 1, win 511, options [nop,nop,TS val 1372876034 ecr 215864], length 188
08:27:57.876355 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 111898:112086, ack 1, win 511, options [nop,nop,TS val 1372876034 ecr 215864], length 188
08:27:57.876358 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 112086:112274, ack 1, win 511, options [nop,nop,TS val 1372876034 ecr 215864], length 188
08:27:57.876361 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 112274:112462, ack 1, win 511, options [nop,nop,TS val 1372876034 ecr 215864], length 188
08:27:57.876364 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 112462:112650, ack 1, win 511, options [nop,nop,TS val 1372876034 ecr 215864], length 188
08:27:57.876395 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 112650:112838, ack 1, win 511, options [nop,nop,TS val 1372876034 ecr 215864], length 188
08:27:57.876398 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 112838:113026, ack 1, win 511, options [nop,nop,TS val 1372876034 ecr 215864], length 188
08:27:57.876401 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 113026:113214, ack 1, win 511, options [nop,nop,TS val 1372876034 ecr 215864], length 188
08:27:57.876404 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 113214:113402, ack 1, win 511, options [nop,nop,TS val 1372876034 ecr 215864], length 188
08:27:57.876407 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 113402:113590, ack 1, win 511, options [nop,nop,TS val 1372876034 ecr 215864], length 188
08:27:57.876410 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 113590:113778, ack 1, win 511, options [nop,nop,TS val 1372876034 ecr 215864], length 188
08:27:57.876413 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 113778:113966, ack 1, win 511, options [nop,nop,TS val 1372876034 ecr 215864], length 188
08:27:57.876416 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 113966:114154, ack 1, win 511, options [nop,nop,TS val 1372876034 ecr 215864], length 188
08:27:57.876446 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 114154:114342, ack 1, win 511, options [nop,nop,TS val 1372876034 ecr 215864], length 188
08:27:57.876450 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [P.], seq 114342:114530, ack 1, win 511, options [nop,nop,TS val 1372876034 ecr 215864], length 188
08:27:57.876905 IP 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 63582, win 257, options [nop,nop,TS val 215864 ecr 1372876030], length 0
08:27:57.877179 IP 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 65274, win 257, options [nop,nop,TS val 215864 ecr 1372876030], length 0
08:27:57.877617 IP 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 66966, win 257, options [nop,nop,TS val 215864 ecr 1372876030], length 0
08:27:57.877802 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 114530:114718, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877875 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 114718:114906, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877882 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 114906:115094, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877887 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 115094:115282, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877894 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 115282:115470, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877900 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 115470:115658, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877907 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 115658:115846, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877913 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 115846:116034, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877918 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 116034:116222, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877923 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 116222:116410, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877928 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 116410:116598, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877933 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 116598:116786, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877938 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 116786:116974, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877943 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 116974:117162, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877949 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 117162:117350, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877954 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 117350:117538, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877959 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 117538:117726, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877964 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 117726:117914, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877969 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 117914:118102, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877974 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 118102:118290, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877979 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 118290:118478, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877984 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 118478:118666, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877990 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 118666:118854, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.877995 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 118854:119042, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.878000 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 119042:119230, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.878004 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 119230:119418, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.878009 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 119418:119606, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.878014 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 119606:119794, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.878019 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 119794:119982, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.878024 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 119982:120170, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.878029 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 120170:120358, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.878035 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 120358:120546, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.878041 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 120546:120734, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.878047 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 120734:120922, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.878052 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 120922:121110, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.878057 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 121110:121298, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.878063 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 121298:121486, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.878068 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [P.], seq 121486:121674, ack 1, win 511, options [nop,nop,TS val 1372876035 ecr 215864], length 188
08:27:57.894202 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 121674:121862, ack 1, win 511, options [nop,nop,TS val 1372876052 ecr 215864], length 188
08:27:58.106175 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 66966:67154, ack 1, win 511, options [nop,nop,TS val 1372876264 ecr 215864], length 188
08:27:58.542199 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 66966:67154, ack 1, win 511, options [nop,nop,TS val 1372876700 ecr 215864], length 188
08:27:59.406359 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 66966:67154, ack 1, win 511, options [nop,nop,TS val 1372877564 ecr 215864], length 188
08:28:01.102305 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 66966:67154, ack 1, win 511, options [nop,nop,TS val 1372879260 ecr 215864], length 188
08:28:04.494214 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 66966:67154, ack 1, win 511, options [nop,nop,TS val 1372882652 ecr 215864], length 188
08:28:07.850260 IP 192.168.10.x.37134 > 192.168.0.x.5201: Flags [P.], seq 176:177, ack 5, win 510, options [nop,nop,TS val 1372886008 ecr 215859], length 1
08:28:07.855152 IP 192.168.0.x.5201 > 192.168.10.x.37150: Flags [F.], seq 1, ack 66966, win 257, options [nop,nop,TS val 216862 ecr 1372876030], length 0
08:28:07.857710 IP 192.168.0.x.5201 > 192.168.10.x.37134: Flags [P.], seq 5:6, ack 177, win 257, options [nop,nop,TS val 216862 ecr 1372886008], length 1
08:28:07.858054 IP 192.168.10.x.37134 > 192.168.0.x.5201: Flags [P.], seq 177:181, ack 6, win 510, options [nop,nop,TS val 1372886015 ecr 216862], length 4
08:28:07.858097 IP 192.168.10.x.37134 > 192.168.0.x.5201: Flags [P.], seq 181:448, ack 6, win 510, options [nop,nop,TS val 1372886015 ecr 216862], length 267
08:28:07.858218 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], ack 2, win 511, options [nop,nop,TS val 1372886016 ecr 216862], length 0
08:28:07.862217 IP 192.168.0.x.5201 > 192.168.10.x.37134: Flags [.], ack 448, win 256, options [nop,nop,TS val 216862 ecr 1372886015], length 0
08:28:07.862719 IP 192.168.0.x.5201 > 192.168.10.x.37134: Flags [P.], seq 6:10, ack 448, win 256, options [nop,nop,TS val 216862 ecr 1372886015], length 4
08:28:07.862786 IP 192.168.0.x.5201 > 192.168.10.x.37134: Flags [P.], seq 10:253, ack 448, win 256, options [nop,nop,TS val 216862 ecr 1372886015], length 243
08:28:07.862805 IP 192.168.0.x.5201 > 192.168.10.x.37134: Flags [P.], seq 253:254, ack 448, win 256, options [nop,nop,TS val 216862 ecr 1372886015], length 1
08:28:07.863034 IP 192.168.10.x.37134 > 192.168.0.x.5201: Flags [.], ack 254, win 509, options [nop,nop,TS val 1372886020 ecr 216862], length 0
08:28:07.863098 IP 192.168.10.x.37134 > 192.168.0.x.5201: Flags [P.], seq 448:449, ack 254, win 509, options [nop,nop,TS val 1372886020 ecr 216862], length 1
08:28:07.863120 IP 192.168.10.x.37134 > 192.168.0.x.5201: Flags [F.], seq 449, ack 254, win 509, options [nop,nop,TS val 1372886021 ecr 216862], length 0
08:28:07.867403 IP 192.168.0.x.5201 > 192.168.10.x.37134: Flags [.], ack 450, win 256, options [nop,nop,TS val 216863 ecr 1372886020], length 0
08:28:07.868341 IP 192.168.0.x.5201 > 192.168.10.x.37134: Flags [F.], seq 254, ack 450, win 256, options [nop,nop,TS val 216863 ecr 1372886020], length 0
08:28:07.868372 IP 192.168.0.x.5201 > 192.168.10.x.37134: Flags [R.], seq 255, ack 450, win 0, length 0
08:28:07.868670 IP 192.168.10.x.37134 > 192.168.0.x.5201: Flags [.], ack 255, win 509, options [nop,nop,TS val 1372886026 ecr 216863], length 0
08:28:11.406218 IP 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 66966:67154, ack 2, win 511, options [nop,nop,TS val 1372889564 ecr 216862], length 188
08:28:16.782327 IP 192.168.10.x.51892 > 192.168.0.x.5201: Flags [.], seq 0:1348, ack 1, win 510, options [nop,nop,TS val 1372894940 ecr 213348], length 1348
08:28:16.786874 IP 192.168.0.x.5201 > 192.168.10.x.51892: Flags [R.], seq 1, ack 1348, win 0, length 0

      And this is how it loks on enc0

      08:27:57.873524 (authentic,confidential): SPI 0x9bbda17c: 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 66590:66778, ack 1, win 511, options [nop,nop,TS val 1372876030 ecr 215863], length 188
08:27:57.873546 (authentic,confidential): SPI 0x9bbda17c: 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], seq 66778:66966, ack 1, win 511, options [nop,nop,TS val 1372876030 ecr 215863], length 188
08:27:57.873853 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 40834, win 257, options [nop,nop,TS val 215863 ecr 1372876026], length 0
08:27:57.873897 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 42714, win 257, options [nop,nop,TS val 215864 ecr 1372876026], length 0
08:27:57.873905 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 43842, win 257, options [nop,nop,TS val 215864 ecr 1372876026], length 0
08:27:57.873911 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 44594, win 257, options [nop,nop,TS val 215864 ecr 1372876026], length 0
08:27:57.873918 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 46474, win 257, options [nop,nop,TS val 215864 ecr 1372876026], length 0
08:27:57.873928 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 48354, win 257, options [nop,nop,TS val 215864 ecr 1372876026], length 0
08:27:57.873939 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 49294, win 257, options [nop,nop,TS val 215864 ecr 1372876026], length 0
08:27:57.873949 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 50234, win 257, options [nop,nop,TS val 215864 ecr 1372876026], length 0
08:27:57.874110 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 52114, win 257, options [nop,nop,TS val 215864 ecr 1372876026], length 0
08:27:57.874739 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 53994, win 257, options [nop,nop,TS val 215864 ecr 1372876026], length 0
08:27:57.874769 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 54934, win 257, options [nop,nop,TS val 215864 ecr 1372876026], length 0
08:27:57.874778 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 55874, win 257, options [nop,nop,TS val 215864 ecr 1372876026], length 0
08:27:57.874847 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 57754, win 257, options [nop,nop,TS val 215864 ecr 1372876026], length 0
08:27:57.874871 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 58130, win 256, options [nop,nop,TS val 215864 ecr 1372876027], length 0
08:27:57.876157 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 60010, win 257, options [nop,nop,TS val 215864 ecr 1372876029], length 0
08:27:57.876242 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 61890, win 257, options [nop,nop,TS val 215864 ecr 1372876029], length 0
08:27:57.876888 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 63582, win 257, options [nop,nop,TS val 215864 ecr 1372876030], length 0
08:27:57.877173 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 65274, win 257, options [nop,nop,TS val 215864 ecr 1372876030], length 0
08:27:57.877611 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [.], ack 66966, win 257, options [nop,nop,TS val 215864 ecr 1372876030], length 0
08:28:07.850292 (authentic,confidential): SPI 0x9bbda17c: 192.168.10.x.37134 > 192.168.0.x.5201: Flags [P.], seq 176:177, ack 5, win 510, options [nop,nop,TS val 1372886008 ecr 215859], length 1
08:28:07.855136 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37150: Flags [F.], seq 1, ack 66966, win 257, options [nop,nop,TS val 216862 ecr 1372876030], length 0
08:28:07.857694 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37134: Flags [P.], seq 5:6, ack 177, win 257, options [nop,nop,TS val 216862 ecr 1372886008], length 1
08:28:07.858068 (authentic,confidential): SPI 0x9bbda17c: 192.168.10.x.37134 > 192.168.0.x.5201: Flags [P.], seq 177:181, ack 6, win 510, options [nop,nop,TS val 1372886015 ecr 216862], length 4
08:28:07.858102 (authentic,confidential): SPI 0x9bbda17c: 192.168.10.x.37134 > 192.168.0.x.5201: Flags [P.], seq 181:448, ack 6, win 510, options [nop,nop,TS val 1372886015 ecr 216862], length 267
08:28:07.858222 (authentic,confidential): SPI 0x9bbda17c: 192.168.10.x.37150 > 192.168.0.x.5201: Flags [.], ack 2, win 511, options [nop,nop,TS val 1372886016 ecr 216862], length 0
08:28:07.862208 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37134: Flags [.], ack 448, win 256, options [nop,nop,TS val 216862 ecr 1372886015], length 0
08:28:07.862713 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37134: Flags [P.], seq 6:10, ack 448, win 256, options [nop,nop,TS val 216862 ecr 1372886015], length 4
08:28:07.862780 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37134: Flags [P.], seq 10:253, ack 448, win 256, options [nop,nop,TS val 216862 ecr 1372886015], length 243
08:28:07.862800 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37134: Flags [P.], seq 253:254, ack 448, win 256, options [nop,nop,TS val 216862 ecr 1372886015], length 1
08:28:07.863040 (authentic,confidential): SPI 0x9bbda17c: 192.168.10.x.37134 > 192.168.0.x.5201: Flags [.], ack 254, win 509, options [nop,nop,TS val 1372886020 ecr 216862], length 0
08:28:07.863103 (authentic,confidential): SPI 0x9bbda17c: 192.168.10.x.37134 > 192.168.0.x.5201: Flags [P.], seq 448:449, ack 254, win 509, options [nop,nop,TS val 1372886020 ecr 216862], length 1
08:28:07.863123 (authentic,confidential): SPI 0x9bbda17c: 192.168.10.x.37134 > 192.168.0.x.5201: Flags [F.], seq 449, ack 254, win 509, options [nop,nop,TS val 1372886021 ecr 216862], length 0
08:28:07.867391 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37134: Flags [.], ack 450, win 256, options [nop,nop,TS val 216863 ecr 1372886020], length 0
08:28:07.868334 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37134: Flags [F.], seq 254, ack 450, win 256, options [nop,nop,TS val 216863 ecr 1372886020], length 0
08:28:07.868354 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.37134: Flags [R.], seq 255, ack 450, win 0, length 0
08:28:07.868676 (authentic,confidential): SPI 0x9bbda17c: 192.168.10.x.37134 > 192.168.0.x.5201: Flags [.], ack 255, win 509, options [nop,nop,TS val 1372886026 ecr 216863], length 0
08:28:16.782353 (authentic,confidential): SPI 0x9bbda17c: 192.168.10.x.51892 > 192.168.0.x.5201: Flags [.], seq 3382332850:3382334198, ack 1144945350, win 510, options [nop,nop,TS val 1372894940 ecr 213348], length 1348
08:28:16.786867 (authentic,confidential): SPI 0xcb0e0fb6: 192.168.0.x.5201 > 192.168.10.x.51892: Flags [R.], seq 1, ack 1348, win 0, length 0

      The packets sent from 192.168.10.x just disappear. And since these are small packets there should be no issues with MTU.

      It seems that once some data is transferred, pfsense stops forwarding the packets, but in one direction only.

      If I do a reverse test (iperf3 -R) packets are not dropped, everything works as it should.
      If I do a bidirectional test (iperf3 --bidir) upload from 192.168.10.x stalls, download to 192.168.10.x works correctly.

      tinfoilmattT P 2 Replies Last reply Reply Quote 0
      • tinfoilmattT Offline
        tinfoilmatt LAYER 8 @Pentium100
        last edited by

        Would be better to upload PCAPs of entire traces and/or text files containing more complete logging.

        1 Reply Last reply Reply Quote 0
        • P Offline
          Pentium100 @Pentium100
          last edited by

          Sorry for double post, but I found something else interesting.
          I am not too familiar with BSD, I am more familiar with Linux, but this looks weird to me:

          If I start the iperf3 test, then while it is running (stalled transfer) I run
          pfctl -s state | grep 192.168.0.x | grep 192.168.10.x | grep 5201
          I get this:

          vtnet2 tcp 192.168.0.x:5201 <- 192.168.10.x:42202       ESTABLISHED:ESTABLISHED
ipsec30 tcp 192.168.10.x:42202 -> 192.168.0.x:5201       SYN_SENT:CLOSED
all tcp 192.168.10.x:42202 -> 192.168.0.x:5201       ESTABLISHED:ESTABLISHED

          So, on "all" and vtnet2 it says ESTABLISHED, but on ipsec30 it says CLOSED.
          Are the states supposed to look like this?

          I checked this on another pfsense router (older version 2.6.0) and all states there are on "all" and not separate interfaces. What is the difference here?

          tinfoilmattT 1 Reply Last reply Reply Quote 0
          • tinfoilmattT Offline
            tinfoilmatt LAYER 8 @Pentium100
            last edited by

            What does the firewall ruleset look like on both IPsec and IPsec30 tabs?

            I checked this on another pfsense router (older version 2.6.0) and all states there are on "all" and not separate interfaces.

            Obvious question, but is that system also using Routed IPsec or no?

            P 1 Reply Last reply Reply Quote 0
            • P Offline
              Pentium100 @tinfoilmatt
              last edited by

              @tinfoilmatt said in TCP traffic over IPSec stalls with some packets not appearing on enc0:

              Obvious question, but is that system also using Routed IPsec or no?

              Yes. It's the only other pfsense with routed IPsec..

              @tinfoilmatt said in TCP traffic over IPSec stalls with some packets not appearing on enc0:

              What does the firewall ruleset look like on both IPsec and IPsec30 tabs?

              IPSec tab has only one rule
              c9283e3a-bac9-4278-9137-0f3d69f5f119-image.png

              ipsec30 interface does not have its own firewall tab. That's probably because I have this setting on:
              afbef87c-da7d-46e8-a926-8bafd9a8c0bc-image.png

              This is because there are policy-based IPsec tunnels too, so I cannot change it.
              This setting is the same on both routers, as both have a combination of routed IPsec and policy-based.

              tinfoilmattT 1 Reply Last reply Reply Quote 0
              • tinfoilmattT Offline
                tinfoilmatt LAYER 8 @Pentium100
                last edited by

                ipsec30 interface does not have its own firewall tab. That's probably because I have this setting on:
                afbef87c-da7d-46e8-a926-8bafd9a8c0bc-image.png

                So then you're aware that that's literally the setting preventing you from using "per-interface rules, NAT, or reply-to" reliably.

                Maybe there's a way to make this behavior more reliable/consistent using static routing?

                P 1 Reply Last reply Reply Quote 0
                • P Offline
                  Pentium100 @tinfoilmatt
                  last edited by

                  @tinfoilmatt Changing the setting will break the policy-based VPNs.

                  I do not really need per-interface rules, as I just have allowed everything in the IPsec tab and it works correctly on the other router with an older pfsense version.

                  As I only have two such routers I cannot say if the other one works because of the version of something else.

                  tinfoilmattT 1 Reply Last reply Reply Quote 0
                  • tinfoilmattT Offline
                    tinfoilmatt LAYER 8 @Pentium100
                    last edited by

                    The issue is not the firewall ruleset.

                    1 Reply Last reply Reply Quote 0
                    • tinfoilmattT Offline
                      tinfoilmatt LAYER 8
                      last edited by

                      Why does this P2 require "Routed (VTI)" mode?

                      tinfoilmattT P 2 Replies Last reply Reply Quote 0
                      • tinfoilmattT Offline
                        tinfoilmatt LAYER 8 @tinfoilmatt
                        last edited by

                        And can we assume you've read this section of the documentation, IPsec Interface Assignment?

                        1 Reply Last reply Reply Quote 0
                        • P Offline
                          Pentium100 @tinfoilmatt
                          last edited by

                          @tinfoilmatt
                          Because using policy based it would be something like 30 phase2 entries and we had a problem where some of them would stop working at random.

                          Anyway, I found a solution to this.
                          Firewall State Policy - set to "Floating States" (default was "Interface Bound States" and apparently this default was different on the older pfsense version)
                          Now it seems to work

                          tinfoilmattT 1 Reply Last reply Reply Quote 0
                          • tinfoilmattT Offline
                            tinfoilmatt LAYER 8 @Pentium100
                            last edited by

                            If it works and it's messy, it still works.

                            P 1 Reply Last reply Reply Quote 0
                            • P Offline
                              Pentium100 @tinfoilmatt
                              last edited by

                              @tinfoilmatt Yeah. I have to use routed ipsec for this and also have to use policy-based tunnels. If there is no other way to make this work, then using floating states (like it is in the older version) seems like a solution.

                              tinfoilmattT 1 Reply Last reply Reply Quote 0
                              • tinfoilmattT Offline
                                tinfoilmatt LAYER 8 @Pentium100
                                last edited by

                                If there is no other way to make this work

                                I think there might be. I'd need to confirm for myself to be sure.

                                1 Reply Last reply Reply Quote 0
                                • First post
                                  Last post
                                Copyright 2026 Rubicon Communications LLC (Netgate). All rights reserved.