Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    PFS 2.0.1 release - keepalive troubles

    Scheduled Pinned Locked Moved General pfSense Questions
    2 Posts 1 Posters 2.9k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S
      Stirlitz
      last edited by

      Hello all.

      I have not found any topic about my problem. If you know about it, sorry.

      I have installed 2.0.1 release after using 1.2.x many months. And on this week I found problem with TCP keepalive mechanism in pfSense when I tried to test 4G USB modem and to reduce keepalive intervals. After a little investigation I found that keepalive packets are wrong. No matter which hardware interface - on all types I have it.

      You can try it too. I set:
      net.inet.tcp.keepidle: 20000
      net.inet.tcp.keepintvl: 5000
      net.inet.tcp.always_keepalive: 1

      After it I run ftp-client on pfSense, connect to server by WAN-interface (server has ip 192.168.178.20), try to get file, but after one minute I have message "remote sever has closed connection". For a test I run a tcpdump with rule "port 21" before sending "mget *" command.

      And what I see:
      (I cut a ip-header info from log for easy reading. First 3 packets are from mget command.)

      12:06:29.510451 192.168.178.42.58459 > 192.168.178.20.21: Flags [P.], cksum 0xd54d (correct), seq 44:82, ack 63, win 520, options [nop,nop,TS val 149273 ecr 136613], length 38
12:06:29.511937 192.168.178.20.21 > 192.168.178.42.58459: Flags [P.], cksum 0x8e46 (correct), seq 63:88, ack 82, win 65403, options [nop,nop,TS val 136613 ecr 149273], length 25
12:06:29.512047 192.168.178.42.58459 > 192.168.178.20.21: Flags [.], cksum 0xb3d8 (correct), seq 82, ack 88, win 520, options [nop,nop,TS val 149273 ecr 136613], length 0

12:07:29.493967 192.168.178.42.58459 > 192.168.178.20.21: Flags [R.], cksum 0x9c64 (correct), seq 82, ack 88, win 520, options [nop,nop,TS val 155273 ecr 136613], length 0

      pfSense sends a RST packet after timeout of keepalive mechanism! After it, a server closes all connection with downloading stream too.
      I disabled a packet filtering, and saw what's wrong:

      11:49:28.415493 192.168.178.42.49283 > 192.168.178.20.21: Flags [P.], cksum 0x8b7f (correct), seq 44:82, ack 63, win 520, options [nop,nop,TS val 47239 ecr 126402], length 38
11:49:28.416913 192.168.178.20.21 > 192.168.178.42.49283: Flags [P.], cksum 0x4478 (correct), seq 63:88, ack 82, win 65403, options [nop,nop,TS val 126402 ecr 47239], length 25
11:49:28.417148 192.168.178.42.49283 > 192.168.178.20.21: Flags [.], cksum 0x6a0a (correct), seq 82, ack 88, win 520, options [nop,nop,TS val 47239 ecr 126402], length 0

11:49:48.412106 192.168.178.42.21 > 192.168.178.20.49283: Flags [.], cksum 0x496e (correct), seq 1173739157, ack 4115639386, win 520, length 0
11:49:48.412518 192.168.178.20.49283 > 192.168.178.42.21: Flags [R], cksum 0xc662 (correct), seq 4115639386, win 0, length 0

11:49:53.411460 192.168.178.42.21 > 192.168.178.20.49283: Flags [.], cksum 0x496e (correct), seq 0, ack 1, win 520, length 0
11:49:53.414050 192.168.178.20.49283 > 192.168.178.42.21: Flags [R], cksum 0xc662 (correct), seq 4115639386, win 0, length 0

11:49:58.410793 192.168.178.42.21 > 192.168.178.20.49283: Flags [.], cksum 0x496e (correct), seq 0, ack 1, win 520, length 0
11:49:58.411215 192.168.178.20.49283 > 192.168.178.42.21: Flags [R], cksum 0xc662 (correct), seq 4115639386, win 0, length 0

11:50:03.414151 192.168.178.42.21 > 192.168.178.20.49283: Flags [.], cksum 0x496e (correct), seq 0, ack 1, win 520, length 0
11:50:03.415462 192.168.178.20.49283 > 192.168.178.42.21: Flags [R], cksum 0xc662 (correct), seq 4115639386, win 0, length 0

11:50:08.409386 192.168.178.42.21 > 192.168.178.20.49283: Flags [.], cksum 0x496e (correct), seq 0, ack 1, win 520, length 0
11:50:08.409836 192.168.178.20.49283 > 192.168.178.42.21: Flags [R], cksum 0xc662 (correct), seq 4115639386, win 0, length 0

11:50:13.415556 192.168.178.42.21 > 192.168.178.20.49283: Flags [.], cksum 0x496e (correct), seq 0, ack 1, win 520, length 0
11:50:13.415951 192.168.178.20.49283 > 192.168.178.42.21: Flags [R], cksum 0xc662 (correct), seq 4115639386, win 0, length 0

11:50:18.414901 192.168.178.42.21 > 192.168.178.20.49283: Flags [.], cksum 0x496e (correct), seq 0, ack 1, win 520, length 0
11:50:18.415320 192.168.178.20.49283 > 192.168.178.42.21: Flags [R], cksum 0xc662 (correct), seq 4115639386, win 0, length 0

11:50:23.414344 192.168.178.42.21 > 192.168.178.20.49283: Flags [.], cksum 0x496e (correct), seq 0, ack 1, win 520, length 0
11:50:23.424502 192.168.178.20.49283 > 192.168.178.42.21: Flags [R], cksum 0xc662 (correct), seq 4115639386, win 0, length 0

11:50:28.412509 192.168.178.42.49283 > 192.168.178.20.21: Flags [R.], cksum 0x5296 (correct), seq 82, ack 88, win 520, options [nop,nop,TS val 53239 ecr 126402], length 0

      When a tcp.keepidle is expired, pfSense sends a packet, but source and destination ports are swapped! And server returns a RST packet, because incoming packet is wrong. So, after 8 intervals pfSnese closes a socket. And packet for closing is OK. Very strange.

      I have tested FreeBSD 8.1 (because pfSense is based on it), but FreeBSD works fine! Here is log:

      15:26:45.617423 192.168.178.44.26215 > 192.168.178.20.ftp: Flags [P.], seq 52:90, ack 82, win 8326, options [nop,nop,TS val 29720 ecr 112737], length 38
15:26:45.618941 192.168.178.20.ftp > 192.168.178.44.26215: Flags [P.], cksum 0x7895 (correct), seq 82:107, ack 90, win 65367, options [nop,nop,TS val 112737 ecr 29720], length 25
15:26:45.712863 192.168.178.44.26215 > 192.168.178.20.ftp: Flags [.], cksum 0xe5b8 (incorrect -> 0x7f7b), seq 90, ack 107, win 8326, options [nop,nop,TS val 29730 ecr 112737], length 0

15:27:05.615989 192.168.178.44.26215 > 192.168.178.20.ftp: Flags [.], cksum 0xe5ac (incorrect -> 0xe518), seq 89, ack 107, win 8326, length 0
15:27:05.616572 192.168.178.20.ftp > 192.168.178.44.26215: Flags [.], cksum 0x9fe1 (correct), seq 107, ack 90, win 65367, options [nop,nop,TS val 112937 ecr 29730], length 0

15:27:25.623195 192.168.178.44.26215 > 192.168.178.20.ftp: Flags [.], cksum 0xe5ac (incorrect -> 0xe518), seq 89, ack 107, win 8326, length 0
15:27:25.624459 192.168.178.20.ftp > 192.168.178.44.26215: Flags [.], cksum 0x9f19 (correct), seq 107, ack 90, win 65367, options [nop,nop,TS val 113137 ecr 29730], length 0

      Each 20 seconds (keepidle time) client sends a packet and socket stays open.

      pfSense 1.2.3 works fine with keealive too. What I must change for correct work here? I have only one idea - set keepidle=14400000 (4 hours) for big timer in pfSense and sending keealive packets from other side, where timer will expire faster.

      PS: I have tested it on VMware after I found it on PC-route. In pfSense I tried to disable/enable all hardware offloading in Advanced-Networking settings. No matter. I have the same result. But with disabling of all offloading the checksums are correct.

      1 Reply Last reply Reply Quote 0
      • S
        Stirlitz
        last edited by

        Can anybody test it? It's my problem or pfSense release? For testing you can connect to ftp server only, without downloading. If problem exists, after you setted these sysctl enviroments, you will be disconnected from sever after ~one minute. Wait a little more and try to send other command like "dir".

        1 Reply Last reply Reply Quote 0
        • First post
          Last post
        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.