Conenction to linux box dies over ipsec

  • set up a new department with new pfsense and at first it seemed all is working.
    first i give will you a overview of our net structure, we have a server net with a pfsense firewall and from there all departments are connected via internet over ipsec.
    the new department is
    all seamed to work fine at first until i set up my a unifi ap, i could not talk to the unifi controller on the server net, it gives timeout errors, i can ping unifi from the ap and ping the ap from the controller.
    ive set any any rules on ipsec interface, seems nothing is blocked.
    next thing i tried to ssh from a server into a linux box and ecery time i connect via ssh and run a command, eg ls the connection dies.
    i have included wireshark dumps from both the pfsense box and pfsense box, where i try to ssh and run ls and it dies. i can see tcp retransmission errors in the logs.

    hope someone will help, there is some free beer to the one who solves it :)


  • could it be a MTU issue, or maybe MTU over ipsec?

  • @jca1981:

    could it be a MTU issue, or maybe MTU over ipsec?

    IP is designed to tolerate different MTUs, especially with TCP, as would be used with SSH.  When a TCP connection is set up, the MTU is depended on the smallest MSS at either end.  Then, if a packet is still too small, it will be fragmented on IPv4.

  • soo, you are saying that it most likley is not MTU.
    any idea what i could try to resolve it?

  • One very useful technique is to examine the traffic, to see what's happening.  This can be done the the pfSense packet capture or Wireshark.  Once you understand what the problem is and where, you can fix it.

  • yea, i have included downloaderble wireshark pcap in this thread, would you mind looking at them to see if you can figure out what is happening, ive tried and like i wrote i found retransmision and malformed packet, but i am not a wireshark master :)

  • In the from I see some 1512 byte packets, with a 1448 packet size.  What is the MTU on IPSec?  You can try pinging with different size packets, to see where it fails.  Use the -s option for this.  Also, I see the do not fragment flag is set, which means path MTU discovery is used, instead of fragmenting.  This should generate some "too big" ICMP messages, but I don't see those.  Have they been filtered out?

  • im have not set mtu on ipsec so i guess it is using 1400, to set mtu on ipsec mss clamping needs to be enabled, right?
    biggest MTU i can get now with ping -s is 1410.
    ive now tried enabling "do not fragment" and "firewall scrub" under system/advanced/firewall & nat, not sure if there is a specific place only for ipsec.

  • ok, so tried setting mss clamping to 1300 and now it works, thanks, triet upping it to 1400 and it stopped working, found 1380 to be working after trying som more,

  • Linux normally uses PMTUD to set packet size.  Do you see the ICMP "too big" messages?  I'm not sure about IPSec settings, as I haven't used IPSec with pfSense.  The MSS is normally used when setting up a TCP connection to tell the other end the maximum supported packet size.  It has nothing to do with any router, including pfSense.  It is PMTUD that's used to determine the maximum packet size that will fit the smallest MTU along the path.

Log in to reply