Site to site IPsec suspect not passing TCP traffic



  • Hi,

    I've been testing and googling this for hours but the keywords produce a lot of irrelevant results. Maybe I need to get better at searching :)

    Anyway... Site to site VPN using IPsec on pfSense. pfSense routers both ends. At one end the pfSense router is not the default gateway but it accessed from WAN by a port forward rule and the VPN traffic from LAN by a route. The tunnel comes up and I can ping from any host on the remote LAN to any host on the local LAN and vice versa. End to end RDP sessions work great. Here comes the issue...

    SSH and HTTP/HTTPS connections fail. Not denied, but time out as if there's no response at all from the other end. Ping and RDP works fine. I believe RDP can use UDP as well as TCP, so I'm wondering if the VPN is simply not letting TCP traffic through. Is there a way to prove this? It sounds like a firewall issue but I've disabled the firewalls where possible and created pass all rules on the ones I can't disable. Makes no difference. SSH and websites work fine using SSH port forwarding from my local PC, so I know the actual services I'm trying to reach through the VPN are online. This is all well and good but it won't help them talk to each other directly, hence creating a site to site VPN.

    Oh.. RDP works in both directions if that helps. I can't test SSH or HTTP the other way around.

    Any hints on what I can test/eliminate are much appreciated.

    Thank you.



  • I have set up a web server locally just for testing. Accessing it from the remote LAN works fine. The screenshot below shows an RDP session to a Windows 10 machine on the remote LAN via the VPN, which is in turn accessing the web server on the local LAN via the VPN. It does not work in the other direction.

    Local network = 10.18.6.0/24
    Remote network = 10.16.3.0/24

    There goes my theory that it's not passing TCP traffic. This has to be a firewall issue right?

    86b11a33-a80f-4c66-a50b-f6725d9b923d-image.png



  • SSH remote to local works too. Here's what works and what doesn't:

    Local --> remote:
    Ping: yes
    SSH: no
    HTTP: no
    RDP: yes

    Remote --> local:
    Ping: yes
    SSH: yes
    HTTP: yes
    RDP: yes



  • I am having issues with my setup as well and they lead to intermittent connectivity. Sometimes things work, sometimes not. Mostly not. They kind of sound like what you're experiencing.

    One thing to consider is that packet size and fragmentation issues can lead to what you're seeing. You might be loading a big web page in one direction which leads to large packets that have a fragmentation problem. In the other direction you might load a small test page with small packets and no fragmentation problem.

    Similarly, ssh might work when you're doing small things but then if you do something that results in a lot of text being sent, the packet size grows and now you have the fragmentation issue again.



  • Hey thanks for that. Yeah I've been reading about packet size and MTU related issues. Thing is, mine either work perfectly or not at all, consistently. There's nothing intermittent about it. This is an SSH connection attempt (with a ping thrown in for good measure):

    Pinging 10.16.3.20 with 32 bytes of data:
    Reply from 10.16.3.20: bytes=32 time=244ms TTL=61
    Reply from 10.16.3.20: bytes=32 time=261ms TTL=62
    Reply from 10.16.3.20: bytes=32 time=243ms TTL=62
    Reply from 10.16.3.20: bytes=32 time=246ms TTL=62
    
    Ping statistics for 10.16.3.20:
        Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
    Approximate round trip times in milli-seconds:
        Minimum = 243ms, Maximum = 261ms, Average = 248ms
    
    C:\Users\grace>ssh grace@10.16.3.20
    ssh: connect to host 10.16.3.20 port 22: Connection timed out
    

    I think I might have to learn how to use wireshark.



  • Wireshark has revealed the problem. Packets are arriving with incorrect checksums when the following conditions are met:

    • Protocol is TCP
    • The packet must pass through the IPsec tunnel in a particular direction
    • The source OS is Linux (possibly a particular kernel, don't know)

    Change any of those three things and the checksums are good.

    The 3-way handshake is failing because my PC is not responding to the SYN-ACK packets with bad checksums.

    63dd8cca-7a41-48f2-bb82-8a0250a3de12-image.png

    This is what good looks like (HTTP server running on Windows):

    5c869d38-fa3e-4494-abd5-ee166ca5609b-image.png

    So now I know what's happening, but not why. Not sure what to try from here.



  • @graciejane wow, well done. I can't imagine what the cause could be.



  • I'm baffled too. I need to put this down for now as have other priorities. Maybe coming back to it with a fresh mind after a few days will help.



  • FYI, my issue turned out to be this IPsec Advanced Setting on my SG-1100:
    b1af9ed6-6a8f-417d-9c08-aad4d9e6730c-image.png

    The time I wasted to find out that that checkbox might as well say " break my SH!# "...



  • Good to hear you found the problem.

    I solved my one too. The OS being Linux wasn't actually one of the required conditions, but it was a big clue. The real problem was the type of virtual NIC (the remote hosts are KVM virtual machines). The default NIC for Linux is VirtIO vs an emulation of a real NIC - the Intel E1000 - for Windows. Swap out the VirtIO for an E1000 on the Linux hosts and problem solved!


Log in to reply