Site to site IPsec suspect not passing TCP traffic

graciejane

Hi,

I've been testing and googling this for hours but the keywords produce a lot of irrelevant results. Maybe I need to get better at searching :)

Anyway... Site to site VPN using IPsec on pfSense. pfSense routers both ends. At one end the pfSense router is not the default gateway but it accessed from WAN by a port forward rule and the VPN traffic from LAN by a route. The tunnel comes up and I can ping from any host on the remote LAN to any host on the local LAN and vice versa. End to end RDP sessions work great. Here comes the issue...

SSH and HTTP/HTTPS connections fail. Not denied, but time out as if there's no response at all from the other end. Ping and RDP works fine. I believe RDP can use UDP as well as TCP, so I'm wondering if the VPN is simply not letting TCP traffic through. Is there a way to prove this? It sounds like a firewall issue but I've disabled the firewalls where possible and created pass all rules on the ones I can't disable. Makes no difference. SSH and websites work fine using SSH port forwarding from my local PC, so I know the actual services I'm trying to reach through the VPN are online. This is all well and good but it won't help them talk to each other directly, hence creating a site to site VPN.

Oh.. RDP works in both directions if that helps. I can't test SSH or HTTP the other way around.

Any hints on what I can test/eliminate are much appreciated.

Thank you.

graciejane

I have set up a web server locally just for testing. Accessing it from the remote LAN works fine. The screenshot below shows an RDP session to a Windows 10 machine on the remote LAN via the VPN, which is in turn accessing the web server on the local LAN via the VPN. It does not work in the other direction.

Local network = 10.18.6.0/24
Remote network = 10.16.3.0/24

There goes my theory that it's not passing TCP traffic. This has to be a firewall issue right?

graciejane

SSH remote to local works too. Here's what works and what doesn't:

Local --> remote:
Ping: yes
SSH: no
HTTP: no
RDP: yes

Remote --> local:
Ping: yes
SSH: yes
HTTP: yes
RDP: yes

scurrier

I am having issues with my setup as well and they lead to intermittent connectivity. Sometimes things work, sometimes not. Mostly not. They kind of sound like what you're experiencing.

One thing to consider is that packet size and fragmentation issues can lead to what you're seeing. You might be loading a big web page in one direction which leads to large packets that have a fragmentation problem. In the other direction you might load a small test page with small packets and no fragmentation problem.

Similarly, ssh might work when you're doing small things but then if you do something that results in a lot of text being sent, the packet size grows and now you have the fragmentation issue again.

graciejane

Hey thanks for that. Yeah I've been reading about packet size and MTU related issues. Thing is, mine either work perfectly or not at all, consistently. There's nothing intermittent about it. This is an SSH connection attempt (with a ping thrown in for good measure):

Pinging 10.16.3.20 with 32 bytes of data:
Reply from 10.16.3.20: bytes=32 time=244ms TTL=61
Reply from 10.16.3.20: bytes=32 time=261ms TTL=62
Reply from 10.16.3.20: bytes=32 time=243ms TTL=62
Reply from 10.16.3.20: bytes=32 time=246ms TTL=62

Ping statistics for 10.16.3.20:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 243ms, Maximum = 261ms, Average = 248ms

C:\Users\grace>ssh grace@10.16.3.20
ssh: connect to host 10.16.3.20 port 22: Connection timed out

I think I might have to learn how to use wireshark.

graciejane

Wireshark has revealed the problem. Packets are arriving with incorrect checksums when the following conditions are met:

Protocol is TCP
The packet must pass through the IPsec tunnel in a particular direction
The source OS is Linux (possibly a particular kernel, don't know)

Change any of those three things and the checksums are good.

The 3-way handshake is failing because my PC is not responding to the SYN-ACK packets with bad checksums.

This is what good looks like (HTTP server running on Windows):

So now I know what's happening, but not why. Not sure what to try from here.

scurrier

@graciejane wow, well done. I can't imagine what the cause could be.

graciejane

I'm baffled too. I need to put this down for now as have other priorities. Maybe coming back to it with a fresh mind after a few days will help.

scurrier

FYI, my issue turned out to be this IPsec Advanced Setting on my SG-1100:

The time I wasted to find out that that checkbox might as well say " break my SH!# "...

graciejane

Good to hear you found the problem.

I solved my one too. The OS being Linux wasn't actually one of the required conditions, but it was a big clue. The real problem was the type of virtual NIC (the remote hosts are KVM virtual machines). The default NIC for Linux is VirtIO vs an emulation of a real NIC - the Intel E1000 - for Windows. Swap out the VirtIO for an E1000 on the Linux hosts and problem solved!

mkit

@scurrier That check box caused me some issues as well. Thanks for sharing. My Netgate could connect to my Sonicwall but not from Sonicwall to Netgate.

taxiarxos

Hello All,

I am having exactly the same issue but when I enable or disable this check box at VPN IPSec Advanced I am still not able to use ssh or http/https. Any other ideas?

Thank you in advance.