Site-to-site link is established but no traffic passes
-
Hi
I have a site-to-site connection between A and B. Until I relocated B, everything was working fine. I recently relocated B to another ip network. After the relocation, the link seems to be set up just fine, but no traffic can pass the link. IPSEC system log reports: ISAKMP-SA/IPsec-SA established on both ends.
From what I see and know, the only difference between before and after the move, is that B is now behind a firewall. This firewall allows in-/outbound UDP port 500 and protocol ESP/AH.
Have I forgotten something? Any ideas of what might be wrong?
Best regards, Søren Knudsen
-
I seem to have narrowed down the problem. When pinging from endpoint B, the request is received at endpoint A and a reply is made. The reply is never seen on endpoint B. Here's a snippet of A's packet capture while pinging from B.
13:21:12.579899 (authentic,confidential): SPI 0x0a2be574: (tos 0x0, ttl 64, id 56228, offset 0, flags [none], proto ICMP (1), length 84)
192.168.13.1 > 192.168.11.1: ICMP echo request, id 52622, seq 0, length 64
13:21:12.579968 (authentic,confidential): SPI 0x02f112cb: (tos 0x0, ttl 64, id 23756, offset 0, flags [none], proto ICMP (1), length 84, bad cksum 5b2 (->848a)!)
192.168.11.1 > 192.168.13.1: ICMP echo reply, id 52622, seq 0, length 6413:21:14.330006 (authentic,confidential): SPI 0x0a2be574: (tos 0x0, ttl 127, id 18791, offset 0, flags [none], proto ICMP (1), length 60)
192.168.13.100 > 192.168.11.3: ICMP echo request, id 1, seq 2442, length 40
13:21:14.330263 (authentic,confidential): SPI 0x02f112cb: (tos 0x0, ttl 127, id 5007, offset 0, flags [none], proto ICMP (1), length 60, bad cksum 8d7a (->8e7a)!)
192.168.11.3 > 192.168.13.100: ICMP echo reply, id 1, seq 2442, length 40Does this make sense?
-
Repeat the same capture on the LAN side of the destination. See if the packet leaves LAN but never returns.
If that is the case, which is likely, then you're looking at one of:
- The target has no gateway or does not use pfSense as its gateway
- The target has a subnet mask that makes it believe that the other side is "local" and not a distinct network
- The target has a local firewall blocking the traffic (e.g. Windows firewall, iptables, etc)
-
Repeat the same capture on the LAN side of the destination. See if the packet leaves LAN but never returns.
If that is the case, which is likely, then you're looking at one of:
- The target has no gateway or does not use pfSense as its gateway
- The target has a subnet mask that makes it believe that the other side is "local" and not a distinct network
- The target has a local firewall blocking the traffic (e.g. Windows firewall, iptables, etc)
I assume that by target, you mean a host on endpoint A's network. In the packet capture above, that would be 192.168.11.3.
This receives the ICMP request and an answer is transmitted from this host to endpoint A, which should relay this over the IPSEC tunnel, which seems to never happen - it is not seen in the packet capture on endpoint B at least.
Thanks for helping.
-
So in the captures you showed, taken on the IPsec interface, you saw both the request and the reply on both sides.
What do you see in captures on the LAN (or other internal interface) on both sides?
-
So in the captures you showed, taken on the IPsec interface, you saw both the request and the reply on both sides.
No. I saw both request and reply on the destination side - in the capture, that would be endpoint A (192.168.11.1). On the source side, I only saw the request.
What do you see in captures on the LAN (or other internal interface) on both sides?
I saw both request and reply on the destination side LAN. On the source side, I only saw the request.
-
OK, I thought the logs you showed were from both sides, not two separate pings from the same side.
On the destination side, in the LAN capture, ensure you have it set to "full" detail and make sure that the MAC address on the ping reply matches the MAC address of the firewall, to make sure that the client ping is actually getting picked up by the firewall and not something else. Just because it shows in the capture doesn't mean that it was actually coming to the firewall.
-
OK, I thought the logs you showed were from both sides, not two separate pings from the same side.
On the destination side, in the LAN capture, ensure you have it set to "full" detail and make sure that the MAC address on the ping reply matches the MAC address of the firewall, to make sure that the client ping is actually getting picked up by the firewall and not something else. Just because it shows in the capture doesn't mean that it was actually coming to the firewall.
Good point. It does show the correct MAC address in the reply.
The capture is performed on the IPSEC interface on the destination side. That the reply is seen here, to me indicates that it should go over the link.
Thanks, Søren
-
OK, so I'm a bit confused, earlier I thought you said you never saw the reply on the side with the target. I must have misread something somewhere.
What exactly do you see at each step? Source LAN, Source IPsec, Destination IPsec, Destination LAN.
-
OK, so I'm a bit confused, earlier I thought you said you never saw the reply on the side with the target. I must have misread something somewhere.
What exactly do you see at each step? Source LAN, Source IPsec, Destination IPsec, Destination LAN.
Host in source LAN network: 192.168.13.100
pfSense on source (B): 192.168.13.1
pfSense on destination (A): 192.168.11.1
Host in destination LAN network: 192.168.11.3Source LAN:
13:53:30.280851 00:15:5d:63:de:00 > 00:15:5d:63:de:04, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 128, id 20547, offset 0, flags [none], proto ICMP (1), length 60)
192.168.13.100 > 192.168.11.3: ICMP echo request, id 1, seq 3524, length 40Source IPsec:
13:53:30.280851 00:15:5d:63:de:00 > 00:15:5d:63:de:04, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 128, id 20547, offset 0, flags [none], proto ICMP (1), length 60)
192.168.13.100 > 192.168.11.3: ICMP echo request, id 1, seq 3524, length 40Destination IPsec:
14:54:39.386746 00:0a:5e:54:40:49 > 90:e6:ba:77:6b:eb, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 126, id 20547, offset 0, flags [none], proto ICMP (1), length 60)
192.168.13.100 > 192.168.11.3: ICMP echo request, id 1, seq 3524, length 40
14:54:39.386951 90:e6:ba:77:6b:eb > 00:0a:5e:54:40:49, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 128, id 1109, offset 0, flags [none], proto ICMP (1), length 60)
192.168.11.3 > 192.168.13.100: ICMP echo reply, id 1, seq 3524, length 40Destination LAN:
14:54:39.386746 00:0a:5e:54:40:49 > 90:e6:ba:77:6b:eb, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 126, id 20547, offset 0, flags [none], proto ICMP (1), length 60)
192.168.13.100 > 192.168.11.3: ICMP echo request, id 1, seq 3524, length 40
14:54:39.386951 90:e6:ba:77:6b:eb > 00:0a:5e:54:40:49, ethertype IPv4 (0x0800), length 74: (tos 0x0, ttl 128, id 1109, offset 0, flags [none], proto ICMP (1), length 60)
192.168.11.3 > 192.168.13.100: ICMP echo reply, id 1, seq 3524, length 40Is this more clear?
-
yes, much more clear.
Go to System > Advanced, Misc tab, uncheck Prefer Old IPsec SA.
-
yes, much more clear.
Go to System > Advanced, Misc tab, uncheck Prefer Old IPsec SA.
Good.
This setting was not checked on either end.
-
Are there multiple IPsec tunnels at each of these locations?
The only other thing that immediately springs to mind is that the reply traffic is going back over some other IPsec tunnel.
The traffic wouldn't show up in the IPsec capture unless it went to an active tunnel. The most common way that can happens is when Prefer old IPsec SA is on and there are "stale" SAs in place.
You might check Status > IPsec, SPDs tab, make sure there isn't something leftover there from a previous config attempt. Maybe even restart racoon under Status > Services.
-
Are there multiple IPsec tunnels at each of these locations?
The only other thing that immediately springs to mind is that the reply traffic is going back over some other IPsec tunnel.
Yes. Two other tunnels are defined at the destination network pfSense. What would make them take precedence?
I checked SPDs and all were fine here.
Thanks for all the pointers btw.
-
Are there multiple IPsec tunnels at each of these locations?
The only other thing that immediately springs to mind is that the reply traffic is going back over some other IPsec tunnel.
Yes. Two other tunnels are defined at the destination network pfSense. What would make them take precedence?
That would be: Two other Phase-1 entries are defined.
There are no other Phase-2 entries in the Phase-1 entry for this link.
-
So long as the networks do not overlap it should be OK, but if any of the tunnels that were entered first have a Phase 2 that would include the network from the other side (maybe one has 192.168.0.0/16, for example) it would break any tunnel after it.
The first match wins, taken by the order the tunnels were created. Phase 2's from all tunnels are considered, in the order of their respective Phase 1's.
-
So long as the networks do not overlap it should be OK, but if any of the tunnels that were entered first have a Phase 2 that would include the network from the other side (maybe one has 192.168.0.0/16, for example) it would break any tunnel after it.
The first match wins, taken by the order the tunnels were created. Phase 2's from all tunnels are considered, in the order of their respective Phase 1's.
Yes. That's what I thought. This is not the case.
What is also weird is that the link was working fine before moving endpoint B.
Would it be possible that a firewall between the two endpoints could block the link traffic, but still allow the link to be established? That seems to be the only difference.
-
Possible but not likely, it would have to only block the ESP traffic in one direction. Kind of an odd behavior.