Routing across an IPSec tunnel

[corentin.deboisset]

I have an issue I'm not quite able to deal with, I've spent many hours trying to debug it but to no avail...

Here is the layout of my networks:

laptop         10.25.0.51
 |
Site A         10.25.0.0/16
pfsense A      10.25.0.1 on LAN ; 10.200.0.1 on IPSEC_VTI
 ||
 ||
IPSec
 ||
 ||
pfsense B      10.101.0.1 on LAN ; 10.200.0.2 on IPSEC_VTI
Site B         10.101.0.0/16
 |
rocket         10.101.50.11

Nothing special here, the routed IPSec works fine (I had to remove some disabled phase 2 items to make it work btw) and it's possible to reach the servers in Site B from the Site A.

However, I need the possibility to access the server rocket inside Site B (10.101.50.11) which acts as a gateway to get access to a set of remote machines (called satellites) that are accessible using fake IPs on the network 172.25.0.0/16. For each one of these IPs, the gateway does some NAT and firewalling before sending the packets to the given satellite.

I thought that adding a static route to send the packets destined to the fake IPs to the gateway would be enough but I can't make it work...

My static routing tables on the pfsense site A:

10.101.0.0/16 -> 10.200.0.2
172.25.0.0/16 -> 10.200.0.2

And on site B:

10.25.0.0/16  -> 10.200.0.1
172.25.0.0/16 -> 10.101.50.11

So far, through investigation, I've deduced the following:

• The tunnel starts, and it's possible to access machines on site B from site A (and vice versa). It's for instance possible to directly access rocket using the IP 10.101.50.11.
• If I ssh into the pfsense B, and run nc 172.35.0.1 5555, I get some output in tcpdump -i any port 5555 on rocket.
  => The firewall uses the routing table, and is able to send the packets to rocket.
• If I ssh into the pfsense A, and run nc 172.35.0.1 5555, the packets show up on the firewall log in pfsense B but there is no output in tcpdump on rocket. Same if I try to run nc from a laptop in Site A.
  => It looks as if somehow after going through the IPSec, the packets do not follow the static routing but instead get lost by the pfsense B.
• I tried the same setup using a policy-based IPSec with exactly the same results.
• I also tried swapping the static route 172.25.0.0/16 -> 10.101.50.11 on the pfSense B with quick floating rules : any packet that have 172.25.0.0/16 as destination, should be allowed and use 10.101.50.11 as a gateway, for both the in and out directions. Unfortunately, it still does not work.

On both sides, I have pfSense 2.6.0. In Site A the hardware is a dedicated firewall, and on Site B, it's a virtual machine hosted by a cloud provider.

The weirdest thing is that I have inherited a legacy infrastructure that works perfectly with basically the same setup: Site A is the same firewall as above and the equivalent of site B has a dedicated hardware firewall running pfSense 2.5.2.

[michmoor]

So right now you cant access rocket over the S2S?

[corentin.deboisset]

@michmoor I can but only using its own IP (for ssh-ing for instance).

If I try to ping/netcat a satellite (ie 172.25.0.1) through the S2S it gives me nothing. The last I hear about the packets is if I run a packet capture on the Site B Lan interface, then they disappear and rocket gets nothing.

Here is my log from the capture:

00:52:42.283626 IP 10.25.0.51.49247 > 172.25.0.1.5555: tcp 0
00:52:43.284518 IP 10.25.0.51.49247 > 172.25.0.1.5555: tcp 0
00:52:44.281234 IP 10.25.0.51.49247 > 172.25.0.1.5555: tcp 0

[michmoor]

@corentin-deboisset
so you can ssh to 10.101.50.11 and that works.

but you try to ssh to 172.25.0.1 from site A and that doesnt work? Im not following where the satellites come into play.

[corentin.deboisset]

@michmoor 172.25.0.1 is a virtual IP for the satellite, it's routed to rocket which handles the communication with the satellite.

Indeed, if I send packets destined to the the satellite from pfsense B they arrive to rocket and get to their destination, but from site A they don't even reach rocket..