Routing across an IPSec tunnel
-
I have an issue I'm not quite able to deal with, I've spent many hours trying to debug it but to no avail...
Here is the layout of my networks:
laptop 10.25.0.51 | Site A 10.25.0.0/16 pfsense A 10.25.0.1 on LAN ; 10.200.0.1 on IPSEC_VTI || || IPSec || || pfsense B 10.101.0.1 on LAN ; 10.200.0.2 on IPSEC_VTI Site B 10.101.0.0/16 | rocket 10.101.50.11
Nothing special here, the routed IPSec works fine (I had to remove some disabled phase 2 items to make it work btw) and it's possible to reach the servers in Site B from the Site A.
However, I need the possibility to access the server rocket inside Site B (
10.101.50.11
) which acts as a gateway to get access to a set of remote machines (called satellites) that are accessible using fake IPs on the network172.25.0.0/16
. For each one of these IPs, the gateway does some NAT and firewalling before sending the packets to the given satellite.I thought that adding a static route to send the packets destined to the fake IPs to the gateway would be enough but I can't make it work...
My static routing tables on the pfsense site A:
10.101.0.0/16 -> 10.200.0.2 172.25.0.0/16 -> 10.200.0.2
And on site B:
10.25.0.0/16 -> 10.200.0.1 172.25.0.0/16 -> 10.101.50.11
So far, through investigation, I've deduced the following:
- The tunnel starts, and it's possible to access machines on site B from site A (and vice versa). It's for instance possible to directly access rocket using the IP
10.101.50.11
. - If I ssh into the pfsense B, and run
nc 172.35.0.1 5555
, I get some output intcpdump -i any port 5555
on rocket.
=> The firewall uses the routing table, and is able to send the packets to rocket. - If I ssh into the pfsense A, and run
nc 172.35.0.1 5555
, the packets show up on the firewall log in pfsense B but there is no output in tcpdump on rocket. Same if I try to runnc
from a laptop in Site A.
=> It looks as if somehow after going through the IPSec, the packets do not follow the static routing but instead get lost by the pfsense B. - I tried the same setup using a policy-based IPSec with exactly the same results.
- I also tried swapping the static route
172.25.0.0/16 -> 10.101.50.11
on the pfSense B with quick floating rules : any packet that have172.25.0.0/16
as destination, should be allowed and use10.101.50.11
as a gateway, for both thein
andout
directions. Unfortunately, it still does not work.
On both sides, I have pfSense 2.6.0. In Site A the hardware is a dedicated firewall, and on Site B, it's a virtual machine hosted by a cloud provider.
The weirdest thing is that I have inherited a legacy infrastructure that works perfectly with basically the same setup: Site A is the same firewall as above and the equivalent of site B has a dedicated hardware firewall running pfSense 2.5.2.
- The tunnel starts, and it's possible to access machines on site B from site A (and vice versa). It's for instance possible to directly access rocket using the IP
-
So right now you cant access rocket over the S2S?
-
@michmoor I can but only using its own IP (for ssh-ing for instance).
If I try to ping/netcat a satellite (ie 172.25.0.1) through the S2S it gives me nothing. The last I hear about the packets is if I run a packet capture on the Site B Lan interface, then they disappear and rocket gets nothing.
Here is my log from the capture:
00:52:42.283626 IP 10.25.0.51.49247 > 172.25.0.1.5555: tcp 0 00:52:43.284518 IP 10.25.0.51.49247 > 172.25.0.1.5555: tcp 0 00:52:44.281234 IP 10.25.0.51.49247 > 172.25.0.1.5555: tcp 0
-
@corentin-deboisset
so you can ssh to 10.101.50.11 and that works.but you try to ssh to 172.25.0.1 from site A and that doesnt work? Im not following where the satellites come into play.
-
@michmoor 172.25.0.1 is a virtual IP for the satellite, it's routed to rocket which handles the communication with the satellite.
Indeed, if I send packets destined to the the satellite from pfsense B they arrive to rocket and get to their destination, but from site A they don't even reach rocket..