Possible IPSec routing issue

Phatsta

I'm a long time user of PFsense and the go-to-guy for a lot of installations. Usually whenever there's a problem, I know it's something I overlooked or misunderstood, because there is rarely any real issues with PFsense. However, this time I'm not so sure, and because I'm pretty sure I tested everything, I just want to use the help of those more experienced than me just in case I did overlook something. Humbleness first, right! :)

Trying to connect a customer with one of our networks for using Veeam over SMB to backup some data through IPSec. I can ping from us to customer server, but not the other way around. I can ping our PFsense from customer server but anything beyond that is considered internet (as far as I can tell).

Customer side:
PFsense 2.4.4 p3: 192.168.75.1/24
Firewall rule for IF IPSec: Any **** (for troubleshooting purposes)
IPSec tunnel is up and running
Veeam server (Win 2019): 192.168.75.254/24 (Win FW turned off)

Our side:
PFsense 2.4.2 p1: 10.10.0.251/24 (not default gw)
Firewall rule for IF IPSec: Any **** (for troubleshooting purposes)
IPSec tunnel is up and running
Backup repository (Win 2016): 10.10.0.2/24 (Win FW turned off)
On this side Win server has permanent route: IP:192.168.75.0 Mask:255.255.255.0 GW:10.10.0.251 Metric:1

Tracert from 192.168.75.254:
Tracing route to 10.10.0.251 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms 192.168.75.1
2 26 ms 26 ms 27 ms 10.10.0.251

Tracing route to 10.10.0.2 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms 192.168.75.1
2 * * * Request timed out.
3 * * * Request timed out.
4 * * * Request timed out.

Like I said, from the other side all is well:
Tracing route to 192.168.75.254 over a maximum of 30 hops
1 <1 ms <1 ms <1 ms 10.10.0.251
2 * * * Request timed out.
3 27 ms 26 ms 26 ms 192.168.75.254

I'm at a loss. If I'm being stupid, just lay it on me.

Phatsta

Bump.

Phatsta

Here is routing table on customer Windows machine:

C:\Windows\system32>route print
===========================================================================
Interface List
  7...00 0c 29 d0 a5 79 ......Intel(R) 82574L Gigabit Network Connection
  1...........................Software Loopback Interface 1
===========================================================================

IPv4 Route Table
===========================================================================
Active Routes:
Network Destination        Netmask          Gateway       Interface  Metric
          0.0.0.0          0.0.0.0     192.168.75.1   192.168.75.254    281
        10.10.0.0    255.255.255.0      10.10.0.251   192.168.75.254     26
        127.0.0.0        255.0.0.0         On-link         127.0.0.1    331
        127.0.0.1  255.255.255.255         On-link         127.0.0.1    331
  127.255.255.255  255.255.255.255         On-link         127.0.0.1    331
     192.168.75.0    255.255.255.0         On-link    192.168.75.254    281
   192.168.75.254  255.255.255.255         On-link    192.168.75.254    281
   192.168.75.255  255.255.255.255         On-link    192.168.75.254    281
        224.0.0.0        240.0.0.0         On-link         127.0.0.1    331
        224.0.0.0        240.0.0.0         On-link    192.168.75.254    281
  255.255.255.255  255.255.255.255         On-link         127.0.0.1    331
  255.255.255.255  255.255.255.255         On-link    192.168.75.254    281
===========================================================================
Persistent Routes:
  Network Address          Netmask  Gateway Address  Metric
          0.0.0.0          0.0.0.0     192.168.75.1  Default
===========================================================================

IPv6 Route Table
===========================================================================
Active Routes:
 If Metric Network Destination      Gateway
  1    331 ::1/128                  On-link
  7    281 fe80::/64                On-link
  7    281 fe80::a86a:5e7e:5151:3a69/128
                                    On-link
  1    331 ff00::/8                 On-link
  7    281 ff00::/8                 On-link
===========================================================================
Persistent Routes:
  None

And routing table on customer PFsense:

IPv4 Routes
Destination	Gateway	Flags	Use	Mtu	Netif	Expire
default	        *wan-ip*	UGS	124614232	1500	igb0	
10.135.3.0/24	10.135.3.2	UGS	4845	1500	ovpns1	
10.135.3.1	link#10	UHS	0	16384	lo0	
10.135.3.2	link#10	UH	587101	1500	ovpns1	
*our-gw-ip*	*wan-ip*	UGHS	670368	1500	igb0	
127.0.0.1	link#6	UH	35967849	16384	lo0	
192.168.1.0/24	link#2	U	21674	1500	igb1	
192.168.1.1	link#2	UHS	0	16384	lo0	
192.168.75.0/24	link#9	U	169108234	1500	bridge0	
192.168.75.1	link#9	UHS	0	16384	lo0	
*wan-nw*	link#1	U	9251379	1500	igb0	
*wan-gw*	link#1	UHS	0	16384	lo0

And interface list of customer PFsense:

WAN   up   100baseTX <full-duplex>   *wan-ip*
IF1   up   1000baseTX <full-duplex>   192.168.1.1
IF2   down autoselect                 n/a
IF3   up   1000baseTX <full-duplex>   n/a
LAN   up   <bridge IF1-3>             192.168.75.1

Phatsta

@jimp or anyone?

lfoerster

@Phatsta said in Possible IPSec routing issue:

Tracing route to 10.10.0.2 over a maximum of 30 hops

Keep in mind that only 10.10.0.251 has a dedicated route to the .75.0 /24 network and most likely 10.10.0.2 has NOT !
Thats why the pings to .2 end up in nirwana.

This is a general problem when you put static routes to end devices where they should NOT be !!
Those static routes belong to the router ! Routers should route in a network and not end devices !

So its more intelligent to place the static route NOT on 10.10.0.251 here, but on the default gateway both .251 and .2 devices (and probably all in the 10.10.0.0 segment) have configured.
This is most likely a layer 3 routing switch or the default router on the 10.10.0.0 /24 segment.
Here, on this gateway you need to set the static route and this will cover than ALL 10.10.0.0 devices who have this gateway as the default gateway !
Thats the right way to set this up and will fix your problem instantly !

Phatsta

@lfoerster I would love to place the static routes in the routers, but it is not possible to put a static route without adding a gateway to route through, and it's not possible to add a gateway from a not-connected network in PFsense. This is why I've had no choice but to put the static routes at the end device as I see it?

I'll have another look at this and consider what you said. Thank you for your input.

Phatsta

@lfoerster said in Possible IPSec routing issue:

So its more intelligent to place the static route NOT on 10.10.0.251 here, but on the default gateway both .251 and .2 devices (and probably all in the 10.10.0.0 segment) have configured.

Of course you were correct. I put a static route in the default GW on our side and it started to work immediately.

I do have to admit it gets me a little confused though, since I've been using static routes on clients before. And, while it says it's a headache when administering multiple clients (which we don't), this article says it should work:
https://docs.netgate.com/pfsense/en/latest/book/ipsec/site-to-site.html

Anyway, thanks a million!