Routing traffic towards another pfsense firewall which is terminating an IPSEC tunnel

[steve.scotter]

I’m trying to redirect traffic destined for another site across an IPSEC tunnel handled by a separate pair of pfsense boxes. Here’s a simplified copy of my topology.

All firewalls are running pfsense 2.7 (I’m aware 2.7.1 is available and I’ll update in due course).

Firewall A1 and B1 are default gateways at both sites.
Firewall A2 and B2 have a IPSEC tunnel configured between the two sites. I’m satisfied the IPSEC tunnel is configured correctly as when I set the default gateway for Server A to FW A2 and Server B to FW B2 I can successfully communicate between the two servers using both ICMP and TCP protocols (and probably UDP, but I didn’t check).

Firewalls A1 has a gateway configured which points to the firewall A2 IP address.

Firewalls B1 has a gateway configured which points to the firewall B2 IP address.

FW A1 has a static route configured to use FW A2 for traffic destined to 172.16.0.0/16

FW B1 has a static route configured to use FW B2 for traffic destined to 192.168.0.0/16

If I attempt to ping from Server A to Server B it works, but I get Redirect Host(New nexthop: 192.168.0.241) responses. Eg…

root@ServerA:~# ping 172.16.0.20
PING 172.16.0.20 (172.16.0.20) 56(84) bytes of data.
From 192.168.0.240: icmp_seq=1 Redirect Host(New nexthop: 192.168.0.241)
64 bytes from 172.16.0.20: icmp_seq=1 ttl=126 time=5.32 ms
From 192.168.0.240: icmp_seq=2 Redirect Host(New nexthop: 192.168.0.241)
64 bytes from 172.16.0.20: icmp_seq=2 ttl=126 time=5.15 ms
From 192.168.0.240: icmp_seq=3 Redirect Host(New nexthop: 192.168.0.241)
64 bytes from 172.16.0.20: icmp_seq=3 ttl=126 time=5.51 ms
From 192.168.0.240: icmp_seq=4 Redirect Host(New nexthop: 192.168.0.241)
64 bytes from 172.16.0.20: icmp_seq=4 ttl=126 time=5.25 ms
From 192.168.0.240: icmp_seq=5 Redirect Host(New nexthop: 192.168.0.241)

Any TCP based connection (eg, SSH, HTTP, HTTPS) doesn’t work.

If I temporarily local static routes to Server A (ip r add 172.16.0.0/16 via 192.168.0.241) and to Server B (ip r add 192.168.0.0/16 via 172.16.0.241) then I can ping without getting nexthop responses AND TCP based connections work.

My research suggests the route cause of this will almost certainly be something to do with asymmetric routing, but having followed the advice in the following two pages (e.g. ticking Bypass firewall rules for traffic on the same interface and adding firewall rules on the LAN and floating interfaces for any TCP flag and sloppy state type) I've been unable to find working configuration.

https://docs.netgate.com/pfsense/en/latest/routing/static.html
https://docs.netgate.com/pfsense/en/latest/troubleshooting/asymmetric-routing.html

The traffic between these two sites is semi temporarily for a migration of services. As a result I really don't want to have to add a local routing entry to all the servers in sites A and B to allow the inter site communication to go over the IPSEC tunnel between FW A2 and FW B2. Setting up an IPSEC tunnel between FW A1 and FW B1 is not possible for reasons I can't go into.

[steve.scotter]

I didn't get any replies, but I did find a solution myself which I thought I'd share in case anyone else find them selves in a similar situation and lands on this post.

My solution was to

• Add an additional VLAN interfaces to firewalls FWA1 and FWB1 (the default gateways) and give them IP addresses of 192.168.1.240/24 and 172.16.1.240/240 respectively.
• Reconfigure firewalls FWA2 and FWB2 to have the IP addresses of 192.168.1.241/24 and 172.16.1.241/24 respectively.
• Reconfigure the VLAN configuration on the network switches (not shown on the diagram as that's a layer 3 diagram, but the connectivity between FWA1 <->FWA2 and FWB1 <-> FWB2 goes via separate managed switches at each site)
• Confirm connectivity between FWA1 <->FWA2 and FWB1 <-> FWB2 worked using ICMP pings.
• Configure a gateway gateway on FWA1 of FWA2 new IP
• Configure a gateway gateway on FWB1 of FWB2 new IP
• Configure a static route on FWA1 for 172.20.0.0/16 to route towards FWA2.
• Configure a static route on FWB1 for 192.168.0.0/16 to route towards FWB2.

After that I was able to communicate between Server A and Server B without adding any local static routes to the endpoints.

[keyser]

@steve-scotter That is technically also better network design. But you could have made it work by simply enabling the bullet called “Static Route Filtering” in SYSTEM -> ADVANCED -> Advanced Options on A1 and B1.
That makes pfsense stop firewalling packet recieved on LAN that should be rerouted to A2/B2

[steve.scotter]

Hi @keyser,

Thanks for taking the time to reply.

That is technically also better network design

Yeah, that's the conclusion I came to after I figured it out and implemented it :) I'm a bit disappointed I hadn't figured it out before I originally posted.

But you could have made it work by simply enabling the bullet called “Static Route Filtering” in SYSTEM -> ADVANCED -> Advanced Options on A1 and B1.
That makes pfsense stop firewalling packet received on LAN that should be rerouted to A2/B2

I did try that as part of my troubleshooting prior to posting.. it was mentioned in my paragraph which starts My research suggests the route cause of this will almost certainly be something to do with asymmetric routing.... For reasons I don't understand it didn't have the desired effect.

With the benefit of hindsight I do believe some sort of firewalling that the root cause of my problem, because when I was migrating the configuration to what I've outlined in my second post I noticed some blocked traffic in the logs. I had however decided by that point that the configuration I was moving towards was a better solution and decided not to investigate further why the original configuration didn't work as expected.

I hope this post someone else in the future.

Thanks

Steve