Policy Based Routing of return traffic

A Former User

Hello,

I'm struggling with the following scenario:

[Internet] ----(WAN)---- [Firewall DC1] 
                      |
                      |
                 (IPSEC VTI)
                      |                        
                      |
[Internet] ----(WAN)---- [Firewall DC2] 
                               |
                               |
                       (Transit Network)
                               |
                               |
                         [Internal LAN]   -------  [Web Servers]

Firewall in DC1 is running a HAProxy and does SSL offloading for the Web Servers in DC2. These Web Servers are reachable through the routed VPN connection between DC1 and DC2. The Server Pool in the HAProxy Config is configured as transparent so client IPs are maintained at Layer3. If a client contacts the HAProxy at DC1 the request is correctly forwarded to a Web Server in DC2 with the public IP of the Client as Source. I can see the SYN-RCVD status on the Web Server for this request. The return traffic will be (as expected) routed out towards the Internet (Default Gateway) of Firewall in DC2. I confirmed this by packet capture on the WAN Interface at FW DC2. Also I can see in the state table of FW2 two state [SYN_SENT:ESTABLISHED] and [ESTABLISHED:SYN_SENT] for one request to the Web Server.

I've tried to implement a policy based routing by tagging the traffic inbound on the IPsec interface at FW DC2 like:

pass in quick on enc0 inet proto tcp from ! <net_rfc1918> to <websrv> flags S/SA keep state label "USER_RULE: Tag Traffic from DC1 to DC2 WebServer" tag DC1_TRAFFIC

In the floating rules section I've created another rule which suppose to send the traffic back to DC1, but it doesn't get any hit:

pass in quick on ix0 route-to (ipsec4000 172.x.x.2) inet from <websrv> to ! <net_rfc1918> flags S/SA keep state (sloppy) label "USER_RULE" tagged DC1_TRAFFIC

Does anyone have an Idea how to write a proper rule set to send return traffic from the Web Servers back to the FW1 over the VPN Tunnel?

A Former User

Is this maybe a limitation to the routed IPSEC Implementation of pfSense / FREEBSD? I can policy based route traffic from dc2 to dc1 and NAT it out the WAN. This is pretty easy to configure.

I still cannot catch traffic from the Internet which is routed trough DC1 to DC2 to route it back to the HARroxy in DC1. IPSEC traffic leaving the enc0 interface seems to get terminated on the Firewall itself and is handled like outgoing traffic initalized by the firewall itself. At least it looks to me that way if I'm looking at the state table of FW2.

Konstanti

@Artes
Hi
This is a known PF problem, and it has been discussed here many times. Via virtual interfaces (VTI, GRE,OpenVPn,...) the function "Reply-to" does not work. Therefore, all external traffic will ALWAYS return through the default gateway DC2 (WAN interface DC2). To solve this problem you need to:
Use NAT Outbound on the interface VTI DC1 for all external traffic that is forwarded for the WEB server (DC2 side)
or change the default gateway to VTI DC1 (DC2 side)

A Former User

Hi @Konstanti ,

thank you for your reply. Currently I'm using a HAProxy Backend configuration in non-transparent Mode with a dedicated internal source IP of DC1 per Backend Group and let the frontend inserting X-Forwarded-For Header.
My Plan was to switch the Backend Server to http, let Snort have a 2nd look on the traffic and block offenders in case. Since snort is not a real inline IPS in the pfSense implementation, I would need to see the real client IPs to make this work. Maybe I take a look at suricata and see if I can accomplish it with this solution.

ybizeul

Thank you, I’m banging my head on that issue for days, it would be nice if a mention was made in the documentation that reply traffic doesn’t go through PBR.

My solution will be to setup a dedicated pfSense that will have a default route through VPN (OpenVPN in my case)