OpenVPN, routed subnet and 1:1 NAT and outbound return path

MrEmbedded

Hi All,

I have done many searches on how to get this configured correctly but I don't have a definitive answer so I am hoping someone can help me.

I have a standard failover pfSense CARP (with /29 on WAN interfaces+ CARP) based cluster at location A.
Location B has a single pfSense firewall but will eventually be converted to a cluster
There is on OpenVPN tunnel connecting the WAN interface of location B to the CARP IP (/29) of location A. This works well
There are rules in place for OpenVPN at both locations to allow for each locations LAN subnet to reach each other. This works well.

Now things get more complex.

There is an external /28 that is routed to the location A CARP (/29) address
I have created a single VIP other on both members of the CARP cluster using 2 of the /28 IPs. This is to bypass the CARP subnet requrement
I have created some of the /28 as CARP addresses on the WAN interface of location A. The addresses are correctly copied to the slave firewall
Those /28 address have been assigned a 1:1 NAT with addresses on the LAN subnet of location B
Rules have been setup at location A to allow traffic to the location B subnet addresses from the Location A WAN interface

Here is what I happening:

When a request is made to a /28 CARP address at location A, the traffic is passed correctly via 1:1 NAT and hits the location B LAN address
The return packet gets lost somewhere and the originating machine doesn't get a response
tcpdump on the WAN interface of location A shows the request from the external host to one of the /28 CARP addresses
tcpdump on the WAN interface of location B shows the request also from the external host to the 1:1 NAT location B LAN address
tcpdump on the LAN interface of location B shows the response from the 1:1 NAT location B LAN address to the external host. This is what I believe to be the problem as I don't think its going back out via the OpenVPN tunnel

I have tried many combinations things with the same results:

Adding push routes on the OpenVPN setup at location A and/or adding routes to the OpenVPN setup at location B
Manual outbound NAT to the /28 CARP address or location A LAN gateway
Manual outbound NAT to location A LAN gateway
Using regular NAT rather than 1:1
Other things I may have forgotten

There is one other clue

I have a loadbalanced cluster setup on location A CARP address (/29)
This is forwarding to machines on location B LAN
Manual outbound NAT is setup on location B WAN interface to route any outbound traffic from those servers to CARP IP (/29) in location A

This is working well. So there is definitely something I am missing regarding the return path. I have seen various posts about this but the answer is just not very clear for me.

MrEmbedded

Ok this is like pulling teeth. I think I want to change the strategy here. Maybe what I should do:

Route the external /28 through the VPN link rather than trying to NAT it through
Setup the CARP VIPs for that /28 on the Location B firewall instead
NAT only from the Location B firewall external to internal interfaces

Does anyone see an issue with this logically?