[Solved] OpenVPN Multiple WAN Asymmetric Routing Issue

tman222

Hi all,

Really scratching my head on this - hopefully someone will be able to help. I have a multiple wan setup (WAN1 and WAN2) in failover configuration (WAN1 is Tier1, WAN2 is Tier2). I wanted to try out the OpenVPN port forwarding method which binds the OpenVPN server to localhost and then forwards requests from both WANs to it, as described in the docs here:

https://docs.netgate.com/pfsense/en/latest/multiwan/openvpn.html#bind-to-localhost-and-setup-port-forwards

Setting this up worked as expected and I can make VPN connections from both WAN1 and WAN2. However, when the connection is made on WAN2, the traffic flows back out WAN1 for some reason leading to asymmetric routing (I have not disabled the reply-to functionality anywhere). Now, I do use FreeRadius as the authentication backend for OpenVPN to utilize 2FA, and from the logs it looks like once the authentication succeeds the traffic goes back out of WAN1 instead of WAN2. More specifically, regardless of whether I connect to the VPN on WAN1 or WAN2, I see a message like this from FreeRadius in the logs:

(0) Login OK: [user_id] (from client pfsenseclient port 1194 cli WAN1_IP:1194)

Could the issue here be with FreeRadius just looking at the currently active gateway (WAN1, Tier1) vs. where the traffic originated from (WAN2, Tier2)? Is there any way to get around by configuring FreeRadius differently or perhaps via policy routing / NAT rules?

Thanks in advance for your help.

viragomann

@tman222
Ensure that no pass rule on an interface group and no floating pass rule matches the incoming OpenVPN packets on the WANs to localhost.

It's tempting to create interface groups for multiple WANs or creating floating rules, since you want to allow the same services on both, but this doesn't work. This can only be used for block rules or internal interfaces.

The pass rules have to be defined on the respective interfaces. If you're unsure if the rules matche enable the logging and check the log after accessing.
Or if you look at the rule and it shows 0 bytes it means, that no traffic matches it.

tman222

@viragomann said in OpenVPN Multiple WAN Asymmetric Routing Issue:

@tman222
Ensure that no pass rule on an interface group and no floating pass rule matches the incoming OpenVPN packets on the WANs to localhost.

It's tempting to create interface groups for multiple WANs or creating floating rules, since you want to allow the same services on both, but this doesn't work. This can only be used for block rules or internal interfaces.

The pass rules have to be defined on the respective interfaces. If you're unsure if the rules matche enable the logging and check the log after accessing.
Or if you look at the rule and it shows 0 bytes it means, that no traffic matches it.

Thanks @viragomann - I have confirmed everything you mentioned. There are no interface groups and no floating pass rules matching OpenVPN traffic. There are two separate pass rules, one on WAN1 and one on WAN2. I can confirm that the connection is made to OpenVPN on WAN2 by looking at the logs and firewall states. However, after the authentication occurs via FreeRADIUS the traffic now exits on WAN1 instead. I have OpenVPN setup to redirect the default gateway and after the connection establishes, I can see the device being connected to WAN2 but its apparent IP is that of WAN1. I'm starting to think this has something do with FreeRADIUS not being properly aware of the source of the traffic and then just choosing the default gateway. Unless you think it may have something to do with the redirect gateway option? Thanks again for your help.

tman222

In other words, what I see happening is basically this:

OpenVPN Connection request to WAN2 --> forwarded to OpenVPN server bound to localhost/127.0.0.1 --> authentication request forwarded to RADIUS server bound to localhost/127.0.0.1 --> upon successful authentication OpenVPN connection established through WAN2 but traffic going through WAN1.

tman222

Well, looks like second time's a charm! I looked again and somehow neglected to realize the first time that I had one floating rule that matched incoming traffic on WAN2. Once I disabled that, things started working properly when connecting to the OpenVPN server through WAN2. I guess I was led astray by the FreeRADIUS message in the logs, but this was irrelevant in the end. Thanks again @viragomann for all your help.

Here also is the link to the documentation that details the issue with floating rules in the inbound direction and the lack of reply-to:

https://docs.netgate.com/pfsense/en/latest/firewall/floating-rules.html#precautions-caveats

tman222

One last update on this post - it looks like I only half-solved this initially: Disabling the floating rule did fix the asymmetric routing issues, but the other problem was that I have Redirect IPv4 Gateway enabled on the OpenVPN server to force all traffic through the tunnel. What essentially happens then is when I connect to the OpenVPN server through WAN2, any requests back out to the internet (e.g. Google) go out through the WAN1 gateway, and then get routed back out through the established OpenVPN tunnel on WAN2. I can force the traffic through the WAN2 gateway instead by setting it explicitly on the allow outbound traffic rule on the OpenVPN interface. I'm guessing the Redirect IPv4 Gateway option just looks at what the default gateway currently is in the routing table and chooses it unless another policy route has been set.

Given all this information, I conclude the following:

The OpenVPN port forward / localhost method would work well in a multiple wan environment if OpenVPN is set up for split tunneling vs. forcing all traffic through the tunnel.
If one wants to force all traffic through the tunnel in a multiple wan environment, I believe one has to set up two (mostly identical) OpenVPN servers along with two OpenVPN interfaces. On the allow outbound traffic rule on OpenVPN interface 1, the WAN1 gateway should be set, on OpenVPN interface 2, the WAN2 gateway should be set on the allow outbound traffic rule.

I hope the above makes sense, but please correct me if these conclusions are incorrect. Is there anyway to force traffic through the correct WAN1 or WAN2 gateway with just one OpenVPN server if Redirect IPv4 Gateway is enabled? Thanks in advance.

viragomann

@tman222
I cannot see any benefit at all in directing upstream traffic from VPN clients out on the interface, where the VPN connection comes in.

The OpenVPN tunnel subnet is considered as a local network. So yes, upstream traffic from VPN clients follows the pfSense routing table of course.

If you really want to realize VPN clients connected to WAN2 going out on WAN2 for whatever reason, yes, you have to run multiple VPN servers listening on the respective interface instead of forwarding and policy route the clients upstream traffic to the desired gateway.
But there are no specific OpenVPN interfaces necessary for this. You can just add the policy routing rules for the different tunnel subnets to the OpenVPN gateway group.

tman222

@viragomann - thanks for the response, I really appreciate it. Can you elaborate what you meant by this:

@viragomann said in [Solved] OpenVPN Multiple WAN Asymmetric Routing Issue:

@tman222
I cannot see any benefit at all in directing upstream traffic from VPN clients out on the interface, where the VPN connection comes in.
...

Why would there be no benefit?

Also, a more general question: What is the overall advantage then of going with the port forward / localhost method for multiwan if the interface on the OpenVPN server can setup using the gateway (failover) group? Is the port forward / localhost method a more robust failover method for OpenVPN compared to using the gateway group?

Thanks again for your help.