Policy routing out Wireguard interface - asymetric routing

dpravd · Mar 6, 2024, 9:36 AM

Hi there,

I thought this would be quite easy - but it doesn't seem to work for some reason.

I have a workround for this - but I don't like it. (Setting the default gateway as the VPN address and removing policy routing)

Here are the gatways:

What I would like to do, is for the lan interface, set the gateway as the vpn interface. (policy route) and nat only the outbound traffic behind the VPN tunnel address. Which is exactly what I do on Opnsense with Openvpn.

This works fine for outbound traffic.

However, for inbound traffic, the reply is going out the WAN, so I'm getting asymetric routing. I read somewhere about turning off "Netgate rules", which I did, but it didn't make any difference.

I have the "interface" for the VPN set with the VPN as the gateway. I find the whole setup for wireguard pretty odd, like you have a "wireguard" under firewall, then the actual VPN interface too. Like why? The VPN interface doesn't do anything here.

Here's the VPN interface:

Any ideas why the return traffic is going out the WAN and not the tunnel? Thanks

Bob.Dig · Mar 6, 2024, 10:49 AM

@dpravd said in Policy routing out Wireguard interface - asymetric routing:

However, for inbound traffic, the reply is going out the WAN,

If it is inbound on WAN, sure it goes out the WAN, why not. Anything else would be asymmetric routing...
Doesn't make any sense what you did and want. And why is the vpn and WAN in the same Gatewaygroup...

dpravd · Mar 6, 2024, 11:40 AM

@Bob-Dig Sorry, I should clarify, it's inbound on the VPN interface (I'm dnatting it from the internet, on the other end of the tunnel)

This setup works perfectly fine on Opnsense and Openvpn. (I haven't tried it with opnsense and wireguard yet but I suspect it would work fine)

That's the point - I don't want to have to specify the VPN as the default gateway. (I put the WAN in there as secondary for when the VPN hasn't come up yet) but it's the only way I can get the pfsense to route the traffic back properly.

If I remove the VPN as the default gateway, and try policy route, it tries to send the replies out the wan... which is wrong

viragomann · Mar 6, 2024, 11:45 AM

@dpravd
The clue is to remove all pass rules from Wireguard and having pass rules for permitting incoming traffic on the VPN only.

dpravd · Mar 6, 2024, 12:34 PM

@viragomann Hmmmm interesting. I'll give this a go, thanks for the info

dpravd · Mar 12, 2024, 10:14 AM

@viragomann I tried what you said, and can see the traffic is hitting this filter in firewall rules.

However, return traffic will not policy route properly - it's still going out the WAN.

So, traffic originating from the lan is fine, it policy routes out the vpn. However traffic coming in on the vpn, goes back out the wan, and skips the policy route for some reason.

I tried setting allow all flags on the lan firewall rules - but still doesn't work.

Of note, the return traffic gets natted properly to the VPN address, but from there instead of going out the VPN, it's going out the WAN.

Here, you can see the SYN acks going out the WAN. The vpn address is 10.11.12.1

Any ideas?

Bob.Dig · Mar 12, 2024, 10:14 AM

@dpravd said in Policy routing out Wireguard interface - asymetric routing:

Any ideas?

Start over with a fresh install. Don't set your VPN as the default gateway, use policy based routing for that. The default gateway should be WAN. And not "Main()", whatever this is.

dpravd · Mar 12, 2024, 10:45 AM

@Bob-Dig Okay - yeah absolutely, the only reason I was setting the VPN as the default gateway as that was the only way to get the return traffic.

I'll start again, thanks.

Bob.Dig · Mar 12, 2024, 11:19 AM

@dpravd And, if your server is on LAN or wherever, you don't need to set the gateway there for a webserver. It would be enough to allow incoming traffic from the VPN to it.

viragomann · Mar 12, 2024, 11:20 AM

@dpravd said in Policy routing out Wireguard interface - asymetric routing:

I tried what you said, and can see the traffic is hitting this filter in firewall rules.

Your screenshot doesn't show any hits on the VPN rule.
Did you reset the states?

Popolou · Mar 12, 2024, 11:25 AM

What do your outbound rules look like?

viragomann · Mar 12, 2024, 11:29 AM

@Popolou said in Policy routing out Wireguard interface - asymetric routing:

What do your outbound rules look like?

The outbound NAT has not even any impact on response packets.

dpravd · Mar 12, 2024, 11:58 AM

I set it up again from scratch. Same thing.

Here is the traffic flow - this works fine, traffic goes out the VPN interface and comes back in the VPN interface

LAN -> VPN Pfsense -> VPN ubuntu -> INTERNET
10.5.0.0/24 -> 10.11.12.1/31 -> 10.11.12.0/31 -> 0.0.0.0/0

Here is the broken traffic - traffic comes in the VPN interface, but then goes out the WAN

INTERNET (dnat to 10.11.12.1) -> VPN ubuntu -> pfsense VPN (port forwards on vpn interface) -> LAN
0.0.0.0/0 -> 10.11.12.0/31 -> 10.11.12.1/31 -> 10.5.0.0/24

In the second flow, the traffic gets to the lan fine, and gets natted to 10.11.12.1 fine, but then goes out the wan.

Lan rule has gatway of VPN.
Wireguard interface has no rules
VPN interface has allow all
Default gateway of box is WAN.

dpravd · Mar 12, 2024, 11:56 AM

@viragomann Yes - sorry, I reset the states just before I took the screenshot

viragomann · Mar 12, 2024, 12:00 PM

@dpravd
To ensure that the proper rule is applied, enable logging in the rule and check the filter log after initiating traffic from remote.
Note the logged rule ID, which is passing the traffic and check if it's the rule you added on the VPN interface.