[SOLVED] Policy-Based Routing Not Consistently Going Out the Specified Gateway

luckman212 · Oct 16, 2017, 10:26 PM

Are your OpenVPN clients set to "Don't pull routes" ? (they should be…)

Finger79 · Oct 16, 2017, 10:29 PM

Here's some of the LAN rules (farther down the rule list) for most of the traffic. For example, the "Allow Web Traffic" rule sends all 80/443 traffic out the VPN gateway.

Finger79 · Oct 16, 2017, 10:31 PM

@luckman212:

Are your OpenVPN clients set to "Don't pull routes" ? (they should be…)

I've played with that setting in the past and had undesirable results. Similar to the guy in this thread, my real IP started leaking instead of going out the VPN.

I'll try it again.

luckman212 · Oct 16, 2017, 10:39 PM

I think you definitely need route-nopull to be checked. If your DNS is "leaking" then you need to adjust the DNS settings on your Tablet so the lookups policy route via the tunnel. If you're using pfSense as the DNS server for the tablet, this isn't going to work since Unbound or dnsmasq upstream queries will be sourced from the firewall itself. The simple fix for that is to manually set a DNS server on the tablet and make sure your policy route traps that traffic (udp/53).

Finger79 · Oct 16, 2017, 10:46 PM

No, not DNS leaking, all traffic leaking such as through 80/443. Even though I have a "NO_WAN_EGRESS" policy-based filtering setup, it doesn't seem to work when I don't pull routes from the VPN provider. (Very not cool!)

I'm getting inconsistent results right now on my laptop on various "What is my IP?" sites.

1) Real WAN IP address (undesirable)
2) VPN connection 1 (expected)
3) VPN connection 2 (expected)

My DNS does not appear to be leaking at all, which is good. All DNS through unbound goes out through the VPN, as configured.

luckman212 · Oct 16, 2017, 10:47 PM

Ah ok sorry I saw "leaking" and jumped to conclusion you were talking about DNS since that is usually what people refer to when talking about leaks.

Did you try the pfctl commands from a few posts ago? And you double checked your outbound NAT rules?

Finger79 · Oct 16, 2017, 10:53 PM

I reverted back to having both OpenVPN client connections pulling routes. Having them not pull routes was 100 times more undesirable since it exposed my entire LAN through the normal gateway and the VPN gateway randomly.

Finger79 · Oct 16, 2017, 10:55 PM

@luckman212:

Ah ok sorry I saw "leaking" and jumped to conclusion you were talking about DNS since that is usually what people refer to when talking about leaks.

Did you try the pfctl commands from a few posts ago? And you double checked your outbound NAT rules?

Not sure what to look for in the outbound NAT rules.
LAN to WAN
LAN to VPN1
LAN to VPN2

I don't have a NAT rule just for the .103 exception… should be included in the above 3 rules.

I did the pfctl command and got inconsistent results. Sometimes it would go out the VPN and other times it would go out the WAN.

Derelict · Oct 16, 2017, 10:58 PM

You people who want to take a complicated setup like policy routing multiple openvpn connections then blame the software when it doesn't work yet obviously have no real grasp on what really needs to happen to make it work simply floor me.

In order for tagging and matching along the NO_WAN_EGRESS vein to work, EVERY packet that should go over the VPN must be tagged.

That is going to be a crap shoot without enabling "don't pull routes."

You are going to have to know exactly how to structure your rules in either case.

Two choices:

Enable "don't pull routes" and policy route VPN traffic

Don't enable "don't pull routes" and policy route clear internet traffic.

With multiple VPN providers, not enabling "don't pull routes" is going to be very complicated because they will both want to enable the 0/1 and 128/1 rules.

luckman212 · Oct 16, 2017, 10:59 PM

Anything in your OpenVPN logs?

My brain started working again & I remembered that you are trying to put your default LAN behind the OpenVPN tunnel and just make an exception for the Tablet. That's the reverse of what I do (I have an alias for devices that I want "hidden" and everything else just uses default routing). So in your case you should probably set the default route of your pfSense is set to the VPN gateway, otherwise your DNS traffic will leak – or, you can set DHCP to hand out an upstream DNS server to LAN clients e.g. 8.8.8.8, etc.

As to why you are getting inconsistent results, I am at a bit of a loss. Maybe this needs a fresh set of eyes. Anyone else got any ideas?

Finger79 · Oct 16, 2017, 11:04 PM

@Derelict:

You people who want to take a complicated setup like policy routing multiple openvpn connections then blame the software when it doesn't work yet obviously have no real grasp on what really needs to happen to make it work simply floor me.

In order for tagging and matching along the NO_WAN_EGRESS vein to work, EVERY packet that should go over the VPN must be tagged.

That is going to be a crap shoot without enabling "don't pull routes."

I think you're misunderstanding. The policy-based filtering floating rule works perfectly (matches the previously tagged packets). That's not what this thread is about.

@Derelict:

In order for tagging and matching along the NO_WAN_EGRESS vein to work, EVERY packet that should go over the VPN must be tagged.

Yep, no issues here. Tagging is working as expected.

@Derelict:

You people who want to take a complicated setup like policy routing multiple openvpn connections

The VPN connections are in one gateway group. The policy-route is set to send all traffic out the VPN gateway. That's working perfectly.

The only thing that's not consistently working is the policy-route for one device on the LAN to go out the normal WAN gateway.

Derelict · Oct 16, 2017, 11:05 PM

OK where is that rule in relation to all your other rules? You have yet to show that.

I was more commenting on the nonsense like this:

No, not DNS leaking, all traffic leaking such as through 80/443. Even though I have a "NO_WAN_EGRESS" policy-based filtering setup, it doesn't seem to work when I don't pull routes from the VPN provider. (Very not cool!)

It works perfectly when configured correctly.

Finger79 · Oct 16, 2017, 11:13 PM

@Derelict:

OK where is that rule in relation to all your other rules? You have yet to show that.

I was more commenting on the nonsense like this:

No, not DNS leaking, all traffic leaking such as through 80/443. Even though I have a "NO_WAN_EGRESS" policy-based filtering setup, it doesn't seem to work when I don't pull routes from the VPN provider. (Very not cool!)

It works perfectly when configured correctly.

Screenshots of that posted earlier. The "Allow Web Traffic" rule sets the policy-based filtering tag "NO_WAN_EGRESS" and also sets the policy-based routing gateway to the VPN_Gateway.

The only time that traffic leaked out the naked WAN was when I told both client VPN connections to not pull routes. Then I got inconsistent results: some traffic went out the VPN, and other traffic went out the WAN. It was random.

Finger79 · Oct 16, 2017, 11:16 PM

@luckman212:

So in your case you should probably set the default route of your pfSense is set to the VPN gateway, otherwise your DNS traffic will leak – or, you can set DHCP to hand out an upstream DNS server to LAN clients e.g. 8.8.8.8, etc.

DNS seems to work perfectly. Unbound sends all traffic through the VPN tunnels and never out the naked WAN. (That interface is unchecked.)

Derelict · Oct 16, 2017, 11:19 PM

You are all over the place. that rule routes traffic to PIA for, presumably, destinations 80 and 443.

All other traffic will go out the default gateway (or the OpenVPN connection that happens to have been able to set the 0.0.0.0/1 and 128.0.0.0/1 rules, which as you found in the other thread, will be the first OpenVPN connection without "don't pull routes" set that connects. The other one will receive errors when trying to set those routes.)

You are indicating there is a problem with some other host that is unable to go out WAN. Where is that rule in relation to all the other rules?

Not blocking out things that really don't matter might help people help you.

Finger79 · Oct 16, 2017, 11:27 PM

@Derelict:

You are indicating there is a problem with some other host that is unable to go out WAN.

Negative. It's not unable to go out WAN. It just doesn't do it consistently.

@Derelict:

Where is that rule in relation to all the other rules?

Answered earlier:
@Finger79:

Here's some of the LAN rules (farther down the rule list) for most of the traffic. For example, the "Allow Web Traffic" rule sends all 80/443 traffic out the VPN gateway.

The .103 tablet exception rule matches first. It just doesn't consistently send traffic out the WAN. Sometimes it does, other times it doesn't. But the rule always fires.

Derelict · Oct 16, 2017, 11:29 PM

Is WANGW flapping?

System > Logs, Gateways

Finger79 · Oct 16, 2017, 11:46 PM

@Derelict:

Is WANGW flapping?

System > Logs, Gateways

I don't see it mentioned anywhere in the Gateway logs.

I had to turn off WANGW gateway monitoring (meaning it's always considered "Up"). The dpinger pings may have pissed it off. I'll turn it back on and see if that helps.

Derelict · Oct 16, 2017, 11:42 PM

Right. If that was happening when you were seeing "random" routing then the same principles that make "NO_WAN_EGRESS" necessary would apply equally to WANGW if it was flagged as down. In that case you would need "NO_VPN_EGRESS."

Finger79 · Oct 16, 2017, 11:52 PM

I get that. Fortunately, gateway flapping appears to not be the reason behind this.

When I do a fresh reboot of pfSense, the tablet traffic consistently goes out the WAN, as expected. Some time later (or some event later), it decides to go out the VPN only, even though the rule still fires that specifies WANGW. It's just ignoring the gateway in the rule but still logging the rule as having fired. Weird.