[SOLVED] Policy-Based Routing Not Consistently Going Out the Specified Gateway

Finger79 · Oct 16, 2017, 5:29 PM

That looks correct - next time the Policy Routing stops working (I'm guessing it's after one of the gateways flaps) & before doing any manual intervention, try to dump /tmp/rules.debug again and see how they compare to what you have above.

It's currently not working. :)

luckman212 · Oct 16, 2017, 9:16 PM

Hmm that is really odd. Is a screenshot of your LAN rules possible? Do you also have an outbound NAT rule to rewrite traffic for the .103 to GWWANGW?

You could also start a ping from the tablet and then run pfctl on pfSense to have a look at the states…

on tablet

ping 75.75.75.75

on pfSense

pfctl -vv -ss | grep -A2 '75.75.75.75'

It will show you the rule# that is being hit, let's say it was rule 193 - you can check that…

pfctl -vv -sr | grep '@193'

Finger79 · Oct 16, 2017, 10:22 PM

@luckman212:

Hmm that is really odd. Is a screenshot of your LAN rules possible? Do you also have an outbound NAT rule to rewrite traffic for the .103 to GWWANGW?

Screenshot of the rule attached. It's the one that gets triggered (as expected).

No outband NAT rule just for the .103, but I do have 3 separate outbound NAT rules for the LAN subnet to go out WAN, VPN1, and VPN2.

@luckman212:

You could also start a ping from the tablet and then run pfctl on pfSense to have a look at the states…

on tablet
ping 75.75.75.75
on pfSense
pfctl -vv -ss | grep -A2 '75.75.75.75'
It will show you the rule# that is being hit, let's say it was rule 193 - you can check that…
pfctl -vv -sr | grep '@193'

My Tablet rule was only configured to pass IPv4 TCP/UDP (which covers the apps I want to go through the normal WAN), so I had to modify the rule to IPv4 * to include ICMP.

As soon as I applied that rule, everything worked as expected. Pinging 75.75.75.75 went through the tablet rule, and every "What is my IP" website on the tablet returned my ISP WAN's address.

I reset all states just to be sure, and then it reverted all traffic from the tablet through the VPN. Damn it!

Seems like the OpenVPN routing rules use stronger magic than policy-based routing rules and override any gateway I set.

luckman212 · Oct 16, 2017, 10:26 PM

Are your OpenVPN clients set to "Don't pull routes" ? (they should be…)

Finger79 · Oct 16, 2017, 10:29 PM

Here's some of the LAN rules (farther down the rule list) for most of the traffic. For example, the "Allow Web Traffic" rule sends all 80/443 traffic out the VPN gateway.

Finger79 · Oct 16, 2017, 10:31 PM

@luckman212:

Are your OpenVPN clients set to "Don't pull routes" ? (they should be…)

I've played with that setting in the past and had undesirable results. Similar to the guy in this thread, my real IP started leaking instead of going out the VPN.

I'll try it again.

luckman212 · Oct 16, 2017, 10:39 PM

I think you definitely need route-nopull to be checked. If your DNS is "leaking" then you need to adjust the DNS settings on your Tablet so the lookups policy route via the tunnel. If you're using pfSense as the DNS server for the tablet, this isn't going to work since Unbound or dnsmasq upstream queries will be sourced from the firewall itself. The simple fix for that is to manually set a DNS server on the tablet and make sure your policy route traps that traffic (udp/53).

Finger79 · Oct 16, 2017, 10:46 PM

No, not DNS leaking, all traffic leaking such as through 80/443. Even though I have a "NO_WAN_EGRESS" policy-based filtering setup, it doesn't seem to work when I don't pull routes from the VPN provider. (Very not cool!)

I'm getting inconsistent results right now on my laptop on various "What is my IP?" sites.

1) Real WAN IP address (undesirable)
2) VPN connection 1 (expected)
3) VPN connection 2 (expected)

My DNS does not appear to be leaking at all, which is good. All DNS through unbound goes out through the VPN, as configured.

luckman212 · Oct 16, 2017, 10:47 PM

Ah ok sorry I saw "leaking" and jumped to conclusion you were talking about DNS since that is usually what people refer to when talking about leaks.

Did you try the pfctl commands from a few posts ago? And you double checked your outbound NAT rules?

Finger79 · Oct 16, 2017, 10:53 PM

I reverted back to having both OpenVPN client connections pulling routes. Having them not pull routes was 100 times more undesirable since it exposed my entire LAN through the normal gateway and the VPN gateway randomly.

Finger79 · Oct 16, 2017, 10:55 PM

@luckman212:

Ah ok sorry I saw "leaking" and jumped to conclusion you were talking about DNS since that is usually what people refer to when talking about leaks.

Did you try the pfctl commands from a few posts ago? And you double checked your outbound NAT rules?

Not sure what to look for in the outbound NAT rules.
LAN to WAN
LAN to VPN1
LAN to VPN2

I don't have a NAT rule just for the .103 exception… should be included in the above 3 rules.

I did the pfctl command and got inconsistent results. Sometimes it would go out the VPN and other times it would go out the WAN.

Derelict · Oct 16, 2017, 10:58 PM

You people who want to take a complicated setup like policy routing multiple openvpn connections then blame the software when it doesn't work yet obviously have no real grasp on what really needs to happen to make it work simply floor me.

In order for tagging and matching along the NO_WAN_EGRESS vein to work, EVERY packet that should go over the VPN must be tagged.

That is going to be a crap shoot without enabling "don't pull routes."

You are going to have to know exactly how to structure your rules in either case.

Two choices:

Enable "don't pull routes" and policy route VPN traffic

Don't enable "don't pull routes" and policy route clear internet traffic.

With multiple VPN providers, not enabling "don't pull routes" is going to be very complicated because they will both want to enable the 0/1 and 128/1 rules.

luckman212 · Oct 16, 2017, 10:59 PM

Anything in your OpenVPN logs?

My brain started working again & I remembered that you are trying to put your default LAN behind the OpenVPN tunnel and just make an exception for the Tablet. That's the reverse of what I do (I have an alias for devices that I want "hidden" and everything else just uses default routing). So in your case you should probably set the default route of your pfSense is set to the VPN gateway, otherwise your DNS traffic will leak – or, you can set DHCP to hand out an upstream DNS server to LAN clients e.g. 8.8.8.8, etc.

As to why you are getting inconsistent results, I am at a bit of a loss. Maybe this needs a fresh set of eyes. Anyone else got any ideas?

Finger79 · Oct 16, 2017, 11:04 PM

@Derelict:

You people who want to take a complicated setup like policy routing multiple openvpn connections then blame the software when it doesn't work yet obviously have no real grasp on what really needs to happen to make it work simply floor me.

In order for tagging and matching along the NO_WAN_EGRESS vein to work, EVERY packet that should go over the VPN must be tagged.

That is going to be a crap shoot without enabling "don't pull routes."

I think you're misunderstanding. The policy-based filtering floating rule works perfectly (matches the previously tagged packets). That's not what this thread is about.

@Derelict:

In order for tagging and matching along the NO_WAN_EGRESS vein to work, EVERY packet that should go over the VPN must be tagged.

Yep, no issues here. Tagging is working as expected.

@Derelict:

You people who want to take a complicated setup like policy routing multiple openvpn connections

The VPN connections are in one gateway group. The policy-route is set to send all traffic out the VPN gateway. That's working perfectly.

The only thing that's not consistently working is the policy-route for one device on the LAN to go out the normal WAN gateway.

Derelict · Oct 16, 2017, 11:05 PM

OK where is that rule in relation to all your other rules? You have yet to show that.

I was more commenting on the nonsense like this:

No, not DNS leaking, all traffic leaking such as through 80/443. Even though I have a "NO_WAN_EGRESS" policy-based filtering setup, it doesn't seem to work when I don't pull routes from the VPN provider. (Very not cool!)

It works perfectly when configured correctly.

Finger79 · Oct 16, 2017, 11:13 PM

@Derelict:

OK where is that rule in relation to all your other rules? You have yet to show that.

I was more commenting on the nonsense like this:

No, not DNS leaking, all traffic leaking such as through 80/443. Even though I have a "NO_WAN_EGRESS" policy-based filtering setup, it doesn't seem to work when I don't pull routes from the VPN provider. (Very not cool!)

It works perfectly when configured correctly.

Screenshots of that posted earlier. The "Allow Web Traffic" rule sets the policy-based filtering tag "NO_WAN_EGRESS" and also sets the policy-based routing gateway to the VPN_Gateway.

The only time that traffic leaked out the naked WAN was when I told both client VPN connections to not pull routes. Then I got inconsistent results: some traffic went out the VPN, and other traffic went out the WAN. It was random.

Finger79 · Oct 16, 2017, 11:16 PM

@luckman212:

So in your case you should probably set the default route of your pfSense is set to the VPN gateway, otherwise your DNS traffic will leak – or, you can set DHCP to hand out an upstream DNS server to LAN clients e.g. 8.8.8.8, etc.

DNS seems to work perfectly. Unbound sends all traffic through the VPN tunnels and never out the naked WAN. (That interface is unchecked.)

Derelict · Oct 16, 2017, 11:19 PM

You are all over the place. that rule routes traffic to PIA for, presumably, destinations 80 and 443.

All other traffic will go out the default gateway (or the OpenVPN connection that happens to have been able to set the 0.0.0.0/1 and 128.0.0.0/1 rules, which as you found in the other thread, will be the first OpenVPN connection without "don't pull routes" set that connects. The other one will receive errors when trying to set those routes.)

You are indicating there is a problem with some other host that is unable to go out WAN. Where is that rule in relation to all the other rules?

Not blocking out things that really don't matter might help people help you.

Finger79 · Oct 16, 2017, 11:27 PM

@Derelict:

You are indicating there is a problem with some other host that is unable to go out WAN.

Negative. It's not unable to go out WAN. It just doesn't do it consistently.

@Derelict:

Where is that rule in relation to all the other rules?

Answered earlier:
@Finger79:

Here's some of the LAN rules (farther down the rule list) for most of the traffic. For example, the "Allow Web Traffic" rule sends all 80/443 traffic out the VPN gateway.

The .103 tablet exception rule matches first. It just doesn't consistently send traffic out the WAN. Sometimes it does, other times it doesn't. But the rule always fires.

Derelict · Oct 16, 2017, 11:29 PM

Is WANGW flapping?

System > Logs, Gateways