[SOLVED] Policy-Based Routing Not Consistently Going Out the Specified Gateway

Derelict · Oct 17, 2017, 12:03 AM

Doubtful. There is some other reason the traffic is not matching that policy routing rule - else it would be policy routed accordingly.

Finger79 · Oct 17, 2017, 12:05 AM

@Derelict:

Doubtful. There is some other reason the traffic is not matching that policy routing rule - else it would be policy routed accordingly.

Then why does the rule match in the Firewall Logs?

As you can see, the rule is very simple. If the source is .103 IPv4, then policy route it through WANGW.

This works perfectly after a fresh reboot of pfSense. Then like I said, after some time or some event, it no longer goes out the WANGW and goes out the VPN. Something is overriding the routing portion of the firewall rule.

Finger79 · Oct 17, 2017, 12:33 AM

Just did some more, all from Firefox on the tablet:

Shows real WAN IP
Google "What is my IP"
iplocation.com
whatismyip.net
privateinternetaccess.com
whatismyip.org
ExpressVPN.com
MXtoolbox.com
ip-address.org
iplocation.net
findipinfo.com
myipaddress.com

Shows VPN IP
TorGuard.net
DuckDuckGo "What is my IP"
whatismyipaddress.com
BearsMyIP.com
ipchicken.com
ipaddress.pro

luckman212 · Oct 17, 2017, 12:24 AM

@Finger79:

Then why does the rule match in the Firewall Logs?

As you can see, the rule is very simple. If the source is .103 IPv4, then policy route it through WANGW.

Do you have a kill switch rule below the policy route for .103 to block all traffic? You need this in case WANGW is down, because rules will be skipped if the GW is flapping, could explain your inconsistent results…

Derelict · Oct 17, 2017, 12:29 AM

When it stops working, run this:

pfctl -vvsr | grep -A3 XX.XX.XX.103

Here: I'll show you one of mine. I'm not afraid of leaking inside addresses:

$ pfctl -vvsr | grep -A3 192.168.223.6

@307(1493852191) pass in quick on igb1.223 route-to (ovpnc3 172.29.114.130) inet from 192.168.223.6 to <openvpn_lan:2>flags S/SA keep state label "USER_RULE: Route OpenVPN Addresses Through OpenVPN"
[ Evaluations: 2386 Packets: 0 Bytes: 0 States: 0 ]
[ Inserted: pid 21796 State Creations: 0 ]

Anyway, that will show the EXACT rules in the active rule set that have anything to do with that address at that specific point in time.</openvpn_lan:2>

Finger79 · Oct 17, 2017, 12:45 AM

@124(10000001) pass in log quick on igb1 inet from 192.168.100.103 to <negate_net ="" works:0="">flags S/SA keep state label "NEGATE_ROUTE: Negate policy routing for de stination"
[ Evaluations: 2572 Packets: 0 Bytes: 0 States: 0 ]
[ Inserted: pid 54887 State Creations: 0 ]
@125(1505701172) pass in log quick on igb1 route-to (igb0 xxx.xxx.xxx.xxx public IP) inet from 192.168.100.103 to any flags S/SA keep state label "USER_RULE: Tablet Out Naked WAN"
[ Evaluations: 865 Packets: 55591 Bytes: 19139183 States: 5 ]
[ Inserted: pid 54887 State Creations: 807 ]
@126(1458032398) block return in log quick on igb1 inet from any to <pfb_africa_ ="" v4:6176="">label "USER_RULE: pfb_Africa"</pfb_africa_ ></negate_net >

Derelict · Oct 17, 2017, 12:51 AM

Looks good to me unless negate_networks includes the wrong destinations, which is pretty unlikely.

Or if there is a rule that matches that source address that won't be shown there.

I'd be happy to look at /tmp/rules.debug if you want to PM it.

Finger79 · Oct 17, 2017, 3:42 AM

Somewhat redacted and edited:

< /tmp/rules.debug removed >

Finger79 · Oct 17, 2017, 1:24 AM

Wonder if this table should be empty:

Diagnostics -> Tables
negate_networks

No entries exist in this table.

Derelict · Oct 17, 2017, 1:41 AM

pass in quick on $LAN $GWPIA_TX_CHI inet proto tcp from any to $Facebook port 443 tag "NO_WAN_EGRESS" tracker 1422073736 flags S/SA keep state label "USER_RULE: Allow Facebook"
pass in quick on $LAN inet proto tcp from any to $CloudFlare port $HTTP_HTTPS tracker 1422073738 flags S/SA keep state label "USER_RULE: CloudFlare"
pass in log quick on $LAN inet from 192.168.100.103 to <negate_networks>tracker 10000001 keep state label "NEGATE_ROUTE: Negate policy routing for destination"
pass in log quick on $LAN $GWWANGW inet from 192.168.100.103 to any tracker 1505701172 keep state label "USER_RULE: Asus Tablet Out Naked WAN"

Anything at $Facebook/443 or $CloudFlare/$HTTP_HTTPS will not match your source 192.168.100.103 rules. That is probably your problem.

Put the most specific rules at the top.</negate_networks>

Finger79 · Oct 17, 2017, 3:23 AM

It was that damned CloudFlare rule.
I re-ran the list of places that previously showed the VPN IP and they all reported the real WAN IP as expected.

I really hope this consistently fixes it. I'll update the thread if it doesn't fix it after I've pulled some hair out.

(BTW, those Facebook IPs are straight from Facebook so only include their CIDR blocks and nobody else. Back when that info was public.)

Shows VPN IP
TorGuard.net –> Shows real IP :)
DuckDuckGo "What is my IP" --> Shows real IP :)
whatismyipaddress.com --> Shows real IP :)
BearsMyIP.com --> Shows real IP :)
ipchicken.com --> Shows real IP :)
ipaddress.pro --> Shows real IP :)

Anecdotally, this also tells me just how many sites are CloudFlare customers (at least the free account). Holy crap it's a lot.