Multi home, policy routing public segments

cuteredstorm

Hi all,

First let me say that I am a big fan of pfSense and it really saved a lot of money and time in our company. We decided to give something back and are contributing to packages git.

I have also searched the forum and tried all the tricks, but no luck.

We are a small LIR, multi (2x) WAN and using 3 IP segments (modified for security):
89.212.96.186/29->this segment is routable only through provider A
193.189.169.154/29->this segment is routable only through provider B
185.134.185.0/24->this segment is routable through both providers (using BGP).

BGP routing works just fine. We can also send out traffic from 185.134.185.0/24 via both providers at the same time, so that's all good.

Now once default route is set to provider A, the IP segment 193.189.169.154/29 is no longer reachable from the internet. And the other way around.
That makes sense really, because the package comes in through provider B (as it should) but pfSense sends the response out through provider A, following the default route.

This is a case calling for policy routing so I jump right at it. Here are the rules for 193.189.169.154/29:

$ pfctl -sr | grep 193.189.169
block drop in log on ! igb3 inet from 193.189.169.152/29 to any
block drop in log inet from 193.189.169.155 to any
block drop in log inet from 193.189.169.154 to any
pass in log quick route-to (igb3 193.189.169.153) inet proto tcp from 193.189.169.152/29 to any flags S/SA keep state label "USER_RULE"

igb3 is the interface of provider B.

However all responses are still flowing to provider A.

All of this was tested against IPs on the pfSense box itself using ICMP (ping) from my home. Maybe these rules only work when the traffic is flowing through another interface?

I have checked "Disable Negate rule on policy routing rules" and "Enable default gateway switching".

Please help.

cuteredstorm

Found this https://redmine.pfsense.org/issues/5476
and this https://forum.pfsense.org/index.php?topic=102053.0

also related
https://forum.pfsense.org/index.php?topic=72275
https://www.reddit.com/r/PFSENSE/comments/3rxka2/policy_based_routing_of_pfsense_internal_traffic/
http://serverfault.com/questions/734713/pfsense-internal-traffic-policy-based-routing

This is a bug in 2.0+ hopefully fixed in 2.3 (no reason to believe so without testing). This is a bug because docs claim it should work:
https://doc.pfsense.org/index.php/What_are_Floating_Rules

rubic

You do not need floating rules in that scenario. You need to explicitly set 'IPv4 Upstream Gateway' in the igb3 settings. This will add 'reply-to' policy routing to your rules for traffic coming in through ISP B.

For example, ping allowed on my second non-default ISP link:

block drop in log on ! em1 inet from 188.128.123.200/30 to any
block drop in log inet from 188.128.123.202 to any
pass out route-to (em1 188.128.123.201) inet from 188.128.123.202 to ! 188.128.123.200/30 flags S/SA keep state allow-opts label "let out anything from firewall host itself"
pass in quick on em1 reply-to (em1 188.128.123.201) inet proto icmp from any to 188.128.123.202 icmp-type echoreq keep state label "USER_RULE"

cuteredstorm

Hi,

Thank you, that really opened new options for me. For anyone else looking:

yes, you need to explicitly specify the upstream gateway on the interface
this is not enough, when you have floating rules accross both interfaces (provider A and B), but want different paths. So create separate rules :)