Multi home, policy routing public segments
First let me say that I am a big fan of pfSense and it really saved a lot of money and time in our company. We decided to give something back and are contributing to packages git.
I have also searched the forum and tried all the tricks, but no luck.
We are a small LIR, multi (2x) WAN and using 3 IP segments (modified for security):
18.104.22.168/29->this segment is routable only through provider A
22.214.171.124/29->this segment is routable only through provider B
126.96.36.199/24->this segment is routable through both providers (using BGP).
BGP routing works just fine. We can also send out traffic from 188.8.131.52/24 via both providers at the same time, so that's all good.
Now once default route is set to provider A, the IP segment 184.108.40.206/29 is no longer reachable from the internet. And the other way around.
That makes sense really, because the package comes in through provider B (as it should) but pfSense sends the response out through provider A, following the default route.
This is a case calling for policy routing so I jump right at it. Here are the rules for 220.127.116.11/29:
$ pfctl -sr | grep 193.189.169 block drop in log on ! igb3 inet from 18.104.22.168/29 to any block drop in log inet from 22.214.171.124 to any block drop in log inet from 126.96.36.199 to any pass in log quick route-to (igb3 188.8.131.52) inet proto tcp from 184.108.40.206/29 to any flags S/SA keep state label "USER_RULE"
igb3 is the interface of provider B.
However all responses are still flowing to provider A.
All of this was tested against IPs on the pfSense box itself using ICMP (ping) from my home. Maybe these rules only work when the traffic is flowing through another interface?
I have checked "Disable Negate rule on policy routing rules" and "Enable default gateway switching".
This is a bug in 2.0+ hopefully fixed in 2.3 (no reason to believe so without testing). This is a bug because docs claim it should work:
You do not need floating rules in that scenario. You need to explicitly set 'IPv4 Upstream Gateway' in the igb3 settings. This will add 'reply-to' policy routing to your rules for traffic coming in through ISP B.
For example, ping allowed on my second non-default ISP link:
block drop in log on ! em1 inet from 220.127.116.11/30 to any block drop in log inet from 18.104.22.168 to any pass out route-to (em1 22.214.171.124) inet from 126.96.36.199 to ! 188.8.131.52/30 flags S/SA keep state allow-opts label "let out anything from firewall host itself" pass in quick on em1 reply-to (em1 184.108.40.206) inet proto icmp from any to 220.127.116.11 icmp-type echoreq keep state label "USER_RULE"
Thank you, that really opened new options for me. For anyone else looking:
yes, you need to explicitly specify the upstream gateway on the interface
this is not enough, when you have floating rules accross both interfaces (provider A and B), but want different paths. So create separate rules :)