Policy based routing, multi-WAN and gateway on same subnet

[makz]

Hello,

I've included coreswitching part and some more informations on this diagram

As seen in older screenshots, two gateway are configured on WAN interface (192.168.0.1 & 192.168.0.5), my rule "Matching source 10.10.3.50 -> Gateway 192.168.0.1" is done in INTERCO interface and it's on top of the rule list.

Thank you.

.png)
.png_thumb)

[makz]

Hello,

I'm back with some news !

Today i've configured a little freebsd router to try my setup on a vanilla bsd.

My basic setup work well on this vanilla router.

pass out quick on $int_if route-to ($ext_if 192.168.0.1) inet from $bck_srv
pass in quick on $int_if route-to ($ext_if 192.168.0.1) inet from $bck_srv

Now i know this is not a freebsd issue but more a pfsense issue.

I've read the pf rules generated by pfsense and i saw some hidden rules

pass out  route-to ( lagg0_vlan2000 192.168.0.5 ) from 192.168.0.10 to !192.168.0.0/24 tracker 1000008011 keep state allow-opts label "let out anything from firewall host itself"
pass out  route-to ( lagg0_vlan2000 192.168.0.5 ) from 192.168.0.12 to !192.168.0.0/24 tracker 1000008012 keep state allow-opts label "let out anything from firewall host itself"

I've removed those rules from pf and magic it work !

So,

I saw in the bug tracker, if i place my rule in "Floating rules" this should disable the hidden rule <https: 1823="" redmine.pfsense.org="" issues="">This still won't work … Here is all my "route-to" rules from my non working pf configuration

pfctl -sa | grep route-to
pass out route-to (lagg0_vlan2000 192.168.0.5) inet from 192.168.0.10 to ! 192.168.0.0/24 flags S/SA keep state allow-opts label "let out anything from firewall host itself"
pass out route-to (lagg0_vlan2000 192.168.0.5) inet from 192.168.0.12 to ! 192.168.0.0/24 flags S/SA keep state allow-opts label "let out anything from firewall host itself"
pass out quick on lagg0_vlan1008 route-to (lagg0_vlan2000 192.168.0.1) inet from <backup_servers> to any flags S/SA keep state label "USER_RULE: TEST ROUTING"
pass in quick on lagg0_vlan1008 route-to (lagg0_vlan2000 192.168.0.1) inet from <backup_servers> to any flags S/SA keep state label "USER_RULE: TEST ROUTING"</backup_servers></backup_servers>

I've tried to inverse the rules (my rule before the hidden rule), this still won't work

I've also tried to remove the route-to argument to the hidden rule

pass out  from 192.168.0.10 to !192.168.0.0/24 tracker 1000008011 keep state allow-opts label "let out anything from firewall host itself"
pass out  from 192.168.0.12 to !192.168.0.0/24 tracker 1000008012 keep state allow-opts label "let out anything from firewall host itself"

and this work great !

For now i have some questions …

What is the purpose of this hidden rule ?
Can i disable it permanently ?
Can i just remove the route-to argument permanently ?
Is there any option to bypass this rule for some cases ?

Thank you !

Have a great day.</https:>

[gslongo]

A bug related to this has been opened here : https://redmine.pfsense.org/issues/7033

Can it be analysed by a developper ?

//EDIT: Also found user having same issue here : https://redmine.pfsense.org/issues/6625

[jimp]

The floating rules do not "disable" the hidden rule, they override it. When you make your floating rules, be sure to check "quick" and that the rules will match the traffic going to your other gateway. The hidden rules do not have quick, so a quick rule will match and the non-quick rule will never be processed.

Post the exact floating rules you made and it should be fairly easy to tell why they aren't working.

[makz]

@jimp:

The floating rules do not "disable" the hidden rule, they override it. When you make your floating rules, be sure to check "quick" and that the rules will match the traffic going to your other gateway. The hidden rules do not have quick, so a quick rule will match and the non-quick rule will never be processed.

Post the exact floating rules you made and it should be fairly easy to tell why they aren't working.

Hello, thank you for the response, but … there's a but ...

My floating rule have the "quick" option, you can see it in my screenshot on the bugtracker (i repost it here in attachment)

Here's the block of rules with the hidden one and mine

pass out inet all flags S/SA keep state allow-opts label "let out anything IPv4 from firewall host itself"
pass out inet6 all flags S/SA keep state allow-opts label "let out anything IPv6 from firewall host itself"
pass out route-to (lagg0_vlan2000 192.168.0.5) inet from 192.168.0.10 to ! 192.168.0.0/24 flags S/SA keep state allow-opts label "let out anything from firewall host itself"
pass out route-to (lagg0_vlan2000 192.168.0.5) inet from 192.168.0.12 to ! 192.168.0.0/24 flags S/SA keep state allow-opts label "let out anything from firewall host itself"
pass in quick on lagg0_vlan1007 proto tcp from any to (lagg0_vlan1007) port = https flags S/SA keep state label "anti-lockout rule"
pass in quick on lagg0_vlan1007 proto tcp from any to (lagg0_vlan1007) port = http flags S/SA keep state label "anti-lockout rule"
pass in quick on lagg0_vlan1007 proto tcp from any to (lagg0_vlan1007) port = rsh-spx flags S/SA keep state label "anti-lockout rule"
anchor "userrules/*" all
pass in quick on lagg0_vlan1008 inet from <backup_servers>to <negate_networks>flags S/SA keep state label "NEGATE_ROUTE: Negate policy routing for destination"
pass in quick on lagg0_vlan1008 route-to (lagg0_vlan2000 192.168.0.1) inet from <backup_servers>to any flags S/SA keep state label "USER_RULE: TEST ROUTING"
pass out quick on lagg0_vlan1008 inet from <backup_servers>to <negate_networks>flags S/SA keep state label "NEGATE_ROUTE: Negate policy routing for destination"
pass out quick on lagg0_vlan1008 route-to (lagg0_vlan2000 192.168.0.1) inet from <backup_servers>to any flags S/SA keep state label "USER_RULE: TEST ROUTING"</backup_servers></negate_networks></backup_servers></backup_servers></negate_networks></backup_servers> 

And when i don't patch the filter.inc to remove the hidden rule, my traffic is routed to the default gateway as explained before.

[root@backup ~]# traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  gateway (10.10.3.1)  0.541 ms  1.298 ms  1.264 ms
 2  192.168.0.5 (192.168.0.5)  0.406 ms  0.404 ms  0.350 ms

[jimp]

Do not set a gateway on those

[makz]

@jimp:

Do not set a gateway on those

Hi,
Thank you for your reply

Not sure I understand well
On which rules ? Because we do not find where to modify the hidden rules.
And if you mean on the floating rules, then where to set the gateway if not in advanced options of these rules?

Thank you for advance

[jimp]

Use a static route

[jimp]

Or policy route on LAN rules

[gslongo]

@jimp:

Or policy route on LAN rules

Hi Jim,

Static routes are not possible in that case
We already tried same rule on LAN interface but same behavior. And if I understand well the solution shoud be a floating rule ? But then why it is not working in that case ?
We tried every possible scenarios (I think) this is why at the end we opened a bug on the bug tracker.

Thank you!

[jimp]

You need both.
Policy route on LAN rule, quick out floating rule.

[gslongo]

@jimp:

You need both.
Policy route on LAN rule, quick out floating rule.

I'm sorry to say that but already tried :-( Just validated right now again..
The most strange is the floating route matches because packet count get incremented… But traceroute shows routing path is not correct...
So as a summary :

Floating rule quick in+out with gateway option
LAN rule with gateway option

Is that right ?

[jimp]

No. Just exactly what I wrote.

[gslongo]

@jimp:

No. Just exactly what I wrote.

:o

I'm sorry to ask again but your answers are very short and not clear to me :-[ I'm not inside your head  ;D

Could you please just describe what should work ? Because it seems what we tested here is according what you suggested ?

Thanks!

[jimp]

Pass rule on LAN with gateway set.
Floating quick pass out on WAN without gateway set.

[makz]

@jimp:

Pass rule on LAN with gateway set.
Floating quick pass out on WAN without gateway set.

Hello, i tried that but still not working, maybe i miss something but i don't know what, screenshots in attachments

anchor "userrules/*" all
pass out quick on lagg0_vlan2000 reply-to (lagg0_vlan2000 192.168.0.5) inet from <backup_servers> to any flags S/SA keep state label "USER_RULE: TEST ROUTING"
pass in quick on openvpn inet all flags S/SA keep state label "USER_RULE: TEMP"
pass in quick on openvpn inet from any to (self) flags S/SA keep state label "USER_RULE: TEMP"
pass in quick on lagg0_vlan2000 reply-to (lagg0_vlan2000 192.168.0.5) inet proto tcp from any to (self) port = https flags S/SA keep state label "USER_RULE: Easy Rule: Passed from Firewall Log View"
pass in quick on lagg0_vlan2000 reply-to (lagg0_vlan2000 192.168.0.5) inet proto tcp from <itop_public_ip> to (self) port = rsh-spx flags S/SA keep state label "USER_RULE"
pass in quick on lagg0_vlan2000 reply-to (lagg0_vlan2000 192.168.0.5) inet proto tcp from <itop_public_ip> to (self) port = https flags S/SA keep state label "USER_RULE"
pass in quick on lagg0_vlan2000 reply-to (lagg0_vlan2000 192.168.0.5) inet proto tcp from any to (self) port = rsh-spx flags S/SA keep state label "USER_RULE: Easy Rule: Passed from Firewall Log View"
pass in quick on lagg0_vlan1007 inet proto carp from any to (self) keep state label "USER_RULE: CARP ALLOWED"
pass in quick on lagg0_vlan1007 inet proto pfsync from any to (self) keep state label "USER_RULE: PFSYNC ALLOWED"
pass in quick on lagg0_vlan2010 inet proto tcp from any to (self) port = http flags S/SA keep state label "USER_RULE: WEB INTERFACE"
pass in quick on lagg0_vlan2010 inet proto tcp from any to (self) port = https flags S/SA keep state label "USER_RULE: WEB INTERFACE"
pass in quick on lagg0_vlan2010 inet proto tcp from any to (self) port = rsh-spx flags S/SA keep state label "USER_RULE: SSH"
pass in quick on lagg0_vlan2010 inet proto icmp from any to (self) keep state label "USER_RULE"
pass in quick on lagg0_vlan2010 inet from any to <hq_lans> flags S/SA keep state label "USER_RULE: oldlan2hqlans"
block drop in quick on lagg0_vlan2010 inet from any to <lans_rfc1918> label "USER_RULE: LAST RULE-1"
pass in quick on lagg0_vlan1008 inet from <backup_servers> to <negate_networks> flags S/SA keep state label "NEGATE_ROUTE: Negate policy routing for destination"
pass in quick on lagg0_vlan1008 route-to (lagg0_vlan2000 192.168.0.1) inet from <backup_servers> to any flags S/SA keep state label "USER_RULE: TEST ROUTING"</backup_servers></negate_networks></backup_servers></lans_rfc1918></hq_lans></itop_public_ip></itop_public_ip></backup_servers>

I also tried to disable the auto generated reply-to on the floating rule but still not working

[root@backup ~]# traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  gateway (10.10.3.1)  1.169 ms  1.115 ms  1.074 ms
 2  192.168.0.5 (192.168.0.5)  0.459 ms  0.432 ms  0.399 ms^C

Thank you in advance.

[jimp]

Hmm, that same style rule works when I use it. Try checking "Disable reply-to" on that rule as well.

[makz]

@jimp:

Hmm, that same style rule works when I use it. Try checking "Disable reply-to" on that rule as well.

As said in the latest message, i already tried this disabling the automatic reply-to. Still not working.

thank you.

[jimp]

OK, so then something your boxes are sending is not matching that rule, but falling through to the other rule. So compare the two:

default rule:

pass out  route-to ( lagg0_vlan2000 192.168.0.5 ) from 192.168.0.10 to !192.168.0.0/24 tracker 1000008011 keep state allow-opts label "let out anything from firewall host itself"

Your rules (assuming you did disable reply-to):

pass in quick on lagg0_vlan1008 inet from <backup_servers>to <negate_networks>flags S/SA keep state label "NEGATE_ROUTE: Negate policy routing for destination"
pass out quick on lagg0_vlan2000 inet from <backup_servers>to any flags S/SA keep state label "USER_RULE: TEST ROUTING"</backup_servers></negate_networks></backup_servers> 

Three things jump out:

#1: The first rule allows packets with ip options, so check that in your floating rule
#2: The first rule does not filter TCP flags, clone that rule and make one that is TCP only but allows any flags
#3: Try disabling the policy route negation rules under System > Advanced, Firewall & NAT tab

[makz]

@jimp:

OK, so then something your boxes are sending is not matching that rule, but falling through to the other rule. So compare the two:

default rule:

pass out  route-to ( lagg0_vlan2000 192.168.0.5 ) from 192.168.0.10 to !192.168.0.0/24 tracker 1000008011 keep state allow-opts label "let out anything from firewall host itself"

Your rules (assuming you did disable reply-to):

pass in quick on lagg0_vlan1008 inet from <backup_servers>to <negate_networks>flags S/SA keep state label "NEGATE_ROUTE: Negate policy routing for destination"
pass out quick on lagg0_vlan2000 inet from <backup_servers>to any flags S/SA keep state label "USER_RULE: TEST ROUTING"</backup_servers></negate_networks></backup_servers> 

Three things jump out:

#1: The first rule allows packets with ip options, so check that in your floating rule
#2: The first rule does not filter TCP flags, clone that rule and make one that is TCP only but allows any flags
#3: Try disabling the policy route negation rules under System > Advanced, Firewall & NAT tab

Thank for the response.

I tried those three things, no one of them works

with or without ip options allowed

Screenshot in attachments

pass out quick on lagg0_vlan2000 inet from <backup_servers>to any flags S/SA keep state allow-opts label "USER_RULE: TEST ROUTING"  # Floating

pass in quick on lagg0_vlan1008 route-to (lagg0_vlan2000 192.168.0.1) inet from <backup_servers>to any flags S/SA keep state allow-opts label "USER_RULE: TEST ROUTING"
pass in quick on lagg0_vlan1008 route-to (lagg0_vlan2000 192.168.0.1) inet proto tcp from <backup_servers>to any flags any keep state allow-opts label "USER_RULE: TEST ROUTING"</backup_servers></backup_servers></backup_servers> 

Here's the full dump of pfctl http://pastebin.com/jw4mXLTf

TCP Traceroute

[root@backup ~]# traceroute -T 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  gateway (10.10.3.1)  1.193 ms  1.143 ms  1.108 ms
 2  192.168.0.5 (192.168.0.5)  0.451 ms  0.433 ms  0.411 ms^C

ICMP Traceroute

[root@backup ~]# traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  gateway (10.10.3.1)  1.144 ms  1.094 ms  1.034 ms
 2  192.168.0.5 (192.168.0.5)  0.456 ms  0.407 ms  0.386 ms

thank you in advance