Policy based routing, multi-WAN and gateway on same subnet

[Derelict]

It can deal with it. It's just that an interface with a gateway defined in the interface configuration is considered a WAN and thus all incoming states get reply-to to the WAN gateway (what 99.999% of installations want) so you need to bypass that so reply traffic on those states is routed according to the routing table.

So on WAN there you would add pass rules sourced from 192.168.2.0/24 with reply-to disabled on that rule.

When dealing with multiple gateways on an inside transit interface you don't run into that because you generally do not configure a gateway on inside interfaces.

[johnpoz]

Great info Derelict..  Yeah I have not actually done this on a pfsense, but I didn't think it would have an issue with it.. Since I know there are many instances of pfsense used in the enterprise with more and more all the time.. Which is fantastic news!!

And if something like this could not be done it would be a big issue ;)

So its clear - this is the checkbox your talking about right Derelict.

[Derelict]

Roger.

[makz]

Hello

I'm gslongo's collegue.

Here is in attachments a little diagram of what we have.

So the user network 10.1.1.0/24 reach internet through the default route -> 192.168.0.5.
But we want the Server (10.10.3.50) to reach internet through the second gateway -> 192.168.0.1.

I've also attached the rules screen, in this screen you'll find BACKUP SERVERS (Alias for 10.10.3.50), WANGW (Gateway name for 192.168.0.1)

In the rule, i've checked "Disable reply-to", the server still use the default route (but i'm not sure if i need to check this box in this rule).

Thank you.

.png)
.png_thumb)

[johnpoz]

And what interface do you have these rule setup on?  Clearly your backup servers on a different vlan than your users..  You sure your placing the rules on the correct interface.

[makz]

Yep, differents VLAN

In fact, we have a lot of VLAN the intervlan routing are done by the core switches and we have a vlan named INTERCO

In all vlan, the default gateway are the core switchs.

All core switches have a default route to the pfsense vIP (10.10.8.12).

Like that, pfsense get all packets destinated of internet in his INTERCO interface and the filtering is done on this interface.

My rule is done in INTERCO interface.

I didn't included the core switching part in my diagram to keep it simple.

[Derelict]

Looks like you kept it even simpler by not including a diagram at all.

[makz]

Hello,

I've included coreswitching part and some more informations on this diagram

As seen in older screenshots, two gateway are configured on WAN interface (192.168.0.1 & 192.168.0.5), my rule "Matching source 10.10.3.50 -> Gateway 192.168.0.1" is done in INTERCO interface and it's on top of the rule list.

Thank you.

.png)
.png_thumb)

[makz]

Hello,

I'm back with some news !

Today i've configured a little freebsd router to try my setup on a vanilla bsd.

My basic setup work well on this vanilla router.

pass out quick on $int_if route-to ($ext_if 192.168.0.1) inet from $bck_srv
pass in quick on $int_if route-to ($ext_if 192.168.0.1) inet from $bck_srv

Now i know this is not a freebsd issue but more a pfsense issue.

I've read the pf rules generated by pfsense and i saw some hidden rules

pass out  route-to ( lagg0_vlan2000 192.168.0.5 ) from 192.168.0.10 to !192.168.0.0/24 tracker 1000008011 keep state allow-opts label "let out anything from firewall host itself"
pass out  route-to ( lagg0_vlan2000 192.168.0.5 ) from 192.168.0.12 to !192.168.0.0/24 tracker 1000008012 keep state allow-opts label "let out anything from firewall host itself"

I've removed those rules from pf and magic it work !

So,

I saw in the bug tracker, if i place my rule in "Floating rules" this should disable the hidden rule <https: 1823="" redmine.pfsense.org="" issues="">This still won't work … Here is all my "route-to" rules from my non working pf configuration

pfctl -sa | grep route-to
pass out route-to (lagg0_vlan2000 192.168.0.5) inet from 192.168.0.10 to ! 192.168.0.0/24 flags S/SA keep state allow-opts label "let out anything from firewall host itself"
pass out route-to (lagg0_vlan2000 192.168.0.5) inet from 192.168.0.12 to ! 192.168.0.0/24 flags S/SA keep state allow-opts label "let out anything from firewall host itself"
pass out quick on lagg0_vlan1008 route-to (lagg0_vlan2000 192.168.0.1) inet from <backup_servers> to any flags S/SA keep state label "USER_RULE: TEST ROUTING"
pass in quick on lagg0_vlan1008 route-to (lagg0_vlan2000 192.168.0.1) inet from <backup_servers> to any flags S/SA keep state label "USER_RULE: TEST ROUTING"</backup_servers></backup_servers>

I've tried to inverse the rules (my rule before the hidden rule), this still won't work

I've also tried to remove the route-to argument to the hidden rule

pass out  from 192.168.0.10 to !192.168.0.0/24 tracker 1000008011 keep state allow-opts label "let out anything from firewall host itself"
pass out  from 192.168.0.12 to !192.168.0.0/24 tracker 1000008012 keep state allow-opts label "let out anything from firewall host itself"

and this work great !

For now i have some questions …

What is the purpose of this hidden rule ?
Can i disable it permanently ?
Can i just remove the route-to argument permanently ?
Is there any option to bypass this rule for some cases ?

Thank you !

Have a great day.</https:>

[gslongo]

A bug related to this has been opened here : https://redmine.pfsense.org/issues/7033

Can it be analysed by a developper ?

//EDIT: Also found user having same issue here : https://redmine.pfsense.org/issues/6625

[jimp]

The floating rules do not "disable" the hidden rule, they override it. When you make your floating rules, be sure to check "quick" and that the rules will match the traffic going to your other gateway. The hidden rules do not have quick, so a quick rule will match and the non-quick rule will never be processed.

Post the exact floating rules you made and it should be fairly easy to tell why they aren't working.

[makz]

@jimp:

The floating rules do not "disable" the hidden rule, they override it. When you make your floating rules, be sure to check "quick" and that the rules will match the traffic going to your other gateway. The hidden rules do not have quick, so a quick rule will match and the non-quick rule will never be processed.

Post the exact floating rules you made and it should be fairly easy to tell why they aren't working.

Hello, thank you for the response, but … there's a but ...

My floating rule have the "quick" option, you can see it in my screenshot on the bugtracker (i repost it here in attachment)

Here's the block of rules with the hidden one and mine

pass out inet all flags S/SA keep state allow-opts label "let out anything IPv4 from firewall host itself"
pass out inet6 all flags S/SA keep state allow-opts label "let out anything IPv6 from firewall host itself"
pass out route-to (lagg0_vlan2000 192.168.0.5) inet from 192.168.0.10 to ! 192.168.0.0/24 flags S/SA keep state allow-opts label "let out anything from firewall host itself"
pass out route-to (lagg0_vlan2000 192.168.0.5) inet from 192.168.0.12 to ! 192.168.0.0/24 flags S/SA keep state allow-opts label "let out anything from firewall host itself"
pass in quick on lagg0_vlan1007 proto tcp from any to (lagg0_vlan1007) port = https flags S/SA keep state label "anti-lockout rule"
pass in quick on lagg0_vlan1007 proto tcp from any to (lagg0_vlan1007) port = http flags S/SA keep state label "anti-lockout rule"
pass in quick on lagg0_vlan1007 proto tcp from any to (lagg0_vlan1007) port = rsh-spx flags S/SA keep state label "anti-lockout rule"
anchor "userrules/*" all
pass in quick on lagg0_vlan1008 inet from <backup_servers>to <negate_networks>flags S/SA keep state label "NEGATE_ROUTE: Negate policy routing for destination"
pass in quick on lagg0_vlan1008 route-to (lagg0_vlan2000 192.168.0.1) inet from <backup_servers>to any flags S/SA keep state label "USER_RULE: TEST ROUTING"
pass out quick on lagg0_vlan1008 inet from <backup_servers>to <negate_networks>flags S/SA keep state label "NEGATE_ROUTE: Negate policy routing for destination"
pass out quick on lagg0_vlan1008 route-to (lagg0_vlan2000 192.168.0.1) inet from <backup_servers>to any flags S/SA keep state label "USER_RULE: TEST ROUTING"</backup_servers></negate_networks></backup_servers></backup_servers></negate_networks></backup_servers> 

And when i don't patch the filter.inc to remove the hidden rule, my traffic is routed to the default gateway as explained before.

[root@backup ~]# traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  gateway (10.10.3.1)  0.541 ms  1.298 ms  1.264 ms
 2  192.168.0.5 (192.168.0.5)  0.406 ms  0.404 ms  0.350 ms

[jimp]

Do not set a gateway on those

[makz]

@jimp:

Do not set a gateway on those

Hi,
Thank you for your reply

Not sure I understand well
On which rules ? Because we do not find where to modify the hidden rules.
And if you mean on the floating rules, then where to set the gateway if not in advanced options of these rules?

Thank you for advance

[jimp]

Use a static route

[jimp]

Or policy route on LAN rules

[gslongo]

@jimp:

Or policy route on LAN rules

Hi Jim,

Static routes are not possible in that case
We already tried same rule on LAN interface but same behavior. And if I understand well the solution shoud be a floating rule ? But then why it is not working in that case ?
We tried every possible scenarios (I think) this is why at the end we opened a bug on the bug tracker.

Thank you!

[jimp]

You need both.
Policy route on LAN rule, quick out floating rule.

[gslongo]

@jimp:

You need both.
Policy route on LAN rule, quick out floating rule.

I'm sorry to say that but already tried :-( Just validated right now again..
The most strange is the floating route matches because packet count get incremented… But traceroute shows routing path is not correct...
So as a summary :

Floating rule quick in+out with gateway option
LAN rule with gateway option

Is that right ?

[jimp]

No. Just exactly what I wrote.