Policy based routing, multi-WAN and gateway on same subnet

Derelict

You might have to put a rule that matches the traffic coming from the routes behind 192.168.0.1 and disable reply-to there in the advanced settings on that rule, else replies will probably be explicitly sent to the gateway set on that interface configuration.

Naturally, the router at 192.168.0.1 would have to know to route the correct traffic back to pfSense, and not 192.168.0.5.

If you have traffic coming from behind 192.168.0.1, being sent to pfSense, and has to then be routed to 192.168.0.5 you get into hairpinning traffic and should consider a redesign.

johnpoz

^ yeah that could be an issue if your not natting.. But I don't see why pfsense would have an issue with gateways on the same transit.. But sure if this .1 router had a default of .5 on the transit you could have issues as Derelict describes.

I think is always easier with a picture. So from what the OP states we could have something like the below drawing. Now if we assume no nat and this .1 router has a default gateway of .5 as well then we have a bit of problem if there is not route on the .1 to the 192.168.1 network in my drawing.

But such a scenario is common. So I don't see why pfsense should have an issue with multiple gateways on the same network.

So pfsense (bottom router) has its default set to the 192.168.0.5 address. This is how his clients get to all networks it doesn't have specific routes for. Doesn't matter if it nats or not.. So if we add a new router into the mix on the transit on how I get to some specific networks.. The 192.168.2 in my drawing. Pfsense needs to know that to get to 192.168.2 he sends traffic to 192.168.0.1

Now this policy route needs to be above a rule that would allow traffic and send it out the default gateway.

So pfsense send traffic to .1, he sends it on to the 192.168.2.x box.. That box answers then .1 router needs to send it back to our pfsense.3 in the example. If we are natting so that traffic looks like it came from .3 we don't have an issue. But we are not natting, then yes that .1 router needs to know to get to the 192.168.1 he needs to route that traffic to the .3 on the transit.. If he sends it over to .5 even if .5 know to send it to .3 you have a problem.

If the 192.168.2 is using some other gateway other than that .1 router you could have problems, etc. etc.

To troubleshoot the OP actual problem we could for sure use more info. But If pfsense can not have more than one gateway in the same transit network then that would be a real serious flaw in its usability wouldn't it… In your typical home user or smb, this would be an uncommon setup to be sure. But in enterprise transit network with multiple routers is very very common.

3routersontransit.jpg_thumb

Derelict

It can deal with it. It's just that an interface with a gateway defined in the interface configuration is considered a WAN and thus all incoming states get reply-to to the WAN gateway (what 99.999% of installations want) so you need to bypass that so reply traffic on those states is routed according to the routing table.

So on WAN there you would add pass rules sourced from 192.168.2.0/24 with reply-to disabled on that rule.

When dealing with multiple gateways on an inside transit interface you don't run into that because you generally do not configure a gateway on inside interfaces.

johnpoz

Great info Derelict.. Yeah I have not actually done this on a pfsense, but I didn't think it would have an issue with it.. Since I know there are many instances of pfsense used in the enterprise with more and more all the time.. Which is fantastic news!!

And if something like this could not be done it would be a big issue ;)

So its clear - this is the checkbox your talking about right Derelict.

disablereplyto.jpg_thumb

Derelict

Roger.

makz

Hello

I'm gslongo's collegue.

Here is in attachments a little diagram of what we have.

So the user network 10.1.1.0/24 reach internet through the default route -> 192.168.0.5.
But we want the Server (10.10.3.50) to reach internet through the second gateway -> 192.168.0.1.

I've also attached the rules screen, in this screen you'll find BACKUP SERVERS (Alias for 10.10.3.50), WANGW (Gateway name for 192.168.0.1)

In the rule, i've checked "Disable reply-to", the server still use the default route (but i'm not sure if i need to check this box in this rule).

Thank you.

![Untitled Diagram (1).png](/public/imported_attachments/1/Untitled Diagram (1).png)
![Untitled Diagram (1).png_thumb](/public/imported_attachments/1/Untitled Diagram (1).png_thumb)
![Screenshot from 2016-12-09 14-06-21.png](/public/imported_attachments/1/Screenshot from 2016-12-09 14-06-21.png)
![Screenshot from 2016-12-09 14-06-21.png_thumb](/public/imported_attachments/1/Screenshot from 2016-12-09 14-06-21.png_thumb)

johnpoz

And what interface do you have these rule setup on? Clearly your backup servers on a different vlan than your users.. You sure your placing the rules on the correct interface.

makz

Yep, differents VLAN

In fact, we have a lot of VLAN the intervlan routing are done by the core switches and we have a vlan named INTERCO

In all vlan, the default gateway are the core switchs.

All core switches have a default route to the pfsense vIP (10.10.8.12).

Like that, pfsense get all packets destinated of internet in his INTERCO interface and the filtering is done on this interface.

My rule is done in INTERCO interface.

I didn't included the core switching part in my diagram to keep it simple.

Derelict

Looks like you kept it even simpler by not including a diagram at all.

makz

Hello,

I've included coreswitching part and some more informations on this diagram

As seen in older screenshots, two gateway are configured on WAN interface (192.168.0.1 & 192.168.0.5), my rule "Matching source 10.10.3.50 -> Gateway 192.168.0.1" is done in INTERCO interface and it's on top of the rule list.

Thank you.

![Screenshot from 2016-12-12 10-19-54.png](/public/imported_attachments/1/Screenshot from 2016-12-12 10-19-54.png)
![Screenshot from 2016-12-12 10-19-54.png_thumb](/public/imported_attachments/1/Screenshot from 2016-12-12 10-19-54.png_thumb)
![Untitled Diagram (5).png](/public/imported_attachments/1/Untitled Diagram (5).png)
![Untitled Diagram (5).png_thumb](/public/imported_attachments/1/Untitled Diagram (5).png_thumb)

makz

Hello,

I'm back with some news !

Today i've configured a little freebsd router to try my setup on a vanilla bsd.

My basic setup work well on this vanilla router.

pass out quick on $int_if route-to ($ext_if 192.168.0.1) inet from $bck_srv
pass in quick on $int_if route-to ($ext_if 192.168.0.1) inet from $bck_srv

Now i know this is not a freebsd issue but more a pfsense issue.

I've read the pf rules generated by pfsense and i saw some hidden rules

pass out  route-to ( lagg0_vlan2000 192.168.0.5 ) from 192.168.0.10 to !192.168.0.0/24 tracker 1000008011 keep state allow-opts label "let out anything from firewall host itself"
pass out  route-to ( lagg0_vlan2000 192.168.0.5 ) from 192.168.0.12 to !192.168.0.0/24 tracker 1000008012 keep state allow-opts label "let out anything from firewall host itself"

I've removed those rules from pf and magic it work !

So,

I saw in the bug tracker, if i place my rule in "Floating rules" this should disable the hidden rule <https: 1823="" redmine.pfsense.org="" issues="">This still won't work … Here is all my "route-to" rules from my non working pf configuration

pfctl -sa | grep route-to
pass out route-to (lagg0_vlan2000 192.168.0.5) inet from 192.168.0.10 to ! 192.168.0.0/24 flags S/SA keep state allow-opts label "let out anything from firewall host itself"
pass out route-to (lagg0_vlan2000 192.168.0.5) inet from 192.168.0.12 to ! 192.168.0.0/24 flags S/SA keep state allow-opts label "let out anything from firewall host itself"
pass out quick on lagg0_vlan1008 route-to (lagg0_vlan2000 192.168.0.1) inet from <backup_servers> to any flags S/SA keep state label "USER_RULE: TEST ROUTING"
pass in quick on lagg0_vlan1008 route-to (lagg0_vlan2000 192.168.0.1) inet from <backup_servers> to any flags S/SA keep state label "USER_RULE: TEST ROUTING"</backup_servers></backup_servers>

I've tried to inverse the rules (my rule before the hidden rule), this still won't work

I've also tried to remove the route-to argument to the hidden rule

pass out  from 192.168.0.10 to !192.168.0.0/24 tracker 1000008011 keep state allow-opts label "let out anything from firewall host itself"
pass out  from 192.168.0.12 to !192.168.0.0/24 tracker 1000008012 keep state allow-opts label "let out anything from firewall host itself"

and this work great !

For now i have some questions …

What is the purpose of this hidden rule ?
Can i disable it permanently ?
Can i just remove the route-to argument permanently ?
Is there any option to bypass this rule for some cases ?

Thank you !

Have a great day.</https:>

gslongo

A bug related to this has been opened here : https://redmine.pfsense.org/issues/7033

Can it be analysed by a developper ?

//EDIT: Also found user having same issue here : https://redmine.pfsense.org/issues/6625

jimp

The floating rules do not "disable" the hidden rule, they override it. When you make your floating rules, be sure to check "quick" and that the rules will match the traffic going to your other gateway. The hidden rules do not have quick, so a quick rule will match and the non-quick rule will never be processed.

Post the exact floating rules you made and it should be fairly easy to tell why they aren't working.

makz

@jimp:

The floating rules do not "disable" the hidden rule, they override it. When you make your floating rules, be sure to check "quick" and that the rules will match the traffic going to your other gateway. The hidden rules do not have quick, so a quick rule will match and the non-quick rule will never be processed.

Post the exact floating rules you made and it should be fairly easy to tell why they aren't working.

Hello, thank you for the response, but … there's a but ...

My floating rule have the "quick" option, you can see it in my screenshot on the bugtracker (i repost it here in attachment)

Here's the block of rules with the hidden one and mine

pass out inet all flags S/SA keep state allow-opts label "let out anything IPv4 from firewall host itself"
pass out inet6 all flags S/SA keep state allow-opts label "let out anything IPv6 from firewall host itself"
pass out route-to (lagg0_vlan2000 192.168.0.5) inet from 192.168.0.10 to ! 192.168.0.0/24 flags S/SA keep state allow-opts label "let out anything from firewall host itself"
pass out route-to (lagg0_vlan2000 192.168.0.5) inet from 192.168.0.12 to ! 192.168.0.0/24 flags S/SA keep state allow-opts label "let out anything from firewall host itself"
pass in quick on lagg0_vlan1007 proto tcp from any to (lagg0_vlan1007) port = https flags S/SA keep state label "anti-lockout rule"
pass in quick on lagg0_vlan1007 proto tcp from any to (lagg0_vlan1007) port = http flags S/SA keep state label "anti-lockout rule"
pass in quick on lagg0_vlan1007 proto tcp from any to (lagg0_vlan1007) port = rsh-spx flags S/SA keep state label "anti-lockout rule"
anchor "userrules/*" all
pass in quick on lagg0_vlan1008 inet from <backup_servers>to <negate_networks>flags S/SA keep state label "NEGATE_ROUTE: Negate policy routing for destination"
pass in quick on lagg0_vlan1008 route-to (lagg0_vlan2000 192.168.0.1) inet from <backup_servers>to any flags S/SA keep state label "USER_RULE: TEST ROUTING"
pass out quick on lagg0_vlan1008 inet from <backup_servers>to <negate_networks>flags S/SA keep state label "NEGATE_ROUTE: Negate policy routing for destination"
pass out quick on lagg0_vlan1008 route-to (lagg0_vlan2000 192.168.0.1) inet from <backup_servers>to any flags S/SA keep state label "USER_RULE: TEST ROUTING"</backup_servers></negate_networks></backup_servers></backup_servers></negate_networks></backup_servers>

And when i don't patch the filter.inc to remove the hidden rule, my traffic is routed to the default gateway as explained before.

[root@backup ~]# traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  gateway (10.10.3.1)  0.541 ms  1.298 ms  1.264 ms
 2  192.168.0.5 (192.168.0.5)  0.406 ms  0.404 ms  0.350 ms

2665af6955eeb7bb48c0472ca4926722.png_thumb

jimp

Do not set a gateway on those

makz

@jimp:

Do not set a gateway on those

Hi,
Thank you for your reply

Not sure I understand well
On which rules ? Because we do not find where to modify the hidden rules.
And if you mean on the floating rules, then where to set the gateway if not in advanced options of these rules?

Thank you for advance

jimp

Use a static route

jimp

Or policy route on LAN rules

gslongo

@jimp:

Or policy route on LAN rules

Hi Jim,

Static routes are not possible in that case
We already tried same rule on LAN interface but same behavior. And if I understand well the solution shoud be a floating rule ? But then why it is not working in that case ?
We tried every possible scenarios (I think) this is why at the end we opened a bug on the bug tracker.

Thank you!

jimp

You need both.
Policy route on LAN rule, quick out floating rule.