Policy based routing, multi-WAN and gateway on same subnet



  • Hi All,

    We are facing to an issue in this particular situation.. We have two gateways on the same subnet 192.168.0.1 and 192.168.0.5. Default gateway on the firewall is 192.168.0.5.
    When creating a rule to route traffic to 192.168.0.1 it is not working. Traceroute show traffic pass through 192.168.0.5.. (and rule is correct).
    When changing, in that rule, for a gateway located into another subnet, traffic is correctly routed (traceroute OK).

    Any idea regarding this issue ?

    Thank you for advance !


  • LAYER 8 Global Moderator

    So you have the gateway created.. And then you created a rule to send specific traffic either on source IP, dest IP, port, etc. out a specific gateway.  And this rule is above the rule that would send said traffic out the default gateway.

    Post up your rules showing how your doing your policy routing.



  • What issue? Gateway in the same subnet don't work.



  • @heper:

    What issue? Gateway in the same subnet don't work.

    Hi,

    Do you know why ?

    Johnpoz, when you say "And this rule is above the rule that would send said traffic out the default gateway", what do you mean ? There is no such rule (policy based rule to default gateway)…

    Thanks guys


  • LAYER 8 Global Moderator

    do you have any rule that allow traffic above your policy based rule?

    Remember rules are top down, first rule to trigger wins no other rules are looked at.

    So if you have  rule that would allow the traffic and no gateway set, it would use the default gateway and normal routing.  And would never get to the rule that says hey go out this specific gateway.

    Is just easier if your post your rules so we can all see and be on the same page.

    As to using more than 1 gateway on the same network - I am not sure on that.  While its highly uncommon setup.  I have never tried it.. So not sure if pfsense would even let you create multiple gateways on the same network?

    But in general common issue people have with policy routing is placement of rules.  Either they place the policy rule on top and wonder why they can not get to their other local networks, or they have a rule above and wonder why they are not using the policy routing.



  • As to using more than 1 gateway on the same network - I am not sure on that.  While its highly uncommon setup.  I have never tried it.. So not sure if pfsense would even let you create multiple gateways on the same network?

    i haven't tried it myself either, but i've read about it in the past.

    i think its the routes on the WAN interfaces that get messed up.  (the gateways themselfs might not be an issue)

    example:

    
    192.168.5.0/24	link#2	U	0	1500	vmx1	
    192.168.5.3	link#2	UHS	0	16384	lo0
    
    
    
    192.168.5.0/24	link#3	U	0	1500	vmx2	
    192.168.5.2	link#3	UHS	0	16384	lo0
    
    

    as I see it, both can't be in the routing table at the same time.


  • LAYER 8 Global Moderator

    You can for sure have more specific routes in the table..  But yeah depending you could have an issue in the routing table that prevents such a thing.  I am remote currently, for sure not going to mess with pfsense gateways remotely in such a manner while I am vpn'd in - don't want to disconnect myself and then no way to get back in ;)  Have to give it test once I get home.


  • LAYER 8 Netgate

    You might have to put a rule that matches the traffic coming from the routes behind 192.168.0.1 and disable reply-to there in the advanced settings on that rule, else replies will probably be explicitly sent to the gateway set on that interface configuration.

    Naturally, the router at 192.168.0.1 would have to know to route the correct traffic back to pfSense, and not 192.168.0.5.

    If you have traffic coming from behind 192.168.0.1, being sent to pfSense, and has to then be routed to 192.168.0.5 you get into hairpinning traffic and should consider a redesign.


  • LAYER 8 Global Moderator

    ^ yeah that could be an issue if your not natting.. But I don't see why pfsense would have an issue with gateways on the same transit.. But sure if this .1 router had a default of .5 on the transit you could have issues as Derelict describes.

    I think is always easier with a picture.  So from what the OP states we could have something like the below drawing.  Now if we assume no nat and this .1 router has a default gateway of .5 as well then we have a bit of problem if there is not route on the .1 to the 192.168.1 network in my drawing.

    But such a scenario is common.  So I don't see why pfsense should have an issue with multiple gateways on the same network.

    So pfsense (bottom router) has its default set to the 192.168.0.5 address.  This is how his clients get to all networks it doesn't have specific routes for.  Doesn't matter if it nats or not..  So if we add a new router into the mix on the transit on how I get to some specific networks.. The 192.168.2 in my drawing.  Pfsense needs to know that to get to 192.168.2 he sends traffic to 192.168.0.1

    Now this policy route needs to be above a rule that would allow traffic and send it out the default gateway.

    So pfsense send traffic to .1, he sends it on to the 192.168.2.x box.. That box answers then .1 router needs to send it back to our pfsense.3 in the example.  If we are natting so that traffic looks like it came from .3 we don't have an issue.  But we are not natting, then yes that .1 router needs to know to get to the 192.168.1 he needs to route that traffic to the .3 on the transit.. If he sends it over to .5 even if .5 know to send it to .3 you have a problem.

    If the 192.168.2 is using some other gateway other than that .1 router you could have problems, etc. etc.

    To troubleshoot the OP actual problem we could for sure use more info.  But If pfsense can not have more than one gateway in the same transit network then that would be a real serious flaw in its usability wouldn't it… In your typical home user or smb, this would be an uncommon setup to be sure.  But in enterprise transit network with multiple routers is very very common.



  • LAYER 8 Netgate

    It can deal with it. It's just that an interface with a gateway defined in the interface configuration is considered a WAN and thus all incoming states get reply-to to the WAN gateway (what 99.999% of installations want) so you need to bypass that so reply traffic on those states is routed according to the routing table.

    So on WAN there you would add pass rules sourced from 192.168.2.0/24 with reply-to disabled on that rule.

    When dealing with multiple gateways on an inside transit interface you don't run into that because you generally do not configure a gateway on inside interfaces.


  • LAYER 8 Global Moderator

    Great info Derelict..  Yeah I have not actually done this on a pfsense, but I didn't think it would have an issue with it.. Since I know there are many instances of pfsense used in the enterprise with more and more all the time.. Which is fantastic news!!

    And if something like this could not be done it would be a big issue ;)

    So its clear - this is the checkbox your talking about right Derelict.



  • LAYER 8 Netgate

    Roger.



  • Hello

    I'm gslongo's collegue.

    Here is in attachments a little diagram of what we have.

    So the user network 10.1.1.0/24 reach internet through the default route -> 192.168.0.5.
    But we want the Server (10.10.3.50) to reach internet through the second gateway -> 192.168.0.1.

    I've also attached the rules screen, in this screen you'll find BACKUP SERVERS (Alias for 10.10.3.50), WANGW (Gateway name for 192.168.0.1)

    In the rule, i've checked "Disable reply-to", the server still use the default route (but i'm not sure if i need to check this box in this rule).

    Thank you.

    ![Untitled Diagram (1).png](/public/imported_attachments/1/Untitled Diagram (1).png)
    ![Untitled Diagram (1).png_thumb](/public/imported_attachments/1/Untitled Diagram (1).png_thumb)
    ![Screenshot from 2016-12-09 14-06-21.png](/public/imported_attachments/1/Screenshot from 2016-12-09 14-06-21.png)
    ![Screenshot from 2016-12-09 14-06-21.png_thumb](/public/imported_attachments/1/Screenshot from 2016-12-09 14-06-21.png_thumb)


  • LAYER 8 Global Moderator

    And what interface do you have these rule setup on?  Clearly your backup servers on a different vlan than your users..  You sure your placing the rules on the correct interface.



  • Yep, differents VLAN

    In fact, we have a lot of VLAN the intervlan routing are done by the core switches and we have a vlan named INTERCO

    In all vlan, the default gateway are the core switchs.

    All core switches have a default route to the pfsense vIP (10.10.8.12).

    Like that, pfsense get all packets destinated of internet in his INTERCO interface and the filtering is done on this interface.

    My rule is done in INTERCO interface.

    I didn't included the core switching part in my diagram to keep it simple.


  • LAYER 8 Netgate

    Looks like you kept it even simpler by not including a diagram at all.



  • Hello,

    I've included coreswitching part and some more informations on this diagram

    As seen in older screenshots, two gateway are configured on WAN interface (192.168.0.1 & 192.168.0.5), my rule "Matching source 10.10.3.50 -> Gateway 192.168.0.1" is done in INTERCO interface and it's on top of the rule list.

    Thank you.

    ![Screenshot from 2016-12-12 10-19-54.png](/public/imported_attachments/1/Screenshot from 2016-12-12 10-19-54.png)
    ![Screenshot from 2016-12-12 10-19-54.png_thumb](/public/imported_attachments/1/Screenshot from 2016-12-12 10-19-54.png_thumb)
    ![Untitled Diagram (5).png](/public/imported_attachments/1/Untitled Diagram (5).png)
    ![Untitled Diagram (5).png_thumb](/public/imported_attachments/1/Untitled Diagram (5).png_thumb)



  • Hello,

    I'm back with some news !

    Today i've configured a little freebsd router to try my setup on a vanilla bsd.

    My basic setup work well on this vanilla router.

    pass out quick on $int_if route-to ($ext_if 192.168.0.1) inet from $bck_srv
    pass in quick on $int_if route-to ($ext_if 192.168.0.1) inet from $bck_srv
    

    Now i know this is not a freebsd issue but more a pfsense issue.

    I've read the pf rules generated by pfsense and i saw some hidden rules

    pass out  route-to ( lagg0_vlan2000 192.168.0.5 ) from 192.168.0.10 to !192.168.0.0/24 tracker 1000008011 keep state allow-opts label "let out anything from firewall host itself"
    pass out  route-to ( lagg0_vlan2000 192.168.0.5 ) from 192.168.0.12 to !192.168.0.0/24 tracker 1000008012 keep state allow-opts label "let out anything from firewall host itself"
    

    I've removed those rules from pf and magic it work !

    So,

    I saw in the bug tracker, if i place my rule in "Floating rules" this should disable the hidden rule <https: 1823="" redmine.pfsense.org="" issues="">This still won't work … Here is all my "route-to" rules from my non working pf configuration

    pfctl -sa | grep route-to
    pass out route-to (lagg0_vlan2000 192.168.0.5) inet from 192.168.0.10 to ! 192.168.0.0/24 flags S/SA keep state allow-opts label "let out anything from firewall host itself"
    pass out route-to (lagg0_vlan2000 192.168.0.5) inet from 192.168.0.12 to ! 192.168.0.0/24 flags S/SA keep state allow-opts label "let out anything from firewall host itself"
    pass out quick on lagg0_vlan1008 route-to (lagg0_vlan2000 192.168.0.1) inet from <backup_servers> to any flags S/SA keep state label "USER_RULE: TEST ROUTING"
    pass in quick on lagg0_vlan1008 route-to (lagg0_vlan2000 192.168.0.1) inet from <backup_servers> to any flags S/SA keep state label "USER_RULE: TEST ROUTING"</backup_servers></backup_servers>
    

    I've tried to inverse the rules (my rule before the hidden rule), this still won't work

    I've also tried to remove the route-to argument to the hidden rule

    pass out  from 192.168.0.10 to !192.168.0.0/24 tracker 1000008011 keep state allow-opts label "let out anything from firewall host itself"
    pass out  from 192.168.0.12 to !192.168.0.0/24 tracker 1000008012 keep state allow-opts label "let out anything from firewall host itself"
    

    and this work great !

    For now i have some questions …

    What is the purpose of this hidden rule ?
    Can i disable it permanently ?
    Can i just remove the route-to argument permanently ?
    Is there any option to bypass this rule for some cases ?

    Thank you !

    Have a great day.</https:>



  • A bug related to this has been opened here : https://redmine.pfsense.org/issues/7033

    Can it be analysed by a developper ?

    //EDIT: Also found user having same issue here : https://redmine.pfsense.org/issues/6625


  • Rebel Alliance Developer Netgate

    The floating rules do not "disable" the hidden rule, they override it. When you make your floating rules, be sure to check "quick" and that the rules will match the traffic going to your other gateway. The hidden rules do not have quick, so a quick rule will match and the non-quick rule will never be processed.

    Post the exact floating rules you made and it should be fairly easy to tell why they aren't working.



  • @jimp:

    The floating rules do not "disable" the hidden rule, they override it. When you make your floating rules, be sure to check "quick" and that the rules will match the traffic going to your other gateway. The hidden rules do not have quick, so a quick rule will match and the non-quick rule will never be processed.

    Post the exact floating rules you made and it should be fairly easy to tell why they aren't working.

    Hello, thank you for the response, but … there's a but ...

    My floating rule have the "quick" option, you can see it in my screenshot on the bugtracker (i repost it here in attachment)

    Here's the block of rules with the hidden one and mine

    pass out inet all flags S/SA keep state allow-opts label "let out anything IPv4 from firewall host itself"
    pass out inet6 all flags S/SA keep state allow-opts label "let out anything IPv6 from firewall host itself"
    pass out route-to (lagg0_vlan2000 192.168.0.5) inet from 192.168.0.10 to ! 192.168.0.0/24 flags S/SA keep state allow-opts label "let out anything from firewall host itself"
    pass out route-to (lagg0_vlan2000 192.168.0.5) inet from 192.168.0.12 to ! 192.168.0.0/24 flags S/SA keep state allow-opts label "let out anything from firewall host itself"
    pass in quick on lagg0_vlan1007 proto tcp from any to (lagg0_vlan1007) port = https flags S/SA keep state label "anti-lockout rule"
    pass in quick on lagg0_vlan1007 proto tcp from any to (lagg0_vlan1007) port = http flags S/SA keep state label "anti-lockout rule"
    pass in quick on lagg0_vlan1007 proto tcp from any to (lagg0_vlan1007) port = rsh-spx flags S/SA keep state label "anti-lockout rule"
    anchor "userrules/*" all
    pass in quick on lagg0_vlan1008 inet from <backup_servers>to <negate_networks>flags S/SA keep state label "NEGATE_ROUTE: Negate policy routing for destination"
    pass in quick on lagg0_vlan1008 route-to (lagg0_vlan2000 192.168.0.1) inet from <backup_servers>to any flags S/SA keep state label "USER_RULE: TEST ROUTING"
    pass out quick on lagg0_vlan1008 inet from <backup_servers>to <negate_networks>flags S/SA keep state label "NEGATE_ROUTE: Negate policy routing for destination"
    pass out quick on lagg0_vlan1008 route-to (lagg0_vlan2000 192.168.0.1) inet from <backup_servers>to any flags S/SA keep state label "USER_RULE: TEST ROUTING"</backup_servers></negate_networks></backup_servers></backup_servers></negate_networks></backup_servers> 
    

    And when i don't patch the filter.inc to remove the hidden rule, my traffic is routed to the default gateway as explained before.

    [root@backup ~]# traceroute 8.8.8.8
    traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
     1  gateway (10.10.3.1)  0.541 ms  1.298 ms  1.264 ms
     2  192.168.0.5 (192.168.0.5)  0.406 ms  0.404 ms  0.350 ms
    
    



  • Rebel Alliance Developer Netgate

    Do not set a gateway on those



  • @jimp:

    Do not set a gateway on those

    Hi,
    Thank you for your reply

    Not sure I understand well
    On which rules ? Because we do not find where to modify the hidden rules.
    And if you mean on the floating rules, then where to set the gateway if not in advanced options of these rules?

    Thank you for advance


  • Rebel Alliance Developer Netgate

    Use a static route


  • Rebel Alliance Developer Netgate

    Or policy route on LAN rules



  • @jimp:

    Or policy route on LAN rules

    Hi Jim,

    Static routes are not possible in that case
    We already tried same rule on LAN interface but same behavior. And if I understand well the solution shoud be a floating rule ? But then why it is not working in that case ?
    We tried every possible scenarios (I think) this is why at the end we opened a bug on the bug tracker.

    Thank you!


  • Rebel Alliance Developer Netgate

    You need both.
    Policy route on LAN rule, quick out floating rule.



  • @jimp:

    You need both.
    Policy route on LAN rule, quick out floating rule.

    I'm sorry to say that but already tried :-( Just validated right now again..
    The most strange is the floating route matches because packet count get incremented… But traceroute shows routing path is not correct...
    So as a summary :

    Floating rule quick in+out with gateway option
    LAN rule with gateway option

    Is that right ?


  • Rebel Alliance Developer Netgate

    No. Just exactly what I wrote.



  • @jimp:

    No. Just exactly what I wrote.

    :o

    I'm sorry to ask again but your answers are very short and not clear to me :-[ I'm not inside your head  ;D

    Could you please just describe what should work ? Because it seems what we tested here is according what you suggested ?

    Thanks!


  • Rebel Alliance Developer Netgate

    Pass rule on LAN with gateway set.
    Floating quick pass out on WAN without gateway set.



  • @jimp:

    Pass rule on LAN with gateway set.
    Floating quick pass out on WAN without gateway set.

    Hello, i tried that but still not working, maybe i miss something but i don't know what, screenshots in attachments

    anchor "userrules/*" all
    pass out quick on lagg0_vlan2000 reply-to (lagg0_vlan2000 192.168.0.5) inet from <backup_servers> to any flags S/SA keep state label "USER_RULE: TEST ROUTING"
    pass in quick on openvpn inet all flags S/SA keep state label "USER_RULE: TEMP"
    pass in quick on openvpn inet from any to (self) flags S/SA keep state label "USER_RULE: TEMP"
    pass in quick on lagg0_vlan2000 reply-to (lagg0_vlan2000 192.168.0.5) inet proto tcp from any to (self) port = https flags S/SA keep state label "USER_RULE: Easy Rule: Passed from Firewall Log View"
    pass in quick on lagg0_vlan2000 reply-to (lagg0_vlan2000 192.168.0.5) inet proto tcp from <itop_public_ip> to (self) port = rsh-spx flags S/SA keep state label "USER_RULE"
    pass in quick on lagg0_vlan2000 reply-to (lagg0_vlan2000 192.168.0.5) inet proto tcp from <itop_public_ip> to (self) port = https flags S/SA keep state label "USER_RULE"
    pass in quick on lagg0_vlan2000 reply-to (lagg0_vlan2000 192.168.0.5) inet proto tcp from any to (self) port = rsh-spx flags S/SA keep state label "USER_RULE: Easy Rule: Passed from Firewall Log View"
    pass in quick on lagg0_vlan1007 inet proto carp from any to (self) keep state label "USER_RULE: CARP ALLOWED"
    pass in quick on lagg0_vlan1007 inet proto pfsync from any to (self) keep state label "USER_RULE: PFSYNC ALLOWED"
    pass in quick on lagg0_vlan2010 inet proto tcp from any to (self) port = http flags S/SA keep state label "USER_RULE: WEB INTERFACE"
    pass in quick on lagg0_vlan2010 inet proto tcp from any to (self) port = https flags S/SA keep state label "USER_RULE: WEB INTERFACE"
    pass in quick on lagg0_vlan2010 inet proto tcp from any to (self) port = rsh-spx flags S/SA keep state label "USER_RULE: SSH"
    pass in quick on lagg0_vlan2010 inet proto icmp from any to (self) keep state label "USER_RULE"
    pass in quick on lagg0_vlan2010 inet from any to <hq_lans> flags S/SA keep state label "USER_RULE: oldlan2hqlans"
    block drop in quick on lagg0_vlan2010 inet from any to <lans_rfc1918> label "USER_RULE: LAST RULE-1"
    pass in quick on lagg0_vlan1008 inet from <backup_servers> to <negate_networks> flags S/SA keep state label "NEGATE_ROUTE: Negate policy routing for destination"
    pass in quick on lagg0_vlan1008 route-to (lagg0_vlan2000 192.168.0.1) inet from <backup_servers> to any flags S/SA keep state label "USER_RULE: TEST ROUTING"</backup_servers></negate_networks></backup_servers></lans_rfc1918></hq_lans></itop_public_ip></itop_public_ip></backup_servers>
    

    I also tried to disable the auto generated reply-to on the floating rule but still not working

    [root@backup ~]# traceroute 8.8.8.8
    traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
     1  gateway (10.10.3.1)  1.169 ms  1.115 ms  1.074 ms
     2  192.168.0.5 (192.168.0.5)  0.459 ms  0.432 ms  0.399 ms^C
    

    Thank you in advance.





  • Rebel Alliance Developer Netgate

    Hmm, that same style rule works when I use it. Try checking "Disable reply-to" on that rule as well.



  • @jimp:

    Hmm, that same style rule works when I use it. Try checking "Disable reply-to" on that rule as well.

    As said in the latest message, i already tried this disabling the automatic reply-to. Still not working.

    thank you.


  • Rebel Alliance Developer Netgate

    OK, so then something your boxes are sending is not matching that rule, but falling through to the other rule. So compare the two:

    default rule:

    pass out  route-to ( lagg0_vlan2000 192.168.0.5 ) from 192.168.0.10 to !192.168.0.0/24 tracker 1000008011 keep state allow-opts label "let out anything from firewall host itself"
    

    Your rules (assuming you did disable reply-to):

    pass in quick on lagg0_vlan1008 inet from <backup_servers>to <negate_networks>flags S/SA keep state label "NEGATE_ROUTE: Negate policy routing for destination"
    pass out quick on lagg0_vlan2000 inet from <backup_servers>to any flags S/SA keep state label "USER_RULE: TEST ROUTING"</backup_servers></negate_networks></backup_servers> 
    

    Three things jump out:

    #1: The first rule allows packets with ip options, so check that in your floating rule
    #2: The first rule does not filter TCP flags, clone that rule and make one that is TCP only but allows any flags
    #3: Try disabling the policy route negation rules under System > Advanced, Firewall & NAT tab



  • @jimp:

    OK, so then something your boxes are sending is not matching that rule, but falling through to the other rule. So compare the two:

    default rule:

    pass out  route-to ( lagg0_vlan2000 192.168.0.5 ) from 192.168.0.10 to !192.168.0.0/24 tracker 1000008011 keep state allow-opts label "let out anything from firewall host itself"
    

    Your rules (assuming you did disable reply-to):

    pass in quick on lagg0_vlan1008 inet from <backup_servers>to <negate_networks>flags S/SA keep state label "NEGATE_ROUTE: Negate policy routing for destination"
    pass out quick on lagg0_vlan2000 inet from <backup_servers>to any flags S/SA keep state label "USER_RULE: TEST ROUTING"</backup_servers></negate_networks></backup_servers> 
    

    Three things jump out:

    #1: The first rule allows packets with ip options, so check that in your floating rule
    #2: The first rule does not filter TCP flags, clone that rule and make one that is TCP only but allows any flags
    #3: Try disabling the policy route negation rules under System > Advanced, Firewall & NAT tab

    Thank for the response.

    I tried those three things, no one of them works

    with or without ip options allowed

    Screenshot in attachments

    pass out quick on lagg0_vlan2000 inet from <backup_servers>to any flags S/SA keep state allow-opts label "USER_RULE: TEST ROUTING"  # Floating
    
    pass in quick on lagg0_vlan1008 route-to (lagg0_vlan2000 192.168.0.1) inet from <backup_servers>to any flags S/SA keep state allow-opts label "USER_RULE: TEST ROUTING"
    pass in quick on lagg0_vlan1008 route-to (lagg0_vlan2000 192.168.0.1) inet proto tcp from <backup_servers>to any flags any keep state allow-opts label "USER_RULE: TEST ROUTING"</backup_servers></backup_servers></backup_servers> 
    

    Here's the full dump of pfctl http://pastebin.com/jw4mXLTf

    TCP Traceroute

    [root@backup ~]# traceroute -T 8.8.8.8
    traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
     1  gateway (10.10.3.1)  1.193 ms  1.143 ms  1.108 ms
     2  192.168.0.5 (192.168.0.5)  0.451 ms  0.433 ms  0.411 ms^C
    

    ICMP Traceroute

    [root@backup ~]# traceroute 8.8.8.8
    traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
     1  gateway (10.10.3.1)  1.144 ms  1.094 ms  1.034 ms
     2  192.168.0.5 (192.168.0.5)  0.456 ms  0.407 ms  0.386 ms
    

    thank you in advance









  • Rebel Alliance Developer Netgate

    clone the floating rule, not the LAN rule (though it won't hurt).

    And in this test you are sure that you are sourcing the traffic from a host in that alias?



  • @jimp:

    clone the floating rule, not the LAN rule (though it won't hurt).

    And in this test you are sure that you are sourcing the traffic from a host in that alias?

    I tried duplicating the floating rule and set only tcp + any flags, same problem

    And yes, the alias is defined




  • Hi Jim,

    Regarding all the tests my colleague has made and his results, do you think it could be a bug ?

    Thank you :-)


Log in to reply