Policy based routing, multi-WAN and gateway on same subnet

gslongo

No. Just exactly what I wrote.

:o

I'm sorry to ask again but your answers are very short and not clear to me :-[ I'm not inside your head ;D

Could you please just describe what should work ? Because it seems what we tested here is according what you suggested ?

Thanks!

jimp

Pass rule on LAN with gateway set.
Floating quick pass out on WAN without gateway set.

makz

@jimp:

Pass rule on LAN with gateway set.
Floating quick pass out on WAN without gateway set.

Hello, i tried that but still not working, maybe i miss something but i don't know what, screenshots in attachments

anchor "userrules/*" all
pass out quick on lagg0_vlan2000 reply-to (lagg0_vlan2000 192.168.0.5) inet from <backup_servers> to any flags S/SA keep state label "USER_RULE: TEST ROUTING"
pass in quick on openvpn inet all flags S/SA keep state label "USER_RULE: TEMP"
pass in quick on openvpn inet from any to (self) flags S/SA keep state label "USER_RULE: TEMP"
pass in quick on lagg0_vlan2000 reply-to (lagg0_vlan2000 192.168.0.5) inet proto tcp from any to (self) port = https flags S/SA keep state label "USER_RULE: Easy Rule: Passed from Firewall Log View"
pass in quick on lagg0_vlan2000 reply-to (lagg0_vlan2000 192.168.0.5) inet proto tcp from <itop_public_ip> to (self) port = rsh-spx flags S/SA keep state label "USER_RULE"
pass in quick on lagg0_vlan2000 reply-to (lagg0_vlan2000 192.168.0.5) inet proto tcp from <itop_public_ip> to (self) port = https flags S/SA keep state label "USER_RULE"
pass in quick on lagg0_vlan2000 reply-to (lagg0_vlan2000 192.168.0.5) inet proto tcp from any to (self) port = rsh-spx flags S/SA keep state label "USER_RULE: Easy Rule: Passed from Firewall Log View"
pass in quick on lagg0_vlan1007 inet proto carp from any to (self) keep state label "USER_RULE: CARP ALLOWED"
pass in quick on lagg0_vlan1007 inet proto pfsync from any to (self) keep state label "USER_RULE: PFSYNC ALLOWED"
pass in quick on lagg0_vlan2010 inet proto tcp from any to (self) port = http flags S/SA keep state label "USER_RULE: WEB INTERFACE"
pass in quick on lagg0_vlan2010 inet proto tcp from any to (self) port = https flags S/SA keep state label "USER_RULE: WEB INTERFACE"
pass in quick on lagg0_vlan2010 inet proto tcp from any to (self) port = rsh-spx flags S/SA keep state label "USER_RULE: SSH"
pass in quick on lagg0_vlan2010 inet proto icmp from any to (self) keep state label "USER_RULE"
pass in quick on lagg0_vlan2010 inet from any to <hq_lans> flags S/SA keep state label "USER_RULE: oldlan2hqlans"
block drop in quick on lagg0_vlan2010 inet from any to <lans_rfc1918> label "USER_RULE: LAST RULE-1"
pass in quick on lagg0_vlan1008 inet from <backup_servers> to <negate_networks> flags S/SA keep state label "NEGATE_ROUTE: Negate policy routing for destination"
pass in quick on lagg0_vlan1008 route-to (lagg0_vlan2000 192.168.0.1) inet from <backup_servers> to any flags S/SA keep state label "USER_RULE: TEST ROUTING"</backup_servers></negate_networks></backup_servers></lans_rfc1918></hq_lans></itop_public_ip></itop_public_ip></backup_servers>

I also tried to disable the auto generated reply-to on the floating rule but still not working

[root@backup ~]# traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  gateway (10.10.3.1)  1.169 ms  1.115 ms  1.074 ms
 2  192.168.0.5 (192.168.0.5)  0.459 ms  0.432 ms  0.399 ms^C

Thank you in advance.

lan_rule.png_thumb

floating_rule.png_thumb

jimp

Hmm, that same style rule works when I use it. Try checking "Disable reply-to" on that rule as well.

makz

@jimp:

Hmm, that same style rule works when I use it. Try checking "Disable reply-to" on that rule as well.

As said in the latest message, i already tried this disabling the automatic reply-to. Still not working.

thank you.

jimp

OK, so then something your boxes are sending is not matching that rule, but falling through to the other rule. So compare the two:

default rule:

pass out  route-to ( lagg0_vlan2000 192.168.0.5 ) from 192.168.0.10 to !192.168.0.0/24 tracker 1000008011 keep state allow-opts label "let out anything from firewall host itself"

Your rules (assuming you did disable reply-to):

pass in quick on lagg0_vlan1008 inet from <backup_servers>to <negate_networks>flags S/SA keep state label "NEGATE_ROUTE: Negate policy routing for destination"
pass out quick on lagg0_vlan2000 inet from <backup_servers>to any flags S/SA keep state label "USER_RULE: TEST ROUTING"</backup_servers></negate_networks></backup_servers>

Three things jump out:

#1: The first rule allows packets with ip options, so check that in your floating rule
#2: The first rule does not filter TCP flags, clone that rule and make one that is TCP only but allows any flags
#3: Try disabling the policy route negation rules under System > Advanced, Firewall & NAT tab

makz

@jimp:

OK, so then something your boxes are sending is not matching that rule, but falling through to the other rule. So compare the two:

default rule:
pass out  route-to ( lagg0_vlan2000 192.168.0.5 ) from 192.168.0.10 to !192.168.0.0/24 tracker 1000008011 keep state allow-opts label "let out anything from firewall host itself"
Your rules (assuming you did disable reply-to):
pass in quick on lagg0_vlan1008 inet from <backup_servers>to <negate_networks>flags S/SA keep state label "NEGATE_ROUTE: Negate policy routing for destination"
pass out quick on lagg0_vlan2000 inet from <backup_servers>to any flags S/SA keep state label "USER_RULE: TEST ROUTING"</backup_servers></negate_networks></backup_servers> 
Three things jump out:

#1: The first rule allows packets with ip options, so check that in your floating rule
#2: The first rule does not filter TCP flags, clone that rule and make one that is TCP only but allows any flags
#3: Try disabling the policy route negation rules under System > Advanced, Firewall & NAT tab

Thank for the response.

I tried those three things, no one of them works

with or without ip options allowed

Screenshot in attachments

pass out quick on lagg0_vlan2000 inet from <backup_servers>to any flags S/SA keep state allow-opts label "USER_RULE: TEST ROUTING"  # Floating

pass in quick on lagg0_vlan1008 route-to (lagg0_vlan2000 192.168.0.1) inet from <backup_servers>to any flags S/SA keep state allow-opts label "USER_RULE: TEST ROUTING"
pass in quick on lagg0_vlan1008 route-to (lagg0_vlan2000 192.168.0.1) inet proto tcp from <backup_servers>to any flags any keep state allow-opts label "USER_RULE: TEST ROUTING"</backup_servers></backup_servers></backup_servers>

Here's the full dump of pfctl http://pastebin.com/jw4mXLTf

TCP Traceroute

[root@backup ~]# traceroute -T 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  gateway (10.10.3.1)  1.193 ms  1.143 ms  1.108 ms
 2  192.168.0.5 (192.168.0.5)  0.451 ms  0.433 ms  0.411 ms^C

ICMP Traceroute

[root@backup ~]# traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  gateway (10.10.3.1)  1.144 ms  1.094 ms  1.034 ms
 2  192.168.0.5 (192.168.0.5)  0.456 ms  0.407 ms  0.386 ms

thank you in advance

advanced_firewall_nat.png_thumb

floating.png_thumb

lan_any.png_thumb

lan_tcp.png_thumb

jimp

clone the floating rule, not the LAN rule (though it won't hurt).

And in this test you are sure that you are sourcing the traffic from a host in that alias?

makz

@jimp:

clone the floating rule, not the LAN rule (though it won't hurt).

And in this test you are sure that you are sourcing the traffic from a host in that alias?

I tried duplicating the floating rule and set only tcp + any flags, same problem

And yes, the alias is defined

alias.png_thumb

gslongo

Hi Jim,

Regarding all the tests my colleague has made and his results, do you think it could be a bug ?

Thank you :-)