Policy-based routing: directly attached interface can never be overridden
-
Here's my setup:
- pfsense with VLAN10 (192.168.10.1/24) and VLAN300 (172.26.1.1/30).
- CBS350 L3 switch: VLAN10 (192.168.10.2/24) and VLAN300 (172.26.1.2/30).
CBS350 is the default gateway doing intervlan routing for VLAN10.
Default 0.0.0.0/0 route on cbs350 is 172.26.1.1.So, all internet traffic is routed from VLAN10 clients -> cbs350 -> pfsense VLAN300 -> WAN.
I keep VLAN10 interface set on pfsense so that it can provide DNS/DHCP/NTP services to VLAN10. I keep it only for lab and educational network testing.
On pfsense, I added a gateway named cbs350_gw with:
- interface = VLAN300
- gateway = 172.26.1.2
On pfsense, I added a NAT outbound in hybrid mode with a manual rule above automatic rules:
- interface = VLAN300
- source = VLAN10 subnets
- destination = any
- translation address = VLAN300 address.
On pfsense VLAN300 interface, I added a firewall rule with:
- source = VLAN10
- target = internet alias
- disable reply-to = checked
- gateway = cbs350_gw
With this, return internet traffic should be symmetric and returning through VLAN300.
However, ping 8.8.8.8 from VLAN10 hosts shows echo requests going out from VLAN300 (expected) but replies still flowing through VLAN10 interface on pfsenseNote: pfsense cannot add static routes for directly connected interfaces.
To summarize the current state:
Forward Path (VLAN10 Client -> Internet):- Client 192.168.10.x -> CBS350 192.168.10.2 (Default Gateway)
- CBS350 routes 0.0.0.0/0 to pfSense 172.26.1.1
- Traffic arrives on pfSense VLAN300 (from 172.26.1.2 / CBS350).
- Outbound NAT Rule on pfSense (Source 192.168.10.0/24 -> 172.26.1.1 on VLAN300) works, as evidenced by echo requests going out VLAN300.
- Traffic goes out pfSense WAN.
Return Path (Internet -> VLAN10 Client):
- Reply arrives on pfSense WAN.
- pfSense de-NATs, destination is 192.168.10.x.
- Problem: pfSense tries to send this reply out its VLAN10 interface (which has 192.168.10.1/24), rather than back through VLAN300 to the CBS350.
Setting a gateway override in VLAN300 should make policy-based routing work as advertised. Why pfsense seems to always priorize locally attached interface over the policy based routing ?
Obviously, the onyl proper fix would be to remove VLAN10 interface from pfsense. Still, technically, why this policy-based routing setup doesn't work ? Is it a bug or a kernel based implementation ? What's teh puropose of the gateway override in firewall rules in that case as they are supposed o make policy-based routing possible ?
Please just answer/explain the technical part I mention and no need to general statements like: just remove VLAN10 interface or use pfsense as gateway
-
@phil80 said in Policy-based routing: directly attached interface can never be overridden:
I keep VLAN10 interface set on pfsense so that it can provide DNS/DHCP/NTP services to VLAN10.
That is going to always cause issues.. A transit network should be used for routing to downstream networks. If you are using vlan10 as a transit - there shouldn't be any hosts on it or yeah your going to run into routing issues.
So you think you should be able to create a route statement that says go to some other IP to get to network A, when the device has an interface in network A? Yeah that is not a thing.
If you want to do such a thing you would have to nat at your cbs350 so the traffic looks like its coming from the cbs350 IP in your transit network vs the actual source IP of 192.168.10..
Or, just because you don't want to hear it doesn't mean its not the correct way to do it - there shouldn't be an interface on your router for vlan 10, if you want to use vlan 300 as your transit to get to the network 192.168.10..
If you want a vlan 10 on pfsense, then use that as the default gateway for devices on vlan 10, and set specific routes on this device to use your cbs350 as route to get to other networks being routed off your 350.
-
This post is deleted! -
@johnpoz
Yes, I fully got your answer. I can confirm that the way I posted is asymmetric internet return traffic with all security issuesWhat's the purpose of the gateway override and disable reply-to in advanced firewall rules in that case ? Why they don't kick for internet return traffic after d-nat on WAN in this case ? I even tried applying them using a floating rule for VLAN300 in out direction but VLAN10, locally attached, is always priorized for return traffic. Am I missing a network fundamental or the policy-based firewall rule only applies to interfaces not directly connected ?
-
@phil80 I think your misunderstanding a policy route.. And you can't tell pfsense to send to IP address on network B to get to network A, when pfsense has an IP in network A.
You can either nat at your 350 so pfsense will send the return traffic back to your 350 for it to route it to the 192.168.10 address. Or you can remove the asymmetrical setup completely.
-
@johnpoz
Sure, I was assuming that:
1- NAT made on outgoing traffic from transit VLAN300 (source VLAN10, translating to VLAN300 address) would kick in for outgoing internet traffic from VLAN10 -> CBS350 -> VLAN300 -> NAT rule on VLAN300 -> WAN
2- If I understand it well, D-Nat happens on return traffic and the source address translates back to VLAN10.
3- Where I thought it would be ok is to set the gateway on VLAN300 firewall rule so that outgoing replies are forwarded to the VLAN300 gateway and not VLAN10.Can you confirm that my last assumption (3) was wrong because the gateway applies only to outgoing traffic from the interface and not to the replies (managed by states and routing table). Routing table priorize local attached networks and those can never be overridden by a static route
So only fix, since cbs350 cannot do NAT, is to route on pfsense or DHCP outside it
Unless the feature to enable kea DHCP feature to serve pools outside of the directly attached interfaces is once implemented -
@phil80 the correct setup for routing traffic via a downstream router is via a transit network. This would eliminate any asymmetrical traffic flow.
The cbs350 can do dhcp - so run dhcp server on it for the networks behind it. Or spin up some other dhcp server that can do scopes for network not attached and then setup relay/ip helper on your cbs switch.
As to kea supporting that - I do believe that kea at some point will enable that. ISC dhcp could do that - but feature was never implemented into pfsense.. Same might be for kea, not sure. I don't recall seeing that as an option as of yet. But maybe at some point.
My take on dhcp in your router has always been its a simple solution for simple networks/needs - if you need more advanced abilities with your dhcp server it prob shouldn't be on your firewall/router anyway.
-
@johnpoz
Thank you
Can I keep the transit VLAN rules that filter and NAT redirect DHCP requests flowing through transit VLAN in pfsense ? This would ensure DOT is done from pfsense and restrict some VLANs to Opendns servers.Can I also keep the DNS resolver and use the transit address in pfsense to expose DNS and NTP services to the VLANs ?
I don't see a reason it won't if I set the source address of the rule to an alias with the VLANs subnet (192.168.10.1/24)
-
@phil80 sure you can clients behind your 350 can get to anywhere they want that are allowed by your rules on your transit network. So if you want to point them to some dns or dot server or ntp. And sure you could provide dns/ntp on your transit network interface on pfsense, or they could point to any other dns/ntp server on your network they can get to.
As to redirecting dhcp via relay/helper on pfsense - not sure about that for downstream networks. But that ip helper should really be on the 350 that is directly attached to the network behind it.
dhcp would never get to pfsense transit interface to be relayed unless you were doing that on your 350. So Not sure what the point of sending that to pfsense transit IP, just to be sent on again to your dhcp server hosting multiple scopes. You would just send them to the dhcp server IP doing that right on your 350.
-
@johnpoz
Thank you for the clarifications
I finally opted to route everything on pfsense and remove SVIs on switch. It's so much easier than to manage the ACLs on switchIf I end up needing more L3 switching throughoutput on some vlans, I can always try a hybrid setup with static routes on pfsense and running the dhcp server on the switch for those vlans
