Asymmetric routing with pfsense and L3 switch
-
@kevindd992002 question - why are you even segmenting the traffic then? If what you want is high speed between clients.. And you don't care about "firewall" between them.. Why not just put them in the same network?
What is the segmentation getting you, other than since you want pfsense to be dhcp server for what amounts to dns registration. Which pfsense is not the best choice of doing that because every time dhcp does something unbound restarts.
So your going to create a asymmetrical mess..
-
@johnpoz said in Asymmetric routing with pfsense and L3 switch:
@kevindd992002 question - why are you even segmenting the traffic then? If what you want is high speed between clients.. And you don't care about "firewall" between them.. Why not just put them in the same network?
What is the segmentation getting you, other than since you want pfsense to be dhcp server for what amounts to dns registration. Which pfsense is not the best choice of doing that because every time dhcp does something unbound restarts.
So your going to create a asymmetrical mess..
I can get ACL rules to control traffic in between VLANs. Separate broadcast domains too. I just have Main, IoT, Guest, CCTV, and Management VLANs.
I know what you mean about the pfsense dns registration restarting. I can just do it for static dhcp leases though, so unbound does not restart.
So that asymmetrical mess is not recommended even though it will technically work?
-
@kevindd992002
If you use a trunk port to connect an L3 switch to pfsense by defining the VLANs on pfsense then the L3 switch is not doing the routing, Pfsense is doing the routing. You have defeated your L3 switch routing and are using it as a L2 switch.To use your L3 switch to route you should not define any VLANs on pfsense and route disconnected networks to your L3 switch and the L3 switch will figure out how to route them.
You cannot use DHCP on pfsense. On my Cisco small business L3 switches I use DHCP in the switch. If that is not an option, then use something like Microsoft's DHCP server.
Your problem has nothing to do with asymmetric routing. It is your network structure. I do asymmetric routing now. In a prior setup I used a transit network. It works both ways.
To setup an L3 switch for routing you create a gateway pointing to the L3 switch on pfsense. On the L3 switch you set the default route on the L3 switch to point to the outbound gateway on pfsense. All the clients on the L3 switch will use the gateway for the VLAN or VLANs on the L3 switch and the L3 switch will route the non-local traffic using the default gateway on the L3 switch to Pfsense and out.
-
@kevindd992002 It won’t work because pfSense will see the packets coming in on one VLAN (Transit) and replies going out on another (IoT and so on). That means firewall states will prevent additional traffic from flowing (states require traffic in both directions).
So apart from being REALLY bad network design it will also not work.
It’s a pain that the DHCP server in pfSense does not support”non local” scopes - or running DHCP relay on some interfaces and DHCP server on others.Does the new KEO DHCP server arriving with 23.09 change any of this?
-
@coxhaus said in Asymmetric routing with pfsense and L3 switch:
@kevindd992002
If you use a trunk port to connect an L3 switch to pfsense by defining the VLANs on pfsense then the L3 switch is not doing the routing, Pfsense is doing the routing. You have defeated your L3 switch routing and are using it as a L2 switch.To use your L3 switch to route you should not define any VLANs on pfsense and route disconnected networks to your L3 switch and the L3 switch will figure out how to route them.
You cannot use DHCP on pfsense. On my Cisco small business L3 switches I use DHCP in the switch. If that is not an option, then use something like Microsoft's DHCP server.
Your problem has nothing to do with asymmetric routing. It is your network structure. I do asymmetric routing now. In a prior setup I used a transit network. It works both ways.
Yes it actually will, as long as you point the default gateway (through pfsense's DHCP server) of the clients to their respective VLAN VE (or SVI in Cisco) IP address. The trunk port is merely for DHCP and DNS traffic, nothing more.
I can only use either DHCP on pfsense or DHCP on the switch. For this specific network, I cannot have a separate DHCP running on another hardware.
@coxhaus said in Asymmetric routing with pfsense and L3 switch:
To setup an L3 switch for routing you create a gateway pointing to the L3 switch on pfsense. On the L3 switch you set the default route on the L3 switch to point to the outbound gateway on pfsense. All the clients on the L3 switch will use the gateway for the VLAN or VLANs on the L3 switch and the L3 switch will route the non-local traffic using the default gateway on the L3 switch to Pfsense and out.
This is exactly what I meant. But with the proposed setup in this post, you will have an interface for the VLANs in pfsense too, again for DHCP and DNS. You solve the DHCP issue but the return traffic does not go back to the L3 switch routing process (internet -> pfsense -> client), leading to some kind of asymmetric routing. And this is what I'm concerned about. But since the forward and return packets go through the same firewall, will that even be a problem? The client won't know anything about this too.
-
@keyser said in Asymmetric routing with pfsense and L3 switch:
@kevindd992002 It won’t work because pfSense will see the packets coming in on one VLAN (Transit) and replies going out on another (IoT and so on). That means firewall states will prevent additional traffic from flowing (states require traffic in both directions).
So apart from being REALLY bad network design it will also not work.
It’s a pain that the DHCP server in pfSense does not support”non local” scopes - or running DHCP relay on some interfaces and DHCP server on others.Does the new KEO DHCP server arriving with 23.09 change any of this?
Oh ok. I didn't know pfsense monitors in terms of VLAN too. In that case, do you have any clue how this guy here made it work for him? I was trying to ask him about it but he isn't replying. But yes, I agree it's really a bad network design, sort of just really a workaround. I guess my best bet for this network is just to use the switch's DHCP server and if I ever need hostname resolving, I just manually add to the entries in pfsense's unbound. It's a flat home network with nothing much going on.
In my other (main) house, that's where I will configure a separate DHCP server through docker so that isn't too much of a big deal.
-
@kevindd992002 said in Asymmetric routing with pfsense and L3 switch:
I just manually add to the entries in pfsense's unbound
That is what I would do.. How many devices do you have? Create a reservation for box X in your switch dhcpd, or some other dhcp, so it always gets 192.168.1.42 or whatever, then create a host override in unbound.. There you go, no asymmetrical just route traffic at your L3 and use a transit to pfsense, to get to the internet.
Or as mentioned just use dhcpd on your network that you can have the switch point to via helper/relay. I do believe isc dhcpd can do that, just not really exposed in the pfsense gui, etc. etc. But if you ran standalone dhcpd. I am not sure what features the dhcpd is going to bring, or what might be exposed to edit in the gui, etc.
-
@johnpoz said in Asymmetric routing with pfsense and L3 switch:
@kevindd992002 said in Asymmetric routing with pfsense and L3 switch:
I just manually add to the entries in pfsense's unbound
That is what I would do.. How many devices do you have? Create a reservation for box X in your switch dhcpd, or some other dhcp, so it always gets 192.168.1.42 or whatever, then create a host override in unbound.. There you go, no asymmetrical just route traffic at your L3 and use a transit to pfsense, to get to the internet.
Or as mentioned just use dhcpd on your network that you can have the switch point to via helper/relay. I do believe isc dhcpd can do that, just not really exposed in the pfsense gui, etc. etc. But if you ran standalone dhcpd. I am not sure what features the dhcpd is going to bring, or what might be exposed to edit in the gui, etc.
Yeah. I'm aware of these proper solutions even before posting here. I was just very curious if this dirty workaround would've worked because that guy in the post "seemed" to have it working for him.
-
@kevindd992002 Well, It can be made to work if the L3 switch supports ICMP redirect and you create a static route to the internet on each VLAN in the switch using the pfSense address of that VLAN. Then the switch will issue a redirect to clients rather than route the packet through the “Transit” VLAN (Which will cause assymetrical routing in pfSense that gets blocked).
So technically it is posible even though my former post suggested otherwise. But it also removes the Transit VLAN.
It’s still not good network design, but it may not really have a heavy impact in this case because the assymmetric routing does not include pfSense (It sees in and outgoing packets for same session arrive and leave on the same interface). -
@kevindd992002 said in Asymmetric routing with pfsense and L3 switch:
that guy in the post "seemed" to have it working for him.
There is a lot of bad design solutions.. Just because some idiot on the internet does it the wrong way, doesn't mean you should follow them off the cliff ;)
Here is what I am going to tell you with like 30+ years in the biz, if your solution is to setup some asymmetric solution - its BAD!! and you shouldn't do it.. Not saying have not done it myself - but its to work around an issue in need of fixing and needing to be able to get to something, etc.. And it something that is done in a pinch for temporary fix until a proper design can be put in place.
Another example running multiple layer 3 on the same L2.. Its bad to do that, can it be done sure, is it a fix or solution to something and its the easy thing to do - again sure.. Have done it myself again, but it is not the perm solution I would ever put in place, etc.
Setting up a known asymmetrical configuration because you want to run dhcp on pfsense, and it has to be connected to the L2 for that to work - would be a bad idea.. Is it something you might do while you bring up the correct solution - ok sure.. But I sure wouldn't plan on being your perm setup..
-
@keyser said in Asymmetric routing with pfsense and L3 switch:
@kevindd992002 Well, It can be made to work if the L3 switch supports ICMP redirect and you create a static route to the internet on each VLAN in the switch using the pfSense address of that VLAN. Then the switch will issue a redirect to clients rather than route the packet through the “Transit” VLAN (Which will cause assymetrical routing in pfSense that gets blocked).
So technically it is posible even though my former post suggested otherwise. But it also removes the Transit VLAN.
It’s still not good network design, but it may not really have a heavy impact in this case because the assymmetric routing does not include pfSense (It sees in and outgoing packets for same session arrive and leave on the same interface).I'm pretty sure I'm going back to my original plan of just doing dhcp on the switch. I realized that the complications of making this work outweight the little gain I get for this flat network (it's for my sister's new house and the people there aren't tech savvy). All there is are 14 IP cameras, 8 AP's, and a couple of wall eth ports per room. That's it, lol.
Out of curiosity though, how would you create a static route in each of the VLAN in the switch and make the next hop the pfsense interface IP on that VLAN if they're on the same subnet (no routing involved)? You lost me here.
-
@johnpoz said in Asymmetric routing with pfsense and L3 switch:
@kevindd992002 said in Asymmetric routing with pfsense and L3 switch:
that guy in the post "seemed" to have it working for him.
There is a lot of bad design solutions.. Just because some idiot on the internet does it the wrong way, doesn't mean you should follow them off the cliff ;)
Here is what I am going to tell you with like 30+ years in the biz, if your solution is to setup some asymmetric solution - its BAD!! and you shouldn't do it.. Not saying have not done it myself - but its to work around an issue in need of fixing and needing to be able to get to something, etc.. And it something that is done in a pinch for temporary fix until a proper design can be put in place.
Another example running multiple layer 3 on the same L2.. Its bad to do that, can it be done sure, is it a fix or solution to something and its the easy thing to do - again sure.. Have done it myself again, but it is not the perm solution I would ever put in place, etc.
Setting up a known asymmetrical configuration because you want to run dhcp on pfsense, and it has to be connected to the L2 for that to work - would be a bad idea.. Is it something you might do while you bring up the correct solution - ok sure.. But I sure wouldn't plan on being your perm setup..
Fair point, lol.
What do you mean running multiple layer 3 on the same layer 2? Are you referring to exactly what I was trying to do?
-
@kevindd992002 It is basically just creating equal cost routes to 0.0.0.0/0 each using the pfSense IP in each VLAN. If the switch support proper route cost evaluation it should send a ICMP redirect on each VLAN to the client instead of actually routing the packet itself (because of cost). This will cause the client to create a temporary entry in its own routing table to send packets for that specific destination IP directly to pfSense instead of the Switch VLAN interface. It will incur a very slight latency penalty for every new session towards the internet (a couple of ms).
It the switch is more advanced it might support VRFs (independant virtual routing domains), and then you can create a specific routing domain for each VLAN.
But still - a terrible solution for such a small network and still very bad network design. If its just for DHCP I would ANY day of the week have the switch do it instead and create the needed static DNS entries myself.
-
@kevindd992002 But then again - I would never have the switch do routing in such a setup. I would any day of the week have pfSense be the gateway for each VLAN and have that route the networks. It gives you statefull firewalling between VLANs (MUCH better than stateless ACLs in switches) and thus much more security. But it would downgrade your use of the switch to have it only be a VLAN capable managed switch (even though it is a full L3)
-
@keyser said in Asymmetric routing with pfsense and L3 switch:
@kevindd992002 But then again - I would never have the switch do routing in such a setup. I would any day of the week have pfSense be the gateway for each VLAN and have that route the networks. It gives you statefull firewalling between VLANs (MUCH better than stateless ACLs in switches) and thus much more security. But it would downgrade your use of the switch to have it only be a VLAN capable managed switch (even though it is a full L3)
Yeah, with such a small setup, I don't think there's even a performance difference between routing in pfsense vs. routing in the switch. But since I have the Brocade ICX6450 for this network, I'd rather use its potential. It's also my first time (since my CCNA days) to handle a mamaged switch 100% through CLI so why not. There's always fun in experimenting with new things
-
@kevindd992002 no but its another common bad setup see around here.. Users think they can create 2 networks say 192.168.0/24 and 192.168.1/24 on 2 interfaces and then just plug them into the same dumb switch..
and they wonder why dhcp isn't working how they want.
Or they will create 192.168.0/24 on the interface, and then wonder why they can't just run 192.168.1.1/24 as a vip and setup devices with static..
Technically you can get that sort of setup to work - but its not a good idea.. You might need to do such a thing while you transition from one IP range to another, etc. Or maybe someone put some device on the network with the wrong IP and you need to be able to get to it so you temp setup that wrong network to run on your current network, etc.
Another bad choice see all the time around here is setting up a bridge.. Can it work - sure ok.. But why, if what you want is a switch port, the correct solution is to actually use a switch ;)
Are there scenarios when a bridge is the answer - yeah sure.. But because there is some port on your device you feel is not being used, so you will think you turn it into a switch port by creating a bridge is not one of them ;)
There is no scenario I can think (at least off the top of my head) of where I would setup on purpose asymmetrical, and think well thats done ;) Let me go blog/post about how cool of setup I have - hehehe Its amazing he didn't put that nonsense on on some youtube video hahah, so other users can setup their network half assed..
I didn't read the whole thread - it was what like 400 some pages? WTF??? Was he going over how you could get something to work while they correct it? Or was he promoting that as a proper setup??
-
@johnpoz said in Asymmetric routing with pfsense and L3 switch:
@kevindd992002 no but its another common bad setup see around here.. Users think they can create 2 networks say 192.168.0/24 and 192.168.1/24 on 2 interfaces and then just plug them into the same dumb switch..
and they wonder why dhcp isn't working how they want.
Or they will create 192.168.0/24 on the interface, and then wonder why they can't just run 192.168.1.1/24 as a vip and setup devices with static..
Technically you can get that sort of setup to work - but its not a good idea.. You might need to do such a thing while you transition from one IP range to another, etc. Or maybe someone put some device on the network with the wrong IP and you need to be able to get to it so you temp setup that wrong network to run on your current network, etc.
Another bad choice see all the time around here is setting up a bridge.. Can it work - sure ok.. But why, if what you want is a switch port, the correct solution is to actually use a switch ;)
Are there scenarios when a bridge is the answer - yeah sure.. But because there is some port on your device you feel is not being used, so you will think you turn it into a switch port by creating a bridge is not one of them ;)
There is no scenario I can think (at least off the top of my head) of where I would setup on purpose asymmetrical, and think well thats done ;) Let me go blog/post about how cool of setup I have - hehehe Its amazing he didn't put that nonsense on on some youtube video hahah, so other users can setup their network half assed..
I didn't read the whole thread - it was what like 400 some pages? WTF??? Was he going over how you could get something to work while they correct it? Or was he promoting that as a proper setup??
Lol, you would think they have some basic idea of how subnet works since they are using pfsense. Two networks in the same switch/VLAN is messed up.
For bridging, I completely agree on that. That's why for AP's that come usually with two ports, I tend to ask if that extra port is bridged with the main uplink port or what. It's almost always a passthrough which for me in the same as bridged. There's just some that make both ports connected to an internal switch.
That thread is one of the longest threads in that forum. It's a support forum for Brocade switches as they're one of the cheapest used enterprise switches you can get in ebay. The OP is pretty good and have been helping a lot of people with his knowledge about these switches. Not sure how that guy came about his post about his network setup but I just happened to also stumble upon his post because it was linked in another thread when I was trying to look for a workaround to the dhcp dns registration issue.
-
@kevindd992002 said in Asymmetric routing with pfsense and L3 switch:
some basic idea of how subnet works since they are using pfsense
Hahaha - dude I ask myself that about many of the posts here, hahah.. Hey I get it, everyone starts somewhere, but I do wonder..
So was the post a hey, I wouldn't suggest you do this sort of post.. But this is "how" you could get it work..
-
@kevindd992002
If you have the VLANs defined on Pfsense then L2 traffic is going to flow across the trunk to Pfsense to route. The L3 switch is not going to route. -
@johnpoz said in Asymmetric routing with pfsense and L3 switch:
@kevindd992002 said in Asymmetric routing with pfsense and L3 switch:
some basic idea of how subnet works since they are using pfsense
Hahaha - dude I ask myself that about many of the posts here, hahah.. Hey I get it, everyone starts somewhere, but I do wonder..
So was the post a hey, I wouldn't suggest you do this sort of post.. But this is "how" you could get it work..
Ahaha, I know what yoy're saying.
Yeah. From the look of it, he was mostly just sharing what he had done and it's not like he was trying to make people do the same.
