Netgate 8200 and Unifi L3 Switch Configuration
-
Hi,
We are trying to replace an EdgeRouter ER-4 with a Netgate 8200 and are having some trouble with the configuration. In the current setup we use the EdgeRouter as a "router on a stick" for a number of VLANs. It handles DHCP, DNS, and routing to the internet and between VLANs. The wrinkle is that there are a couple more VLANs that are routed by a 10GbE Unifi L3 switch for performance reasons (the hosts all have 10GbE NICs).
With the EdgeRouter everything works fine. It has an interface created for each VLAN, for example: 192.168.10.1 for a regular VLAN or 192.168.30.2 (because the switch is 192.168.30.1) for a VLAN that is routed by the L3 switch. Because the EdgeRouter has an interface in every VLAN it is able to provide DHCP (dnsmasq) to all of them by creating a DHCP server for each VLAN. Basically, we followed the Unifi docs on how to set this up.
When trying to substitute in the Netgate 8200 we aren't able to create the static routes needed to handle the Unifi switches magic VLAN 4040. In EdgeOS it was as simple as creating a static route for each L3 routed VLAN with the next hop set to 10.255.253.2 (as per the Unifi docs). Seems like in pfSense we can't have a static IP assigned a virtual interface and use it as the destination network in a static route? This results in a error when trying to save the static route. I think we might just be misunderstanding how pfSense works though.
Any insight on how to configure pfSense would be appreciate.
Thanks.
** edit, I should add that we can't use DHCP on the L3 switch because we need the router to handle it so it can inject the dhcp configured hosts into the DNS relay **
-
@angusmh said in Netgate 8200 and Unifi L3 Switch Configuration:
each L3 routed VLAN with the next hop set to 10.255.253.2 (as per the Unifi docs)
Huh? So your trying to use that network as your transit network? What actual L2 are you running it over, that you would setup a vip on pfsense?
When setting up a downstream router, here is a good drawing
As to your dhcp wanting to be on pfsense - yeah that is going to be a problem. Pfsense can not provide dhcp to networks it doesn't have an interface in. If you can not run it on your switch because of limitation in the dhcpd on that being able to do what you want for options or whatever.. Prob best to fire up something else on your network to be your dhcpd, that can provide scope for not attached networks via a helper/relay.
-
Mmm, hard to know for sure what you're doing here since 10.255.253.2 doesn't appear on the diagram. Or any other IP in that subnet. Or VLAN 4040.
But, yeah, dhcp will be a problem if pfSense doesn't have an IP in the 10G subnets. The actual routing is no problem though as long as you remove whatever obfuscation Unifi is doing to make it 'easy'.
-
Thanks you both for the response and suggestions. I suppose running dnsmasq separately instead of using the pfSense for DHCP/DNS isn't a huge deal.
What I am now trying to figure out is why the EdgeRouter is able to provide DHCP to all the VLANs, including the L3 Routed ones, because it is also using dnsmasq (same as the pfSense can). I made a slightly better diagram with some configuration listed.
I think the reason it works now is because the EdgeRouter has the L3 switch VLANs defined AND a static route (the thing pfSense won't let me do). I can see that the routing is "asymmetric" when using
traceroute:Outside the L3 VLAN to inside the L3 VLAN
$ traceroute 192.168.40.100 traceroute to 192.168.40.100 (192.168.40.100), 30 hops max, 60 byte packets 1 192.168.10.1 (192.168.10.1) 0.623 ms 0.580 ms 0.556 ms 2 192.168.40.100 (192.168.40.100) 0.534 ms 0.509 ms 0.487 msInside the L3 VLAN to outside the L3 VLAN
$ traceroute 192.168.10.100 traceroute to 192.168.10.100 (192.168.10.100), 30 hops max, 60 byte packets 1 192.168.40.100 (192.168.40.100) 0.548 ms 0.926 ms 1.026 ms 2 10.255.253.1 (10.255.253.1) 0.162 ms 0.114 ms 0.114 ms 3 192.168.10.100 (192.168.10.100) 0.262 ms 0.281 ms 0.258 msI am still not exactly sure how this lets DHCP broadcast requests out of the L3 VLAN though, I assume they should be blocked. If I delete virtual interface VLAN 40 on the EdgeRouter then the traffic looks "symmetric", notice the additional hop, but DHCP stops working:
Outside the L3 VLAN to inside the L3 VLAN
$ traceroute 192.168.40.100 traceroute to 192.168.40.100 (192.168.40.100), 30 hops max, 60 byte packets 1 192.168.10.1 (192.168.10.1) 0.623 ms 0.580 ms 0.556 ms 2 10.255.253.2 (10.255.253.2) 0.162 ms 0.114 ms 0.114 ms 3 192.168.40.100 (192.168.40.100) 0.534 ms 0.509 ms 0.487 msInside the L3 VLAN to outside the L3 VLAN
$ traceroute 192.168.10.100 traceroute to 192.168.10.100 (192.168.10.100), 30 hops max, 60 byte packets 1 192.168.40.100 (192.168.40.100) 0.548 ms 0.926 ms 1.026 ms 2 10.255.253.1 (10.255.253.1) 0.162 ms 0.114 ms 0.114 ms 3 192.168.10.100 (192.168.10.100) 0.262 ms 0.281 ms 0.258 msI assume the second option is how it would look in pfSense and also why DHCP can't work. Do you know the downsides of doing it the way the EdgeRouter is doing it? Looks like a feature not a bug right now but I might be mistaken. Either way it seems like the path forward is to create static routes only on the pfSense, spin up a DHCP server, and use DHCP relaying on the pfSense and the L3 switch to point the VLANs to the new DHCP server.
Thanks again for your time.
-
@angusmh said in Netgate 8200 and Unifi L3 Switch Configuration:
DHCP broadcast requests out of the L3 VLAN though
This is done with a helper or relay - different makers call it different things - its the same thing.
Seems the unifi stuff doesn't do it other than on their gateway devices.. They also require that stupid 4040 vlan for the transit?
Note: It is not possible to modify the UniFi switch network range (10.255.253.0/24) or use a different VLAN than 4040. The third-party router needs to match the UniFi configuration.
Yet another thing not to like about rest of their stuff I guess.. Your going to ask them how to do dhcp through their L3 switches.. My guess is you don't.. Which means you have to compromise your whole network by putting something with a leg in your all your vlans just do to dhcp relay to whatever you run for your dhcpd.
The pfsense limitation has been gone over - I don't recall the specific reason, but I can tell you it doesn't do it. Pfsense needs to have an interface in each network to provide dhcp. It can act as a relay if you don't run dhcp on it. But you can not forward dhcp to it with the source network, ie with a helper or relay and have it hand out dhcp.
-
Yeah in pfSense you cannot have a static route to a subnet that already exists as a locally connected subnet. You could have a smaller subnet locally, so a /32 for example. But that wouldn't help because DHCP would not run on that.
What I expect it happening here is that the L3 switch is running dhcp relay and the Edgerouter is able to reply to queries from that. dhcpd in pfSense can do that but there's no front end to configure it. It is possible to force that with a manual config file but that's not recommended.https://github.com/pfsense/pfsense/pull/1406
Steve
-
Thanks for the help, I am going to put the resolution here in case anyone else comes across this in the future.
I had to double check but there are definitely no DHCP relays active on the L3 switch currently, Unifi does have an option for that though. It seems that the EdgeRouter has no problem adding itself to a VLAN subnet AND having a static route set, this lets it run a DHCP/DNS server at 192.168.40.2 and the L3 switch be the default gateway at 192.168.40.1. It totally works but does means there are two routers on a single VLAN and leads to asymmetric routing as show above.
With Netgate it is not possible to have the above config. We have to run our own instance of dnsmasq on one of the VLANs and use the DHCP relay functionality of the Unifi L3 switch and pfSense to make sure DHCP gets relayed from all the VLANs regardless of the router servicing them. Dnsmasq can handle this just by changing the DHCP scopes, it doesn't need an interface on each VLAN:
# By default DNS will only respond to requests on the local subnet, specifying the address fixes this listen-address=127.0.0.1,192.168.10.2 # VLAN10 dhcp-range=set:VLAN10,192.168.10.10,192.168.10.220 dhcp-option=tag:VLAN10,option:router,192.168.10.1 # VLAN20 dhcp-range=set:VLAN20,192.168.20.10,192.168.20.220 dhcp-option=tag:VLAN20,option:router,192.168.20.1 # VLAN30 dhcp-range=set:VLAN30,192.168.30.10,192.168.30.220 dhcp-option=tag:VLAN30,option:router,192.168.30.1 # VLAN40 dhcp-range=set:VLAN40,192.168.40.10,192.168.40.220 dhcp-option=tag:VLAN40,option:router,192.168.40.1In the above config the DHCP/DNS server is 192.168.10.2 and it will hand out leases for all four VLANs. The routers must support relaying DHCP to the server for their respective VLANs.
Thanks again.
-
@angusmh
I find it hard to use the 'L3' term with UniFi switches given how threadbare it is. They can be the DHCP server just for those problematic legs though; would that help your aim?(Not that I think 2 DHCP servers is ideal but UniFi...)
️ -
@robbiett Thanks for the suggestion. We did test the Unifi L3 DHCP server and it works, the issue for us is that we use the DHCP/DNS integration of dnsmasq, i.e., all the hosts that use DHCP need to be added to our local DNS. Because of this having a "unified" DHCP/DNS server is required, we can't have some on the router and some on the L3 switch.
I have to admit I do find myself saying "but Unifi..." too.
-
@angusmh
I don't want to push this too much as it is a kludge on top of a kludge but depending on how dynamic things are you could write to the hosts files so that Dnsmasq knows what is going on. I've had to do something like that before, plus leaving myself # instructions in the file so I know what I did:#Entries for cat /etc/hosts #Using vi /etc/hosts to edit #To include the dhcp config stored in a different file: #source /path/to/file #source /etc/dnsmasq.d/dnsmasq-dhcp-config.conf #cat /etc/dnsmasq.d/dnsmasq-dhcp-config.conf 127.0.0.1 localhost.localdomain localhost 127.0.1.1 router-4 #vyatta entry 127.0.1.1 router-4.redacted.me #vyatta entry 2a02:390:8641:3::1 router-4.redacted.me router-4 10.0.1.1 router-4.redacted.me router-4 10.0.1.2 switch-24p.redacted.me switch-24p 10.0.1.3 switch-24x.redacted.me switch-24x 10.0.1.4 switch-5p.redacted.me switch-5pIt's just a thought but it can get you out of a jam sometimes.
️ -
@robbiett said in Netgate 8200 and Unifi L3 Switch Configuration:
've had to do something like that before, plus leaving myself # instructions in the file so I know what I did:
Thanks for the suggestion. We have a bunch of ephemeral linux containers spinning up and down so I would not want to have to manage updating the host file in our case I think. I could see this being an option if things weren't moving around so much though.
Thanks again.
