Netgate 8200 and Unifi L3 Switch Configuration
-
@angusmh said in Netgate 8200 and Unifi L3 Switch Configuration:
each L3 routed VLAN with the next hop set to 10.255.253.2 (as per the Unifi docs)
Huh? So your trying to use that network as your transit network? What actual L2 are you running it over, that you would setup a vip on pfsense?
When setting up a downstream router, here is a good drawing
As to your dhcp wanting to be on pfsense - yeah that is going to be a problem. Pfsense can not provide dhcp to networks it doesn't have an interface in. If you can not run it on your switch because of limitation in the dhcpd on that being able to do what you want for options or whatever.. Prob best to fire up something else on your network to be your dhcpd, that can provide scope for not attached networks via a helper/relay.
-
Mmm, hard to know for sure what you're doing here since 10.255.253.2 doesn't appear on the diagram. Or any other IP in that subnet. Or VLAN 4040.
But, yeah, dhcp will be a problem if pfSense doesn't have an IP in the 10G subnets. The actual routing is no problem though as long as you remove whatever obfuscation Unifi is doing to make it 'easy'.
-
Thanks you both for the response and suggestions. I suppose running dnsmasq separately instead of using the pfSense for DHCP/DNS isn't a huge deal.
What I am now trying to figure out is why the EdgeRouter is able to provide DHCP to all the VLANs, including the L3 Routed ones, because it is also using dnsmasq (same as the pfSense can). I made a slightly better diagram with some configuration listed.
I think the reason it works now is because the EdgeRouter has the L3 switch VLANs defined AND a static route (the thing pfSense won't let me do). I can see that the routing is "asymmetric" when using
traceroute:Outside the L3 VLAN to inside the L3 VLAN
$ traceroute 192.168.40.100 traceroute to 192.168.40.100 (192.168.40.100), 30 hops max, 60 byte packets 1 192.168.10.1 (192.168.10.1) 0.623 ms 0.580 ms 0.556 ms 2 192.168.40.100 (192.168.40.100) 0.534 ms 0.509 ms 0.487 msInside the L3 VLAN to outside the L3 VLAN
$ traceroute 192.168.10.100 traceroute to 192.168.10.100 (192.168.10.100), 30 hops max, 60 byte packets 1 192.168.40.100 (192.168.40.100) 0.548 ms 0.926 ms 1.026 ms 2 10.255.253.1 (10.255.253.1) 0.162 ms 0.114 ms 0.114 ms 3 192.168.10.100 (192.168.10.100) 0.262 ms 0.281 ms 0.258 msI am still not exactly sure how this lets DHCP broadcast requests out of the L3 VLAN though, I assume they should be blocked. If I delete virtual interface VLAN 40 on the EdgeRouter then the traffic looks "symmetric", notice the additional hop, but DHCP stops working:
Outside the L3 VLAN to inside the L3 VLAN
$ traceroute 192.168.40.100 traceroute to 192.168.40.100 (192.168.40.100), 30 hops max, 60 byte packets 1 192.168.10.1 (192.168.10.1) 0.623 ms 0.580 ms 0.556 ms 2 10.255.253.2 (10.255.253.2) 0.162 ms 0.114 ms 0.114 ms 3 192.168.40.100 (192.168.40.100) 0.534 ms 0.509 ms 0.487 msInside the L3 VLAN to outside the L3 VLAN
$ traceroute 192.168.10.100 traceroute to 192.168.10.100 (192.168.10.100), 30 hops max, 60 byte packets 1 192.168.40.100 (192.168.40.100) 0.548 ms 0.926 ms 1.026 ms 2 10.255.253.1 (10.255.253.1) 0.162 ms 0.114 ms 0.114 ms 3 192.168.10.100 (192.168.10.100) 0.262 ms 0.281 ms 0.258 msI assume the second option is how it would look in pfSense and also why DHCP can't work. Do you know the downsides of doing it the way the EdgeRouter is doing it? Looks like a feature not a bug right now but I might be mistaken. Either way it seems like the path forward is to create static routes only on the pfSense, spin up a DHCP server, and use DHCP relaying on the pfSense and the L3 switch to point the VLANs to the new DHCP server.
Thanks again for your time.
-
@angusmh said in Netgate 8200 and Unifi L3 Switch Configuration:
DHCP broadcast requests out of the L3 VLAN though
This is done with a helper or relay - different makers call it different things - its the same thing.
Seems the unifi stuff doesn't do it other than on their gateway devices.. They also require that stupid 4040 vlan for the transit?
Note: It is not possible to modify the UniFi switch network range (10.255.253.0/24) or use a different VLAN than 4040. The third-party router needs to match the UniFi configuration.
Yet another thing not to like about rest of their stuff I guess.. Your going to ask them how to do dhcp through their L3 switches.. My guess is you don't.. Which means you have to compromise your whole network by putting something with a leg in your all your vlans just do to dhcp relay to whatever you run for your dhcpd.
The pfsense limitation has been gone over - I don't recall the specific reason, but I can tell you it doesn't do it. Pfsense needs to have an interface in each network to provide dhcp. It can act as a relay if you don't run dhcp on it. But you can not forward dhcp to it with the source network, ie with a helper or relay and have it hand out dhcp.
-
Yeah in pfSense you cannot have a static route to a subnet that already exists as a locally connected subnet. You could have a smaller subnet locally, so a /32 for example. But that wouldn't help because DHCP would not run on that.
What I expect it happening here is that the L3 switch is running dhcp relay and the Edgerouter is able to reply to queries from that. dhcpd in pfSense can do that but there's no front end to configure it. It is possible to force that with a manual config file but that's not recommended.https://github.com/pfsense/pfsense/pull/1406
Steve
-
Thanks for the help, I am going to put the resolution here in case anyone else comes across this in the future.
I had to double check but there are definitely no DHCP relays active on the L3 switch currently, Unifi does have an option for that though. It seems that the EdgeRouter has no problem adding itself to a VLAN subnet AND having a static route set, this lets it run a DHCP/DNS server at 192.168.40.2 and the L3 switch be the default gateway at 192.168.40.1. It totally works but does means there are two routers on a single VLAN and leads to asymmetric routing as show above.
With Netgate it is not possible to have the above config. We have to run our own instance of dnsmasq on one of the VLANs and use the DHCP relay functionality of the Unifi L3 switch and pfSense to make sure DHCP gets relayed from all the VLANs regardless of the router servicing them. Dnsmasq can handle this just by changing the DHCP scopes, it doesn't need an interface on each VLAN:
# By default DNS will only respond to requests on the local subnet, specifying the address fixes this listen-address=127.0.0.1,192.168.10.2 # VLAN10 dhcp-range=set:VLAN10,192.168.10.10,192.168.10.220 dhcp-option=tag:VLAN10,option:router,192.168.10.1 # VLAN20 dhcp-range=set:VLAN20,192.168.20.10,192.168.20.220 dhcp-option=tag:VLAN20,option:router,192.168.20.1 # VLAN30 dhcp-range=set:VLAN30,192.168.30.10,192.168.30.220 dhcp-option=tag:VLAN30,option:router,192.168.30.1 # VLAN40 dhcp-range=set:VLAN40,192.168.40.10,192.168.40.220 dhcp-option=tag:VLAN40,option:router,192.168.40.1In the above config the DHCP/DNS server is 192.168.10.2 and it will hand out leases for all four VLANs. The routers must support relaying DHCP to the server for their respective VLANs.
Thanks again.
-
@angusmh
I find it hard to use the 'L3' term with UniFi switches given how threadbare it is. They can be the DHCP server just for those problematic legs though; would that help your aim?(Not that I think 2 DHCP servers is ideal but UniFi...)
️ -
@robbiett Thanks for the suggestion. We did test the Unifi L3 DHCP server and it works, the issue for us is that we use the DHCP/DNS integration of dnsmasq, i.e., all the hosts that use DHCP need to be added to our local DNS. Because of this having a "unified" DHCP/DNS server is required, we can't have some on the router and some on the L3 switch.
I have to admit I do find myself saying "but Unifi..." too.
-
@angusmh
I don't want to push this too much as it is a kludge on top of a kludge but depending on how dynamic things are you could write to the hosts files so that Dnsmasq knows what is going on. I've had to do something like that before, plus leaving myself # instructions in the file so I know what I did:#Entries for cat /etc/hosts #Using vi /etc/hosts to edit #To include the dhcp config stored in a different file: #source /path/to/file #source /etc/dnsmasq.d/dnsmasq-dhcp-config.conf #cat /etc/dnsmasq.d/dnsmasq-dhcp-config.conf 127.0.0.1 localhost.localdomain localhost 127.0.1.1 router-4 #vyatta entry 127.0.1.1 router-4.redacted.me #vyatta entry 2a02:390:8641:3::1 router-4.redacted.me router-4 10.0.1.1 router-4.redacted.me router-4 10.0.1.2 switch-24p.redacted.me switch-24p 10.0.1.3 switch-24x.redacted.me switch-24x 10.0.1.4 switch-5p.redacted.me switch-5pIt's just a thought but it can get you out of a jam sometimes.
️ -
@robbiett said in Netgate 8200 and Unifi L3 Switch Configuration:
've had to do something like that before, plus leaving myself # instructions in the file so I know what I did:
Thanks for the suggestion. We have a bunch of ephemeral linux containers spinning up and down so I would not want to have to manage updating the host file in our case I think. I could see this being an option if things weren't moving around so much though.
Thanks again.
