pfSense + Layer 3 + Access Point

[Zipping8761]

I have been using pfSense as my network firewall for over 10 years. I have many rules with separate vpns set up for different clients. I tried to segment clients as best as I could without being able to use vlans because I didn't have the hardware for it. Now I've acquired some used equipment that is capable of it and would like to move my network over. I've read many topics on how to do it but am still new to vlans and layer 3 switches. Here is a list of topics I have been reading to better understand my options:

Layer 3 Switch w/ PFSense - Configuring InterVLAN Routing with a Layer 3 Switch and pfSense - pfSense and Brocade - pfSense baseline guide with VPN, Guest and VLAN support - Configuring the L3 switch - VLAN configuration on Ruckus ICX switches for Ruckus Wireless WLAN's

pfSense currently handles dhcp and local dns. It also is the ntp server for all clients and I run pfBlockerNG and Suricata. I also run a traffic shaper to restrict clients, especially mobile ones, from using more bandwidth than they need. Our internet here is bad. I'm wondering how layer 3 can handle working with those pfSense features. I'm hoping by moving dhcp to layer 3 that I do not have to give up any of the pfSense features that I am using but I simply don't know enough yet.

Here is a diagram of what I am hoping the final product will produce. In terms of internet access, it's similar to what I already have now. I just don't have the network separation. The cleaner vlan design with the new segmented wifi ssid's look nice.

Will I still be able to allow traffic from one vlan to another vlan from a specific port, etc.? I know the rules on the layer 3 switch will not be as feature rich as on pfSense. Should I use the switch as layer 2 and give up all the benefits of faster routing and keep the restrictions and rules on pfSense?

It looks like if I use static routes on pfSense, I can keep dhcp on the layer 3 switch and still redirect the correct clients over the proper vpn with seperate firewall rules.

I currently have the layer 3 switch setup with an ip of 192.168.99.8 and have a new interface in pfSense with a gateway of 192.168.99.1. I have it connected to a third nic port so that I can work on this new setup without effecting my current network at all. I'm very new to interacting with this switch and the aruba ap, so I need my hand held here and would appreciate an experts guidance. Thank you all.

[johnpoz]

@zipping8761 said in pfSense + Layer 3 + Access Point:

Should I use the switch as layer 2 and give up all the benefits of faster routing

And what are you doing that you need to route at the switch? Are you dumping GB of files to your nas or writing them on the fly as you edit video?

I also run a traffic shaper to restrict clients, especially mobile ones, from using more bandwidth than they need.

From that statement I would think not ;)

What are you running through pfsense now that it can not route at wire speed? Are you trying to bump up to 10g or 2.5ge and switches can do that and have those interfaces, but your pfsense box does not? Even if you were jumping to higher than gig via switch ports, why do you have to route these devices that want to talk at the higher speed? Why can they not just be in the same L2? Trying to do ACLs on a L3 switch isn't point and click like in pfsense ;) Now they can be useful sure, I use multicast ACLs on my L2 networks to reduce noise, etc. But trying to actual "firewall" isn't as simple as pfsense.

There is almost zero reason for a home user or a smb to route at the switch, other than a learning experience in setting it up - since you would need to understand what a transit network is, etc. Or you would be doing it wrong.

So unless you can point out why your pfsense box can not route at speed you need it to route at?? There would be zero reason to route at the switch.. Other than lab and you can, and your a geek and want to, and you don't care about firewall rules between the segments your going to route at the switch, etc.

I would say do yourself a favor and keep your routing and firewalling at pfsense, and just use L2 on your switch.. You know if you have bandwidth constraints you can also use multiple uplinks.. Just because your using vlans, doesn't mean all those vlans have to share the bandwidth of 1 physical switch port. You could spread your vlans out across multiple interfaces, not putting vlans that have lots of intervlan traffic on the same physical interface, etc.

[Zipping8761]

Thank you for your wisdom.

I see your point and it makes sense. If it isn't broke, don't try to mess with it. However, I needed a challenge and learning layer 3 or even layer 2 switching is something new. I saw on the Serve The Home forums (that I was not permitted to join) you were brought up as the go to networking guy here and you were running layer 2 switching with your pfSense firewall and highly recommend it. I'm still not sure which way I will be going but I since I am using a separate network port, I can play around without making permanent changes or effect the existing network.

I started by creating VLAN 99 and adding an single untagged port to it. I set up both pfSense and layer 3 with a /30 so no other access is granted on that port. I added a gateway to pfSense and gave it the layer 3 ip address and created some static routes pointing to that gateway. This will let me see what layer 3 is like and I can always put it back.

Now I am in the process of setting up an access point. Here is how I setup that port: ICX 6450 VLAN Help. I'm hoping this is the correct way to do it and provide the ap with a main ssid for vlan 40 and two extra ssid's for iot (42) and guests (44). The ap was given an ip address through dhcp on the layer 3 switch via a static mapping and I can ping the ap through all clients. This ap turned out to be the wrong one and needs an Aruba controller so I will be exchanging it for the instant access one. I won't be able to go any further with this section yet.

Some have said that the dhcp services on the layer 3 switch are not good, so I was thinking about running isc dhcp on one of my linux clients. Any thoughts on this? I'd rather use the layer 3 switch for it if the dhcp service works correctly. As I said, this is just playing / learning at this point so if I go back to layer 2 I can leave dhcp on pfSense.

If I do run an isc dhcp service on a separate machine, I was thinking of using separate nic ports to isolate the services and giving it it's own vlan and ip address. Maybe this is stupid and ridiculous but I'd like to get your thoughts on it. These 2 old topics were all I could find on this subject and I haven't tried to implement it yet in testing: Two Default Gateways on One System - Add network eth with a separate gateway.

Along the same line, I have a docker container running a service on the same machine and would like to isolate it as well giving it its own ip address and vlan on a separate nic port. This would allow me to setup ACLs and restrict access on the layer 3 switch. I could still do this in layer 2 as well and pass it on to pfSense to be firewalled with rules. I still haven't quite figured this one out yet. If I setup the routing tables on linux, how do I tell docker to use that routing table for its container? This is what I found about this subject: Setting Default Gateway to a container - Routing packets from a specific Docker container through a specific outgoing interface - Routing from docker containers using a different physical network interface and default gateway. Maybe this is also a ridiculous thought process.

I'll continue sharing what I learn as I go in case anyone else finds it of interest.

[johnpoz]

@zipping8761 if you want to play/learn with downstream router - that is great, and more power to you.

But other than a learning experience - I don't really see a need to make your life harder ;)

Keep in mind that if your going to use your L3 switch as a downstream router, that the connection needs to be on a transit network (no hosts on this network) or you going to more than likely run into asymmetrical traffic flow.

Here is good diagram for how to do routing to a downstream router (L3 switch doing routing).

🔒 Log in to view

As to running some other dhcp sever on your network and provide dhcp for all your other networks - sure quite possible and pfsense could be even be the relay, on cisco called helper address.. So dhcp from 1 layer 2 can be sent to a dhcp server on a different network - as long as your dhcp server supports this function - and sure isc dhcp does.. Just need to set it up.

With pfsense though, it can only provide dhcp services to networks it is attached to, it can not provide dhcp to network that are downstream via a different router.

As to your docker question - I have min exp with docker, on synology nas I run a few dockers - but docker normally nats, etc. Networking with docker where the docker actually has a IP on the same network as the actual host running the dockers is bit more complicated. Stuff about macvlan settings and the like - it can be done, but pretty sure out of the box dockers are only natted to the host IP. And would take on the routing of the host, etc.

[Zipping8761]

Well your right about upstream confusion. I needed to setup a tftp link to flash my Aruba access point and I couldn't get it to work on my normal LAN; connected into a separate NIC port than my switch. I had to setup another vlan (10) so it would connect to vlan 40 (which is the Aruba ap). So I must have an issue somewhere in my pfSense rules but I couldn't find any. Devices could access the internet and each other, ping always showed good. I don't know why it couldn't download from the local tftp but since I won't be separating upstream traffic in the end (it will all go on one interface) I didn't spend any time on it. I just hooked the tftp device up to the switch and got what I needed to do done.

The problem I'm working on now...

I have my ICX switch setup incorrectly or the Aruba access point. Maybe both but I thought I had everything tagged correctly. If I don't tag traffic as vlan 40 on the Aruba 'Mobile' SSID, the mobile device can connect. It even gets the correct ip address in the 192.168.40.x range. ICX shows it getting a lease in dhcp. But no internet access on the device. If I connect to the 'Guest' SSID also untagged, it does the same thing and it gives it another ip address in the 192.168.40.x range. It's supposed to be using the 192.168.44.x range for the 'Guest' SSID. So it's supposed to be tagged in Aruba, obviously or it's just going to keep using the default 40. When I set the tags though, the devices can't connect. I don't know if it's an Aruba problem or an ICX problem.

VLANs on ICX:

🔒 Log in to view

Routing Table on Aruba:

The 172.x.x.x address is the default SSID which is now deleted but I don't know if it can be removed from the routing table.

🔒 Log in to view

192.168.40.8 is the ip address of the Aruba AP and Mobile is the SSID:

What does the virtual controller ip mean? I couldn't figure it out. There is also something called an uplink vlan. I set it to 40 and I couldn't load the web ui so I had to change it back to 0 in the serial console.

🔒 Log in to view 🔒 Log in to view

I can ping 1.1.1.1 from the Aruba serial console. I can ping the ICX port the Aruba (192.168.40.1) is plugged into. I'd test the Aruba by bypassing the switch if I could but it requires POE and my ICX switch is all I have with POE. Once I can figure this one out I'd like to think I will be closer to understanding it all but probably not.

[Zipping8761]

As a follow up: I forgot to make vlan 42 and vlan 44 into router interfaces and assign them an ip address. I also didn't have dhcp servers setup for them. Now I can connect to all 3 of the ssids and receive the proper ip address assigned to the vlan the ssid is on.

I believe for best security I should re-assign the mobile ssid to vlan 46 instead of vlan 40. Since the access point is managed by vlan 40, it should be given a /30 and not assign any other devices to it. This is how I have the connection to pfSense setup and I believe it is the proper way to go, never use the management vlan that 2 switches are connected to each other on as being used by other devices. Correct? I just need to figure out how to deny all dhcp leases on vlan 40 unless a static mapping is set. I only want the access points I assign to have dhcp leases. Easy to do in pfSense but not straightforward on the layer 3 switch.

Now the devices still have no internet access. I am able to ping from the devices 1.1.1.1 but it will not resolve the hostname so it looks like DNS is not working properly. DNS is being passed on to pfSense and the firewall is not blocking it. I can run a traceroute on the switch and it is able to process DNS. The DNS is not being passed along to the access point. I can see the DNS requests coming into pfSense through the states table.

[Zipping8761]

On the switch:

🔒 Log in to view

On the access point:

🔒 Log in to view

[johnpoz]

@zipping8761 google.com is invalid means what? Can you resolve google.com or not?

If you can not resolve it - then no your not going to be able to ping it

[Zipping8761]

I don't know. Maybe the access point is incapable of accepting the second option as a host and requires an ipv4/ipv6 address only. Or maybe it means it can't resolve it because it cannot connect to the DNS server. I wish I knew. I'm assuming it can't access the DNS server but as you can see above that it is able to ping 192.168.99.1 which is where the DNS resolver on pfSense lives. The switch has no problem resolving DNS.

While running traceroute from a proper linux laptop connected to the switch via vlan 10:

traceroute google.com
google.com: Temporary failure in name resolution
Cannot handle "host" cmdline arg `google.com` on position 1 (argc 1)

Traceroutes work with any ipv4 addresses but not host names which require DNS lookup. I don't know how pfSense is blocking DNS to not go beyond the switch to the hosts connected to the switch. I can change the DNS servers to 9.9.9.9 in /etc/resolv.conf and fix everything but that's not a solution to this problem. Something about switch routing or pfSense I'm not understanding here.

[johnpoz]

@zipping8761 if you have a downstream network, and you want this downstream network to use unbound, you need to added it to the acls

out of the box pfsense only auto adds its locally attached networks, and or tunnel networks of vpn server setups.

If you have say downstream network 10.0.0/24 via your 172.16.0.0/30 transit interface, you have to add 10.0.0/24 to your unbound ACL

[Zipping8761]

@johnpoz

Thank you for that. That was the magic setting that I didn't know about.

[Zipping8761]

I have successfully transferred my entire network over to my layer 3 switch. It was quite the chore to give all of my devices a new ip address. I had been using static ip addresses through pfsense's dhcp server. Most devices were unimportant but a few servers serve all the clients with their static ip addresses and were last to be transferred over. I am still using the dhcp server on the layer 3 switch and haven't decided to migrate from it yet. Maybe later.

I have managed to separate services on my server through a second network interface card and docker. My mistake was trying to use a /30 subnet where docker needed to have an ip address assigned from the same subnet but with a /30 that wasn't possible. So a workaround was to use a /24 subnet and block dhcp from assigning any more addresses. You can make up any ip address you want for docker as long as it has the same subnet setting as the NIC. I can share specifics if anyone else wants to try this.

I'm moving on to learning access lists. I couldn't understand the in and out directions. This video finally explained it in a way that I could understand. It's been a fun if not difficult learning experience so far. The only reason I think it's been so difficult is because I wanted to slowly transfer my network over and keep everything working during the whole process. If I had started from scratch I think all of my challenges would have been lessened.

Is it better or more efficient to have pfsense paired with a layer 2 switch instead of an layer 3 switch? Possibly. Your losing some functionality of pfsense. Time will tell how I like it. I know I am learning a lot of new things and that's good for my brain. Hopefully no one else is afraid to take on this challenge, I know I am glad I did.

[johnpoz]

@zipping8761 it can be a great learning experience.. But yeah its going to be more difficult than just clicking whatever rules you want in the pfsense gui ;)

And what are you using for dhcp - pfsense can not provide dhcp to clients that its not directly attached too.. So some downstream networks that are routed at your L3 switch would not be able to use the dhcpd on pfsense for their ips, etc.

Glad your having fun and learning..

[Zipping8761]

@johnpoz

I have an interface on pfsense connected to the layer 3 switch with a /30 subnet. I have the dhcp server enabled for that interface. There is no lease present in dhcp for the switch but there is an entry in arp. Since I would have had to create an interface for each vlan in order to use pfsense's dhcp server, I am currently using the downstream layer 3 switch for all dhcp.

I ran into an issue here with a smart tv that would not get an ip address. After a few days I had to let the wifi access point handle the dhcp for the smart tv. Since it's the only device on that vlan, it'll work for now. I couldn't figure out why it was failing. For the first few days I thought it was some wrong wifi band causing auth failure but once I was able to figure out how to view the access point's debug log it kept showing successful authentication. I am planning on running ISC DHCP at some point but as you stated it's still going to be downstream from pfsense.

I'm still using pfSense for DNS because I'm running pfBlockerNG and want that to continue to work.

I guess thinking about it now I am going to miss pfSense with Suricata acting as an intrusion protection system between the vlans (internal network) themselves. It almost makes me want to go back to a layer 2 switch mode. :) But for the most part all I ever really had between clients is false positives so will I really miss much? When I have my ACLs in place, none of the noisy devices will ever be able to access the desktops or servers. Plus everything is linux based with their own intrusion systems already so I may not miss the ips from pfSense in regards to that. It's still working for internet based blocking.

Now that everyone has a new static ip address assigned, I could easily switch over to layer 2 and setup vlan interfaces in pfsense in a few minutes should I decide I don't like layer 3.

[Zipping8761]

I'll probably go back to layer 2 eventually because these ACLs are harder to understand than originally thought. A simple ping test block for example. To block all pings from source 192.168.48.x devices to destination devices on 192.168.10.x would be:

ip access-list extended VLAN48-ACL
deny icmp any 192.168.10.0/24
permit ip any any
exit

interface ve 48
ip access-group VLAN48-ACL in
exit

As expected pings from 192.168.48.x devices to 192.168.10.x devices are blocked but pings from 192.168.10.x devices to 192.168.48.x devices are also blocked.

A device on vlan 48 sends packets in on vlan 48 and they go out on vlan 10 before hitting the destination device on vlan 10 without ever going out vlan 48 or in on vlan 10.

In order to achieve the desired results you also have to think about a reply from a ping coming back. The echo would be blocked by the deny icmp any 192.168.10.0/24 rule unless you add echo to the end of the rule.

deny icmp any 192.168.10.0/24 echo

How to deny only incoming traffic to a certain VLAN

[johnpoz]

@zipping8761 haha - I warned you, but it a good learning experience ;)