pfSense + Layer 3 + Access Point

[johnpoz]

@zipping8761 if you want to play/learn with downstream router - that is great, and more power to you.

But other than a learning experience - I don't really see a need to make your life harder ;)

Keep in mind that if your going to use your L3 switch as a downstream router, that the connection needs to be on a transit network (no hosts on this network) or you going to more than likely run into asymmetrical traffic flow.

Here is good diagram for how to do routing to a downstream router (L3 switch doing routing).

As to running some other dhcp sever on your network and provide dhcp for all your other networks - sure quite possible and pfsense could be even be the relay, on cisco called helper address.. So dhcp from 1 layer 2 can be sent to a dhcp server on a different network - as long as your dhcp server supports this function - and sure isc dhcp does.. Just need to set it up.

With pfsense though, it can only provide dhcp services to networks it is attached to, it can not provide dhcp to network that are downstream via a different router.

As to your docker question - I have min exp with docker, on synology nas I run a few dockers - but docker normally nats, etc. Networking with docker where the docker actually has a IP on the same network as the actual host running the dockers is bit more complicated. Stuff about macvlan settings and the like - it can be done, but pretty sure out of the box dockers are only natted to the host IP. And would take on the routing of the host, etc.

[Zipping8761]

Well your right about upstream confusion. I needed to setup a tftp link to flash my Aruba access point and I couldn't get it to work on my normal LAN; connected into a separate NIC port than my switch. I had to setup another vlan (10) so it would connect to vlan 40 (which is the Aruba ap). So I must have an issue somewhere in my pfSense rules but I couldn't find any. Devices could access the internet and each other, ping always showed good. I don't know why it couldn't download from the local tftp but since I won't be separating upstream traffic in the end (it will all go on one interface) I didn't spend any time on it. I just hooked the tftp device up to the switch and got what I needed to do done.

The problem I'm working on now...

I have my ICX switch setup incorrectly or the Aruba access point. Maybe both but I thought I had everything tagged correctly. If I don't tag traffic as vlan 40 on the Aruba 'Mobile' SSID, the mobile device can connect. It even gets the correct ip address in the 192.168.40.x range. ICX shows it getting a lease in dhcp. But no internet access on the device. If I connect to the 'Guest' SSID also untagged, it does the same thing and it gives it another ip address in the 192.168.40.x range. It's supposed to be using the 192.168.44.x range for the 'Guest' SSID. So it's supposed to be tagged in Aruba, obviously or it's just going to keep using the default 40. When I set the tags though, the devices can't connect. I don't know if it's an Aruba problem or an ICX problem.

VLANs on ICX:

Routing Table on Aruba:

The 172.x.x.x address is the default SSID which is now deleted but I don't know if it can be removed from the routing table.

192.168.40.8 is the ip address of the Aruba AP and Mobile is the SSID:

What does the virtual controller ip mean? I couldn't figure it out. There is also something called an uplink vlan. I set it to 40 and I couldn't load the web ui so I had to change it back to 0 in the serial console.

I can ping 1.1.1.1 from the Aruba serial console. I can ping the ICX port the Aruba (192.168.40.1) is plugged into. I'd test the Aruba by bypassing the switch if I could but it requires POE and my ICX switch is all I have with POE. Once I can figure this one out I'd like to think I will be closer to understanding it all but probably not.

[Zipping8761]

As a follow up: I forgot to make vlan 42 and vlan 44 into router interfaces and assign them an ip address. I also didn't have dhcp servers setup for them. Now I can connect to all 3 of the ssids and receive the proper ip address assigned to the vlan the ssid is on.

I believe for best security I should re-assign the mobile ssid to vlan 46 instead of vlan 40. Since the access point is managed by vlan 40, it should be given a /30 and not assign any other devices to it. This is how I have the connection to pfSense setup and I believe it is the proper way to go, never use the management vlan that 2 switches are connected to each other on as being used by other devices. Correct? I just need to figure out how to deny all dhcp leases on vlan 40 unless a static mapping is set. I only want the access points I assign to have dhcp leases. Easy to do in pfSense but not straightforward on the layer 3 switch.

Now the devices still have no internet access. I am able to ping from the devices 1.1.1.1 but it will not resolve the hostname so it looks like DNS is not working properly. DNS is being passed on to pfSense and the firewall is not blocking it. I can run a traceroute on the switch and it is able to process DNS. The DNS is not being passed along to the access point. I can see the DNS requests coming into pfSense through the states table.

[Zipping8761]

On the switch:

On the access point:

[johnpoz]

@zipping8761 google.com is invalid means what? Can you resolve google.com or not?

If you can not resolve it - then no your not going to be able to ping it

[Zipping8761]

I don't know. Maybe the access point is incapable of accepting the second option as a host and requires an ipv4/ipv6 address only. Or maybe it means it can't resolve it because it cannot connect to the DNS server. I wish I knew. I'm assuming it can't access the DNS server but as you can see above that it is able to ping 192.168.99.1 which is where the DNS resolver on pfSense lives. The switch has no problem resolving DNS.

While running traceroute from a proper linux laptop connected to the switch via vlan 10:

traceroute google.com
google.com: Temporary failure in name resolution
Cannot handle "host" cmdline arg `google.com` on position 1 (argc 1)

Traceroutes work with any ipv4 addresses but not host names which require DNS lookup. I don't know how pfSense is blocking DNS to not go beyond the switch to the hosts connected to the switch. I can change the DNS servers to 9.9.9.9 in /etc/resolv.conf and fix everything but that's not a solution to this problem. Something about switch routing or pfSense I'm not understanding here.

[johnpoz]

@zipping8761 if you have a downstream network, and you want this downstream network to use unbound, you need to added it to the acls

out of the box pfsense only auto adds its locally attached networks, and or tunnel networks of vpn server setups.

If you have say downstream network 10.0.0/24 via your 172.16.0.0/30 transit interface, you have to add 10.0.0/24 to your unbound ACL

[Zipping8761]

@johnpoz

Thank you for that. That was the magic setting that I didn't know about.

[Zipping8761]

I have successfully transferred my entire network over to my layer 3 switch. It was quite the chore to give all of my devices a new ip address. I had been using static ip addresses through pfsense's dhcp server. Most devices were unimportant but a few servers serve all the clients with their static ip addresses and were last to be transferred over. I am still using the dhcp server on the layer 3 switch and haven't decided to migrate from it yet. Maybe later.

I have managed to separate services on my server through a second network interface card and docker. My mistake was trying to use a /30 subnet where docker needed to have an ip address assigned from the same subnet but with a /30 that wasn't possible. So a workaround was to use a /24 subnet and block dhcp from assigning any more addresses. You can make up any ip address you want for docker as long as it has the same subnet setting as the NIC. I can share specifics if anyone else wants to try this.

I'm moving on to learning access lists. I couldn't understand the in and out directions. This video finally explained it in a way that I could understand. It's been a fun if not difficult learning experience so far. The only reason I think it's been so difficult is because I wanted to slowly transfer my network over and keep everything working during the whole process. If I had started from scratch I think all of my challenges would have been lessened.

Is it better or more efficient to have pfsense paired with a layer 2 switch instead of an layer 3 switch? Possibly. Your losing some functionality of pfsense. Time will tell how I like it. I know I am learning a lot of new things and that's good for my brain. Hopefully no one else is afraid to take on this challenge, I know I am glad I did.

[johnpoz]

@zipping8761 it can be a great learning experience.. But yeah its going to be more difficult than just clicking whatever rules you want in the pfsense gui ;)

And what are you using for dhcp - pfsense can not provide dhcp to clients that its not directly attached too.. So some downstream networks that are routed at your L3 switch would not be able to use the dhcpd on pfsense for their ips, etc.

Glad your having fun and learning..

[Zipping8761]

@johnpoz

I have an interface on pfsense connected to the layer 3 switch with a /30 subnet. I have the dhcp server enabled for that interface. There is no lease present in dhcp for the switch but there is an entry in arp. Since I would have had to create an interface for each vlan in order to use pfsense's dhcp server, I am currently using the downstream layer 3 switch for all dhcp.

I ran into an issue here with a smart tv that would not get an ip address. After a few days I had to let the wifi access point handle the dhcp for the smart tv. Since it's the only device on that vlan, it'll work for now. I couldn't figure out why it was failing. For the first few days I thought it was some wrong wifi band causing auth failure but once I was able to figure out how to view the access point's debug log it kept showing successful authentication. I am planning on running ISC DHCP at some point but as you stated it's still going to be downstream from pfsense.

I'm still using pfSense for DNS because I'm running pfBlockerNG and want that to continue to work.

I guess thinking about it now I am going to miss pfSense with Suricata acting as an intrusion protection system between the vlans (internal network) themselves. It almost makes me want to go back to a layer 2 switch mode. :) But for the most part all I ever really had between clients is false positives so will I really miss much? When I have my ACLs in place, none of the noisy devices will ever be able to access the desktops or servers. Plus everything is linux based with their own intrusion systems already so I may not miss the ips from pfSense in regards to that. It's still working for internet based blocking.

Now that everyone has a new static ip address assigned, I could easily switch over to layer 2 and setup vlan interfaces in pfsense in a few minutes should I decide I don't like layer 3.

[Zipping8761]

I'll probably go back to layer 2 eventually because these ACLs are harder to understand than originally thought. A simple ping test block for example. To block all pings from source 192.168.48.x devices to destination devices on 192.168.10.x would be:

ip access-list extended VLAN48-ACL
deny icmp any 192.168.10.0/24
permit ip any any
exit

interface ve 48
ip access-group VLAN48-ACL in
exit

As expected pings from 192.168.48.x devices to 192.168.10.x devices are blocked but pings from 192.168.10.x devices to 192.168.48.x devices are also blocked.

A device on vlan 48 sends packets in on vlan 48 and they go out on vlan 10 before hitting the destination device on vlan 10 without ever going out vlan 48 or in on vlan 10.

In order to achieve the desired results you also have to think about a reply from a ping coming back. The echo would be blocked by the deny icmp any 192.168.10.0/24 rule unless you add echo to the end of the rule.

deny icmp any 192.168.10.0/24 echo

How to deny only incoming traffic to a certain VLAN

[johnpoz]

@zipping8761 haha - I warned you, but it a good learning experience ;)