Weird VLAN problems

djheadcrash · Apr 18, 2017, 5:28 PM

I've got a wireless router which is connected to a 16-port managed switch.

While all devices connected directly to the switch should be able to communicate to each other, I want to route all WiFi traffic (LAN + WAN) through pfSense, so that I am able to enable custom firewall rules for guests and some IoT devices which should not communicate with other LAN devices in my network.
As a side note, all the devices are in the same subnet 192.168.1.1/20 and should stay there so I won't have any trouble with UPNP discovery and broadcast Messages etc.

I enabled in the managed switch 3 untagged 802.11q VLANs:

VLAN 1 - Default - Containing all ports
VLAN 10 - All ports except Wifi (12)
VLAN 11 - pfSense (6) + Wifi (12)

I assigned the following VLAN IDs to the ports:
Port 6 - pfSense (VLAN 1)
Port 12 - WiFi-Router (VLAN 11)
All other ports - VLAN 10

While this actually sends port 12 traffic to pfSense (checked via tracert), there are several side-effects:

Some devices don't communicate anymore with any device through port 12 (especially Unix-based NAS and Raspberry PI - even not answering on pings)
RDP sessions to windows devices loose connectivity after a few seconds

The switch has a "so-called" MTU VLAN feature (which actually doesn't have to do anything with MTU) forcing to route all ports to a specific uplink port, which I also tried. Even with this VLAN mode enabled, the problem appears and of course affects the remaining ports additionally.

Is there anything in pfSense that has to be enabled or setup, so that the routing will be stable? Might this problem be introduced due to a buggy implementation of VLANs within the switch? Or did I get something completely wrong, so that such scenario can't work out of the box?

johnpoz · Apr 18, 2017, 7:19 PM

I enabled in the managed switch 3 untagged 802.11q VLANs:
VLAN 1 - Default - Containing all ports
VLAN 10 - All ports except Wifi (12)
VLAN 11 - pfSense (6) + Wifi (12)

Huh?? If your going to use untagged?? HOw are you going to connect pfsense to this? You do not run multiple untagged networks.. According to your above pfsense would only be connected to vlan 11??

How do you have pfsense connected.. Where is this unmanaged switch and where is your managed switch. So pfsense is going to be a downstream router? Your managed switch is doing your L3 routing?? How many interfaces does pfsense have??

Your above seems to have multiple untagged vlans on the same ports..

"I want to route all WiFi traffic (LAN + WAN) through pfSense,"

What does that even mean?? How are you planning on routing wifi traffic through pfsense if the wifi devices are connecting to your wifi router at the edge connected to your ISP?

This is how it should/could be setup.. You could do it different if you have more ports in pfsense you want to use, etc.

So your lan interface would be your untagged.. This could be vlan 1 or any other vlan you want.. Pfsense is not going to care if this is your untagged network. Other vlans you want to run into pfsense would need to be tagged so you would need to create vlan interfaces that ride on this lan interface. Lets call it em1 while your wan is em0 in this setup which is connected to your internet.

You then could put ports on your managed switch on any vlan you want. Your really going to want to put your AP behind pfsense. Best would be to get a AP that does vlans. But if just going to use some old wifi router as AP it could be connected to whatever vlan you want. Vlan 10 in this drawing, you could then have other vlans (11 in this example). You would then connect your dumb switch to whatever vlan you wanted to.. but if we are going to use vlan 1 as your lan.. Then lets put it on your lan.. Then all devices connected to that dumb switch would be in your lan.

You then create whatever rules you want to allow intervlan access between your vlans.

djheadcrash · Apr 18, 2017, 8:46 PM

Hi johnpoz

Thank you so much for your guidance on that issue!

Basically, my setup is exactly the same as the one, you recommend to me (cable modem <-> pfSense (DHCP server, IPsec etc.) <-> managed switch).
My idea was that if I enable three untagged VLANs on the managed switch, the switch will avoid direct communication between the WiFi router and the remaining ports, routing all traffic to the pfSense and pfSense will forward the communication back to the managed switch (when passing my firewall rules which I will add later) but it leads me to the already mentioned side effects (packet loss etc.)…

Can you give me some hints how one can create tagged VLANs interfaces in pfSense which will play nicely on the same subnet (192.168.1.1/20)? All tutorials I've found are based on 1 subnet per VLAN which would cause additional headache when working with broadcasts etc.
When trying to set it up by myself, I endup in loosing all connectivity. Do I need to create pass rules between VLAN and LAN in pfSense? If so, what's the source and destination?

Derelict · Apr 18, 2017, 9:17 PM

Can you give me some hints how one can create tagged VLANs interfaces in pfSense which will play nicely on the same subnet (192.168.1.1/20)?

This doesn't make any sense. The reason to create multiple VLANs is to isolate networks, not combine them.

djheadcrash · Apr 18, 2017, 11:17 PM

Ok. If VLAN doesn't make any sense in this scenario, how can this requirement be fulfilled otherwise?
I want to ensure that clients connected to the Wifi-device can't communicate without being routed to pfSense…
I would be glad to have some hints. Thanks a lot!

Derelict · Apr 18, 2017, 11:38 PM

VLANs make perfect sense. What doesn't make sense is having everything on the same 192.168.1.0/20

VLAN 100 - 192.168.100.0/24
VLAN 101 - 192.168.101.0/24
VLAN 102 - 192.168.102.0/24

To your firewall they are separate interfaces and can be treated as such, with rules passing or blocking traffic between them.

djheadcrash · Apr 19, 2017, 6:41 AM

The only reason why I am currently not going for this solution is that this will break multicast and device discovery and I have absolutely no idea how I can forward such packets from one subnet to another…

Would it be a better idea to add another physical network port (e.g. USB) to my pfSense and connect the Wifi router directly to that port?
I mean I am about routing and filtering network traffic within my network.

doktornotor · Apr 19, 2017, 8:24 AM

No, having a physical port will not change anything (except for making things massively worse if it's USB.)

With the nonsense you are producing with VLANs, just ditch the VLANs and hang everything on the same dumb switch on a single subnet.

johnpoz · Apr 19, 2017, 6:27 PM

^ exactly… What multicast are you trying to pass exactly?

And what devices need to use this multicast.. There are plenty of ways to design the network to allow your multicast to work while still isolating other devices, etc. Simple solution is to just put the device that uses multicast and the devices that access this device via multicast on the same layer 2.

What exact devices are you working with?? Its quite possible you don't actually need multicast but just need to proxy some info like mdns which could be done with avahi..

Example.. My ipad and phones using airprint to my printer.. So I could of created some dns stuff to allow them to find the printer.. But it was easier to just put the wired printer on the same layer 2 as the wifi network those devices connect to. My other devices that use the printer don't use airprint and just need the IP of the printer to send prints across the segments, etc.

If we know the devices and exact protocols you are working with or think you need, which might not be the case we can work out how best to segment your network and provide the security boundaries you want.

djheadcrash · Apr 20, 2017, 8:00 AM

Hi johnpoz,

the devices in question are basically several IoT devices and gateways, such as an Orvibo AllOne (IR+RF 433MHz blaster), Xiaomi Gateway (ZigBee hub), Xiaomi Mi Robot (Robot vacuum) among others. Most of the mobile apps which control these devices use UDP multicasts for discovery each time you start to establish a communication.

While I am using Domoticz on a Raspberry Pi to keep things together (actually you don't want to use plenty of different apps to switch a light etc.), the apps are required for non-daily tasks (like driving the vacuum into another room).

As soon as I put these devices into a different subnet… Well, you know the result. ))
On the other hand, leaving my network the way as it is now is also a no-go.

While digging deeper into possible solutions for what I am trying to do, I came across this tutorial (Author derelict?):
https://www.infotechwerx.com/blog/Creating-a-Simple-pfSense-Bridge

I really don't want to make you guys busy with my nonsense because I am already so glad that you all took the time to give me some advise about do's and dont's.

Anyway, wouldn't it be possible to bridge two interfaces in pfSense (one interface - managed Switch, other interface - dumb Wifi router) and add firewall rules on the bridge interface (like: block 192.168.5.1 -> 192.168.1.1 - 192.168.2.255)?
If so, should I really not go for such setup? It would make things much more easier.

johnpoz · Apr 20, 2017, 9:42 AM

Dude you do not bridge interfaces and use different networks? 192.168.5, 192.168.2

In your example why would your iot devices not all be on the same layer 2 network? ie the same wifi network.. what devices are you using to control them - why wold they not be on the same wifi or layer 2 network via a wire?

Not seeing here why you think you have an issue?

"Most of the mobile apps which control these devices use UDP multicasts"

Why would you not just connect the device to that wifi network??

Derelict · Apr 20, 2017, 5:06 PM

IMHO any "IoT" device that:

Does not play nicely with IoT network isolation
requires any inbound port forwards or special rules

should be promptly returned for a refund.

You might have some joy using a bridge with filters on the member interface but IoT is supposed to make life better/easier. And as johnpoz states, they would not be different layer 3 networks/subnets in that case.

johnpoz · Apr 20, 2017, 7:19 PM

Good point about returning the devices that don't support isolation.. I would agree!

I have not run into any iot devices that I can not isolate on their own layer 2 and 3.. None of my iot devices have access into any of my "normal" networks. I have access into them from my normal into them.. But there are no forwards from the outside into them either.

But I don't see any sort of issue that if you want to control some iot device with a mobile app just putting them on the same wifi network that would be 1 layer 2, and its own layer 3 and then isolating this network from the rest of your network(s)..

Derelict · Apr 20, 2017, 7:51 PM

Any device that uses archaic discovery methods such as broadcast should also have some way to nudge/force it using DHCP, DNS, etc, else it should be returned.

mDNS/bonjour should be solvable with avahi, as has been stated.