VLAN traffic blocked on LAN

[Talon9804]

Hello Everybody!
I've been pulling my hair out trying to troubleshoot some odd behavior on my Netgate SG-3100 at home, hoping for some insight.

The Context:
My network is configured in what I think is simple:
Main LAN configured on Opt 1 (mventa0) of the SG-3100 (the solo port, not the switch ports!). Using the default LAN as my management network. - 192.168.4.0/24
Four VLANs configured with mventa0 as the parent interface:
-VLAN 24 for home devices - 192.168.24.0/24
-VLAN 34 for IoT devices - 192.168.34.0/24
-VLAN 54 for Camera network - 192.168.54.0/24
-VLAN 94 for Guest devices - 192.168.94.0/24

Each of them are enabled, have DHCP servers enabled, and all successfully point to my DNS server (PiHole) which is homed on my management default LAN with working firewall rules. Clients all get correct IP addresses, and when things work, they successfully get DNS and internet access.

In terms of topology, I have the following (I have a 16-port switch on the way that will simplify this, but for now this is what I have):

I am using a Unifi Switch 8 60w as a primary switch, with a few other switches connected downstream to feed different rooms or different functions. The trunk port to pfSense is configured as All in the Unifi Controller, as are all of the downstream switch trunk ports.

I also have separate SSID's for each of the VLANs configured in my Unifi Controller (CloudKey Gen 2+), and I will post their configs below.

The Problem:
Intermittently, client traffic from any VLAN starts to appear on the LAN interface, and subsequently gets blocked by the firewall. For simplicity sake, I'll focus on VLAN 24, where my family members' devices are homed, but I see similar behavior across other VLANS.

If I take my phone and connect to my VLAN 24 SSID (BergieNet), my phone can access the internet for about 30 seconds. After that, the firewall starts to block all traffic from my phone with the TCP:S flag. If I do pings, I cannot ping 8.8.8.8, but I can ping the VLAN gateway at 192.168.24.1 and other local devices on the 192.168.24.x subnet. Oddly, I can also ping some devices on the native LAN (192.168.4.x) (which should be blocked by firewall rules!) but no internet traffic.
Internet connectivity will come and go--sometimes if I let the ping run, I'll be able to hit 8.8.8.8, and for a short time I'll have access like I should, with everything behaving as expected according to firewall rules. After a short time again, however, all outbound traffic gets blocked again.

Notice in this case that the Interface listed is LAN, which has a subnet of 192.168.4.0/24, but the source IP is from VLAN 24 with 192.168.24.0/24, and the destination is a WAN address.
Why is my VLAN traffic appearing on my LAN interface?
This happens most of all with wireless devices, but also occurs briefly with wired clients, but tends to resolve more quickly on the wired devices.

Or in other situations, I have devices on the native management LAN that should be permitted to access devices on the VLANs that are still getting blocked. For example, my CloudKey is homed in the native management LAN (subnet 192.168.4.0/24), and that interface has a LAN Net/Any firewall rule. However, it gets blocked when trying to access some cameras on the Camera VLAN (192.168.54.0 subnet).

As I've researched, as far as I can tell this seems like asymmetric routing behavior, but for my life I cannot figure out why this is occuring: I have only one router, I've specifically tagged all client ports on my switches with the specific VLAN port profile they need, all switches are connected via All-profile trunks, and in general I've tried to cross my T's and dot my I's. For troubleshooting, I've reset the state table, played with firewall rules rebooted the Netgate box and switches several times, attempted to sacrifice to the technology gods, done a few rain dances--all to no avail. I'm hoping that others can look over what I've got set up and help me see what I am missing.

Here are my native management LAN rules (192.168.4.0/24):

VLAN 24 (Private/family devices) Firewall Rules:

And the VLAN 54 (Camera, local access only) Rules:

Also, here is an example of my VLAN configurations in the Unifi Controller for the Unifi switches (all are set similarly for their respective VLAN):

And this is how the VLAN 24 SSID is configured (called BergieNet) (all other SSIDS's are set similarly for their respective VLAN):

Thank you all in advance for any input you can provide!

[johnpoz]

@talon9804 said in VLAN traffic blocked on LAN:

it gets blocked when trying to access some cameras on the Camera VLAN (192.168.54.0 subnet).

The way that looks to me is 192.168.54.34 talking to 4.2 to port 7550 wasn't seen by pfsense, so when the 4.2 answers its SA sending to its gateway, pfsense had no state so you see the SA blocked.

This could occur if for some reason you didn't have L2 isolation and 192.168.54.34 was able to talk to 4.2 directly be it wrong mask on the 54.34 box, or some other routing that allowed it to talk to 4.2 without it going through pfsense.

[Talon9804]

@johnpoz Thank you for your input!
And right—definitely that’s what’s going on, but what is killing me is why/how it is happening!

In this case, one router is going to one switch, to one access point. Each switch client port is set for the specific VLAN, and all SSID’s are set up for their respective VLANs. How is it that my wireless client VLAN traffic (on 24.x or 54.x) take a direct route back to LAN devices (4.x).

Or more concerning/important to me—why is my outbound 24.x traffic appearing on my LAN interface (and thus getting blocked) when its source is the 24.x subnet, and it’s destination is WAN? Why would it pass through the LAN interface at all?

[johnpoz]

@talon9804 possible your switches have the wrong PVID set..

So for example on the port you have connect that you have in vlan X, so traffic gets to say 4.2 box, but when 4.2 sends back its SA, the pvid on the port is not vlan X and is your default lan pvid, default is normally 1.. This would for explain what your seeing.

When you set a port in some vlan, you also have to set the pvid to match.. I don't have a lot of experience with the unifi switches, I do have 1 of their flexmini switch but its currently not on my network.

But an example of mismatched pvid on a different switch would look like this for example

So see here I have port 7 in vlan 3, and the pvid on port 7 is also 3.. This tells the switch hey any traffic inbound into this port is in vlan 3.

But if was like this where the pvid was 1

Then you could get the sort of traffic your seeing.

[Talon9804]

@johnpoz Thanks again for your help and input!

Wanted to give an update:
I did receive my 16-port Unifi Switch Lite 16, swapped it in and moved some switches around. Doing so, I was able to take out two Ubiquiti Edge Router X's (in VLAN Switch mode) and a simply unmanaged switch. Now the only brand of switch I am using is Unifi switches.

After taking out the Edgerouters, things started to work as expected. I'm very familiar with the Unifi switches, but a little less so with the Edgerouters. Despite my best attempts to set them up properly with the correct VID's and PVID's for my different VLANS, ports, and trunks downstream from my primary switch, I must have still gotten something wrong and been creating some sort of STP issue.

As I said, now that I am using only Unifi switches, things are working as expected, so we seem to be all clear! Thanks again for all of your help and input!