Broadcast packets duplicated across VLAN

[nazuro]

Hi, really puzzling issue here I can't get to the bottom of. Any advice much appreciated. Issue shown here on pcap on LAN interface on pfSense:

The issue also seems present not just on DHCP broadcasts but any other broadcast too. I am connected on VLAN 11 and milliseconds later the broadcast packet is duplicated to VLAN 13.

Setup is pfSense with four VLANs on the LAN (igb1) interface:

MAIN (VLAN11): igb1.1
IOT (VLAN12): igb1.2
VPN (VLAN 13): igb1.3
GUEST (VLAN 14): igb1.4

LAN is connected to Netgear GS116Ev2 which has 802.1Q VLAN tagging configured in a fairly standard configuration:

Port 9 is trunk port to pfSense firewall
Port 11 is trunk from Unifi AP
Port 16 is for a wired device on VLAN 13
All other ports for devices on VLAN 11.

Any ideas or suggestions for how I can further diagnose much appreciated. Thank you

[JKnott]

@nazuro

If you had a TP-Link switch, I'd say toss it. However, I suspect it may be a configuration issue with the switch. I don't have a Netgear switch, so I can't help you with the details.

BTW, please attach files to your post instead of using a link. Who knows where that link leads to.

[nazuro]

@jknott Apologies, I have now added the screenshots.

[JKnott]

@nazuro

I didn't say screenshot. I said files. For example, you can attach the capture file, which contains a lot more info than a screen capture. Also, one thing you can do in Wireshark to make things a bit easier is add a column for VLAN ID. That way, you can tell it at a glance, rather than hunt for it in the data. That column, like the others, can be enabled or disabled as needed. Another thing which I do, is change the layout, so the 2nd and 3rd panels are across the bottom part of the screen. This results in what I find to be the best display. I adjust the width of the 3rd panel so that has just enough space to hold the hex data, leaving maximum space for panel 2.

Here's how my Wireshark display looks:

Also, why are you using a VLAN for the main LAN? Normally, that would be native LAN.

[viragomann]

@nazuro said in Broadcast packets duplicated across VLAN:

Port 11 is trunk from Unifi AP

So why is this port untagged assigned to VLAN 11?

[nazuro]

@jknott Thanks for the tips! I have made those adjustments and it's indeed much easier to follow. Here are the DHCP packets I captured pcap1.pcap

If it's best practice to use a native LAN for the main network instead of a VLAN then I can certainly give that a go. Kinda makes sense actually.

@viragomann This is the configuration I was advised as the default network on Unifi cannot be VLAN tagged.

[viragomann]

@nazuro said in Broadcast packets duplicated across VLAN:

This is the configuration I was advised as the default network on Unifi cannot be VLAN tagged.

Really? The management IP cannot be on a VLAN?
So I'm wondering why so many people are using these crappy Unify parts.

Anyway, I'd remove this port from VLAN 11 for testing to rule out the AP is the backdoor into VLAN 13.

[JKnott]

@nazuro

I have a Unifi AC-LITE AP with the main network on native and guest WiFi on VLAN3. I configured pfsense, the AP and appropriate switch ports to pass VLAN3 as well as native. AP management is on the native LAN.

Sometimes, when you have weird problems, the best thing to do is start from scratch and do things one step at a time.

You seem to be having trouble between VLAN11 & VLAN13. VLAN11 is your main system, but VLAN 13 is your VPN. Why is that on a VLAN? That would normally be native traffic on the WAN port. Also, what on 13 is responding to DHCP requests? VPNs normally use TUN, which means they are routed. But DHCP uses broadcasts, which should not be routed.

[JKnott]

@nazuro

One other Wireshark tip. Some columns appear to run out of space, losing the rightmost info. To fix that, right click on the column title and select Resize Column to Width. This will cause the column to include all the info. This is really handy with IPv6, where the addresses can be really long.

[nazuro]

@jknott Yes really not sure why the issue is only between VLAN 11 & 13. VPN is not natively on the WAN port as I do not want all my traffic going through the VPN. The DHCP server running on the VPN virtual interface is responding to the DHCP requests.

I've pretty much based my setup around this guide which I found to be quite comprehensive: https://nguvu.org/pfsense/pfsense-baseline-setup/

I'm thinking about adjusting my configuration such that I am using the native LAN for management, however realised that Unifi for some reason only allows me to have four (total) networks. I'd in effect want to have MAIN,IOT,VPN,GUEST,native_management which would be five.

Alternatively I could use the native LAN for my "Main" traffic but I can't quite get my head around how this will work in pfSense, given that I have several firewall rules configured between VLANs at the moment and it's quite easy to do as each VLAN is on it's own virtual interface. Not sure how I would enforce rules on the "Main" traffic as I'd have to apply these on the LAN interface which could potentially get in the way of VLAN traffic coming in on LAN. Admittedly I am a bit out of my depth here and potentially talking complete nonsense so am happy to be corrected.

Alternatively I do have three spare NIC ports on pfSense so in theory could have a physical interface for each VLAN.

Thanks and appreciate your help

[johnpoz]

@nazuro said in Broadcast packets duplicated across VLAN:

for some reason only allows me to have four (total) networks. I'd in effect want to have MAIN,IOT,VPN,GUEST,native_management which would be five.

If you want more than 4, disable link monitoring.. Then you can have up to 8 per radio.. And you can not use wireless uplink.

Another option for more than 4 networks is dynamic assigned vlans.

[JKnott]

@nazuro said in Broadcast packets duplicated across VLAN:

VPN is not natively on the WAN port as I do not want all my traffic going through the VPN.

That depends on how you configure the VPN, not by using VLANs. The setting in OpenVPN server config is Redirect IPv4 Gateway, which can force all traffic through the VPN. This might be used for those using the Wifi in a coffee shop, etc..

I'm thinking more you should start over from scratch.

[nazuro]

@johnpoz Thanks for that, news to me! I made a change and gave myself four VLANs on Unifi and one native LAN. I could then update the switch so all four VLANs were tagged on port 11. However, same issue persists and I also noticed the issue persists on wired connection to the switch (ruling out Unifi issue).

@JKnott
I am dreading the thought of starting again if I'm honest. Am seemingly so close with the perfect setup for me it's just these damn broadcast packets that seem to get duplicated. Perhaps I have a dodgy switch or NIC on my pfSense box. I just can't think what else could cause these duplicated broadcast packets.

Looking in the traffic logs, I also see the odd occasion where the interface is "VLAN" when it should be "MAIN" (as 192.168.11.103 is a device on the MAIN network). So confusing! And throwing off the hypothesis that it's just broadcast traffic being duplicated

[JKnott]

@nazuro said in Broadcast packets duplicated across VLAN:

I am dreading the thought of starting again if I'm honest. Am seemingly so close with the perfect setup for me it's just these damn broadcast packets that seem to get duplicated. Perhaps I have a dodgy switch or NIC on my pfSense box. I just can't think what else could cause these duplicated broadcast packets.

The thing is I don't think you really have it configured correctly. I can see only 2 things that should be VLANs, guest WiFi and IoT. I still have absolutely no idea why your VPN is on a VLAN, when it shouldn't even be on the LAN side at all. A VPN is used to connect computers or networks securely over the Internet and normally terminate on the firewall/router.

[johnpoz]

So yeah that doesn't make any sense.. What is this 13.147 address?

And this 11.103 is no way its multihomed? And or could tag traffic? Is it a wired device or wireless?

Pfsense shouldn't ever see that traffic unless sent to its mac (its the gateway).. Unless it was broadcast..

can you get a sniff of this traffic? On pfsense interface and post the pcap..

Those firewall entries are 9 seconds apart, so its not like a duplication of traffic.. It would have to be sent separately. I can't see how something duplicating traffic, or any sort of loop, etc. showing the hits 9 seconds apart..

So to me - these were actually sent by the .103 device to 2 different gateway. Too the vpn pfsense IP, and the main pfsense IP.. How these device could talk to these different pfsense IPs when they are in different vlans makes no sense - if the vlans are correctly isolated and being tagged correctly..

edit: To his vpn vlan, I take it that is just what he is calling this vlan, because he wants to route this out his vpn connection.. That is my take on that name.

edit2: If its a wireless device this 103 device - could it be switching wifi ssids? That could explain sending traffic to vpn vlan gateway on vlan X, and then few seconds latter sending same traffic to vlan Y gateway.

[JKnott]

@nazuro

Further on this. Several years ago, I had one job that had 3 VLANs. It was a senior citizens residence, occupying 3 towers. In addition to the main native LAN for the office, there were VLANs for VoIP, network management and Internet to the resident's rooms. That is the only time I've had that many VLANs and I don't understand why you need 4.

[johnpoz]

@jknott said in Broadcast packets duplicated across VLAN:

why you need 4.

You don't understand why someone might want to isolate wifi to more than 4 vlans? Come on!

I have 4, and really could use more.. My main trusted vlan, my psk vlan, my roku vlan, and then a guest vlan.

I could see busting out a few more.. I have a few different devices sharing my psk vlan.. I could see breaking those out to different vlans for different devices, etc. For example when I setup some camera's I would want those on their own camera vlan. And I should really break out all my lightbulbs to their own vs being on the same vlan as my other iot devices.. Just lazy and let them share the same vlan..

edit: Thinking about this for a few minutes - my "guess" is this 103 device is wireless and its changing the ssid its connected to. Say for example when it cant get to that 13.147 address via the vpn ssid, it jumps to another ssid to try and get there. That would explain the 9 second span between firewall hits.

[JKnott]

@johnpoz

I can understand why there might be a need for more VLANs. What I don't understand is why he needs 4, based on what he's mentioned. For example, what's with the VPN on a VLAN? Where does the VLAN terminate? If other than on pfsense, that complicates routing. As discussed, the main LAN also shouldn't be on a VLAN. That leaves 2. Sure, I could put my TV on it's own VLAN, but the guest WiFi is good enough for that. Even then, the only reason the TV isn't on my main LAN is it chokes on the 63 character password I use on it. I suppose I could put my IPTV boxes on a separate VLAN, but haven't bothered.

[johnpoz]

@jknott said in Broadcast packets duplicated across VLAN:

what's with the VPN on a VLAN?

My take and have seen many users do this.. They create a vlan, that all devices on this vlan are routed out a vpn connection. So for example if he wants to use the vpn on his phone for "something" he just changes the ssid he is connected to. Maybe netflix doesn't work via the vpn, so when he wants to watch netflix on is phone he changes to different ssid that is not routed out the vpn.

[JKnott]

@johnpoz

I suppose I could give my dog & cat their own VLANs.

[johnpoz]

@jknott clearly - they should not be on the same ;) that is for sure - they would just fight ;)

[nazuro]

@jknott @johnpoz

Apologies for any confusion. John is correct that VPN is just what I'm calling my VLAN. It enables me to connect physical / wireless devices to it and that traffic then exits via OpenVPN to the internet via an AirVPN endpoint.

My 11.103 address is MacBook Pro wireless. 13.147 is my iPad. I did just have a particularly bad bout of wireless problems and observed this in the logs. I don't believe my MacBook is switching SSIDs and did previously confirm this by forgetting all SSIDs apart from the Main.

I have followed this guide pretty closely https://nguvu.org/pfsense/pfsense-baseline-setup/. Perhaps I am out of my depth - I am working through CISSP at the moment but still much different in practice than in theory. I found it easier conceptually to leave the parent interface (LAN) as just that, and have four different VLANs that I would use operationally and could easily apply firewall rules to each of the virtual interfaces without interfering with the parent interface.

Things do seem quite a bit worse now since I tweaked the Unifi settings and minor changes on the switch. I'm going to try put everything back to how it was, and do more analysis specifically on what I'm seeing with these firewall logs and get another pcap. Appreciate both of your help very much so.

edit: Perhaps if there's some way of me locally tagging traffic from my MacBook on VLAN 11 I could plug directly in to the LAN on pfSense and rule out an issue with the switch.

[nazuro]

I've just taken a short pcap on LAN port on pfSense. 192.168.11.103 is still my MacBook and I renewed the DHCP lease twice during the pcap pcap7.pcap

[johnpoz]

And here you go.. Here is arp announcement from your apple - tagged with both 11 and 13

Here is your dhcp also 1 tagged 11, other 13

So yes pfsense would see this on both of its vlan interfaces, because the traffic tagged. So either your apple is tagging this traffic, or your wifi is doing it.. But that explains why pfsense is seeing the traffic on both interfaces - because its tagged for both vlans

[nazuro]

@johnpoz Understood. Problem is it's not just my Apple but also other devices (wired and wireless) including windows too. Must be something happening on the Netgear switch then I suppose, although it's a fairly standard 802.1Q setup I have got going on.

[johnpoz]

What wired device in that sniff? So I can look for those.

Nothing else jumped out at me..

This is ODD.. you have this unifi mac asking for who is 11.1 with 11 and 13 tagged

what is 11.9?

[nazuro]

@johnpoz 192.168.11.117 in this pcap is wired in to the switch on an Untagged VLAN 11 port (wireless on this laptop is turned off) pcap8.pcap

edit: 11.9 is the IP address of the Unifi AP

[johnpoz]

who is this

asking for 13.100, from 13.1 in both 11 and 13..

Something for sure is all messed up..

[nazuro]

@johnpoz 00:1b:21:33:9e:e9 is the MAC address of the LAN NIC (igb1) in pfSense. Therefore igb1.1, igb1.2, igb1.3, igb1.4 also share the same MAC address

[nazuro]

This post is deleted!

[nazuro]

@johnpoz said in Broadcast packets duplicated across VLAN:

asking for 13.100, from 13.1 in both 11 and 13..
Something for sure is all messed up..

In case you are interested, I contacted Netgear support and they say it's an issue with my switch:

"As I have this inquired to my senior experts, seems like the behavior of GS116ev2 Plus Switch is causing the issue for the certain VLAN. Since GS116Ev2 does not have native VLAN nor management VLAN ID, any DHCP request is being sent to all ports. As advised and to have this be corrected, Smart Pro switches are recommended."