Navigation

    Netgate Discussion Forum
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search

    Broadcast packets duplicated across VLAN

    L2/Switching/VLANs
    4
    31
    1065
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • N
      nazuro last edited by nazuro

      Hi, really puzzling issue here I can't get to the bottom of. Any advice much appreciated. Issue shown here on pcap on LAN interface on pfSense:

      Screenshot 2021-07-05 at 19.37.16.png Screenshot 2021-07-05 at 19.38.01.png

      The issue also seems present not just on DHCP broadcasts but any other broadcast too. I am connected on VLAN 11 and milliseconds later the broadcast packet is duplicated to VLAN 13.

      Setup is pfSense with four VLANs on the LAN (igb1) interface:

      MAIN (VLAN11): igb1.1
      IOT (VLAN12): igb1.2
      VPN (VLAN 13): igb1.3
      GUEST (VLAN 14): igb1.4

      LAN is connected to Netgear GS116Ev2 which has 802.1Q VLAN tagging configured in a fairly standard configuration:

      Port 9 is trunk port to pfSense firewall
      Port 11 is trunk from Unifi AP
      Port 16 is for a wired device on VLAN 13
      All other ports for devices on VLAN 11.

      Screenshot 2021-07-05 at 21.02.22.png Screenshot 2021-07-05 at 21.02.30.png

      Any ideas or suggestions for how I can further diagnose much appreciated. Thank you

      JKnott V 2 Replies Last reply Reply Quote 0
      • JKnott
        JKnott @nazuro last edited by

        @nazuro

        If you had a TP-Link switch, I'd say toss it. However, I suspect it may be a configuration issue with the switch. I don't have a Netgear switch, so I can't help you with the details.

        BTW, please attach files to your post instead of using a link. Who knows where that link leads to.

        PfSense running on Qotom mini PC
        i5 CPU, 4 GB memory, 32 GB SSD & 4 Intel Gb Ethernet ports.
        UniFi AC-Lite access point

        I haven't lost my mind. It's around here...somewhere...

        N 1 Reply Last reply Reply Quote 0
        • N
          nazuro @JKnott last edited by

          @jknott Apologies, I have now added the screenshots.

          JKnott 1 Reply Last reply Reply Quote 0
          • JKnott
            JKnott @nazuro last edited by

            @nazuro

            I didn't say screenshot. I said files. For example, you can attach the capture file, which contains a lot more info than a screen capture. Also, one thing you can do in Wireshark to make things a bit easier is add a column for VLAN ID. That way, you can tell it at a glance, rather than hunt for it in the data. That column, like the others, can be enabled or disabled as needed. Another thing which I do, is change the layout, so the 2nd and 3rd panels are across the bottom part of the screen. This results in what I find to be the best display. I adjust the width of the 3rd panel so that has just enough space to hold the hex data, leaving maximum space for panel 2.

            Here's how my Wireshark display looks:

            aa13b6df-b07b-420b-b861-d30bac5bab6b-image.png

            Also, why are you using a VLAN for the main LAN? Normally, that would be native LAN.

            PfSense running on Qotom mini PC
            i5 CPU, 4 GB memory, 32 GB SSD & 4 Intel Gb Ethernet ports.
            UniFi AC-Lite access point

            I haven't lost my mind. It's around here...somewhere...

            N 1 Reply Last reply Reply Quote 0
            • V
              viragomann @nazuro last edited by

              @nazuro said in Broadcast packets duplicated across VLAN:

              Port 11 is trunk from Unifi AP

              So why is this port untagged assigned to VLAN 11?

              1 Reply Last reply Reply Quote 0
              • N
                nazuro @JKnott last edited by

                @jknott Thanks for the tips! I have made those adjustments and it's indeed much easier to follow. Here are the DHCP packets I captured pcap1.pcap

                If it's best practice to use a native LAN for the main network instead of a VLAN then I can certainly give that a go. Kinda makes sense actually.

                @viragomann This is the configuration I was advised as the default network on Unifi cannot be VLAN tagged.

                V JKnott 3 Replies Last reply Reply Quote 0
                • V
                  viragomann @nazuro last edited by

                  @nazuro said in Broadcast packets duplicated across VLAN:

                  This is the configuration I was advised as the default network on Unifi cannot be VLAN tagged.

                  Really? The management IP cannot be on a VLAN?
                  So I'm wondering why so many people are using these crappy Unify parts.

                  Anyway, I'd remove this port from VLAN 11 for testing to rule out the AP is the backdoor into VLAN 13.

                  1 Reply Last reply Reply Quote 0
                  • JKnott
                    JKnott @nazuro last edited by

                    @nazuro

                    I have a Unifi AC-LITE AP with the main network on native and guest WiFi on VLAN3. I configured pfsense, the AP and appropriate switch ports to pass VLAN3 as well as native. AP management is on the native LAN.

                    Sometimes, when you have weird problems, the best thing to do is start from scratch and do things one step at a time.

                    You seem to be having trouble between VLAN11 & VLAN13. VLAN11 is your main system, but VLAN 13 is your VPN. Why is that on a VLAN? That would normally be native traffic on the WAN port. Also, what on 13 is responding to DHCP requests? VPNs normally use TUN, which means they are routed. But DHCP uses broadcasts, which should not be routed.

                    PfSense running on Qotom mini PC
                    i5 CPU, 4 GB memory, 32 GB SSD & 4 Intel Gb Ethernet ports.
                    UniFi AC-Lite access point

                    I haven't lost my mind. It's around here...somewhere...

                    N 1 Reply Last reply Reply Quote 0
                    • JKnott
                      JKnott @nazuro last edited by

                      @nazuro

                      One other Wireshark tip. Some columns appear to run out of space, losing the rightmost info. To fix that, right click on the column title and select Resize Column to Width. This will cause the column to include all the info. This is really handy with IPv6, where the addresses can be really long.

                      PfSense running on Qotom mini PC
                      i5 CPU, 4 GB memory, 32 GB SSD & 4 Intel Gb Ethernet ports.
                      UniFi AC-Lite access point

                      I haven't lost my mind. It's around here...somewhere...

                      1 Reply Last reply Reply Quote 0
                      • N
                        nazuro @JKnott last edited by

                        @jknott Yes really not sure why the issue is only between VLAN 11 & 13. VPN is not natively on the WAN port as I do not want all my traffic going through the VPN. The DHCP server running on the VPN virtual interface is responding to the DHCP requests.

                        I've pretty much based my setup around this guide which I found to be quite comprehensive: https://nguvu.org/pfsense/pfsense-baseline-setup/

                        I'm thinking about adjusting my configuration such that I am using the native LAN for management, however realised that Unifi for some reason only allows me to have four (total) networks. I'd in effect want to have MAIN,IOT,VPN,GUEST,native_management which would be five.

                        Alternatively I could use the native LAN for my "Main" traffic but I can't quite get my head around how this will work in pfSense, given that I have several firewall rules configured between VLANs at the moment and it's quite easy to do as each VLAN is on it's own virtual interface. Not sure how I would enforce rules on the "Main" traffic as I'd have to apply these on the LAN interface which could potentially get in the way of VLAN traffic coming in on LAN. Admittedly I am a bit out of my depth here and potentially talking complete nonsense so am happy to be corrected.

                        Alternatively I do have three spare NIC ports on pfSense so in theory could have a physical interface for each VLAN.

                        Thanks and appreciate your help

                        johnpoz JKnott 2 Replies Last reply Reply Quote 0
                        • johnpoz
                          johnpoz LAYER 8 Global Moderator @nazuro last edited by johnpoz

                          @nazuro said in Broadcast packets duplicated across VLAN:

                          for some reason only allows me to have four (total) networks. I'd in effect want to have MAIN,IOT,VPN,GUEST,native_management which would be five.

                          If you want more than 4, disable link monitoring.. Then you can have up to 8 per radio.. And you can not use wireless uplink.

                          Another option for more than 4 networks is dynamic assigned vlans.

                          An intelligent man is sometimes forced to be drunk to spend time with his fools
                          If you get confused: Listen to the Music Play
                          Please don't Chat/PM me for help, unless mod related
                          SG-4860 23.01 | Lab VMs CE 2.6, 2.7

                          1 Reply Last reply Reply Quote 0
                          • JKnott
                            JKnott @nazuro last edited by

                            @nazuro said in Broadcast packets duplicated across VLAN:

                            VPN is not natively on the WAN port as I do not want all my traffic going through the VPN.

                            That depends on how you configure the VPN, not by using VLANs. The setting in OpenVPN server config is Redirect IPv4 Gateway, which can force all traffic through the VPN. This might be used for those using the Wifi in a coffee shop, etc..

                            I'm thinking more you should start over from scratch.

                            PfSense running on Qotom mini PC
                            i5 CPU, 4 GB memory, 32 GB SSD & 4 Intel Gb Ethernet ports.
                            UniFi AC-Lite access point

                            I haven't lost my mind. It's around here...somewhere...

                            N 1 Reply Last reply Reply Quote 0
                            • N
                              nazuro @JKnott last edited by nazuro

                              @johnpoz Thanks for that, news to me! I made a change and gave myself four VLANs on Unifi and one native LAN. I could then update the switch so all four VLANs were tagged on port 11. However, same issue persists and I also noticed the issue persists on wired connection to the switch (ruling out Unifi issue).

                              @JKnott
                              I am dreading the thought of starting again if I'm honest. Am seemingly so close with the perfect setup for me it's just these damn broadcast packets that seem to get duplicated. Perhaps I have a dodgy switch or NIC on my pfSense box. I just can't think what else could cause these duplicated broadcast packets.

                              Looking in the traffic logs, I also see the odd occasion where the interface is "VLAN" when it should be "MAIN" (as 192.168.11.103 is a device on the MAIN network). So confusing! And throwing off the hypothesis that it's just broadcast traffic being duplicated 😢
                              Screenshot 2021-07-06 at 20.53.04.png

                              JKnott johnpoz 3 Replies Last reply Reply Quote 0
                              • JKnott
                                JKnott @nazuro last edited by

                                @nazuro said in Broadcast packets duplicated across VLAN:

                                I am dreading the thought of starting again if I'm honest. Am seemingly so close with the perfect setup for me it's just these damn broadcast packets that seem to get duplicated. Perhaps I have a dodgy switch or NIC on my pfSense box. I just can't think what else could cause these duplicated broadcast packets.

                                The thing is I don't think you really have it configured correctly. I can see only 2 things that should be VLANs, guest WiFi and IoT. I still have absolutely no idea why your VPN is on a VLAN, when it shouldn't even be on the LAN side at all. A VPN is used to connect computers or networks securely over the Internet and normally terminate on the firewall/router.

                                PfSense running on Qotom mini PC
                                i5 CPU, 4 GB memory, 32 GB SSD & 4 Intel Gb Ethernet ports.
                                UniFi AC-Lite access point

                                I haven't lost my mind. It's around here...somewhere...

                                1 Reply Last reply Reply Quote 0
                                • johnpoz
                                  johnpoz LAYER 8 Global Moderator @nazuro last edited by johnpoz

                                  So yeah that doesn't make any sense.. What is this 13.147 address?

                                  And this 11.103 is no way its multihomed? And or could tag traffic? Is it a wired device or wireless?

                                  Pfsense shouldn't ever see that traffic unless sent to its mac (its the gateway).. Unless it was broadcast..

                                  can you get a sniff of this traffic? On pfsense interface and post the pcap..

                                  Those firewall entries are 9 seconds apart, so its not like a duplication of traffic.. It would have to be sent separately. I can't see how something duplicating traffic, or any sort of loop, etc. showing the hits 9 seconds apart..

                                  So to me - these were actually sent by the .103 device to 2 different gateway. Too the vpn pfsense IP, and the main pfsense IP.. How these device could talk to these different pfsense IPs when they are in different vlans makes no sense - if the vlans are correctly isolated and being tagged correctly..

                                  edit: To his vpn vlan, I take it that is just what he is calling this vlan, because he wants to route this out his vpn connection.. That is my take on that name.

                                  edit2: If its a wireless device this 103 device - could it be switching wifi ssids? That could explain sending traffic to vpn vlan gateway on vlan X, and then few seconds latter sending same traffic to vlan Y gateway.

                                  An intelligent man is sometimes forced to be drunk to spend time with his fools
                                  If you get confused: Listen to the Music Play
                                  Please don't Chat/PM me for help, unless mod related
                                  SG-4860 23.01 | Lab VMs CE 2.6, 2.7

                                  1 Reply Last reply Reply Quote 0
                                  • JKnott
                                    JKnott @nazuro last edited by

                                    @nazuro

                                    Further on this. Several years ago, I had one job that had 3 VLANs. It was a senior citizens residence, occupying 3 towers. In addition to the main native LAN for the office, there were VLANs for VoIP, network management and Internet to the resident's rooms. That is the only time I've had that many VLANs and I don't understand why you need 4.

                                    PfSense running on Qotom mini PC
                                    i5 CPU, 4 GB memory, 32 GB SSD & 4 Intel Gb Ethernet ports.
                                    UniFi AC-Lite access point

                                    I haven't lost my mind. It's around here...somewhere...

                                    johnpoz 1 Reply Last reply Reply Quote 0
                                    • johnpoz
                                      johnpoz LAYER 8 Global Moderator @JKnott last edited by johnpoz

                                      @jknott said in Broadcast packets duplicated across VLAN:

                                      why you need 4.

                                      You don't understand why someone might want to isolate wifi to more than 4 vlans? Come on!

                                      I have 4, and really could use more.. My main trusted vlan, my psk vlan, my roku vlan, and then a guest vlan.

                                      I could see busting out a few more.. I have a few different devices sharing my psk vlan.. I could see breaking those out to different vlans for different devices, etc. For example when I setup some camera's I would want those on their own camera vlan. And I should really break out all my lightbulbs to their own vs being on the same vlan as my other iot devices.. Just lazy and let them share the same vlan..

                                      edit: Thinking about this for a few minutes - my "guess" is this 103 device is wireless and its changing the ssid its connected to. Say for example when it cant get to that 13.147 address via the vpn ssid, it jumps to another ssid to try and get there. That would explain the 9 second span between firewall hits.

                                      An intelligent man is sometimes forced to be drunk to spend time with his fools
                                      If you get confused: Listen to the Music Play
                                      Please don't Chat/PM me for help, unless mod related
                                      SG-4860 23.01 | Lab VMs CE 2.6, 2.7

                                      JKnott 1 Reply Last reply Reply Quote 0
                                      • JKnott
                                        JKnott @johnpoz last edited by

                                        @johnpoz

                                        I can understand why there might be a need for more VLANs. What I don't understand is why he needs 4, based on what he's mentioned. For example, what's with the VPN on a VLAN? Where does the VLAN terminate? If other than on pfsense, that complicates routing. As discussed, the main LAN also shouldn't be on a VLAN. That leaves 2. Sure, I could put my TV on it's own VLAN, but the guest WiFi is good enough for that. Even then, the only reason the TV isn't on my main LAN is it chokes on the 63 character password I use on it. I suppose I could put my IPTV boxes on a separate VLAN, but haven't bothered.

                                        PfSense running on Qotom mini PC
                                        i5 CPU, 4 GB memory, 32 GB SSD & 4 Intel Gb Ethernet ports.
                                        UniFi AC-Lite access point

                                        I haven't lost my mind. It's around here...somewhere...

                                        johnpoz 1 Reply Last reply Reply Quote 0
                                        • johnpoz
                                          johnpoz LAYER 8 Global Moderator @JKnott last edited by

                                          @jknott said in Broadcast packets duplicated across VLAN:

                                          what's with the VPN on a VLAN?

                                          My take and have seen many users do this.. They create a vlan, that all devices on this vlan are routed out a vpn connection. So for example if he wants to use the vpn on his phone for "something" he just changes the ssid he is connected to. Maybe netflix doesn't work via the vpn, so when he wants to watch netflix on is phone he changes to different ssid that is not routed out the vpn.

                                          An intelligent man is sometimes forced to be drunk to spend time with his fools
                                          If you get confused: Listen to the Music Play
                                          Please don't Chat/PM me for help, unless mod related
                                          SG-4860 23.01 | Lab VMs CE 2.6, 2.7

                                          JKnott 1 Reply Last reply Reply Quote 1
                                          • JKnott
                                            JKnott @johnpoz last edited by

                                            @johnpoz

                                            I suppose I could give my dog & cat their own VLANs. 😉

                                            PfSense running on Qotom mini PC
                                            i5 CPU, 4 GB memory, 32 GB SSD & 4 Intel Gb Ethernet ports.
                                            UniFi AC-Lite access point

                                            I haven't lost my mind. It's around here...somewhere...

                                            johnpoz 1 Reply Last reply Reply Quote 0
                                            • johnpoz
                                              johnpoz LAYER 8 Global Moderator @JKnott last edited by

                                              @jknott clearly - they should not be on the same ;) that is for sure - they would just fight ;)

                                              An intelligent man is sometimes forced to be drunk to spend time with his fools
                                              If you get confused: Listen to the Music Play
                                              Please don't Chat/PM me for help, unless mod related
                                              SG-4860 23.01 | Lab VMs CE 2.6, 2.7

                                              1 Reply Last reply Reply Quote 0
                                              • N
                                                nazuro last edited by nazuro

                                                @jknott @johnpoz

                                                Apologies for any confusion. John is correct that VPN is just what I'm calling my VLAN. It enables me to connect physical / wireless devices to it and that traffic then exits via OpenVPN to the internet via an AirVPN endpoint.

                                                My 11.103 address is MacBook Pro wireless. 13.147 is my iPad. I did just have a particularly bad bout of wireless problems and observed this in the logs. I don't believe my MacBook is switching SSIDs and did previously confirm this by forgetting all SSIDs apart from the Main.

                                                Screenshot 2021-07-06 at 21.30.27.png
                                                I have followed this guide pretty closely https://nguvu.org/pfsense/pfsense-baseline-setup/. Perhaps I am out of my depth - I am working through CISSP at the moment but still much different in practice than in theory. I found it easier conceptually to leave the parent interface (LAN) as just that, and have four different VLANs that I would use operationally and could easily apply firewall rules to each of the virtual interfaces without interfering with the parent interface.

                                                Things do seem quite a bit worse now since I tweaked the Unifi settings and minor changes on the switch. I'm going to try put everything back to how it was, and do more analysis specifically on what I'm seeing with these firewall logs and get another pcap. Appreciate both of your help very much so.

                                                edit: Perhaps if there's some way of me locally tagging traffic from my MacBook on VLAN 11 I could plug directly in to the LAN on pfSense and rule out an issue with the switch.

                                                N 1 Reply Last reply Reply Quote 0
                                                • N
                                                  nazuro @nazuro last edited by

                                                  I've just taken a short pcap on LAN port on pfSense. 192.168.11.103 is still my MacBook and I renewed the DHCP lease twice during the pcap pcap7.pcap

                                                  johnpoz 1 Reply Last reply Reply Quote 0
                                                  • johnpoz
                                                    johnpoz LAYER 8 Global Moderator @nazuro last edited by johnpoz

                                                    And here you go.. Here is arp announcement from your apple - tagged with both 11 and 13

                                                    arp.png

                                                    Here is your dhcp also 1 tagged 11, other 13

                                                    dhcp.png

                                                    So yes pfsense would see this on both of its vlan interfaces, because the traffic tagged. So either your apple is tagging this traffic, or your wifi is doing it.. But that explains why pfsense is seeing the traffic on both interfaces - because its tagged for both vlans

                                                    An intelligent man is sometimes forced to be drunk to spend time with his fools
                                                    If you get confused: Listen to the Music Play
                                                    Please don't Chat/PM me for help, unless mod related
                                                    SG-4860 23.01 | Lab VMs CE 2.6, 2.7

                                                    N 1 Reply Last reply Reply Quote 0
                                                    • N
                                                      nazuro @johnpoz last edited by

                                                      @johnpoz Understood. Problem is it's not just my Apple but also other devices (wired and wireless) including windows too. Must be something happening on the Netgear switch then I suppose, although it's a fairly standard 802.1Q setup I have got going on.

                                                      johnpoz 1 Reply Last reply Reply Quote 0
                                                      • johnpoz
                                                        johnpoz LAYER 8 Global Moderator @nazuro last edited by johnpoz

                                                        What wired device in that sniff? So I can look for those.

                                                        Nothing else jumped out at me..

                                                        This is ODD.. you have this unifi mac asking for who is 11.1 with 11 and 13 tagged

                                                        odd.png

                                                        what is 11.9?

                                                        An intelligent man is sometimes forced to be drunk to spend time with his fools
                                                        If you get confused: Listen to the Music Play
                                                        Please don't Chat/PM me for help, unless mod related
                                                        SG-4860 23.01 | Lab VMs CE 2.6, 2.7

                                                        N 1 Reply Last reply Reply Quote 0
                                                        • N
                                                          nazuro @johnpoz last edited by nazuro

                                                          @johnpoz 192.168.11.117 in this pcap is wired in to the switch on an Untagged VLAN 11 port (wireless on this laptop is turned off) pcap8.pcap

                                                          edit: 11.9 is the IP address of the Unifi AP

                                                          johnpoz 1 Reply Last reply Reply Quote 0
                                                          • johnpoz
                                                            johnpoz LAYER 8 Global Moderator @nazuro last edited by johnpoz

                                                            who is this

                                                            whoisthis.png

                                                            asking for 13.100, from 13.1 in both 11 and 13..

                                                            Something for sure is all messed up..

                                                            An intelligent man is sometimes forced to be drunk to spend time with his fools
                                                            If you get confused: Listen to the Music Play
                                                            Please don't Chat/PM me for help, unless mod related
                                                            SG-4860 23.01 | Lab VMs CE 2.6, 2.7

                                                            N 1 Reply Last reply Reply Quote 0
                                                            • N
                                                              nazuro @johnpoz last edited by

                                                              @johnpoz 00:1b:21:33:9e:e9 is the MAC address of the LAN NIC (igb1) in pfSense. Therefore igb1.1, igb1.2, igb1.3, igb1.4 also share the same MAC address

                                                              N 1 Reply Last reply Reply Quote 0
                                                              • N
                                                                nazuro @nazuro last edited by nazuro

                                                                This post is deleted!
                                                                1 Reply Last reply Reply Quote 0
                                                                • N
                                                                  nazuro last edited by nazuro

                                                                  @johnpoz said in Broadcast packets duplicated across VLAN:

                                                                  asking for 13.100, from 13.1 in both 11 and 13..
                                                                  Something for sure is all messed up..

                                                                  In case you are interested, I contacted Netgear support and they say it's an issue with my switch:

                                                                  "As I have this inquired to my senior experts, seems like the behavior of GS116ev2 Plus Switch is causing the issue for the certain VLAN. Since GS116Ev2 does not have native VLAN nor management VLAN ID, any DHCP request is being sent to all ports. As advised and to have this be corrected, Smart Pro switches are recommended."

                                                                  1 Reply Last reply Reply Quote 0
                                                                  • First post
                                                                    Last post