Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Wierd behaviour on VLANs jumping between interfaces.

    Scheduled Pinned Locked Moved L2/Switching/VLANs
    34 Posts 4 Posters 4.7k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A
      antonkristensen
      last edited by

      I have basicly tried about every single configuration that i have in my head and i just am not getting this through.

      I have a pfsense with 2 physical interfaces, a zyxel GS1900-24E and a Unifi AC wireless dude.

      I have configured 3 vlans

      vlan 10: for my normal regular networking and nerding.
      vlan 40: for my guests and visitors and possible hijackers outside of my house
      vlan 50: for all my iot devices

      Now, when i plug the ports into the Unifi AC wireless and assign the vlans everything works just as it is expected, all the vlans are correctly configured and everything comes out just as it should, the firewall rules and interfaces behave in a normal way.

      BUT then i have a ubuntu server that i want to have on all 3 vlans, on a single NIC, a 18.04 one that i have configured with netplan and, to be honest everything seems to work just as it should, i could even manage to get DHCP provision to all the 3 networks.

      I do some curling on each of the interfaces and the UDP packages show up in my firewall logs on the correct interface.

      NOW, as soon as a TCP package is supposed to go trough it gets reverted to the default interface, looking like there is something or someone stripping my packages of their VLAN tag, resulting in the firewall logs showing the TCP request on the wrong interface.

      The unifi does everything correctly, and all the vlans work properly and show up on the correct interface in pfsense.

      The ports on the zyxel are set to PVID 1 and trunked, with 10,40 and 50 tagged, both for the unifi and the server.

      My thoughts are that this is an issue with the server NIC or the zyxel doing something wierd to the packages.

      Now, i know that there are alot of smart people here, many that are waaay more smarter than i am when it comes to these thing, i am doing this to learn and to utilize the devices i have to their extent, so comments like "why are you not just routing the traffic between vlans" and "why are you doing this" or "you have to learn so much more to understand what is going on" or maby "this won't work" are not going to help me and can be left out.

      And "google" has been my main website for the last week trying to figure this out.

      I include a screenshot of the firewall blockings.
      ![alt text](image url)Screenshot 2019-10-28 at 18.51.34.png !

      Perhaps someone has encountered this before and can provide me with some information, where to look, what to make sure is correct and so on.

      JKnottJ 1 Reply Last reply Reply Quote 0
      • JKnottJ
        JKnott @antonkristensen
        last edited by

        @antonkristensen said in Wierd behaviour on VLANs jumping between interfaces.:

        NOW, as soon as a TCP package is supposed to go trough it gets reverted to the default interface, looking like there is something or someone stripping my packages of their VLAN tag, resulting in the firewall logs showing the TCP request on the wrong interface.

        Perhaps you can do some packet captures to see what's actually happening with those tags. PfSense has Packet Capture built in and you can run Wireshark on the Ubuntu box. That will help narrow things down a bit. Also, when you use Packet Capture, you want to download the capture and view it in Wireshark.

        PfSense running on Qotom mini PC
        i5 CPU, 4 GB memory, 32 GB SSD & 4 Intel Gb Ethernet ports.
        UniFi AC-Lite access point

        I haven't lost my mind. It's around here...somewhere...

        A 1 Reply Last reply Reply Quote 0
        • johnpozJ
          johnpoz LAYER 8 Global Moderator
          last edited by johnpoz

          @antonkristensen said in Wierd behaviour on VLANs jumping between interfaces.:

          BUT then i have a ubuntu server that i want to have on all 3 vlans, on a single NIC

          Well that is borked to be honest, and circumvents any firewall rules might put into place..

          The ports on the zyxel are set to PVID 1

          This means any untagged traffic gets put into vlan 1.. So if server puts out a packet that is not tagged, it will get put into vlan 1..

          What exactly are you trying to show with those blocks? That you have a source of this 40.2 IP hitting lan and Guest?

          I take it this 40.2 IP is this server that has an interface with multiple vlans on it?

          An intelligent man is sometimes forced to be drunk to spend time with his fools
          If you get confused: Listen to the Music Play
          Please don't Chat/PM me for help, unless mod related
          SG-4860 24.11 | Lab VMs 2.8, 24.11

          JKnottJ 1 Reply Last reply Reply Quote 0
          • JKnottJ
            JKnott @johnpoz
            last edited by

            @johnpoz said in Wierd behaviour on VLANs jumping between interfaces.:

            BUT then i have a ubuntu server that i want to have on all 3 vlans, on a single NIC

            Well that is borked to be honest, and circumvents any firewall rules might put into place..

            Why do you say that, considering VLANs are logically separate interfaces?

            PfSense running on Qotom mini PC
            i5 CPU, 4 GB memory, 32 GB SSD & 4 Intel Gb Ethernet ports.
            UniFi AC-Lite access point

            I haven't lost my mind. It's around here...somewhere...

            johnpozJ 1 Reply Last reply Reply Quote 0
            • A
              antonkristensen @JKnott
              last edited by

              @JKnott
              I have done some captures both from the server and the pfsense

              The DNS(UDP) packages are coming into and answered with a vlan tag that is correct to their behaviour.
              While the TCP packages are coming in with an vlan tag of 10 and then obviously not answered since they are on the wrong interface

              I have to run through this one more time before i give you the results from the server, they were wierd tbh.

              @johnpoz
              Yes all the ip with .2 are the server

              normal vlan has: 10.0.10.0/24
              Guest vlan has: 10.0.40.0/24
              IOT vlan has: 10.0.50.0/24

              1 Reply Last reply Reply Quote 0
              • A
                antonkristensen
                last edited by

                @JKnott
                I ran some captures on the main interface of the server.

                All the UDP packages get the correct 802.1q vlan ID of 40 (When capturing the guest network) when they are leaving the interface.

                All the TCP packages are somehow changed to 802.1q vlan id 10 when they are leaving the interface, therefore they are getting to the wrong place on the pfsense... holy hell.

                Does anyone know what might be causing this or how it could be fixed.

                1 Reply Last reply Reply Quote 0
                • DerelictD
                  Derelict LAYER 8 Netgate
                  last edited by

                  Show said captures. If it is happening on egress from the server host it is not a pfSense problem. Perhaps you do not have the proper routes on that host that would be required to choose the correct NIC in that multi-NIC setup.

                  Chattanooga, Tennessee, USA
                  A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                  DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                  Do Not Chat For Help! NO_WAN_EGRESS(TM)

                  A 1 Reply Last reply Reply Quote 0
                  • A
                    antonkristensen @Derelict
                    last edited by

                    @Derelict
                    vlan0.40.pcap

                    1 Reply Last reply Reply Quote 0
                    • DerelictD
                      Derelict LAYER 8 Netgate
                      last edited by

                      Look at packet 26 there. It is probably being sent according to the routing table which has the default gateway as something on VLAN 10 but the traffic is sourced from an address on VLAN 40. You have to be sure that all outbound connections are sourced from an address on the same subnet as the gateway. This is why multiple NICs in a host is complicated. You have to know exactly what you are doing. You can use the other NICs to access the subnet the NICs are on, but routing anything is going to get extremely complicated.

                      Chattanooga, Tennessee, USA
                      A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                      DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                      Do Not Chat For Help! NO_WAN_EGRESS(TM)

                      1 Reply Last reply Reply Quote 0
                      • johnpozJ
                        johnpoz LAYER 8 Global Moderator @JKnott
                        last edited by johnpoz

                        @JKnott said in Wierd behaviour on VLANs jumping between interfaces.:

                        Why do you say that, considering VLANs are logically separate interfaces?

                        Not that they are on the same interface, that the server is in more than 1 no matter how.. You just circumvented any firewall rules.. If that server is compromised then access to other vlans is right there, no firewall protecting anything... So you might as well just be in 1 flat network if you going to put all your devices into multiple networks.

                        Unless the box is your router/firewall then it should be in 1 network... If it needs to be in other than those should be special networks like storage or backup for that server...

                        And as Derelict touched on, there shouldn't be any freaking gateways on these other interfaces if your going to multihome a box... It should just use that interface to talk to that network, not be able to route through it.

                        An intelligent man is sometimes forced to be drunk to spend time with his fools
                        If you get confused: Listen to the Music Play
                        Please don't Chat/PM me for help, unless mod related
                        SG-4860 24.11 | Lab VMs 2.8, 24.11

                        A JKnottJ 2 Replies Last reply Reply Quote 0
                        • A
                          antonkristensen @johnpoz
                          last edited by

                          @johnpoz

                          Again, im not putting all the devices on all the networks, just the one server, mainly to learn but also to run some services.

                          It seems as @Derelict perhaps might have somekind of an answer for me:
                          So that would say that i would have to do

                          routes:
                            - to: 0.0.0.0/0
                              via: 10.0.10.1
                              metric: 100
                          

                          Or something in that sense? perhaps what is confusing me is the placement of each configuration on the server.

                          Sorry for those who annoy themselves over me talking about something else than pfsense, i thought originally this was a pfsense issue but it came out to be something else, if you are in the interest of giving me few pointers then i thank you from the depths of my heart.

                          Im using netplan to configure this on a 18.04 server, it amazes me that the UDP packages get assigned the correct tag but not the TCP xD

                          On the physical NIC i havent defined anything, then i have defined vlans linked to the phyNIC and then bridges for each vlan to configure all the other necessary things.

                          Does that sound idiotic ?

                          I have spent too many hours on this ! haha!

                          1 Reply Last reply Reply Quote 0
                          • JKnottJ
                            JKnott @johnpoz
                            last edited by

                            @johnpoz said in Wierd behaviour on VLANs jumping between interfaces.:

                            Not that they are on the same interface, that the server is in more than 1 no matter how.. You just circumvented any firewall rules.. If that server is compromised then access to other vlans is right there, no firewall protecting anything...

                            I read the original post said 3 VLANs from the firewall and then the server also on the 3 VLANs. The server is not the firewall, so the rules are not compromised, provided they're configured properly. I don't agree either with the way he does things, such as why the IoT stuff is connected to a server, but he may have a valid reason for that. Also, VLAN 40 shouldn't be anywhere near the server. It should allow access to the Internet only.

                            Regardless, your comment was "BUT then i have a ubuntu server that i want to have on all 3 vlans, on a single NIC Well that is borked to be honest" It sure sounds like your issue is with 3 VLANs on a single NIC.

                            PfSense running on Qotom mini PC
                            i5 CPU, 4 GB memory, 32 GB SSD & 4 Intel Gb Ethernet ports.
                            UniFi AC-Lite access point

                            I haven't lost my mind. It's around here...somewhere...

                            1 Reply Last reply Reply Quote 0
                            • DerelictD
                              Derelict LAYER 8 Netgate
                              last edited by

                              The source address also has to change.

                              That will be sent to pfSense on VLAN 10.

                              pfSense will have a route back to 10.0.40.2 on VLAN 40.

                              You didn't want to be asked "why" but what are you trying to accomplish besides making your routing unnecessarily complicated?

                              Chattanooga, Tennessee, USA
                              A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                              DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                              Do Not Chat For Help! NO_WAN_EGRESS(TM)

                              A 1 Reply Last reply Reply Quote 0
                              • DerelictD
                                Derelict LAYER 8 Netgate
                                last edited by Derelict

                                @JKnott said in Wierd behaviour on VLANs jumping between interfaces.:

                                It sure sounds like your issue is with 3 VLANs on a single NIC.

                                It's fine, ON A ROUTER. OP has found himself deep in the weeds as is usually the case when these techniques are employed. That and people make all sorts of changes to fix this asymmetric routing when they should just properly-design their network.

                                Chattanooga, Tennessee, USA
                                A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                JKnottJ 1 Reply Last reply Reply Quote 0
                                • JKnottJ
                                  JKnott @Derelict
                                  last edited by

                                  @Derelict said in Wierd behaviour on VLANs jumping between interfaces.:

                                  It's fine, ON A ROUTER.

                                  Have you not heard of multi homed servers? Do you see a significant difference if the multiple networks are connected to the server via multiple NICs or multiple VLANs?

                                  While the OP may have made a mess, I don't see that the mess was caused by VLANs, but rather the inappropriate use of them. An example I mentioned was connecting the guest VLAN to the server, unless there's a need for such.

                                  PfSense running on Qotom mini PC
                                  i5 CPU, 4 GB memory, 32 GB SSD & 4 Intel Gb Ethernet ports.
                                  UniFi AC-Lite access point

                                  I haven't lost my mind. It's around here...somewhere...

                                  DerelictD 1 Reply Last reply Reply Quote 0
                                  • DerelictD
                                    Derelict LAYER 8 Netgate
                                    last edited by

                                    Plenty - as I explained you have to know what you are doing.

                                    Chattanooga, Tennessee, USA
                                    A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                    DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                    Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                    1 Reply Last reply Reply Quote 0
                                    • DerelictD
                                      Derelict LAYER 8 Netgate @JKnott
                                      last edited by

                                      @JKnott said in Wierd behaviour on VLANs jumping between interfaces.:

                                      Do you see a significant difference if the multiple networks are connected to the server via multiple NICs or multiple VLANs?

                                      No difference to me at all. It's all about the routing.

                                      Chattanooga, Tennessee, USA
                                      A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                      DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                      Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                      1 Reply Last reply Reply Quote 0
                                      • A
                                        antonkristensen @Derelict
                                        last edited by

                                        @Derelict
                                        Well i want to have certain services accessible to different devices on all 3 networks, and as i wrote i know i could make it with routing over the firewall but i wanted to try and see if i could manage to get it to work

                                        I tried to assign the vlan10 to the main interface and then only setup vlan 40 and 50 and it just did the same thing except the data got tagged with vlan 40 instead of vlan 10 😰

                                        This is ofcourse not something that is in a production environment so security is not at all my top concern, im mainly fascinated by the technology and want to see if i can manage to get it to work, in a similar way that the unifi access point manages to do things, i know it is different yet the device is somehow taking in all those vlans and separating them in a correct way and delivering what needs to be delivered to the correct places.

                                        Of course people always "should" know what they are doing, but somewhere people have to begin to find that knowledge to be able to utilize it later on.

                                        @JKnott
                                        When you build alot of your own iot devices it can be nice to have a database to store values, thats mainly why the iot would be connected to the server.

                                        1 Reply Last reply Reply Quote 0
                                        • DerelictD
                                          Derelict LAYER 8 Netgate
                                          last edited by

                                          You can do it but you have to understand the routing caveats.

                                          You can't have traffic sourced from a subnet the upstream firewall has a route for via another path.

                                          Chattanooga, Tennessee, USA
                                          A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                          DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                          Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                          1 Reply Last reply Reply Quote 0
                                          • johnpozJ
                                            johnpoz LAYER 8 Global Moderator
                                            last edited by johnpoz

                                            I have to deal with multihomed servers all the time.. Freaking jump boxes, and they have interface in customer network, and interface in our network. Then they have an interface for storage, etc.

                                            Not sure how you do not get about how that can compromise a firewall.

                                            User gains access to server with legs in vlan A and vlan B.. From vlan A - he is not suppose to have access to vlan B, from the firewall point of view vlan A is not allowed to talk to vlan B... But since say I found rdp password to this server with legs in both, I now have direct access to vlan B from the server - which I was not supposed to have access. The "compromised" statement I made.

                                            I have been doing this for many years just like you - that you think its common to have multihomed servers just blows my mind..

                                            I am with Derelict, while yes it is possible, and yes it can be done - you better freaking know what your doing.. Or what do you know you run into shit like the OP is running into ;)

                                            In my above scenario, to talk to this server via our interface you have to auth to even get on that network.. Freaking tics card, etc.. And even when you auth to the network, your only allowed to talk to the specific devices you have access to, and then you have to auth to the server to get in.. But the point is still that once I get access to this server I now have unfettered access to the networks its connected to, or that it knows how to route to.. These server are directly attached to the customer network.. And gives me any access into that network that is on that vlan, and anything past that vlan that their firewall allow. No services listen on this box in the customer network. Its a two way street, if something from the customer leg compromised that server they would have access to our administration vlan.. But there is nothing on that specific vlan other than other jump boxes for the same customer.. So they wouldn't get far, etc.

                                            Multihoming is NOT for the newb to networking... Even in my company I see shit all the time where someone setup a box that isn't suppose to register its name in dns, and then its registering the IPs from the wrong freaking interfaces in dns, etc.

                                            1 server = 1 connection to a network.. If your putting the thing in multiple networks... Stop! Think what your doing, why are you doing it, what steps need to happen to ensure its not going to cause issues Be it access its not suppose to have, be it asymmetrical routing, be it a list of a lot of other things that can go wrong..

                                            An intelligent man is sometimes forced to be drunk to spend time with his fools
                                            If you get confused: Listen to the Music Play
                                            Please don't Chat/PM me for help, unless mod related
                                            SG-4860 24.11 | Lab VMs 2.8, 24.11

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.