Wierd behaviour on VLANs jumping between interfaces.

johnpoz

@JKnott said in Wierd behaviour on VLANs jumping between interfaces.:

Why do you say that, considering VLANs are logically separate interfaces?

Not that they are on the same interface, that the server is in more than 1 no matter how.. You just circumvented any firewall rules.. If that server is compromised then access to other vlans is right there, no firewall protecting anything... So you might as well just be in 1 flat network if you going to put all your devices into multiple networks.

Unless the box is your router/firewall then it should be in 1 network... If it needs to be in other than those should be special networks like storage or backup for that server...

And as Derelict touched on, there shouldn't be any freaking gateways on these other interfaces if your going to multihome a box... It should just use that interface to talk to that network, not be able to route through it.

antonkristensen

@johnpoz

Again, im not putting all the devices on all the networks, just the one server, mainly to learn but also to run some services.

It seems as @Derelict perhaps might have somekind of an answer for me:
So that would say that i would have to do

routes:
  - to: 0.0.0.0/0
    via: 10.0.10.1
    metric: 100

Or something in that sense? perhaps what is confusing me is the placement of each configuration on the server.

Sorry for those who annoy themselves over me talking about something else than pfsense, i thought originally this was a pfsense issue but it came out to be something else, if you are in the interest of giving me few pointers then i thank you from the depths of my heart.

Im using netplan to configure this on a 18.04 server, it amazes me that the UDP packages get assigned the correct tag but not the TCP xD

On the physical NIC i havent defined anything, then i have defined vlans linked to the phyNIC and then bridges for each vlan to configure all the other necessary things.

Does that sound idiotic ?

I have spent too many hours on this ! haha!

JKnott

@johnpoz said in Wierd behaviour on VLANs jumping between interfaces.:

Not that they are on the same interface, that the server is in more than 1 no matter how.. You just circumvented any firewall rules.. If that server is compromised then access to other vlans is right there, no firewall protecting anything...

I read the original post said 3 VLANs from the firewall and then the server also on the 3 VLANs. The server is not the firewall, so the rules are not compromised, provided they're configured properly. I don't agree either with the way he does things, such as why the IoT stuff is connected to a server, but he may have a valid reason for that. Also, VLAN 40 shouldn't be anywhere near the server. It should allow access to the Internet only.

Regardless, your comment was "BUT then i have a ubuntu server that i want to have on all 3 vlans, on a single NIC Well that is borked to be honest" It sure sounds like your issue is with 3 VLANs on a single NIC.

Derelict

The source address also has to change.

That will be sent to pfSense on VLAN 10.

pfSense will have a route back to 10.0.40.2 on VLAN 40.

You didn't want to be asked "why" but what are you trying to accomplish besides making your routing unnecessarily complicated?

Derelict

@JKnott said in Wierd behaviour on VLANs jumping between interfaces.:

It sure sounds like your issue is with 3 VLANs on a single NIC.

It's fine, ON A ROUTER. OP has found himself deep in the weeds as is usually the case when these techniques are employed. That and people make all sorts of changes to fix this asymmetric routing when they should just properly-design their network.

JKnott

@Derelict said in Wierd behaviour on VLANs jumping between interfaces.:

It's fine, ON A ROUTER.

Have you not heard of multi homed servers? Do you see a significant difference if the multiple networks are connected to the server via multiple NICs or multiple VLANs?

While the OP may have made a mess, I don't see that the mess was caused by VLANs, but rather the inappropriate use of them. An example I mentioned was connecting the guest VLAN to the server, unless there's a need for such.

Derelict

Plenty - as I explained you have to know what you are doing.

Derelict

@JKnott said in Wierd behaviour on VLANs jumping between interfaces.:

Do you see a significant difference if the multiple networks are connected to the server via multiple NICs or multiple VLANs?

No difference to me at all. It's all about the routing.

antonkristensen

@Derelict
Well i want to have certain services accessible to different devices on all 3 networks, and as i wrote i know i could make it with routing over the firewall but i wanted to try and see if i could manage to get it to work

I tried to assign the vlan10 to the main interface and then only setup vlan 40 and 50 and it just did the same thing except the data got tagged with vlan 40 instead of vlan 10

This is ofcourse not something that is in a production environment so security is not at all my top concern, im mainly fascinated by the technology and want to see if i can manage to get it to work, in a similar way that the unifi access point manages to do things, i know it is different yet the device is somehow taking in all those vlans and separating them in a correct way and delivering what needs to be delivered to the correct places.

Of course people always "should" know what they are doing, but somewhere people have to begin to find that knowledge to be able to utilize it later on.

@JKnott
When you build alot of your own iot devices it can be nice to have a database to store values, thats mainly why the iot would be connected to the server.

Derelict

You can do it but you have to understand the routing caveats.

You can't have traffic sourced from a subnet the upstream firewall has a route for via another path.

johnpoz

I have to deal with multihomed servers all the time.. Freaking jump boxes, and they have interface in customer network, and interface in our network. Then they have an interface for storage, etc.

Not sure how you do not get about how that can compromise a firewall.

User gains access to server with legs in vlan A and vlan B.. From vlan A - he is not suppose to have access to vlan B, from the firewall point of view vlan A is not allowed to talk to vlan B... But since say I found rdp password to this server with legs in both, I now have direct access to vlan B from the server - which I was not supposed to have access. The "compromised" statement I made.

I have been doing this for many years just like you - that you think its common to have multihomed servers just blows my mind..

I am with Derelict, while yes it is possible, and yes it can be done - you better freaking know what your doing.. Or what do you know you run into shit like the OP is running into ;)

In my above scenario, to talk to this server via our interface you have to auth to even get on that network.. Freaking tics card, etc.. And even when you auth to the network, your only allowed to talk to the specific devices you have access to, and then you have to auth to the server to get in.. But the point is still that once I get access to this server I now have unfettered access to the networks its connected to, or that it knows how to route to.. These server are directly attached to the customer network.. And gives me any access into that network that is on that vlan, and anything past that vlan that their firewall allow. No services listen on this box in the customer network. Its a two way street, if something from the customer leg compromised that server they would have access to our administration vlan.. But there is nothing on that specific vlan other than other jump boxes for the same customer.. So they wouldn't get far, etc.

Multihoming is NOT for the newb to networking... Even in my company I see shit all the time where someone setup a box that isn't suppose to register its name in dns, and then its registering the IPs from the wrong freaking interfaces in dns, etc.

1 server = 1 connection to a network.. If your putting the thing in multiple networks... Stop! Think what your doing, why are you doing it, what steps need to happen to ensure its not going to cause issues Be it access its not suppose to have, be it asymmetrical routing, be it a list of a lot of other things that can go wrong..

antonkristensen

Would the routes be defined on the main physical NIC or would it be defined on each bridge/vlan nic?

johnpoz

Routes are not defined on a nic, they are defined in the OS!!

The OS determines routing, a nic is only a interface into a specific network - or a way to talk to something else that network is attached to to get to a different network.. Ie route.

antonkristensen

@johnpoz
lol yeah that is what im working with, im not standing above the physical network port and trying to push some actual routes on it.

but i have to define each nic on the server, derived from the physical nic

this is what i am working with(if there is any interest):

network:
  version: 2
  renderer: networkd
  ethernets:
    enp4s0:
      match:
       macaddress: 20:cf:30:c1:c2:55
      dhcp4: no
      dhcp6: no
      accept-ra: no
  vlans:
    vlan10:
      accept-ra: no
      id: 10
      link: enp4s0
    vlan40:
      accept-ra: no
      id: 40
      link: enp4s0
      optional: yes
    #vlan50:
      #accept-ra: no
      #id: 50
      #link: enp4s0
      #optional: yes
  bridges:
    br10:
      dhcp4: no
      dhcp6: no
      accept-ra: no
      interfaces:
        - vlan10
      addresses: [10.0.10.2/24]
      nameservers:
        addresses: [10.0.10.1]
      gateway4: 10.0.10.1
      routes:
         - to: 10.0.10.0/24
           via: 10.0.10.1
           table: 102
         - to: 0.0.0.0/0
           via: 10.0.10.1
           table: 102
      routing-policy:
         - from: 10.0.10.1
           table: 102
         - to: 10.0.10.1
           table: 102
    br40:
      dhcp4: no
      dhcp6: no
      accept-ra: no
      interfaces:
        - vlan40
      addresses: [10.0.40.2/24]
      nameservers:
        addresses: [10.0.40.1]
      gateway4: 10.0.40.1
      routes:
         - to: 10.0.40.0/24
           via: 10.0.40.1
           table: 102
         - to: 0.0.0.0/0
           via: 10.0.40.1
           table: 102
      routing-policy:
         - from: 10.0.40.1
           table: 102
         - to: 10.0.40.1
           table: 102
    #br50:
     # dhcp4: yes
      #dhcp6: no
      #accept-ra: no
      #interfaces:
       # - vlan50
      #addresses: [10.0.50.2/24]
      #nameservers:
       # addresses: [10.0.50.1]
      #gateway4: 10.0.50.1

and yes you can tell me if im way off or if im on the right path,

the vlan50 and br50 is commented out just to not have to work with 3 of them at the moment.

johnpoz

Your creating bridges of these different interfaces into different vlans?

When I have to deal with interfaces on ubuntu, I just use sub interfaces.. I just use vlan and vconfig.. But its not very often - since even with 30 some years I say away from multihoming any devices ;) Its not a design, goal - its a workaround ;)

I had a box setup recently at home that was multihomed.. But its sub interfaces (vlan interfaces) only used to listen for arps and arp to validate stuff was online.. It was a monitoring tool... You could only talk to it from its admin interface in a different vlan.. You didn't route anything from any of the interfaces, and none of those interfaces had gateways.. And I well aware of the security implications of what I was doing - and that box wasn't open to the internet or anything on any of its interfaces.

While the box I run vms one has a trunk port connected to one of its interfaces, so vms can be on any vlan I want them to be on.. The vms are never multihomed. And the VM hosts doesn't use or talk on any of those vlans - you can only get to it from its main normal, admin vlan..

Routing should be done on your "router" ;)

antonkristensen

@johnpoz

the only physical that has a network cable connected to it is the enp4s0.
The rest are virtual nics.
I made the bridges because i thought that was necessary to get this thing to perhaps work! heh!

Im learning as i go through it, it fascinates me how Unifi devices just do this with a click of a button, and then it seems to be über hard to configure it manually on regular devices found everywhere... heh!

Derelict

Unifi wireless is not routing. Each wireless network is its own broadcast domain talking to the correct firewall gateway address for that subnet. They are layer 2 bridges, not layer 3 routing interfaces like you will find on a multi-homed host.

johnpoz

What is the goal here? Other than a learning experience? What is the point of putting this server into 3 networks at the same time? If you do want to do that - why would you route over anything then the network you want to use as the main network.. If you want the server to get to your iot vlan, why can you not just route it over the lan vlan to the iot vlan, through pfsense.

There is little reason for this server to be connected to all 3 networks at the same time..

Where you run into another issue. When you multihome is even if you send the traffic over say your lan connection to get to your iot network.. But your server has an interface in the iot.. And say you want the iot device to talk to the server.. What does the fqdn resolve to when the IOT looks it up? Is it the lan IP, or the iot IP? Lets say its the lan IP - and you allow it to through the firewall.. When the server goes to answer back, hey I have an IP in that interface I will just send the traffic back using that. Now your asymmetrical - now you can run into all kinds of issues, states on the firewall. Device not accepting the traffic because wtf.. I sent that to my gateway at mac xyz, why is the return traffic coming from mac address abc..

Multihoming a server is a whole can of worms your opening up.. And while sure it can be a learning experience... Be prepared for a huge amount of issues that you have not even thought of yet. Say for example ok fine will just make sure devices in IOT resolve server.domain.tld to its iot IP.. How do you do that? Are you setting up views in your dns?

If your goal is learning about vlans and tagging - great, but no reason to also complex that up with multihoming a server.

antonkristensen

@johnpoz
Yeah i guess it makes sense, and i had thought about routing over the pfsense instead of vlan, just figured it would be a good learning thing to try it over separate vlans, as i have mentioned before.

but i see what you are talking about and that exacly the issues i have been facing here, i managed to get things to work somehow but not optimally, like @Derelict said it was going to end up as a routing issue, and it was, i removed those bridges and configured directly on the vlan interface and threw in some routing rules that made sense to me, i managed to get vlan40(guest network) to speak to the correct vlan and get the responses back on with that vlan tag, as did the vlan10, on the correct interface to and from pfsense, but there was as you mention more too it, i could access all the services from both vlans, tried to throw in some firewall rulez but ofcourse the computers didn't give a f* about them once they knew where to directly talk to eachother, so i would have had to define the NIC for every single service on the server itself, instead of routing it just via the firewall and have everything restricted from there.

So i have learned alot from this, probably one of the thing was that this is possible, but there does exist an easier way to do it. Perhaps i should upgrade some of my devices and holy cow (preferably the pfsense that is running on a 32bit computer), as simple as they have managed to create it for regular people just plugging in an ethernet cable and everything works this sure as hell is a complicated tech! hah!

Not that i don't understand it, but there are so many different caviars to it. but it is fun to learn it and it really is a good skill to have in the backpack, I have actually solved business and enterprise issues with tech support on the line on cisco devices and they didn't have a clue how it was done and why i knew those things! hah.

Thank you both for beating some sense into me and creating an interesting conversation about the topic, thanks for your time!

johnpoz

@antonkristensen said in Wierd behaviour on VLANs jumping between interfaces.:

pfsense that is running on a 32bit computer

So your running 2.3.x? Dude EOL long time ago - really need to update that... What is it running on that is only 32bit? Ouch.. If need be just run pfsense as VM... That is something worth playing with ;) Not multihoming servers, hehehe