Wierd behaviour on VLANs jumping between interfaces.
-
Would the routes be defined on the main physical NIC or would it be defined on each bridge/vlan nic?
-
Routes are not defined on a nic, they are defined in the OS!!
The OS determines routing, a nic is only a interface into a specific network - or a way to talk to something else that network is attached to to get to a different network.. Ie route.
-
@johnpoz
lol yeah that is what im working with, im not standing above the physical network port and trying to push some actual routes on it.but i have to define each nic on the server, derived from the physical nic
this is what i am working with(if there is any interest):
network: version: 2 renderer: networkd ethernets: enp4s0: match: macaddress: 20:cf:30:c1:c2:55 dhcp4: no dhcp6: no accept-ra: no vlans: vlan10: accept-ra: no id: 10 link: enp4s0 vlan40: accept-ra: no id: 40 link: enp4s0 optional: yes #vlan50: #accept-ra: no #id: 50 #link: enp4s0 #optional: yes bridges: br10: dhcp4: no dhcp6: no accept-ra: no interfaces: - vlan10 addresses: [10.0.10.2/24] nameservers: addresses: [10.0.10.1] gateway4: 10.0.10.1 routes: - to: 10.0.10.0/24 via: 10.0.10.1 table: 102 - to: 0.0.0.0/0 via: 10.0.10.1 table: 102 routing-policy: - from: 10.0.10.1 table: 102 - to: 10.0.10.1 table: 102 br40: dhcp4: no dhcp6: no accept-ra: no interfaces: - vlan40 addresses: [10.0.40.2/24] nameservers: addresses: [10.0.40.1] gateway4: 10.0.40.1 routes: - to: 10.0.40.0/24 via: 10.0.40.1 table: 102 - to: 0.0.0.0/0 via: 10.0.40.1 table: 102 routing-policy: - from: 10.0.40.1 table: 102 - to: 10.0.40.1 table: 102 #br50: # dhcp4: yes #dhcp6: no #accept-ra: no #interfaces: # - vlan50 #addresses: [10.0.50.2/24] #nameservers: # addresses: [10.0.50.1] #gateway4: 10.0.50.1and yes you can tell me if im way off or if im on the right path,
the vlan50 and br50 is commented out just to not have to work with 3 of them at the moment.
-
Your creating bridges of these different interfaces into different vlans?
When I have to deal with interfaces on ubuntu, I just use sub interfaces.. I just use vlan and vconfig.. But its not very often - since even with 30 some years I say away from multihoming any devices ;) Its not a design, goal - its a workaround ;)
I had a box setup recently at home that was multihomed.. But its sub interfaces (vlan interfaces) only used to listen for arps and arp to validate stuff was online.. It was a monitoring tool... You could only talk to it from its admin interface in a different vlan.. You didn't route anything from any of the interfaces, and none of those interfaces had gateways.. And I well aware of the security implications of what I was doing - and that box wasn't open to the internet or anything on any of its interfaces.
While the box I run vms one has a trunk port connected to one of its interfaces, so vms can be on any vlan I want them to be on.. The vms are never multihomed. And the VM hosts doesn't use or talk on any of those vlans - you can only get to it from its main normal, admin vlan..
Routing should be done on your "router" ;)
-
the only physical that has a network cable connected to it is the enp4s0.
The rest are virtual nics.
I made the bridges because i thought that was necessary to get this thing to perhaps work! heh!Im learning as i go through it, it fascinates me how Unifi devices just do this with a click of a button, and then it seems to be über hard to configure it manually on regular devices found everywhere... heh!
-
Unifi wireless is not routing. Each wireless network is its own broadcast domain talking to the correct firewall gateway address for that subnet. They are layer 2 bridges, not layer 3 routing interfaces like you will find on a multi-homed host.
-
What is the goal here? Other than a learning experience? What is the point of putting this server into 3 networks at the same time? If you do want to do that - why would you route over anything then the network you want to use as the main network.. If you want the server to get to your iot vlan, why can you not just route it over the lan vlan to the iot vlan, through pfsense.
There is little reason for this server to be connected to all 3 networks at the same time..
Where you run into another issue. When you multihome is even if you send the traffic over say your lan connection to get to your iot network.. But your server has an interface in the iot.. And say you want the iot device to talk to the server.. What does the fqdn resolve to when the IOT looks it up? Is it the lan IP, or the iot IP? Lets say its the lan IP - and you allow it to through the firewall.. When the server goes to answer back, hey I have an IP in that interface I will just send the traffic back using that. Now your asymmetrical - now you can run into all kinds of issues, states on the firewall. Device not accepting the traffic because wtf.. I sent that to my gateway at mac xyz, why is the return traffic coming from mac address abc..
Multihoming a server is a whole can of worms your opening up.. And while sure it can be a learning experience... Be prepared for a huge amount of issues that you have not even thought of yet. Say for example ok fine will just make sure devices in IOT resolve server.domain.tld to its iot IP.. How do you do that? Are you setting up views in your dns?
If your goal is learning about vlans and tagging - great, but no reason to also complex that up with multihoming a server.
-
@johnpoz
Yeah i guess it makes sense, and i had thought about routing over the pfsense instead of vlan, just figured it would be a good learning thing to try it over separate vlans, as i have mentioned before.but i see what you are talking about and that exacly the issues i have been facing here, i managed to get things to work somehow but not optimally, like @Derelict said it was going to end up as a routing issue, and it was, i removed those bridges and configured directly on the vlan interface and threw in some routing rules that made sense to me, i managed to get vlan40(guest network) to speak to the correct vlan and get the responses back on with that vlan tag, as did the vlan10, on the correct interface to and from pfsense, but there was as you mention more too it, i could access all the services from both vlans, tried to throw in some firewall rulez but ofcourse the computers didn't give a f* about them once they knew where to directly talk to eachother, so i would have had to define the NIC for every single service on the server itself, instead of routing it just via the firewall and have everything restricted from there.
So i have learned alot from this, probably one of the thing was that this is possible, but there does exist an easier way to do it. Perhaps i should upgrade some of my devices and holy cow (preferably the pfsense that is running on a 32bit computer), as simple as they have managed to create it for regular people just plugging in an ethernet cable and everything works this sure as hell is a complicated tech! hah!
Not that i don't understand it, but there are so many different caviars to it. but it is fun to learn it and it really is a good skill to have in the backpack, I have actually solved business and enterprise issues with tech support on the line on cisco devices and they didn't have a clue how it was done and why i knew those things! hah.
Thank you both for beating some sense into me and creating an interesting conversation about the topic, thanks for your time!
-
@antonkristensen said in Wierd behaviour on VLANs jumping between interfaces.:
pfsense that is running on a 32bit computer
So your running 2.3.x? Dude EOL long time ago - really need to update that... What is it running on that is only 32bit? Ouch.. If need be just run pfsense as VM... That is something worth playing with ;) Not multihoming servers, hehehe
-
@johnpoz
Yes! Heh!
Most of my network is from devices that i have aquired and scavenged, my networks cost is probably lower than the amount of money i have paid for the electricity that run them! heh!The box is from an old POS touchscreen from a store, it does the job quite and the main reason i haven't changed it out is because of the built in 2 NICS and the tiny form factor. 2-400$ is to much for this hobby yet, as i have around 15 other hobbies that also cost alot xD
-
Well this server your using - could prob just run pfsense as VM ;)
-
@antonkristensen said in Wierd behaviour on VLANs jumping between interfaces.:
I made the bridges because i thought that was necessary to get this thing to perhaps work! heh!
No, the only connection between VLANs is supposed to be through a router or layer 3 switch. It's no wonder you're getting weird behaviour.
-
@antonkristensen said in Wierd behaviour on VLANs jumping between interfaces.:
The box is from an old POS touchscreen from a store
POS? Piece of s***?
One thing I learned a long time ago is that keeping hardware beyond it's best before date isn't a great idea, as it can be frustrating when it fails to meet needs. Check out the stores selling used & refurb gear for something better. Even 32 bit Linux distros are getting scarce. I've been running 64 bit here for 13 - 14 years.
BTW, my first computer was 8 bit.
Also, many years ago, I used to be a computer tech, working on the old mini computers. I realized I was in a dead end job, when I read about how the Intel 80386 CPU was as powerful as the CPU in a VAX 11/780 computer. Several of the computers I maintained were VAXes, but I had several other systems that weren't a powerful as a VAX.
Bottom line, time moves on and sticking with ancient gear is pointless.
-
@JKnott
I found an older i5 hp machine in the attic that i just set up with an 64bit pfsense! updated finally!
Guess it was the smartest thing to do.
