Interface + VLANs setup advice

johnpoz

@gwaitsi said in Interface + VLANs setup advice:

or am i not getting it?

Your not getting it... Why would you have to or ever want to bridge over a lags? If you have free ports on your router and switch - either make a bigger lagg (more ports).. Or break them up into multiple lags that carry different vlans, or break out the interfaces to just carry only specific vlans.

So if i connect each switch directly to the 8 port box

So your question is this... You have the top, and your wondering how to do the bottom?

You wouldn't set that up - the whole point of distribution and access layers in a switch design..

You could do something like this

How to best setup would really depend on your vlans connection to your core or routing point would depend on traffic flows, etc. etc.

akuma1x

@gwaitsi I also think you're getting your IP address space and networks wrong, like you listed above.

igb1 lan 192.168.1.1/26
igb2 lan 192.168.1.65/26
igb3 lan 192.168.1.129/26

I don't believe that those are different networks. These are technically different networks, and will VLAN much easier/better:

igb1 lan 192.168.1.0 /24
igb2 lan 192.168.2.0 /24
igb3 lan 192.168.3.0 /24

I'm actually surprised pfsense let you make separate interfaces with those addresses. I would have assumed that it would have squawked at you about that. I don't think a network size of /26 or /24 really matters in this case. It's usually common practice to start with size /24 and adjust up or down from there, depending on your needs.

johnpoz

@akuma1x said in Interface + VLANs setup advice:

I don't believe that those are different networks.

Exactly - pfsense wouldn't allow him to do that on different interfaces.

4o4rh

@johnpoz exactly, the 2nd diagram. but these are access layer switches i am talking about.

my house has 3 floors, with 2 managed switch. 1 on the upper floor and 1 in the basement.
i was using the lagg to increase the bandwidth between the two.

on the upper floor switch is an openwrt tplink for wireless access and the switch ports are used as unmanaged vlan ports

on the basement switch is two openwrt tplink for wireless access on the ground and basement with switch ports configured in unmanaged vlan ports.

on each floor are media devices that connect to the internet i.e. gaming, netflix, etc. these are all connected to the same MEDIA vlan like in pic 1

on each floor is a wifi and wifi-guest on separate vlan and the switch ports have untagged vlans for either work PCs or home or kids. everything works well in this config on the J1900 but i don't get full performance of the 1GB ports

I figured, the performance would be better if i hang each of the managed switches and tplinks, directly off pfsense but am struggling with how a setup my MEDIA vlan for example to then service all switches devices.

johnpoz

@gwaitsi said in Interface + VLANs setup advice:

i don't get full performance of the 1GB ports

Well you could be running into your issue with same physical path with your intervlan traffic. Which if you broke out multiple uplinks you couldn't run into that problem..

Lets look at it like this.

Here you have you have setup lagg from your dist switch connected to pfsense.

And this traffic happens to go up down over the same physical link.. This could be hindered.. And you don't see full gig.

If you broke out your uplinks to different physical uplinks you control the path.. So vlan X is on 1 phsyical path, and vlan Z is on different physical path and you don't have a hairpin.

Your issue could be that for whatever reason your pfsense can not just route at full wire speed? But hairpin over a physical interface would never be a helper..

Maybe you do 2 port lag for vlan X and 3 port lagg for vlans Y and Z.. All depends on your traffic flow - what is talking to what and how much..

Trying to bridge interfaces in pfsense sure and the hell not going to make for better performance that is for sure..

Without understanding the data flow and amounts trying to optimize your physical paths is just guess work.. My point from before was that a lagg gives you simplicity and redundancy on a failure - but it does come at a cost of actual control of what physical path traffic might take.

4o4rh

@johnpoz if i take the use case to more detail.

PFSENSE -> WAN
        |-    SW1 (basement 8 port netgear) <- TPLINK basement (WIFI + media + home + IoT)
                                                                 <- TPLINK ground (WIFI + media + VOIP)
              SW2 (3rd floor 12 port netgear)    <- TPLINK 3rd (WIFI + work + office  + MEDIA + NAS + IoT)

So the majority of traffic is internet outgoing.
Internal traffic from SW1 -> SW2 is home media server, nas (smb/nfs) + IoT servers
VLANs dedicated for MEDIA, VOIP, IOT, SERVERS, HOME, WORK, OFFICE

Now I change from the J1900 to i3 8 port, I want to maximize the port usage for pfsense and performance overall.

So, what do you recommend as the way to best optimize the usage of the ports and performance overall

johnpoz

@gwaitsi what are those specific switch models - do they do vlans.. tplink has had some serious issues with vlans before..

What is your internet traffic, unless its higher than gig.. That comes out of the equation completely.. Doesn't matter if you had 100ge coming into pfsense if internet is only 1 gig, etc.

So is the majority of your intervlan traffic.. What mostly talks to your nas for example - home or media?

4o4rh

@johnpoz said in Interface + VLANs setup advice:

@gwaitsi what are those specific switch models - do they do vlans.. tplink has had some serious issues with vlans before..

What is your internet traffic, unless its higher than gig.. That comes out of the equation completely.. Doesn't matter if you had 100ge coming into pfsense if internet is only 1 gig, etc.

So is the majority of your intervlan traffic.. What mostly talks to your nas for example - home or media?

TPLinks are WR1024ND running latest OpenWRT so they work quite well with vlans.
i use the lan/wan bridge as unmanaged

there are two wan connections with 120mg cable and 100mg fibre
the nas box is also server for media and some IOT apps with total 4x vlans, so yeh traffic is mostly local PCs to NAS, media devices to nas on media vlan, etc.

4o4rh

@akuma1x just did some testing and configured

igb2 192.168.1.1/26 LAN_igb2
igb3 192.168.1.65/26 LAN_igb3
igb4 192.168.1.129/26 LAN_igb4
igb5 192.168.1.193/26 LAN_igb5

then created an interface group
LAN comprising LAN_igb2, LAN_igb3, LAN_igb4, LAN_igb5

put some test firewall rules in LAN group

gives the desired effect, which means in an alias I can use a network address of 192.168.1.1/24
i can apply the same principle for the VLANs i.e. if i have devices for VLAN10 on igb3 and igb5, just create 2 VLAN10 with 26 subnet, then a VLAN10 group to simpify rule management

4o4rh

@johnpoz
Hi John, i created a bridge for the separate LAN ports, so i can plug each switch directly into the quotom. they are only 8 port and 12 port netgears, but i need to free up some ports, which is why i got the 8 port qotom as an upgrade.

that approach also worked for the VLANs, but i am finding that increasing the total interfaces, causes the CPU to be running hotter even though the load is minimal (still on the bench)

some of the VLANs, i can isolate to upstairs or downstairs (thereby reducing a bridge + 1 vlan interface) that leaves me with the WIFI.

Currently i have 3 different access points, 2 connected to downstairs switch, 1 to upstairs.

I have internal WIFI network and a GUEST network.
I have them all setup with the same SSID and channel.
They are all on the same vlan (with the present setup) and work seemlessly.

with the new setup, i also bridged the vlans and it seems to work.

My question is;
if i put them on a separate vlan for upstairs and downstairs, will i still get seemless switching when moving between levels?

akuma1x

@gwaitsi said in Interface + VLANs setup advice:

that approach also worked for the VLANs, but i am finding that increasing the total interfaces, causes the CPU to be running hotter even though the load is minimal (still on the bench)

I think you're getting higher temperatures on your pfsense box because you are literally asking it do do everything on your network - switching, routing, firewalling, lagging, all of it. That's not normal for a router/firewall device on a small home computer network.

In my opinion, and I know you probably don't want to hear it, you don't have your network setup correctly.

johnpoz

@akuma1x said in Interface + VLANs setup advice:

you don't have your network setup correctly.

From these statement I would agree

@gwaitsi said in Interface + VLANs setup advice:

i created a bridge for the separate LAN ports

@gwaitsi said in Interface + VLANs setup advice:

i also bridged the vlans and it seems to work.

Generally speaking if your "bridging" your doing it wrong ;) If you need switch ports - get a switch.. While there are legit reasons to create a bridge, normally it would be when you need to change media types.. A typical legit reason for a "bridge" would be in say a AP, which is designed be a bridge from ethernet to wifi..

Seems like he bought a router with 8 discrete interfaces, just so he could use it as switch that he could of gotten for like $40...

I would like to see a drawing if this final setup - but yeah from comments of bridging I would say its not optimal at all..

4o4rh

@johnpoz here is my original/current network with a Qotom J1900 that runs with a temp of 47degC at 25% CPU utilization

The only additional packages to standard are pfblocker and suricata. everything works like a charm

J1900_Design.ndg.png

4o4rh

@johnpoz here is the new design, with the introduction of an additional managed switch for the IOT which has more than the 2 depicted.

i bridge the Lagg0/ProPOE/GS108 with the LAN 192.168.x.x/24 for the original LAN
i bridge the Lagg0/GS108 with the VLAN Home of the original 192.168.x.x/24
i bridge the Lagg0/GS108 with the VLAN WIFI of the original 192.168.x.x/24

according to the docs, the bridge will use spanning tree, so most of the traffic will go directly to the internet, but some will go to the server or home vlans.

the reason for 2 and 3 was to keep them backward compatible to not impact my windows /smb networking setup.

the IOT i made as 2 vlans and created an interface group to apply the rules to both
the VLAN GUEST i made as 2 vlans and created an interface group to apply the rules to both
WRK1/WRK2/MEDIA are single interface VLANS so backward compatible rule set
- media server will move into the SVR VLAN

as with the original, i have pfblocker and suricata plus ntopng. CPU utilization is at 5-10% with avg temp 50-55degC. increasing the number of interfaces makes the temp go up, by about 5deg as i added them in one by one and monitored.

On this config, there are 21 interfaces in total vs 15 of the J1900 config

i3_Design.ndg.png

4o4rh

@akuma1x see my current and upgrade designs. i am open to advice

4o4rh

@johnpoz Hey John, so i am wondering if this would be a better construct. would get rid of the lagg on the qotom, but don't see that it would help reduce the interfaces or firewall/routing greatly.

Media, home, wifi all pass to svrs through firewall rules only.
media, guest, DMZ/IOT/VOIP pass exclusively to internet
media/iot pass to svrs through firewall rules only.

i could reduce interfaces on the quotom for example by

vlan guest to only gs108 therefore 1st floor would switch through gs110 to gs108
vlan wifi to only gs110 therefore wifi ground/basement would switch through gs108 to gs110

what are your thoughts?

2022-03-22 18_43_11-C__Users_Stevo_OneDrive_Documents_i3_Designv2.ndg.png

johnpoz

@gwaitsi looks like a LOOP to me with that purple line connecting gs108t and your gs110

I would never do it like that.. I would never bridge on my router - get switch.. That becomes the distribution layer.

If your cameras are writing to your nas - I would put them in the same vlan, or your routing all that traffic.. And your hairpinning the traffic over physical interface. I take it purple is trunk.

If you want to firewall routing. Then you should have multiple interfaces as uplinks.. I would put in a switch where your router is.

Something like this..

And then use different physical interfaces for your different vlans - x y and z.. So your not haiprinning traffic over a physical interfaces and your not bridging across interfaces.

You understand in a bridge all traffic will be sent across all of those interfaces. The performance of a software bridge is never going to be the same a switch..

As I stated before and posted typical core, distribution, access layer model. If you have lots of traffic flowing between your switches, or even between vlans, etc. You could lagg where needed to provide for more bandwidth, etc. between multiple devices talking to each other, etc.

Be that a lagg for vlan X to your router and then laggs for your trunks between switches, etc.

If you can not home run all your switches to your distribution layer, then you can daisy chain some of them like this

4o4rh

@johnpoz thanks for the thoughts.

the cameras although they write to zoneminder, also write to addresses in China...like most stuff from there ;-)
same witht the IoT stuff like doorbells, etc.

That's why I have the servers on an isolated vlan so i can control what goes to them.
Same with the media stuff. i have a samsung bluray which loves to talk to russia and has most likely been backdoored. Sat boxes, playstation, etc. i keep them isolated on only allow certain boxes to reach the media server.

in summary, everything has to pass through the firewall to get out, or to get to an allowed internal device. i block all out going traffic and only allow based on rules too.

the purple, i currently have bridged as LAN which is only used as mgmt lan to each device. everything else is on a vlan and the switches are all L3 managed switches. so the alternative is to go back to a 192.168.x.x/26 for each interface or a 192.168.x.x/22

johnpoz

@gwaitsi said in Interface + VLANs setup advice:

are all L3 managed switches.

But your not using them for routing - your using them as just layer 2, ie your vlans.

I get why you want to isolate the stuff, but bridging at pfsense is a horrible idea.. especially if your going to bridge every interface for all your vlans.

Bite the bullet and get a switch to use.. Some $40 8 port gig smart switch would give you say 4 ports for your vlans to pfsense and 4 runs to to your access port switches, etc.

Or get a 16 port switch to allow for more growth, or more vlans... Trying to leverage your router as a switch via bridging is not a solution to anything..

8 port prob work just fine since you could trunk some vlans onto same physical - the vlans that don't do a lot of talking to each other.. And the vlans that do a lot of intervlan talking - put them on their own physical..

4o4rh

@johnpoz i think that is what i was suggesting in the 2nd setup.

e.g. wifi/guest vlans on the gs110 pass on the interface to the gs108 (netgear) which then pass on the single interface to pfsense. there won't be loops because they both support spanning tree. so in effect the vlans will be on cascaded switches.