high availability w/ redundant layer 2 switches causing loop on my test network

se_marc · Apr 21, 2021, 6:51 PM

hey everyone, I'm attempting to test High availability with two layer 2 redundant switches and I'm causing a loop somehow. My lab network is set up a bit different than the official netgate diagram (see second diagrams).

Is Unmanaged Switch A from the second diagram causing the loop or might it be the unifi 8 port PoE managed switch? My gut is telling me its the managed switch.
The managed switch has RSTP enabled.
When I plug the cable between Unmanaged Switch B and Unmanaged Switch C, the loop occurs and my 192.168.1.0/24 network loses local connectivity as well as internet.
When I unplug the cable, 192.168.1.0/24 regains connectivity to both local and internet.
The reason I am using the managed switch is so VLAN 100 can act as an ISP. I am using a VLAN due to lack of ports on my EdgeRouter 4 for a second physical network as a simulated ISP

Is there a way I can prevent the loop from happening with my current setup, maybe creating a rule that segregates each LAN from each other?

Official Netgate Diagram:

My full network (minus connected clients):

johnpoz · Apr 21, 2021, 7:24 PM

@se_marc said in high availability w/ redundant layer 2 switches causing loop on my test network:

When I plug the cable between Unmanaged Switch B and Unmanaged Switch C, the loop occurs

Well Duh! ;)

I do not know of any unmanaged switch that supports STP.. So yeah you have a loop.. How would some upstream or downstream managed switch fix that?

This will not work with unmanaged switches - you will have a loop

If you want to do that - get managed/smart switches that support spanning tree so the switches can shut down the redundant path(s) when not needed. Ie no loop.

se_marc · Apr 21, 2021, 7:37 PM

@johnpoz much appreciated.

so it sounds like i'll need to free up a port on my EdgeRouter4 move my VLAN to that port? This will allow my to remove the managed switch from my diagram completely.

With that being said, I still see the same loop. if STP is non-existent and the only network at play is 10.10.1.0/24, will the setup below work?

johnpoz · Apr 21, 2021, 7:58 PM

No you can not connected dumb switches like that.. Just pick up some smart 5 ports.. You can find them for like $25-40 for sure..

Or juts replace all 3 of those with 1 smart switch ;)

se_marc · Apr 21, 2021, 8:25 PM

@johnpoz okay so, i probably should just ask what i actually need to know without the whole lab network. Can I set up HA with 2 ISPs and redundant switches? This is how I imagine it would need to be setup. Do you see any issues with this? Since I will be using the switches directly connected to the ISP modem, I feel like I can use dumb switches.

According to this documentation: "When using multiple switches, the switches should be interconnected. As long as there is a single connection between the two switches, and no bridge on either of the firewalls, this is safe with any type of switch."

se_marc · Apr 21, 2021, 8:31 PM

@johnpoz im trying to combine THIS and THIS

johnpoz · Apr 21, 2021, 9:20 PM

that is not what you have.. You have dumb switch A..

se_marc · Apr 21, 2021, 9:26 PM

@johnpoz i don't have an actual second ISP to test with so I was using LANs as my two "ISPs". I will get some managed switches to test on my lab internally but would the most recent diagram I posted work if i actually have 2 ISPs and keep the dumb switches?

johnpoz · Apr 21, 2021, 9:27 PM

You can keep the dumb switches, but you have dumb switch A connecting them, and then you connect them together again with a cable = LOOP!

se_marc · Apr 21, 2021, 9:32 PM

@johnpoz sorry, i'm a bit confused. i didn't include dumb switch A in the most recent diagram.

Derelict · Apr 21, 2021, 10:16 PM

@se_marc This is how you would do it.

This cannot be done with "cheap" gear.

Some gear that can do this that used to be expensive is now not very expensive on eBay, etc.

johnpoz · Apr 22, 2021, 6:13 PM

@se_marc said in high availability w/ redundant layer 2 switches causing loop on my test network:

i didn't include dumb switch A in the most recent diagram.

My point was what you posted as how to set it up, what not what you had before.

A dumb switch is not a ISP device that would not bridge the connections together..

Your drawing showing 2 isp, you can not replace that isp device with a dumb switch - which becomes really a wire.. And now you have 2 wires connecting 2 dumb switches - big fat loop!

To derelicts point - what your trying to do with a HA setup is not meant to be done with cheap dumb home gear. If you want to play with a lab to do this sort of stuff, you need equipment that can do a min amount of enterprise level sort of stuff. Which stp is common on even the cheapest of smart switches.

Or yeah get on ebay and get some enterprise gear for cheap..

se_marc · Apr 22, 2021, 6:01 PM

@johnpoz @Derelict I haven't given up just yet lol. I took some time to think things over and I feel like I'm getting closer. The managed switches will be ubiquiti so RSTP will be enabled. To my understanding that means ports 3 & 4 on Managed Switches B and D will be blocked unless something happens the the either ports 5 & 6 on Managed Switch A or one of the WAN ports on pfSense Main. same would go for the second ISP. Would i need to connect Managed Switch A to Managed Switch B, and then C to D?

johnpoz · Apr 22, 2021, 6:16 PM

Trying to lagg into 2 disconnected switches not something that is normally an option. When you lagg into multiple switches like you have it draw. The switches are part of stack.

Are your A and B, and then C,D stacked?

Notice the drawing Derelict posted.

se_marc · Apr 22, 2021, 6:27 PM

@johnpoz crud, to my knowledge unifi switches do not support official stacking but a common trend is to connect them together via SFP ports and then relying on RSTP. This causes half speed of the LAGG router ports.

johnpoz · Apr 22, 2021, 7:03 PM

Then its not really a lagg now is it ;) hehehe

Derelict · Apr 22, 2021, 7:18 PM

People call all sorts of things a "lagg."

I consider everything except proper LACP which generally requires either stacking or multi-chassis trunking, etc.

AndyRH · Apr 22, 2021, 7:22 PM

I wanted to play a network guy on TV, so I bought one of these on eBay. It will do everything in your drawing in one switch. Great for play time.

Aruba S2500-48P-US <- has 10Gb SFP+ ports, POE, VLANs, semi-loud fans... All the enterprise features for under $200.

johnpoz · Apr 22, 2021, 7:26 PM

@derelict said in high availability w/ redundant layer 2 switches causing loop on my test network:

People call all sorts of things a "lagg."

Very true - its a kind of a catch all.. I was thinking lacp, which yeah you need a stack..