Question about switchs to be used between WAN CARP and ISP's

SipriusPT

Hello everyone,

I am setting up two netgates XG-7100 to be used with High Availability, but I really dont know want kind (and even models or brands) of switchs I should use for WAN side, for this purpose, and the issues that I could encounter.

I am looking for two desktop switches with 4 or 8 ports at maximum, with at least 2 SPF+ ports, and dont want to buy cheap (with possible weak hardware...) manageable switchs for 30€ used at home or small offices, to add in a system like this. I am looking for something around 100€ or a bit higher, with a good MTBF

Thanks in advance!

DaddyGo

@sipriuspt said in Question about switchs to be used between WAN CARP and ISP's:

I am looking for two desktop switches with 4 or 8 ports at maximum, and dont want to buy cheap

Hi,

maybe it won't disappoint you:
https://mikrotik.com/product/css610_8g_2s_in

johnpoz

You don't find many higher end switches in such a low port density to be honest.

10 ports is normally lowest they go.. You could look at the sg350-10 models, if budget is not a concern you could look to say the SG350X-8PMD model.. That give you poe and multigig ports.. plus two sfp+ ports..

The sg300-10 are still available, they are not eos let. This is your 8 ports, with 2 combo ports as bonus.

The 350x models are stackable - which is what i would think you would be looking for if your wanting to setup a HA configuration.

DaddyGo

@johnpoz

if we have so much money , we prefer this instead of the SMB series Cisco:
https://www.cisco.com/c/en/us/support/switches/catalyst-3560cx-12pc-s-switch/model.html

johnpoz

Yeah those would work ;)

DaddyGo

@johnpoz said in Question about switchs to be used between WAN CARP and ISP's:

Yeah those would work ;)

I’m not a MikroTik fan (Cisco is my love:)), but I got a switch like this in my hand....

with this switch chip Marvell 88E6193X and it has serious performance in addition to simplicity

I surprised...

BTW:

I replaced my "mobile" test switch which was a TP-Link TL-SG105E (because VLAN and port mirror, etc.) for this Mikrotik stuff

SipriusPT

@johnpoz I forgot to mention that my budget is an issue lol, in this case I am looking for something around 100€ or a bit higher, with a good MTBF (I have updated my previous post).

@DaddyGo I will check that mikrotik, it has a very small finger print, and good features such as 2 portas with SPF+ with 11 years of MTBF . Do you know if its possible to fix it well using only one support on rackmount? From what I am seeing its possible to have two aligned in just 1U.

Thank you both for the help!

DaddyGo

@sipriuspt said in Question about switchs to be used between WAN CARP and ISP's:

Do you know if its possible to fix it well using only one support on rackmount? From what I am seeing its possible to have two aligned in just 1U.

As I have just tried, it seems possible...

I hope not a little DIY is far from you as the "rack ears" need to be redesigned,
and the two units must be connected from the inside, but there are holes for this with screw thread.

The two units next to each other are shorter 1 - 2 cm than the standard 19 ", therefore, the "ears" must be reshaped using a saw.
Overall, I say 20 minutes of work

SipriusPT

I've found one with 4 SPF+ ports,

https://mikrotik.com/product/crs305_1g_4s_in

with MTBF of 20 years. Seems like its dificult to found a small print to use just a few ports. I have read a review saying that can overheat when using both 4 ports, but since it will be on a server room with AC, it will not be a problem.

https://mikrotik.com/product/crs305_1g_4s_in
https://www.youtube.com/watch?v=bUmIzmuWtEs&ab_channel=LawrenceSystems

I've only used HP smartswitchs, but from what I have saw on Lawrence Systems channel, it seems to be easy.

I totally forgot that if needed, I would have to use 3 ports SPF+ ports at least, one from ISP, and the other two for firewalls WANs ...

What do you think about this unit @DaddyGo and @johnpoz ?

DaddyGo

@sipriuspt said in Question about switchs to be used between WAN CARP and ISP's:

I have read a review saying that can overheat when using both 4 ports, but since it will be on a server room with AC, it will not be a problem.

This is also good stuff.
We only use it as an optical form converter because it actually overheats with 4 x 10Gig SFP.

I do not recommend this to you as it is very sensitive to SFP modules and only accepts few SFP modules.
It works almost exclusively with Cisco and MikroTik compatible modules.

as you can see in the picture, we also leave empty slots next to each other due to warming up, the house (cassis) in which he lives is too small, and poor ventilation due to few holes...

MOBO 37 celsius with 2 pcs. SFP

+++edit:

jah and it is also in an air-conditioned room...

SipriusPT

@daddygo thank you a lot for those photos and screenshots!

I dont have an idea of expected temperatures per port. In this environment I dont have temperature sensors in SFP ports to have a reference.

Regarding that switch MOBO, seems a bit high under AC (assuming a AC running at the lowest temperature, ~17ºC), with two devices, it was under what kind of traffic usage (almost idle, youtube, mails, web,...) ?

For now, I just need 1Gbp per port, but in future 10Gbps could be needed per port. On firewalls side, there are SPF+ free ports, to be used in the future, for both LAN and WAN. In terms of WAN usage, right now, I am using VPNs (one site-to-site, and two client-to-side), around 70 devices with internet access. There are backups being done through VPN site-to-site as well as to third party storages, so sometimes we have around 100Mbps for each backup being done in simultaneously for a couple of days per week.

When you also say that SFP modules are very sensitive, can you be more precise?

Thanks a lot for the help!!

SipriusPT

Seems like this model with 4 SFP+ ports, as a 8 port model with a hugh upgrade on heatsink:

https://mikrotik.com/product/crs309_1g_8s_in#fndtn-gallery

@DaddyGo have you ever used this one?

DaddyGo

@sipriuspt said in Question about switchs to be used between WAN CARP and ISP's:

I dont have an idea of expected temperatures per port. In this environment I dont have temperature sensors in SFP ports to have a reference.

if your switch can read DDM / DOM information and you buy DDM / DOM capable SFP modules you can see the temperature and other parameters:
https://en.wikipedia.org/wiki/Small_form-factor_pluggable_transceiver
and f.e.: https://community.fs.com/blog/how-to-view-the-ddm-information-of-optical-transceiver-via-snmp.html

@SipriusPT sirius "When you also say that SFP modules are very sensitive, can you be more precise?"

each manufacturer has a module compatibility chart...
https://wiki.mikrotik.com/wiki/MikroTik_SFP_module_compatibility_table

well, for this model (CRS305) of MikroTik, this allows very few other manufacturers
(many manufacturers limit these parameters / modules, to buy their branded MikroTik SFP module)

BTW:

on average, in a well-cooled switch, this value is between 35 and 45 (50) degrees Celsius

DaddyGo

@sipriuspt said in Question about switchs to be used between WAN CARP and ISP's:

have you ever used this one?

Yes
it is a device with dual boot ability that can run both RouterOS and SwOS

RouterOS requires a lot of learning as it has a completely different philosophy than a standard SOHO device

I've been working with MikroTik devices for a long time and sometimes I still scratch my neck, but it's pretty professional stuff anyway

+++edit:
for an SMB (small office!) optical core switch, it can be an excellent choice

DaddyGo

@sipriuspt said in Question about switchs to be used between WAN CARP and ISP's:

Seems like this model with 4 SFP+ ports, as a 8 port model

if you want a specifically "SFP" switch I recommend this: https://www.cisco.com/c/en/us/support/switches/sg350-10sfp-10-port-gigabit-managed-sfp-switch/model.html
(stable reliable but no 10Gig option)

and you can use such RJ45 copper modules too:
https://www.fs.com/de-en/products/23681.html
(there are plenty of SFP manf. codes available)

I can still recommend for 10Gig:

https://www.ui.com/unifi-switching/unifi-switch-16-xg/
(here you have to reach into your wallet, deep down)

SipriusPT

@daddygo so after all this time, since I've started this thread, I went in an even bigger project, not only having HA on WAN side, but also HA (active, passive) on firewalls (with both extended cards changed from original 4 NICs with 1Gbps to 4 NICs with 10Gbps), and HA on intranet (only for VMware cluster and some equipment with 2 or more NICs), with a vertical backbone of 10Gbps.

Important note, there are several ways to achieve the same goals, and this was what worked for me. This is just a brief resume of the all adventure from "top rack to bottom".

It was a big struggle with my ISP (ALTICE in Portugal) regarding getting a simple subnet of IPv4's with 3 IPs, because they didnt had that possibility, only through a Border Gateway Protocol (BGP), something that I didnt had an idea on how to setup and still dont have with both XG-7100 units. So I had to get stick with my main goal. This was something that I only discovered weeks after purchasing those two Mikrotiks CRS309-1G-8S+IN with eight 10Gbps tranceivers. Fortunatly Mikrotik its dual boot (!), so I had to move from switch to firewall, and had to learn how Mikrotik Router OS worked in a way that I would DMZ at least 1 IP (the VIP WAN IP for 7100 cluster) for all TCP and UDP ports, from WAN side to a subnet LAN where both 7100s were. I had also to ask for another dedicated fiber from ISP, as well as a reverse DNS and dedicated IP (we only had 1). OMG the documention for Mikrotik is such a mess, god bless youtube and foruns!

From 7100 units, I've changed both expansion cards with 4x1Gbp NICs to one with 4x10Gbps capable of receiving tranceivers, and use it to sync both units in one link, and setup all intranet VLANs in another (leaving empty more 10Gbps ports to use in the future on intranet). At the time I had to backup previously both units with 4x1Gbp cards, and only then installed those 4x10Gbp cards(around 500€ each), but was not easy to reuse that config, so end up factoring reseting both units, and wrote my self the all changes on backup files to restore after on each firewall, using each terminal to verify if each card and NIC was detected, as well as the name of each interface (IXL).On built in 10Gbp ports (IX1 and IX0), I used those as uplinks to those Mikrotiks. Had to setup lots of services that was in the appliance that was in production but to work with HA, like DHCP servers, SQUID+SQUIDGUARD, NAT, VPNs (site-to-site and site-to-clients), VIPs and 7100 IPs for each VLAN (I had to change some devices IPs on production to had some logic on those VIPs and IPs per 7100). I've also take this oportunity to clean an simplify all rules that I had in production during last 4 years, taking into account scalability. During this year I had to pass through lots of testings and backups, where even found one bug in one pfsense version, on dynamic DNS entries not being updated when using a tier 2 connection of a WAN group.

On those HP Aruba switchs, got two 48x1Gbp ports with 4x10Gbps each, as "core switchs" (in the future I will use those to a dedicated cabinet for servers and replace those with real core switchs to connect the all infrastructure), with one with around 300W PoE for our Ubiquiti APs. Those were configurated with three links under trunks, two links between Aruba's, and one each to 7100 units, on those 4x10Gbps.

Regarding all webconfigs and ssh consoles, I've setup all to be placed in one single VLAN that I already had in production.

Those transceivers, for Intel and HP cards, I got some generic transceivers 10GBase-SR SFP+
from 10GTEK, and on Mikrotik I went into their transceivers. For all transceivers, I got spare units just in case, mostly for those generic transceivers that I bought directly from 10GTEK (China), were I didnt had another solution because each intel or HP transceivers are a fortune (some around 300€ compared with those 10GTEK at 15$ each).

This was by far my biggest achivement as a network administrator, that till last year I was only setting up pfsense appliances and VMs as standalones, moving to HA was a big step in all terms. The worst part of doing this was that being the only guy in the show as a IT Manager + System Administrator + Network Administrator + Support all tiers + IT Technician in a 100 enduser company where the core business is not a IaaS, SaaS or PaaS, got this project being delayed several times.

Here is a photo of the all setup (aka eiffel tower) still to be installed in a cabinet:

20220303_180643 (1).jpg

Sorry for the long text but I really feel very proud about this.

SipriusPT

More photos: