Multiple Switches cross connect or each port into pfsense box



  • I have a new install, this is going to be my first small business install with pfsense. There will be a total of 233 live ports. Only about 200 users total. They will access a 1GB/1GB Comcast fiber internet connection. Everything they use is on the cloud no servers or local resources onsite. This is purely internet.

    I will have 5 48 port switches. It was suggested that I plug each switch into it own port on the pfsense box (I can/will add 4 port intel nics), but now I am reading that bridged network performance is not great. Should I just chain the switches into each other, and one port for LAN on pfsense. Or should I bridge the ports together and give each switch its own port, or should I use vlans to cut up the switches.


  • Rebel Alliance Global Moderator

    What is the layout for these 5 switches? Are they spread about in different closets? Just all in the same IDF/MDF room?

    So you do not have these switches stacked - so guess they are spread out.. You sure and the F shouldn't daisy chain them..

    Everything they use is on the cloud no servers or local resources onsite

    Then why would you need to bridge on pfsense to put them in 1 network? Just put each switch on its own vlan/network and plug direct into interface on pfsense. So you would have 5 networks.. So you would want 5 interfaces on pfsense for these networks. If you have 5 interfaces min, unless you were going to lagg for redundancy on the ports, but there really is no reason to tag these networks, ie vlans. You would have just 5 different networks all physical isolated.

    Comes down to where the switches are located, if stacked, etc.. if there is local traffic where devices all need to be on the same L2 ,etc.

    I you say they have no need for local resources prob best to use private vlans on the switches so devices can not even talk ti each other. This way when billing gets hit with ransomware ;) you don't have to worry about it worming its way across the whole office.

    More than happy to help discuss best plans here - just need more info.. Physical location of where these switches are can help in how best to connect them to the edge (pfsense).


  • Netgate Administrator

    If the clients are all going to be on the same subnet then don't use multiple bridged interfaces in pfSense. A switch will always do a better job there unless you need to filter between network segments within the same subnet.
    You might opt to run several interfaces to the switches as a lagg group for some redudancy.
    Depending on the switch capabilites you might stack rather then chain them.
    If you have physical separation you might have, for example, a switch for each floor in the building and core switch connecting them.

    Many possiblities.

    Steve


  • Rebel Alliance Global Moderator

    From what he has stated I see zero reason for these devices to be on the same network.. Unless they are gonig to be doing file sharing between the machines... Isolate the shit of each machine!!! You know billy he is going to get infected in like about 3.2 ms... And then its going to worm its away across every machine that has not been patched, etc.. ;)


  • Netgate Administrator

    I agree, I wouldn't choose to have one subnet there.

    And I also agree I would use VLANs to subnet rather than, say, each switch on a separate interface. Waaaay more flexible.

    Still don't do bridges in pfSense though! (unless you really know you need to)

    Steve



  • @johnpoz said in Multiple Switches cross connect or each port into pfsense box:

    What is the layout for these 5 switches? Are they spread about in different closets? Just all in the same IDF/MDF room?

    They are in one room. A computer closet with 2 Racks and a ton of network cables. The building is one floor.

    Then why would you need to bridge on pfsense to put them in 1 network? Just put each switch on its own vlan/network and plug direct into interface on pfsense. So you would have 5 networks.. So you would want 5 interfaces on pfsense for these networks. If you have 5 interfaces min, unless you were going to lagg for redundancy on the ports, but there really is no reason to tag these networks, ie vlans. You would have just 5 different networks all physical isolated.

    So this was what I was wondering, I am trying to figure out which way is faster, brigded or Vlans, or just chain the switches together. Your saying to use Vlans to isolate the switches. That was a answer I was looking for. I could also install another 8 or 12 port switch to connect all the other switches together (so they are not daisy chained) and then one cable to the pfsense box.

    Comes down to where the switches are located, if stacked, etc.. if there is local traffic where devices all need to be on the same L2 ,etc.

    I you say they have no need for local resources prob best to use private vlans on the switches so devices can not even talk ti each other. This way when billing gets hit with ransomware ;) you don't have to worry about it worming its way across the whole office.

    So i should have said been more clear there will be printers (Airprint and Macs) which I can then just open up the correct ports to allow for printing.

    More than happy to help discuss best plans here - just need more info.. Physical location of where these switches are can help in how best to connect them to the edge (pfsense)

    Thanks for your advice, let me know what other info you need.

    Also I am using Ubiquiti 48 Port switches, and Ubiquiti AC access points for wifi (was planing on Wifi being on its own VLAN and the Guest Wifi as well)
    The specs for the pfsense box is i3-7100, 16GB DDR4, 60GB SSD drive, depending on how I setup things one x2 port intel nic, or two x4 port Intel nics.


  • Rebel Alliance Global Moderator

    If all the switches in 1 room in the same rack why not just stack? Do the 48 port unifi not stack?

    If you want them all on same L2 for airprint then yeah use a distribution layer switch. Ie your smaller 8 port switch to connect them. Then connect to pfsense from that switch.. You only need 1 lan interface on pfsense then ;)

    Printer(s) would be local resource btw...

    If the user devices do not need to talk to each other I would still isolate the shit of them.. That billy loves to get infected!! ;)



  • @johnpoz said in Multiple Switches cross connect or each port into pfsense box:

    If all the switches in 1 room in the same rack why not just stack? Do the 48 port unifi not stack?

    They don't, and this was a requirement from the customer (they have some IT guys in California who told them Unifi was the only way to go. I talked them out of the Unfi gateway they wanted to use)

    If you want them all on same L2 for airprint then yeah use a distribution layer switch. Ie your smaller 8 port switch to connect them. Then connect to pfsense from that switch.. You only need 1 lan interface on pfsense then ;)

    Perfect this will do perfect

    Printer(s) would be local resource btw...

    Yes sorry I was not clear, to be honest I forgot about the 3 printers they have until you guys started talking about VLANS, but I was thinking about isolating the printers anyway and just opening up the ports for airprint.

    If the user devices do not need to talk to each other I would still isolate the shit of them.. That billy loves to get infected!! ;)

    Ok I will consider isolating them. They are all running Mac so infection is a little less of a worry (I know MACs can get viruses, just a little harder then a Windows Machines).

    What do you think about the HW for the router good enough? and also I can get away with a 2 port nic if I just use a smaller switch?


  • Rebel Alliance Global Moderator

    @manwdaplan said in Multiple Switches cross connect or each port into pfsense box:

    anyway and just opening up the ports for airprint.

    Its a bit more complicated than that.. Airprint is designed for flat layer 2 networks for discovery of the printer. You could use avhai to allow the discovery.

    Why are you using SSD? Are you going to run proxy? Why not just go with netgate hardware.. The XG-7100 would seem like a good fit for such a deployment. And it has the 8 port switch so now you don't need to buy that other switch..

    If you can wait a bit the SG-5100 be good too.. Im more a fan of router interfaces vs switch ports. You can always get a small switch if you need switch ports ;)



  • @johnpoz said in Multiple Switches cross connect or each port into pfsense box:

    @manwdaplan said in Multiple Switches cross connect or each port into pfsense box:

    Its a bit more complicated than that.. Airprint is designed for flat layer 2 networks for discovery of the printer. You could use avhai to allow the discovery.

    Leave it to Apple to make this hard, ok maybe we will just use IP to print that will depend on what the customer wants to do. Thanks for that by the way I am not great with Apple and MACOS.

    Why are you using SSD? Are you going to run proxy? Why not just go with netgate hardware.. The XG-7100 would seem like a good fit for such a deployment. And it has the 8 port switch so now you don't need to buy that other switch..

    SSDs are cheap and pretty darn reliable. I was planing on using Squid like I have setup as home :) This is my first real deployment of pfsense in a business setting, all of my customers are very small (no more then 10 users) and a ASUS router (with Merlin firmware) as been fine for most everybody so far. This is by far my largest customer which is why I decided to go with pfsense and not a retail router. So the router budget is $500.00 which is why I was looking at a DYI build (I have most of the parts on hand due to the nature of my business, I have inventory)

    If you can wait a bit the SG-5100 be good too.. Im more a fan of router interfaces vs switch ports. You can always get a small switch if you need switch ports ;)

    I am open to suggestions on Netgate devices if you can recomend one for under $500 that can handle that amount of users. But I will need the device in less then two weeks.



  • @johnpoz said in Multiple Switches cross connect or each port into pfsense box:

    The XG-7100 would seem like a good fit for such a deployment.

    I can recommend this device...got two of them. ✌

    -Rico


  • Rebel Alliance Global Moderator

    The 3100 could prob do it.. It handles one of my locations without any issues.. But its not on a full gig.. The 3100 should be able to do gig.. But then others come back and say oh it doesn't do vpn at full speed, etc. etc.. <rolleyes>

    But sure the 3100 should be able to do it.. You can look about for others running it - since without local resources you prob have no need for road warrior vpn, etc.

    That is under your $500 and has switch ports as well..

    BTW while I do like the unifi for AP... The price point for the switches - which one did you get the 48 poe (750w) one? For same price point you could of gone with L3 sg500x poe from cisco..



  • @johnpoz

    Thanks for all of your help on this, I will look at the SG-3100 right now. You have been very helpful :)


  • Rebel Alliance Global Moderator

    check out my edit curious on the switches you got.. Guessing you got POE, but they do not stack and they are only L2 right.. From that pricepoint the sg500x from cisco prob better small business choice.



  • @johnpoz

    So they are the non POE 48 Port switches. They are on Amazon here. I was just going to use injectors to power the APs, (just two of them) but now I might just pickup a 16 port Unifi with POE as the switch to attach everything together. I am kinda stuck using Unifi due to the customer mandating them. Like I said before I talked them out of the USG-PRO-4 which I have read very bad reviews on, and understand that the interface is very hard to use, and can not hit GB speeds all the time. I want to use pfsense because it has a large following and I know I can find answers if I have questions (like today)


  • Rebel Alliance Global Moderator

    Oh if there is only 2 of them - then sure you could just use the injectors. But yeah since your getting smaller switch - sure go with POE for your AP..

    What you going to the run the controller on? One of the cloud keys from unifi?



  • @johnpoz said in Multiple Switches cross connect or each port into pfsense box:

    What you going to the run the controller on? One of the cloud keys from unifi?

    Yep the cloud key is on order already. I have never used that device before but it looks pretty easy, to setup, and I have plans for VLAN setup for the wifi networks.

    The location is not very big square footage wise, about 10000 sqft open space with some offices and cubicles, I was going to place one AP on one side and one on the other. My sub contractor is working on the wiring now that is the largest part of the job (233 drops). Like I said much larger then the normal 10 to 20 drops I do myself. I had to sub-contract it out.


  • Rebel Alliance Global Moderator

    Have fun! When project suppose to be complete? Reach out to sales if you have any questions on the 3100.. This is less than your budget and should be fine..

    Or post question in the Official hardware section.. I have a 3100, but just do not have the connection to stress it up to gig.. I know for sure the sg4860 can do it, since it doesn't break a sweat on my 500mbps connection and my buddy has his on a gig and has no problem with that..



  • I am watching some reviews on it now (Youtube), The project should be done Sept 15, so I need the HW installed about 3 days before so I can be assured everything is configured and ready to go once they start moving into this office. Thanks again for all your help!!



  • Sounds like you are already committed to the UniFi APs, but for a small network like that, the HPE OfficeConnect OC20 is a really nice product.

    https://www.hpe.com/us/en/product-catalog/networking/networking-wireless/pip.hpe-officeconnect-oc20-802dot11ac-series-access-points.1010099011.html



  • @coreybrett

    Thanks for the recommendation, I will keep this on my list of stuff I can use. With a little luck I might be some more bigger clients like this.



  • Where are you located?



  • @coreybrett said in Multiple Switches cross connect or each port into pfsense box:

    Where are you located?

    I am in Colorado, Denver metro area. I run a very small computer company called SnowbeePC


  • Galactic Empire

    @manwdaplan said in Multiple Switches cross connect or each port into pfsense box:

    @johnpoz

    So they are the non POE 48 Port switches. They are on Amazon here. I was just going to use injectors to power the APs, (just two of them) but now I might just pickup a 16 port Unifi with POE as the switch to attach everything together. I am kinda stuck using Unifi due to the customer mandating them. Like I said before I talked them out of the USG-PRO-4 which I have read very bad reviews on, and understand that the interface is very hard to use, and can not hit GB speeds all the time. I want to use pfsense because it has a large following and I know I can find answers if I have questions (like today)

    Look at a midspan unit rather than POE injectors, if you aren’t looking at POE switches, most of the midspan units enable power to be switched off remotely.

    Makes things much easier if you need to power cycle access-points.

    I’d look at stacked switches if I was doing this myself.


  • Netgate

    You have 200 seats in a 100m radius? OK 10000 sqft is not that big. I would certainly check that no copper runs are more than 100m though.

    I would certainly stack the switches if possible. Personally I would not even consider a closet of 5 x 48-port switches that weren't stacked.

    The SG-3100 is a great product and will probably move the traffic, but it is really more geared to the SOHO market. I'd probably look at a pair of XG-1537s instead. Probably LACP to the switch stack.

    I would also do a wifi survey. 200 seats with 2 APs sounds a little dense. Figure at least a phone for everyone. That's 100 associations for each AP/Radio. I know brands (Ruckus) that can do that with one radio tied behind their backs. Not sure about the Unifi gear. Never seen a dense deployment first-hand. Something like NetSpot for the Mac can get you some pretty effective, if rudimentary, measurements of the wifi propagation through that particular construction. You'll have to estimate it for all the 160-pound bags of mostly-water that will be walking around. Ceiling mounting makes this a lot easier to predict. Survey for 5GHz. 2.4 GHz will propagate further.

    200 Seats. How many thousands of dollars per hour do you/they lose if it goes down?


  • Galactic Empire


  • Netgate

    @nogbadthebad said in Multiple Switches cross connect or each port into pfsense box:

    https://dl.ubnt.com/datasheets/unifi/UniFi_UAP-AC-HD_DS.pdf

    I'd personally have to see it to believe it.

    2 APs still sounds like not enough for that space even if they can handle that association density.


  • Galactic Empire

    @derelict said in Multiple Switches cross connect or each port into pfsense box:

    @nogbadthebad said in Multiple Switches cross connect or each port into pfsense box:

    https://dl.ubnt.com/datasheets/unifi/UniFi_UAP-AC-HD_DS.pdf

    I'd personally have to see it to believe it.

    2 APs still sounds like not enough for that space even if they can handle that association density.

    Agreed, also depends on the shape of the building, is it square or rectangular.

    If you have the floor plans, I’ve got a copy of Aruba Visual RF I could do a quick predictive plan using a standard Aruba AP with integrated antennas.



  • @nogbadthebad said in Multiple Switches cross connect or each port into pfsense box:

    @derelict said in Multiple Switches cross connect or each port into pfsense box:

    @nogbadthebad said in Multiple Switches cross connect or each port into pfsense box:

    https://dl.ubnt.com/datasheets/unifi/UniFi_UAP-AC-HD_DS.pdf

    I'd personally have to see it to believe it.

    2 APs still sounds like not enough for that space even if they can handle that association density.

    Agreed, also depends on the shape of the building, is it square or rectangular.

    If you have the floor plans, I’ve got a copy of Aruba Visual RF I could do a quick predictive plan using a standard Aruba AP with integrated antennas.

    I will get you a floor plan later tonight when I am at my desk, thank you this will be very helpful


  • Galactic Empire

    @manwdaplan said in Multiple Switches cross connect or each port into pfsense box:

    @nogbadthebad said in Multiple Switches cross connect or each port into pfsense box:

    @derelict said in Multiple Switches cross connect or each port into pfsense box:

    @nogbadthebad said in Multiple Switches cross connect or each port into pfsense box:

    https://dl.ubnt.com/datasheets/unifi/UniFi_UAP-AC-HD_DS.pdf

    I'd personally have to see it to believe it.

    2 APs still sounds like not enough for that space even if they can handle that association density.

    Agreed, also depends on the shape of the building, is it square or rectangular.

    If you have the floor plans, I’ve got a copy of Aruba Visual RF I could do a quick predictive plan using a standard Aruba AP with integrated antennas.

    I will get you a floor plan later tonight when I am at my desk, thank you this will be very helpful

    Draw on the plan where you think the APs are going, also can you dimension the longest side of the building.


  • Rebel Alliance Global Moderator

    @derelict said in Multiple Switches cross connect or each port into pfsense box:

    Not sure about the Unifi gear.

    Bit off topic but did you see their new HIGH Density stuff
    https://www.ubnt.com/unifi/unifi-ap-xg/

    This one is for Stadiums and such
    https://www.ubnt.com/unifi/unifi-wifi-basestation-xg/

    200 Seats. How many thousands of dollars per hour do you/they lose if it goes down?

    This is such a good point!!! But they have 500$ budget for router.. Should be HA and multiple lines.. Everything in the cloud remember.. What happens when that 1 isp is offline?


  • Galactic Empire


  • Rebel Alliance Global Moderator

    Thanks... They do look sweet - nice to see a real world example.. I will have to look deployment story on the smaller XG model.. Might be something to look at for this posters deployment... But sounds like they have a real limited budget to be honest.. The switches are not what I would of used that is for sure - I have no experience with the unifi switches - but if you can not stack them they don't seem to be good fit for such a deployment.


  • Galactic Empire



  • So I am having a issue uploading the floor plan what am I missing just gives me a error when I upload the PNG file.

    I might have to go three APs, I was planing on putting in the two I have on order and making sure they are enough, So right now they have about 70 people, they plan on growing to 200, so I do have a little time if expansion is needed.

    As far as the backup is concerned, I will have a extra switch waiting onsite, and they have a Linksys EA9000 series router that they are using now, so I will set it up to be swapped in if the router breaks, if one of the 48 port switches goes out, I will just switch those users over to the wifi till I get it fixed. I know its not enterprise worthy but it is cheap, which this customer is very. They have a call center in California that rolls over to this location when there is volume issues.


  • Rebel Alliance Global Moderator

    I don't think your floorplan made it through the upload? Not seeing it only

    [0_1535726299255_Floor Plan.pdf](Uploading 100%)

    You can for sure do things on a tight budgets - but Derelicts statement of how much money is lost per hour/minute is the big thing these companies need to get through their skulls when it comes to networking infrastructure.. On the shelf router you can swap in - ok, how long does that take you to swap in and make sure is working? 1 hour? 10 minutes - I take it your not going to be on site for example. What about off hours? etc.. Redundancy and failover are huge factors that need to be considered when "if it goes down" your talking $$.. Because in the big picture its not if but when it goes down.

    What happens when the ISP is offline because of say a fiber cut and you might be offline for a couple of days.. We had a recent fiber cut with 1 isp that took them 3 days to get back up. Not an issue because the backup link was there, etc.

    You have to work with in the constraints of the customers budget sure.. Problem is these customers don't understand IT ;) If they did they would not be hiring you... So while it is sometimes a hard sell, make sure they they understand that they could be down for quite some time "when" a part of the network fails.. And since they are depending on that internet connection - when it goes down they are going to be down for the duration.. Hope they have a great SLA with the isp, with major compensation if they do not meet the SLA...



  • @johnpoz

    I completely agree, normally I work it in slowly, once they get used to me helping with stuff, and maybe we go 3 months in when the next quarter and budgets get refreshed I talk to them about what else is needed, second ISP line (they have comcast fiber with a pretty good SLA) redundant routers, that sort of thing, it harder to hit them with it all at once, but I make it clear where the weak spots are and if something happens, they understand. Now I am not onsite, but I am close and have a process to handle emergency issues, they are not a 24x7 place, so that's a little easier.





  • So I had another question that is a side note, so I know you guys work for Netgate (maybe not all of you) and they are here to make money, but do you normally not recommend a DIY builds for PFsense? Whats the positive and negatives of a DYI vs Netgate (sorry if this gets a little off this topic).

    What do you think of the performance of the SG-1000 how much can it handle user and internet wise? That price sadly for my customers is about the limit they want to spend on routers. (I work with very small customers mom and pop normally that is kinda my nitch most IT guys don't like touching these smaller businesses)


  • Galactic Empire

    After a quick look at the plans i’d say you’ll need more than a couple of access points.

    I’d get the cabling contractor to run more copper between the network closet and the ceiling, you can always get some long rj45 patch leads to run above the ceiling to the access points.