Network Design with Multi ISPs for 400+ Users

[ashima]

Hello Everyone,

We have a 400+ seat office space with multiple ISPs.

1. How to Add Load balancer to get effective 1200 Mbps or 1.2 Gbps  with what ever practical limitations

2. How to have Content Filter in place

3. How to have effective firewall services

4. How to enable IP Guard    ( so that if a user changes his machine ip address,  it should not affect the network )

5. Control Rogue AP ( WiFi Access point /  issues arising due to Hot Spotting )

6. Control Rogue DHCP  ( if by any chance a user introduces  for example  Xbox in the network which has DHCP enabled …  etc    devices )

Hardware components available for pfsense box / boxes  integration.

Processor

Intel Core i3-4130 Processor (3M Cache, 3.40 GHz , 2 Core - 4 Thread )
Intel Core i5-4440 Processor (6M Cache, up to 3.30 GHz, 4 Core - 4 Thread )

Motherboard

Gigabyte GA-B85-D3H Motherboard
Ethernet Port : Onboard - Realtek Gigabit LAN 10/100/1000 Mb/s

PCIe x16 Slots : 1  Ver 3.0
PCI Express x16 slot : 1  running at x4 (PCIEX4)
PCI Slots : 2

Ethernet LAN Cards

Intel PWLA8492MT PRO/1000 MT PCI/PCI-X Dual Port Ethernet Adapter
Intel Gigabit E1G42ET Dual Port PCI Express 2.0 Ethernet Adapter

RAM
8  GB DDR3 - 1333 MHZ    ( 4GB x 2 modules  for Dual Channel memory access )
16 GB DDR3 - 1600 MHZ    ( 8GB x 2 modules  for Dual Channel memory access )
32 GB DDR3 - 1600 MHZ    ( 8GB x 4 modules  for Dual Channel memory access )

Hard Disk Drive

Seagate ST1000DM003 1tb sata 6gb/s 7200rpm 64mb
or suggest if SSD would be better.

Display / Graphics Card

AMD/ATI Radeon HD5450 1 GB DDR3 Graphics Card
so that the system memory is not shared  & is dedicated for pfsence

ISP Link available
cable broadband 200Mbps each  x  6 nos  ( to get effective 1200 Mbps or 1.2 Gbps with which ever practical limitations )

I understand we'll need a seperate box for Load Balancing and content filtering. How do I go about designing the entire network.
Can integrate  multiple pfsense boxes.

Any help is greatly appreciated.

Thank you
Regards,
Ashima

[jahonix]

You might want to consider this option: https://portal.pfsense.org/professional-services.php

It is a sign of personal strength to get external expertise, NOT a sign of weakness!

@ashima:

1. How to Add Load balancer to get effective 1200 Mbps or 1.2 Gbps  with what ever practical limitations

No load balancing will ever extend your existing download rates.
2x 100Mb connections are still 2x 100 Mb, you can only spread users across these two. But you won't get a single 200Mb connection.

[ashima]

Thank you jahonix,

Point Taken and understood.

With the available nos of ISPs and bandwidth + hardware available, what and which way is the best implementation ?

Would appreciate rough sketch and direction guidance if possible.

Regards
Ashima

[jahonix]

@ashima:

With the available nos of ISPs and bandwidth + hardware available, what and which way is the best implementation ?

You wrote "cable broadband 200Mbps each  x  6 nos"
Is that one cable coming to your facility? Since cable is a shared media across all attached users you won't even exceed the 200Mb. You would just route through another modem to the same central gateway. And if all cable modems get the same gateway from a single ISP then that will be another drawback for your project.

If you want it working reliably then your best bet is still: https://forum.pfsense.org/index.php?topic=111413.msg620518#msg620518

[ashima]

No, it is from 6 different ISPs.

And Surely I don't mind taking a professional help. Just that I want things to be clear in my mind before I ask for a professional help.

Coming to number of pfsense  boxes required. I'll need 1 box working as load balancer and another as content filter. A managed switch after that to take care of Rogue IP, AP and DHCP.

Can you provide  some help for this.

Regards,
Ashima

[ashima]

Actually,  Internet  will  be  :  6  different connections    from  6  different ISPs.

Coming to number of pfsense  boxes required.

I will  probably need 1 box working as load balancer  and    another as content filter.

A managed switch after that to take care of Rogue IP, AP and DHCP.

One more thing :    Is there a way to allot  Usage Quote  as  "x"  GBs per user    per week  or  per day  ?

Regards,
Ashima

[coxhaus]

Six different ISPs seems a little over the top for redundancy.  You are going to have 6 different default gateways to deal with.  Is it really worth the effort?

[ashima]

Well it is 3 different ISps with 2 connections from each ISPs to be precise. Its a 4 storey building with 400 users.

The idea is to provide reliable  fast internet to all the users. How many number of pfsense boxes will I need to act as load balancer, content filter. I'll be introducing a couple of managed switch.  A rough sketch of the network will be really appreciated.

Regards,
Ashima

[jahonix]

6 different ISPs as you wrote earlier or 3 ISPs with 2 lines from each?
Makes a real difference in gateways since each ISP usually hands out the same GW to different subscribers/lines. And routing two WANs to the same external gateway is … tricky at least.

Will it be cable, DSL or fiber connections and will you be using PPPoE?

What about the services your users need? Is that one company or a couple of different one, do they need fixed IPs for services they run on premise (like mail- or web-servers, own-cloud installs, ...)

Is that a new install, a redo or an upgrade?

How many company policies to implement?

pfSense scales pretty well with the hardware you throw at it. There's non need for separate boxes if HW is sized right. I would add a second unit as failover in a CARP cluster. But I would want to run it on more mature hardware like this:
https://store.pfsense.org/HIGH-AVAILABILITY-SG-4860-1U-pfSense-Systems-P47.aspx
or this:
https://store.pfsense.org/HIGH-AVAILABILITY-SG-8860-1U-pfSense-Systems-P48.aspx

Better run DHCP from pfSense, not the switches. Get switches with port security features to lock-down ports for unknown hosts and to dynamically assign VLANs to known ones.
User authentication is done where? You have one or several ADs/Radius servers around? Consider that for your DNS servers as well.

Sizing your gateway router/firewall is somewhere in the middle of the design process of your network. It shouldn't be your first thing to consider.

I can't help it but this project seems to be a bit too big for your current knowledge.
Before I repeat myself I better shut-up now...

[Guest]

XG-2758 from the pfSense store with stacked switches and 10 GBit/s uplinks to the DMZ and LAN switches
would be good. In the pfSense store it will be also a 4 Port LAN card able to buy too, that offers you more
ports.

[Keljian]

400 devices. It's a lot to manage with limited networking experience.

You are trying to use consumer hardware for an enterprise load. This is Bad.

Using a graphics card in the box is just silly. Integrated graphics can be configured to 32meg (which is nothing even on a 2gig ram setup), and considering you're not displaying anything, it isn't going to be used much if at all.

I would be looking at enterprise gear with support for this application.

If you must use that gear, I would be putting together two boxes, both with 2x i350v2-T4 cards, in redundant setup. I would probably opt for the i5 as the processor of choice, though an i3 would suffice in a pinch.

Then I would feed that into multiple separate vlans on numerous switches and access points as required.

As for IP locking and what not, I would use captive portal for dhcp for a start.

Really it does sound like you are out of your depth and should get a network engineer in.

[Derelict]

If it were me, I'd get enough Layer 3 switch ports to give a connection to each suite and enough public IP addresses to give everyone a /30, and install a HA cluster at the edge to take care of the multi-wan and the big picture. Let them worry about their own content filtering.

Since it's no longer a flat network you don't have to worry about rogue DHCP servers, rogue access points, or "IP guard" any more.

You can't run an ISP like you run an office network. Completely different things.

Don't buy graphics cards. Put your money into switches instead.

Since you'll have the edge firewall you can do typical ISP things like blocking outbound tcp/25.

400 devices is a lot.

400 devices is next-to-nothing.

[Keljian]

I corrected myself, 400 is a lot to manage for someone with limited networking experience

[Derelict]

@Keljian:

I corrected myself, 400 is a lot to manage for someone with limited networking experience

Two inside subnets of however-many devices is a lot for someone who doesn't know what they're doing.

[jahonix]

@Derelict:

…give everyone a /30, and install a HA cluster at the edge to take care of the multi-wan and the big picture. Let them worry about their own content filtering.

IIRC, we don't know about that yet. Might as well be only one company's big office and ashima is the IT guy by accident and/or interests.

Anyway, starting the network design process from the perimeter firewall is just the wrong direction and trying to size hardware from a more than incomplete picture might make it worse. But you may as well have nailed it. Who knows right now?

[Derelict]

I was keying on clues like:

Is there a way to allot  Usage Quote  as  "x"  GBs per user    per week  or  per day  ?

and

The idea is to provide reliable  fast internet to all the users.

People get way too concerned with the hardware necessary instead of the design. Buying gear is the easy part.

[ashima]

Hello everyone,

Let me explain the scenario.

The premises is a Coworking Space, a building of 4 floors and a total seating capacity of 400 seater.
There is no  file server or database servers or email servers  etc . . .  as its a coworking space environment.

Various Teams of different sizes are expected to hire the seating for a month or two ,  bringing their own laptops and using the internet provided by the premises.

To be able to provide comfortable internet browsing speeds for all the users  (400 users),  following plan comes to mind.

Plan was to take  6 Broadband Connctions from 3 ISPs as follows :-
200 Mbps Cable Connection from  provider - A    x 2 nos
200 Mbps Cable Connection from  provider - B    x 2 nos
32  Mbps DSL    Connection from  provider - C    x 2 nos

planning to terminate all the ISPs into a pfsense load balancer ,  followed by another pfsense machine configured with common content filer + DHCP Server + Captive Portal + Free Radius Server ( for usage quota in GBs for each team / user )

Planning to put a managed switch for each floor (4nos)  with VLAN Tagging per  AP,  in turn connected to  4 Access Points (ENgenius EAP350) with different SSIDs    as all the users will be on Wireless Network from their personal Laptops.

I am neither a Network Engineer not i have been hired by anyone,  doing this project due to sheer passion for pfsense.

Regards,
Ashima

[Derelict]

doing this project due to sheer passion for pfsense.

Interesting priority but OK. Most would probably do something like this to try to make more/some money.

So, like I thought, you want to be an ISP in a multi-tenant building. That really doesn't change much except you are going to need to deal with larger groups for whom one wired jack isn't enough and they want more than one location tied together without isolation.  Admin headache any way you do it. You can:

• Manually place certain ports on certain VLANs as needed. Admin overhead there.

• Use something like 802.1x to automatically place certain logins on certain VLANs. Admin overhead there maintaining authentication backend and helping people deal with 802.1x.

• This might actually be a use case for actual private VLANs, but you would still have the overhead in point 1 but you wouldn't have to allocate a separate layer 3 network. This has the overhead of making sure all your gear understands private VLANs on tagged ports. I have not yet seen a line of Access Points that does (but haven't looked lately). That causes problems.

• Give them one port and let them worry about their switching behind it or they can VPN between multiple ports.

[Derelict]

OK so no wired. That's easier.

You need to do a survey.  Place an AP and take a walk around your space with something like NetSpot. Only use 5GHz when you do this. 2.4GHz will cover better than that.

[ashima]

Thank you Derelict for the response.

This is what I am Planning now.

In each floor,

3    X  Engenius EAP300  running on same SSID connected to unmanaged switch.
4    X  Network Printers connected to same unmanaged switch.

The unmanaged switch from each floor is connected to 28 port Cisco SG300 Managed switch. The ports are protected so there is no communication between floors. This will prevent users from one floor sending Print command to Printer connected to another floor.

The SG300 Cisco is connected to Pfsense Box  #1 which will have the following settings :

1)  LAN IP 192.168.4.1/22    ( since I have ~ 400 users)
2)  DHCP server
3)  Common Captive Portal
4)  Freeradius to keep a check on each users monthly quota of Internet Usage.
5)  A simple proxy ( Squid + Squidguard) to prevent access to unwanted sites.

The Pfsense Box #1 is connected to a Load Balancer (PFsense Box #2).

So now there are no VLAns. ( The VLANs things were getting too complicated as it is Coworking environment with ~50-60 teams of different sizes, not feasible to provide that).

Are there any flaws in this setup. Is there something that I should take care.

Thank you all for your support. The reason I love Pfsense is the support I get from you all.

Regards,
Ashima