Network Design with Multi ISPs for 400+ Users

[jahonix]

6 different ISPs as you wrote earlier or 3 ISPs with 2 lines from each?
Makes a real difference in gateways since each ISP usually hands out the same GW to different subscribers/lines. And routing two WANs to the same external gateway is … tricky at least.

Will it be cable, DSL or fiber connections and will you be using PPPoE?

What about the services your users need? Is that one company or a couple of different one, do they need fixed IPs for services they run on premise (like mail- or web-servers, own-cloud installs, ...)

Is that a new install, a redo or an upgrade?

How many company policies to implement?

pfSense scales pretty well with the hardware you throw at it. There's non need for separate boxes if HW is sized right. I would add a second unit as failover in a CARP cluster. But I would want to run it on more mature hardware like this:
https://store.pfsense.org/HIGH-AVAILABILITY-SG-4860-1U-pfSense-Systems-P47.aspx
or this:
https://store.pfsense.org/HIGH-AVAILABILITY-SG-8860-1U-pfSense-Systems-P48.aspx

Better run DHCP from pfSense, not the switches. Get switches with port security features to lock-down ports for unknown hosts and to dynamically assign VLANs to known ones.
User authentication is done where? You have one or several ADs/Radius servers around? Consider that for your DNS servers as well.

Sizing your gateway router/firewall is somewhere in the middle of the design process of your network. It shouldn't be your first thing to consider.

I can't help it but this project seems to be a bit too big for your current knowledge.
Before I repeat myself I better shut-up now...

[Guest]

XG-2758 from the pfSense store with stacked switches and 10 GBit/s uplinks to the DMZ and LAN switches
would be good. In the pfSense store it will be also a 4 Port LAN card able to buy too, that offers you more
ports.

[Keljian]

400 devices. It's a lot to manage with limited networking experience.

You are trying to use consumer hardware for an enterprise load. This is Bad.

Using a graphics card in the box is just silly. Integrated graphics can be configured to 32meg (which is nothing even on a 2gig ram setup), and considering you're not displaying anything, it isn't going to be used much if at all.

I would be looking at enterprise gear with support for this application.

If you must use that gear, I would be putting together two boxes, both with 2x i350v2-T4 cards, in redundant setup. I would probably opt for the i5 as the processor of choice, though an i3 would suffice in a pinch.

Then I would feed that into multiple separate vlans on numerous switches and access points as required.

As for IP locking and what not, I would use captive portal for dhcp for a start.

Really it does sound like you are out of your depth and should get a network engineer in.

[Derelict]

If it were me, I'd get enough Layer 3 switch ports to give a connection to each suite and enough public IP addresses to give everyone a /30, and install a HA cluster at the edge to take care of the multi-wan and the big picture. Let them worry about their own content filtering.

Since it's no longer a flat network you don't have to worry about rogue DHCP servers, rogue access points, or "IP guard" any more.

You can't run an ISP like you run an office network. Completely different things.

Don't buy graphics cards. Put your money into switches instead.

Since you'll have the edge firewall you can do typical ISP things like blocking outbound tcp/25.

400 devices is a lot.

400 devices is next-to-nothing.

[Keljian]

I corrected myself, 400 is a lot to manage for someone with limited networking experience

[Derelict]

@Keljian:

I corrected myself, 400 is a lot to manage for someone with limited networking experience

Two inside subnets of however-many devices is a lot for someone who doesn't know what they're doing.

[jahonix]

@Derelict:

…give everyone a /30, and install a HA cluster at the edge to take care of the multi-wan and the big picture. Let them worry about their own content filtering.

IIRC, we don't know about that yet. Might as well be only one company's big office and ashima is the IT guy by accident and/or interests.

Anyway, starting the network design process from the perimeter firewall is just the wrong direction and trying to size hardware from a more than incomplete picture might make it worse. But you may as well have nailed it. Who knows right now?

[Derelict]

I was keying on clues like:

Is there a way to allot  Usage Quote  as  "x"  GBs per user    per week  or  per day  ?

and

The idea is to provide reliable  fast internet to all the users.

People get way too concerned with the hardware necessary instead of the design. Buying gear is the easy part.

[ashima]

Hello everyone,

Let me explain the scenario.

The premises is a Coworking Space, a building of 4 floors and a total seating capacity of 400 seater.
There is no  file server or database servers or email servers  etc . . .  as its a coworking space environment.

Various Teams of different sizes are expected to hire the seating for a month or two ,  bringing their own laptops and using the internet provided by the premises.

To be able to provide comfortable internet browsing speeds for all the users  (400 users),  following plan comes to mind.

Plan was to take  6 Broadband Connctions from 3 ISPs as follows :-
200 Mbps Cable Connection from  provider - A    x 2 nos
200 Mbps Cable Connection from  provider - B    x 2 nos
32  Mbps DSL    Connection from  provider - C    x 2 nos

planning to terminate all the ISPs into a pfsense load balancer ,  followed by another pfsense machine configured with common content filer + DHCP Server + Captive Portal + Free Radius Server ( for usage quota in GBs for each team / user )

Planning to put a managed switch for each floor (4nos)  with VLAN Tagging per  AP,  in turn connected to  4 Access Points (ENgenius EAP350) with different SSIDs    as all the users will be on Wireless Network from their personal Laptops.

I am neither a Network Engineer not i have been hired by anyone,  doing this project due to sheer passion for pfsense.

Regards,
Ashima

[Derelict]

doing this project due to sheer passion for pfsense.

Interesting priority but OK. Most would probably do something like this to try to make more/some money.

So, like I thought, you want to be an ISP in a multi-tenant building. That really doesn't change much except you are going to need to deal with larger groups for whom one wired jack isn't enough and they want more than one location tied together without isolation.  Admin headache any way you do it. You can:

• Manually place certain ports on certain VLANs as needed. Admin overhead there.

• Use something like 802.1x to automatically place certain logins on certain VLANs. Admin overhead there maintaining authentication backend and helping people deal with 802.1x.

• This might actually be a use case for actual private VLANs, but you would still have the overhead in point 1 but you wouldn't have to allocate a separate layer 3 network. This has the overhead of making sure all your gear understands private VLANs on tagged ports. I have not yet seen a line of Access Points that does (but haven't looked lately). That causes problems.

• Give them one port and let them worry about their switching behind it or they can VPN between multiple ports.

[Derelict]

OK so no wired. That's easier.

You need to do a survey.  Place an AP and take a walk around your space with something like NetSpot. Only use 5GHz when you do this. 2.4GHz will cover better than that.

[ashima]

Thank you Derelict for the response.

This is what I am Planning now.

In each floor,

3    X  Engenius EAP300  running on same SSID connected to unmanaged switch.
4    X  Network Printers connected to same unmanaged switch.

The unmanaged switch from each floor is connected to 28 port Cisco SG300 Managed switch. The ports are protected so there is no communication between floors. This will prevent users from one floor sending Print command to Printer connected to another floor.

The SG300 Cisco is connected to Pfsense Box  #1 which will have the following settings :

1)  LAN IP 192.168.4.1/22    ( since I have ~ 400 users)
2)  DHCP server
3)  Common Captive Portal
4)  Freeradius to keep a check on each users monthly quota of Internet Usage.
5)  A simple proxy ( Squid + Squidguard) to prevent access to unwanted sites.

The Pfsense Box #1 is connected to a Load Balancer (PFsense Box #2).

So now there are no VLAns. ( The VLANs things were getting too complicated as it is Coworking environment with ~50-60 teams of different sizes, not feasible to provide that).

Are there any flaws in this setup. Is there something that I should take care.

Thank you all for your support. The reason I love Pfsense is the support I get from you all.

Regards,
Ashima