Scaling PFSENCE 50,000+ Users
-
For the VPN It's not particularly heavy usage 100Mbps MAX mostly around 10-20Mbps.
We have server class H/W for this already, four identical machines ready for failover
-
For the VPN It's not particularly heavy usage 100Mbps MAX mostly around 10-20Mbps.
We have server class H/W for this already, four identical machines ready for failover
If you can provide the basic hardware specs/models, we can possibly give you some more relevant info/opinions/recommendations.
-
I can't help but to giggle at this thread. If you're an admin with the hardware to support 50,000+ users and a 1/2 million concurrent connections, then you can afford to pay the support fee to make sure PfSense can handle the load. Contact PfSense technical support directly and have your credit card ready. ;D
-
Whilst I agree with what you are saying I can completely see why the original poster did so.
It would seem like a waste to take out a support contract if the answer to the question has been 'not at all'. ;) (though I'm sure in fact they wouldn't take your money to tell you that!)
No harm in getting a few free opinions first.Steve
-
Keep in mind that each actual client connection results in two states - one in, one out. (client -> firewall, firewall -> wan or vice versa) so if you really need 500,000 simultaneous client connections, plan on enough RAM for 1,000,000+ states.
-
at about 100 states per user
My users average about 120 states each.
To the OP, what you want to do isn't terribly difficult, and not even all that expensive, but I would echo the others, pay the $600 and get yourself a support contract. I've used all of 30 minutes of mine but that 30 minutes was worth the cost.
If this is a critical system you should consider using two boxes & CARP for failover. It's worth it just for the "anytime" maintenance window.
-
Don't forget newer inte CPU's (Sandy Bridge and later) have AEI-NI which significantly speeds up the AES encryption. Something you will want to make sure you have if you want a decent amount of VPN capability.
-
Don't forget newer inte CPU's (Sandy Bridge and later) have AEI-NI which significantly speeds up the AES encryption. Something you will want to make sure you have if you want a decent amount of VPN capability.
As far as I know, that's not supported yet.
-
Also keep in mind, the pf process is still in a giant-locked single thread. Multiple cores won't help there.
-
Yup, which is why I recommend that most people go with a dual-quad or quad-core with as high a clock frequency as possible unless they're going to be using snort, squid, vpn, etc. where you'd be able to use more than 2 cores.
There have been quite a few people here who've bought expensive dual quads or hexes with low clock speeds, only to find out that they're slower than an i3 or i5.
-
Don't forget newer inte CPU's (Sandy Bridge and later) have AEI-NI which significantly speeds up the AES encryption. Something you will want to make sure you have if you want a decent amount of VPN capability.
As far as I know, that's not supported yet.
It may only be supported in 2.1, and in that case it will be out very soon.
-
Yup, which is why I recommend that most people go with a dual-quad or quad-core with as high a clock frequency as possible unless they're going to be using snort, squid, vpn, etc. where you'd be able to use more than 2 cores.
There have been quite a few people here who've bought expensive dual quads or hexes with low clock speeds, only to find out that they're slower than an i3 or i5.
I would not recommend multiple physical multi-core processors for a physical pfSense box. If you're stuck with single or even 2 threads, having 2 separate processor dies can slow individual processes down if there aren't enough threads to occupy the logical CPUs. Many OSs will cycle the threads between logical CPUs, which, when on a single physical multi-core processor, that's fine since the cache is local to the cores, but when it cycles between sockets it has to transfer the cache over the CPU bus (or pull from main memory.) While it can do it fairly quickly, it's not helping. Some multi-core designs share a cache, but even if they don't, the cores can usually snoop in to the other core's cache really fast.
For a pfSense box, I would recommend to not have multiple physical CPUs; multiple sockets on the motherboard is fine, just don't occupy it. Not only would you be spending extra on a CPU that you're not effectively using, both in the initial purchase, but also the power to run it, but you may actually be slowing the machine down.
Of course, if it's a VM host, cores are handy so the extra sockets may be beneficial, but that goes the other way from the original question, here.
I'm right behind your clock cycle recommendation, though. So a fast dual or quad core is probably a sweet spot for a high volume pfSense install. "Server" grade boxes will use a Xeon, but that's not going to buy you much here. The main thing a Xeon really gives you is usually an option for more cache, better multiple physical CPU support, and the possibility of more cores, neither of which is going to make a huge difference for you. It'll be hard to get away from it in a "serious" OEM server, like a Dell or HP, but you can find lower end servers with "desktop" type CPUs for less, it just might not have the options for dual power supplies and such; which I would want for a router with a bunch of people behind it.
I don't think the box needs to be parallel large, just singularly fast. Same with other components, like a smaller amount of fast RAM rather than gobs of slower RAM. (When I say smaller amount, I mean in the 16GB to 32GB range, rather than the 192GB+ side.) You don't need a bunch of disk spindles, it's not a file server or a database, I don't think any data transfer tasks wait on disk reads or writes (after boot-up, of course), as long as there is enough RAM it shouldn't hit swap (much), so a single or mirrored SSD is good.
Fast NICs on a fast bus, but, and this may sound counter intuitive, maybe limit smaller alternative networks to 100Mb physical links (DMZ, guest wireless, etc.) to reduce the potential routing load. Not a hard recommendation, just something to keep in mind, that "faster always" is not always helpful, there are other (if Low-Fi) ways to introduce bandwidth limits. Especially with things like a Guest Wireless network with a bunch of APs that could easily introduce a lot of bandwidth at times; it takes CPU from your pfSense box to do bandwidth throttling, off-load it to "hardware". Same with a DMZ network, or if you have a DEV network that's islanded and, well, no offense to DEVs, but ya'll do some wacky stuff sometimes and generate a lot of not always legit bandwidth, sorry, you get 100Mb.
If you don't have any of that, and you're really doing simple routing for 50k people, consider breaking it up in to multiple routers. Even if large amounts of people need to be on the same broadcast network(s), you can still run multiple DHCP servers with overlapping subnets, just serving portions each, then each DHCP scope points to a different router. You've effectively load balanced your routing without fancy hardware. With 4 medium machines, each in a active/passive CARP pool, you've got some pretty robust routing for fairly cheap. Need more, add more and re-organize your DHCP scopes. You could do this with some fairly cheap 1U [insert OEM or whitebox manufacturer/reseller here] single core servers. Many OEM servers have at least 2 good Gb NICs, some 4. Hell, there's 1/2 U servers, or more accurately, 2 servers in 1U that might work well for this. I would just be sure to separate the CARP pools across chassis so if a whole chassis went down you don't lose both the primary and secondary of a CARP pool.
Hell, you could put together a proof of concept with a few desktops, if you have 'em handy. It's very common to find older desktops for very cheap. Off the top of my head, you could get old Dell GX260's, I see 'em fairly often for $50 or less in local "old-stuff" / PC Recycler shops. Toss in some Intel 100Pro PCI nics for the WAN side, the onboard should be an Intel 1000Pro (maybe Broadcom, mine are Intel.) They probably have at least 1GB of ram, maybe 2, should be enough for some light proof of concept testing. Oh, these should have Hyper-Threading P4's, around 2.8Ghz.
For the most part, that's mainly testing failover and load balancing scenarios. The DHCP servers should be a first respond decision, so, assuming the machines are roughly similar, they should balance themselves decently (if a machine is getting hit hard, odds are, DHCP is going to respond slower anyway, so the idle machine should service the call.)
If you want more throughput testing, Dell GX280's have PCI-Express x16 slots and SATA, which means if you get the dual or quad port Intel cards now, you'll be able to simply migrate them to your production environment, same with SSDs. Other popular options are HP DC7600's (P4 or Pentium-D, 4 GB of RAM, PCI-Express x16), HP DC7700's (Pentium-D or C2d, 8GB, PCI-E x16) HP DC7800 (Pentium-D, C2D, Quad, 8GB, PCI-E x16.) All those are constantly available on ebay for between $70 and $200 shipped (C2Q might be a bit more.)
Congratulations if you've read this far ;)
