PfSense enterprise grade setup



  • Hi guys,

    We have been running pfSense in our organization for about two and a half years now and we started with a single vmWare ESXi free version host running a single pfSense virtual appliance (in a single desktop PC). About a year and a half ago we moved on to clustered Netgate c2758 hardware appliances and as were(and still are) are very happy with the Netgate c2758 hardware (has not failed us at all so far) then we decided to proceed with also setting up remote installations for our customers.

    The first two remote installations we did with Netgate C2758 appliances clustered as we were familiar with the hardware by that time but I did have some people (mostly my colleagues in other companies) raising concerns over the true "enterprise grade" availability of these appliances as these appliances have a single hard drive and a single power supply only. A hard drive or power supply failure thus would render one of the cluster members unusable and while our configuration is redundant and failover would happen within seconds(we have tested this numerous times) then having to fly to another country to service hardware and re-install from backup the failed unit is not very attractive.

    As I have been working with Supermicro hardware for some years now (for 24/7 real time video encoding and streaming) and I have observed that over time (around 2 years mark or so) the PSU units can fail(as they would fail with any other manufacturers hardware too) then I decided to build our own fully redundant configuration based on Supermicro hardware for the pfSense firewall clusters(we don't assemble the units from parts, the equipment would be assembled by a Supermicro based on our specifications).

    So, I came up with 2 configuration options, both are presented below. Option 1 has less CPU cores/threads (8 cores with 8 threads) and option 2 has quite a few more (16 cores and 32 threads). While for firewalling/nat and load balancing to web based applications in our application cluster option 1 is more than enough (we were easily able to reach 3000 HTTP requests per second to a remote site using the Netgate C2758 cluster firewalling and load balancing our application cluster, at that rate we ran out of internet bandwidth as we have 30Mbps only, the firewalls were quite capable to take in much more along with application server cluster that it was load balancing traffic in for) then we are also considering to do IDS and IPS in the near future, thus option 2 comes to picture.

    I have ordered the 2 below options fully as it is and these should be delivered in about 2 weeks or so but I am wondering if anyone here would have any comments/suggestions/thoughts about what we are going into here.

    Our use case and configuration is as below per remote site:

    Internet bandwidth: 25Mbps with 1Gbe redundant handover from the ISP (we utilize CDN heavily, thus our internet bandwidth requirement is low)
    Site-to-Site VPN using pfSense OpenVPN configuration (for remote management only)
    2x Catalyst 2960-X in stack (we have previously used 2x Cisco sg500x-48 in stack, works well but has single PSU per unit)
    2x pfSense (based on one of the below configurations, clustered with CARP) with interfaces in lagg/LACP
    1x Supermicro Microcloud blade systems for application and database clusters (8x E5-2670 V2, 128GB RAM, vmWare vSphere, LAMP)
    1x dual controller SSD only ISCSI storage with 4x 10Gb ISCSI interfaces.
    We don't firewall ISCSI traffic and we do use Layer 3 capabilities on the switches.
    Incoming internet traffic is firewalled to ports 80 and 443 only and we use pfSense load balancing (we don't use HAproxy though).
    We do not use pfSense IDS/IPS yet (but are considering) as our ISP offers this service as of today.

    Below are the 2 pfSense configurations I mentioned earlier.

    pfSense configuration 1, consists of 2x units as described below.

    System: Supermicro 6017R-M7RF
    2x Intel Xeon E5-2609 V2 (8 cores total, 8 threads total @ 2.5 GHz)
    4x 4GB DDR3-1866 1Rx8 ECC REG.
    4x 3.5" 500GB SATA 6Gb/s 7.2K RPM 64M RE4 HDD
    1x 4-port GbE card based on Intel i350, OEM and Bundle only AOC-SGP-I4
    Redundant PSU configuration

    pfSense configuration 2, consists of 2x units as described below.

    System: Supermicro 6017R-M7RF
    2 x Intel Xeon E5-2650 V2 (16 cores, 32 threads total @ 2.6 GHz - 3.4 GHz)
    4 x 8GB DDR3 1866 ECC Reg.
    4x 3.5" 500GB SATA 6Gb/s 7.2K RPM 64M RE4 HDD
    1x 4-port GbE card based on Intel i350, OEM and Bundle only AOC-SGP-I4
    Redundant PSU configuration

    Sorry for the long post and bad English.

    Martin



  • Maybe I'm reading your use case wrong, but either of those configurations sounds like way too much hardware for your scenario.  Not that it's going to hurt, but IMHO you're wasting money.  I'd be more concerned with redundant PSUs and NIC count, and less concerned with core count and lots of RAM.  I'd go with a single CPU and a mirrored disk config; no need for high speed drives.

    For comparison, we're running an old HP DL-360 Gen 5 with a Xeon E5420 and 4GB of RAM and it never comes close to full utilization.  It's backed up by a VM on ESXi as a failover pair, and it's hard to tell the difference when it fails over.

    As always, your mileage may vary, but with a single modern CPU you should be able to aggregate 4 NICs and achieve as much throughput as you need.



  • Supermicro mainboard socket 1150
    2 x 8 GB - 16 GB DDR3 - 1600L EEC RAM
    Enterprise SSD Samsung840 Pro or Intel 530
    Intel Xeon E3-12xxv3 / 4 cpu cores / 3,x GHz
    Supermicro 1U or 2U chassis with dual PSU integrated
    Intel I210-T1 server network adapter as dedicated WAN port
    Comtech AHA AHA363PCIE 5 GBS de/compression accelerator card

    Would do the job more energy efficient or power saving, but powerful enough.



  • @whosmatt:

    Maybe I'm reading your use case wrong, but either of those configurations sounds like way too much hardware for your scenario.  Not that it's going to hurt, but IMHO you're wasting money.  I'd be more concerned with redundant PSUs and NIC count, and less concerned with core count and lots of RAM.  I'd go with a single CPU and a mirrored disk config; no need for high speed drives.

    For comparison, we're running an old HP DL-360 Gen 5 with a Xeon E5420 and 4GB of RAM and it never comes close to full utilization.  It's backed up by a VM on ESXi as a failover pair, and it's hard to tell the difference when it fails over.

    As always, your mileage may vary, but with a single modern CPU you should be able to aggregate 4 NICs and achieve as much throughput as you need.

    Hi Matt,

    Thank you for the reply.  I did go with more CPU power and RAM as I plan to run IDS and IPS also in there but in general you are right, it is possibly quite an overkill. I also have run pfSense on HP Proliant equipment, quite similar to yours in specifications, it is fairly enough. I did run pfSense as a VM on those Proliants and it does indeed work quite well. My long term goal is also to have a more powerful firewalls with the first purchase as I might need to later use them as a dedicated database cluster and possibly downsize the firewall specs in terms of CPU count into some other appliance.

    Martin



  • @BlueKobold:

    Supermicro mainboard socket 1150
    2 x 8 GB - 16 GB DDR3 - 1600L EEC RAM
    Enterprise SSD Samsung840 Pro or Intel 530
    Intel Xeon E3-12xxv3 / 4 cpu cores / 3,x GHz
    Supermicro 1U or 2U chassis with dual PSU integrated
    Intel I210-T1 server network adapter as dedicated WAN port
    Comtech AHA AHA363PCIE 5 GBS de/compression accelerator card

    Would do the job more energy efficient or power saving, but powerful enough.

    Hi Frank,

    Thank you for the reply. Your specification is something that I might try next. In regards to your suggestion to have a dedicated WAN port though I would not do that, it removes redundancy, rather have lagg/LACP running on multiple ports and use the WAN port as a vlan in there. We have had some events here where network cables go bad or someone disconnects them by mistake, a dedicated port solution would impact the network.

    Martin



  • @whosmatt:

    Maybe I'm reading your use case wrong, but either of those configurations sounds like way too much hardware for your scenario.  Not that it's going to hurt, but IMHO you're wasting money.  I'd be more concerned with redundant PSUs and NIC count, and less concerned with core count and lots of RAM.  I'd go with a single CPU and a mirrored disk config; no need for high speed drives.

    Indeed, this sounds like overkill. But I think your C2758 things are too slow, you should send them to me. Really, trust me!

    Use a chassis with redundant PSU and perhaps a mirror setup of SSDs. I would recommend those Intel DC S3500 drives. Two of those in a mirror is also overkill. 8 GB of drive space should be enough for anybody ;)

    No, but really, there are some things that don't change much. There are some people and magazines believing you would need the most state of the art computer for video editing. But before HD came along they were telling us the same thing for over 15 years. Of course it was valid, a long time ago. But sometimes the computers change a lot, and the requirements stay the same.

    I use a C2558 platform. The CPU has so little to do it's building its own greenhouse and managing a few blogs. With 8 GB of memory it still has 93% free. Even with IDS, and all sorts of gadgets, this isn't going to use a lot of resources. What it does need is IPMI, a nice PSU and a disk that's solid enough.

    That stuff you ordered, it's all trash. Just send it to me, I will be happy to help you get rid of your junk.

    Seriously, if you have broken PSU's, maybe we can open them up and look if anything is broken. If you were living close by it would be my pleasure. (I love fixing things like that.)



  • Hi,

    The C2758 is definately a nice piece … for it's purpose. But let's imagine a bit more demanding situation here.

    ---The challenge---

    1. All of your firewall cluster is installed in another country 4 hours flight away from where you are.
    2. This firewall cluster has to firewall, load balance and run IDS/IPS on hundreds of incoming requests per second from the internet (we actually got as far as over 3000 requests per second firewalled and load balanced to application pools before we ran out of internet bandwidth on that circuit, that Netgate C2758 probbably would deliver quite a lot more).
    3. If you miss for some reason 10% of those requests, your customers will not be happy at all

    ---The disaster situation---

    1. You don't run your storage devices in a redundant way in your firewalls and the single disk fails, whatever the firewall was doing, it is lost for good, the request initiator has to send a new request
    2. You don't have redundant PSUs and your hosting partner is doing power maintenance, your active firewall member goes down, again, whatever the firewall was doing, it is lost for good, the request initiator has to send a new request
    3. In a case where a disk or a PSU fails, you suffer, need to fly for a few hours and to replace a failed part in the equipment, not a nice situation at all

    While the C2758 chip is quite an amazing one (we have over 50 devices with them so far doing different things) then having it running in an appliance that has basic single points of failure like hard drives and power supply units will one day for sure ruin your day.

    By the way, I'm not going to give away or sell the Netgate C2758 units we have, that is for sure :) These short depth, low voltage and quiet appliances are really good for R&D and experimentation purposes or to run them in environments where it is possible to access and fix them when something fails and 1 second delay for cluster member failover is not an issue.

    Martin



  • Uh, no reason you can't use those boards in a chassis with a redundant power supply. And two hard drives.