The perferct pfSense box 2016?



  • Hi all,

    after a long period of lurking this forum and testing pfSense on virtualbox, I decided that I reallyreally wanted to have my own pfSense box at home.
    So after checking out the prices of the prebuild boxes on the pfSense store, and me being from Europe (prices are too high… :(), I decided to build my own.
    It's in production now, and it works really well, so I wanted to share my config in case someone wants to build his own as well and use some hardware known to work:

    • Super Micro SUPERMICRO A1SRI-2758F - Motherboard - Mini-ITX - Intel Atom C2758 - USB3.0 - 4 x Gigabit LAN - Onboard-Grafik (A1SRI-2758F-O)

    • MS-Tech CI-57 - Ultra Small Form Factor - Mini-ITX - 120 Watt (CI-57-120)

    • SanDisk SSD - SSD - 64GB - 6,4 cm (2.5") - SATA-600 (SDSSDP-064G-G25)

    • Kingston KVR16LSE11/8HB

    It cost me exactly 500€.

    It is probably overkill for my 100Mbit/s Down, 30Mbit/s Up wan line, but it does what it has to do.



  • Sounds nice! Wish I had access to that hardware when I built mine.



  • I recently did a similar build, but I opted for a low power full Haswell chip instead of Atom.  (I'm a little bit biased when it comes to Atom based chips and their capabilities).

    My build was:

    • Supermicro X10SLV-Q motherboard with dual onboard Intel NIc's (i210AT and i217)
    • Core i5-4570T (Haswell dual core with HT, base clock 2.9Ghz, Turbo 3.6Ghz)
    • 2x 1GB 1600MZ 1.35v DDR3L SO-DIMM's
    • 16GB Sandisk USB stick (to boot from)
    • Mini-Box M350 case
    • 60w PicoPSU kit with 60W power brick.
    • Rosewill low profile CPU cooler

    My total cost was just under $400 (US), but I saved some cost by buying the motherboard with an open box discount and getting the CPU on eBay.

    I'm pretty happy thus far, I have low power use during regular loads (12-13W  at the wall) due to using power saving features, the PicoPSU (which is very efficient) a slow fan and turning off HT.

    I think you'll be very happy with your hardware, some comments though:

    1.) 8 Cores are probably not the best fit for pfSense.  pfSense tends to benefit more from fewer fast cores than from more slower cores.  I'd go with 1-2 really fast cores over more slower cores.  Unless you are doing some REALLY heavy lifting with that box, they will probably be fine either way though.

    2.) Unless you have either multiple LAN's or multiple WAN's, more than 2 ethernet ports are a waste.  A lot of people buy quad port adapters hoping to use the extra ports like switches, but that's not how they work.  You CAN bridge them, and make them behave like a switch, but performance will be rather low, as this is not what they are designed for.  An Actual switch will be much better for this task.

    3.) 8GB of RAM is total overkill for a pfSense build.  pfSense doesn't use very much RAM unless you use lots of packages, and even when you do, it's tough to get it to use a lot.  1-2GB is more than enough for most.

    4.) 64GB is an absolutely HUGE amount of storage for pfSense.  The base install without adding packages is under 600MB.  Add a few packages you can add to that a bit, but you likely won't add much.  I've run pfSense for 5 years as a virtual machine with only a 4GB virtual drive image.  It can be difficult to find SSD's that are smaller than 64GB which is probably why you went with that, but many people successfully boot their systems from USB sticks or use small SATA DOM's to avoid wasting a large drive on something that only requires a handful of GB.

    Anyway, I think you'll be very happy with the performance of that build.  It should do everything you need it to :)  There might ave been a few opportunities for cost savings, but as long as you are happy with it, who cares :p



  • @mattlach:

    I'm pretty happy thus far, I have low power use during regular loads (12-13W  at the wall) due to using power saving features, the PicoPSU (which is very efficient) a slow fan and turning off HT.

    I think you'll be very happy with your hardware, some comments though:

    1.) 8 Cores are probably not the best fit for pfSense.  pfSense tends to benefit more from fewer fast cores than from more slower cores.  I'd go with 1-2 really fast cores over more slower cores.  Unless you are doing some REALLY heavy lifting with that box, they will probably be fine either way though.

    2.) Unless you have either multiple LAN's or multiple WAN's, more than 2 ethernet ports are a waste.  A lot of people buy quad port adapters hoping to use the extra ports like switches, but that's not how they work.  You CAN bridge them, and make them behave like a switch, but performance will be rather low, as this is not what they are designed for.  An Actual switch will be much better for this task.

    3.) 8GB of RAM is total overkill for a pfSense build.  pfSense doesn't use very much RAM unless you use lots of packages, and even when you do, it's tough to get it to use a lot.  1-2GB is more than enough for most.

    4.) 64GB is an absolutely HUGE amount of storage for pfSense.  The base install without adding packages is under 600MB.  Add a few packages you can add to that a bit, but you likely won't add much.  I've run pfSense for 5 years as a virtual machine with only a 4GB virtual drive image.  It can be difficult to find SSD's that are smaller than 64GB which is probably why you went with that, but many people successfully boot their systems from USB sticks or use small SATA DOM's to avoid wasting a large drive on something that only requires a handful of GB.

    Anyway, I think you'll be very happy with the performance of that build.  It should do everything you need it to :)  There might ave been a few opportunities for cost savings, but as long as you are happy with it, who cares :p

    FreeBSD pf supports multi-threading now, so pfSense will also be beneficial from a multi-core CPU.
    The Intel Avoton/Rangeley Atom CPU are server grade, 8-core one was proven by a benchmark that it runs faster than i3, and about 50% of an entry level Xeon, and it does support ECC memory, also….at full load it's eating only 20W power and doesn't require cpu fan (I am using N2930 now, I don't even have a case fan for it).

    Quad port at least allows you to have dual WAN, I did this at home with 1 x 1000M + 1 x 500M FTTH broadband

    To OP: Probably A1SRi-2358F will fit your need, this platform is capable to run a 1000M WAN-LAN NAT throughput already, and TDP is even lower.



  • @edwardwong:

    FreeBSD pf supports multi-threading now, so pfSense will also be beneficial from a multi-core CPU.

    Well, FreeBSD has always supported multithreading, or at least as long as theyre have been multi-CPU servers, which is quite a while.  That was never the problem.

    It's whether or not the software is written for multithreading that is the issue.

    I'm pretty certain that the NAT and Firewall functions are still single threaded, and that's typically where most of the load is on basic setups.

    Other installed packages, QoS and other types of load, will likely make use of more threads, if you are doing a more complex setup, but I am fairly certain that the basic NAT and firewall operations are not very well threaded.

    if I recall, this is because the type of operations they perform don't lend themselves very well to threading, without thread locks and other problems.  Not all workloads can be multithreaded.

    So on a many-core pfSense box, you'll likely have one core loaded up with all your NAT and Firewall load, and everything else spread across the other cores.  So, one, highly loaded core, and the rest with very light load.



  • @mattlach:

    @edwardwong:

    FreeBSD pf supports multi-threading now, so pfSense will also be beneficial from a multi-core CPU.

    Well, FreeBSD has always supported multithreading, or at least as long as theyre have been multi-CPU servers, which is quite a while.  That was never the problem.

    It's whether or not the software is written for multithreading that is the issue.

    I'm pretty certain that the NAT and Firewall functions are still single threaded, and that's typically where most of the load is on basic setups.

    Other installed packages, QoS and other types of load, will likely make use of more threads, if you are doing a more complex setup, but I am fairly certain that the basic NAT and firewall operations are not very well threaded.

    if I recall, this is because the type of operations they perform don't lend themselves very well to threading, without thread locks and other problems.  Not all workloads can be multithreaded.

    So on a many-core pfSense box, you'll likely have one core loaded up with all your NAT and Firewall load, and everything else spread across the other cores.  So, one, highly loaded core, and the rest with very light load.

    I may have to take part of this back.

    Apparently some work has been done on multithreading pf, which when I last read up on this was considered to be a workload that COULD NOT be multithreaded well.

    I wonder if it is - like I've seen elsewhere - multithreading in name only, or if it actually load balances well between cores.  Maybe I can throw htop on it and load it up in a console while I do some performance tests to see what it looks like.

    Damn, it's amazing how quickly things can go from "CAN NEVER BE DONE" to "DONE" :p  it feels like just yesterday I was reading up about this, but now that I think about it, it was probably 2010-2011 some time.  While that feels like just yesterday, I guess it's 5+ years ago now :p



  • @mattlach:

    @mattlach:

    @edwardwong:

    FreeBSD pf supports multi-threading now, so pfSense will also be beneficial from a multi-core CPU.

    Well, FreeBSD has always supported multithreading, or at least as long as theyre have been multi-CPU servers, which is quite a while.  That was never the problem.

    It's whether or not the software is written for multithreading that is the issue.

    I'm pretty certain that the NAT and Firewall functions are still single threaded, and that's typically where most of the load is on basic setups.

    Other installed packages, QoS and other types of load, will likely make use of more threads, if you are doing a more complex setup, but I am fairly certain that the basic NAT and firewall operations are not very well threaded.

    if I recall, this is because the type of operations they perform don't lend themselves very well to threading, without thread locks and other problems.  Not all workloads can be multithreaded.

    So on a many-core pfSense box, you'll likely have one core loaded up with all your NAT and Firewall load, and everything else spread across the other cores.  So, one, highly loaded core, and the rest with very light load.

    I may have to take part of this back.

    Apparently some work has been done on multithreading pf, which when I last read up on this was considered to be a workload that COULD NOT be multithreaded well.

    I wonder if it is - like I've seen elsewhere - multithreading in name only, or if it actually load balances well between cores.  Maybe I can throw htop on it and load it up in a console while I do some performance tests to see what it looks like.

    Damn, it's amazing how quickly things can go from "CAN NEVER BE DONE" to "DONE" :p  it feels like just yesterday I was reading up about this, but now that I think about it, it was probably 2010-2011 some time.  While that feels like just yesterday, I guess it's 5+ years ago now :p

    Actually the history of FreeBSD's pf becomes "multi-threading capable" was just about a year, for pfSense it's v2.2, so really not that long time ago.

    If you've take a look to the thread with my new build, I did included some screen captures, on my N2930 platform (quad-core processor), running 1Gbps throughput in single direction will use about 30% cpu load, to a quad-core system, this means just a little bit more than full load on single core. A bi-directional test showing around 55-60% loading, simple calculation you'll know it's occupying just more than 2 cores, so the multithreading is working pretty well!



  • A bi-directional test showing around 55-60% loading, simple calculation you'll know it's occupying just more than 2 cores, so the multithreading is working pretty well!

    Please don´t forget that your Internet account is not PPPoE based and there fore multiple CPU cores are running
    or will be in usage. If PPPoE will be in use then that will be not true, because that is at the moment only a single
    CPU core threated part. So both can´t be really compared against in my eyes. And on top of this a CPU core is
    not really comparable to any other CPU core, likes an Intel Atom @2,16GHz against an Intel Xeon E3 or E5 core
    @3,0GHz.

    And on top of this might be the point that not really often the CPU will not be really saturated, but more the
    memory system of that pfSense appliance! So a really fast amount of RAM will be really important too in my eyes.



  • @edwardwong:

    Actually the history of FreeBSD's pf becomes "multi-threading capable" was just about a year, for pfSense it's v2.2, so really not that long time ago.

    You are confusing multithreading capability of the operating system itself, and that of pf, the software that handles firewall/NAT and other IP transactions.

    FreeBSD itself has been able to run multithreaded code since the introduction of SMP systems, some point in the 90's.

    pf, the software that puts the pf in pfSense - however - has only very recently been multithreaded.  (And I am still not convinced it does it well, based on previous statements I have read that pf just wasn't suitable for multithreading.  (Not all code is, in fact most code has trouble in one way or another with threading)

    It is a common misconception in hardware circles, that if only software developers weren't lazy, all code would be better threaded, and able to fully take advantage of their many core systems..  The truth is that a lot of workloads simply can not be threaded.



  • @mattlach:

    @edwardwong:

    Actually the history of FreeBSD's pf becomes "multi-threading capable" was just about a year, for pfSense it's v2.2, so really not that long time ago.

    You are confusing multithreading capability of the operating system itself, and that of pf, the software that handles firewall/NAT and other IP transactions.

    FreeBSD itself has been able to run multithreaded code since the introduction of SMP systems, some point in the 90's.

    pf, the software that puts the pf in pfSense - however - has only very recently been multithreaded.  (And I am still not convinced it does it well, based on previous statements I have read that pf just wasn't suitable for multithreading.  (Not all code is, in fact most code has trouble in one way or another with threading)

    It is a common misconception in hardware circles, that if only software developers weren't lazy, all code would be better threaded, and able to fully take advantage of their many core systems..  The truth is that a lot of workloads simply can not be threaded.

    No….I didn't confuse.....see my phrase 'FreeBSD's pf becomes "multi-threading capable"' , which I was focused on PF. I knew *BSD has multithreading capability long time ago but not for PF. If I remember correctly PF started to have multithreading support was since FreeBSD 10, which is what 2.2 based on.



  • 2 8GB RAM Module, DDR3L 1600MHz  Kingston, KVR16LN11/8 
    1 AMD FX-6-Core Black Edition, 6-Core Processor, AMD FX-6300
    1 Asus M5A97 LE R2.0, MotherBoard, Asus M5A97 LE R2.0
    1 PRO/1000 PT Quad Port Server Adapter, Ethenet Card, Intel D47316-004
    1 ATX Mid Tower Case, Computer Case, Deep Cool TESSERACT BF
    1 2 TB HDD/64MB Cache SATA, Hard Drive, Toshiba P300 HDWD120XZSTA

    Total $409.23
    Avg Cost per item $58.46

    I have been monitoring this pfsense box and have not even come close to 10% total usage with heavy usage.  I have OpenVPN, Backup, RRD Summary and full Squid Packages running. I have 38 varying devices from phones to computers to bluray players to chromecast. With almost all of them running internet connected activities at the same time my cpu maxed out at about 11% my memory max was around 14% and load average is now about 5.2.  This is truly overkill for a system like this but I just needed the functionality and I wanted some level of "future proofing" for the next 5 years.  Most of these parts were on sale so it is a good setup for me. All other networking is gigabit with cat6 cables and wireless ac access point.  My backups are sent to my CentOS box nightly with 1TB dedicated to just these files to keep some archives "just in case" (I'm a bit paranoid).  That CentOS box has 5 4TB HD's in RAID 5 and that is box is also encrypted archived at friends house several miles away on his CentOS box (his is archived on mine also).



  • Axiomtek has also very nice boxes in the desktop or 19" 1U form factor.
    With additional add on modules for the "R" (rack mount) series
    NA342 & NA342R
    NA361 & NA361R



  • @mattlach:

    I recently did a similar build, but I opted for a low power full Haswell chip instead of Atom.  (I'm a little bit biased when it comes to Atom based chips and their capabilities).

    Avoton though technically an Atom was designed as a server chip. Intel severely limits how this chip can be used because of how good it is. It's not as good as a Xeon but it's a very good low power chip designed for server applications. TDP is 20W if I recall correctly at 2.4ghz and its got 8 real cores (no hyperthreading fake cores) supports all the virtualization extensions AES extension, and up to 64gb of ECC memory. You won't find all that in anything but a Xeon at twice the price and 2 to 4 times the power consumption.

    Avoton is the perfect firewall chip IMO. Pfsense even sells one as their highest end hardware. https://store.pfsense.org/XG-2758/



  • Avoton though technically an Atom was designed as a server chip.

    Both are "server grade" SoCs and both are Intel Atom platforms, they are split into two platforms
    Avoton is more for servers likes Apache and Samba servers or NAS devices and the Rangeley is more
    for network appliances such as firewalls and routers.
    Rangeley comes with AES-NI and Intel QuickAssist
    Avoton comes with AES-NI and TurboBoost

    Intel severely limits how this chip can be used because of how good it is. It's not as good as a Xeon but it's a very good low power chip designed for server applications. TDP is 20W if I recall correctly at 2.4ghz and its got 8 real cores (no hyperthreading fake cores) supports all the virtualization extensions AES extension, and up to 64gb of ECC memory. You won't find all that in anything but a Xeon at twice the price and 2 to 4 times the power consumption.

    Yes this might be right on the first look, but on the second view a real Xeon E3-12xxv3
    is really heavy routing multiple 1 GBit/s at the WAN and strong enough to run a fully
    featured pfSense UTM device. There will be nothing you are missing. And better then
    the common Intel Core i3, i5 and i7 CPUs related to the power consuming.

    Avoton is the perfect firewall chip IMO. Pfsense even sells one as their highest end hardware. https://store.pfsense.org/XG-2758/

    It is the Intel Atom C2x58 ("Rangeley") platform
    or SoC and not the Avoton which they are selling !



  • I wanted a simple, reasonably low energy use set up. Went for the following, using vlans with the switch:

    $175  PC:  Intel NUC BOXNUC5PPYH Barebone Kit - Pentium N3700
    $20    RAM:  Kingston SO-DIMM KVR16LS11/4 135V (Low Voltage) 4G DDR3 1600 Notebook Ram
    $25    SSD:  32Gb SATA3 2.5inch
    SWITCH:  I already had a D-Link DGS-1100-16 16 Port Gigabit Switch, so used that. Otherwise would have used something like:
    $34  TP-Link TL-SG105E 5-Port Gigabit Easy Smart Switch
    –-------------------
    $254  TOTAL

    Works just fine for me.



  • Here is what I ordered direct from PC Engines, with 2 extra AC adapters, it was $196 including 3 day shipping from Switzerland to Oregon

    http://pcengines.ch/apu2c4.htm

    1      apu2c4  APU.2C4 system board 4GB
    1      case1d2u        Enclosure 3 LAN, alu, USB
    3      ac12vus2        AC adapter 12V US plug for IT equipment
    1      msata16d        SSD M-Sata 16GB MLC Phison

    Without the extra AC adapters, I think this would ship for about $170.  It can run a couple hundred mbps worth of OpenVPN, and about 600mbps of basic NAT/routing traffic at about 8w total consumption.



  • @Zebibyte:

    It can run a couple hundred mbps worth of OpenVPN, …

    I agree a great little board… but that seems quite high for OpenVPN on that board, how did you test ?

    I would not expect any more than 40 Mbps for a single OpenVPN connection.



  • I would not expect any more than 40 Mbps for a single OpenVPN connection.

    The APU2 comes with 4 Core CPU and only the PPPoE WAN part is single core using, the entire
    OpenVPN part is fully multi CPU core usage and so you will see perhaps numbers owed to this
    circumstance that you was not expecting before. But I would be counting more on the AES-NI
    and IPSec (AES-GCM) that should be more pushing the entire VPN part, for sure not OpenVPN
    but really fast.



  • @BlueKobold:

    I would not expect any more than 40 Mbps for a single OpenVPN connection.

    The APU2 comes with 4 Core CPU and only the PPPoE WAN part is single core using, the entire
    OpenVPN part is fully multi CPU core usage and so you will see perhaps numbers owed to this
    circumstance that you was not expecting before. But I would be counting more on the AES-NI
    and IPSec (AES-GCM) that should be more pushing the entire VPN part, for sure not OpenVPN
    but really fast.

    I just tested my APU2, (on Linux in my test), disabled lzo-compression, "cipher AES-256-CBC" and consistently saw 58-62 Mbps using iperf.  Note iperf was not running on the APU2, and the APU2 was an OpenVPN server.

    My version of iperf did not support randomized data, so I had to disable lzo-compression for a closer real-world test.

    @BlueKobold, looking at "htop" on the APU2, it seemed only one core was running at 50-100% during the test.



  • @lra:

    @BlueKobold:

    I would not expect any more than 40 Mbps for a single OpenVPN connection.

    The APU2 comes with 4 Core CPU and only the PPPoE WAN part is single core using, the entire
    OpenVPN part is fully multi CPU core usage and so you will see perhaps numbers owed to this
    circumstance that you was not expecting before. But I would be counting more on the AES-NI
    and IPSec (AES-GCM) that should be more pushing the entire VPN part, for sure not OpenVPN
    but really fast.

    I just tested my APU2, (on Linux in my test), disabled lzo-compression, "cipher AES-256-CBC" and consistently saw 58-62 Mbps using iperf.  Note iperf was not running on the APU2, and the APU2 was an OpenVPN server.

    My version of iperf did not support randomized data, so I had to disable lzo-compression for a closer real-world test.

    @BlueKobold, looking at "htop" on the APU2, it seemed only one core was running at 50-100% during the test.

    Update,

    I retested on the APU2 running iperf3 (client) on the APU2 itself, while the remote end iperf3 (server) bound to the tunnel IP of the OpenVPN client, the result was 92 Mbps.

    It seems testing downstream off an external interface made the test somewhat "choppy" so a consistent, solid stream did not happen (a short pause every few seconds) and hence slower throughput.


Log in to reply