PfSense on micro-ITX - What to use to virtualize and what to buy - help



  • Hello everyone. I have two questions.

    I was looking at the smallest form factor computer I could put together with dual gigabit LAN ports that will work with pfSense. I don't know if pfSense still only works with a single core or if it can now load balance between cores. This will determine which i7 processor I get. Speed, is crucial for me especially for VPN/encryption. So asking how fast or $ will be irrelevant for this conversation.

    My second question is; would anyone recommend me running this inside ESXi so I can have other VM's (thus I can save on power by removing my other old computer and have an all in 1 option?

    Let me also state that this won't be the first time working with pfSense - amazing stuff.

    I previously had a rack mount computer running ESXi 5.1 with pfSense as 1 VM, and then about 4 other VM's on the same box. This worked great. Had 1 core dedicated to pfSense on a AMD 1090T processor and testing with another user over WAN did around 300,000 packets per second inbound at 64-byte size before maxing out the processor core and affecting the internet. That's why I love pfSense.



  • pfSense 2.2 has multi-core support for the pf filter as well, so yes, more cores/threads helps.
    Older versions had multithreaded components but the pf part was not multithreaded so that only went as fast as 1 core could.

    Smallest form factor would be NUC-sized, but you will need to add a gigabit ethernet mPCIe card and a 8P8C/RJ45 hole to have 2 ports.
    If you want more customisations, you should have a look at ITX sized systems. It's a standard with multiple suppliers and manufacturers.

    On ITX systems you can pretty much run anything you want, the limitations are usually in supported TDP for the CPU and the I/O options.
    You won't be able to punt 1024GB of RAM in an ITX box for example ;-) Usually they come with one PCIe slot and sometimes one or more mPCIe and mSATA slots.
    Some brands have special connectors for daughter boards that have additional I/O like 4 extra Gbe ports.

    So that should answer the 2nd question about small form factor systems and virtualisation: it'll work just fine. One problem you might encounter is ESXi itself which can be rather problematic if the hardware isn't on the HCL. But there are different (and better IMHO) options for virtualisation. Check out Xen (the standalone packaged version) or XenServer (the complete enterprise Hypervisor distro, used to be Citrix commercial), both are free and open source and work on pretty much anything that has normal-ish hardware and doesn't have crazy firmware bugs that break hypervisors.



  • @Roltec:

    I was looking at the smallest form factor computer I could put together with dual gigabit LAN ports that will work with pfSense.

    Do you just want it smallish or do you really want the absolute smallest possible? What are your exact requirements or limits?

    This will determine which i7 processor I get.

    Why is only i7 possible? Xeon maybe?

    Speed, is crucial for me…

    Speed is pretty closely linked to high TDPs and high TDPs usually don't work well at all with a small form factor. Therefore I think you have to choose which is more important to you, small or speed?



  • Thank you both!

    I was looking at ITX. Honestly this is because I want to be able to place it on a small shelf in the basement next to my equipment. I do not need anything smaller.

    i7 sure, Xeon, why not. But I am not trying to spend $1000+ on a processor just yet until I can figure out the motherboard/form factor involved. The build will obviously be maybe 16Gb ram, SSD only. Going from a 150/150 connection to possibly 500/500Mbit soon, I want to make sure that VPN between this and other sites, plus all the gamers in the house don't ever experience any slowdowns.

    I would love to shut down my server and replace it with a single box that has 2 VM's, server OS and pfSense. I have a NAS device and that will be for storage as it already is now.

    A year or more ago I wasn't able to find an ITX board that has NIC's on the HCL for pfSense (or should I say the Linux distro behind it all). Maybe things have changed now?



  • @Roltec:

    Thank you both!

    I was looking at ITX. Honestly this is because I want to be able to place it on a small shelf in the basement next to my equipment. I do not need anything smaller.

    i7 sure, Xeon, why not. But I am not trying to spend $1000+ on a processor just yet until I can figure out the motherboard/form factor involved. The build will obviously be maybe 16Gb ram, SSD only. Going from a 150/150 connection to possibly 500/500Mbit soon, I want to make sure that VPN between this and other sites, plus all the gamers in the house don't ever experience any slowdowns.

    I would love to shut down my server and replace it with a single box that has 2 VM's, server OS and pfSense. I have a NAS device and that will be for storage as it already is now.

    A year or more ago I wasn't able to find an ITX board that has NIC's on the HCL for pfSense (or should I say the Linux distro behind it all). Maybe things have changed now?

    pfSense is based on FreeBSD. The 'pf' from pfSense is the name of the filter, which is called 'pf' in FreeBSD. Linux, Unix and BSD are different terms for different systems. Solaris is another, to make it more complex ;-) If you want a rough split: there are Unices (yes, the multiple of Unix is Unices as far as I know), and there are Unix-alikes. But since Linux is pretty well known and pretty common, people sometimes think everything that is not Windows or Mac OS has therefore to be Linux :p

    Anyway, back to your topic.

    If you want to run VM's, you have to decide what sort of VM setup you want. Most commonly used are things like ESXi and KVM, but there are better choices out there. Take Xen for example, a robust and fast system (one of the fastest, if not the fastest). You can get it as a package within a Linux distribution, or, if you want a fancy interface to control it from (from Windows for example; it's called XenCenter in that case), use the full-fledged XenServer distro. Other choices might be stuff like Hyper-V or bhyve. Which one you take is up to you, but best performance is had with PVHVM where HVM is used for everything that doesn't get any performance increase from virtualising it, and VirtIO or some other enlightenment interface is used to speed up stuff like I/O for disks, networks, RAM etc. Xen does this very well, KVM does it pretty well as long as it's not loaded down, ESXi has mixed results. The others I don't know about with recent versions, the old ones weren't super good.

    For pfSense, you will mostly benefit from really good NICs, AES-NI instructions on the CPU and skipping any crappy virtual switches (use OpenVSwitch instead, XenServer has it by default I believe and Xen-normal has it as an option you can just configure). Make sure you have 4GB of RAM if you're going to pull a lot of packages in on your pfSense setup.

    Since you are pretty much building a proper server here, check out the Xeon CPU's, especially the E3 version 3 series. They don't cost that much and perform rather well (I can push 900Mbit over VPN with no problems, but that was tested with light encryption - it really depends on your settings).

    Have a look at Supermicro motherboards too, they come with proper NICs and if you need more interfaces you can add a good Intel card for that. Any card will pretty much work, I'd suggest using one that is at least 1 year old so the drivers have matured, and if you only need 1 extra port, buy a dual port one instead as upgrading later on will make you want to bang your head on the desk for having to replace the card you thought was 'good enough'.

    I have plenty of boxes running on the X10SLV-Q (or Q-O at the end, I don't know the code exactly) with 16GB RAM, a quad ethernet card, mSATA SSD, and two 2.5" laptop HDD's for bulk storage in RAID1. Most of the time they run pfsense + Windows Server Essentials 2012R2 and one or two VM's for internal services such as a Linux server for websites and mail and a different Windows Server for server-side applications that you don't want running on the AD windows server. All runs pretty smooth, pfSense gets 1GB RAM, the two Windows VM's get 8GB and the rest usually gets split up between whatever other VM's you need.

    Then the Hypervisor runs on the video console, pfSense on the serial console (a front panel RS-232 port), BIOS and bootloader on the second RS-232 port, and every thing else is just blind from the outside. For some devices I use USB passthrough or complete PCI passthrough if you need a bunch of external devices like sensors, RFID scanners or sometimes fax modems, 3G modems etc. for use in one of the VM's (or in case of 3G modems, directly for pfSense). The ports are all internally plugged in to OpenVSwitches which in turn are plugged in to pfSense's virtual interfaces. This way, the VM's can attach to any of the interfaces as needed, which is great if you need to run a SIP server on your VoIP network, or a webserver on your WAN link, or maybe a fileserver on LAN only. Freedom everywhere! All of that also fits nicely in a 1U box, in most cases I get them at supermicro as well, but there are plenty to choose from. I like my network appliances to have front-panel I/O, so I usually get one what only has power on the back, and everything else on the front. The CPU gets a Dynatron K2 or K199 cooler (depending on the socket) so it stays cool, but is low enough so the lid still fits on the box.

    CPU, Mobo, RAM, SSD, Case, Cooler can be done for about 600. PSU comes with the case I think.

    A totally different approach would be checking out the stuff Lanner makes, or the stuff in the pfSense store. Some of them have decent CPU's for virtualisation which would make for a great pre-assembled solution.



  • It looks like that Supermicro board you posted is one of the best right now that I could find, alongside possibly one from ASRock (or a future unreleased board micro-ITX board).

    I also started reading that E3-1200 V4 may be coming out soon, along with new micro-ITX boards for socket 2011 or 1151? Yet who knows when the processors to match will be released either. Who knows if they will be faster then V3 right now though.

    There are still parts that confuse me now going away from ESXi and towards another piece of software/different CPU&form factor. The most important part now is to understand which is the best setup for running a virtual environment (mind you all, this is in a home and downtime isn't a problem from time to time). Its just a fast internet connection; thus I want to match it with something I can abuse if necessary and not be affected by high PPS networking or high demand either.

    I see you mentioned that Xen is the best and fastest most likely. Would PVHVM be a way that a VM is created under Xen? Kind of lost here as I have only used ESXi before.

    Also, the motherboard mentioned from Supermicro says HD4600 while one of the processors I could get says P4600. Would this matter or cause a conflict?


Log in to reply