Powerful Pfsense box hardware selection.
-
Hello Netgate community,
I'm planning a pfsense deployment for a medium size business and would like to use supermicro hardware.
Connectivity requirements: (not necessarily bandwidth goals).
Dual WAN - 1Gbps/1Gbps + 120Mbps/20Mbps. Support incoming openVPN full tunnel connections from road warriors to connect to the trusted network (full tunnel, layer 2 tap).
10Gbps to local building data network - untrusted network. Support incoming openVPN connections on this interface to connect to the trusted network (full tunnel, layer 2 tap).
10Gbps to trusted network. DC/fileserver access. IPS+IDS enabled.
10Gbps mirror of trusted network traffic to SIEM appliance. (Still debating about whether to pull the mirror here or at the 10G switch on the trusted network... compromises each way).
1Gbps to local building phone network - untrusted. QoS (this actually runs on the same physical network as the data network, but on a tagged VLAN with QoS, prefer to give a dedicated link to router)
1Gbps to local building tenant network - untrusted.
So... Yea, I need at minimum 4 X 1G interfaces and 3 X 10G interfaces. I understand I'm not going to get full 10G performance with pfsense, especially to any single client or connection, and that's fine. The goal for the 10G link here is not 10G, it's to prevent locally connected users who are full-tunnel VPNing to the local server, from unnecessarily reducing available bandwidth for road warrior connected users VPNing from the WAN side, and vice versa.... In other-words, If I get 2Gbps I'll be happy.
Anyway.. Assume for a moment that most of the traffic traversing this thing will be openVPN traffic. (At least, most business related traffic, as we are moving in a direction of requiring full tunnel VPN so that we can capture/monitor all traffic at an SIEM), and that, I'll likely host numerous instances of openVPN servers on different ports in order to scale the openVPN server side bandwidth to multiple cores more effectively.
Options:
- 1019C-FHTN8 + E-2278G + 2X Chelsio 2x10G or 1 X Intel X710 4X10G. (8 skylake cores @ 3.4-5.0GHZ)
- 1019D-16C-FHN13TP (16 skylake cores @2.2-3.0GHZ, SoC/X5579 10G)
- 1019D-14CN-FHN13TP (14 skylake cores @1.9-3.0GHZ +QuickAssist, SoC/X557 10G)
- 1019D-FHN13TP (8 skylake cores @ 2.2-3.0GHZ +QuickAssist, SoC/X557 10G)
Assume for a moment, that cost doesn't really matter much here. The cost of time to set it all up and get it working and document how it meets our security requirements will be far greater than the hardware anyway. I would configure any of these with ~32GB RAM and an M.2 NVME boot device because RAM and NVME is cheap enough that it really doesn't matter.
Worth noting, we do have a number of power users with laptops/desktops running lake CPU's with turbo clocks in the 4.0-4.6GHZ range. It would be nice to have something on the firewall side that could "match" the openVPN throughput for these users that their machines are capable of.
Is QuickAssist support on those D-2 series Xeon's a go in pfsense yet? soon? Will it accelerate openVPN? If so... that 8 core 1019D looks appealing. If support is expected within the next year or 2, I'd be willing to pay the premium for the 14 core 1019D with QuickAssist, knowing I'd have enough compute to handle going without it in the mean time. Otherwise, I think the 8 core "E" xeon (5GHZ turbo) looks best.
What do you guys think? Have a preference? How about- which one would you be most interested in seeing someone deploy and report back on the experience with? (Honestly we all know these will all work fine).
Thanks!
-Eric -
Anyone have preference among that hardware list there?
-
If you want to be able to match clients high power machines you will want the fastest single thread performance you can get so go with the E-2278G out of those. You may not get the same total throughput though.
Quick Assist is unfortunately not yet supported in pfSense and there is no ETA on it though I believe there is a FreeBSD driver in development. Whilst it may well accelerate OpenSSL and hence OpenVPN that's not really where the limiting issue is for OpenVPN. It's in the context switching between kernel and user mode.
Personally I'd like to see what throughput you can see with a 5GHz Xeon and some fast client.
Steve
-
Hi stephenw10,
Thank you! I do appreciate the information about the state of quickassist. That steers me towards the E-2278G as well.
I was hoping to get a quad 10G card. I don't see anything from Chelsio that does that. Do the X710 cards tend to work well in pfsense with similar offload or do you think it's worth occupying both PCIE slots for the chelsio cards in this application?
Thanks,
-Eric -
@eric-marshall - Check out the Chelsio T540-SO-CR:
https://store.chelsio.com/collections/server-offload-adapters
I actually use two of these in my network - one inside a pfSense box, the other in a Proxmox box. Cards work great under both Linux and FreeBSD (pfSense).
Hope this helps.
-
Hi tman222,
Looks like a great suggestion. I guess I was looking primarily for a quad card with 10G Base T, but this could work with custom DAC cables to connect to the Cisco switches, and maybe an SFP+ transceiver for the 10G Base T to go to a security appliance...
I don't see an Ethernet transceiver on Chelsio's site, is there a 3rd party alternative known to work?
Thanks!
-
@eric-marshall - my mistake, I didn't realize you were looking for quad 10Gbit copper card. You are right, I don't think Chelsio makes one of those.
I'm not 100% sure regarding X710 support in the current version of pfSense (which is based on FreeBSD 11.2), it looks like more support was added in FreeBSD 12 (which is what pfSense 2.5 will be based on).
11.2: https://www.freebsd.org/cgi/man.cgi?query=ixl&apropos=0&sektion=4&manpath=FreeBSD+11.2-RELEASE+and+Ports&arch=default&format=html
12.0: https://www.freebsd.org/cgi/man.cgi?query=ixl&apropos=0&sektion=4&manpath=FreeBSD+12.0-RELEASE+and+Ports&arch=default&format=html
Maybe @stephenw10 can comment more on this?
Now having said that, I have used fs.com in the past for SFP+ SR fiber transceivers and haven't had a problem using them in Chelsio cards. These SFP+ 10Gbit RJ45 transceivers might be worth taking a look at to convert over to copper:
https://www.fs.com/products/66613.html
Hope this helps
-
I would personally never use direct-attach cables for that. Or copper.
I would use Cisco modules on the Cisco side and whatever the Intel or Chelsio modules specify/like on that side, and fiber patch cables. It's a short distance so you can use multi-mode which makes the modules cheaper. Fiber cables are easier to manage, etc, etc.
An SFP module has code in it. You can get custom DACs made with different code on each side but why bother with uni-tasker cables? The Cisco fiber modules can stay with the Cisco and the other side can stay with that hardware even if you move/change to a different configuration.
-
Thanks tman222!
I think that's a good idea. FS actually has the custom option there to select chelsio compatibility.
I appreciate your thoughts on the matter.
In my experience, fiber has it's own set of reliability issues (physically weak, requires careful handling, no pinching/bending/pulling, dust and debris can quickly ruin a connection.)
Not saying that it's worse than a custom DAC or CAT 6/7 solution for patching, but I'm just not convinced it's absolutely superior either. I'll think about it. I think I like the elegance of a custom DAC, especially when FS can make them for so cheap anyway... If the system changes at a later date, I don't mind the DACs being throwaways if they served their purpose.
At least one of the 10G connections out of the pfsense box needs to be a 10GBaseT to go to the Intel 10G integrated on the motherboard of the SIEM appliance, so in some ways I'm a bit married to some copper here.
