pfSense 10G hardware advice
-
I'm looking to put together a pfSense build for my SOHO network. I've done plenty of reading but still have many questions. I'm hoping folks here can point me in the right direction to select hardware.
Use Case
- 500Mbps WAN connection
- 3x 1G VLANs (mix of wired and wifi)
- 1x guest WIFI VLAN
- 2x 10G VLAN (fibre)
- OpenVPN to access my network on the road
NICs
- i350T-4V2 (4x1G)
- either T420-CR(2x10G) or T422-CR(2x10G + 2x1G)
Am I understanding correctly that the T400 series CR cards are better for this use case than the T400 series SO-CR cards due to the latter's heavier reliance on the system's cpu?
Onboard Controller vs NIC
If a board has quality onboard 1G controllers how much of an issue is it to get more ports on the board?
For example, which is better
- i350-T4V2 + T422-CR + 2x onboard 1G links powered by an I210-AT
- i350T4V2 + T420-CR + 4x onboard 1G links powered by an I350-AM4
Processor
If I am hoping to achieve near 10G speeds between my two 10G VLANs and switching within the same 10G VLAN what kind of processor am I looking at? Looking at some of the hardware selection guides it looks like perhaps an 8-core 2.5GHz cpu is what I'm looking at. Is this correct? And if so are the Atom C3758 (2.2GHz 8core/8threads) or AMD EPYC Embedded 3251 (2.5GHz 8core/16threads) ideal or should I be considering other options such as the i3-8100?
Motherboard
Is it likely that I'll find myself in a situation where I'd prefer a traditional socket over an SoC design so I can upgrade the cpu later?
Memory
How important is ECC memory here? Am I wasting my money considering builds with ECC memory? I imagine either way I'm looking at ~8GB?
Thanks in advance for any thoughts or advice here.
-
Hi @PhiloEpisteme - at 10Gbit speeds and above it will become increasingly important how many packets per second (pps) the firewall can actually process. I would say that with regular sized packets (~1500 bytes) it should be fairly easy to max out a 10Gbit link with some decent hardware, but as you go down in packet size it will become more challenging (since the firewall will need to be able to process more and more packets per second to fill the entire 10Gbit link). If you don't mind me asking, what is the intended use case for the 10Gbit VLAN's? Are you just planning on moving big files back and forth (e.g. using a NAS)?
To put all this into a little more context, I did some 10Gbit testing fairly recently using my pfSense firewall (which is based on a Xeon D-1518). This thread might be worth reading through to get an idea as to what kind of performance to expect above 1Gbi and to see where limitations come int:
https://forum.netgate.com/topic/132394/10gbit-performance-testing
Since you are planning on using OpenVPN, I would recommend going with a higher clocked CPU to take full advantage of your fast WAN connection. As a matter of fact, I built a pfSense firewall based on the Intel i3-8100 processor earlier this year and it has worked out very well. This little guy offers great bang for the buck.
Hope this helps.
-
To get close to 10Gbps you will want the most powerful processor you can get. Most 10GbE NICs have drivers supporting multiple queues (maybe all) which helps utilise CPUs with multiple cores. However you're still probably better having fewer cores at a higher clock rate if you have a choice and you are looking for greatest throughput for a single connection.
The Chelsio cards support a bunch of hardware offloading to different extents but much of it is not relevant to a router or not supported. I would try to avoid an Intel NIC using the ixl driver currently as there is a known issue with VLANs:
https://redmine.pfsense.org/issues/9123
Those using the ix driver are fine.
I would not use ECC ram personally if the cost is significant.Steve
-
Hi @tman222 and @stephenw10, thanks for replying I appreciate it very much.
@tman222 said in pfSense 10G hardware advice:
If you don't mind me asking, what is the intended use case for the 10Gbit VLAN's? Are you just planning on moving big files back and forth (e.g. using a NAS)?
It will be used often for the NAS, yes. I make very heavy use of the NAS regularly move large volumes of data back and forth with a mix of large and small files.
@tman222 said in pfSense 10G hardware advice:
https://forum.netgate.com/topic/132394/10gbit-performance-testing
This thread is extremely helpful, thank you for helping me find it.
@stephenw10 said in pfSense 10G hardware advice:
The Chelsio cards support a bunch of hardware offloading to different extents but much of it is not relevant to a router or not supported.
Thanks for the advice. I'll take this as you suggesting that the T420-SO-CR is an acceptable card vs the T420-CR?
After doing further research into the various options available I've got a few things I'm looking at.
Supermicro A2SDI-TP8F board
- Intel Atom Processor C3858 2.0GHz 12 cores/12 threads[/li]
- 25W TDP[/li]
- 4x 1G links via Intel i350-AM4'[/li]
- 2x 10GBase-T via Intel X557-AT2
- 2x SPF+ 10G links via CS4227 (Is this the same controller as above?)
16GB ram 2x 8GB via Crucial CT8G4SFD824A non ECC
Pros:
- The board already has 8 interfaces and an extra PCIe slot should I need to expand in the future.
- Memory is fairly cheap
- All I'd need to purchase after this is the chassis and power supply
- Precludes purchasing additional NICs or CPUS, possibly offsets high price?
- High core count
- Low TDP
Cons:
- This board is every expensive. Though possibly offset as above?
- Doesn't use Chelsio 10G NICs, this could be a problem?
- Unsure of exactly what controller is used on the 2x SFP+ links
- Low clock speed
The real trouble with cost is the NICs. If I get a board without a ton of onboard controllers I'll have to purchase some 1G and 10G NICs and the genuine ones are expensive; almost so much so that it may make sense to pay for the high end board with them built-in despite the high cost.
Given the suggestions from @stephenw10 perhaps I may be happier with a build utilizing a faster CPU with fewer cores. To that end I've come up with the following.
Supermicro M11SDV-8C+-LN4F
- 8-core/16thread 2.5Ghz CPU
- 4x 1G Link via i350-AM4
Pros:
- Higher clock speed per core
- Allows for use of Chelsio NICs for 10G interface
Cons:
- More expensive than above option
- Requires an additional NIC for my use case and thus less expandable without getting into port bifurcation and finding cases that easily accommodate cabling for this.
- Lower core count
- Would be mixing Chelsio 1G NICs with Intel 1G NICs via T422-CR.
I'm having a hard time finding better options that are lower on power consumption. If I come up with another option I'll update here.
-
@PhiloEpisteme said in pfSense 10G hardware advice:
Supermicro M11SDV-8C+-LN4F
I have never used that AMD CPU or anything close to it so I can't really say how that would perform.
There is nothing wrong with using Intel NICs for this. The newer 10G NICs like the X550 work great.
The C3K SoC has 4 10G NICs built in so those 4 interfaces will be using the same NICs.
I would go for less cores at higher clock rate if you can.
Steve
-
@stephenw10 said in pfSense 10G hardware advice:
The C3K SoC has 4 10G NICs built in so those 4 interfaces will be using the same NICs.
Just to be clear, are you suggesting that the 4x 10G ports on a single NIC will affect performance? It seems very common in the 1G space to have 4 links such as the i350-AM4 NIC.
I'm having trouble finding higher clock speeds at 8-cores with low-ish power requirements. I'll keep searching. I'm aiming in the 8-core count because most of the 10G pre-built machines I'm seeing are 8-core; many of them around the 2.0Ghz range; of course they are using hardware and cpus specially designed for L3 switches.
-
No, sorry, you were asking if the SFP ports would be using the same controller as the 10Gbase-T ports and it's almost certain they are on that as the CPU has 4 on-board. There's no problem with that.
You can see from @tman222's testing that the D-1518 can pass close to 10Gbps but only with multiple streams and all available cores are at 100%. Moving a single large file over a single TCP connection would be a lot less, somewhere near the 3Gbps he saw. 4 cores at 4GHz would likely perform far better in that situation.
It would actually be interesting to see what effect disabling hyper-threading has on the single stream test.
Steve
-
@stephenw10 Thanks for the clarification. I may look at a build similar to the one mentioned above using a Xeon processor. I'd assume that the power draw while at idle would be fairly minimal. I'm a bit worried about noise, but can maybe solve that with quality fans.
-
I would not expect much from a D-1518 at idle I've never tested myself. @tman222 may be able to tell you though.
Steve
-
Hi @PhiloEpisteme - my pfSense box is actually based on the Sumpemicro 5018D - F8NT 1U barebones system:
https://www.supermicro.com/en/products/system/1U/5018/SYS-5018D-FN8T.php
I believe they also make a stand alone or desktop version of this as well (i.e. with the same CPU). With respect to noise, I would not call this system quiet, and the primary reason for that is of course the small form factor. With a 1U chassis you are limited in terms of the types of fans you can use and to get any decent airflow you'll need several small fans operating at quite high RPM's (which means more noise). While this system doesn't sound like a jet plane taking off, one would definitely notice the noise in an office setting. I haven't measured the power consumption on just this system specifically (only on my entire network stack), but with a CPU TDP of just 35 Watts it will be on the lower side. Consider also that the CPU wont' be running at full speed the whole time (unless the firewall is consistently loaded down), but any expansion cards you add will contribute a few extra watts. If you are looking to build a system with this CPU (or similar) it might be a good idea to just get the motherboard and CPU combo and run the whole setup in a larger (2U or bigger) case, which would allow you to use bigger fans.
Now having said that, given that your use case involves wanting to utilize 10Gbit speeds between subnets, I would recommend looking at a higher frequency CPU than the Xeon D's as @stephenw10 already suggested. The quad core Intel i3-8100 or newer generation i3-9100 would make good choices and are decent bang for the buck IMHO. Couple that with a solid motherboard (that has appropriate expansion slots), a 4 port 1Gbit NIC, and a 2 -4 port 10Gbit NIC and you'll have powerful system that will also handle OpenVPN quite well. The i3's I referenced do have a little higher TDP (65 Watts) but again, unless the firewall is loaded down the entire time, the CPU will scale back the frequency and power consumption will be lower on average.
I hope this helps - please let me know if you have any other questions.