Choosing hardware based on throughput performance

nikkon

Hi all,

Same old question seems to be again in discussion.
Got this task to choose a box for a fast wan connection (1000 Mbps fiber).
I know CPU matters, then Lan Adaptor and Ram
My question is : What's the best choice between all SoC's on the market? N3700, J1900, C2750 etc and also does it matter to have more than 2 Lan's on the same mobo? There are also SUpermicro models with 4x but still they share the same controller.
Forgot to tell you : wan will be over PPPoE

Thank you

Keljian

Simple answer: The absolute best SOC out there is the xeon-D, but you're not asking for that, you're asking for what is the best SOC for you, and the answer is the C2758

nikkon

does it matter the network adaptor?
Usually supermicro uses Intel.

Keljian

@nikkon:

does it matter the network adaptor?
Usually supermicro uses Intel.

Intel is ideal.

Same old question seems to be again in discussion.

Why not, mostly all is pending on the installed packages, offered services, amount of users and
the entire WAN speed. This can be very difficult from case to case and varying each time.

Got this task to choose a box for a fast wan connection (1000 Mbps fiber).
I know CPU matters, then Lan Adaptor and Ram

And this is all, we are only talking about 1 GBit/s WAN speed?

My question is : What's the best choice between all SoC's on the market? N3700, J1900, C2750 etc

This could be also really owed to your budget for sure!
But if we are talking about 1 GBit/s WAN speed only, you could take many of them to handle that.

For the private or home usage I would consider or taking something from that hardware here:
Budget:
Jetway NF9HG-2930 would do the job with ease
More horse power but also more power consuming:
Intel Core i3 or Core i5 biggest you can get your hands on will do the job also fine, if parts are there
If you have the money take it!
C2758 would be the best balance between power consuming and power serving

For business usage I would at this days going more to tend on this hardware:
Intel Xeon D-1520 or D-1540
Intel Xeon E3-12xx v3/v4

Nothing beats an Intel Xeon CPu at these days as I see it right.

and also does it matter to have more than 2 Lan's on the same mobo?

WAN Port
DMZ Port
LAN Port

Would be mostly the best and common way to solve the most things out as today.
So it would be better to have some spare ports then you have one or two to less!

There are also SUpermicro models with 4x but still they share the same controller.

There are many different options served by the Supermicro from 4 ports, 5 ports and 7 ports with mSATA
slot or SATA only, likes you want or need it! All based on the Intel Atom C2x58 (Rangeley) platform.

Forgot to tell you : wan will be over PPPoE

For the hardware this might be not interesting.

does it matter the network adaptor?
Usually supermicro uses Intel.

Intel gets at this time the best driver support in pfSense as I see it right and so it would be good to get
hardware attached or sorted with Intel NICs for sure. But the real concern with Intel NICs is the following:
Consumer or onBoard NICs:

Intel NICs are coming with a small chip that handles the parity and this is then offloading the CPU less
or more and there fore I would be even prefer to go with the more expensive but better sorted Intel NICs.
Server grade NICs:
This NICs are sorted with a real DSP (Digital Signal Processor) and they are offloading many things more
then the consumer grade NICs but this makes them often also much more expensive!

nikkon

ok,
for now i have 3 choices:

supermicro A1SAi-2750F
asus P9A-I/C2750/SAS/4L
gigabyte GA-9SISL

I expect all to be better in terms of bandwith-throughput that the actual sollution : cisco asa 5506-x

Keljian

@nikkon:

ok,
for now i have 3 choices:

supermicro A1SAi-2750F

asus P9A-I/C2750/SAS/4L

gigabyte GA-9SISL

I expect all to be better in terms of bandwith-throughput that the actual sollution : cisco asa 5506-x

Please note I said C2758, not C2750, the 8 variant has quickassist.

nikkon

i know…but it seems that is not so importan...as far as i read on google.will this help in my case?

update: yes...it does make sense...thanks for the warning! :)

nikkon

then i only have this option: A1SAi-2758F

supermicro A1SAi-2750F

This is a board from the Avoton series with TurboBoost and thats better for running a server
likes apache or samba or any other server.

The Intel Atom C2xx8 series is the Rangeley platform and does not have TurboBoost, but AES-NI
and QuickAssist that would be perhaps more important in the future and is also more tended to
building security appliances likes pfSense.

then i only have this option: A1SAi-2758F

Cool thing as I see it right and balanced out for power serving and power saving.

nikkon

A1SAi-2758F + sc101i case must be the choice.
The only concern is now if i can add an extra network adaptor on pci-e.

heper

A1SRi-2758F has 4x gbit interface. you need more ?

nikkon

Ideally 2 more for a nas lagg :P

A1SAi-2758F + sc101i case must be the choice.

I personally was also thinking exactly at first likes you!
You have 4 common, mainly or mostly used case free to choose from for that board.
1U Rackmount

1U SC505-203 with Front I/O (Ports)
1U SC504-203 with Rear I/O Ports
Mini-ITX
Mini-ITX SC101i with Front Ports
SC721TQ-250B Mini-ITX Tower (But with 1 PCIe Slot outbreak)
The M350 Mini-ITX also without an PCIe Slot outbreak

The only concern is now if i can add an extra network adaptor on pci-e.

This might be not really able with the mini ITX form factor for the case.
But if you perhaps swap over to a 1U chassis or case, there are more way to help you out.
I don´t know from where you are nikkon and what are the price for this hardware will be
that you have to pay for it but Supermicro offers also other boards in other form factors.
So let us imagine you would tend to a 1U form factor then you could go with this one.
A1SRM-LN7F-2758
C2758
8 Cores

1 Intel IPMI LAN Port
7 Intel GB LAN Ports
AES-NI & QuickAssist
8 GB LAN Ports direct on the mainboard no other cards are needed
mSATA Slot (half high) direct on the mainboard no expensive SSD is needed
Normal DDR3 RAM modules can be used and not the more expensive SO-DIMMs

If you need more GB LAN ports or perhaps different Ports for your appliance you
might be happy with this appliances from Axiomtek, they will be not cheap but comes
with some interesting features on top and also more Ports as usual.
Axiomtek NA361 C2358 Dual Core
Axiomtek NA361R C2558 Quad Core or C2758 8 Core
In the middle of the NA361R are a module bay to insert modules for the NA361R.
- VPN Modul
- 4 SFP+
- 4 RJ45 PoE
- 8 RJ45
- 8 SFP

nikkon

i deviced to get supermicro case SC101i and another ethernet adaptor for wan only: INTEL Gigabit CT or http://www.supermicro.nl/products/accessories/addon/AOC-CGP-i2.cfm
hope to be enought space in the case for it.
any other ideas? pci-express 8x ?

or http://www.supermicro.nl/products/accessories/addon/AOC-CGP-i2.cfm

Is this then for the PCIe Slot on the A1SRi-2758F? Is this compatible together?

nikkon

I belive they are. Still need to confirm with super micro dealer.

Keljian

Why don't you get a secondhand pair of Chelsio 310s or Mellanox connectx-2 cards and whack one in the pfsense box?

Or just get a Chelsio t420 for pfsense ($75-80USD on eBay) and Mellanox (Connectx-2 - $19USD) for your server.

10gig is much easier than lagg.

nikkon

I have 2 concerns here :
-how's gonna sell this in romania

if will feat in the case

Keljian

@nikkon:

I have 2 concerns here :
-how's gonna sell this in romania

if will feat in the case

EBay sellers will ship to Romania.
With a low profile bracket, and some dremelling, you'll get a chelsio t420 in that case, it's a very small card. (100mm. x 69mm), you won't find much smaller that isn't specific (like the card mentioned above). The 4xPci-e slot will be able to keep up with the two 10gbe ports.

NegativeEntropy

@nikkon:

…
Got this task to choose a box for a fast wan connection (1000 Mbps fiber).
...
Forgot to tell you : wan will be over PPPoE
...

Careful with Rangely and this set of conditions. According to here: https://forum.pfsense.org/index.php?topic=100900.0

PPOE connections are currently limited to a single core for some of the related CPU load. On the Rangely SoC, this then limits throughput to less than 1 Gigabit. According to the link and the linke thread(s), you'd need a faster CPU (i.e. more single core performance) to be able to use the full Gigabit.

@NegativeEntropy:

@nikkon:

…
Got this task to choose a box for a fast wan connection (1000 Mbps fiber).
...
Forgot to tell you : wan will be over PPPoE
...

Careful with Rangely and this set of conditions. According to here: https://forum.pfsense.org/index.php?topic=100900.0

PPOE connections are currently limited to a single core for some of the related CPU load. On the Rangely SoC, this then limits throughput to less than 1 Gigabit. According to the link and the linke thread(s), you'd need a faster CPU (i.e. more single core performance) to be able to use the full Gigabit.

If he gets a real or pure fiber modem (not a router) he could set up the data inside if the modem and then he
gets a better or a faster throughput. So this problem can be pushed away or solved out, for sure not in all cases
but many where the ISP is using for that pure modems or the router is able to be setup in the so called "bridged mode".

Jailer

@nikkon:

Ideally 2 more for a nas lagg :P

http://www.supermicro.com/products/motherboard/Atom/X10/A1SRM-LN7F-2758.cfm

nikkon

anyone has any clue how many pps can handle this kinda of setup? the one i asked for (c2758 based)
I was just reading some reviews for Vyos and miktorik and they look impresive.

I was just reading some reviews for Vyos and miktorik

Ok for sure this might be a total other thing to go with different OS!
If this both named OS are doing the usage of multicore instead of the single Core usage
by going to use pppoe, this would be really able to see an impressive goal or number of pps.

nikkon

I know Pfsense uses only 1 core for PPPoE…. For now I hope.
Waiting for a testfix in the near future.

zax123

I'm in the same boat.

I need a rig that can run 1gigabit fiber without choking. I currently have an AMD G-T40E and it tops out at about 350mbps.

Pricing out the Supermicro with memory and a case and an SSD comes to something like $500 US. Am I the only one who thinks that's a lot of money for a router?

Is there no cheaper way to do it and still take advantage of 1gigabit WAN?

I also run 3 IPSEC VPN tunnels…

I'm in the same boat.

Likes many peoples

I need a rig that can run 1gigabit fiber without choking.

There are so many out of them that we can´t really count and name them here all, but for sure a
bunch of should be placed for you here.

I currently have an AMD G-T40E and it tops out at about 350mbps.

Hmm, just curious we got something around till ~550 MBit/s and ~650 MBit/s - ~750 MBit/s
with enabled PowerD (Hi adaptive. Sometimes and surely pending on the configuration it might
be good to set up a higher mbuf size. (measured with iPerf from PC (on the WAN Port) to PC (on the LAN Port)

Pricing out the Supermicro with memory and a case and an SSD comes to something
like $500 US. Am I the only one who thinks that's a lot of money for a router?

Not really alone, but if you need only a capable routing device you might be better going
with a dedicated router software likes OpenWRT or DD-WRT, pfSense is a fully featured
software firewall that can be turned with ease to a full UTM device!!! It depends on what
services you might be offering to your network and how many packets you will be installing
and on top how many functions and options given by pfSense will be turned on.

I also run 3 IPSEC VPN tunnels…

This is to less input I think.
Offered services, activated functions, turned on options, used features, installed packets, number of
rules, DMZ & LAN devices and switches, users, CP, Proxy, DPI, IDS/IPS and other things on top likes
needed throughput on WAN, VPN and LAN or DMZ would be fine to know.

Is there no cheaper way to do it and still take advantage of 1gigabit WAN?

For sure it is and there is a really great amount of devices that could match your wish for sure!
At first I want to tell you that the pfSense store is combining the best out of from both "worlds"
the PC Engines APU boards 3 miniPCIe slots + 1 SIM slot and the Intel Atom C2x58 SoC mostly
to find on the well known Supermicro boards. So the SG-2220 for $299 would be matching also
fine to the small told case of usage.

Budget suggestion 1:
pfSense SG-2220 & miniPCIe WIFi card & M.2 mSATA drive
pfSense shop ready to go and with pre-installed pfSense
~$299 + add on cards if needed

Budget suggestion 2:
Intel Celeron G3260 @3,2GHz Mini-ITX Board
Is capable to route 1 GBit/s at the WAN port and running snort for sure.

self made box ~$200 - $300

Most common suggestion:
Intel Celeron J1900 Quad Core CPU based boards often are able to turn the pfSense
both into a real UTM with ease for ~$299 to ~$320 fully assembled and ready to go.

Jetway board based
Axiomtek
$299 - $599

alternatively

Intel Bay Trail Dual Core CPU @2,16 GHz with AES-NI
Will be right sorted for a home appliance but very powerful

Axiomtek
Jetway boards based
$299 - $599
alternatively

Intel Atom C2358, C2558 or C2758 from Dual Core @1,7GHz to Octa Core @2,4GHz
with AES-NI and Intel QuickAssist
pfSense shop as SG units ~$300 - $600
Supermicro Boards from resellers or amazon ~$500
Axiomtek appliances from resellers or directly from Axiomtek ~$600

Really powerful appliances:
Intel Atom C2758, Intel Xeon D-1540, Intel Xeon E3 or E5 based with AES-NI
partial with Intel QuickAssist support.

Lanner appliances ~$400 - open end
Lannic appliances ~$400 - open end
Axiomtek appliances ~$800 open end

Axiomtek NA342 (J1900)
Axiomtek NA342R (J1900)
Axiomtek NA361 (C2558 or C2758)
Axiomtek NA361R (C2558 or C2758)

nikkon

I have great expectations from the version 2.2.5 in terms of performance improvement and bigger from 3.x :)
C2758 + 8Gb RAM + SSD + PCI-X 4x intel Gigabit for WAN must be capable to route 1 GBit/s .

zax123

Wow, that was an EXTREMELY comprehensive reply.

Thank you!

I do make use of a lot of pfSense's features, so I don't want to lose pfSense. I do IGMP proxy, VPN, lots of port mapping, I like debugging tools, etc…

From your list of suggestions, it looks like $500 is the sweet spot for a decent pfSense machine, so I guess I have to go with that.

Which SG unit from pfSense shop would be equivalent to the C2758 SuperMicro board which I will likely go with at the end of all this. :)

Thanks so much for the reply!

Rob

zax123

@BlueKobold:

I currently have an AMD G-T40E and it tops out at about 350mbps.

Hmm, just curious we got something around till ~550 MBit/s and ~650 MBit/s - ~750 MBit/s
with enabled PowerD (Hi adaptive. Sometimes and surely pending on the configuration it might
be good to set up a higher mbuf size. (measured with iPerf from PC (on the WAN Port) to PC (on the LAN Port)

Out of curiosity, how does one enable PowerD/Hi Adaptive in pfSense?

Thanks!

Aluminum

Unless you are infatuated with having a tiny router, if you really want to leverage the most out of a lucky home with gigabit internet, grab one of the SMB "starter server" deals.

You can often find a couple models with cheap E3 xeons, I often see Lenovo TS140 and Dell T220 in the low $300s or less. Last week there was a killer one day deal for $280: 1225v3 @3.2ghz, 4GB ECC, 1TB HDD, free shipping.
Add a dual port intel 82571 nic for $20 off fleabay, quads not much more. (1000 pro pt, expi9402pt, x3939, nc360t are common variants of the duals)

Those two models for sure are actually quiet (seen a few in person) and relatively efficient for the performance they provide (can't cheat physics) there may be others models and vendor types that are similar.

zax123

@Aluminum:

Unless you are infatuated with having a tiny router, if you really want to leverage the most out of a lucky home with gigabit internet, grab one of the SMB "starter server" deals.

You can often find a couple models with cheap E3 xeons, I often see Lenovo TS140 and Dell T220 in the low $300s or less. Last week there was a killer one day deal for $280: 1225v3 @3.2ghz, 4GB ECC, 1TB HDD, free shipping.
Add a dual port intel 82571 nic for $20 off fleabay, quads not much more. (1000 pro pt, expi9402pt, x3939, nc360t are common variants of the duals)

Those two models for sure are actually quiet (seen a few in person) and relatively efficient for the performance they provide (can't cheat physics) there may be others models and vendor types that are similar.

Thank you for this.

I think this is the way I'm going to go. I have a room with "servers" in it and one more won't hurt much.

I checked out the Dell website and doesn't look like the T220 exists anymore. The lowest price T320 was $1200+, so I'm not sure how they get to $280. Was that the Dell or the Lenovo? Was it a used machine?

Thanks again for the suggestion!

Rob

Aluminum

Sorry it was T20, this was the deal last week, expired:

http://slickdeals.net/f/8192191-dell-poweredge-t20-tower-server-xeon-e3-1225-v3-quad-4gb-ddr3-1tb-hdd-dvdrw-279-w-free-s-h-dell-back

Best deal I've seen so far, but with cyber week coming up I think there will be plenty of good buys soon.

Often much better than the consumer bloatware crap because no windows tax and nicer components: with the E3 12_5 models you get a C2_6 chipset motherboard with intel NIC and usually a full x16 from cpu along with x4 and/or x1 off the PCH for expansion. PSU might be slightly custom (lenovo uses different harness you can adapt) but tends to be a more efficient one too.

I believe HP has a similar model in this price class as well, makes sense as the big vendors all try to cover the SMB server niche.

zax123

That's awesome! Now I know what to look for, thank you so much.

I'll wait till Black Friday and grab the best deal possible on a similar config. I'd even be fine if there were no hard drive, I'm much rather an SSD anyway. Hopefully those servers use SATA.

Out of curiosity, how does one enable PowerD/Hi Adaptive in pfSense?

System > Advanced > System Tunables > and then start/activate PowerD & reboot the pfSense box.
After doing this have a look in the Dashboard for the correct MHz/GHz entry will be there.
Do a iPerf test with one PC on the WAN interface and one on the LAN interface, thats it.

If you are using a mSATA drive that is capable of the TRIM support, I would also suggest to activate
TRIM support on your pfSense box.

Do a full install on the SSD or mSATA
reboot to the single user mode
at the prompt type in| cd /sbin/ and then ./tunefs -t enable /
./reboot
to verify that it is activated open a shell and type in| tunefs -p / and tunefs -p / var

dopey

Just to pipe in, I am quite happy with my A1SRi-2758F.

With a few packages running, routing performance is still more than capable of hitting gigabit throughput.

The biggest problem is the PPPoE issue. For me, this is a residential connection - but I work from home and use alot of bandwidth, 700-750mbit/s is plenty. So I'm pretty happy with it.

But if you absolutely need the gig throughput and use PPPoE there are other choices that wlil probably work better, and even crazier, use less power. Some of the low power i3/i5 CPUs are pretty capable, and may use more power at peak, but for the load that routing presents to them, they end up actually coming in at or lower than the rangeley.

nikkon

Mobo is on the way…will be here in 2 weeks.
In the mean time i found a gbit ethernet adaptor:
DELL 0X3959 2 x 10/100/1000 Mbps PCI-E 4X REF
Or
Intel D33745-B PRO-1000 Single Port Gigabit Ethernet PCI-E U3867
What do you think ? Will handle gigabit pppoe?

zax123

Thanks for chiming in @dopey,

I'm pretty convinced that I'll go with a Dell T20 or a Lenovo TS140 E3 server. I have lots of room in my "server room" and I really don't want to worry about getting something with not enough power. And I find the Supermicro to just be too expensive.

I'm hoping I'll get a T20 or TS140 with no HDD and I'll put a small (64gig or so) SSD on it. I'm trying to never buy an HDD again. :)

Thanks all!

awebster

@zax123
Beware the Lenovo TS140…its a great box, but you'll be pretty unhappy if you try to put an aftermarket SSD in it.
It uses custom brackets and power cables, which you only get in the box if you buy a Lenovo branded drive.