Why does Gigabit throughput require such high end hardware?
-
I've been helping a friend of mine pick out a new router for his home setup. He's really into online gaming so I've been looking to the best gaming routers out there. I would love to see him set something up with pfSense, a little switch, and an AP, but it might be a little over his technical ability to manage. Looking into routers I ran into this website http://www.smallnetbuilder.com/, where they test and rate home routers. I found they test WAN to LAN throughput, among other things, and some of these little home routers are reaching 900+ Mbps WAN to LAN.
http://www.smallnetbuilder.com/tools/charts/router/bar/74-wan-to-lan
Here they describe how they do testing.
http://www.smallnetbuilder.com/lanwan/lanwan-howto/31103-how-we-test-hardware-routers-revision-3
The top router in the WAN to LAN graph is the Netgear Nighthawk R7000. It can allegedly bring down 931.4 Mpbs! Why is it that the APU units can't achieve those speeds? Both pieces of hardware have dual core 1Ghz processors.
I've seen some forums where people weren't getting that kind of throughput on the R7000 in the real world, but I'm still curious as to how those numbers are achieved. If somebody used the same test on an APU, could it possibly achieve those types of numbers? I understand it would be without any packages going.
I've looked around the forums and see that people have achieved close to gigabit throughput from small PCs running Celeron CPUs. I see the C2758 is highly recommended. I'm sure it can do more than the 941 Mbps described in the table from it's listing in the pfSense store. I'm guessing it would just require 10 GB NICs basically. There aren't any numbers, but the FW-7551 in the store says it can do gigabit throughput, and that has the C2358.
This is no way a dig against pfSense. I'm just plain curious. I've been a pfSense user for about a year now, and I would never willingly go back. There are so many benefits to using a pfSense system, plus more I know nothing about. I'm just looking to learn a little something new so when I get the opportunity to build my own box I'll be a little more confident in knowing what I'll get. Personally, I'm waiting for the Netgate C2358 boards to be released!
Any insight would be greatly appreciated!
-
It's because the procesors used in those SOHO routers have dedicated packet filtering/forwarding hardware. Often they require proprietary kernel modules to use them. For example the R7000 running dd-wrt tops out at 360Mbps. Interestingly they quote a much lower number for the Netgear firmware:
http://dd-wrt.com/wiki/index.php/DD-WRT_on_R7000Not sure how they justify disabling NAT and SPI for the test. Doesn't seem like real world conditions.
Steve
-
In some of those "little routers" the switch has a mode to do NAT (and the subsequent forwarding). The CPU isn't involved (after the initial setup, unless the current frame is, somehow, an exception. As an example, some of the Atheros switches (yes, I said switches) have this NAT mode. Consider the AR8327N, for example.
Others of these "little routers" have special-purpose hardware in the CPU, like the much-discussed IPv4 forwarding parts in the Cavium CPUs used in the Ubiquiti Edge Router series. Some of the ARM SoCs have hardware-assist for packet filtering (that is so much like the old Netgate p-code engine that it causes me to smile.)
We're actively investigating netmap / DPDK for a next generation architecture. 1Gbps is no longer the goal, 10Gbps and beyond is.
Specific to the APU, the (Realtek) ethernet parts are … not ideal. This problem is so bad that the owner of PC Engines is on-record stating that he will move away from Realtek ethernet in future boards.
I'm guessing it would just require 10 GB NICs basically.
That's how we use them internally. :-X
Personally, I'm waiting for the Netgate C2358 boards to be released!
and we thank you.
-
My thanks are for this, which is welcome news:
This problem is so bad that the owner of PC Engines is on-record stating that he will move away from Realtek ethernet in future boards.
PTL.
-
I really appreciate all the feedback on this! Some of this is frankly over my head a little, but it gives me something to learn more about. In the case of the C2758 or C2358, do they have some sort of hardware acceleration that allows them to achieve the gigabit speeds? I understand how the C2758 wouldn't need it, considering it has 8 cores, but the C2358 seems to be a pretty lean CPU in terms of core speed and cache. I'm sure part of that is the Intel NICs as well.
-
It's because the procesors used in those SOHO routers have dedicated packet filtering/forwarding hardware. Often they require proprietary kernel modules to use them. For example the R7000 running dd-wrt tops out at 360Mbps. Interestingly they quote a much lower number for the Netgear firmware:
http://dd-wrt.com/wiki/index.php/DD-WRT_on_R7000Not sure how they justify disabling NAT and SPI for the test. Doesn't seem like real world conditions.
Steve
Adding to what you said
Some of them have known bugs and the only way to fix the bugs is to disable the hardware network engines, which makes your router crazy slow. Many times those hardware engines store states in their own integrated memory, and that memory is highly restricted and non-expandable. The connection limits are high for home users, but not for business users, like 30k-ish.
-
Adding to what you said
Some of them have known bugs and the only way to fix the bugs is to disable the hardware network engines, which makes your router crazy slow. Many times those hardware engines store states in their own integrated memory, and that memory is highly restricted and non-expandable. The connection limits are high for home users, but not for business users, like 30k-ish.
I did see that when I was checking out the SOHO routers for my friend. I did also overlook that piece of info when writing my OP. Part of me was thinking how in the world do they get the prices where they are and achieve that throughput compared to what it takes to build a high throughput pfSense box. His big thing has been keeping cost low and I'd really like to be able to justify the investment in a pfSense box to him, but I'm not sure if it would make sense.
I see running a pfSense box as a long term investment, even to the home user. It works so well with minimal configuration that anybody who wants to figure out how to configure a few things could have it going exactly the way they want in no time. The main wizard takes care of almost everything. The forums and documentation are great if you need more help!
If you were just trying to basically replicate a small home situation I imagine you could do something like this:
-
Netgate C2358 boards with case and power for roughly $300. Get your own msata SSD if you want, let's round up to $350 total.
-
Some sort of AP. I found this D-Link DAP-2660 for $150. Maybe not the fastest or best, but seems ok. http://www.newegg.com/Product/Product.aspx?Item=9SIA24G1S89336
-
Get a switch. If you don't need a lot, maybe a little 5-8 port gigabit unmanged one will do you fine. $50 at the most.
You're talking $550 for a decent little setup. Nothing amazing, but if you just want a solid home network, that could last you a really long time. Want to upgrade WiFi? Get a new AP. Need a better switch? Get one with everything you could possibly want! I would imagine any business class AP would be more stable and be higher quality than these SOHO routers. In the long run, I don't think you would spend any more going with this more professional setup than buying a new "nice" SOHO router every couple of years. If you get a new AP with the pfSense setup, you don't have to reconfigure everything like you would if you bought a new router for better WiFi. You also get the the added security and features that come with pfSense and it's community.
This is how I look at it. Please, correct me if I'm wrong or missing something here.
-
-
You can get nice little "web-managed" 8 port switches for $50-ish. This is what I ran until I covered my home network to 10G. ;D ;D
Otherwise, you're right. I run Apple Airports at home. We run Ubiquiti at work. Wireless is mostly for when you're mobile (phone, pad, working at the kitchen table or a coffee shop.) At my desk I'm always plugged in to at least 1Gbps.
I see running a pfSense box as a long term investment
Sure. At least we keep pfSense updated (for free!) You won't find that with most "home router" setups.
-
I really appreciate all the feedback on this! Some of this is frankly over my head a little, but it gives me something to learn more about. In the case of the C2758 or C2358, do they have some sort of hardware acceleration that allows them to achieve the gigabit speeds? I understand how the C2758 wouldn't need it, considering it has 8 cores, but the C2358 seems to be a pretty lean CPU in terms of core speed and cache. I'm sure part of that is the Intel NICs as well.
There are a couple things here.
The Intel C2000 product family has a subset of server products that include enhanced communications features. This communications focused product line codenamed “Rangeley” extends the base C2000 product family with communications reliability profile, longer product lifecycle, enhanced thermal profiles, and QuickAssist Technology to accelerate cryptographic workloads. QuickAssist isn't supported in pfSense today, but we are actively working on a driver (with deep assist from Intel) to go back into the FreeBSD tree.
The product models that have some or all of the additional communications capabilities can be identified by an 8 at the end of the product model number (i.e. C2758, C2738, etc.), whereas the base product model numbers will end in 0 (i.e. C2750, C2730, etc.).
The C2000 cores support out of order execution, which is a huge difference from Intel's previous Atom-series CPUs (and the original Core (but not Core2) CPUs. If you've ever looked at the source code for "pf", it involves a lot of branching, and OOE helps a lot with branch miss penalties.
Next, the i354 that is part of every C2000 system (except for a couple vendors who disable the i354 and instead put 1-2 i210s on the board. (I'm looking at you, Asrock. http://www.asrockrack.com/general/productdetail.asp?Model=C2750D4I#Specifications). Like the i350 and 82580, the i354 supports 8 reception queues and 8 transmission queues and supports MSI-X interrupts.
According to a 2009 Intel benchmark using Linux, using MSI reduced the latency of interrupts by a factor of almost three when compared to I/O APIC delivery. http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/msg-signaled-interrupts-paper.pdf FreeBSD has similar work in it.
Other 'server-class' Intel GigE controllers include:
The 82575, which supports 4 reception queues and 4 transmission queues. MSI-X is not enabled due to hardware errata. Under MSI or legacy interrupt mode, 2 reception queues are enabled for hardware RSS hash and only 1 transmission queue is enabled.
The 82576, which supports 16 reception queues and 16 transmission queues. MSI-X is enabled by default. However, due to the number of MSI-X vectors (10), at most 8 reception queues and 8 transmission queues will be enabled under MSI-X mode. When polling(4) is enabled on the device, at most 16 reception queues and 16 transmission queues will be enabled.
Other 'desktop' Intel GigE controllers include:
The i210, which supports 4 reception queues and 4 transmission queues, and supports MSI-X interrupts.
The i211, which supports 2 reception queues and 2 transmission queues, and supports MSI-X interrupts.While typically one does not need more queues than CPU cores, there are architectures that can advantage some. Without giving too much away, we are looking at these for future variants of pfSense, as well as adding support for RSS.
Some vendors (perhaps unknowingly) cut corners. For instance, Deciso, the real vendor behind the OPNsense project, put 4 Intel 82574L parts on their "Netboard-A10". http://www.deciso.com/netboard-a10/#.VLiN8VusnGk
Like the i210, the Intel 82574L supports 2 reception queues and 2 transmission queues, and supports MSI-X interrupts. Unlike the i210, (which, like the 82575, 82576, i211, i350 and i354), the 82574L uses the em driver. The biggest difference between the two drivers is that those in the igb (and igbx, which is the driver for most of Intel's 10Gbps parts) use a different descriptor format, called 'advanced descriptors'.
-
Thank you for all that info! I will have to reread it many more times to get it all to sink in. ;D I was hesitant to post my question, but I'm really glad I did!
@gonzopancho:
You can get nice little "web-managed" 8 port switches for $50-ish. This is what I ran until I covered my home network to 10G. ;D ;D
Otherwise, you're right. I run Apple Airports at home. We run Ubiquiti at work. Wireless is mostly for when you're mobile (phone, pad, working at the kitchen table or a coffee shop.) At my desk I'm always plugged in to at least 1Gbps.
10G in my house would be awesome! I've been faking it as an IT guy for a couple years ;) and I can't tell you how much I hate wireless. It's basically good for surfing the web in my opinion. It's convenient, but if you're going to get some real work done, plug it in. That's what I like to tell people.
@gonzopancho:
Sure. At least we keep pfSense updated (for free!) You won't find that with most "home router" setups.
I really appreciate the pfSense project being open and free. I'm a huge fan of the ideas behind open source, and those who can, should help the project by donating or buying something from the store.
-
I can't tell you how much I hate wireless. It's basically good for surfing the web in my opinion. It's convenient, but if you're going to get some real work done, plug it in.
Yep, can't agree more. :)
Steve
-
I picked up an Ubiquity Edge Router Lite and have been fooling with it, not what I need but an interesting little $100 box.
The web GUI (that I really need as I"m not a network expert) is really limited and much of the configuration must be done at the command line mode. I can figure it out to get it done but a month later I'm having to figure it out again as it has slipped from my memory.
The offloading chip for faster throughput (that I don't really need due to my slow net connection) gets a lot of discussion on their forums. Many things, most beyond me, seem to prevent the chip from being used and put the data back on the CPU.
I don't regret buying it and as long as your needs are simple or your command line skills high you'd likely find something to do with it. Someday it may even run a pfSense.
–-------
I really appreciate the technical details of the chips and cards here, it really helps me figure out a lot of what has been confusing, Thanks
-
Yeah, wireless is good for hooking phones and tablets up, and somebody surfing with a laptop.
I'm stunned by how many people don't think ethernet matters anymore.
I live in a town of 15,000. There was exactly one non-managed gigabit switch for sale in town. It was a little dusty. I needed it in a hurry so I bought it.
We're supposed to have gigabit Internet connections available in the next couple years, according to my ISP. Based on past claims and commitments they're very conservative in their promises and very aggressive on delivery. They upgraded my connection from 30 mbps to 60 mbps without saying anything, and without charging more. I actually called them and asked what was up. Turns out right now I can order 200 mbps and get it tomorrow. Once my pfSense gear is built and working I intend to do that.
So getting back to the point, since you can get 200 mbps throughput in town right now, you would expect to be able to find a switch that can deliver that throughput. Nope. And the dummies at the stores look at you funny when you ask.
When I get things set up, the wifi is going to have severely limited access to the secure side of things.
-
The increased cost of 10gb yet alone the increase power usage of 10gb switches and NICs is hard to justify at home. I hope some next gen ASICs bring down power/heat and cost.
-
The increased cost of 10gb yet alone the increase power usage of 10gb switches and NICs is hard to justify at home. I hope some next gen ASICs bring down power/heat and cost.
You know this is almost exactly what people said about GigE 12+ years ago, right?
-
Good thing is that actually happened, Gb is really mainstream now.
Hoping though it won't take another 12y to make 10Gb equally accessible for the crowd ;D -
"Luckily" I have no requirement for 10gb Ethernet currently for my personal use.
It would be nice to have but unless I upgraded all my drives and interfaces internally as well as 10gb internet suddenly becoming commonly available to consumers at a medium consumer price, I'd see no improvement.Everything else would need to improve dramatically also for me to notice a difference.
-
@gonzopancho:
You know this is almost exactly what people said about GigE 12+ years ago, right?
And 12+ years ago that was true. I had nothing in my house that was capable of filling a 1Gbps connection anyway just as today a 10Gbps network at home would be completely wasted.
The only real consideration here, for me at least, is that 10Gb Networks will inevitably become mainstream in likely less than 10 years but the time period between house re-wiring is substantially longer than that. For many years the maxim 'put Cat5 everywhere' has held true but should we now be fitting Cat6? Or something else, fibre everywhere? If that going to entail much more rigorous cable routing requirements?Steve
-
At some point I will probably go with TLC drives for storage archive only and probably will continue to run a raid of either standard platter drives or SLC raid for the OS. If that happens and if my wan speed multiplies by more than 20x and if I have something distant on the web to also connect to that and that can also support greater than gigabit at the ISP, I will definitely be replacing all my networking stuff…
But I think we are about 10 years out for that in most of the world,. Maybe 5 in Asia. For the usa, its hard to know. There are pockets of excellence with google fiber here and there, but mostly network speeds have been stagnant for years. USA is trying to come in dead last technology wise lately with everything except missiles and drones.
For USA it might be a 20 year wait for GB network speed to be common place at the home ISP providers.
Notice I said GB... Not 10GB. Lord only knows how long that may take.
By then I can grab a 10GB switch for $50 and 10GB NICs used on ebay for $15
-
For USA it might be a 20 year wait for GB network speed to be common place at the home ISP providers.
I dont know about that… Well maybe for rural areas which is a problem right now for HSD.. But I do know of 2 MSOs that will be deploying 1GB speeds within 2 years. There are a few companies that offer fiber which can do GB but I only know of 1 right now that offers it... Of course you'll have to live in a dense area because ROI is there for the MSOs.
Cracks me up.. The US is normally the first to deploy gen1 infrastructure before most other countries then within 5-10+ years, we're behind and stay behind :-(