Why does Gigabit throughput require such high end hardware?



  • I've been helping a friend of mine pick out a new router for his home setup.  He's really into online gaming so I've been looking to the best gaming routers out there.  I would love to see him set something up with pfSense, a little switch, and an AP, but it might be a little over his technical ability to manage.  Looking into routers I ran into this website http://www.smallnetbuilder.com/, where they test and rate home routers. I found they test WAN to LAN throughput, among other things, and some of these little home routers are reaching 900+ Mbps WAN to LAN.

    http://www.smallnetbuilder.com/tools/charts/router/bar/74-wan-to-lan

    Here they describe how they do testing.

    http://www.smallnetbuilder.com/lanwan/lanwan-howto/31103-how-we-test-hardware-routers-revision-3

    The top router in the WAN to LAN graph is the Netgear Nighthawk R7000.  It can allegedly bring down 931.4 Mpbs!  Why is it that the APU units can't achieve those speeds?  Both pieces of hardware have dual core 1Ghz processors.

    I've seen some forums where people weren't getting that kind of throughput on the R7000 in the real world, but I'm still curious as to how those numbers are achieved.  If somebody used the same test on an APU, could it possibly achieve those types of numbers?  I understand it would be without any packages going.

    I've looked around the forums and see that people have achieved close to gigabit throughput from small PCs running Celeron CPUs.  I see the C2758 is highly recommended.  I'm sure it can do more than the 941 Mbps described in the table from it's listing in the pfSense store.  I'm guessing it would just require 10 GB NICs basically.  There aren't any numbers, but the FW-7551 in the store says it can do gigabit throughput, and that has the C2358.

    This is no way a dig against pfSense.  I'm just plain curious.  I've been a pfSense user for about a year now, and I would never willingly go back.  There are so many benefits to using a pfSense system, plus more I know nothing about.  I'm just looking to learn a little something new so when I get the opportunity to build my own box I'll be a little more confident in knowing what I'll get.  Personally, I'm waiting for the Netgate C2358 boards to be released!

    Any insight would be greatly appreciated!


  • Netgate Administrator

    It's because the procesors used in those SOHO routers have dedicated packet filtering/forwarding hardware. Often they require proprietary kernel modules to use them. For example the R7000 running dd-wrt tops out at 360Mbps. Interestingly they quote a much lower number for the Netgear firmware:
    http://dd-wrt.com/wiki/index.php/DD-WRT_on_R7000

    Not sure how they justify disabling NAT and SPI for the test. Doesn't seem like real world conditions.

    Steve



  • In some of those "little routers" the switch has a mode to do NAT (and the subsequent forwarding).  The CPU isn't involved (after the initial setup, unless the current frame is, somehow, an exception.  As an example, some of the Atheros switches (yes, I said switches) have this NAT mode.  Consider the AR8327N, for example.

    Others of these "little routers" have special-purpose hardware in the CPU, like the much-discussed IPv4 forwarding parts in the Cavium CPUs used in the Ubiquiti Edge Router series.  Some of the ARM SoCs have hardware-assist for packet filtering (that is so much like the old Netgate p-code engine that it causes me to smile.)

    We're actively investigating netmap / DPDK for a next generation architecture.  1Gbps is no longer the goal, 10Gbps and beyond is.

    Specific to the APU, the (Realtek) ethernet parts are … not ideal.  This problem is so bad that the owner of PC Engines is on-record stating that he will move away from Realtek ethernet in future boards.

    @Jason:

    I'm guessing it would just require 10 GB NICs basically.

    That's how we use them internally.  :-X

    @Jason:

    Personally, I'm waiting for the Netgate C2358 boards to be released!

    and we thank you.


  • Netgate

    My thanks are for this, which is welcome news:

    This problem is so bad that the owner of PC Engines is on-record stating that he will move away from Realtek ethernet in future boards.

    PTL.



  • I really appreciate all the feedback on this!  Some of this is frankly over my head a little, but it gives me something to learn more about.  In the case of the C2758 or C2358, do they have some sort of hardware acceleration that allows them to achieve the gigabit speeds?  I understand how the C2758 wouldn't need it, considering it has 8 cores, but the C2358 seems to be a pretty lean CPU in terms of core speed and cache.  I'm sure part of that is the Intel NICs as well.



  • @stephenw10:

    It's because the procesors used in those SOHO routers have dedicated packet filtering/forwarding hardware. Often they require proprietary kernel modules to use them. For example the R7000 running dd-wrt tops out at 360Mbps. Interestingly they quote a much lower number for the Netgear firmware:
    http://dd-wrt.com/wiki/index.php/DD-WRT_on_R7000

    Not sure how they justify disabling NAT and SPI for the test. Doesn't seem like real world conditions.

    Steve

    Adding to what you said

    Some of them have known bugs and the only way to fix the bugs is to disable the hardware network engines, which makes your router crazy slow. Many times those hardware engines store states in their own integrated memory, and that memory is highly restricted and non-expandable. The connection limits are high for home users, but not for business users, like 30k-ish.



  • @Harvy66:

    Adding to what you said

    Some of them have known bugs and the only way to fix the bugs is to disable the hardware network engines, which makes your router crazy slow. Many times those hardware engines store states in their own integrated memory, and that memory is highly restricted and non-expandable. The connection limits are high for home users, but not for business users, like 30k-ish.

    I did see that when I was checking out the SOHO routers for my friend.  I did also overlook that piece of info when writing my OP.  Part of me was thinking how in the world do they get the prices where they are and achieve that throughput compared to what it takes to build a high throughput pfSense box.  His big thing has been keeping cost low and I'd really like to be able to justify the investment in a pfSense box to him, but I'm not sure if it would make sense.

    I see running a pfSense box as a long term investment, even to the home user.  It works so well with minimal configuration that anybody who wants to figure out how to configure a few things could have it going exactly the way they want in no time.  The main wizard takes care of almost everything.  The forums and documentation are great if you need more help!

    If you were just trying to basically replicate a small home situation I imagine you could do something like this:

    • Netgate C2358 boards with case and power for roughly $300.  Get your own msata SSD if you want, let's round up to $350 total.

    • Some sort of AP.  I found this D-Link DAP-2660 for $150.  Maybe not the fastest or best, but seems ok. http://www.newegg.com/Product/Product.aspx?Item=9SIA24G1S89336

    • Get a switch.  If you don't need a lot, maybe a little 5-8 port gigabit unmanged one will do you fine. $50 at the most.

    You're talking $550 for a decent little setup.  Nothing amazing, but if you just want a solid home network, that could last you a really long time.  Want to upgrade WiFi?  Get a new AP.  Need a better switch?  Get one with everything you could possibly want!  I would imagine any business class AP would be more stable and be higher quality than these SOHO routers.  In the long run, I don't think you would spend any more going with this more professional setup than buying a new "nice" SOHO router every couple of years.  If you get a new AP with the pfSense setup, you don't have to reconfigure everything like you would if you bought a new router for better WiFi.  You also get the the added security and features that come with pfSense and it's community.

    This is how I look at it.  Please, correct me if I'm wrong or missing something here.



  • You can get nice little "web-managed" 8 port switches for $50-ish.  This is what I ran until I covered my home network to 10G.  ;D ;D

    Otherwise, you're right.  I run Apple Airports at home.  We run Ubiquiti at work.  Wireless is mostly for when you're mobile (phone, pad, working at the kitchen table or a coffee shop.)  At my desk I'm always plugged in to at least 1Gbps.

    I see running a pfSense box as a long term investment

    Sure.  At least we keep pfSense updated (for free!)  You won't find that with most "home router" setups.



  • @Jason:

    I really appreciate all the feedback on this!  Some of this is frankly over my head a little, but it gives me something to learn more about.  In the case of the C2758 or C2358, do they have some sort of hardware acceleration that allows them to achieve the gigabit speeds?  I understand how the C2758 wouldn't need it, considering it has 8 cores, but the C2358 seems to be a pretty lean CPU in terms of core speed and cache.  I'm sure part of that is the Intel NICs as well.

    There are a couple things here.

    The Intel C2000 product family has a subset of server products that include enhanced communications features.  This communications focused product line codenamed “Rangeley” extends the base C2000 product family with communications reliability profile, longer product lifecycle, enhanced thermal profiles, and QuickAssist Technology to accelerate cryptographic workloads.  QuickAssist isn't supported in pfSense today, but we are actively working on a driver (with deep assist from Intel) to go back into the FreeBSD tree.

    The product models that have some or all of the additional communications capabilities can be identified by an 8 at the end of the product model number (i.e. C2758, C2738, etc.), whereas the base product model numbers will end in 0 (i.e. C2750, C2730, etc.).

    The C2000 cores support out of order execution, which is a huge difference from Intel's previous Atom-series CPUs (and the original Core (but not Core2) CPUs.  If you've ever looked at the source code for "pf", it involves a lot of branching, and OOE helps a lot with branch miss penalties.

    Next, the i354 that is part of every C2000 system (except for a couple vendors who disable the i354 and instead put 1-2 i210s on the board. (I'm looking at you, Asrock.  http://www.asrockrack.com/general/productdetail.asp?Model=C2750D4I#Specifications). Like the i350 and 82580, the i354 supports 8 reception queues and 8 transmission queues and supports MSI-X interrupts.

    According to a 2009 Intel benchmark using Linux, using MSI reduced the latency of interrupts by a factor of almost three when compared to I/O APIC delivery.  http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/msg-signaled-interrupts-paper.pdf  FreeBSD has similar work in it.

    Other 'server-class' Intel GigE controllers include:

    The 82575, which supports 4 reception queues and 4 transmission queues.  MSI-X is not enabled due to hardware errata.  Under MSI or legacy interrupt mode, 2 reception queues are enabled for hardware RSS hash and only 1 transmission queue is enabled.

    The 82576, which supports 16 reception queues and 16 transmission queues.  MSI-X is enabled by default.  However, due to the number of MSI-X vectors (10), at most 8 reception queues and 8 transmission queues will be enabled under MSI-X mode.  When polling(4) is enabled on the device, at most 16 reception queues and 16 transmission queues will be enabled.

    Other 'desktop' Intel GigE controllers include:
    The i210, which supports 4 reception queues and 4 transmission queues, and supports MSI-X interrupts.
    The i211, which supports 2 reception queues and 2 transmission queues, and supports MSI-X interrupts.

    While typically one does not need more queues than CPU cores, there are architectures that can advantage some.  Without giving too much away, we are looking at these for future variants of pfSense, as well as adding support for RSS.

    Some vendors (perhaps unknowingly) cut corners.  For instance, Deciso, the real vendor behind the OPNsense project, put 4 Intel 82574L parts on their "Netboard-A10".  http://www.deciso.com/netboard-a10/#.VLiN8VusnGk

    Like the i210, the Intel 82574L supports 2 reception queues and 2 transmission queues, and supports MSI-X interrupts.  Unlike the i210, (which, like the 82575, 82576, i211, i350 and i354), the 82574L uses the em driver.  The biggest difference between the two drivers is that those in the igb (and igbx, which is the driver for most of Intel's 10Gbps parts) use a different descriptor format, called 'advanced descriptors'.



  • Thank you for all that info!  I will have to reread it many more times to get it all to sink in.  ;D  I was hesitant to post my question, but I'm really glad I did!

    @gonzopancho:

    You can get nice little "web-managed" 8 port switches for $50-ish.  This is what I ran until I covered my home network to 10G.  ;D ;D

    Otherwise, you're right.  I run Apple Airports at home.  We run Ubiquiti at work.  Wireless is mostly for when you're mobile (phone, pad, working at the kitchen table or a coffee shop.)  At my desk I'm always plugged in to at least 1Gbps.

    10G in my house would be awesome!  I've been faking it as an IT guy for a couple years  ;) and I can't tell you how much I hate wireless.  It's basically good for surfing the web in my opinion.  It's convenient, but if you're going to get some real work done, plug it in.  That's what I like to tell people.

    @gonzopancho:

    Sure.  At least we keep pfSense updated (for free!)  You won't find that with most "home router" setups.

    I really appreciate the pfSense project being open and free.  I'm a huge fan of the ideas behind open source, and those who can, should help the project by donating or buying something from the store.


  • Netgate Administrator

    @Jason:

    I can't tell you how much I hate wireless.  It's basically good for surfing the web in my opinion.  It's convenient, but if you're going to get some real work done, plug it in.

    Yep, can't agree more.  :)

    Steve



  • I picked up an Ubiquity Edge Router Lite and have been fooling with it, not what I need but an interesting little $100 box.

    The web GUI (that I really need as I"m not a network expert) is really limited and much of the configuration must be done at the command line mode. I can figure it out to get it done but a month later I'm having to figure it out again as it has slipped from my memory.

    The offloading chip for faster throughput (that I don't really need due to my slow net connection) gets a lot of discussion on their forums. Many things, most beyond me, seem to prevent the chip from being used and put the data back on the CPU.

    I don't regret buying it and as long as your needs are simple or your command line skills high you'd likely find something to do with it. Someday it may even run a pfSense.

    –-------

    I really appreciate the technical details of the chips and cards here, it really helps me figure out a lot of what has been confusing, Thanks



  • Yeah, wireless is good for hooking phones and tablets up, and somebody surfing with a laptop.

    I'm stunned by how many people don't think ethernet matters anymore.

    I live in a town of 15,000.  There was exactly one non-managed gigabit switch for sale in town.  It was a little dusty.  I needed it in a hurry so I bought it.

    We're supposed to have gigabit Internet connections available in the next couple years, according to my ISP.  Based on past claims and commitments they're very conservative in their promises and very aggressive on delivery.  They upgraded my connection from 30 mbps to 60 mbps without saying anything, and without charging more.  I actually called them and asked what was up.  Turns out right now I can order 200 mbps and get it tomorrow.  Once my pfSense gear is built and working I intend to do that.

    So getting back to the point, since you can get 200 mbps throughput in town right now, you would expect to be able to find a switch that can deliver that throughput.  Nope.  And the dummies at the stores look at you funny when you ask.

    When I get things set up, the wifi is going to have severely limited access to the secure side of things.



  • The increased cost of 10gb yet alone the increase power usage of 10gb switches and NICs is hard to justify at home. I hope some next gen ASICs bring down power/heat and cost.



  • @Harvy66:

    The increased cost of 10gb yet alone the increase power usage of 10gb switches and NICs is hard to justify at home. I hope some next gen ASICs bring down power/heat and cost.

    You know this is almost exactly what people said about GigE 12+ years ago, right?



  • Good thing is that actually happened, Gb is really mainstream now.
    Hoping though it won't take another 12y to make 10Gb equally accessible for the crowd  ;D



  • "Luckily" I have no requirement for 10gb Ethernet currently for my personal use. 
    It would be nice to have but unless I upgraded all my drives and interfaces internally as well as 10gb internet suddenly becoming commonly available to consumers at a medium consumer price, I'd see no improvement.

    Everything else would need to improve dramatically also for me to notice a difference.


  • Netgate Administrator

    @gonzopancho:

    You know this is almost exactly what people said about GigE 12+ years ago, right?

    And 12+ years ago that was true. I had nothing in my house that was capable of filling a 1Gbps connection anyway just as today a 10Gbps network at home would be completely wasted.
    The only real consideration here, for me at least, is that 10Gb Networks will inevitably become mainstream in likely less than 10 years but the time period between house re-wiring is substantially longer than that. For many years the maxim 'put Cat5 everywhere' has held true but should we now be fitting Cat6? Or something else, fibre everywhere? If that going to entail much more rigorous cable routing requirements?

    Steve



  • At some point I will probably go with TLC drives for storage archive only and probably will continue to run a raid of either standard platter drives or SLC raid for the OS.  If that happens and if my wan speed multiplies by more than 20x and if I have something distant on the web to also connect to that and that can also support greater than gigabit at the ISP, I will definitely be replacing all my networking stuff…

    But I think we are about 10 years out for that in most of the world,.  Maybe 5 in Asia.  For the usa, its hard to know.  There are pockets of excellence with google fiber here and there, but mostly network speeds have been stagnant for years.  USA is trying to come in dead last technology wise lately with everything except missiles and drones.

    For USA it might be a 20 year wait for GB network speed to be common place at the home ISP providers.

    Notice I said GB...  Not 10GB.  Lord only knows how long that may take.

    By then I can grab a 10GB switch for $50 and 10GB NICs used on ebay for $15



  • @kejianshi:

    For USA it might be a 20 year wait for GB network speed to be common place at the home ISP providers.

    I dont know about that… Well maybe for rural areas which is a problem right now for HSD.. But I do know of 2 MSOs that will be deploying 1GB speeds within 2 years. There are a few companies that offer fiber which can do GB but I only know of 1 right now that offers it... Of course you'll have to live in a dense area because ROI is there for the MSOs.

    Cracks me up.. The US is normally the first to deploy gen1 infrastructure before most other countries then within 5-10+ years, we're behind and stay behind :-(



  • Yep - I have fiber right to the outer wall of a house in Maryland and I'm sure the network and the backbone could carry gigabit to each and every home so equipped.  Do they do it?

    Nope - Not while they can make their customers think 50/50 is fast and should cost $100…

    Part of it is the stupidity of the customers collectively for not just unplugging til the companies become competitive.

    I've turned off cell services, internet, cable you name it when I didn't like the cost and the companies think they have me over a barrel.



  • I have fiber to my house with 1Gbps service offerings from two different providers (AT&T and Grande) now, and Google is coming this year.  Grande terminates a peered OC-48 at the data center next door to work.  (We have 2 x 10Gbps between the buildings, and a cabinet there, so the slow part of the path is the first hop.  RTT is nearly always < 5ms.)

    Price is circa $65/mo.  I won't rest until I can run IPSec to work at 1Gbps, either.

    People at work thought I was insane when I pit 10Gbps into all the servers.  Then I pointed out that the economics were better for 10G.  10Gb Ethernet NICs are under $250 (NEW).  10G switch ports are under $100 now.  40G NICs and switch ports are less than 2X these prices.

    Of course, I belive that $50 switches and $15 NICs are only good enough for the most basic of setups.



  • Well yeah - Like home when your WAN speed seems permanently stuck in 2005…  Like most of the USA.

    I'm just saying - I'm not going to remodel my houses internet in hopes that my ISP will get its crap together.

    I think I will do it when they get their crap together.

    When my home 10/100 broadcom nic gets outpaced by the ISP (lets hope soon), and I'm finally using 1/10th of what my 1GB network can give me, I'll consider 10GB.

    And don't even get me talking about the slow "broadband" connection from here in the Philippines back to my home network in Maryland...

    Its slightly faster than using pigeons.



  • "Minneapolis residents to get 10-gigabit fiber, for $400 per month. Single gigabit access is already available for $65 per month."
    http://arstechnica.com/information-technology/2014/12/minneapolis-residents-to-get-10-gigabit-fiber-for-400-per-month/

    NG2-PON is entering testing in the USA. 10gb/2.5gb + WDM, making the head unit port capable of 320gb/s bidirectional. The only reason for the 2.5gb up is the more expensive lasers needed in the customer's ONT. The actual spec is 10/10, but will be 10/2.5 for practical cost reasons.



  • There are pockets of excellence.



  • One of those "pockets of excellence" is about to be New York (both the state and city):

    http://www.nysbroadband.ny.gov/broadband-goals

    Basically, the minimum service in the state will be 100Mbps down / 50Mbps up unless you're super-rural, in which case it's 25Mbps down.

    The US has a more general plan to deliver similar speeds to at 100 million homes by 2020.



  • Plans are nice…


  • Netgate

    So is government force.



  • Government forcing standards and prodding along progress would be nice.
    I have a very bleak outlook on plans and promises.
    I think it would be nice if we got back at trying to be the best in the world with tech and industry.
    Having blazing fast internet couldn't hurt.  If that got combined with dirt cheap power it would make even more difference.



  • I picked up an Ubiquity Edge Router Lite and have been fooling with it, not what I need but an interesting little $100 box.

    For testing Vyatta style CLI out or learning this the router is really one of the best.
    For a small router inside of SMB networks (LAN) it could be also a really good router.

    The web GUI (that I really need as I"m not a network expert) is really limited and much of the configuration must be done at the command line mode.

    But if you have to install and configure 150 routers, you will love it because you
    can use a script to do the most things!

    The increased cost of 10gb yet alone the increase power usage of 10gb switches and NICs is hard to justify at home. I hope some next gen ASICs bring down power/heat and cost.

    Not the best network company, not the best switch, but 2 SFP+ ports!
    D-Link DGS-1510-20 for ~200 €!



  • Power isn't that big of an issue if you are not running lots of hardware in a data center or running off solar.



  • The max wattage on several 24 port 10Gb switches is in the 200-300 range, which is about the same as my *computer, my wife's *computer, my 24port 1Gb switch and firewall put together. Idle power isn't much better, typically only low double digit percentages less.

    *Excluding monitors

    Once we start seeing 10Gb fabbed in the 20nm or 15nm ranges, the power consumption should be dramatically reduced.

    There is already talk about 2.5Gb and 5Gb standards. I guess these can easily done with current fabs and are able to get full rates at 100m over CAT5e and CAT6 respectively. 2.5Gb is supposed to be nearly identical power consumption as 1Gb.



  • @Harvy66:

    The max wattage on several 24 port 10Gb switches is in the 200-300 range, which is about the same as my *computer, my wife's *computer, my 24port 1Gb switch and firewall put together. Idle power isn't much better, typically only low double digit percentages less.

    *Excluding monitors

    Once we start seeing 10Gb fabbed in the 20nm or 15nm ranges, the power consumption should be dramatically reduced.

    There is already talk about 2.5Gb and 5Gb standards. I guess these can easily done with current fabs and are able to get full rates at 100m over CAT5e and CAT6 respectively. 2.5Gb is supposed to be nearly identical power consumption as 1Gb.

    2.5G is a backplane technology.  The only IEEE standards activity in this area is a "study group", which is not a "working group".

    The call for forming a working group has been voted down, repeatedly: http://www.ieee802.org/3/minutes/nov03/1103_CFI_2_5G_report.pdf,
    but the parties concerned managed to pass a vote for a "study group" last November.

    So good luck with that.  Low-power 10GbaseT PHYs are already starting to appear, and many manufacturers are starting to "build in" 10Gbps on their SoC and motherboard designs.  Once you have it on the desktop, it will start to appear in SMB and Enterprise.  The datacenter will always be fiber (cross cabinet) or coax (in-cabinet).

    Netgear sells an 8-port 10Gb switch that consumes 58.8W, max.  How many computers did you have in your house again?



  • I don't think the power requirement of my switch is a big deal for my house…

    So many places to save power that if my switch is not super-green, I will live.

    Its nice to find a good fast green one, but 60w isn't going to break me.

    I'm not dismissing the need for low power equipment.  I'm just saying.

    Unless you are running ALOT of switches, one burning more than 5w isn't the end of the world for a home network.


  • Banned

    All powered by the NSA :D



  • Well - For those guys, 1w differences on hardware adds up quick.  60w switches would be a huge deal when you have 1000s upon 1000s of switches.  haha


  • Banned

    :D Thats why they put the new hub in Utah….Lots of sun all year round! :D

    Only the sun can generate enough power unless they build a nuclear PP next to it #love #USA #freedom



  • @gonzopancho:

    The datacenter will always be fiber (cross cabinet) or coax (in-cabinet).

    Coax? Seriously? Color me skeptical…



  • OK, Twinax, which is just coax with two conductors instead of one.

    Twinax is even good enough for 40Gbps to about 5m.  How tall is your cabinet again?



  • @kejianshi:

    Power isn't that big of an issue if you are not running lots of hardware in a data center or running off solar.

    In either case, power is the biggest 'deal'.