Why does Gigabit throughput require such high end hardware?
-
Adding to what you said
Some of them have known bugs and the only way to fix the bugs is to disable the hardware network engines, which makes your router crazy slow. Many times those hardware engines store states in their own integrated memory, and that memory is highly restricted and non-expandable. The connection limits are high for home users, but not for business users, like 30k-ish.
I did see that when I was checking out the SOHO routers for my friend. I did also overlook that piece of info when writing my OP. Part of me was thinking how in the world do they get the prices where they are and achieve that throughput compared to what it takes to build a high throughput pfSense box. His big thing has been keeping cost low and I'd really like to be able to justify the investment in a pfSense box to him, but I'm not sure if it would make sense.
I see running a pfSense box as a long term investment, even to the home user. It works so well with minimal configuration that anybody who wants to figure out how to configure a few things could have it going exactly the way they want in no time. The main wizard takes care of almost everything. The forums and documentation are great if you need more help!
If you were just trying to basically replicate a small home situation I imagine you could do something like this:
-
Netgate C2358 boards with case and power for roughly $300. Get your own msata SSD if you want, let's round up to $350 total.
-
Some sort of AP. I found this D-Link DAP-2660 for $150. Maybe not the fastest or best, but seems ok. http://www.newegg.com/Product/Product.aspx?Item=9SIA24G1S89336
-
Get a switch. If you don't need a lot, maybe a little 5-8 port gigabit unmanged one will do you fine. $50 at the most.
You're talking $550 for a decent little setup. Nothing amazing, but if you just want a solid home network, that could last you a really long time. Want to upgrade WiFi? Get a new AP. Need a better switch? Get one with everything you could possibly want! I would imagine any business class AP would be more stable and be higher quality than these SOHO routers. In the long run, I don't think you would spend any more going with this more professional setup than buying a new "nice" SOHO router every couple of years. If you get a new AP with the pfSense setup, you don't have to reconfigure everything like you would if you bought a new router for better WiFi. You also get the the added security and features that come with pfSense and it's community.
This is how I look at it. Please, correct me if I'm wrong or missing something here.
-
-
You can get nice little "web-managed" 8 port switches for $50-ish. This is what I ran until I covered my home network to 10G. ;D ;D
Otherwise, you're right. I run Apple Airports at home. We run Ubiquiti at work. Wireless is mostly for when you're mobile (phone, pad, working at the kitchen table or a coffee shop.) At my desk I'm always plugged in to at least 1Gbps.
I see running a pfSense box as a long term investment
Sure. At least we keep pfSense updated (for free!) You won't find that with most "home router" setups.
-
I really appreciate all the feedback on this! Some of this is frankly over my head a little, but it gives me something to learn more about. In the case of the C2758 or C2358, do they have some sort of hardware acceleration that allows them to achieve the gigabit speeds? I understand how the C2758 wouldn't need it, considering it has 8 cores, but the C2358 seems to be a pretty lean CPU in terms of core speed and cache. I'm sure part of that is the Intel NICs as well.
There are a couple things here.
The Intel C2000 product family has a subset of server products that include enhanced communications features. This communications focused product line codenamed “Rangeley” extends the base C2000 product family with communications reliability profile, longer product lifecycle, enhanced thermal profiles, and QuickAssist Technology to accelerate cryptographic workloads. QuickAssist isn't supported in pfSense today, but we are actively working on a driver (with deep assist from Intel) to go back into the FreeBSD tree.
The product models that have some or all of the additional communications capabilities can be identified by an 8 at the end of the product model number (i.e. C2758, C2738, etc.), whereas the base product model numbers will end in 0 (i.e. C2750, C2730, etc.).
The C2000 cores support out of order execution, which is a huge difference from Intel's previous Atom-series CPUs (and the original Core (but not Core2) CPUs. If you've ever looked at the source code for "pf", it involves a lot of branching, and OOE helps a lot with branch miss penalties.
Next, the i354 that is part of every C2000 system (except for a couple vendors who disable the i354 and instead put 1-2 i210s on the board. (I'm looking at you, Asrock. http://www.asrockrack.com/general/productdetail.asp?Model=C2750D4I#Specifications). Like the i350 and 82580, the i354 supports 8 reception queues and 8 transmission queues and supports MSI-X interrupts.
According to a 2009 Intel benchmark using Linux, using MSI reduced the latency of interrupts by a factor of almost three when compared to I/O APIC delivery. http://www.intel.com/content/dam/www/public/us/en/documents/white-papers/msg-signaled-interrupts-paper.pdf FreeBSD has similar work in it.
Other 'server-class' Intel GigE controllers include:
The 82575, which supports 4 reception queues and 4 transmission queues. MSI-X is not enabled due to hardware errata. Under MSI or legacy interrupt mode, 2 reception queues are enabled for hardware RSS hash and only 1 transmission queue is enabled.
The 82576, which supports 16 reception queues and 16 transmission queues. MSI-X is enabled by default. However, due to the number of MSI-X vectors (10), at most 8 reception queues and 8 transmission queues will be enabled under MSI-X mode. When polling(4) is enabled on the device, at most 16 reception queues and 16 transmission queues will be enabled.
Other 'desktop' Intel GigE controllers include:
The i210, which supports 4 reception queues and 4 transmission queues, and supports MSI-X interrupts.
The i211, which supports 2 reception queues and 2 transmission queues, and supports MSI-X interrupts.While typically one does not need more queues than CPU cores, there are architectures that can advantage some. Without giving too much away, we are looking at these for future variants of pfSense, as well as adding support for RSS.
Some vendors (perhaps unknowingly) cut corners. For instance, Deciso, the real vendor behind the OPNsense project, put 4 Intel 82574L parts on their "Netboard-A10". http://www.deciso.com/netboard-a10/#.VLiN8VusnGk
Like the i210, the Intel 82574L supports 2 reception queues and 2 transmission queues, and supports MSI-X interrupts. Unlike the i210, (which, like the 82575, 82576, i211, i350 and i354), the 82574L uses the em driver. The biggest difference between the two drivers is that those in the igb (and igbx, which is the driver for most of Intel's 10Gbps parts) use a different descriptor format, called 'advanced descriptors'.
-
Thank you for all that info! I will have to reread it many more times to get it all to sink in. ;D I was hesitant to post my question, but I'm really glad I did!
@gonzopancho:
You can get nice little "web-managed" 8 port switches for $50-ish. This is what I ran until I covered my home network to 10G. ;D ;D
Otherwise, you're right. I run Apple Airports at home. We run Ubiquiti at work. Wireless is mostly for when you're mobile (phone, pad, working at the kitchen table or a coffee shop.) At my desk I'm always plugged in to at least 1Gbps.
10G in my house would be awesome! I've been faking it as an IT guy for a couple years ;) and I can't tell you how much I hate wireless. It's basically good for surfing the web in my opinion. It's convenient, but if you're going to get some real work done, plug it in. That's what I like to tell people.
@gonzopancho:
Sure. At least we keep pfSense updated (for free!) You won't find that with most "home router" setups.
I really appreciate the pfSense project being open and free. I'm a huge fan of the ideas behind open source, and those who can, should help the project by donating or buying something from the store.
-
I can't tell you how much I hate wireless. It's basically good for surfing the web in my opinion. It's convenient, but if you're going to get some real work done, plug it in.
Yep, can't agree more. :)
Steve
-
I picked up an Ubiquity Edge Router Lite and have been fooling with it, not what I need but an interesting little $100 box.
The web GUI (that I really need as I"m not a network expert) is really limited and much of the configuration must be done at the command line mode. I can figure it out to get it done but a month later I'm having to figure it out again as it has slipped from my memory.
The offloading chip for faster throughput (that I don't really need due to my slow net connection) gets a lot of discussion on their forums. Many things, most beyond me, seem to prevent the chip from being used and put the data back on the CPU.
I don't regret buying it and as long as your needs are simple or your command line skills high you'd likely find something to do with it. Someday it may even run a pfSense.
–-------
I really appreciate the technical details of the chips and cards here, it really helps me figure out a lot of what has been confusing, Thanks
-
Yeah, wireless is good for hooking phones and tablets up, and somebody surfing with a laptop.
I'm stunned by how many people don't think ethernet matters anymore.
I live in a town of 15,000. There was exactly one non-managed gigabit switch for sale in town. It was a little dusty. I needed it in a hurry so I bought it.
We're supposed to have gigabit Internet connections available in the next couple years, according to my ISP. Based on past claims and commitments they're very conservative in their promises and very aggressive on delivery. They upgraded my connection from 30 mbps to 60 mbps without saying anything, and without charging more. I actually called them and asked what was up. Turns out right now I can order 200 mbps and get it tomorrow. Once my pfSense gear is built and working I intend to do that.
So getting back to the point, since you can get 200 mbps throughput in town right now, you would expect to be able to find a switch that can deliver that throughput. Nope. And the dummies at the stores look at you funny when you ask.
When I get things set up, the wifi is going to have severely limited access to the secure side of things.
-
The increased cost of 10gb yet alone the increase power usage of 10gb switches and NICs is hard to justify at home. I hope some next gen ASICs bring down power/heat and cost.
-
The increased cost of 10gb yet alone the increase power usage of 10gb switches and NICs is hard to justify at home. I hope some next gen ASICs bring down power/heat and cost.
You know this is almost exactly what people said about GigE 12+ years ago, right?
-
Good thing is that actually happened, Gb is really mainstream now.
Hoping though it won't take another 12y to make 10Gb equally accessible for the crowd ;D -
"Luckily" I have no requirement for 10gb Ethernet currently for my personal use.
It would be nice to have but unless I upgraded all my drives and interfaces internally as well as 10gb internet suddenly becoming commonly available to consumers at a medium consumer price, I'd see no improvement.Everything else would need to improve dramatically also for me to notice a difference.
-
@gonzopancho:
You know this is almost exactly what people said about GigE 12+ years ago, right?
And 12+ years ago that was true. I had nothing in my house that was capable of filling a 1Gbps connection anyway just as today a 10Gbps network at home would be completely wasted.
The only real consideration here, for me at least, is that 10Gb Networks will inevitably become mainstream in likely less than 10 years but the time period between house re-wiring is substantially longer than that. For many years the maxim 'put Cat5 everywhere' has held true but should we now be fitting Cat6? Or something else, fibre everywhere? If that going to entail much more rigorous cable routing requirements?Steve
-
At some point I will probably go with TLC drives for storage archive only and probably will continue to run a raid of either standard platter drives or SLC raid for the OS. If that happens and if my wan speed multiplies by more than 20x and if I have something distant on the web to also connect to that and that can also support greater than gigabit at the ISP, I will definitely be replacing all my networking stuff…
But I think we are about 10 years out for that in most of the world,. Maybe 5 in Asia. For the usa, its hard to know. There are pockets of excellence with google fiber here and there, but mostly network speeds have been stagnant for years. USA is trying to come in dead last technology wise lately with everything except missiles and drones.
For USA it might be a 20 year wait for GB network speed to be common place at the home ISP providers.
Notice I said GB... Not 10GB. Lord only knows how long that may take.
By then I can grab a 10GB switch for $50 and 10GB NICs used on ebay for $15
-
For USA it might be a 20 year wait for GB network speed to be common place at the home ISP providers.
I dont know about that… Well maybe for rural areas which is a problem right now for HSD.. But I do know of 2 MSOs that will be deploying 1GB speeds within 2 years. There are a few companies that offer fiber which can do GB but I only know of 1 right now that offers it... Of course you'll have to live in a dense area because ROI is there for the MSOs.
Cracks me up.. The US is normally the first to deploy gen1 infrastructure before most other countries then within 5-10+ years, we're behind and stay behind :-(
-
Yep - I have fiber right to the outer wall of a house in Maryland and I'm sure the network and the backbone could carry gigabit to each and every home so equipped. Do they do it?
Nope - Not while they can make their customers think 50/50 is fast and should cost $100…
Part of it is the stupidity of the customers collectively for not just unplugging til the companies become competitive.
I've turned off cell services, internet, cable you name it when I didn't like the cost and the companies think they have me over a barrel.
-
I have fiber to my house with 1Gbps service offerings from two different providers (AT&T and Grande) now, and Google is coming this year. Grande terminates a peered OC-48 at the data center next door to work. (We have 2 x 10Gbps between the buildings, and a cabinet there, so the slow part of the path is the first hop. RTT is nearly always < 5ms.)
Price is circa $65/mo. I won't rest until I can run IPSec to work at 1Gbps, either.
People at work thought I was insane when I pit 10Gbps into all the servers. Then I pointed out that the economics were better for 10G. 10Gb Ethernet NICs are under $250 (NEW). 10G switch ports are under $100 now. 40G NICs and switch ports are less than 2X these prices.
Of course, I belive that $50 switches and $15 NICs are only good enough for the most basic of setups.
-
Well yeah - Like home when your WAN speed seems permanently stuck in 2005… Like most of the USA.
I'm just saying - I'm not going to remodel my houses internet in hopes that my ISP will get its crap together.
I think I will do it when they get their crap together.
When my home 10/100 broadcom nic gets outpaced by the ISP (lets hope soon), and I'm finally using 1/10th of what my 1GB network can give me, I'll consider 10GB.
And don't even get me talking about the slow "broadband" connection from here in the Philippines back to my home network in Maryland...
Its slightly faster than using pigeons.
-
"Minneapolis residents to get 10-gigabit fiber, for $400 per month. Single gigabit access is already available for $65 per month."
http://arstechnica.com/information-technology/2014/12/minneapolis-residents-to-get-10-gigabit-fiber-for-400-per-month/NG2-PON is entering testing in the USA. 10gb/2.5gb + WDM, making the head unit port capable of 320gb/s bidirectional. The only reason for the 2.5gb up is the more expensive lasers needed in the customer's ONT. The actual spec is 10/10, but will be 10/2.5 for practical cost reasons.
-
There are pockets of excellence.
-
One of those "pockets of excellence" is about to be New York (both the state and city):
http://www.nysbroadband.ny.gov/broadband-goals
Basically, the minimum service in the state will be 100Mbps down / 50Mbps up unless you're super-rural, in which case it's 25Mbps down.
The US has a more general plan to deliver similar speeds to at 100 million homes by 2020.