Google Fiber and AT&T Uverse GigaPower - what do I need?



  • All,

    I've done some searching, and there have been some useful and helpful threads. But I haven't gotten a consensus on what's a good solution for me.

    Google Fiber has been announced in two locales (Kansas City, MO, and Austin, TX). In reaction, AT&T is bringing "Gigapower" to my Austin suburban neighborhood. Next month I will be getting up to 300Mbit symmetrical service over my FTTP connection (Fiber to my house, GigE Copper from there). Promises of true Gigabit in 2014.

    I currently run an ALIX 256MB 2LAN embedded and it doesn't have the throughput I am going to need. But I still love PFSense for managing DHCP and monitoring traffic.  So I need something beefier but still quiet and low power.

    Now, I realize that most web sites won't push 300Mbit individually. But I don't want to leave any bandwidth that I am paying for on the floor either. I have one daughter that's a power user (heavy YouTube uploader, she's got a channel that she updates frequently) and I'm also pushing a lot of still photos to Flickr on a weekly basis. My wife and I both work from home and in her case, she's dealing with scanned documents in her corporate system, downloading and uploading constantly.

    My requirements are as follows:

    Silent/Fanless (or as close to silent as I can get it). Will live in a large vented walk-in closet off master bathroom.
    Ability to push hundreds of Mbits or full Gbit WAN side

    Other Notes:
    Might like 3 ethernet for some interesting LAN/WAN/DMZ configuration I might try, but this is flexible
    Minimal VPN requirement (I might use it from time to time but no hard bandwidth requirements)
    Minimal packages (today I use pfblocker)
    With or without traffic shaper (I may not care with high upstream bandwidth)

    Comments? Would an Intel D2700 pretty much cover it or do I need to step up to the low end of the desktop platforms?



  • 300/300 is going to require something faster than an Atom, even just for firewall+NAT.



  • I have another option, but the more complex this gets, the more likely something will go wrong and it will go down when I'm not around (Wife Acceptance Factor).

    The line from outside comes into our master bedroom closet. I could send it from there to my office or the garage, and set up the router there, and send it back IN to the main switch via VLAN. I've toyed with setting this up as a VM under my ESXi hosts (done that previously) and using my old ALIX as a CARP failover. That may be the ticket: Slow but working if my ESXi cluster is borked.

    The biggest hassle is the transition from where I have it today to the final configuration. Between me, the wife, and my youngest daughter (older is at college), finding free time to reconfig everything and get it right is tough.


  • Netgate Administrator

    Yep, the d2700 will likely not do more than 600Mbps (in one direction). Do you actually need 300Mbps up and down simultaneously? As you may have read in other threads you'll need to step up to a low end desktop CPU and that means having a better cooling solution.
    You should be able to switch to different network setup in stages. Install and test the cabling first for example.

    Steve



  • Steve,

    Thanks.

    As I noted, given a pipe that fast, it's not like I'm going to get 300Mbit from any single connection most of the time. That's basic Internet 101. What I am saying is that if I were to go up all the way to their max speed (who of us wouldn't mind trying it out?), I certainly don't want my pfSense router to be the bottleneck. If I have a 100Mbit capable router, then there's no way I'd pay for 300Mbit service, I'd only pay for 100Mbit service.

    As far as testing the alternative network layout: There is VERY little cabling changes involved.  I have Cat5e everywhere, and it's capable of Gigabit speeds already (tested). It's all logical changes in the sense that I need to do a couple of different connections in the main closet and then the rest is VLANs. I can map it out on paper all I like, but the day I implement it, I'll be down for the duration while I change all the VLANs and go to a new router configuration for the very first time that will be doing some funky things (like sending a WAN VLAN through a couple of switches to my office, through a PFsense router, then back UP the same wire on a LAN  VLAN to the same closet for distribution to the rest of the house). I'll probably be walking back and forth between switches with a laptop making changes with direct attached cables.


  • Netgate Administrator

    Hmm, sounds like fun! Easy to make a mistake and incur the wrath of your users though.  ;)
    There are some passive cooling solutions for low end i3s or similar but they aren't cheap.

    Steve



  • Yup. I really need to send the ladies off for a spa weekend somewhere and then tear my network apart and put it back together!



  • If you don't mind me asking, what part of Austin are you in? I live in the Milwood area in Rattan Creek in MW and have not been able to get a straight answer from AT&T about GigaPower. I got Uverse as soon as it was available in my neighborhood and have been happy with it but with the news of Google Fiber and then GigaPower right after that, I wondered how long it would take.



  • Honestly your requirements are a little bit ridiculous. You want a fanless passively cooled system that will handle 300+ Mb/s throughput? You can't even do that with ASIC devices. Find a closet you can put a small server in. Even a dual core AMD with 8 GB of RAM will do the job.


  • Netgate Administrator

    I disagree. A passively cooled atom will firewall 300Mbps easily, in one direction. If you need to run other services like squid or snort then, yes, you'll need something a lot more powerful. Even so there are passive cooling solutions for 35W cpus for a reasonable cost, the Akasa Euler for example.

    Steve



  • @Jason:

    300/300 is going to require something faster than an Atom, even just for firewall+NAT.

    Not really, no.



  • @timthetortoise:

    Honestly your requirements are a little bit ridiculous. You want a fanless passively cooled system that will handle 300+ Mb/s throughput? You can't even do that with ASIC devices.

    LOL  ;)

    Even the 7541 (dual core Atom D525) does 1Gbps throughputs.



  • @bplein:

    Google Fiber has been announced in two locales (Kansas City, MO, and Austin, TX). In reaction, AT&T is bringing "Gigapower" to my Austin suburban neighborhood. Next month I will be getting up to 300Mbit symmetrical service over my FTTP connection (Fiber to my house, GigE Copper from there). Promises of true Gigabit in 2014.

    You are aware that ESF is located in Austin, right?

    Chris lives here (currently on AT&T at home).  I live here (currently on Grande at home, AT&T FTTH in my neighborhood.)
    The office is next door to what used to be CoreNAP (now zColo), and we run a 10Gbps fiber between our rack there and the office.

    Gimme moar bandwidth.  ;D ;D ;D


  • Netgate Administrator

    @gonzopancho:

    Even the 7541 (dual core Atom D525) does 1Gbps throughputs.

    Really?
    I've not tested an atom for throughput but I know that many others have tried and failed to get close to Gigabit. I'm aware that you guys have done testing on that box so I'm sure you know better than me what it's capable of. How did you test it for Gigabit throughput? Most other users seem to top out at ~600Mbps.

    Steve

    Edit: typo



  • sorry, my mind is going.  Happens when you age.  :-X

    In brief testing, with a standard ruleset, the alix passed 85 Mpbs, the 7541 passed 500 Mbps, and pfSense running in a VM running on a Dell R200 R200 passed 850 Mbps.



  • @gonzopancho:

    @Jason:

    300/300 is going to require something faster than an Atom, even just for firewall+NAT.

    Not really, no.

    In my experience, no Atom, including the D2800 I have at home with i350 NICs, can FW+NAT more than 500-600Mbit/s.  I haven't tried the current gen, but that has more to do with the fact that none of the NICs will work under FreeBSD 8.3.  I suspect they'd be able to push 1Gbit/s or more through FW+NAT given the strong increase in IPC the reviews are reporting but we may not be able to actually test with pfSense until the FreeBSD 10 move.



  • @timthetortoise:

    Honestly your requirements are a little bit ridiculous. You want a fanless passively cooled system that will handle 300+ Mb/s throughput? You can't even do that with ASIC devices. Find a closet you can put a small server in. Even a dual core AMD with 8 GB of RAM will do the job.

    I've built a i3-3220T + DQ77KB in an akasa euler thin-itx fanless case, it is rated to 35W TDP cpus. Dual intel NIC onboard, with some DIY on the backplate you could put another one or two GbE ports from a minicard.

    I believe it can do more depending on environment and the very conservative TDP intel uses on recent cpus, so if you really need something faster the 45W quad core E3 1265Lv2 is quite powerful though not cheap.

    So far I have not seen a haswell thin-itx as it would be nice to get AES-NI at i3 prices, and the intel DQ77KB is hard to find at a fair price. (also a caveat about certain NICs not being supported properly, 82576 = no, i350 = single channel ram issue)

    There are also some passive cases coming out for NUCs, if you can figure out a way to have two physical ports* with a dongle NIC that doesn't suck those could be quite nice.
    (*without doing the Very Bad Idea™ of using only VLANs on a switch to separate WAN from LAN)



  • Just go for a simple i3 with 4GB RAM config. You should be good for years. Check out the latest Haswell processor and motherboard compatibility (If I am not mistaken, its on these forums somewhere.. folks have done it). Stick a passive heat sink on it and it should be quiet enough.



  • @Jason:

    @gonzopancho:

    @Jason:

    300/300 is going to require something faster than an Atom, even just for firewall+NAT.

    Not really, no.

    In my experience, no Atom, including the D2800 I have at home with i350 NICs, can FW+NAT more than 500-600Mbit/s.  I haven't tried the current gen, but that has more to do with the fact that none of the NICs will work under FreeBSD 8.3.  I suspect they'd be able to push 1Gbit/s or more through FW+NAT given the strong increase in IPC the reviews are reporting but we may not be able to actually test with pfSense until the FreeBSD 10 move.

    You seem to have missed the part where we have several of these in-house now.



  • @gonzopancho:

    @Jason:

    @gonzopancho:

    @Jason:

    300/300 is going to require something faster than an Atom, even just for firewall+NAT.

    Not really, no.

    In my experience, no Atom, including the D2800 I have at home with i350 NICs, can FW+NAT more than 500-600Mbit/s.  I haven't tried the current gen, but that has more to do with the fact that none of the NICs will work under FreeBSD 8.3.  I suspect they'd be able to push 1Gbit/s or more through FW+NAT given the strong increase in IPC the reviews are reporting but we may not be able to actually test with pfSense until the FreeBSD 10 move.

    You seem to have missed the part where we have several of these in-house now.

    Several of what?  I don't know what you're talking about when you reference "these".  I just skimmed the thread again and aside from some benchmark numbers you posted for an older Atom, I don't see you mentioning having anything specific in-house.  If you've got one of the new 8-core atoms working, NICs and all, please post up some numbers; I'd love to see them.



  • @captainkirkw:

    If you don't mind me asking, what part of Austin are you in? I live in the Milwood area in Rattan Creek in MW and have not been able to get a straight answer from AT&T about GigaPower. I got Uverse as soon as it was available in my neighborhood and have been happy with it but with the news of Google Fiber and then GigaPower right after that, I wondered how long it would take.

    I am in Leander, in Crystal Falls. New development with Fiber to the Home.



  • @timthetortoise:

    Honestly your requirements are a little bit ridiculous. You want a fanless passively cooled system that will handle 300+ Mb/s throughput? You can't even do that with ASIC devices. Find a closet you can put a small server in. Even a dual core AMD with 8 GB of RAM will do the job.

    Honestly your response is a little bit ridiculous.

    I asked a question because I don't know these things. Where is the secret decoder on how many MIPs it takes per Megabit of throughput? If you point it out to me, I'll do my own calculations. Besides, there are many types of passively cooled systems that are very high power. They just get cost prohibitive.



  • @gonzopancho:

    You are aware that ESF is located in Austin, right?

    Chris lives here (currently on AT&T at home).  I live here (currently on Grande at home, AT&T FTTH in my neighborhood.)
    The office is next door to what used to be CoreNAP (now zColo), and we run a 10Gbps fiber between our rack there and the office.

    Gimme moar bandwidth.  ;D ;D ;D

    Wow, didn't realize that!  I relocated to the Austin area in 2012 as well (just picked that up from your web site)



  • I may have solved my challenge for now.

    I have installed pfsense in a VM on one of my ESXi hosts. I have set it up with CARP to fail over to my ALIX. Complicated due to use of VLANs to pipe the various networks between the network panel in my closet to my office where the VMware cluster lives, but it works.

    If it works well and isn't flakey, I'll keep this setup and the ALIX will just be my slow speed fallback when my VM is down.



  • I will just leave this here … It might a very awesome build.

    GA-Q87TN : Thin Mini-ITX Motherboard
    Core i5-4570 : 84W TDP  (Couldnt fing the 4570T 35W ...)
    4GB Ram
    30 GB mSata SSD
    Power Adapter
    Lian-Li PC-Q05B Thin Mini-ITX Case

    http://www.newegg.com/Product/Product.aspx?Item=N82E16813128670&Tpk=GA-Q87TN
    http://www.newegg.com/Product/Product.aspx?Item=N82E16819116896&Tpk=Core i5-4570  (couldnt find the i5-4570T : 35W TDP)
    http://www.newegg.com/Product/Product.aspx?Item=N82E16820231470
    http://www.newegg.com/Product/Product.aspx?Item=9SIA2W014A9300
    http://www.mini-box.com/19v-8-4A-160-Watt-AC-DC-Power-Adapter
    http://www.newegg.com/Product/Product.aspx?Item=N82E16811112357&Tpk=PC-Q05b



  • @timthetortoise:

    Honestly your requirements are a little bit ridiculous. You want a fanless passively cooled system that will handle 300+ Mb/s throughput? You can't even do that with ASIC devices. Find a closet you can put a small server in. Even a dual core AMD with 8 GB of RAM will do the job.

    Nope you can : Ubiquity Edge Router Lite : 99$ 1.3 Gbps Simultaneous WAN/LAN, LAN/WAN Throuput



  • @Altheran:

    I will just leave this here … It might a very awesome build.

    GA-Q87TN : Thin Mini-ITX Motherboard
    Core i5-4570 : 84W TDP  (Couldnt fing the 4570T 35W ...)
    4GB Ram
    30 GB mSata SSD
    Power Adapter
    Lian-Li PC-Q05B Thin Mini-ITX Case

    That board might not work out (at the moment).  Most of the Haswell boards come with 8111G and/ or i217 NICs - neither are supported out-of-box in pfSense 2.1.

    The Pentium G3220 (54W SDP) will more than suffice for >1Gbps NAT throughput and >100Mbps AES-256 VPN throughput.  In fact, you should be able to squeeze out 200Mbps of VPN throughput from it, albeit at increased power consumption since it doesn't have AES-NI ASIC to offload the processor.



  • @Altheran:

    @timthetortoise:

    Honestly your requirements are a little bit ridiculous. You want a fanless passively cooled system that will handle 300+ Mb/s throughput? You can't even do that with ASIC devices. Find a closet you can put a small server in. Even a dual core AMD with 8 GB of RAM will do the job.

    Nope you can : Ubiquity Edge Router Lite : 99$ 1.3 Gbps Simultaneous WAN/LAN, LAN/WAN Throuput

    I love pfSense, but that seems like the 'easy button'.

    I need to see what features will be coming with the new gateway device that Uverse uses for Gigapower. I'd love to ditch it and use my own (tried it with pfSense and it works) but I cannot find a way to put the RG they supply behind my router in order to have phones work (our Uverse is also for phone). If  it was as simple as port forwarding and I had an exhaustive list of ports, I could do it.



  • @Altheran:

    Nope you can : Ubiquity Edge Router Lite : 99$ 1.3 Gbps Simultaneous WAN/LAN, LAN/WAN Throuput

    Has anyone verified this after Ubiquiti was forced to stop using the Vyatta 'secret' codebase?

    Or is this just the "Tolly" report outcome?



  • For the last month or so I've run pfSense in a VM in order to take advantage of my 300Mbit symmetrical service. The added complexity of using VLANs at my closet to pipe the WAN over to my ESXi hosts was a bit excessive, so I finally bit the bullet and bought some dedicated hardware.

    I went with an Intel DC2500CCE dual-core Atom motherboard with 2 onboard Intel GigE nics http://www.amazon.com/gp/product/B007MS9OI2/ref=oh_details_o00_s00_i00?ie=UTF8&psc=1, as well as this case http://www.amazon.com/gp/product/B005PI1Z7O/ref=oh_details_o00_s00_i01?ie=UTF8&psc=1

    The motherboard is fanless, as is the case. I used some old laptop RAM I had lying around as well as an old laptop HDD. I will eventually put an SSD inside just to make it even more quiet.

    I can run speed tests pushing the full 300Mbit in either direction, and the pfSense dashboard indicates it's pulling 42-45% CPU during that peak.

    So, does this imply that I should be able to get about 600 Mbit through it when AT&T increases our speed? Surely I'll have a tough time using that kind of speed, but just trying to get a feel for what my new hardware can do.

    Finally, are there any tips to getting lower CPU utilization out of it? Thanks in advance!



  • @bplein:

    300Mbit symmetrical service

    I went with an Intel DC2500CCE dual-core Atom

    I can run speed tests pushing the full 300Mbit in either direction, and the pfSense dashboard indicates it's pulling 42-45% CPU during that peak.

    So, does this imply that I should be able to get about 600 Mbit through it when AT&T increases our speed?

    That Atom will top out somewhere between 500-600Mbit/s TOTAL.  That means that if you have 300/300 service you can max it out in both directions, or if you had 600/600 you could (come close to) max it out in one direction.  This assumes that all you are doing is firewall & NAT.  If you add in packages, traffic shaping, etc. then you will fall way short.

    @bplein:

    Finally, are there any tips to getting lower CPU utilization out of it? Thanks in advance!

    Buy better hardware or downgrade your connection.  That's about it.



  • Is simultaneous 100% utilization in and out really a realistic use case? I think as long as he can max out each direction individually, the OP can sleep soundly knowing he couldn't have gotten the same performance from a lower bandwidth tier (which seems to be the objective here).


  • Netgate Administrator

    That Atom doesn't do hyper-threading right? So you see 2 logical cores?
    Depending on how you are measuring the CPU usage you might be closer to the limit than you think. The PF process is (currently) limited to a single thread so it can only use one core. If that core is at 80% usage then the total CPU usage, shown on the dashboard, might only show 45%. Check with 'top -SH' at the console to see how the cores are being loaded.
    Have a read through this thread: https://forum.pfsense.org/index.php/topic,67411.0.html
    User maverick_slo eventually managed to get 640Mbps from his D2500CC after some tuning. I can't remeber if that was between the two on board NICs or not. (probably was though since additional NICs would have to be PCI)

    Steve



  • @Jason:

    That Atom will top out somewhere between 500-600Mbit/s TOTAL.  That means that if you have 300/300 service you can max it out in both directions, or if you had 600/600 you could (come close to) max it out in one direction.  This assumes that all you are doing is firewall & NAT.  If you add in packages, traffic shaping, etc. then you will fall way short.

    Didn't you say earlier that "300/300 is going to require something faster than an Atom, even just for firewall+NAT."?

    I'm really just asking if my estimations are based correctly: that CPU for throughput (assuming no other apps are running) is linear.

    @bplein:

    Finally, are there any tips to getting lower CPU utilization out of it? Thanks in advance!

    @Jason:

    Buy better hardware or downgrade your connection.  That's about it.

    Now that's a non-answer! I didn't say "how can I get better throughput". I asked if pfSense had any optimizations for reducing CPU utilization.

    My guess is that you hate Atom. You were the first to answer incorrectly in this thread, and now you are hopping in recommending that I get "better hardware".

    Please do tell me what hardware is better for my needs? Right now I can do 300/300 (unidirectional at least)  with a fanless solution I can keep in my master bedroom closet. If you have another better, faster, still fanless solution that meets my requirements, please do chime in.

    I really do appreciate valuable input. I just don't see how repeatedly claiming that Atom won't do it is adding value. It's already doing it…



  • @razzfazz:

    Is simultaneous 100% utilization in and out really a realistic use case? I think as long as he can max out each direction individually, the OP can sleep soundly knowing he couldn't have gotten the same performance from a lower bandwidth tier (which seems to be the objective here).

    I am not even using the full capabilities of my 300/300 today! Oh, I can speedtest it unidirectionally but realistically the line is underutilized almost all the time, but my wife and I both enjoy the rapid upload of large digital documents to our employers. It's helps us immensely. I probably wouldn't notice if it was 100/100 except for a few types of downloads and uploads that we do.

    My goal is to make sure my current router is as optimized as possible, so that when I do hit a peak (especially after the free upgrade to Gigabit later this year), I am getting the most of my link. If I can get 600 one way, I'll be happy with that. If I could stretch it to 700 via optimizations, that's just smart router management!



  • @stephenw10:

    That Atom doesn't do hyper-threading right? So you see 2 logical cores?
    Depending on how you are measuring the CPU usage you might be closer to the limit than you think. The PF process is (currently) limited to a single thread so it can only use one core. If that core is at 80% usage then the total CPU usage, shown on the dashboard, might only show 45%. Check with 'top -SH' at the console to see how the cores are being loaded.
    Have a read through this thread: https://forum.pfsense.org/index.php/topic,67411.0.html
    User maverick_slo eventually managed to get 640Mbps from his D2500CC after some tuning. I can't remeber if that was between the two on board NICs or not. (probably was though since additional NICs would have to be PCI)

    Steve

    Steve,

    Thanks for the thoughtful reply.

    It has two cores, pfSense sees it as two cores (not hyperthreaded). It appears from the "top -SH" command that the CPU is being utilized by the interrupts, and all are showing up on CPU 0. During these test runs, I ignored the WebUI and only looked at "top", and it looks like you are correct, maybe 85% (of 200%) are eaten up by the "intr" command and all are running in CPU0. If I extrapolate that, and if it can't be distributed across cores (maybe it makes no sense to do so), then I am truly near topped out.

    I'll check out that other thread. Thanks a bunch.


Log in to reply