Hardware required to saturate Comcast Gigabit Pro (2gbps + 1 gbps)



  • Hello,

    I'm moving into an area which offers Comcast Gigabit Pro service. This service brings 2gbps fiber + 1gbps copper into the home. From reading various sources online, these two WANs can be aggregated to a 3gbps connection. (I believe a single client however may only see 2gbps). Comcast provides a Juniper ACX2100 or ACX2200 router for the install.

    I would like to build a pfSense box that will saturate 3gbps. Ideally, I can future proof myself and target 10gbps.

    It looks like the Intel Atom Rangeley series and Intel Xeon D series supports a lot of features which may get us there, but are not yet baked into pfSense. (QuickAssist, AES-NI)

    I see two options in the store which offer 10gb SFP+:
    XG-2758 and XG-1540

    The 1540 is a significantly more powerful processor than the Atom in the 2758, but does it offer significantly more performance? It seems to me like sending packets may be more a function of clock speed than cores??

    Ideally, it looks like the Pentium-D 1508 would be the ideal processor?
    http://goo.gl/hECzvu
    2C/4T @ 2.2 - 2.6Ghz and 25 watts

    or, possibly the Xeon 1518
    http://goo.gl/W2QVQT]
    4C/8T @ 2.2Ghz and 35 watts

    Of these four options, which would provide the highest throughput? (Using current pfSense 2.3.1)

    Other thoughts:
    Will the Juniper device do link aggregation and provide full bandwidth out of the 10GbE SFP+ port?
    Or, will I need to bring both the copper and fiber into the pfSense box and do the aggregation there?

    I see some of the SuperMicro 5018D offerings offer i210 ethernet, and others i350-AM4. Will that make a significant difference?

    When pfSense supports QuickAssist/AES-NI or whatever new shiny ?netmap? goodness is available, will the Atom 2758 be enough to push what I'm looking for?

    My preference would be for the lowest power draw box that does what I'm looking for.

    Thanks!



  • Congrats on the net connection.  Sounds like you have a lot of research to do.  I was just reading a thread about bridging at layer 2 in Linux and the fact that just doing this will engage Linux to manage the bridging.

    The reason I mention this is that a lot of routers seem to have some type of hardware to handle some of the things we expect software to handle just great.  If you plan on bridging links you may want to worry about PCIe port speeds, but I do not know.

    The fact is you are going to have to start with X amount of packets per second, see how the devices are going to handle it, and keep going.  Pay attention to anything talking about offloading.  NICs offload things:  https://en.wikipedia.org/wiki/TCP_offload_engine

    There is hardware that offloads encryption (AES is one) that allow for fast encrypted links.  But then you figure out that OpenVPN does not support that yet, IPSEC does, but a lot do not like IPSEC.

    You are also talking about link aggregation.  I am not familiar with the Comcast fiber service but is the ISP supporting it on their side @ layer 2?

    You are asking about support and processor speed…what are you running is what you need to figure out.  Are you installing packages that interact with the traffic flow?  Are you logging traffic to disk?



  • @webdawg:

    Congrats on the net connection.  Sounds like you have a lot of research to do.  I was just reading a thread about bridging at layer 2 in Linux and the fact that just doing this will engage Linux to manage the bridging.

    The reason I mention this is that a lot of routers seem to have some type of hardware to handle some of the things we expect software to handle just great.  If you plan on bridging links you may want to worry about PCIe port speeds, but I do not know.

    The fact is you are going to have to start with X amount of packets per second, see how the devices are going to handle it, and keep going.  Pay attention to anything talking about offloading.  NICs offload things:  https://en.wikipedia.org/wiki/TCP_offload_engine

    There is hardware that offloads encryption (AES is one) that allow for fast encrypted links.  But then you figure out that OpenVPN does not support that yet, IPSEC does, but a lot do not like IPSEC.

    You are also talking about link aggregation.  I am not familiar with the Comcast fiber service but is the ISP supporting it on their side @ layer 2?

    You are asking about support and processor speed…what are you running is what you need to figure out.  Are you installing packages that interact with the traffic flow?  Are you logging traffic to disk?

    Yes, lots of research indeed! I'm glad there are great resources here to search through! I was hoping to find a Comcast gigabit pro specific thread in the forum, but didn't see anything. Hopefully this one will spawn some great research for others coming online with the service.

    Looking at https://doc.pfsense.org/index.php/LAGG_Interfaces
    It appears that LACP link aggregation is not really what I want, as it assumes similar speeds for each link. That would only get me 1gb x 2, which is the same as the fiber handoff. It looks like I'd could do a round robin configuration… but then 2gbps capable devices might get routed to the 1gb link?

    Maybe it would be best to keep is simple, use the 2gbps fiber link, and keep the 1gb link for failover.

    Connect up a 10gb capable switch, and hang everything off that?

    As for software, I plan to run squid, and squid guard, possibly snort. I'm also interested in running openVPN. Of those, I believe openVPN will be the cpu bottleneck. I don't expect to be able to saturate the 2gb link with an openVPN connection... but it would be cool! I don't plan to do any significant logging unless I need them for debugging or analyzing a threat.



  • I'm moving into an area which offers Comcast Gigabit Pro service. This service brings 2gbps fiber + 1gbps copper into the home.

    Is this a 2 x 1 GBit/s MLPPP (MPLS) service Comcast is offering plus a single 1 GBit/s line on top or
    is it a 3 x 1 GBit/s line sold as 2 x 1 GBit/s plus 1 GBit/s? You can easily email them to find it out I
    would guess. It should be clear fo your first before you are buying devices in my opinion.

    From reading various sources online, these two WANs can be aggregated to a 3gbps connection.

    I would assume this might be then load balancing or MLPPP (MPLS) and this is a extra service not common as
    todays ISPs are offering this only for more money.

    (I believe a single client however may only see 2gbps). Comcast provides a Juniper ACX2100 or ACX2200 router for the install.

    The Juniper ACX2200 is at ~7.300 € here at the time and a smaller TIER-3 carrier router for the so called
    carrier ethernet 2.0 services. Nice device, why you don´t want to run one of this devices to ensure that the
    offered service is able to run false free.

    I would like to build a pfSense box that will saturate 3gbps. Ideally, I can future proof myself and target 10gbps.

    I would personally set up a DMZ and LAN switch then that is connected over a 10 GbE or SFP+ Link to ensure
    that the switches are not creating a bottleneck to that network zones and that all 3 GBit/s will be available there.

    It looks like the Intel Atom Rangeley series and Intel Xeon D series supports a lot of features which may get us there, but are not yet baked into pfSense. (QuickAssist, AES-NI)

    To route multiple GBit/s streams at the WAN interfaces I would prefer then more something likes a
    Intel Xeon E3-1275v3 or E3-1286v3 4 core CPU not under 3,0GHz this might be a really good chance
    to get all routet well. The NIC for that would be nice to have a Chelsio adapter that is able to fully offload
    the NAT and VLAN workload.

    Of these four options, which would provide the highest throughput? (Using current pfSense 2.3.1)

    I would try out more 2.2.6 and if this is running 100% false free I would change.

    Will the Juniper device do link aggregation and provide full bandwidth out of the 10GbE SFP+ port?

    I guess they are using MLPPP (MLPS) services such as link aggregation at the WAN.

    Or, will I need to bring both the copper and fiber into the pfSense box and do the aggregation there?

    Would be pending on what service the ISP Comcast is offering! You can´t mix up all services as
    you want or need it! MLPPP (MPLS) is a service that is offered as a both ending service. But if not you can
    try out doing load balancing over several methods such as;

    • policy based routing
    • session based routing
    • service based routing

    With a failover rule then it would be running fine without any kind of problems for you, but again it is
    all based on what kind of service your ISP is offering you.

    I see some of the SuperMicro 5018D offerings offer i210 ethernet, and others i350-AM4. Will that make a significant difference?

    If there is a strong enough CPU or SoC working in the background you will be not seeing any differences
    but also again I would prefer to use the Chelsio NICs from the pfSense shop and especially the one that
    is able to fully offload the NAT process the other one then for the connection to the switches as told above.

    When pfSense supports QuickAssist/AES-NI or whatever new shiny ?netmap? goodness is available, will the Atom 2758 be enough to push what I'm looking for?

    Others may think different on this but I
    personally would have a look at the Juniper ACX2200 to ensure that the offered service is running fine then.



  • Here is a link to the Comcast documentation
    https://drive.google.com/file/d/0B8e0wvBZ26DadUR2OXp1blg3azVrSzJZRjFUMjRabzFQQ3Nv/view?usp=sharing

    From what I can understand, they are only using the Juniper device for the link handoff.
    This router is rented to me for a very low price of $20 USD per month. They are not making any money on the rental for sure.

    Specifically, to use the fiber link, they suggest I need the following equipment:
      10G capable Router/Layer 3 switch with at least 1 10Gbps SFP+ cage
      10G SFP+ 850nm MMF Transceiver
      MMF LC Jumper

    Comcast will provide one IPv4 static IP address for the 1Gb connection, and one for the IPv6 2Gbps fiber connection.



  • if i lived in the US, then i'd just call up pfSense support and ask for their advice…

    afaik there is no pfSense/freebsd device on the planet that can move 10Gbit wirespeed at this time. ESF&Netgate are working hard to change this;  they will be best suited to point you towards hardware that can handle 10Gbe in the future.



  • I don't have the speed fast as that, but I have 300mbps/150mbps fiber and I'm using:

    Intel(R) Xeon(R) CPU D-1537 @ 1.70GHz
    16 CPUs: 1 package(s) x 8 core(s) x 2 SMT threads

    As my pfsense box.  I used to have 2758 but ran into issues between vlan.

    The 1537 is the updated version of 1540 using less power.  I think the turbo on my SoC is 2.1Ghz.  Anyhow, I never see the load go passed 5-7% anyways.

    What works with this is the onboard 10GB SFP+ ports which connect to my C2960X switch with 2 10GB SFP+.  I have no issues getting nearly 1GBS throughput between machines in different vlans and of course maximum throughput on fiber.

    Not sure if I can test and saturate 10GB link somehow within the network between vlans…



  • Thank you for the detailed informations about that Xeon D-15xx platform, it is not so widely spread
    in the wild now together with pfSense I assume.

    Not sure if I can test and saturate 10GB link somehow within the network between vlans..

    At the PRTG homepage you will be able to download a freeware tool called "server stress tool" and with them
    you might be able to produce a real huge amount of network traffic that will be perhaps able to saturate the
    entire network. If you are interested to test it out here is the link to the download. Link
    Please be careful with that tool it is really powerful and can freeze a whole network.



  • @iamlucas:

    Here is a link to the Comcast documentation
    https://drive.google.com/file/d/0B8e0wvBZ26DadUR2OXp1blg3azVrSzJZRjFUMjRabzFQQ3Nv/view?usp=sharing

    From what I can understand, they are only using the Juniper device for the link handoff.
    This router is rented to me for a very low price of $20 USD per month. They are not making any money on the rental for sure.

    Specifically, to use the fiber link, they suggest I need the following equipment:
      10G capable Router/Layer 3 switch with at least 1 10Gbps SFP+ cage
      10G SFP+ 850nm MMF Transceiver
      MMF LC Jumper

    Comcast will provide one IPv4 static IP address for the 1Gb connection, and one for the IPv6 2Gbps fiber connection.

    You should be able to use a pfSense rig with a SFP+ NIC (either onboard or using the Chelsio adapter) and a SFP+ direct attached cable (this is significantly cheaper than buying optical transceivers).

    I have a c2758 pfSense box running in my office with Suricata and inter-VLAN routing.
    I doubt it can do >4Gbps (which you can hit if your lines are symmetric) with IDS turned on but that remains to be seen since I don't have a use case that requires more than 2Gbps transfers.
    Aside from that, it all works reasonably well. Have 16 vlans running on a single Lagg group (4 x 1GbE) across 2 switches and it hasn't thrown me any curveballs yet.



  • I have a c2750 pfsense box running at home on a gigabit connection. With Sucicata turned on the CPU hits 100% at around 210Mbps. With any sort of IPS/IDS feature turned on, you will have to go with a Xeon processor AFAIK to push 1Gbps or higher.



  • I have a c2750 pfsense box running at home on a gigabit connection.

    That board has no Intel QuickAssist, but it comes together with TurboBoost and so did you
    enable the PowerD (hi adaptive)?

    With Sucicata turned on the CPU hits 100% at around 210Mbps.

    Suricata is now multi CPU core usage and that is then the side effect of lower end Atoms!

    With any sort of IPS/IDS feature turned on, you will have to go with a Xeon processor AFAIK to push 1Gbps or higher.

    An Intel Core i3 or i5 will do the job too, but the Xeon E3 is more electric power saving.



  • That board has no Intel QuickAssist, but it comes together with TurboBoost and so did you
    enable the PowerD (hi adaptive)?

    I did indeed. And although it doesn't have QuickAssist, it does have AES-NI on chip. For the small amount of encryption I'm doing for home, it seems to be plenty.

    An Intel Core i3 or i5 will do the job too, but the Xeon E3 is more electric power saving.

    I'm not a expert, and I could very well be completely off base, perhaps you're correct. I'll just say I'd have to see it to believe it. An i3 doing IPS inspection at 1Gbps seems like a far stretch to me.



  • @Hegemon:

    That board has no Intel QuickAssist, but it comes together with TurboBoost and so did you
    enable the PowerD (hi adaptive)?

    I did indeed. And although it doesn't have QuickAssist, it does have AES-NI on chip. For the small amount of encryption I'm doing for home, it seems to be plenty.

    An Intel Core i3 or i5 will do the job too, but the Xeon E3 is more electric power saving.

    I'm not a expert, and I could very well be completely off base, perhaps you're correct. I'll just say I'd have to see it to believe it. An i3 doing IPS inspection at 1Gbps seems like a far stretch to me.

    An i3 does 150Mbps with IPS using about 2-3% of its capacity with decent sized rules loaded. With Snort (fully loaded with all rules) it hovers around 6-8%. I have tested this on the latest 2.3.1 with 8GB RAM. 85% of my RAM gets used for loading all Snort rules plus Squid with ClamAV and SquidGuard. Moved to an i5 a little while ago or else I would had posted a snapshot of the CPU usage.

    The CPU processing would ofcourse change as the speed increases, but I presume it should be able to do at least 500Mbps without breaking a sweat.



  • @iamlucas:

    Here is a link to the Comcast documentation
    https://drive.google.com/file/d/0B8e0wvBZ26DadUR2OXp1blg3azVrSzJZRjFUMjRabzFQQ3Nv/view?usp=sharing

    From what I can understand, they are only using the Juniper device for the link handoff.
    This router is rented to me for a very low price of $20 USD per month. They are not making any money on the rental for sure.

    Specifically, to use the fiber link, they suggest I need the following equipment:
      10G capable Router/Layer 3 switch with at least 1 10Gbps SFP+ cage
      10G SFP+ 850nm MMF Transceiver
      MMF LC Jumper

    Comcast will provide one IPv4 static IP address for the 1Gb connection, and one for the IPv6 2Gbps fiber connection.

    This looks like a single 2Gbps connection to me.  They're making it easy to connect with commodity equipment by providing a router that can handle up to 1Gbps; anything beyond that will require 10Gbps networking in your home.  That's how I read it anyway.  With two static IPs you could set up both your pfsense router and leave theirs up and running, or ditch theirs and do everything on your (presumably) 10Gbps network.



  • @Hegemon:

    I'm not a expert, and I could very well be completely off base, perhaps you're correct. I'll just say I'd have to see it to believe it. An i3 doing IPS inspection at 1Gbps seems like a far stretch to me.

    Running Suricata on a Pentium G3220 (which is slower than a Core i3) and Suricata uses ~80% at 937Mbps (about the limits of my gigabit line).



  • I'd love to see your follow up on what you ended up doing.  I'm moving in December to a home with a 1GbE handoff from AT&T and I'm going to need to replace my SG-2220 firewall with something that can handle the increased throughput.

    I don't want to venture away from PFsense but I'm looking around at alternatives simply because of price.  The PFsense sales team told me the hardware they sell that can handle 1GbE will cost me $1,799 to own.

    I'm a fan of doing things with open source software but it's hard to say that it's worth $1,500 more to buy a pfsense unit when a competitor is so much more cost effective.

    Either way I'll end up buying pfsense gold because this project is awesome.


  • Banned

    @cenal:

    I'd love to see your follow up on what you ended up doing.  I'm moving in December to a home with a 1GbE handoff from AT&T and I'm going to need to replace my SG-2220 firewall with something that can handle the increased throughput.

    I don't want to venture away from PFsense but I'm looking around at alternatives simply because of price.  The PFsense sales team told me the hardware they sell that can handle 1GbE will cost me $1,799 to own.

    I'm a fan of doing things with open source software but it's hard to say that it's worth $1,500 more to buy a pfsense unit when a competitor is so much more cost effective.

    Either way I'll end up buying pfsense gold because this project is awesome.

    Check Point 750. Can be bought for under $600. Provides throughput of 1 Gbps with encryption throughput of 500 Mbps.



  • You are asking questions without listing your requirements.  Give us your exact requirements and we can help you out.

    Most soho pfsense devices handle gigabit but I think your problem is 2gbps….

    I would just build something.



  • Im kind of in the same boat.  I have a 1 gig synchronous fiber to the home connection.  I am having a hard time finding something that can handle the throughput without dropping packets.  I am using an old Lanner Fw8760 that has an i3 in it, and 4 gig of ram with 8 Intel nics that works great, but I need to put it back in my datacenter so I need something at home that will work just as good.



  • What did you end going with? I'll be getting service in a few weeks.

    @CubedRoot:

    Im kind of in the same boat.  I have a 1 gig synchronous fiber to the home connection.  I am having a hard time finding something that can handle the throughput without dropping packets.  I am using an old Lanner Fw8760 that has an i3 in it, and 4 gig of ram with 8 Intel nics that works great, but I need to put it back in my datacenter so I need something at home that will work just as good.



  • @Hegemon:

    That board has no Intel QuickAssist, but it comes together with TurboBoost and so did you
    enable the PowerD (hi adaptive)?

    I did indeed. And although it doesn't have QuickAssist, it does have AES-NI on chip. For the small amount of encryption I'm doing for home, it seems to be plenty.

    An Intel Core i3 or i5 will do the job too, but the Xeon E3 is more electric power saving.

    I'm not a expert, and I could very well be completely off base, perhaps you're correct. I'll just say I'd have to see it to believe it. An i3 doing IPS inspection at 1Gbps seems like a far stretch to me.

    I have an i3 @4.1  with snort and suricata(for testing purposes)  and i get 950 of a gigabit link with 40/50 % of cpu usage. If they are correctly configured, it proves that one must not underestimate an i3.