WAN Port Bonding / Link Aggregation Group / DOCSIS 3.1
-
Hello,
I am new to pfsense but so far from doing research on what my options are pfsense seems to make the most sense. Recently, my cable internet provider, Cox, has implemented the new DOCSIS 3.1 standard, currently offering me an internet plan of 1Gbps Download and 35Mbps Upload speeds. Since I plan on upgrading to this tier I went ahead and did research on DOCSIS 3.1 modems and decided to go with the Motorola MB8600 ( https://motorolanetwork.com/catalog/product/view/id/1056/s/mb8600/category/111/ ). The Motorola MB8600 has 4 x 1 Gigabit ports and with its current firmware it can support LAG for Port Bonding using its ports 1 and 2 but was also told by a Motorola engineer that in the future they plan on enabling the use of all 4 Gigabit ports for LAG / Port Bonding for speeds closer to 4 Gbps.
My issue is that I can't find a "consumer level" router that has 2 to 4 WAN ports that would support Port Bonding / Link Aggregation Group to achieve higher throughput. Hence, I found pfsense and I am starting to believe this will be the way to go.
Can anyone clarify if I will be able to use pfsense to fully take advantage of this modem and future proof my home router/routing capabilities? I can also post the hardware I was thinking of using to build my router with as well if interested and maybe get your input on that?
-
Ive got the Motorola but have the earlier firmware still. Modems Ive bought for remote sites up till last week still have that firmware. The newest firmware being rolled out by some ISP's give the user the ability to turn on the port bonding on the modem GUI.
8600-6.1.1.19 is what your looking for.
Testing the 2.4.2 snapshots I can finally get my LAGG pair to pick up an address over the modem. Earlier versions of pfSense did not seem to work with it.
But keep in mind I cannot yet make my modem bond. Just testing the LAGG on pfSense right now.https://www.dslreports.com/forum/r31416209-Zoom-Zoom-s-Motorola-MB8600-DOCSIS-3-1-Modem-Arrives-This-Month
Ive been beating on this modem pretty good for about 6 months now. Stable unit!
-
That's good to know. I went ahead and checked and right now it seems that I am currently on Software Version 8600-6.1.1.13 . I reached out to Motorola tech support and they said the following when they replied:
"I logged into the modem to verify your questions, as it turns out since this is a bridge modem, there are no settings to be changed. The LAG should work automatically if your router has it enabled."
Keep in mind that the tech told me this and my modem software version is 8600-6.1.1.13 and not *19. So its either a case of the tech not knowing the fine details or they know something that I don't. I will also reach out to Cox tech support and see what they have to say about LAG / Port Bonding on this modem.
–
All that being said with the Motorola, I'd like to ask advice on my future pfsense build planning and/or get feedback to see what some people think or to question me and perhaps give me a better option?
-
So far this is what I have planned but I am not firm or 100% on the parts exactly. This is just my first or second effort in choosing my parts. So when it is possible, I plan on using the integrated dual gigabit ports to be my LAG configured WAN Ports coming from my Motorola MB8600 and in the future once it is enabled turning my Inter PRO/1000 QuadPort NIC in the a LAG / WAN of the 4 ports.
Let me know if any of this might feel a bit too overboard? But I'd like to think this can be somewhat future proof with me just upgrading NICs at some point?
Motherboard (This MB has dual ethernet ports / integrated 2 x 1 Gigabit Ports):
Intel S1200BTSR Micro ATX Server Motherboard LGA 1155 DDR3 1333 - https://www.newegg.com/Product/Product.aspx?Item=9SIA4MR6HG1153&cm_re=S1200BTSR--9SIA4MR6HG1153--ProductProcessor:
Intel Xeon E3-1270 V2 Ivy Bridge 3.5GHz (3.9GHz Turbo) 4 x 256KB L2 Cache 8MB L3 Cache LGA 1155 69W CM8063701098301 Server Processor - https://www.newegg.com/Product/Product.aspx?Item=9SIABB75KY5965Memory:
4 x 4GB DDR3 240-pin Unbuffered ECC (not sure which brand etc)NIC1:
Intel PRO/1000 PTQuadPort - https://www.amazon.com/gp/product/B000OZC98C/ref=ox_sc_act_title_4?smid=A3O6GNX9CB6SWD&psc=1NIC2:
10Gtek for Intel E10G42BTDA 82599ES Chip 10GbE Ethernet Converged Network Adapter X520-DA2, PCI-E X8 Dual SFP+ Port - https://www.amazon.com/gp/product/B01DCZCA3O/ref=ox_sc_act_title_5?smid=AE2OZG2NN3099&psc=1Power Supply:
EVGA 400 N1, 400W, 2 Year Warranty, Power Supply 100-N1-0400-L1 - https://www.amazon.com/gp/product/B00LV8TZAG/ref=ox_sc_act_title_2?smid=ATVPDKIKX0DER&psc=1 -
what would be the point of this? you only get 1 gbit from isp … i see no reason to setup a lag.
also: 1+1 is not 2
-
1+1 is not 2 seems to fall into the transit network area of understanding around here ;)
Vs the support for lagg the modem should support 802.3bz or 10ge on the Ethernet side.. So you can actually get above 1ge Docsis 3.1 supports what 10ge max down with 1 up.. How does the 4ge in a lagg help in that?
-
Bare with me as I try to explain myself.
From what I have read DOCSIS 3.1 is capable of being full duplex of 10 Gbit/s Downstream and 10 Gbit/s Upstream, supports fully symmetrical speeds. That being said if my ISP would offer it to me or if I would even want to pay for that is a totally different question.
Currently my ISP (Cox) is only offering for their top tier 1 Gbit/s Downstream and 35 Mbit/s Upstream. I know that currently, some people that I know that have this service (depending on the strength of the signal their cable modem is receiving) are getting speeds of more or less around 930-975 Mbit/s Downstream and their Upstream is consistently at least 35-42 Mbit/s Upstream. They are all feeding their Router from a single 1ge ethernet port and no one is ever getting exactly 1 Gbit/s Downstream or more. (I have heard of people around the country with 1 Gbit/s tier internet packages and using lagg over 2 x 1ge ethernet ports getting consistently around 1-1.2 Gbit/s since their medium is capable of ~2 Gbit/s).
Previously they had an internet tier (which is currently my tier) that is 300 Mbit/s Downstream and 30 Mbit/s Upstream but get on average around 320-355 Mbit/s Downstream and 30-35 Mbit/s Upstream.
The reason I would want to use lagg is to make sure I had more than enough overhead to accomodate 1 Gbit/s since I would be paying for 1 Gbit/s I'd want to get that at minimum. The modem has this capability ( https://motorolamentor.zendesk.com/hc/en-us/articles/115010589128 ) and I would like to utilize it and not to mention be able to use the higher packages that would be offered in the future 2 Gbit/s etc. Theoretically, the Motorola MB8600 should be able to handle speeds up to 3.8 Gbit/s from the ISP if they offered a 4 Gbit/s tier, but due to the overhead I couldn't get exactly 4 Gbit/s, the same way I wouldn't be able to get the full 1 Gbit/s if I upgraded to the 1 Gbit/s tier using only 1 ethernet Gbit/s port.
I'm not sure what you are referring to with the 1+1 analogy? I thought that if you had setup 802.3ad LACP or lagg on two 1ge ports that they would be capable of higher throughput of something closer to 2ge?
Also, keep in mind that the 4ge Port Bonding is not yet available but will be in the future as stated by Motorola, to support higher throughput and internet speeds offered via DOCSIS 3.1.
-
"From what I have read DOCSIS 3.1 is capable of being full duplex of 10 Gbit/s Downstream and 10 Gbit/s Upstream"
No what you read is that "full duplex" docsis 3.1, which is an extension of docsis 3.1 is touting symmetrical 10/10
" I thought that if you had setup 802.3ad LACP or lagg on two 1ge ports that they would be capable of higher throughput of something closer to 2ge?"
It is with lots of clients and lots of destination macs.. Ie like over an uplink in a switching setup.. Not all laggs are created equal.. There is a HASHING method that determines what physical interface/wire the packets go out on, etc..
When you talking 1 mac talking to 1 mac.. Good luck seeing traffic load share across the physical connections. In the case this sort of connection.. What mac would be talking to what mac.. Your pfsense mac to your gateway mac..
So what method are they using with this lagg that is going to load share the connections over the pipe.. When you start moving traffic over different physical devices. The nic and the wire and then the other nics on the other end of your lagg. When you split up the gain more bandwidth you no introduce issues with out of order packets.. Than just cause slow down in the traffic, retrans, etc. etc. Buffering if trying to queue them and put them back in the correct order, etc..
If you have different sessions sending data, client A could use path 1, client B talking could use path 2 and you no don't introduce out of order packets, etc.
If they want to provide 10ge up and down - then the solution is 10gig interface.. Or higher even… Not putting multiple smaller connections together in a lagg or bond.. Not the correct solution!!
Here is good starting article on the whole 1+1 not = 2 thing... its from 2010... But it should give you the idea about how throwing more wires at the thing does not mean you get 1+1+1+1 etc.. for your pipe.. Its just you have 4 1ge pipes, not a 4ge pipe..
http://packetpushers.net/the-scaling-limitations-of-etherchannel-or-why-11-does-not-equal-2/So how exactly are they going to hash the traffic to send it across your 2 or 4 paths that allow for any client to see more than the limit of physical pipe.. And not run into any sort of packets out of order issue? Keep in mind then in normal ipv4 setup, especially in the home you will be behind a napt and source IP is always going to be the same.. For all your sessions, etc.
The point heper is trying to make I believe and I agree with is 1+1 does not equal 2 just because you lagg, etherchannel, port channel, nic bonding - whatever term you want to use, etc. So you thinking you are going to see something with this lagg is highly unlikely, even your service was more than 1+ up and down.. What are you clients connected at on the lan side btw? Are 10 or 802.3bz? What sort of traffic are you going to be doing that you will be exceeding more than 1 gig on the uplink? How many boxes will you have streaming netflix? ;)
edit:
Think of it this way.. You have a 4 lane highway.. Can your car drive in all 4 lanes at once? There is a speedlimit in this lane. Say you have 4 people and you all get in different cars and drive in different lanes.. Are you going to get there any faster? But now the traffic in these different lanes all have different amounts of cars in them.. So your 4 people don't get there in order when they are suppose to.. The person that left 3rd might actually get there before the person that left 1st.. The waiter not going to give you your table until your all there.. Shoot he might even say sorry come back again... So now you got to all leave the house again and hope you arrive at the same time in the correct order.So while you could all get in your cars and drive in the same lane right after each other, so now you get there in the proper order.. You have to obey the speedlimit of that 1 lane. But another party going somewhere else could drive in the other lane. So if you work out total bandwidth if you put lots parties all driving their group of cars staying in their own lanes, but using all 4 lanes then yeah you get 4 x the speedlimit.. But no party going to 1 dest even sees more than the speedlimit of the 1 lane.
-
I think I "semi-understand" some of what you just mentioned johnpoz. Please correct me if I understood incorrectly:
"I thought that if you had setup 802.3ad LACP or lagg on two 1ge ports that they would be capable of higher throughput of something closer to 2ge?"
"It is with lots of clients and lots of destination macs.. Ie like over an uplink in a switching setup.. Not all laggs are created equal.. There is a HASHING method that determines what physical interface/wire the packets go out on, etc.."So you are saying that even if the lagg is 1+1 "2ge" that it won't hit that speed or throughput mark unless there is enough traffic going through it that adds up to ~2ge? Even if this was the case I would still like to have the capability of doing it even if I never hit that mark with all my clients on from the LAN side, my traffic wouldn't be limited to ~1ge throughput but ~2ge throughput instead. To use an analogy I guess you can say that "I'd like a Ferrari and will most likely drive it around 65-80 miles an hour most of the time but would like to know that if I wanted to I can go 150 miles an hour and know that I can handle that speed, plus I have an extra $200,000 to buy a Ferrari and I can". Instead of being just at the limit of what I can handle I'd like to be able to have the capability of handling more even if currently most of the time I stick to my current limit. I hope I am making sense? lol ;)
"If they want to provide 10ge up and down - then the solution is 10gig interface.. Or higher even… Not putting multiple smaller connections together in a lagg or bond.. Not the correct solution!!"
I fully agree with you on the 10gig interface or higher part. But check out the Motorola MB8600 technical specifications, why would they even design it the way they did if it wasn't to be used in that way?
In the end, at the consumer level there are no routers that can do what I want or even work with the Motorola MB8600 to its capabilities. That is why building my own pfsense router to me makes the most sense. I'm not a network engineer but can configure and figure out any consumer level router to do what I want or what it is capable of no problem. I am eager to learn more on pfsense after building my own.
"What are you clients connected at on the lan side btw? Are 10 or 802.3bz? What sort of traffic are you going to be doing that you will be exceeding more than 1 gig on the uplink? How many boxes will you have streaming netflix? ;)"
My main client (PLEX server) is using TN9510 10GBase-T/NBASE-T Ethernet Adapter SPF+ ( https://www.amazon.com/gp/product/B00QX4XTE6/ref=oh_aui_search_detailpage?ie=UTF8&psc=1 ) and I know that I am not using anywhere near its full potential but that being said I know for sure that I have enough overhead to support multiple 4K stream concurrently ;D "set it and forget it type mentality".
-
what would be the point of this?
Educational purposes.
https://www.freebsd.org/doc/handbook/network-aggregation.html
:)
edit- Im using FEC on mine right now.. I am passing traffic on both ports. Whether its working "correctly" or not… ? I haven't added any latency to the connection, but in fact have lost 2ms on my WAN according to the graph since employing LAGG.
-
Did anyone in this thread try LAG with their Motorola MB8600? I have it and pfSense, too.
Really curious if any more progress has been made on this front, as I just signed up for gigabit with Comcast last night. :o
Thanks!
-
I have, it works. Enable Port Bonding on your MB8600. Then configure the pfSense to use a LACP LAGG to connect to ports 1,2 on the MB8600. Assign that LAG to your WAN interface, and you should be good to go.
-
I found this old topic searching after running into problems with LACP on my Arris SB8200.
I have gigabit service but have only been getting 750-800mbps and thought it might be an issue with a bottleneck somewhere in my hardware. (using a supermicro C2758 motherboard) Recently, a firmware update was pushed out to enable LAG in the SB8200. Figured I might as well. I set up an LACP under the LAGG interface and connected my second ethernet cable.
Results...I noticed no difference in speed. Of course this makes sense since my network is nowhere near being even close to saturated. What I did notice is a few errors, all corrected, on all of my downstream bonded channels. Also, in pfSense, several hundred errors in the "errors out" on my WAN interface. I ended up turning off the LAG and going back to a single interface and all the errors went away.
-
I too didn't quite understand how LAG worked and tried bonding my desktop to my switch in the hope that if one file transfer was hitting 1Gig, I could still access the other HDDs in the NAS at the same time (which has 10Gig). It didn't work reliably at all, just slowed down all file transfers!
Even maxing out 1Gig to my NAS and using the other link to access the Internet seemed sub-optimal, possible due to the second NIC being USB.
Clearly as mentioned above, LAG doesn't benefit a client, it benefits the server when multiple clients are hitting it at the same time. Or switch to switch uplinks where again it will load balance the clients, not the individual traffic from a single client.
-
I too had issues enabling LAG on my SB8200.
When enabled on the modem and using LACP for my lag interface in pfsense this is what i noticed.
1: lots of corrected errors. millions per second showing up on the modem status page
2: Router would cyclically lose internet on both ipv4 and ipv6 every 50 minutes.
3: When "enabled lag" was checked I could not access the spectrum analyzer on port 8080
I'm going to guess it's an issue with the sb8200 firmware and lag.
Or maybe it does not use LACP?The modem losing internet every 50 minutes would cause the alerts to start piling up in the pfsense dashboard.
-
Did anyone get running LAG with a Technicolor TC4400 cable modem? I got it running but the speed is much worse than without.
