New TCP congestion algorithm - BBR

Harvy66

https://patchwork.ozlabs.org/patch/671069/

Google has been using it, lowers latency, increases throughput, works with modern AQMs, does not get starved by other algorithms in use. What's not to like? Best yet, does not require the receiver to interact in anyway special to get full benefit. Just upgrade your servers and watch the Internet be a nicer place for all.

heper

What's not to like?

it probably injects ads into your tcp steam :D
the patch appears to be for linux
unclear (for me) that is would have any gain to run this on a router

Harvy66

The patch may be for Linux, but the algorithm could be implemented by anyone, assuming they don't lay claim to a patent.

It wouldn't help a router/firewall, but it would be useful for any servers.

Nullity

I saw this post over on HackerNews and thought you might be interested, Harvy. The poster is a FreeBSD kernel dev. https://news.ycombinator.com/item?id=12681091

(Regarding TCP improvements in FreeBSD 11.)

It matters most for people doing 10-100gbps throughput, CPU usage will be lower and more stable in all cases though.
There has been a lot of improvement to many network card drivers in 11, and I am helping to push/fund the final integration of Matt Macy's "iflib" for the common intel em/igb/ixgbe drivers.
There are a lot of goodput improvements coming soon, which will affect all TCP users. I had Matt Macy upgrade TCP CUBIC to match 2016 RFC and most Linux behaviors (HyStart). Hiren Panchasara has been working full time for almost 2 years to address many other goodput and correctness issues in the TCP stack. Some of these are in 11, but the majority will hit in 11.1.
Another company is working on the recently announced BBR congestion control from Google and a TCP stack with RACK/PRR https://wiki.freebsd.org/DevSummit/201606/Transport. The end result of all this will be a more tightly integrated and coherent TCP implementation, which should make FreeBSD have the best network stack again in 2017 after falling behind for a while.

yon

How start the BBR

Fujitsu Labors was doing it in 2013 too, perhaps not the same but 30 times faster then the ordinary TCP
protocol we are using until today, Fujitsu Laboratories Ltd. - Press release

Harvy66

@BlueKobold:

Fujitsu Labors was doing it in 2013 too, perhaps not the same but 30 times faster then the ordinary TCP
protocol we are using until today, Fujitsu Laboratories Ltd. - Press release

That's an interesting read, but it seems Fujitsu's tech was a tunneling tech that wrapped around TCP between two tunneled networks and acted similar to a TCP accelerator by reducing ACK latency, compensating for packetloss, and allowing smaller transmission windowed TCP connections from being limited to bandwidth by high RTTs.

yon

now i using BBR in unbuntu17.10, it is work now.

i want to change the pfsense to BBR or cubic, it is can't change.

i add cc_cubic_load="YES" in loader.conf and net.inet.tcp.cc.algorithm=cubic in sysctl.conf

Shell Output - sysctl net.inet.tcp.cc.available show net.inet.tcp.cc.available: newreno

still can't change it.

who know how change it?

tman222

I know I'm late to the party, but I actually found out about this algorithm just recently as I was searching for network settings to tune for Linux hosts.

Ran some tests using TCP BBR and I have to say I'm quite impressed with the performance:

Performing a local test using Flent between two 10Gbit Linux hosts using TCP BRR and sitting on different network segments (i.e. the test was done across the firewall) resulted in more stable data transfer and lower latency. Using TCP - BBR I had no trouble pushing 14 - 16Gbit of traffic across the pfSense firewall (Flent is a bi-directional test) with latencies on average between 1 - 2 ms during the test. Using the prior (default) TCP congestion algorithm (Cubic) data transfer was less stable (more variability in bandwidth) and total bandwidth was a little lower as well. Latencies were closer to the 3 - 6ms range.
Performing a WAN test I also got better upload performance than before. I have a 1Gbit symmetric Fiber connection and using TCP BBR I saw higher upload speeds, especially over longer distances (e.g. between East Coast and West Coast). I use fq_codel to manage WAN traffic since I have 10Gbit hosts sending traffic into a 1Gbit interface -- it all seems to work quite well still with TCP BBR enabled on the hosts.

Harvy66

@yon Google is working on BBR2. Lots of improvements in making it both more friendly and more resilient.

tman222

@harvy66 said in New TCP congestion algorithm - BBR:

@yon Google is working on BBR2. Lots of improvements in making it both more friendly and more resilient.

Hi @Harvy66 - any ideas what specifically they are working on changing/updating? Thanks in advance.

yon 0

@harvy66

how i get the new BBR in freebsd or linux?

Sergei_Shablovsky

@tman222 said in New TCP congestion algorithm - BBR:

I know I'm late to the party, but I actually found out about this algorithm just recently as I was searching for network settings to tune for Linux hosts.

Ran some tests using TCP BBR and I have to say I'm quite impressed with the performance:

Performing a local test using Flent between two 10Gbit Linux hosts using TCP BRR and sitting on different network segments (i.e. the test was done across the firewall) resulted in more stable data transfer and lower latency. Using TCP - BBR I had no trouble pushing 14 - 16Gbit of traffic across the pfSense firewall (Flent is a bi-directional test) with latencies on average between 1 - 2 ms during the test. Using the prior (default) TCP congestion algorithm (Cubic) data transfer was less stable (more variability in bandwidth) and total bandwidth was a little lower as well. Latencies were closer to the 3 - 6ms range.

Performing a WAN test I also got better upload performance than before. I have a 1Gbit symmetric Fiber connection and using TCP BBR I saw higher upload speeds, especially over longer distances (e.g. between East Coast and West Coast). I use fq_codel to manage WAN traffic since I have 10Gbit hosts sending traffic into a 1Gbit interface -- it all seems to work quite well still with TCP BBR enabled on the hosts.

How You install it on pfSense ?

Please describe in details, if possible.

Sergei_Shablovsky

@tman222 said in New TCP congestion algorithm - BBR:

Using the prior (default) TCP congestion algorithm (Cubic) data transfer was less stable (more variability in bandwidth) and total bandwidth was a little lower as well. Latencies were closer to the 3 - 6ms range.

Cubic - is VERY old CC algorithm, and outdated even in 2012...

Better to compare QUIC and BBR2/BBR.

BTW, BBR (and BBR2) more pushed by Netflix (due they need effective netflow with less latency for their server farms), and QUIC are more pushed by Google (due they need effective netflow with less latency & big quantity of packet drops because last 8-9 years traffic goes more “mobile”).

yon 0

https://github.com/netflix/tcplog_dumper

Sergei_Shablovsky

@yon-0 said in New TCP congestion algorithm - BBR:

https://github.com/netflix/tcplog_dumper

If I understand the page You previously post (translate from China to Eng) there are only one way - rcompiling the kernel.

Using a FreeBSD -head (r363032 minimum, to have the extra TCP stack headers installed), compile a new kernel with BBR and extra TCP stack enabled:

And because pfSense CE open source, I able doing that, but in TNSR - definitely no.

Am I right ?

johneggers

This post is deleted!