New TCP congestion algorithm - BBR

yon 0 · Sep 4, 2022, 10:57 AM

Compile New Kernel
Now we are ready to compile the new kernel to activate the TCP BBR.
Create a new file RACK (you can use any name you want) in the folder /usr/src/sys/amd64/conf/RACK. Inside the file will need to add the options for TCP BBR and the file should look like this:
$ cat /usr/src/sys/amd64/conf/RACK

include GENERIC

ident RACK

makeoptions WITH_EXTRA_TCP_STACKS=1

options RATELIMIT

options TCPHPTS
Next step is to run the following commands (in order) to compile the kernel (this step will take a while)
1) make -j 16 KERNCONF=RACK buildkernel

2) make installkernel KERNCONF=RACK KODIR=/boot/kernel.rack

3) reboot -k kernel.rack
The old kernel will be available but with the name "kernel.old". After rebooting, will use the new kernel because of the command "reboot -k kernel.rack", however to make it persistent will require to adjust couple of files (will explain later in this article).
Once you have built, installed and rebooted to the new kernel we need to load the RACK kernel module tcp_bbr.ko:
kldload /boot/kernel.rack/tcp_bbr.ko 
Now you should see the new module in the functions_available report, by typing the command:
sysctl net.inet.tcp.functions_available
The output will be:
net.inet.tcp.functions_available

Stack                           D Alias                            PCB count

freebsd                         * freebsd                          3

bbr                              bbr                               0:
Now will require to change the default to TCP BBR:
sysctl net.inet.tcp.functions_default=bbr
and the output will be:
net.inet.tcp.functions_default: freebsd -> bbr

root@freebsd # sysctl net.inet.tcp.functions_available

net.inet.tcp.functions_available:

Stack                           D Alias                            PCB count

freebsd                           freebsd                          3

bbr                            * bbr                               0k
After rebooting, will use the old Kernel, but we can make it persistent.
Modify the Loader
To force FreeBSD to use the new Kernel after rebooting, will require to adjust 3 files:
/etc/sysctl.conf
/etc/rc.conf
/boot/loader.conf
Inside /etc/sysctl.conf we can also add command for optimisation, including the command to enable TCP BBR as a default congestion control function.
The file should looks like this:
$ cat /etc/sysctl.conf

# $FreeBSD$

#

#  This file is read when going to multi-user and its contents piped thru

#  ``sysctl'' to adjust kernel values.  ``man 5 sysctl.conf'' for details.

#




# Uncomment this to prevent users from seeing information about processes that

# are being run under another UID.
#security.bsd.see_other_uids=0




# set to at least 16MB for 10GE hosts

kern.ipc.maxsockbuf=16777216

# set autotuning maximum to at least 16MB too

net.inet.tcp.sendbuf_max=16777216  

net.inet.tcp.recvbuf_max=16777216

# enable send/recv autotuning

net.inet.tcp.sendbuf_auto=1

net.inet.tcp.recvbuf_auto=1

# increase autotuning step size 

net.inet.tcp.sendbuf_inc=16384 

net.inet.tcp.recvbuf_inc=524288 

# set this on test/measurement hosts

net.inet.tcp.hostcache.expire=1

# Set congestion control algorithm to Cubic or HTCP

# Make sure the module is loaded at boot time - check loader.conf

# net.inet.tcp.cc.algorithm=cubic  

net.inet.tcp.cc.algorithm=htcp

net.inet.tcp.functions_default=bbr

net.inet.tcp.functions_inherit_listen_socket_stack=0
The 2nd change is to add the following line inside /etc/rc.conf:
kld_list="/boot/kernel.rack/tcp_bbr.ko"
and finally the last change is to modify the /boot/loader.conf file, should look like this:
$ cat /boot/loader.conf

###  Basic configuration options  ############################                                                                                         

kernel="kernel.rack"         # /boot sub-directory containing kernel and modules                                                                                                                    
bootfile="kernel.rack"       # Kernel name (possibly absolute path)

module_path="/boot/kernel.rack"	# Set the module search path

cc_htcp_load="YES"
After modifying the files, reboot the server and you should see the HTCP algorithm as well as TCP BBR function as the chosen options:
$ sudo   sysctl net.inet.tcp.cc.available

net.inet.tcp.cc.available: 

CCmod           D PCB count

newreno           0

htcp            * 6




$ sudo sysctl net.inet.tcp.functions_available

net.inet.tcp.functions_available: 

Stack                           D Alias                            PCB count

freebsd                           freebsd                          5

bbr                             * bbr                              1

yon 0 · Sep 4, 2022, 11:02 AM

https://www.linkedin.com/pulse/frebsd-13-tcp-bbr-congestion-control-andrew-antonopoulos/?trk=articles_directory

who try install bbr in freebsd?

yon 0 · Sep 4, 2022, 11:01 AM

In addition, I recently tested using the Quic protocol for network transmission, and the vpn test is more than 5-10 times faster than the existing wireguard. Especially in the case of a bad network environment, it is more obvious.
I think pfsense should be more aggressive in innovating technology instead of using very, very old technology. It always feels outdated.

mikehu44444 · Jun 18, 2024, 3:27 PM

Hi Guys
Late to the party, I just wrote an article on how to build custom pfSense bbr kernel.
You can try out my custom build kernel at your own risk.
Here is the link:
https://github.com/mikehu404/pfsense-bbr

w0w · Jun 19, 2024, 2:58 AM

@mikehu44444 said in New TCP congestion algorithm - BBR:

https://github.com/mikehu404/pfsense-bbr

Great work!
In reality, there are very few situations where pfSense acts as a client or a server.
It would be nice to re-test the pfSense client's speed, but not pfSense itself.
I don't think we will see any difference. The tuning that is applicable should be applied to all FreeBSD kernels during the test.

mikehu44444 · Jun 20, 2024, 8:22 AM

@w0w said in New TCP congestion algorithm - BBR:

In reality, there are very few situations where pfSense acts as a client or a server.

Yes, most people use pfSense as a gateway,
but for me, I use pfSense on VPS as a web app server & VPN server.
This is mainly due to costs, by doing so I only need to rent one server instead of three. Without the complex networking between servers.
And it can be further improve upon by using Unix sockets to connect app in jails with haproxy in pfSense without sacrificing security.

That's why I also enable jail vnet & fusefs capability in the custom kernel.
I believe BBR would be beneficial to haproxy and vpn service.

w0w · Jun 20, 2024, 9:51 AM

@mikehu44444
Actually, this makes the situation even more interesting. Is it possible to run tests from client PCs or virtual machines, with BBR enabled on pfSense and without, not VPN but NAT? Can you do it? So we have full picture and theories confirmed.

Sergei_Shablovsky · Jun 23, 2024, 2:36 PM

@mikehu44444 said in New TCP congestion algorithm - BBR:

Hi Guys
Late to the party, I just wrote an article on how to build custom pfSense bbr kernel.
You can try out my custom build kernel at your own risk.

THANK YOU SO MUCH!

Is it possible to make this also in Plus version of pfSense?

Could You be so please to make PR or proposal to Netgate’s DevTeam to implement this BBR/BBR2 support in both CE and pfSense+ versions?

Sergei_Shablovsky · Jun 20, 2024, 10:39 AM

@w0w said in New TCP congestion algorithm - BBR:

@mikehu44444
Actually, this makes the situation even more interesting. Is it possible to run tests from client PCs or virtual machines, with BBR enabled on pfSense and without, not VPN but NAT? Can you do it? So we have full picture and theories confirmed.

Agree! Please, make this test carefully!

Sergei_Shablovsky · Jun 23, 2024, 2:47 PM

@yon-0 said in New TCP congestion algorithm - BBR:

In addition, I recently tested using the Quic protocol for network transmission, and the vpn test is more than 5-10 times faster than the existing wireguard. Especially in the case of a bad network environment, it is more obvious.

VERY INTERESTING!

Please explain details how You make working QUIC congestion control in TCP/IP stack in pfSense CE 2.7.X ?
(I thinking for that You may recompiling the FreeBSD kernel with certain options in configuration file, like

# Congestion control algorithms
options TCP_BBR            # Enable BBR
options TCP_BBR2           # Enable BBR2
options TCP_CDG            # Enable CDG
options TCP_QUIC           # Enable QUIC

and also

net.inet.tcp.cc.algorithm=bbr2

must be appended to /boot/loader.conf.

Am wrong ?

Is it possible to make this also in Plus version of pfSense?
(I thinking no, because only Netgate DevTeam have access to private Netgate’s repo with pieces of proprietary code.)

I think pfsense should be more aggressive in innovating technology instead of using very, very old technology. It always feels outdated.

Heh! I wrote the same many times on this forum… ;)

mikehu44444 · Jun 24, 2024, 9:22 PM

@Sergei_Shablovsky said in New TCP congestion algorithm - BBR:

Please explain details how You make working QUIC congestion control in TCP/IP stack in pfSense CE 2.7.X ?

It is possible to use QUIC for VPN, but currently QUIC is mainly use for http3.
You should ask how to enable QUIC with haproxy on pfSense, as this is the right question.

@Sergei_Shablovsky said in New TCP congestion algorithm - BBR:

options TCP_BBR # Enable BBR
options TCP_BBR2 # Enable BBR2

To enable bbr on freebsd.
I don't think BBR2 is available on freebsd yet.

@Sergei_Shablovsky said in New TCP congestion algorithm - BBR:

options TCP_CDG # Enable CDG

cc/cc_cdg          ## Enable CDG

@Sergei_Shablovsky said in New TCP congestion algorithm - BBR:

net.inet.tcp.cc.algorithm=bbr2
must be appended to /boot/loader.conf.

You should add this option via System Tunables

@Sergei_Shablovsky said in New TCP congestion algorithm - BBR:

Is it possible to make this also in Plus version of pfSense?

Although I never use the plus version before, but I believe the CE version and the plus version shares the same kernel, since you can just upgrade to plus version from CE version.

w0w · Jun 25, 2024, 7:59 PM

@mikehu44444 said in New TCP congestion algorithm - BBR:

but I believe the CE version and the plus version shares the same kernel, since you can just upgrade to plus version from CE version.

No. Plus is freebsd 15 and CE is 14, so kernels are different.
Actually you cant just modify plus version, because it is not free and not open source.
https://www.netgate.com/support/frequently-asked-questions-pfsense-plus#:~:text=Is%20pfSense%20Plus%20software%20open,Routing%2C%20and%20of%20course%20FreeBSD.

Sergei_Shablovsky · Jul 6, 2024, 12:59 AM

@mikehu44444 said in New TCP congestion algorithm - BBR:

To enable bbr on freebsd.

So, is this mean that this script would be working in pfSense CE version ?

#!/bin/sh

# Function to log messages
log_message() {
    echo "$1"
    logger -p local0.notice "RACK Enabler: $1"
}

# Function to restart networking services
restart_networking() {
    log_message "Restarting networking services..."
    service netif restart && service routing restart
    log_message "Networking services restarted"
}

# Function to display countdown
display_countdown() {
    local duration="$1"
    local message="$2"
    echo -n "$message"
    while [ "$duration" -gt 0 ]; do
        echo -n "$duration "
        sleep 1
        duration=$((duration - 1))
    done
    echo "0"
}

# Check if running as root
if [ "$(id -u)" != "0" ]; then
    log_message "This script must be run as root"
    exit 1
fi

# Step 1: Check available TCP stacks
log_message "Checking available TCP stacks..."
available_stacks=$(sysctl net.inet.tcp.functions_available)
log_message "$available_stacks"

if ! echo "$available_stacks" | grep -q "rack"; then
    log_message "RACK stack not available. Please ensure you're running FreeBSD 14 or higher."
    exit 1
fi

# Step 2: Load the RACK kernel module
if ! kldstat | grep -q tcp_rack; then
    log_message "Loading RACK kernel module..."
    if kldload tcp_rack; then
        log_message "RACK kernel module loaded successfully"
    else
        log_message "Failed to load RACK kernel module"
        exit 1
    fi
else
    log_message "RACK kernel module is already loaded"
fi

# Step 3: Set RACK as the default TCP stack
log_message "Setting RACK as the default TCP stack..."
if sysctl net.inet.tcp.functions_default=rack; then
    log_message "RACK set as default TCP stack"
else
    log_message "Failed to set RACK as default TCP stack"
    exit 1
fi

# Step 4: Verify the change
log_message "Verifying the change..."
new_default=$(sysctl net.inet.tcp.functions_available | grep -E "rack.*\*")
log_message "New default TCP stack: $new_default"

# Step 5: Make the change persistent
log_message "Making the change persistent..."
if grep -q "net.inet.tcp.functions_default=rack" /etc/sysctl.conf; then
    log_message "Persistent setting already exists in /etc/sysctl.conf"
else
    echo "net.inet.tcp.functions_default=rack" >> /etc/sysctl.conf
    log_message "Added persistent setting to /etc/sysctl.conf"
fi

# Step 6: Suggest restart
log_message "RACK has been enabled and set as the default TCP stack"
log_message "Networking services will be restarted automatically unless cancelled"

# Step 7: Wait for ESC key and restart networking if not pressed
log_message "Press ESC within 10 seconds to cancel automatic restart of networking services..."

if [ -t 0 ]; then  # Check if script is running in a terminal
    # Start countdown in background
    display_countdown 10 "Time remaining: " &
    countdown_pid=$!

    # Wait for user input
    read -t 10 -n 1 key

    # Kill countdown process
    kill $countdown_pid 2>/dev/null

    if [ "$key" = $'\e' ]; then
        echo  # Move to a new line after countdown
        log_message "ESC key pressed. Skipping network restart."
    else
        echo  # Move to a new line after countdown
        restart_networking
    fi
else
    log_message "Not running in an interactive terminal. Proceeding with network restart."
    restart_networking
fi

log_message "RACK enabler script completed successfully"

stephenw10 · Jul 6, 2024, 2:25 PM

@Sergei_Shablovsky said in New TCP congestion algorithm - BBR:

net.inet.tcp.functions_available

It would only work if the tcp_rack kernel module is present and it isn't included in 2.7.2.

Additionally pfSense use the gui system tunables instead of /etc/sysctl.conf so the persistent part would not work.

Sergei_Shablovsky · Jul 6, 2024, 3:18 PM

@stephenw10 said in New TCP congestion algorithm - BBR:

@Sergei_Shablovsky said in New TCP congestion algorithm - BBR:

net.inet.tcp.functions_available

It would only work if the tcp_rack kernel module is present and it isn't included in 2.7.2.

So, if I understand You correctly, we return to old queston "when Netgate compile kernel with CDG/BBR2/RACK/QUICK support for pfSense+ or CE ??!!!!" or workaround "how to recompile kernel with CDG/BBR2/RACK/QUICK support for pfSense CE" (as we see discussion abowe) ?

Additionally pfSense use the gui system tunables instead of /etc/sysctl.conf so the persistent part would not work.

No problem to modifying script to editing "system tunables" in config.xml ...:)

Sergei_Shablovsky · Jul 6, 2024, 4:04 PM

@w0w said in New TCP congestion algorithm - BBR:

@mikehu44444 said in New TCP congestion algorithm - BBR:

https://github.com/mikehu404/pfsense-bbr

Great work!
In reality, there are very few situations where pfSense acts as a client or a server.
It would be nice to re-test the pfSense client's speed, but not pfSense itself.
I don't think we will see any difference. The tuning that is applicable should be applied to all FreeBSD kernels during the test.

With all appreciation to Your R&D, but let's to make some small note that may explain so big difference in numbers:

Mostly, ISPs installing own separate Speedtest OOKLA server on their infrastructure (and sometime even routing all requests from users to own Seedtest server, if this possible) : because OOKLA Speedtest app (desktop, mobile or webapp) AUTOMATICALLY using the nearest Speedtest's server (selected by lowest lathency) and 90%+ OF USERS NOT CHOOSING OTHER TEST SERVER IN OTHER LOCATION in app settings, - THE SPEEDTEST's RESULTS WOULD BE a little INCORRECT (5-25%) from real;
Mostly, ISP making their own "Speedtest page" where in code it's own Speedtest's server are HARDLY PRE-SELECTED. So, the situation goes to item.1 above;
Because A LOT OF ISPs (in some US states (probably in a middle states or with less developed infrastucture states, in Western Europe, in MIddle Asia this percentage may be 80-95%) using core network equipment, and equipment on aggregate levels with simply OLD CONGESTION CONTROL (CC) like Tahoe, Reno, Westwood+, CUBIC, etc., this mean THE MODERN TCP/IP CC protocols WOULD BE ALWAYS WINNER IN REAL LIFE (probably NEXT 10-20years, up to the time, when MOST of equipment/os switch to modern TCP/IP CC, and then would be "next round").
So, ordinary unmodified FreeBSD v.14+ by default using CUBIC as the default TCP/IP CC algorithm. Previous versions used New Reno. However, FreeBSD supports a number of other choices.

P.S.
May be this is one of reasons why Netgate till now not including Speedtest (even as separate pkg) in CE or + version.
Because very hardly to explain to ordinary users "what this pesky Speedtest measurements mean".

BUT I HOPE one day I have a time to make separate pkg for Speedtest, FAST, Librespeed and even smokeping to giving ability to all pfSense users flawlessly using this tools to measure WAN UPLINKs on initial pfSense setup at home or small office.

w0w · Jul 7, 2024, 9:02 AM

@Sergei_Shablovsky said in New TCP congestion algorithm - BBR:

May be this is one of reasons why Netgate till now not including Speedtest (even as separate pkg) in CE or + version.
Because very hardly to explain to ordinary users "what this pesky Speedtest measurements mean".
BUT I HOPE one day I have a time to make separate pkg for Speedtest, FAST, Librespeed and even smokeping to giving ability to all pfSense users flawlessly using this tools to measure WAN UPLINKs on initial pfSense setup at home or small office.

Honestly, I don't know why WAN speeds need to be measured on the firewall. By its nature, pfSense is simply not designed for this, as it works in a pass-through mode, meaning the maximum speeds should be on the client PCs connected to pfSense, not the other way around. Of course, it would be great if the speeds were the same on both pfSense and the clients, but as I understand it, this is impossible and unnecessary overall.

As for BBR, I would conduct tests if I had the opportunity, but I'm not sure if it would offer any advantages in my configuration. Probably none.

I can't recall the exact reason why Netgate didn't even consider including this algorithm and compiling the necessary options in the kernel. Compatibility and stability, obviously, @stephenw10?

Sergei_Shablovsky · Jul 7, 2024, 4:01 PM

@w0w said in New TCP congestion algorithm - BBR:

@Sergei_Shablovsky said in New TCP congestion algorithm - BBR:

May be this is one of reasons why Netgate till now not including Speedtest (even as separate pkg) in CE or + version.
Because very hardly to explain to ordinary users "what this pesky Speedtest measurements mean".
BUT I HOPE one day I have a time to make separate pkg for Speedtest, FAST, Librespeed and even smokeping to giving ability to all pfSense users flawlessly using this tools to measure WAN UPLINKs on initial pfSense setup at home or small office.

Honestly, I don't know why WAN speeds need to be measured on the firewall. By its nature, pfSense is simply not designed for this, as it works in a pass-through mode, meaning the maximum speeds should be on the client PCs connected to pfSense, not the other way around.

Honestly, please spend 20min for searching in this pfSense Users Forum how much times users asking again and again "why my speeed to internet so low..."! A LOT OF!
Of course, part of this questions are from newbies which not read pfSense docs carefully and because of this make some misconfiguration OR just not really understanding all ground aspects about Networking.

But there are also a common PURE QUESTION: what is my UPLINK TO ISP SPEED?

@w0w, please told me, are YOU PERSONALLY not interested in to know WHAT IS REAL WAN UPLINK SPEED (i mean between Your WAN Eth port and Your ISP)?
Because may be something in ISP's hardware, cables, aggregate switches, etc. that IN YOUR CERTAIN CASE GIVE YOU LESS BANDWIDTH that you pay for?

And how for example personally You measure UPLINK TO ISP BANDWITH ?

Of course, it would be great if the speeds were the same on both pfSense and the clients, but as I understand it, this is impossible and unnecessary overall.

I not wrote about that. (That's not possible at all technically for a couple of reasons.)

As for BBR, I would conduct tests if I had the opportunity, but I'm not sure if it would offer any advantages in my configuration. Probably none.

Thank You so much! I'm waiting ! :)

I can't recall the exact reason why Netgate didn't even consider including this algorithm and compiling the necessary options in the kernel. Compatibility and stability, obviously, @stephenw10?

Stability? Compability ? Please explain, what You are about in details?
CDG, BBR/BBR2, RACK, QUIC working last 5 years with no any issues in a lot of facilities and DC.

And I still thinking the reason why Netgate not including the CDG, BBR/BBR2, RACK, QUIC support in their pfSense only because not making R&D bot on simulated (TRex or similar utility) and a real traffic and obtain a certain dataset that quickly pushing it to take a decision which CC to compile in kernel.

This decision also follow to impact on next 2(two) IMPORTANT items:

Is current Netgate applience (not U-rack, small models for home / small office use) able to working well with may be increased workload on NICs controllers and whole CPU/RAM/bus ?
Changing the factory's production process in current economical situation may be not good way. And of coarse this is INVESTING IN A FUTURE with NO IMMEDIATELY REVENUE RISE UP.
Because "Each congestion control algorithm is implemented as a loadable kernel module." would be a some sort of problem to EXPLAIN TO END USERS which option (CDG, BBR/BBR2, RACK, QUIC) make by default and (more important!) WHY USER MAY NEED TOCHANGE this DEFAULT CC.
As we here all see, 90%+ of network appliances users in SOHO become less and less educated, and even more and more stupid...

P.S.
Of what we are speaking? Look at reality: this tread are in "Off-topic section", other threads related to CC on this pfSense Users Forum are with significant small views.
At the same moment this 'small' changing the CC in pfSense may increasing bandwith utilisation, speed, etc. to + 10-30% (depend on setup and use case).

w0w · Jul 20, 2024, 6:04 AM

Good evening everyone

Did quick BBR and no-BBR tests, and the theory is absolutely confirmed. Even if you want to test your ISP speed with BBR on pfSense, you might not get the full speed, but it will be very close to it.
There's no benefit on client PC when BBR is active. The difference is statistically negligible, even with a small sample size.

Sergei_Shablovsky · Aug 21, 2024, 1:14 PM

@mikehu44444
So, what is about tests?