Playing with fq_codel in 2.4

zwck

@mattund I watched the video linked before and end up with more question then before. Do we use masks or not. ? Do we use ECN or not, i remember on the upload side we should not use ECN.

occamsrazor

Not sure what changed but now my setup seems to be working OK...

https://www.dslreports.com/speedtest/39073625

This while doing speedtest:

Shell Output - ipfw sched show

00001:  50.000 Mbit/s    0 ms burst 0
q65537  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0      653   974276  0    0   0
00002:  40.000 Mbit/s    0 ms burst 0
q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2
  0 ip           0.0.0.0/0             0.0.0.0/0       59     3149  0    0   0

Is fq_codel adaptive to bandwidth in some way? I'm wondering what to put for the WAN bandwidth values in the limiter, as my speeds can sometimes vary during the day.

Also, I am running pfBlocker developer version. Should the fq_codel floating rule be above or below all the pfb rules?

xciter327

@occamsrazor said in Playing with fq_codel in 2.4:

Not sure what changed but now my setup seems to be working OK...

https://www.dslreports.com/speedtest/39073625

This while doing speedtest:
Shell Output - ipfw sched show

00001:  50.000 Mbit/s    0 ms burst 0
q65537  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0      653   974276  0    0   0
00002:  40.000 Mbit/s    0 ms burst 0
q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2
  0 ip           0.0.0.0/0             0.0.0.0/0       59     3149  0    0   0
Is fq_codel adaptive to bandwidth in some way? I'm wondering what to put for the WAN bandwidth values in the limiter, as my speeds can sometimes vary during the day.

Also, I am running pfBlocker developer version. Should the fq_codel floating rule be above or below all the pfb rules?

The pfb rules are block/reject so it should not matter. This should only apply on pass/match rules depending on what You are trying to achieve.

ps. - I don't thin fq_codel is adaptive bandwidth. Normally I set it to 90% link speed a peak hours(like 8-9PM) in the evening.

KOM

@occamsrazor Conventional wisdom says to measure your WAN link speed several times throughout the week, at various times of the day, and then use 90-95% of your LOWEST value.

occamsrazor

@kom Thanks. I was doing some testing last night with the fq_codel download bandwidth set to 50Mbit, and when I was running a popular torrent I hit a sustained download rate of over 100Mbit. So it would seem to me that this "limit" isn't actually a hard limit. Can that be right? Or does fq_codel only affect TCP traffic (my torrent client is set to use uTP)

KOM

@occamsrazor I have no idea. I'm fairly dumb when it comes to QoS, and know just enough to be dangerous. I haven't played with codel at all.

dtaht

you should not be able to achieve a higher rate than htb (hfsc?) specifies. There are a lot of ways to configure htb wrong (like not classifying all packets correctly), there have also been bugs in some versions.

Your initial result (2mbits!?) implied to me you'd got the kbit or mbit setting wrong.

The oddity of your current result implies you are putting utp traffic into another bin entirely. try to build the simplest possible thing that feeds all packets into a single fq_codel instance. Post your code.

The fq part of fq codel works on all packets, the codel part works well on anything that is tcp-friendly, which certainly includes utp and quic.

I don't know enough about bsd to help, we found trying to configure this stuff in the linux sqm-scripts so painful that we ended up replacing it all with the one-liner that is sch_cake.

occamsrazor

@dtaht said in Playing with fq_codel in 2.4:

you should not be able to achieve a higher rate than htb (hfsc?) specifies. There are a lot of ways to configure htb wrong (like not classifying all packets correctly), there have also been bugs in some versions.

What is HTB? And my understanding is with fq_codel we are not using HFSC? I followed the video so I have the WANDown and WANup limiters under the limiters section, but when I click on "By Interface" > "WAN" I don't have anything enabled in that section, and nothing in the "By Queue" section....... should I?

Post your code.

I'm doing it all through the GUI. Is there a command to show the configuration easily?

@kom said in Playing with fq_codel in 2.4:

@occamsrazor ............know just enough to be dangerous.

Same here :-)

dtaht

What you folk are calling a limiter, I call a shaper. The two most common algorithms are token bucket and hfsc. google 'em. I have no idea which or an alternative are being used in the "limiter" concept here (?), nor do I have any idea how this gui is configured, I'm hoping others that are more familiar with bsd and pfsense can chime in. I'm just the co-founder of the bufferbloat project and care deeply seeing about seeing rfc8290 get deployed properly everywhere.

I do think you need to enable things on the wan and queue sections but all I can point to at the moment is how to set up "sqm" properly on things like openwrt. Perhaps one of those tutorials would help in creating an analogy for this OS?

https://openwrt.org/docs/guide-user/network/traffic-shaping/sqm

gsmornot

I'm trying this with a gigabit service and so far no matter what instructions I follow codelq performs better for me. With codelq I see an A each time on the dslreports test. With fq_codel I see a C.

00001: 900.000 Mbit/s 0 ms burst 0
q65537 50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
sched 1 type FQ_CODEL flags 0x0 0 buckets 0 active
FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
Children flowsets: 1
00002: 940.000 Mbit/s 0 ms burst 0
q00002 50 sl. 0 flows (1 buckets) sched 2 weight 1 lmax 0 pri 0 AQM CoDel target 5ms interval 100ms ECN
sched 2 type FQ_CODEL flags 0x0 0 buckets 0 active
FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
Children flowsets: 2

Hugovsky

Remember to flush firewall states between each test.

EDIT: what is your hardware and what packages do you use?

gsmornot

Hardware is SG-3100 and packages are Avahi, nut, openvpn client export, and pfblockerng. So, not a lot. Overall the network is great, the only time I see the score is testing. I'm just saying my score is better with codelq than fq-codel after following the video guide. I reset states between testing.

dtaht

@gsmornot your post was fq_codel both times... Does your bandwidth change vs a vs codelq? If you are running out of cpu...

sensemann

Hello!

I want to priorize two things:

video-livestream (one server in my DMZ)
VoIP
So thats guaranteed that these to applications always have enough bandwidth.
-> If the videostream need all, give them all (expect some 100kbit for the telephones).

How can I reach this goal with fq_codel in 2.4.4?

I already did the setup which was shown in the youtube-video/hangout with the effect, that the download is now around 175Mbit and the upload only 170KBit! (my cable line is around 220 down, 20Mbit up)

00001: 215.000 Mbit/s    0 ms burst 0
q65537  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
 sched 1 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1
00002:  20.000 Mbit/s    0 ms burst 0
q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
 sched 2 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2

Thanks for help in advance!
sensemann

gsmornot

@dtaht said in Playing with fq_codel in 2.4:

@gsmornot your post was fq_codel both times... Does your bandwidth change vs a vs codelq? If you are running out of cpu...

Not sure I follow the reply. In each case for testing I use pretty much all of the CPU. The SG3100 is a fanless ARM processor. Enough for line rate gigabit but maybe pushing it for fq_codel but that is just a guess. CodelQ, setup via shaper (vs limiter for fq_codel) seems to perform just a bit better for my setup. Do you think its CPU related?

adx442

So, I was using the System Patches package to apply the fq_codel GUI elements to 2.4.3.

When I move to 2.4.4, I assume if I don't apply that patch, everything will remain as it is now? Just checking before I upgrade.

goodthings

Hello!
My connection without any limiters is 105/16, overall A, bufferbloat B, quality A.
When I activate fq_codel as per https://youtu.be/o8nL81DzTlU?t=518 with 100/15, the upload bufferbloat improves to only ~22ms but download bufferbloat skyrockets (1000ms avg, 3000ms max) in dslreports. Overall C, bufferbloat F, quality A+.

I tried a lot of things but the result is the same, even when I drop the bandwidths to 75% of actual, download buffer bloat always ends up way worse than without any limiters. Tried a lot of things, ECN on/off, firewall rules floating or LAN, etc, all the same. I did exactly what the youtube video suggested, why would I get such a result?

q65539  50 sl. 0 flows (1 buckets) sched 3 weight 0 lmax 0 pri 0 droptail
 sched 3 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 5120 flows 1024 ECN
   Children flowsets: 1
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0     63870 95546322 76 114000 363
00004:  15.000 Mbit/s    0 ms burst 0
q65540  50 sl. 0 flows (1 buckets) sched 4 weight 0 lmax 0 pri 0 droptail
 sched 4 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 NoECN
   Children flowsets: 2
  0 ip           0.0.0.0/0             0.0.0.0/0      263    14216  0    0   0

any ideas/suggestions would be welcome.

@goodthings What hardware do you have? It might be that fq_codel being active is causing your CPU to spike too hard.

If you've already got great bufferbloat figures you can rest happy anyway I'd think.

@sensemann FQ_CODEL doesn't give you knobs to tweak flows, if you want that, you want to use ALTQ.
The idea of FQ_CODEL is that it doesn't let any single flow starve out the others, so your live videostream and your VoIP should just keep working even if a user triggers off a large torrent.

Given the terrible upload speed, maybe you have a CPU limitation, what hardware are you doing this on?

goodthings

@muppet My hardware is a Netgate SG-2220. There are no other packages running, so the CPU (in top via ssh) seems to peak at 25% load at 100 Mbit.

I experimented some more and find that fq_codel on upload and no queue on download seems to give me the best results. As soon as I enable fq_codel for downloads the bufferbloat jumps from 100ms to 2000ms under load and 15ms to 750ms idle according to dslreports. This is even with 30 Mbit set in the limiter. Similar results with PIE. Curious what could cause this. Tried from different clients, same results.