Playing with fq_codel in 2.4

sensemann

Hello!

I want to priorize two things:

video-livestream (one server in my DMZ)
VoIP
So thats guaranteed that these to applications always have enough bandwidth.
-> If the videostream need all, give them all (expect some 100kbit for the telephones).

How can I reach this goal with fq_codel in 2.4.4?

I already did the setup which was shown in the youtube-video/hangout with the effect, that the download is now around 175Mbit and the upload only 170KBit! (my cable line is around 220 down, 20Mbit up)

00001: 215.000 Mbit/s    0 ms burst 0
q65537  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
 sched 1 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1
00002:  20.000 Mbit/s    0 ms burst 0
q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
 sched 2 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2

Thanks for help in advance!
sensemann

gsmornot

@dtaht said in Playing with fq_codel in 2.4:

@gsmornot your post was fq_codel both times... Does your bandwidth change vs a vs codelq? If you are running out of cpu...

Not sure I follow the reply. In each case for testing I use pretty much all of the CPU. The SG3100 is a fanless ARM processor. Enough for line rate gigabit but maybe pushing it for fq_codel but that is just a guess. CodelQ, setup via shaper (vs limiter for fq_codel) seems to perform just a bit better for my setup. Do you think its CPU related?

adx442

So, I was using the System Patches package to apply the fq_codel GUI elements to 2.4.3.

When I move to 2.4.4, I assume if I don't apply that patch, everything will remain as it is now? Just checking before I upgrade.

goodthings

Hello!
My connection without any limiters is 105/16, overall A, bufferbloat B, quality A.
When I activate fq_codel as per https://youtu.be/o8nL81DzTlU?t=518 with 100/15, the upload bufferbloat improves to only ~22ms but download bufferbloat skyrockets (1000ms avg, 3000ms max) in dslreports. Overall C, bufferbloat F, quality A+.

I tried a lot of things but the result is the same, even when I drop the bandwidths to 75% of actual, download buffer bloat always ends up way worse than without any limiters. Tried a lot of things, ECN on/off, firewall rules floating or LAN, etc, all the same. I did exactly what the youtube video suggested, why would I get such a result?

q65539  50 sl. 0 flows (1 buckets) sched 3 weight 0 lmax 0 pri 0 droptail
 sched 3 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 5120 flows 1024 ECN
   Children flowsets: 1
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0     63870 95546322 76 114000 363
00004:  15.000 Mbit/s    0 ms burst 0
q65540  50 sl. 0 flows (1 buckets) sched 4 weight 0 lmax 0 pri 0 droptail
 sched 4 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 NoECN
   Children flowsets: 2
  0 ip           0.0.0.0/0             0.0.0.0/0      263    14216  0    0   0

any ideas/suggestions would be welcome.

@goodthings What hardware do you have? It might be that fq_codel being active is causing your CPU to spike too hard.

If you've already got great bufferbloat figures you can rest happy anyway I'd think.

@sensemann FQ_CODEL doesn't give you knobs to tweak flows, if you want that, you want to use ALTQ.
The idea of FQ_CODEL is that it doesn't let any single flow starve out the others, so your live videostream and your VoIP should just keep working even if a user triggers off a large torrent.

Given the terrible upload speed, maybe you have a CPU limitation, what hardware are you doing this on?

goodthings

@muppet My hardware is a Netgate SG-2220. There are no other packages running, so the CPU (in top via ssh) seems to peak at 25% load at 100 Mbit.

I experimented some more and find that fq_codel on upload and no queue on download seems to give me the best results. As soon as I enable fq_codel for downloads the bufferbloat jumps from 100ms to 2000ms under load and 15ms to 750ms idle according to dslreports. This is even with 30 Mbit set in the limiter. Similar results with PIE. Curious what could cause this. Tried from different clients, same results.

@goodthings It's a good question - I'm unsure how much testing FQ_CODEL has had on the ARM chipsets, maybe it's just not compatible?

EDIT: This is bollocks, the SG-2200 is not ARM based.

w0w

@muppet

What ARM chipset are you talking about?

@w0w Sorry, I totally farked up. I was conflating the SG-3100 post above which is ARM based, but @goodthings has an Intel Atom.

I think I will stop posting about stuff I don't fully understand, my apologies all.

w0w

@goodthings
Do you have any other shaper enabled simultaneously with limiters?
What packages are you using on firewall?
Post result of "ipfw sched show" and "pfctl -sr | grep dnqueue" commands and full result of dslreports test when you have those huge jumps, also

bafonso

I have a 120/20 connection and I just tried the FQ_CODEL approach but I get a strange result. When I use 20Mbit for upload limiter I get 2mbit effective upload on speedtest (ECN or not)... if I use 40Mbit, then I get 4Mbit. I have found if I simply use CODELQ I have better buffer bloat scores on dsl reports and actually 20mbit upload.

goodthings

This post is deleted!

goodthings

I collected all the info but the forum software / akismet detects my postings as spam when i try to attach a dslreport. This all seems more trouble than it's worth so I'm giving up for now, thank you for trying to help.

occamsrazor

A few newbie questions..... What is the best way to monitor what is going on with the fq_codel limiter: "ipfw sched show" from the command prompt, or Diagnostics > Limiter Info.... or it's the same? The latter seems to provide an additional line of info:

Queues:
q00001  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0  AQM CoDel target 5ms interval 100ms ECN
q00002  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0  AQM CoDel target 5ms interval 100ms ECN

though I guess it is same information.

Can anyone give some information (or point me to a resource elsewhere) on how to interpret the output of "ipfw sched show"? Here is a sample I posted above, but I'm confused about what each part means.....

Shell Output - ipfw sched show

00001:  50.000 Mbit/s    0 ms burst 0
q65537  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0        2     2932  0    0   0
00002:  40.000 Mbit/s    0 ms burst 0
q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2
  0 ip           0.0.0.0/0             0.0.0.0/0        2      112  0    0   0

Why 0 buckets 1 active?
Why 0 flows?
Tot_pkt/bytes Pkt/Byte Drp? What do each of these refer to?
If you don't see any Drp (drops?) does that mean fq_codel was not having any effect?
What does target 5ms interval 100ms refer to?

PS - It would be really cool if there were some kind of dashboard widget to show what's going on the traffic shaper/limit in realtime.

Edit: Also I still don't understand the relationship between the bandwidth limit set and what is subsequently possible. Here is a well-seeded torrent running with bandwidth set to 50mbit download 40mbit upload.....

00001:  50.000 Mbit/s    0 ms burst 0
q65537  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0      164   236431 13 19396   0
00002:  40.000 Mbit/s    0 ms burst 0
q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2
  0 ip           0.0.0.0/0             0.0.0.0/0       43     2132  0    0   0

....and yet download bandwidth was able to far exceed the limit set.

w0w

@goodthings said in Playing with fq_codel in 2.4:

I collected all the info but the forum software / akismet detects my postings as spam when i try to attach a dslreport. This all seems more trouble than it's worth so I'm giving up for now, thank you for trying to help.

I am not sure what are you talking about, I wanted just the link like that http://www.dslreports.com/speedtest/39412173

w0w

@bafonso
Can you post you configuration, screenshots or what ever you like?
You are not the first person who reported this misbehavior, but I am not sure what is the reason for it.

xciter327

Is everybody applying their rules "WAN" side? I was a bit surprised that the hangout adds them "WAN" side. I normally either apply them on the "LAN" side or via a floating rule on multiple "LAN" side interfaces.

TheNarc

Almost all of my rules are applied LAN-side, because almost all flows are initiated by hosts on the LAN. But if you run a WAN-facing server (e.g. SSH) for which traffic that initiates a flow will come in on the WAN interface, you'd want to assign it to a limiter queue there. At least, that's my understanding of how it works.

xciter327

Would this also be handy, for example, if one would want to assign different limiters to each segment. For example:

WAN - 480/480
LAN1 - 90/90
LAN2 - 200/200
etc.

What would happen if the product of the bandwidths of the LANs is more than the WAN? I would assume that fq_codel should do its magic.

Another handy scenario I can think of:
WAN - 300/300
LAN1 - 10/10 per host/IP
LAN2 - 100/100 per host/IP
etc.