CoDel - How to use

Nullity

I have turned to the dark-side and chose full bandwidth over best latency on my incoming traffic. The average latency increases to ~70ms (~10ms idle) during a link saturating download, but even with artificial throughput limiting I was reaching ~40ms. Addiction to throughput probably stems from years of dial-up… the experience still haunts me. :)

The difference on uploads is huge though; ~600ms vs ~50ms.

My ISP is the not-so-great-but-improving Windstream. I have wondered what other people experience on other ISPs and CPE.

Harvy66

Since I use HFSC even for download traffic shaping, I can make sure Netflix/Youtube buffering is kept to a minimum. I pretty much can't tell the difference, even when using P2P and saturating the downstream.

I have 100Mb with a 2:1 ratio between Normal:Low. P2P will be sitting around 97Mb, then I start of Netflix and I pretty much immediately see P2P drop down to ~32Mb/s and Netflix pushes through ~64Mb/s. Even with DSLReports, I still see an A/A+ during the download test. But if P2P is using 0 and DSLReports is using max, and suddenly P2P jumps up to 33, the bufferbloat can get bad, like C/D, but that's because available bandwidth is decreasing instead of increasing and I'm still stuck with regular CoDel and not fq_CoDel.

Nullity

CoDel's default "target/interval" values should be fixed in 2.2.3. Dave mentioned earlier that the "target" is pretty good at dynamically adjusting, so the fix probably has little effect in that area.

The "interval" is more vital and is noted in the IETF draft as being the only parameter that is required for CoDel to to function, so perhaps changing from a value of 5ms to a more optimal value of 100ms will improve our CoDel experience. :)

Disappointingly, I tried loading up a CODELQ queue in 2.2.3, and I still got the old values… Either the install needs to be fresh or my patch is shit and needs more work. See if it is fixed on any of your 2.2.3 setups.

Harvy66

I wonder how it'll affect my bufferbloat. Right now I have about 1ms, but it can burst quite high given enough TCP streams growing at the same time.

kieranc

@Nullity:

CoDel's default "target/interval" values should be fixed in 2.2.3. Dave mentioned earlier that the "target" is pretty good at dynamically adjusting, so the fix probably has little effect in that area.

The "interval" is more vital and is noted in the IETF draft as being the only parameter that is required for CoDel to to function, so perhaps changing from a value of 5ms to a more optimal value of 100ms will improve our CoDel experience. :)

Disappointingly, I tried loading up a CODELQ queue in 2.2.3, and I still got the old values… Either the install needs to be fresh or my patch is shit and needs more work. See if it is fixed on any of your 2.2.3 setups.

I did a fresh install of 2.2.3 and the target/interval are still inverted.

[2.2.3-RELEASE][admin@pfSense.localdomain]/root: pfctl -vs queue | grep -i codel
altq on em0 codel( target 50 interval 5) bandwidth 600Kb tbrsize 1500 
altq on em1 codel( target 50 interval 5) bandwidth 6Mb tbrsize 6000

Nullity

@kieranc:

@Nullity:

CoDel's default "target/interval" values should be fixed in 2.2.3. Dave mentioned earlier that the "target" is pretty good at dynamically adjusting, so the fix probably has little effect in that area.

The "interval" is more vital and is noted in the IETF draft as being the only parameter that is required for CoDel to to function, so perhaps changing from a value of 5ms to a more optimal value of 100ms will improve our CoDel experience. :)

Disappointingly, I tried loading up a CODELQ queue in 2.2.3, and I still got the old values… Either the install needs to be fresh or my patch is shit and needs more work. See if it is fixed on any of your 2.2.3 setups.

I did a fresh install of 2.2.3 and the target/interval are still inverted.
[2.2.3-RELEASE][admin@pfSense.localdomain]/root: pfctl -vs queue | grep -i codel
altq on em0 codel( target 50 interval 5) bandwidth 600Kb tbrsize 1500 
altq on em1 codel( target 50 interval 5) bandwidth 6Mb tbrsize 6000 

Fudge…

Thank you for letting me know. I will get them to revert my commits.

How can a little fix be so illusive?

Why must you be so confusing, pfSense-tools!?

kieranc

@Nullity:

@kieranc:
@Nullity:

CoDel's default "target/interval" values should be fixed in 2.2.3. Dave mentioned earlier that the "target" is pretty good at dynamically adjusting, so the fix probably has little effect in that area.

The "interval" is more vital and is noted in the IETF draft as being the only parameter that is required for CoDel to to function, so perhaps changing from a value of 5ms to a more optimal value of 100ms will improve our CoDel experience. :)

Disappointingly, I tried loading up a CODELQ queue in 2.2.3, and I still got the old values… Either the install needs to be fresh or my patch is shit and needs more work. See if it is fixed on any of your 2.2.3 setups.

I did a fresh install of 2.2.3 and the target/interval are still inverted.
[2.2.3-RELEASE][admin@pfSense.localdomain]/root: pfctl -vs queue | grep -i codel
altq on em0 codel( target 50 interval 5) bandwidth 600Kb tbrsize 1500 
altq on em1 codel( target 50 interval 5) bandwidth 6Mb tbrsize 6000 
Fudge…

Thank you for letting me know. I will get them to revert my commits.

How can a little fix be so illusive?

Why must you be so confusing, pfSense-tools!?

It looks like you're doing it right but the patch hasn't been merged. I've created a pull request on github to do the same thing, let's see what happens

Do you know why the current codel_alloc(100, 5, 0) results in values of target 50 interval 5? target 5 interval 50 would be better but I dunno where the value of 50 is coming from.

doktornotor

Was never fixed AFAICT? https://redmine.pfsense.org/issues/4692

kieranc

@doktornotor:

Was never fixed AFAICT? https://redmine.pfsense.org/issues/4692

Indeed, has anyone tried the patch? I don't really want to have to build pfsense in order to test it, that's a lot of repos to clone…

edit: PR has been merged, new values should be applied in 2.2.4 or anything built from here on...

Harvy66

I wish this could be set via config instead of compile time. One problem at a time though, ehh?

50/5 seems to be working well for me right now. I guess I'll see how my bufferbloat is affected once this change finally makes it. I'm getting 0ms of bufferbloat and full throughput already. According to DSLReports, my bloat can spike, but rarely. More of an issue when doing 32 upload streams.

Nullity

@doktornotor:

Was never fixed AFAICT? https://redmine.pfsense.org/issues/4692

https://github.com/pfsense/pfsense-tools/commit/3108a902bd816036a3abffd3ec669767140891a7

I dunno. I am unsure of many things. :(

I probably should have updated the redmine submission. The redmine patch was a initial code to show what I had found, hooefulky to help a dev pinpoint the problem.

The github patches were the best I could do, but I should probably stop trying to patch pfSense considering that I cannot build pfSense to test my code. :(

kieranc

So this is from the latest nightly:

[2.2.4-DEVELOPMENT][admin@pfSense.localdomain]/root: pfctl -vs queue
altq on em0 codel( target 50 interval 100) bandwidth 600Kb tbrsize 1500

Interval successfully changed, now we just have to figure out where the target of 50 is coming from….

Edit: I just set the 'queue limit' to 25 in the GUI and my target is now 25.... Victory?

Edit2: From 2.2.4 19/07/2015 nightly, with queue limit set to 5:

[2.2.4-DEVELOPMENT][admin@pfSense.localdomain]/root: pfctl -vs queue
altq on em1 codel( target 5 interval 100) bandwidth 6Mb tbrsize 6000 
  [ pkts:         85  bytes:       9938  dropped pkts:      0 bytes:      0 ]
  [ qlength:   0/ 50 ]

So it wasn't anything I did yesterday that fixed it, but it does seem to be fixed/workable in 2.2.4

Nullity

If qlimit is 0, it defaults to 50, and codel gets the (initial?) target value from qlimit.

kieranc

@Nullity:

If qlimit is 0, it defaults to 50, and codel gets the (initial?) target value from qlimit.

is qlimit the queue length, or something else entirely?

Nullity

@kieranc:

@Nullity:

If qlimit is 0, it defaults to 50, and codel gets the (initial?) target value from qlimit.

is qlimit the queue length, or something else entirely?

qlimit is the queue length which becomes useless when codel is axtive, since codel dynamically controls queue length (AQM).

kieranc

@Nullity:

@kieranc:

@Nullity:

If qlimit is 0, it defaults to 50, and codel gets the (initial?) target value from qlimit.

is qlimit the queue length, or something else entirely?

qlimit is the queue length which becomes useless when codel is axtive, since codel dynamically controls queue length (AQM).

So when using codel the 'queue limit' setting seems to change the target instead… handy, but not very obvious..
Thanks!

Nullity

Yeah, it is pretty confusing but I'll take CoDel however I can get it. :)
Ermal ported himself, iirc. Ahead of the curve, that guy! :)

I still dunno how to view or set codel's parameters when it is a sub-discipline though. Default or gtfo, I suppose…

Harvy66

The whole target qlimit thing applies to CoDel for both the scheulder and the child discipline?

Do you know if the interval changes? The interval is supposed to be 20x the target.

kieranc

@Nullity:

Yeah, it is pretty confusing but I'll take CoDel however I can get it. :)
Ermal ported himself, iirc. Ahead of the curve, that guy! :)

I still dunno how to view or set codel's parameters when it is a sub-discipline though. Default or gtfo, I suppose…

I've just had a tinker and I can't find anything, but that certainly doesn't mean it's not there.
I've rarely used BSD, is there some /proc type interface where the information comes from that can be queried directly?

Nullity

@Harvy66:

The whole target qlimit thing applies to CoDel for both the scheulder and the child discipline?

Do you know if the interval changes? The interval is supposed to be 20x the target.

iirc, the sub-discipline setup is purely configured by hard-coded defaults and has no user configurable/viewable params that I am aware of. Hopefully, there is a simple way for a user to view/set the params in that situation. ermal? ;)

interval is the only value required by codel, so I do not think it changes. Technically, the target should be set based on the interval value, not vice versa.
afaik, current codel implementations do not automagically set interval to live RTT.

The CoDel building blocks are able to adapt to different or time-
varying link rates, to be easily used with multiple queues, to have
excellent utilization with low delay and to have a simple and
efficient implementation. The only setting CoDel requires is its
interval value, and as 100ms satisfies that definition for normal
internet usage, CoDel can be parameter-free for consumer use.

See: https://tools.ietf.org/id/draft-nichols-tsvwg-codel-02.txt

I have tried to run a thought-experiment concerning how a 5ms interval should negatively affect codel's performance, but I cannot fully comprehend it. I need to setup a bufferbloat lab…