Playing with fq_codel in 2.4
-
@zwck Yes, those settings look fine for a "set it and forget it" fq_codel implementation. How are your bufferbloat scores on dslreports.com/speedtest using the settings in those screenshots??
-
@pfsvrb
Its A all around, however i noticed that when checking ipfw sched show periodically during both phases (up and down) there are quite some dropped packages, maybe my NICs are not that great, they are intel but not the recommended 350s00001: 900.000 Mbit/s 0 ms burst 0 q65537 50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 NoECN Children flowsets: 1 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 ip 0.0.0.0/0 0.0.0.0/0 295104 439672423 275 410213 492 00002: 900.000 Mbit/s 0 ms burst 0 q65538 50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 NoECN Children flowsets: 2 0 ip 0.0.0.0/0 0.0.0.0/0 2405 101107 0 0 000001: 900.000 Mbit/s 0 ms burst 0 q65537 50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 NoECN Children flowsets: 1 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 ip 0.0.0.0/0 0.0.0.0/0 745 31427 0 0 0 00002: 900.000 Mbit/s 0 ms burst 0 q65538 50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 NoECN Children flowsets: 2 0 ip 0.0.0.0/0 0.0.0.0/0 614966 917451569 207 309100 616i am quite sure there are some errors in my config, i exchanged to the 350 intel nics and bufferbloat is back to F
-
@mattund would you mind checking the post above , because i can get a working fq_codel setup as well without the in and out queue, and just selecting download / upload in the lan rules. So i am kindof confused what a minimal fq_codel set up should look like.
-
I feel like that should work; I personally ended up doing floating rules on the WAN interfaces I need to shape (I have a cable/DSL setup). Now, given you have your shaping set to 900/900, maybe you need to increase the slot size/queue length? That's a pretty fast connection. Then again a while ago I think I refuted this and said that queue sizes literally don't matter... the internal behavior of this scheduler as it is on FreeBSD is not well documented, there are some research papers by a particular organization but besides that there's not much else.
Generally seeing drops is actually OK, perhaps even good, because FQ_CoDel should be favoring dropping some traffic over allowing it to sit in queue for a while; in my experience FQ_CoDel is a very "all or nothing" algorithm. This doesn't explain your F's though. I find it odd you're getting that while also seeing activity on the queues. Has to be some sort of parameter issue...
Here is my
ipfw sched show:00001: 128.000 Mbit/s 0 ms burst 0 q65537 50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail sched 1 type FQ_CODEL flags 0x0 0 buckets 0 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 NoECN Children flowsets: 1 00002: 11.128 Mbit/s 0 ms burst 0 q65538 50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail sched 2 type FQ_CODEL flags 0x0 0 buckets 0 active FQ_CODEL target 5ms interval 10ms quantum 1514 limit 10240 flows 1024 NoECN Children flowsets: 2 00003: 25.000 Mbit/s 0 ms burst 0 q65539 50 sl. 0 flows (1 buckets) sched 3 weight 0 lmax 0 pri 0 droptail sched 3 type FQ_CODEL flags 0x0 0 buckets 0 active FQ_CODEL target 10ms interval 100ms quantum 1514 limit 10240 flows 1024 NoECN Children flowsets: 3 00004: 2.000 Mbit/s 0 ms burst 0 q65540 50 sl. 0 flows (1 buckets) sched 4 weight 0 lmax 0 pri 0 droptail sched 4 type FQ_CODEL flags 0x0 0 buckets 0 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 NoECN Children flowsets: 41 - Cable download
2 - Cable upload
3 - DSL download
4 - DSL upload -
@w0w Do we have a redmine number? I can help out; I don't see any PR's yet to fix this
get_single_sysctl("net.inet.ip.dummynet.codel.target")I call this because the script needs to pull in the default values for several of the algorithms and I didn't want to hard-code it.
My first impression is this is returning something funky. Anyone's new VMs result of
sysctl -a | grep net.inet.ip.dummynet.codel.targetshowing as empty or a non-numeric value? I getnet.inet.ip.dummynet.codel.target: 5000Or, they changed or renamed
get_single_sysctlIt's those items or something with my PHP that is screwy in a recent change; I actually don't do PHP very often so this is likely...
Thinking this out, it may be that without the dummynet kernel module loaded (or something like that) there is no sysctl ready for the script to use. Maybe I need to load the kernel module first before I start trying to get these values. I bet you it's not loaded at the very first execution of the script.
If this is the case,
/sbin/kldload dummynetin the command line or in the UI's command executor should fix this. -
Would you mind screenshotting
/ creating a catch one and all floating rule? I am not sure how I am supposed to set this up. -
@zwck https://forum.netgate.com/post/772130
I posted that earlier for reference, I am still using that configuration I believe
-
@mattund perfect I'll give that a try. Thank you so much
-
@mattund
No I did not created any redmine issues. -
@mattund said in Playing with fq_codel in 2.4:
Now, given you have your shaping set to 900/900, maybe you need to increase the slot size/queue length? That's a pretty fast connection.
Hey Matti,
in the mean time i switched to a floating rule setup, and the rule pretty much catches all similarly to what you described. I also played with bucket size and queue length to no avail, when i download or upload most of the time the scheduler uses sub 15 buckets and the default is 256 or so. I doubled the limit as well to 20480 and flows up to 2048 there was no difference. Then i read the fq_codel manual and the standard values put in place to cover basically everything from 1mbit to 1000mbit. Maybe the drop i see is just what i have.
Thanks for your help.
-
I am getting these errors in 2.4.4
kernel config_aqm Unable to configure flowset, flowset busy!
Anyone else seeing these?
-
@mattund said in Playing with fq_codel in 2.4:
@slowgrind said in Playing with fq_codel in 2.4:
So after applying the patch do you just fill in the settings under limiters?
Here's what I'm doing. This might be a little more than what you need, but I figure I would share my configuration in case others have a crazy Multi-WAN multi-LAN setup like I do. I've constructed a series of limiters, one for download and one for upload, each with its own associated queue (you can make the queue with the "+ Add new Queue" button on the bottom of a Limiter's settings page) :
(I have more for my second ISP following that naming scheme: lINTERFACEDownload/lINTERFACEUpload and qINTRERFACEDownload/qINTERFACEUpload children)
I'm assigning FQ_CoDel to the scheduler on the parent limiter and leaving everything else alone. You can either edit the parameters, or leave them at default if you have a typical connection (FQ_CoDel is supposed to be "knobless" after all).
According to the following diagram, this is how the traffic will flow inside dummynet:
(flow_mask|sched_mask) sched_mask +---------+ weight Wx +-------------+ | |->-[flow]-->--| |-+ -->--| QUEUE x | ... | | | | |->-[flow]-->--| SCHEDuler N | | +---------+ | | | ... | +--[LINK N]-->-- +---------+ weight Wy | | +--[LINK N]-->-- | |->-[flow]-->--| | | -->--| QUEUE y | ... | | | | |->-[flow]-->--| | | +---------+ +-------------+ | +-------------+via: https://www.freebsd.org/cgi/man.cgi?query=ipfw&manpath=FreeBSD+9-current&format=html
Dissection: firewall traffic is assigned to a queue, which then generates flows defined by the mask, which pipe into the scheduler (set to FQ_CoDel), which then outputs to the pipe/link at the specified max bitrate.
To assign your traffic to queues, you could do something like I did, which is to use floating rules. I have two WANs, and I need independent shaping and all that, so if you're on a single WAN it may be different for you/you may have better options.
How I set the rules up:
- Interface: WAN A or B interface
- Direction: out
- Address Family: IPv4 or IPv6; I had to do two rules, one for each IP version
- Gateway: Select the applicable IPv4 or IPv6 gateway consistent with how traffic should be routed on that IP stack
- In / Out pipe: qCHARTERUpload / qCHARTERDownload
I have some filtering rules in play here as you can see in my screenshot, but that's only since I'm testing some issues I mentioned previously. It's up to you if you want to match certain protocols/ports, etc.
Anybody know why when i enable floating rules both upload and download speeds get cut in half? As soon as i disable it, speeds are back to normal.
As far as i can see, traffic both download and upload are getting matched by the interface rules anyway, what is the floating rule for?
-
Wrong numbers in the limiters? maybe post some images, so people can help you
-
@mattund said in Playing with fq_codel in 2.4:
@slowgrind said in Playing with fq_codel in 2.4:
So after applying the patch do you just fill in the settings under limiters?
Here's what I'm doing. This might be a little more than what you need, but I figure I would share my configuration in case others have a crazy Multi-WAN multi-LAN setup like I do. I've constructed a series of limiters, one for download and one for upload, each with its own associated queue (you can make the queue with the "+ Add new Queue" button on the bottom of a Limiter's settings page) :
(I have more for my second ISP following that naming scheme: lINTERFACEDownload/lINTERFACEUpload and qINTRERFACEDownload/qINTERFACEUpload children)
I'm assigning FQ_CoDel to the scheduler on the parent limiter and leaving everything else alone. You can either edit the parameters, or leave them at default if you have a typical connection (FQ_CoDel is supposed to be "knobless" after all).
According to the following diagram, this is how the traffic will flow inside dummynet:
(flow_mask|sched_mask) sched_mask +---------+ weight Wx +-------------+ | |->-[flow]-->--| |-+ -->--| QUEUE x | ... | | | | |->-[flow]-->--| SCHEDuler N | | +---------+ | | | ... | +--[LINK N]-->-- +---------+ weight Wy | | +--[LINK N]-->-- | |->-[flow]-->--| | | -->--| QUEUE y | ... | | | | |->-[flow]-->--| | | +---------+ +-------------+ | +-------------+via: https://www.freebsd.org/cgi/man.cgi?query=ipfw&manpath=FreeBSD+9-current&format=html
Dissection: firewall traffic is assigned to a queue, which then generates flows defined by the mask, which pipe into the scheduler (set to FQ_CoDel), which then outputs to the pipe/link at the specified max bitrate.
To assign your traffic to queues, you could do something like I did, which is to use floating rules. I have two WANs, and I need independent shaping and all that, so if you're on a single WAN it may be different for you/you may have better options.
How I set the rules up:
- Interface: WAN A or B interface
- Direction: out
- Address Family: IPv4 or IPv6; I had to do two rules, one for each IP version
- Gateway: Select the applicable IPv4 or IPv6 gateway consistent with how traffic should be routed on that IP stack
- In / Out pipe: qCHARTERUpload / qCHARTERDownload
I have some filtering rules in play here as you can see in my screenshot, but that's only since I'm testing some issues I mentioned previously. It's up to you if you want to match certain protocols/ports, etc.
@mattund I have a question here, when you define your floating rule it states in the discription the following:
"If creating a floating rule, if the direction is In then the same rules apply, if the direction is Out the selections are reversed, Out is for incoming and In is for outgoing." Are you doing this or not ? -
@mattund I have a question here, when you define your floating rule it states in the description the following:
"If creating a floating rule, if the direction is In then the same rules apply, if the direction is Out the selections are reversed, Out is for incoming and In is for outgoing." Are you doing this or not ? -
wow
-
I am delighted to see all the enthusiasm for fq_codel. For theoretical discussions about how it works please visit the cake or codel mailing lists at lists.bufferbloat.net.
I've been meaning to code review the bsd implementation for a while, I know it has a few limitations and differences from the linux version. In reviewing this thread and all its comments just now I have a few comments. Note I'm primarily an emailer, not a web forum person, but I'll try to pay some attention here, now that I know this thread exists, while you sort out teething pains and new bugs.
-
do try the simplest possible config first - one shaper + fq_codel. I've generally found that this eliminates the former need for a lot of rules. sqm-scripts has a few of the common rules (like deprioritizing ping) we use in the openwrt world.
-
does this OS allow for compensating for frame size and spacing (as in dsl/cable/etc). Otherwise if you try to get close, you get bitten by that.
A lot of our motivation for fq_codel and now cake was driven by: https://www.bufferbloat.net/projects/bloat/wiki/Wondershaper_Must_Die/
which references the framing problem.
and if you are going to shape, shape everything, including ping.
-
your nic shouldn't matter
-
I don't know if the bulk-dropper in the linux version of fq_codel is in the bsd version. It helps on extreme overloads.
-
It does sound like there a memory reallocation bug in this version on a reconfigure?
-
anyone up for a sch_cake port? It's SQM on steroids. https://arxiv.org/pdf/1804.07617
I think that's most of my takeaway from this thread. Another plug is that the flent.org tool we created and use a lot to look hard at all sorts of networking problems - you'll find a lot of "rrul" tests in particular and I'd love to see some of those against your various configurations.
Happy debloating!
--
dave taht
co-founder bufferbloat project -
-
I also note that fq_pie was done by the same great folk that did fq_codel, and it benched out pretty good. Even though fq_codel is sort of my adopted baby, I'm pretty agnostic - I just want to beat bufferbloat across the entire internet before I kick the bucket. From what I could see of fq_pie - it looked good also! and I'd really like it if more people also gave it a shot on real world traffic and reported back. thx!
--
dave -
@dtaht et.al thank you!
-
@dtaht is the man!
