Playing with fq_codel in 2.4

Nullity

Thanks for this post. :)

PS - "Downloads not randomly halved versus codel." shouldn't be happening.

qubit

ipfw -a list

That shows your rules and you can see what is matching to validate you have your rules correctly.


[2.4.0-BETA][admin@pfsense.lan]/root: ipfw -a list
ipfw: retrieving config failed: Protocol not available

dummynet is used with pf via "dnqueue" in pf rules which shows up in firewall rules via pfctl with limiters enabled.

@Nullity:

Thanks for this post. :)

PS - "Downloads not randomly halved versus codel." shouldn't be happening.

Again probably related to my hardware. speedtests would show the full limited speeds on altq but most downloads wouldn't even reach that. Oddities like: dslreports would max out but fast.com would top out to about 40 megabits, as well as downloads via multiple browsers. Works fine without altq now. I think it was related to the igb driver as on my 2440 all networking would sometimes die and require a reboot if I disable the altq codel. Probably fixed recently by https://github.com/pfsense/FreeBSD-src/commit/42a5f2897e93d1e42833eac551c64c1373119ff9 but I haven't touched it in a while as this setup has been working great.

w0w

I got all three A+ on dslreports, even having active porn downloading on qbittorent. 8)

w0w

I found that traffic equalization (share bandwidth evenly) works differently with FQ_CODEL. If I start speedtest without FQ_CODEL (pipe settings remain the same, only 'sched x config pipe x type fq_codel' line removed) on two LAN PCs, then I see full equalization, for 300Mbps link I get 150 on both PCs. If I activate FQ_CODEL it gives different result. I see some fluctuations, but the first PC started download always wins with at least 60% of accumulated bandwidth.

qubit

@w0w:

I found that traffic equalization (share bandwidth evenly) works differently with FQ_CODEL. If I start speedtest without FQ_CODEL (pipe settings remain the same, only 'sched x config pipe x type fq_codel' line removed) on two LAN PCs, then I see full equalization, for 300Mbps link I get 150 on both PCs. If I activate FQ_CODEL it gives different result. I see some fluctuations, but the first PC started download always wins with at least 60% of accumulated bandwidth.

These are the default sysctls which may need tweaking depending on traffic and bandwidth

net.inet.ip.dummynet.fqcodel.limit: 10240
net.inet.ip.dummynet.fqcodel.flows: 1024
net.inet.ip.dummynet.fqcodel.quantum: 1514
net.inet.ip.dummynet.fqcodel.interval: 100000
net.inet.ip.dummynet.fqcodel.target: 5000

Technical details can be found here: http://caia.swin.edu.au/freebsd/aqm/papers.html

So far I found the default ok

w0w

I've played a bit, but I think that default are really OK.
Now I am using only IPFW FQ_CODEL shaper and disabled ALTQ, this gives me about +4Mbps on 300Mbps bandwidth if I compare with ALTQ shaper tested maximum.
So far, so good.

shinzo

Thanks alot. i have been looking for something like this for a while. I plan to use it for a while to see how things go. ;D

chrcoluk

interesting i may try this out at some point thanks for sharing the information.

shinzo

I already had the limiters setup. i Was looking on how to make it easier and i came up with this. Instead of modifying anything i just input the ipfw command to enable fq_codel and worked.

Limiters:
00001: 30.000 Mbit/s 0 ms burst 0
q131075 50 sl. 0 flows (1 buckets) sched 65539 weight 0 lmax 0 pri 0 droptail
sched 65539 type FIFO flags 0x0 0 buckets 0 active
00002: 6.00 Mbit/s 0 ms burst 0
q131076 50 sl. 0 flows (1 buckets) sched 65540 weight 0 lmax 0 pri 0 droptail
sched 65540 type FIFO flags 0x0 0 buckets 0 active

Queues:
q00001 50 sl. 0 flows (256 buckets) sched 3 weight 1 lmax 0 pri 0 droptail
mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
q00002 50 sl. 0 flows (256 buckets) sched 4 weight 1 lmax 0 pri 0 droptail
mask: 0x00 0xffffffff/0x0000 -> 0x00000000/0x0000

ipfw sched 1 config pipe 1 type fq_codel
ipfw sched 2 config pipe 2 type fq_codel
ipfw sched show

And Done. While it wont survive a reboot i am sure i can set something up.

w0w

Survivng reboot and update also is that what I needed.
Here is my patch (use System_patches package)


--- shaper.inc	Mon Feb 20 18:14:04 2017
+++ shaper.inc	Sun Mar 05 07:33:23 2017
@@ -4596,7 +4596,8 @@
 				"net.inet.ip.dummynet.pipe_slot_limit" => $max_qlimit
 		));
 		file_put_contents("{$g['tmp_path']}/rules.limiter", $dn_rules);
-		mwexec("/sbin/ipfw {$g['tmp_path']}/rules.limiter");
+		#mwexec("/sbin/ipfw {$g['tmp_path']}/rules.limiter");
+		mwexec("/sbin/ipfw /root/rules.limiter");
 	}
 }

Also, remember, you need to reboot firewall manually after update is completed or disable/enable rule where you have limiters used, like in OP first post.

codelFQ.jpg_thumb

Nullity

Regarding the sysctl defaults, this link is likely the most official source for details, particularly the "Parameters" section: https://tools.ietf.org/html/draft-ietf-aqm-fq-codel-06

w0w

One Sunday morning I have found that bufferbloat rating is B or even C and no drops on my side. I've tried to play with bandwidth limiting and after changing it to twice smaller I got A rating again, looks like it's a problem on the ISP side. OK, I was thinking there is nothing to do, but why not to try to use delay instead of limiting bandwidth.
SO I changed limiter config to
pipe 1 config delay 0ms for both pipes
And looks like this did the trick, now I have A+ bufferbloat and A or A+ Quality ratings.
Certainly, I need to do advanced tests before draw some conclusions, but it looks hopefully.

obrienmd

Can't WAIT for this to get into the UI.

FQ_codel's fair queuing is incredible, and HFSC + CODEL, FAIRQ + CODEL and CODELQ in pfSense can't provide multi-bucket fair queuing nearly as well.

I tested this using shellcmd so it will persist through reboots: "ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codel" runs on reboot, with limiters and firewall pipes configured in the UI. It performs just as good as Linux's fq_codel that I have running on LEDE, IPFire and a few other boxes. pfSense getting fq_codel and wireguard would let me move entirely to pfSense / BSD on the networking side :)

w0w

As for GUI I was thinking about building some package, but I am not any kind of php programmer and the best would be mainstream implementation into pfsense by professionals, core team.
We can also vote for bounty and see what happens.

moscato359

It's literally an on/off setting, and a kernel module

w0w

@moscato359:

It's literally an on/off setting, and a kernel module

Not so simple. You need to enable limiters at least and use it in pf rule. So it's a lot of GUI and code change if we going to make it on the traffic shaper side. If we going to make it on the limiters side, then yes it's much more simpler, we need scheduler type selection and bandwidth OR delay limiting. Since I use delay limiting for pipe, it's not enough to use only bandwidth limit.
BTW delay limiting with 0ms gives me the best result with bufferbloat test, since enabled, I have tested it multiple times per day and it's always A/A+ regarding to ISP mainstream router load.
The best thing that comes with delay setting is that you don't limit your traffic when it's really don't need to be limited. For example my real bandwidth varies from 250 to 300Mbit and sometimes to make it work without bufferbloat I need to limit bandwidth down to 100. I am not sure why delay limiting helps in this case but it really works at least with my ISP and I have no bandwidth limit on my side.

Nullity

@w0w:

@moscato359:

It's literally an on/off setting, and a kernel module

Not so simple. You need to enable limiters at least and use it in pf rule. So it's a lot of GUI and code change if we going to make it on the traffic shaper side. If we going to make it on the limiters side, then yes it's much more simpler, we need scheduler type selection and bandwidth OR delay limiting. Since I use delay limiting for pipe, it's not enough to use only bandwidth limit.
BTW delay limiting with 0ms gives me the best result with bufferbloat test, since enabled, I have tested it multiple times per day and it's always A/A+ regarding to ISP mainstream router load.
The best thing that comes with delay setting is that you don't limit your traffic when it's really don't need to be limited. For example my real bandwidth varies from 250 to 300Mbit and sometimes to make it work without bufferbloat I need to limit bandwidth down to 100. I am not sure why delay limiting helps in this case but it really works at least with my ISP and I have no bandwidth limit on my side.

Thanks for trying to explain it. When it comes to traffic-shaping, even from a user perspective (disregarding the developer implementation), rarely is anything as simple as "It's literally an on/off setting, and a kernel module".

I've been guilty of back-seat driving myself… and I'm totally, fully, absolutely awesome. ::)

moscato359

Why wouldn't it be a check box next to where we already have codel, random, random in and out, and explicit congestion notification

All of those things are already implemented.

It's just a different control algorithm tied in at the same place

Nullity

@moscato359:

Why wouldn't it be a check box next to where we already have codel, random, random in and out, and explicit congestion notification

All of those things are already implemented.

It's just a different control algorithm tied in at the same place

One big reason is because the area you're referring to is in the queues (ALTQ) section while fq_codel was implemented in limiters (dummynet) section.

Why don't we "just" send humans to Mars? We already have robots there.

Like I said, back-seat driving is easy.

moscato359

@Nullity:

@moscato359:

Why wouldn't it be a check box next to where we already have codel, random, random in and out, and explicit congestion notification

All of those things are already implemented.

It's just a different control algorithm tied in at the same place

One big reason is because the area you're referring to is in the queues (ALTQ) section while fq_codel was implemented in limiters (dummynet) section.

Why don't we "just" send humans to Mars? We already have robots there.

Like I said, back-seat driving is easy.

Why is it under limiter, when the rest of them are under altq?