Playing with fq_codel in 2.4

dtaht

@gsakes said in Playing with fq_codel in 2.4:

@tman222 In my case, I'd prefer to use PFSense for firewalling and shaping, but my testing showed that Linux/Cake performs better than PfSense/fq_codel, albeit not by much, maybe 10-15% depending on the load.

As far as the architecture is concerned you don't need to run a firewall on the linux host, you can simply configure it as a router; you'd need two network interfaces, where you'd configure cake using the 'layer.cake' script from the cake github repo on the egress interface.

Taking from @mattund 's example, this is how my setup looks like:
            |   pfSense FW          | Router                    |                   |
            |   (no shaping)        | Ubuntu Server             |                   |
            |   (192.168.0.0/24)|   | (10.18.9.0/24)            |                   |
   LAN -->  |                       | eth1 --> (cake) --> eth2  | --> Cable Modem   | --> WAN
            |                       |                           |                   |

While this is a cool idea, you lose a cake feature by not doing nat on it: per host fq. None of our testing here to date has established the coolness of this feature, you need two or more source ip addresses to test from to see it work.

(yep, flent can do this test too. Not gonna describe how here).

Anyway, if you go this route, use "flows" instead of triple-isolate on cake, and nonat, save some cpu. But I thought we were dealing with a nat bug in pfsense in the first place?

dtaht

@gsakes You can even get rid of any need to have the cake box have ips. Just create a bump in the wire:

https://apenwarr.ca/log/?m=201808

tman222

Thanks @dtaht - I have seen that link you shared before. Now I'm curious how that "bump" should be spec'd hardware wise to shape at gigabit speeds. Does anyone have any thoughts on that? Also, does OpenWRT run on a normal linux box? if not, is it possible to duplicate the functionality on just a regular Linux install?

gsakes

@dtaht Thanks for the tips Dave, as always much appreciated:) I'm decommissioning my PFSense box for now. I've been using PFSense since 2010, and I don't think there's anything better, but I'll put it on ice until fq_codel matures and/or Cake is implemented. I'm slowly building out this Ubuntu box to be my firewall, using Firehol/Fireqos, netdata and PiHole.

So yes - I will be doing NAT on the box:)

gsakes

@dtaht said in Playing with fq_codel in 2.4:

triple-isolate

Yep, it's the 'per-host-fq' that is a real big factor, compared to fq_codel - frankly the biggest reason for me switching over:)

BTW - Anyone wanting to learn about fq_codel, Cake and the design of both should read this:

Piece of CAKE: A Comprehensive Queue
Management Solution for Home Gateways

uptownVagrant

So I read this paper Dummynet AQM v0.1 – CoDel and FQ-CoDel for FreeBSD’s ipfw/dummynet framework

The paper is written by the folks that implemented Codel and FQ-CoDel into FreeBSD ipfw/dummynet. I know @dtaht knows this because he reviewed the source and there is correspondence between them and he back in the day. I'm just catching up - thanks for your patience.

Looking at the examples in the paper, I'm wondering why the Codel AQM is selected in the pfSense WebUI in the August 2018 hangout? Per the FQ-CoDel examples in the paper above, it does not seem appropriate and removing Codel as the AQM from the pipe and queue removes the "flowset busy" error @mattund mentioned 4 months ago. @dtaht this is why I was stating codel+fq-codel - when I first learned about FQ-CoDel being added to pfSense 2.4.4, it was in the hangout video which it instructs to choose Codel as the AQM.

Concerning buckets and CPU utlization, I played with net.inet.ip.dummynet.hash_size which is the closest thing I could find to what you were explaining - pfSense defaults to 256 and I doubled the value on each flent rrul test up to 16384. I had to use sysctl -w net.inet.ip.dummynet.hash_size=$value on the fly in the console because /etc/inc/shaper.inc overwrites the setting to 256 any time you make a change to the limiters. I did not find setting this above 256 to provide real value.

So, unfortunately I haven't made much progress...

markn6262

@xraisen
I found using CODEL for QMA & FQ_PIE for Scheduler, along with CODEL for queue QMA does NOT error with “config_aqm Unable to configure flowset, flowset busy!” Hope this is helpful to those reading this thread late.

xRaisen

@markn6262 Thanks for the suggestion. But it doesn't work on my end. codel/tail drop+fq_codel do wonders even it nags “config_aqm Unable to configure flowset, flowset busy!”

strangegopher

Thanks for cake + OpenWRT in bridge mode suggestion. For my 150mbps+ speeds I was suggested ipq806x based or mvebu (cortexa9) device on the irc. Can't wait for it to arrive next week.

dtaht

yes the ipx8xx and a15 gear is good to a couple hundred mbit.

For a gbit the lowest end x86 I recommend is the apu2 or an i3.

Is there any way to push harder on the pfsense nat bug?

dtaht

flent has now been packaged up and made available for freebsd.

https://github.com/tohojo/flent/commit/c928c03a301258c26c7d045c74ecce6dfeaa3d5a

markn6262

@xraisen Your right FQ_PIE initally didn't exhibit the error but later did in some cases. Your recommendation appears more solid so I'm now using it as well. Thanks.

tman222

@uptownvagrant said in Playing with fq_codel in 2.4:

Looking at the examples in the paper, I'm wondering why the Codel AQM is selected in the pfSense WebUI in the August 2018 hangout? Per the FQ-CoDel examples in the paper above, it does not seem appropriate and removing Codel as the AQM from the pipe and queue removes the "flowset busy" error @mattund mentioned 4 months ago. @dtaht this is why I was stating codel+fq-codel - when I first learned about FQ-CoDel being added to pfSense 2.4.4, it was in the hangout video which it instructs to choose Codel as the AQM.

I have wondered the same as well (if you look up a few posts I shared some thoughts on this based on my current understanding of Dummynet and Limiters). In most situations, the scheduler chosen is just that - a scheduler only. In that case controlling the traffic flowing to the scheduler in the queue(s) makes sense to me. However, fq_codel combines scheduling and AQM into one, so having Codel on the input queue(s) seems a bit redundant to me. Now having said that, I currently have both enabled and it provides the best performance in my case. I'm still trying to figure out exactly as to why, but it might be because I'm trying to push a lot of packets from a 10Gbit LAN link into a 1Gbit WAN link and the additional AQM helps keeps things orderly.

I would be very interested to see some comparisons of Codel + fq_codel vs. just fq_codel as I do wonder at which point it actually starts to make a difference vs. just using additional processing without any real benefit.

strangegopher

Just wanted to report that my ipq806x based router tp-link archer c2600 could only do about 85-119 mbps download with qos on but was able to max out my speed with qos off. Guess I'll be returning the router.

zwck

@dtaht said in Playing with fq_codel in 2.4:

flent has now been packaged up and made available for freebsd.

https://github.com/tohojo/flent/commit/c928c03a301258c26c7d045c74ecce6dfeaa3d5a

This has not gone unnoticed, it took me while to take a look!
In order to use the flent pkg you most likely have to be on the latest pkg train (often its the quarterly branch). For the FreeNAS users:

Create the file /usr/local/etc/pkg/repos/FreeBSD.conf as noted but override the URL instead of disabling the repository:
Code:

FreeBSD: {
  url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest"
}

then in shell

pkg update
pkg install flent
# first test
flent rrul -p all_scaled -l 60 -H flent-london.bufferbloat.net -t no_shaper -o RRUL_no_shaper.png

cheers!

strangegopher

@mattund wouldn't there be a security issue with wan facing bridge interfaces and lan facing management interface on the same machine?

mattund

@strangegopher

Eh, it's an unaddressed layer 2 bridge and just passes the traffic straight through. It won't route to the internal net from the external side unless it's through pfSense. On the internal side, I guess, but this is all home usage. I think it's ultimately up to the implementer to decide how they best want to manage it, my choice to use that management interface doesn't impact the functionality of what it accomplishes (although, like dtaht said, you lose per-host FQ if you neglect the NAT to pfSense like I am, but I am OK with that)

I do recommend disabling IPv6 on the machine's "default" interfaces (everything but eth0) though: net.ipv6.conf.default.disable_ipv6=1, so it will only accept forward Layer2 stuff, and just double-check ip route, ip -6 route are all via eth0.... let me know if there's something I'm missing.

strangegopher

@mattund I see, just checking if it was an issue.

Rasool

Hello everyone,

I am the developer of Dummynet AQM (CoDel/PIE/FQ-CoDel and FQ-PIE). I have read part of this very long interesting thread and I can see some of you have the "config_aqm Unable to configure flowset, flowset busy!" error.
That error tells you that you are trying to reconfigure a flowset (pipe or queue) while there is an actual traffic uses that flowset (it has an active scheduler). I prevent reconfiguring an active flowset because I had some difficulties freeing/reallocating individual AQM (CoDel and PIE) memory space (as well as timeout function used by PIE which can cause a kernel panic). Definitely, the issue can be solved with some work but I hadn't (and I still don't have) enough time to fix that issue.
However, there is an easy workaround to avoid this error. If you make sure that there's no traffic passes through the pipe/queue that you want to reconfigure, then you can reconfigure the pipe/queue without problems. To achieve that you can use "skipto" action (in ipfw) to skip the rules that include the pipes/queues you would like to configure. Here is an example:

00010  70451180  71665155748 skipto 65534 ip from any to any
00100    188029      9852488 pipe 1 ip from 172.16.10.0/24 to 172.16.11.0/24 out
65534 165525594 168354591831 allow ip from any to any
65535         0            0 deny ip from any to any

Now, pipe 1 can be reconfigured without problems.

ipfw pipe 1 config bw 10mbit/s codel

Please note that sometimes you should wait for a short time (based on net.inet.ip.dummynet.expire sysctl) to allow queues/schedulers to drain before reconfiguring the pipe/queue.
For scheduler/AQM cases (i.e. fq_codel and fq_pie), you have to skip the queues instead of pipes.

I hope this workaround works for you. Please let me know if you still have issues and I will try to do my best to provide solutions.

Regards,
Rasool Al-Saadi

Rasool

Additionally, I cannot understand what is the purpose of using Codel + fq_codel?
I want to clarify some dummynet/AQM internals to understand what does Codel + fq_codel mean. (all figures are from [http://caia.swin.edu.au/reports/160708A/CAIA-TR-160708A.pdf]).

CoDel and PIE:
In a simple setup, when a new dummynet pipe is configured, a queue (Droptail queue), two schedulers (FIFO and WF2Q+) and a link (traffic shaper and delay emulation) are created. The queue is connected to FIFO scheduler and the scheduler is connected to a link. If we reconfigure the pipe, dummynet will configure the queue and the link.

For example, the following command will set traffic shaping to 10mbit/s for the link and set Codel AQM (instead of droptail) to be used with the queue. Note that CoDel and PIE can be used to manage the queue (not the pipe).

ipfw pipe 1 config bw 10mbit/s codel

fq_codel and fq_pie:
fq_codel and fq_pie algorithms are implemented as dummynet schedulers and they have internal sub-queues as shown in the figure below.

We need a queue, a scheduler and a link to configure fq_codel/fq_pie. The queue is not "really" used to buffer packets but just to add it the ipfw rules. All packets are stored in fq_codel/fq_pie sub-queues directly. The scheduler is full fq_codel and fq_pie implementation (sub-queue buffers, DDR and codel/pie). The link is a normal dummynet link (traffic shaper).
We have to configure a pipe to create a link since we cannot configure a link alone (according to my knowledge).
Example:

ipfw pipe 1 config  bw 800Mb
ipfw sched 1 config pipe 1 type fq_codel target 5ms interval 100ms quantum 1514 limit 1024 flows 1024 ecn
ipfw queue 1 config pipe 1 sched 1

Then we add queue 1 to ipfw rules (in my case)

 ipfw add 100 queue 1 ip from 172.16.10.0/24 to 172.16.11.0/24 out

Now, Codel + fq_codel is basically just fq_codel if the firewall is configured using the method above.

Finally, I saw some configurations use queue with "mask" option and Codel. That is useful if you want to try different scheduling algorithms with codel/pie and flow separation to produce something similar to fq_codel. For example, you use that method to produce WF2Q+_codel, qfq_pie, qfq_codel, ... etc. This method produces similar results to fq_codel in general but not exactly the same (no priority for new flows for example).

Hopefully, the explanation above was clear and useful.

Regards,
Rasool