Playing with fq_codel in 2.4

w0w

@cplmayo
Install System_Patches package, go to System->Patches, Press Add New Patch button and copy-paste URL https://github.com/pfsense/pfsense/compare/RELENG_2_4_3...mattund:RELENG_2_4_3.diff into URL/Commit ID field.
Also, write something into description field, press SAVE button. After that you will see you newly created patch in the list and Fetch button on the right, press it, then press appeared Test button, you should see

Patch can be applied cleanly (detail)
Patch can NOT be reverted cleanly (detail)

If you see that, then you can press Apply button and enjoy your new Limiters

SlowGrind

So after applying the patch do you just fill in the settings under limiters?

New here to traffic shaping so just looking for an easy way to decrease latency. Thanks.

w0w

@slowgrind
Generally yes, but I prefer to share bandwidth between all users evenly.
Unfortunately looks like foxale08 guide is broken, it was here but now it's completely gone :(
I'll provide some guide later, if you want it.

SlowGrind

@w0w

Thank you and I just have a simple network setup. 1 wan and 1 lan with 1000 Mbps download and 50 Mbps upload.

mattund

I'm testing my patch, and I am seeing some unexpected behavior (maybe intentional).

pipe 2 config  bw 122800Kb queue 100 buckets 256 droptail
sched 2 config pipe 2 type fq_codel target 35ms interval 35ms quantum 1514 limit 10240 flows 1024 noecn
queue 1 config pipe 2 queue 100 droptail

pipe 1 config  bw 12000Kb queue 100 droptail
sched 1 config pipe 1 type fq_codel target 5ms interval 10ms quantum 1514 limit 10240 flows 1024 noecn
queue 2 config pipe 1 queue 100 droptail

Using the above generated ruleset, if I start a speedtest, and have an ICMP ping up at the same time, upon reaching the ceiling of ~120Mbps, pings drop completely. Not bufferbloat, but full drops. Now, of course that would happen, the limiter is supposed to limit the bandwidth. However, is there a way around this? I figured burst might help, but it doesn't appear to -- FQ_CoDel assumes the new burst value as part of the total bandwidth share and doesn't kick in.

It's almost as though FQ_CoDel isn't active, and the limiter is putting itself into the drop state before CoDel can say, "hey, this teeny tiny little ICMP packet isn't part of a busy flow, so let's let that through". Unless I am mistaken on something.

What I want:

Small flow traffic to not get dropped (think my little ping)
Busy flow traffic to be shaped (dropped) appropriately
Not allow the limiter pipe drop packets before FQ_CoDel has a chance to process them (?)

I was under the impression FQ_CoDel was designed around those objectives. Am I doing this right?

cplmayo

@mattund said in Playing with fq_codel in 2.4:

I'm testing my patch, and I am seeing some unexpected behavior (maybe intentional).
pipe 2 config  bw 122800Kb queue 100 buckets 256 droptail
sched 2 config pipe 2 type fq_codel target 35ms interval 35ms quantum 1514 limit 10240 flows 1024 noecn
queue 1 config pipe 2 queue 100 droptail

pipe 1 config  bw 12000Kb queue 100 droptail
sched 1 config pipe 1 type fq_codel target 5ms interval 10ms quantum 1514 limit 10240 flows 1024 noecn
queue 2 config pipe 1 queue 100 droptail
Using the above generated ruleset, if I start a speedtest, and have an ICMP ping up at the same time, upon reaching the ceiling of ~120Mbps, pings drop completely. Not bufferbloat, but full drops. Now, of course that would happen, the limiter is supposed to limit the bandwidth. However, is there a way around this? I figured burst might help, but it doesn't appear to -- FQ_CoDel assumes the new burst value as part of the total bandwidth share and doesn't kick in.

It's almost as though FQ_CoDel isn't active, and the limiter is putting itself into the drop state before CoDel can say, "hey, this teeny tiny little ICMP packet isn't part of a busy flow, so let's let that through". Unless I am mistaken on something.

Am I doing this right?

I applied your patch to my install and so far it is working as intended. I haven't noticed any huge difference between it and manual approach I have been using for months.

I did noticed that when I tried setting weights for different queues and if I set the scheduler in the queues no traffic would pass. I went back and reread your instructions on GitHub and once I removed the weights and scheduler options from the queue it works.

I am still doing some testing to see results in dslreports but so far your interface is simple and just seems to work.

Great job!

mattund

@slowgrind said in Playing with fq_codel in 2.4:

So after applying the patch do you just fill in the settings under limiters?

Here's what I'm doing. This might be a little more than what you need, but I figure I would share my configuration in case others have a crazy Multi-WAN multi-LAN setup like I do. I've constructed a series of limiters, one for download and one for upload, each with its own associated queue (you can make the queue with the "+ Add new Queue" button on the bottom of a Limiter's settings page) :

(I have more for my second ISP following that naming scheme: lINTERFACEDownload/lINTERFACEUpload and qINTRERFACEDownload/qINTERFACEUpload children)

I'm assigning FQ_CoDel to the scheduler on the parent limiter and leaving everything else alone. You can either edit the parameters, or leave them at default if you have a typical connection (FQ_CoDel is supposed to be "knobless" after all).

According to the following diagram, this is how the traffic will flow inside dummynet:

			    (flow_mask|sched_mask)  sched_mask
		    +---------+	  weight Wx  +-------------+
		    |	      |->-[flow]-->--|		   |-+
	       -->--| QUEUE x |	  ...	     |		   | |
		    |	      |->-[flow]-->--| SCHEDuler N | |
		    +---------+		     |		   | |
			...		     |		   +--[LINK N]-->--
		    +---------+	  weight Wy  |		   | +--[LINK N]-->--
		    |	      |->-[flow]-->--|		   | |
	       -->--| QUEUE y |	  ...	     |		   | |
		    |	      |->-[flow]-->--|		   | |
		    +---------+		     +-------------+ |
					       +-------------+

via: https://www.freebsd.org/cgi/man.cgi?query=ipfw&manpath=FreeBSD+9-current&format=html

Dissection: firewall traffic is assigned to a queue, which then generates flows defined by the mask, which pipe into the scheduler (set to FQ_CoDel), which then outputs to the pipe/link at the specified max bitrate.

To assign your traffic to queues, you could do something like I did, which is to use floating rules. I have two WANs, and I need independent shaping and all that, so if you're on a single WAN it may be different for you/you may have better options.

How I set the rules up:

Interface: WAN A or B interface
Direction: out
Address Family: IPv4 or IPv6; I had to do two rules, one for each IP version
Gateway: Select the applicable IPv4 or IPv6 gateway consistent with how traffic should be routed on that IP stack
In / Out pipe: qCHARTERUpload / qCHARTERDownload

I have some filtering rules in play here as you can see in my screenshot, but that's only since I'm testing some issues I mentioned previously. It's up to you if you want to match certain protocols/ports, etc.

mattund

@mattund said in Playing with fq_codel in 2.4:

I'm testing my patch, and I am seeing some unexpected behavior (maybe intentional).

This stopped happening as soon as I enabled masks on the offending download queue correlating to the one ICMP traffic was being dropped on. Not sure why.

cplmayo

@mattund said in Playing with fq_codel in 2.4:

@slowgrind said in Playing with fq_codel in 2.4:

So after applying the patch do you just fill in the settings under limiters?

Here's what I'm doing. This might be a little more than what you need, but I figure I would share my configuration in case others have a crazy Multi-WAN multi-LAN setup like I do. I've constructed a series of limiters, one for download and one for upload, each with its own associated queue (you can make the queue with the "+ Add new Queue" button on the bottom of a Limiter's settings page) :

(I have more for my second ISP following that naming scheme: lINTERFACEDownload/lINTERFACEUpload and qINTRERFACEDownload/qINTERFACEUpload children)

I'm assigning FQ_CoDel to the scheduler on the parent limiter and leaving everything else alone. You can either edit the parameters, or leave them at default if you have a typical connection (FQ_CoDel is supposed to be "knobless" after all).

According to the following diagram, this is how the traffic will flow inside dummynet:
			    (flow_mask|sched_mask)  sched_mask
		    +---------+	  weight Wx  +-------------+
		    |	      |->-[flow]-->--|		   |-+
	       -->--| QUEUE x |	  ...	     |		   | |
		    |	      |->-[flow]-->--| SCHEDuler N | |
		    +---------+		     |		   | |
			...		     |		   +--[LINK N]-->--
		    +---------+	  weight Wy  |		   | +--[LINK N]-->--
		    |	      |->-[flow]-->--|		   | |
	       -->--| QUEUE y |	  ...	     |		   | |
		    |	      |->-[flow]-->--|		   | |
		    +---------+		     +-------------+ |
					       +-------------+
via: https://www.freebsd.org/cgi/man.cgi?query=ipfw&manpath=FreeBSD+9-current&format=html

Dissection: firewall traffic is assigned to a queue, which then generates flows defined by the mask, which pipe into the scheduler (set to FQ_CoDel), which then outputs to the pipe/link at the specified max bitrate.

To assign your traffic to queues, you could do something like I did, which is to use floating rules. I have two WANs, and I need independent shaping and all that, so if you're on a single WAN it may be different for you/you may have better options.

How I set the rules up:

Interface: WAN A or B interface

Direction: out

Address Family: IPv4 or IPv6; I had to do two rules, one for each IP version

Gateway: Select the applicable IPv4 or IPv6 gateway consistent with how traffic should be routed on that IP stack

In / Out pipe: qCHARTERUpload / qCHARTERDownload

I have some filtering rules in play here as you can see in my screenshot, but that's only since I'm testing some issues I mentioned previously. It's up to you if you want to match certain protocols/ports, etc.

When I setup my floating rules I tried using the out direction a few times but have run into problems each and every time.

For instance if I set the rule direction to "out" and then set the gateway I get what appears to be a routing loop. Traffic forwards find and I can hit external hosts however when I do a traceroute from an internal host I don't see the hop routers I just see the destination IP over and over. Are you seeing this or have you changed your setup?

I have also been considering using the dynamic pipes. Anyone tried this with the fq_codel scheduler?

mattund

@cplmayo I also get a routing loop in traceroutes. Are you using the in direction?

cplmayo

@mattund I have been, I would prefer to use out but it makes any troubleshooting I may need to do difficult. Not that I do traceroutes a lot but it annoyed me.

I have wondered if this was due to how the rule is creating states. Like it is rerouting the packet once the rule is applied; very weird.

There are two floating rules both in; one on WAN the other all of my vlans.

I wish I could just apply them to the WAN with in & out as the directions.

xRaisen

Hello, I am new here. I have been following this thread since last week. Tried the new patch and it's working as intended just like the manual. I leave everything in default, I just add the Upload/Download bandwidth Queue Management Algorithm
CoDel and FQ_Codel for Scheduler.

I have tested it, hogging my connections doing 20 torrents and 1 download on IDM. Played Online game to see the ping and CMD ping www.google.com -t, I experienced 1 to 2 packet loss, and 3 to 5 ping dancing and I can still play like as if there were no downloads happening on the background.

Some issues on the syslog, but fixed it by this command: ifconfig <nics> -promisc

0_1529003640259_4f840f82-1253-4255-a783-eb3ad481a2d3-image.png

Just a heads up! This one though, it's just showing on the syslogs everytime.

0_1529003711093_e9ff68bc-48a9-450f-8db7-ed6bda851357-image.png

By the way, I have Floating rules active and limiters(for LAN scheds, no queues).

Overall, thank you. This helps me a lot.

cplmayo

@xraisen Your nics shouldn't need to be in promiscuous mode. That is a message that I would ignore.

Promiscuous mode is just going to have your NIC accept all packets.

Non-Promiscuous mode will have you NIC drop packets not destined for it.

Unless you are doing packet captures I can't think of a reason you would want to run in promiscuous mode.

xRaisen

@cplmayo This was enabled default because I don't remember enabling this. But thanks for the info!

mattund

@xraisen

I'm also seeing the flowset busy log entries occasionally, I think it has to do with the fact that the rules don't always get deleted (just reconfigured). However, I am not positive on that yet. Others may have gotten around this or have other ideas...

EDIT:

We're a lot like opnsense with this patch, because they are using this (dummynet - not my patch lol) for their shaping I think. Well, regardless, this ticket:

https://github.com/opnsense/core/issues/1279

Due to the fact that the internal variables of each AQM instance is allocated and freed dynamically, re-configuring AQM (e.g. select CoDel for a pipe that currently uses PIE)
can lead the AQM code to access unallocated memory (freed during AQM re-configuration) and kernel panic could happen.
Therefore, I tried to avoid any potential kernel panic by preventing re-configuring AQM for busy pipe/queue.

He suggests:

Temporarily block the traffic (using IPFW rules) from passing through AQM PIE/queue until you finish your configuration.
Delete the old pipes/queues that need to be re-configured with AQM and create them again with all the desired settings.
Reboot the firewall

I think this really would need to be a code-level patch somehow. Somehow "unbusy" it, then apply

EDIT 2:

I find this kind of hilarious and coincidental, but I'm experiencing this right now. I'm going to screw around with my shaper.inc and see what it takes to shake this error.

EDIT 3:

Deleting and recreating the rules does nothing to alleviate the error, however setting the Queue Management Algorithm to on the Parent Limiter droptail worked to get it to stop throwing each apply for me.

cplmayo

@mattund From my experience with limiters and floating rules; always apply after any single change to a limiter. Each time I make a change I save and apply. I ran into problems at first when I would save several changes and apply at once.

May help may not.

For floating rules always reset states after making a change. This is due to how pfSense handles floating rules. It just helps to ensure that your floating rules are being applied to all of your traffic. If you add a floating rule and do not reset states only new traffic will be evaluated against the rule.

w0w

This post is deleted!

xRaisen

@cplmayo said in Playing with fq_codel in 2.4:

@xraisen Your nics shouldn’t need to be in promiscuous mode. That is a message that I would ignore.
Promiscuous mode is just going to have your NIC accept all packets.
Non-Promiscuous mode will have you NIC drop packets not destined for it.
Unless you are doing packet captures I can’t think of a reason you would want to run in promiscuous mode.

I found out why it was enabled. Because I use the NTOPNG package. I have uninstalled it, now it's gone. The package turned on the promiscuous mode to monitor the packets.

xRaisen

@mattund said in Playing with fq_codel in 2.4:

He suggests:

Temporarily block the traffic (using IPFW rules) from passing through AQM PIE/queue until you finish your configuration.
Delete the old pipes/queues that need to be re-configured with AQM and create them again with all the desired settings.
Reboot the firewall

I have tried this method, but when doing "ipfw sched show" this will show up

0_1529008156466_c20bb1cc-e2f3-45d8-bce7-1d1bb14456d2-image.png

Notice the "Drop tail" cannot be changed to CoDel, PIE, RED and GRED and It's very persistent.

Please confirm this on your end.

EDIT 1:

Changing the Queue Management Algorithm to Tail Drop, the "config_aqm Unable to configure flowset, flowset busy!" was gone.

EDIT 2:

Trying Tail Drop + FQ_CODEL on scheds and CODEL on queues and use queues instead of scheds, gives me a better results without the syslog nagging. Hogging the bandwidth again for testing, I got zero packet loss and ping on www.google.com and CSGO are dancing to 0ms to 1ms swiftly like nothing happened. This is the best bandwidth management setup I have experienced so far on pfsense. Take note, I have an active traffic shaper (fairq+codel) on floating rules for voice, games and browsing priorities.

zwck

@xraisen Would you mind sharing some screenshots of your config :D