Playing with fq_codel in 2.4

[SlowGrind]

@w0w

Thank you and I just have a simple network setup. 1 wan and 1 lan with 1000 Mbps download and 50 Mbps upload.

[mattund]

I'm testing my patch, and I am seeing some unexpected behavior (maybe intentional).

pipe 2 config  bw 122800Kb queue 100 buckets 256 droptail
sched 2 config pipe 2 type fq_codel target 35ms interval 35ms quantum 1514 limit 10240 flows 1024 noecn
queue 1 config pipe 2 queue 100 droptail

pipe 1 config  bw 12000Kb queue 100 droptail
sched 1 config pipe 1 type fq_codel target 5ms interval 10ms quantum 1514 limit 10240 flows 1024 noecn
queue 2 config pipe 1 queue 100 droptail

Using the above generated ruleset, if I start a speedtest, and have an ICMP ping up at the same time, upon reaching the ceiling of ~120Mbps, pings drop completely. Not bufferbloat, but full drops. Now, of course that would happen, the limiter is supposed to limit the bandwidth. However, is there a way around this? I figured burst might help, but it doesn't appear to -- FQ_CoDel assumes the new burst value as part of the total bandwidth share and doesn't kick in.

It's almost as though FQ_CoDel isn't active, and the limiter is putting itself into the drop state before CoDel can say, "hey, this teeny tiny little ICMP packet isn't part of a busy flow, so let's let that through". Unless I am mistaken on something.

What I want:

• Small flow traffic to not get dropped (think my little ping)
• Busy flow traffic to be shaped (dropped) appropriately
• Not allow the limiter pipe drop packets before FQ_CoDel has a chance to process them (?)

I was under the impression FQ_CoDel was designed around those objectives. Am I doing this right?

[cplmayo]

@mattund said in Playing with fq_codel in 2.4:

I'm testing my patch, and I am seeing some unexpected behavior (maybe intentional).

pipe 2 config  bw 122800Kb queue 100 buckets 256 droptail
sched 2 config pipe 2 type fq_codel target 35ms interval 35ms quantum 1514 limit 10240 flows 1024 noecn
queue 1 config pipe 2 queue 100 droptail

pipe 1 config  bw 12000Kb queue 100 droptail
sched 1 config pipe 1 type fq_codel target 5ms interval 10ms quantum 1514 limit 10240 flows 1024 noecn
queue 2 config pipe 1 queue 100 droptail

Using the above generated ruleset, if I start a speedtest, and have an ICMP ping up at the same time, upon reaching the ceiling of ~120Mbps, pings drop completely. Not bufferbloat, but full drops. Now, of course that would happen, the limiter is supposed to limit the bandwidth. However, is there a way around this? I figured burst might help, but it doesn't appear to -- FQ_CoDel assumes the new burst value as part of the total bandwidth share and doesn't kick in.

It's almost as though FQ_CoDel isn't active, and the limiter is putting itself into the drop state before CoDel can say, "hey, this teeny tiny little ICMP packet isn't part of a busy flow, so let's let that through". Unless I am mistaken on something.

Am I doing this right?

I applied your patch to my install and so far it is working as intended. I haven't noticed any huge difference between it and manual approach I have been using for months.

I did noticed that when I tried setting weights for different queues and if I set the scheduler in the queues no traffic would pass. I went back and reread your instructions on GitHub and once I removed the weights and scheduler options from the queue it works.

I am still doing some testing to see results in dslreports but so far your interface is simple and just seems to work.

Great job!

[mattund]

@slowgrind said in Playing with fq_codel in 2.4:

So after applying the patch do you just fill in the settings under limiters?

Here's what I'm doing. This might be a little more than what you need, but I figure I would share my configuration in case others have a crazy Multi-WAN multi-LAN setup like I do. I've constructed a series of limiters, one for download and one for upload, each with its own associated queue (you can make the queue with the "+ Add new Queue" button on the bottom of a Limiter's settings page) :

(I have more for my second ISP following that naming scheme: lINTERFACEDownload/lINTERFACEUpload and qINTRERFACEDownload/qINTERFACEUpload children)

I'm assigning FQ_CoDel to the scheduler on the parent limiter and leaving everything else alone. You can either edit the parameters, or leave them at default if you have a typical connection (FQ_CoDel is supposed to be "knobless" after all).

According to the following diagram, this is how the traffic will flow inside dummynet:

			    (flow_mask|sched_mask)  sched_mask
		    +---------+	  weight Wx  +-------------+
		    |	      |->-[flow]-->--|		   |-+
	       -->--| QUEUE x |	  ...	     |		   | |
		    |	      |->-[flow]-->--| SCHEDuler N | |
		    +---------+		     |		   | |
			...		     |		   +--[LINK N]-->--
		    +---------+	  weight Wy  |		   | +--[LINK N]-->--
		    |	      |->-[flow]-->--|		   | |
	       -->--| QUEUE y |	  ...	     |		   | |
		    |	      |->-[flow]-->--|		   | |
		    +---------+		     +-------------+ |
					       +-------------+

via: https://www.freebsd.org/cgi/man.cgi?query=ipfw&manpath=FreeBSD+9-current&format=html

Dissection: firewall traffic is assigned to a queue, which then generates flows defined by the mask, which pipe into the scheduler (set to FQ_CoDel), which then outputs to the pipe/link at the specified max bitrate.

To assign your traffic to queues, you could do something like I did, which is to use floating rules. I have two WANs, and I need independent shaping and all that, so if you're on a single WAN it may be different for you/you may have better options.

How I set the rules up:

• Interface: WAN A or B interface
• Direction: out
• Address Family: IPv4 or IPv6; I had to do two rules, one for each IP version
• Gateway: Select the applicable IPv4 or IPv6 gateway consistent with how traffic should be routed on that IP stack
• In / Out pipe: qCHARTERUpload / qCHARTERDownload

I have some filtering rules in play here as you can see in my screenshot, but that's only since I'm testing some issues I mentioned previously. It's up to you if you want to match certain protocols/ports, etc.

[mattund]

@mattund said in Playing with fq_codel in 2.4:

I'm testing my patch, and I am seeing some unexpected behavior (maybe intentional).

This stopped happening as soon as I enabled masks on the offending download queue correlating to the one ICMP traffic was being dropped on. Not sure why.

[cplmayo]

@mattund said in Playing with fq_codel in 2.4:

@slowgrind said in Playing with fq_codel in 2.4:

So after applying the patch do you just fill in the settings under limiters?

Here's what I'm doing. This might be a little more than what you need, but I figure I would share my configuration in case others have a crazy Multi-WAN multi-LAN setup like I do. I've constructed a series of limiters, one for download and one for upload, each with its own associated queue (you can make the queue with the "+ Add new Queue" button on the bottom of a Limiter's settings page) :

(I have more for my second ISP following that naming scheme: lINTERFACEDownload/lINTERFACEUpload and qINTRERFACEDownload/qINTERFACEUpload children)

I'm assigning FQ_CoDel to the scheduler on the parent limiter and leaving everything else alone. You can either edit the parameters, or leave them at default if you have a typical connection (FQ_CoDel is supposed to be "knobless" after all).

According to the following diagram, this is how the traffic will flow inside dummynet:

			    (flow_mask|sched_mask)  sched_mask
		    +---------+	  weight Wx  +-------------+
		    |	      |->-[flow]-->--|		   |-+
	       -->--| QUEUE x |	  ...	     |		   | |
		    |	      |->-[flow]-->--| SCHEDuler N | |
		    +---------+		     |		   | |
			...		     |		   +--[LINK N]-->--
		    +---------+	  weight Wy  |		   | +--[LINK N]-->--
		    |	      |->-[flow]-->--|		   | |
	       -->--| QUEUE y |	  ...	     |		   | |
		    |	      |->-[flow]-->--|		   | |
		    +---------+		     +-------------+ |
					       +-------------+

via: https://www.freebsd.org/cgi/man.cgi?query=ipfw&manpath=FreeBSD+9-current&format=html

Dissection: firewall traffic is assigned to a queue, which then generates flows defined by the mask, which pipe into the scheduler (set to FQ_CoDel), which then outputs to the pipe/link at the specified max bitrate.

To assign your traffic to queues, you could do something like I did, which is to use floating rules. I have two WANs, and I need independent shaping and all that, so if you're on a single WAN it may be different for you/you may have better options.

How I set the rules up:

• Interface: WAN A or B interface
• Direction: out
• Address Family: IPv4 or IPv6; I had to do two rules, one for each IP version
• Gateway: Select the applicable IPv4 or IPv6 gateway consistent with how traffic should be routed on that IP stack
• In / Out pipe: qCHARTERUpload / qCHARTERDownload

I have some filtering rules in play here as you can see in my screenshot, but that's only since I'm testing some issues I mentioned previously. It's up to you if you want to match certain protocols/ports, etc.

When I setup my floating rules I tried using the out direction a few times but have run into problems each and every time.

For instance if I set the rule direction to "out" and then set the gateway I get what appears to be a routing loop. Traffic forwards find and I can hit external hosts however when I do a traceroute from an internal host I don't see the hop routers I just see the destination IP over and over. Are you seeing this or have you changed your setup?

I have also been considering using the dynamic pipes. Anyone tried this with the fq_codel scheduler?

[mattund]

@cplmayo I also get a routing loop in traceroutes. Are you using the in direction?

[cplmayo]

@mattund I have been, I would prefer to use out but it makes any troubleshooting I may need to do difficult. Not that I do traceroutes a lot but it annoyed me.

I have wondered if this was due to how the rule is creating states. Like it is rerouting the packet once the rule is applied; very weird.

There are two floating rules both in; one on WAN the other all of my vlans.

I wish I could just apply them to the WAN with in & out as the directions.

[xRaisen]

Hello, I am new here. I have been following this thread since last week. Tried the new patch and it's working as intended just like the manual. I leave everything in default, I just add the Upload/Download bandwidth Queue Management Algorithm
CoDel and FQ_Codel for Scheduler.

I have tested it, hogging my connections doing 20 torrents and 1 download on IDM. Played Online game to see the ping and CMD ping www.google.com -t, I experienced 1 to 2 packet loss, and 3 to 5 ping dancing and I can still play like as if there were no downloads happening on the background.

Some issues on the syslog, but fixed it by this command: ifconfig <nics> -promisc

Just a heads up! This one though, it's just showing on the syslogs everytime.

By the way, I have Floating rules active and limiters(for LAN scheds, no queues).

Overall, thank you. This helps me a lot.

[cplmayo]

@xraisen Your nics shouldn't need to be in promiscuous mode. That is a message that I would ignore.

Promiscuous mode is just going to have your NIC accept all packets.

Non-Promiscuous mode will have you NIC drop packets not destined for it.

Unless you are doing packet captures I can't think of a reason you would want to run in promiscuous mode.

[xRaisen]

@cplmayo This was enabled default because I don't remember enabling this. But thanks for the info!

[mattund]

@xraisen

I'm also seeing the flowset busy log entries occasionally, I think it has to do with the fact that the rules don't always get deleted (just reconfigured). However, I am not positive on that yet. Others may have gotten around this or have other ideas...

EDIT:

We're a lot like opnsense with this patch, because they are using this (dummynet - not my patch lol) for their shaping I think. Well, regardless, this ticket:

https://github.com/opnsense/core/issues/1279

Due to the fact that the internal variables of each AQM instance is allocated and freed dynamically, re-configuring AQM (e.g. select CoDel for a pipe that currently uses PIE)
can lead the AQM code to access unallocated memory (freed during AQM re-configuration) and kernel panic could happen.
Therefore, I tried to avoid any potential kernel panic by preventing re-configuring AQM for busy pipe/queue.

He suggests:

• Temporarily block the traffic (using IPFW rules) from passing through AQM PIE/queue until you finish your configuration.
• Delete the old pipes/queues that need to be re-configured with AQM and create them again with all the desired settings.
• Reboot the firewall

I think this really would need to be a code-level patch somehow. Somehow "unbusy" it, then apply

EDIT 2:

I find this kind of hilarious and coincidental, but I'm experiencing this right now. I'm going to screw around with my shaper.inc and see what it takes to shake this error.

EDIT 3:

Deleting and recreating the rules does nothing to alleviate the error, however setting the Queue Management Algorithm to on the Parent Limiter droptail worked to get it to stop throwing each apply for me.

[cplmayo]

@mattund From my experience with limiters and floating rules; always apply after any single change to a limiter. Each time I make a change I save and apply. I ran into problems at first when I would save several changes and apply at once.

May help may not.

For floating rules always reset states after making a change. This is due to how pfSense handles floating rules. It just helps to ensure that your floating rules are being applied to all of your traffic. If you add a floating rule and do not reset states only new traffic will be evaluated against the rule.

[w0w]

This post is deleted!

[xRaisen]

@cplmayo said in Playing with fq_codel in 2.4:

@xraisen Your nics shouldn’t need to be in promiscuous mode. That is a message that I would ignore.
Promiscuous mode is just going to have your NIC accept all packets.
Non-Promiscuous mode will have you NIC drop packets not destined for it.
Unless you are doing packet captures I can’t think of a reason you would want to run in promiscuous mode.

I found out why it was enabled. Because I use the NTOPNG package. I have uninstalled it, now it's gone. The package turned on the promiscuous mode to monitor the packets.

[xRaisen]

@mattund said in Playing with fq_codel in 2.4:

He suggests:

Temporarily block the traffic (using IPFW rules) from passing through AQM PIE/queue until you finish your configuration.
Delete the old pipes/queues that need to be re-configured with AQM and create them again with all the desired settings.
Reboot the firewall

I have tried this method, but when doing "ipfw sched show" this will show up

Notice the "Drop tail" cannot be changed to CoDel, PIE, RED and GRED and It's very persistent.

Please confirm this on your end.

EDIT 1:

Changing the Queue Management Algorithm to Tail Drop, the "config_aqm Unable to configure flowset, flowset busy!" was gone.

EDIT 2:

Trying Tail Drop + FQ_CODEL on scheds and CODEL on queues and use queues instead of scheds, gives me a better results without the syslog nagging. Hogging the bandwidth again for testing, I got zero packet loss and ping on www.google.com and CSGO are dancing to 0ms to 1ms swiftly like nothing happened. This is the best bandwidth management setup I have experienced so far on pfsense. Take note, I have an active traffic shaper (fairq+codel) on floating rules for voice, games and browsing priorities.

[zwck]

@xraisen Would you mind sharing some screenshots of your config :D

[xRaisen]

@zwck I have a 50Mbps Symmetrical connection. So, I will give 40Mbps to Wired, and 10Mbps to Wifi. What will happen, when the wired are full, it will not eat up all the bandwidth reserved for the wifi devices and vice versa. These are my config as follows.

STEP 1: Applied this patch for ez life.

@w0w said in Playing with fq_codel in 2.4:

Applied this diff https://github.com/pfsense/pfsense/compare/RELENG_2_4_3…mattund:RELENG_2_4_3.diff
Adding new queue by pressing button on limiter settings page opens new page and you can add new queue save and apply it, but it does not appear in the list and I don't see it anywhere. But it can be only in my case, just because I have some April version of 2.4.4, before PHP7 preparations were made.

STEP 2: I created Aliases to separate Wired and WiFi

STEP 3: Create a limiters.

Download:

Upload:

Same to other limiters, just change the bandwidth.

STEP 5: Create child queues

Download queue:

Upload queue:

STEP 4: Create LAN rules and assign the pipe in/out as queue I've created.

And so on...

My rules arrangements:

Why I use queue than sched? I believe its a Flexible “fair use” bandwidth limiter

References:
#1 https://www.gridstorm.net/pfsense-traffic-limiting-fair-share/
#2 https://www.reddit.com/r/PFSENSE/comments/3e67dk/flexible_vs_fixed_limiters_troubleshooting_with/

My Traffic shaping rules for priorities. I am used to be a fan of HFSC, but I find this better than HFSC for my needs:

WAN: FAIRQ

WAN Child example:

The rest are self-intuitive.

My floating rules:

In my 4 years experience on pfsense, this is the best config I get so far for my requirements.

Many thanks to all members of this community. Ideas are from them, I just built it based from their brilliant minds. Credits and kudos to them!

EDIT 1:

I installed two packages only for my box. I have uninstalled squid and squidguard because it's not helping getting my desired QoS and I will stick to pfblockerng-devel by bbcan because it has a new feat that whatever squidguard had. I have installed FreeRadius for Captive Portal full control over it.

EDIT 2: Added this because it's really helpful.

[cplmayo]

@mattund So I'm an idiot and just figured out a way around the apparent routing loop. I redid my floating rules; created two rules one in and one out. However, I set the protocol to TCP/UDP since that is all I really want to apply QoS to anyway. No, that ICMP is not having the rule applied to it when I run a traceroute from a vLAN I can see the correct information.

So far for me, this has worked much better because I apply the rules directly to the WAN interface and not worry about creating extra rules to prevent shapping of local traffic.

So obvious and I can't believe I just thought of it today.

[strangegopher]

@xraisen According to openwrt wiki (only firewall where I was able to get A+ bufferbloat with fq_codel) ecn should only be enabled on inbound packets.