Playing with fq_codel in 2.4

belt9

I'm not sure one is better than the other.

It depends on what you're trying to do.

ALTQ gives you a lot of options and control.

Dummynet is much less specific, but is very easy to implement. IMO, fq_codel is the only dummynet algorithm worth messing around with for most home users. But there are reasons to use the other algorithms.

So really it depends on what you are trying to do and why.

As far as testing your specific network under different types of traffic shaping, I would recommend FLENT.

https://flent.org/index.html

tman222

@belt9:

I'm not sure one is better than the other.

It depends on what your trying to do.

ALTQ gives you a lot of options and control.

Dummynet is much less specific, but is very easy to implement. IMO, fq_codel is the only dummynet algorithm worth messing around with for most home users. But there are reasons to use the other algorithms.

So really it depends on what you are trying to do and why.

As far as testing your specific network under different types of traffic shaping, I would recommend FLENT.

https://flent.org/index.html

Thanks again for your help and explanation - I will check out that link.

EricE

@superbree:

I really wish I could find a way to limit a subnet to say 100Mbs and then limit each ip host address in the subnet to 5 Mbs. And then have each IP address dynamically shaped if the overall link was approaching the 100Mbs total.

Take a look at the message in thread: https://forum.pfsense.org/index.php?topic=99503.msg642533#msg642533

You should be able to tweak the config files from that thread to match your networks and do exactly what you want.

Harvy66

fq_Codel is a zero-config AQM. All it needs is to be hooked up to a shaper of some sort and and works magic. You really need to understand how to traffic shape to do better than it. Eve then, it's great.

I hope they get the kinks worked out of Cake, but they keep trying to add the kitchen sink and has caused some back-and-forth regression.

belt9

@Harvy66:

fq_Codel is a zero-config AQM. All it needs is to be hooked up to a shaper of some sort and and works magic. You really need to understand how to traffic shape to do better than it. Eve then, it's great.

Agreed, it really is very impressive - probably one of the more impressive things I've seen in pfSense.

It's a huge improvement for very little config, and the config you have to do is not complicated even for a non-tech-savvy home user.

Netgate should implement some sort of automatic bandwidth limiting, and place that in the UI next to dummynet using fq_codel. Maybe 2.4.2?

The net result of the above would be that pfSense would dramatically improve the quality of even the crappiest connections from ISP with a sub 5 minute configuration for even the least experienced user.

I will grant you that pfSense can already do that (very, very well) with HFSC and limiting your bandwidth manually to below the lowest values you ever see. But HFSC you have to learn how to do, and as Harvy noted - even if you know what you're doing you will have to spend some time getting it as good as fq_codel can be just by turning it on. The result of that is most people either don't use it or don't use it well.

Also, many WAN connection speeds dip dramatically during peak hours. No one wants to cut their bandwidth down by a large percentage all the time just so their limiter can catch the traffic during peak hours.
Either an automatic speedtest similar to ubiquiti's, or an automatic latency test similar to gargoyle could be leveraged to automatically keep bandwidth limited just below the current WAN speeds so your limiter is always catching the traffic and you are always making the most of your available bandwidth.

fq_codel + automatic bandwidth limiter = killer app - huge bullet point for pfSense & Netgate.

EricE

If anyone has an fq_codel resource they can point me to that demonstrates how to de-priortize traffic to a group of specific subnets, I'd love to see it.

I'm trying to de-prioritize traffic to Backblaze servers as outlined in this thread. I don't want to limit it, just make it a lower priority for any other traffic that happens, including starving out Backblaze entirely if there is ANY other traffic. But if there isn't other traffic, let Backblaze consume all the available bandwidth of the internet connection.

I would have thought not only would this be a simple thing to do, but it would also be fairly common - ha! I have found precious few examples of how to do ANY traffic shaping to specific subnets - everything I have found so far is all around specific ports or services, which won't work in this instance since all the Backblaze traffic all over SSL. They do provide a list of all the subnets their servers are on, so I have (one would think!) an easy way to classify their traffic

belt9

@EricE:

If anyone has an fq_codel resource they can point me to that demonstrates how to de-priortize traffic to a group of specific subnets, I'd love to see it.

I'm trying to de-prioritize traffic to Backblaze servers as outlined in this thread. I don't want to limit it, just make it a lower priority for any other traffic that happens - but if there isn't other traffic, consume all the available bandwidth.

For this, set your up and down limiters like normal.

Within each limit set two queues, lets say you call one normal and the other backblaze.

Set the subnet to match your network (probably /24). Down=destination up=source

If you wanted to prioritize normal traffic to have 90% bandwidth and backblaze to get 10% when the pipe is full. Then weight normal as 90 and backblaze as 10.

If the pipe is empty backblaze can still use it all.

EricE

I'm going to ask what may seem like dumb or trivial questions, just because I have seen so much conflicting information I don't want to leave anything to assumptions. Thanks in advance.

@belt9:

For this, set your up and down limiters like normal.

So are we talking Limiters in the Firewall/Traffic Shaper/Limiters or Firewall/Traffic Shaper/By Interface?

@belt9:

Within each limit set two queues, lets say you call one normal and the other backblaze.

OK - right now I've got CODELQ queues in Interfaces side and that doesn't support sub-queues, but it was also the only thing that appeared to touch buffer bloat. Sounds like I need to be in the limiters instead - that might be where I went wrong.

@belt9:

Set the subnet to match your network (probably /24). Down=destination up=source

I'm assuming your talking about a firewall match or pass rule to classify the traffic and assign it to a queue. If I'm using a floating rule I want the interface to be WAN and the Destination to have the Backblaze networks, right? I have an Alias with all the subnets for the BackBlaze servers.

I never did get a floating rule to work, but a Pass rule directly on the LAN interface worked with the Backblaze subnet list alias in the Destination section. It's just the wizard configs for traffic shaping didn't seem to touch buffer bloat - latency and overall bandwidth was horrible. But CODELQ only handles buffer bloat wonderfully but I didn't see how to shape the Backblaze traffic.

It sounds like I really need to play with the limiters instead. Thanks again for the hints.

belt9

@EricE:

I'm going to ask what may seem like dumb or trivial questions, just because I have seen so much conflicting information I don't want to leave anything to assumptions. Thanks in advance.

@belt9:

For this, set your up and down limiters like normal.

So are we talking Limiters in the Firewall/Traffic Shaper/Limiters or Firewall/Traffic Shaper/By Interface?

Firewall/Traffic Shaper/Limiters

@EricE:

@belt9:

Within each limit set two queues, lets say you call one normal and the other backblaze.

OK - right now I've got CODELQ queues in Interfaces side and that doesn't support sub-queues, but it was also the only thing that appeared to touch buffer bloat. Sounds like I need to be in the limiters instead - that might be where I went wrong.

CODELQ is under the ALTQ system - which does certainly work, it's just a much more involved setup.

@EricE:

@belt9:

Set the subnet to match your network (probably /24). Down=destination up=source

I'm assuming your talking about a firewall match or pass rule to classify the traffic and assign it to a queue. If I'm using a floating rule I want the interface to be WAN and the Destination to have the Backblaze networks, right? I have an Alias with all the subnets for the BackBlaze servers.

When you make the queues in dummynet there will be an area to enter your subnet size so that it can share traffic between clients, set that to match which probably means setting it to /24, and the Download limiter is "destination" and Upload Limiter is "source".
You will also need to apply your queues to firewall rules.

In order to make sure everything is working and set correctly, I would temporarily set your up and down speeds to something way under your upload speed and set a unique number so that you will easily recognize it on speedtest.
What I mean by that is, if your normal download/up speeds are 40/10, then on dummynet set download to something like 4200Kbps and set upload to something like 650Kbps.
The only point of this is so that if you've accidentally reversed the upload and download queues in your firewall rules you will easily recognize that and fix it when you run a speedtest if you see upload at 4.2Mbps and download at 0.65Mbps. If you already know it's all setup correctly then just skip all that stuff.

@EricE:

I never did get a floating rule to work, but a Pass rule directly on the LAN interface worked with the Backblaze subnet list alias in the Destination section. It's just the wizard configs for traffic shaping didn't seem to touch buffer bloat - latency and overall bandwidth was horrible. But CODELQ only handles buffer bloat wonderfully but I didn't see how to shape the Backblaze traffic.

It sounds like I really need to play with the limiters instead. Thanks again for the hints.

I'm sorry for the convoluted explanation. I'm not near a pfSense box I can access and won't be for awhile. Otherwise I would just give you a screenshot to explain this, it's very easy I'm just trying to explain this from memory of what the config GUI looks like.

w0w

Patch for Limiter Info page with schedulers information and refresh interval of 500ms


--- diag_limiter_info.php	Wed Sep 07 00:26:47 2016
+++ diag_limiter_info.php	Sun Oct 01 08:20:33 2017
@@ -40,5 +40,5 @@
 	echo $text;
-	$text = `/sbin/ipfw queue show`;
+	$text = `/sbin/ipfw sched show`;
 	if ($text != "") {
-		echo "\n\n" . gettext("Queues") . ":\n";
+		echo "\n\n" . gettext("Shedulers") . ":\n";
 		echo $text;
@@ -72,3 +76,3 @@
 	events.push(function() {
-		setInterval('getlimiteractivity()', 2500);
+		setInterval('getlimiteractivity()', 500);
 		getlimiteractivity();

Chrismallia

@belt9:

@Harvy66:

fq_Codel is a zero-config AQM. All it needs is to be hooked up to a shaper of some sort and and works magic. You really need to understand how to traffic shape to do better than it. Eve then, it's great.

Agreed, it really is very impressive - probably one of the more impressive things I've seen in pfSense.

It's a huge improvement for very little config, and the config you have to do is not complicated even for a non-tech-savvy home user.

Netgate should implement some sort of automatic bandwidth limiting, and place that in the UI next to dummynet using fq_codel. Maybe 2.4.2?

The net result of the above would be that pfSense would dramatically improve the quality of even the crappiest connections from ISP with a sub 5 minute configuration for even the least experienced user.

I will grant you that pfSense can already do that (very, very well) with HFSC and limiting your bandwidth manually to below the lowest values you ever see. But HFSC you have to learn how to do, and as Harvy noted - even if you know what you're doing you will have to spend some time getting it as good as fq_codel can be just by turning it on. The result of that is most people either don't use it or don't use it well.

Also, many WAN connection speeds dip dramatically during peak hours. No one wants to cut their bandwidth down by a large percentage all the time just so their limiter can catch the traffic during peak hours.
Either an automatic speedtest similar to ubiquiti's, or an automatic latency test similar to gargoyle could be leveraged to automatically keep bandwidth limited just below the current WAN speeds so your limiter is always catching the traffic and you are always making the most of your available bandwidth.

fq_codel + automatic bandwidth limiter = killer app - huge bullet point for pfSense & Netgate.

Agreed with all you said. They should look into implementing it asap

superbree

@w0w:

Patch for Limiter Info page with schedulers information and refresh interval of 500ms


--- diag_limiter_info.php	Wed Sep 07 00:26:47 2016
+++ diag_limiter_info.php	Sun Oct 01 08:20:33 2017
@@ -40,5 +40,5 @@
 	echo $text;
-	$text = `/sbin/ipfw queue show`;
+	$text = `/sbin/ipfw sched show`;
 	if ($text != "") {
-		echo "\n\n" . gettext("Queues") . ":\n";
+		echo "\n\n" . gettext("Shedulers") . ":\n";
 		echo $text;
@@ -72,3 +76,3 @@
 	events.push(function() {
-		setInterval('getlimiteractivity()', 2500);
+		setInterval('getlimiteractivity()', 500);
 		getlimiteractivity();

Would love to try this patch out. This will show fq_codel on the limiter info page? Is there are kind soul who could explain how to implement this to the lay person?

belt9

There's a redmine feature request to get an automatic bandwidth limiter added to dummynet.

If anyone is interested and technically inclined please chime in!

Check out the links in my signature for more info.

https://redmine.pfsense.org/issues/7904

cplmayo

I finally got fq_codel limiters applied to just my WAN connection via floating rules.

From what I am seeing I think I like it better than using my vlan's interfaces. From what I am seeing in my own testing the jitter seems lower and I see fewer latency spikes on my upload bandwidth tests. Also since this is queuing all traffic on the WAN interface I feel like it is handling separate flows better than it did before.

I could be wrong and all of this is anecdotal or a placebo affect from all of my messing around with shappers and limiters.

If anyone is interested in trying it out the setup is fairly easy.

Firewall > Rules > Floating

*Add new rule

*Change "Action" from "Pass" to "Match"

*Select "WAN" in Interface

*Set "Direction" to "Out"

*Set "Protocol" to "any"

*Source to "any"

*Destination to "any"

Advanced settings

*Set Gateway (Cannot leave as default; you have to specifically set it to your configured gateway)

*Set In/Out (Because it is a floating rule and it is set to "Out" it gets a little confusing. It reverses In/Out ie In is for outgoing and Out is for incoming.)

Harvy66

dslreports.com has a good bufferbloat test.

w0w

@superbree:

Would love to try this patch out. This will show fq_codel on the limiter info page? Is there are kind soul who could explain how to implement this to the lay person?

You need "System patches" package.
Create new patch and apply it. See attachment.

limiters.jpg_thumb

johnpoz

I got asked in a PM to post some screenshots of my settings.. Figured post it here as reference.

Just apply the in/out pipe to firewall rule on your interface.. So that these do not effect your intervlan traffic if you have any. Put a rule above to allow access to your other vlans without the pipe's applied.

These settings changed my bufferbloat tests on dslreports to A..

fqcodel.png_thumb

belt9

Why a /32 IPv4 mask?

johnpoz

Because that is what comes up in the gui when this is the rules.limiter

[2.4.0-RELEASE][root@pfsense.local.lan]/root: cat /tmp/rules.limiter

pipe 1 config bw 85Mb
queue 1 config pipe 1 mask dst-ip6 /128 dst-ip 0xffffffff

pipe 2 config bw 11Mb
queue 2 config pipe 2 mask src-ip6 /128 src-ip 0xffffffff

Is something wrong there? It was working great!!!

belt9

Haha, I don't know to be honest. I had mine set the same way until I noticed that, then set it to /24 to match my network (I'm IPv4 only). I haven't been on that network in awhile now but I don't remember noticing a difference. My config is otherwise pretty much the same as yours.

Maybe someone can chime in on whether that setting matters or not and exactly what it is doing?

I know that in some parts of traffic shaping GUI there are options presented that don't apply to all types of shaping.