Playing with fq_codel in 2.4
-
If anyone has an fq_codel resource they can point me to that demonstrates how to de-priortize traffic to a group of specific subnets, I'd love to see it.
I'm trying to de-prioritize traffic to Backblaze servers as outlined in this thread. I don't want to limit it, just make it a lower priority for any other traffic that happens - but if there isn't other traffic, consume all the available bandwidth.
For this, set your up and down limiters like normal.
Within each limit set two queues, lets say you call one normal and the other backblaze.
Set the subnet to match your network (probably /24). Down=destination up=source
If you wanted to prioritize normal traffic to have 90% bandwidth and backblaze to get 10% when the pipe is full. Then weight normal as 90 and backblaze as 10.
If the pipe is empty backblaze can still use it all.
-
I'm going to ask what may seem like dumb or trivial questions, just because I have seen so much conflicting information I don't want to leave anything to assumptions. Thanks in advance.
For this, set your up and down limiters like normal.
So are we talking Limiters in the Firewall/Traffic Shaper/Limiters or Firewall/Traffic Shaper/By Interface?
Within each limit set two queues, lets say you call one normal and the other backblaze.
OK - right now I've got CODELQ queues in Interfaces side and that doesn't support sub-queues, but it was also the only thing that appeared to touch buffer bloat. Sounds like I need to be in the limiters instead - that might be where I went wrong.
Set the subnet to match your network (probably /24). Down=destination up=source
I'm assuming your talking about a firewall match or pass rule to classify the traffic and assign it to a queue. If I'm using a floating rule I want the interface to be WAN and the Destination to have the Backblaze networks, right? I have an Alias with all the subnets for the BackBlaze servers.
I never did get a floating rule to work, but a Pass rule directly on the LAN interface worked with the Backblaze subnet list alias in the Destination section. It's just the wizard configs for traffic shaping didn't seem to touch buffer bloat - latency and overall bandwidth was horrible. But CODELQ only handles buffer bloat wonderfully but I didn't see how to shape the Backblaze traffic.
It sounds like I really need to play with the limiters instead. Thanks again for the hints.
-
I'm going to ask what may seem like dumb or trivial questions, just because I have seen so much conflicting information I don't want to leave anything to assumptions. Thanks in advance.
For this, set your up and down limiters like normal.
So are we talking Limiters in the Firewall/Traffic Shaper/Limiters or Firewall/Traffic Shaper/By Interface?
Firewall/Traffic Shaper/Limiters
Within each limit set two queues, lets say you call one normal and the other backblaze.
OK - right now I've got CODELQ queues in Interfaces side and that doesn't support sub-queues, but it was also the only thing that appeared to touch buffer bloat. Sounds like I need to be in the limiters instead - that might be where I went wrong.
CODELQ is under the ALTQ system - which does certainly work, it's just a much more involved setup.
Set the subnet to match your network (probably /24). Down=destination up=source
I'm assuming your talking about a firewall match or pass rule to classify the traffic and assign it to a queue. If I'm using a floating rule I want the interface to be WAN and the Destination to have the Backblaze networks, right? I have an Alias with all the subnets for the BackBlaze servers.
When you make the queues in dummynet there will be an area to enter your subnet size so that it can share traffic between clients, set that to match which probably means setting it to /24, and the Download limiter is "destination" and Upload Limiter is "source".
You will also need to apply your queues to firewall rules.
In order to make sure everything is working and set correctly, I would temporarily set your up and down speeds to something way under your upload speed and set a unique number so that you will easily recognize it on speedtest.
What I mean by that is, if your normal download/up speeds are 40/10, then on dummynet set download to something like 4200Kbps and set upload to something like 650Kbps.
The only point of this is so that if you've accidentally reversed the upload and download queues in your firewall rules you will easily recognize that and fix it when you run a speedtest if you see upload at 4.2Mbps and download at 0.65Mbps. If you already know it's all setup correctly then just skip all that stuff.
I never did get a floating rule to work, but a Pass rule directly on the LAN interface worked with the Backblaze subnet list alias in the Destination section. It's just the wizard configs for traffic shaping didn't seem to touch buffer bloat - latency and overall bandwidth was horrible. But CODELQ only handles buffer bloat wonderfully but I didn't see how to shape the Backblaze traffic.
It sounds like I really need to play with the limiters instead. Thanks again for the hints.
I'm sorry for the convoluted explanation. I'm not near a pfSense box I can access and won't be for awhile. Otherwise I would just give you a screenshot to explain this, it's very easy I'm just trying to explain this from memory of what the config GUI looks like.
-
Patch for Limiter Info page with schedulers information and refresh interval of 500ms
--- diag_limiter_info.php Wed Sep 07 00:26:47 2016 +++ diag_limiter_info.php Sun Oct 01 08:20:33 2017 @@ -40,5 +40,5 @@ echo $text; - $text = `/sbin/ipfw queue show`; + $text = `/sbin/ipfw sched show`; if ($text != "") { - echo "\n\n" . gettext("Queues") . ":\n"; + echo "\n\n" . gettext("Shedulers") . ":\n"; echo $text; @@ -72,3 +76,3 @@ events.push(function() { - setInterval('getlimiteractivity()', 2500); + setInterval('getlimiteractivity()', 500); getlimiteractivity();
-
fq_Codel is a zero-config AQM. All it needs is to be hooked up to a shaper of some sort and and works magic. You really need to understand how to traffic shape to do better than it. Eve then, it's great.
Agreed, it really is very impressive - probably one of the more impressive things I've seen in pfSense.
It's a huge improvement for very little config, and the config you have to do is not complicated even for a non-tech-savvy home user.
Netgate should implement some sort of automatic bandwidth limiting, and place that in the UI next to dummynet using fq_codel. Maybe 2.4.2?
The net result of the above would be that pfSense would dramatically improve the quality of even the crappiest connections from ISP with a sub 5 minute configuration for even the least experienced user.
I will grant you that pfSense can already do that (very, very well) with HFSC and limiting your bandwidth manually to below the lowest values you ever see. But HFSC you have to learn how to do, and as Harvy noted - even if you know what you're doing you will have to spend some time getting it as good as fq_codel can be just by turning it on. The result of that is most people either don't use it or don't use it well.
Also, many WAN connection speeds dip dramatically during peak hours. No one wants to cut their bandwidth down by a large percentage all the time just so their limiter can catch the traffic during peak hours.
Either an automatic speedtest similar to ubiquiti's, or an automatic latency test similar to gargoyle could be leveraged to automatically keep bandwidth limited just below the current WAN speeds so your limiter is always catching the traffic and you are always making the most of your available bandwidth.fq_codel + automatic bandwidth limiter = killer app - huge bullet point for pfSense & Netgate.
Agreed with all you said. They should look into implementing it asap
-
@w0w:
Patch for Limiter Info page with schedulers information and refresh interval of 500ms
--- diag_limiter_info.php Wed Sep 07 00:26:47 2016 +++ diag_limiter_info.php Sun Oct 01 08:20:33 2017 @@ -40,5 +40,5 @@ echo $text; - $text = `/sbin/ipfw queue show`; + $text = `/sbin/ipfw sched show`; if ($text != "") { - echo "\n\n" . gettext("Queues") . ":\n"; + echo "\n\n" . gettext("Shedulers") . ":\n"; echo $text; @@ -72,3 +76,3 @@ events.push(function() { - setInterval('getlimiteractivity()', 2500); + setInterval('getlimiteractivity()', 500); getlimiteractivity();
Would love to try this patch out. This will show fq_codel on the limiter info page? Is there are kind soul who could explain how to implement this to the lay person?
-
There's a redmine feature request to get an automatic bandwidth limiter added to dummynet.
If anyone is interested and technically inclined please chime in!
Check out the links in my signature for more info.
https://redmine.pfsense.org/issues/7904
-
I finally got fq_codel limiters applied to just my WAN connection via floating rules.
From what I am seeing I think I like it better than using my vlan's interfaces. From what I am seeing in my own testing the jitter seems lower and I see fewer latency spikes on my upload bandwidth tests. Also since this is queuing all traffic on the WAN interface I feel like it is handling separate flows better than it did before.
I could be wrong and all of this is anecdotal or a placebo affect from all of my messing around with shappers and limiters.
If anyone is interested in trying it out the setup is fairly easy.
Firewall > Rules > Floating
*Add new rule
*Change "Action" from "Pass" to "Match"
*Select "WAN" in Interface
*Set "Direction" to "Out"
*Set "Protocol" to "any"
*Source to "any"
*Destination to "any"
Advanced settings
*Set Gateway (Cannot leave as default; you have to specifically set it to your configured gateway)
*Set In/Out (Because it is a floating rule and it is set to "Out" it gets a little confusing. It reverses In/Out ie In is for outgoing and Out is for incoming.)
-
dslreports.com has a good bufferbloat test.
-
Would love to try this patch out. This will show fq_codel on the limiter info page? Is there are kind soul who could explain how to implement this to the lay person?
You need "System patches" package.
Create new patch and apply it. See attachment.
-
I got asked in a PM to post some screenshots of my settings.. Figured post it here as reference.
Just apply the in/out pipe to firewall rule on your interface.. So that these do not effect your intervlan traffic if you have any. Put a rule above to allow access to your other vlans without the pipe's applied.
These settings changed my bufferbloat tests on dslreports to A..
-
Why a /32 IPv4 mask?
-
Because that is what comes up in the gui when this is the rules.limiter
[2.4.0-RELEASE][root@pfsense.local.lan]/root: cat /tmp/rules.limiter
pipe 1 config bw 85Mb
queue 1 config pipe 1 mask dst-ip6 /128 dst-ip 0xffffffffpipe 2 config bw 11Mb
queue 2 config pipe 2 mask src-ip6 /128 src-ip 0xffffffffIs something wrong there? It was working great!!!
-
Haha, I don't know to be honest. I had mine set the same way until I noticed that, then set it to /24 to match my network (I'm IPv4 only). I haven't been on that network in awhile now but I don't remember noticing a difference. My config is otherwise pretty much the same as yours.
Maybe someone can chime in on whether that setting matters or not and exactly what it is doing?
I know that in some parts of traffic shaping GUI there are options presented that don't apply to all types of shaping.
-
The person that asked for the screenshot says its working great for him as well..
I just am not knowledgeable enough when it comes to shaping and limiters to know one way or the other either. I understand the basic principles is about all. I just took the settings as given and applied them to my bandwidth at the time and yeah it drastically reduced the bufferbloat test without noticing any serious hit to the top end numbers on speedtest or during normal use.
But to be honest I had not really noticed any issues before that ;) Other than the test showing me my bufferbloat was bad..
Looking forward to when I can apply it to my new 500/50 line when get new pfsense hardware. I can tell you for sure that on the usg that currently stuck with that when you turn on their smart queues my download is limited to 80ish down vs the 530 I see on speedtest currently. Seems to handle the upload ok but the download gets shit on..
-
Yikes, that's pretty limited!
-
Which is why its not on ;) When you turn on their queues you loose the hardware offload it seems.. So yeah speed takes a hit ;)
-
And that is why I am thankful for pfSense!
-
Oh believe me I will be back to pfsense as soon as get new hardware that can handle the speed.. The usg was a temp solution that was cheap enough to sneak through the budget committee (wife).. its was only a 100$ ;)
It can handle the speed in hardware offload.. But its feature set is so lacking.. Still running my pfsense vm for dhcp and dns since those features on usg need a huge amount of work to be viable in anything other than the most basic of home user networks.. And really just forget about ipv6 and or openvpn without manipulate of json files and having to reload them any time you reprovision the usg from the controller.. And the firewall rules are just nuts to setup on it as well.. I counting the days til I have pfsense back that is for sure ;)
-
I ran this on my router at my LAN party and it worked out great. 184 people with a 300mbit modem and 2 100mbit modems , made 2 download shapers and 1 upload shaper.
i made the system patches as well so it would apply after updates.
-
I should skip this since I don't know what I'm doing but still really curious to make it work. I have gigabit service and get D's and F's on buffer bloat.
I'm sure its in the post and I have indeed read though but still don't understand. What are the steps to enable this? I have 2.4 installed.
Looks like install patches package, run patch posted on page 8 which I was going to do until it said I could not remove this so I thought I better study a bit before I keep going. If you have the energy, please tell me what are the steps and I will follow them. Thanks.
-
You don't have to install the patch.
Just set up limiters (look at Johns screenshots a few pages above this) then run the ipfw commands for fq_codel and add them to shellcmd.
Run a speed test and set your limiters to 95% of the speeds you get.
Now go to your firewall rules to pass traffic and in the advanced section just select the queues you just made.
That's it.
-
You don't have to install the patch.
Just set up limiters (look at Johns screenshots a few pages above this) then run the ipfw commands for fq_codel and add them to shellcmd.
Run a speed test and set your limiters to 95% of the speeds you get.
Now go to your firewall rules to pass traffic and in the advanced section just select the queues you just made.
That's it.
I don't think it's that simple. If you don't override rules.limiter with own one like TS suggests by patching php code, then any firewall config or even WAN IP change that wants and would reload this file will destroy your manually configured fq_codel, until you manually run ipfw commands again or restart firewall to let shellcmd to do it. Am I wrong?
-
No sorry it is that simple.. You do not need to make any files changes at all.. Just create the limiters and then put in the commands via shellcmd to put them in every time you reboot, etc.
-
Yeah, I just tried adding and deleting firewall rules then checking ipfw and it still has my fq_codel flows.
If there's some other action you're worried might remove fq_codel then just try doing that action then check ipfw after to see if fq_codel is still in place.
ipfw sched show
-
OK so may be quick start quide?
1. RTFM for FQ_CODEL http://caia.swin.edu.au/freebsd/aqm/patches/README-0.2.1.txt
2. Config limiters (pipes) via GUI.
3. View /tmp/rules.limiterfor example it will be
pipe 1 config bw 280576Kb queue 1 config pipe 1 mask src-ip6 /128 src-ip 0xffffffff pipe 2 config bw 280576Kb queue 2 config pipe 2 mask dst-ip6 /128 dst-ip 0xffffffff
4. USE shellcmd package to recreate pipes with commands like
ipfw pipe flush ipfw pipe 1 config bw 280576Kb ipfw sched 1 config pipe 1 type fq_codel target 7ms quantum 2000 flows 2048 ipfw queue 1 config pipe 1 mask src-ip6 /128 src-ip 0xffffffff ipfw pipe 2 config bw 280576Kb ipfw sched 2 config pipe 2 type fq_codel target 7ms quantum 2000 flows 2048 ipfw queue 2 config pipe 2 mask dst-ip6 /128 dst-ip 0xffffffff
5. Add your limiters to firewall rules (IN/OUT pipes), this step can be any after step 2 actually.
Is it correct?
Maybe it's better to run script at startup? Just placing it into /usr/local/etc/rc.d? I found that using shellcmd is a little bit uncomfortable with multiple command lines at once, have I missed something? -
Excuse my ignorance on this. I've just learned about and started using pfSense a couple weeks ago.
I have my limiters attached to my "Default allow LAN to any rule" in order to evenly split bandwidth to my LAN clients. And then fq_codel applied to those limiters. Seems to be working great for reducing bufferbloat, ensuring low latency for all clients, etc. Thanks for all the guidance in this thread!
Is there any benefit or harm to doing it that way vs. attaching the limiters to a floating rule as @johnpoz did?
Also, how does all this apply to OpenVPN clients (with pfSense as the server)? Would either setup also work with the OpenVPN clients, or is one setup better than the other?
Thanks for all your help!
-
Floating rules vs interface rules won't make a difference. It will also work well on VPN clients. VPN traffic will always have higher latency relative to the same traffic not routed through a VPN. fq_codel can't fix that, but it will still work with fairly queuing the traffic and reducing bufferbloat.
-
I came back here to say thanks because it works well. I completed my setup differently than some of what has just been posted.
I setup limiters just as seen in the screenshots. (post 121)(upload, download, wan, lan)
I ran the single command for IPFW pipes. (ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2)
I installed shellcmd and added the single IPFW statement.
Modified the two stock LAN firewall rules (IPV4 and IPV6 advanced configuration) so that wan and lan would be used just as seen in the screenshots.
I restarted the firewall.That is all I have done. Prior my buffer bloat was a D to F. Post I get an A each time. I may/may not be setup correctly but whatever it is works. I originally used the wizard for setup of traffic shaping which used HFSC and which gave @425 upload on my gigabit connection. This new setup gives @750. So, good for me.
-
Definitely I am blind what screenshots are you all talking about? :D
-
@w0w:
Definitely I am blind what screenshots are you all talking about? :D
Reply 121 of this thread.
-
Thanks. :)
-
Floating rules vs interface rules won't make a difference. It will also work well on VPN clients. VPN traffic will always have higher latency relative to the same traffic not routed through a VPN. fq_codel can't fix that, but it will still work with fairly queuing the traffic and reducing bufferbloat.
I tested floating rules vs. lan rules and they both give excellent results. Latency results in bufferbloat tests seemed to be just slightly lower with the lan rules, but that's just splitting hairs.
I had very poor bufferbloat results when testing through my OpenVPN connection as a client connected to the OpenVPN server in pfSense. Is there any way to fix this? Should I be creating limiters to apply to the OpenVPN interface rules in the firewall and then selecting fq_codel on those limiters, as well?
-
Yes you would need to apply limiters to your openvpn interface in order to queue your clients traffic. However, you can only fix your end, if the client is connecting to you via a poor connection then you can't get any better than the worst link.
-
Yes you would need to apply limiters to your openvpn interface in order to queue your clients traffic. However, you can only fix your end, if the client is connecting to you via a poor connection then you can't get any better than the worst link.
Thanks, that makes sense.
I’ll try it out and see how much it helps. -
Finally got around to trying this again, and everything worked great! John's screenshots in reply 121 are spot on and there is no need to edit any files if one uses shellcmd.
I actually recently changed to a 100/100 Fiber connection - here are results (using the DSL Reports speed test which has a nice Bufferbloat check):
Before (no shaping):
Using ALTQ FAIRQ + Codel Active Queue Management; 100Mbit Limit on Both WAN and LAN:
Using fq_codel and 100Mbit Limit on Both Upload and Download:
What's interesting to me here is that fq_codel appears to perform a bit better than the ALTQ emulation of fq_codel (using FAIRQ + Codel) - I find this very interesting. Anyone have any thoughts as to why?
I also ran a more intense FLENT test on another system with fq_codel enabled and the results looked great as well (stable ping and stable download/upload over the course of the test).
Given the relatively little effort required to get this to work on pfSense, it's a fantastic way to improve the stability of a connection.
-
Finally got around to trying this again, and everything worked great! John's screenshots in reply 121 are spot on and there is no need to edit any files if one uses shellcmd.
I actually recently changed to a 100/100 Fiber connection - here are results (using the DSL Reports speed test which has a nice Bufferbloat check):
Before (no shaping):
Using ALTQ FAIRQ + Codel Active Queue Management; 100Mbit Limit on Both WAN and LAN:
Using fq_codel and 100Mbit Limit on Both Upload and Download:
What's interesting to me here is that fq_codel appears to perform a bit better than the ALTQ emulation of fq_codel (using FAIRQ + Codel) - I find this very interesting. Anyone have any thoughts as to why?
I also ran a more intense FLENT test on another system with fq_codel enabled and the results looked great as well (stable ping and stable download/upload over the course of the test).
Given the relatively little effort required to get this to work on pfSense, it's a fantastic way to improve the stability of a connection.
As I understand it, the biggest difference between FAIRQ + CoDel and fq_codel is that fq_codel individually applies codel to each per-flow pseudo-queue while FAIRQ + CoDel applies codel to the entire queue. There are also other subtle differences between codel and fq_codel, like the "fq" in fq_codel being a bit smarter than standard "fair queueing".
Either way, the 4ms difference you observed in best-case latency could just be a fluke.
Thanks for sharing the comparisons, btw.
-
I really don't get much difference. I was using OPNSense and fq_codel prior as it seemed to just work better for me.
With the new release, I changed back and just use HFSC queues with codel checked and some very basic rules to make sure my gaming traffic is first and my non important (downloads for media and other odd plex related download stuff) is limited. Works like a champ.
Only thing for me always comes back to making sure my upload and download limits match close to reality what I expect out of my link so I use 940 down and 880 on Verizon's Gigabit FIOS with 1000 queue. No drops and no bufferbloat that I've been able to make happen.
-
Thanks all for the feedback. i do have a quick follow up question as I think that I may have misconfigured something:
I actually ended up creating two limiters, one at 100Mbit up/down, the other at 25Mbit up/down to use on a guest network. Went through the same process and enabled fq_codel on the second set of limiters. Applied the limiters inside the firewall rules on the guest network, but for some reason when I try to test out the configuration with a machine on the guest network I'm able to go faster than the limited speed of 25Mbit. However, the interesting thing is that does not seem to be consistent - for instance:
- When running a speedtest on speedtest.net I'm limited to just 25Mbit (as expected)
- When running a speedtest on DSLReports I'm able to go well beyond 25Mbit (almost to full speed).
I haven't been able to try an iperf3 test yet unfortunately. Could it be that something is misconfigured and that the 25Mbit limit is applied per flow vs. the queue as a whole?
Thanks in advance for any insight you might have.
P.S. Some thoughts regarding fq_codel vs. FAIRQ + Codel: At least in my case, using fq_codel consistently results in a bufferbloat average (for both upload/download) under 10ms. Using FAIRQ + Codel it often goes beyond that, but never higher than 15-20ms. Ultimately, I suppose it's not really a big deal, but I found it interesting nonetheless.
-
Thanks all for the feedback. i do have a quick follow up question as I think that I may have misconfigured something:
I actually ended up creating two limiters, one at 100Mbit up/down, the other at 25Mbit up/down to use on a guest network. Went through the same process and enabled fq_codel on the second set of limiters. Applied the limiters inside the firewall rules on the guest network, but for some reason when I try to test out the configuration with a machine on the guest network I'm able to go faster than the limited speed of 25Mbit. However, the interesting thing is that does not seem to be consistent - for instance:
- When running a speedtest on speedtest.net I'm limited to just 25Mbit (as expected)
- When running a speedtest on DSLReports I'm able to go well beyond 25Mbit (almost to full speed).
I haven't been able to try an iperf3 test yet unfortunately. Could it be that something is misconfigured and that the 25Mbit limit is applied per flow vs. the queue as a whole?
Thanks in advance for any insight you might have.
P.S. Some thoughts regarding fq_codel vs. FAIRQ + Codel: At least in my case, using fq_codel consistently results in a bufferbloat average (for both upload/download) under 10ms. Using FAIRQ + Codel it often goes beyond that, but never higher than 15-20ms. Ultimately, I suppose it's not really a big deal, but I found it interesting nonetheless.
Looks like the issue I was experiencing has to do with the Squid Proxy running on the guest network. Similar to what was described here:
https://forum.pfsense.org/index.php?topic=132960.0
I'll go ahead and start a separate thread as I may need some help configuring the proper rules to get this work.