Playing with fq_codel in 2.4
-
I keep seeing these errors in my system log-
fq_codel_enqueue maxidx = 797 fq_codel_enqueue over limit fq_codel_enqueue maxidx = 797 fq_codel_enqueue over limit fq_codel_enqueue maxidx = 797 fq_codel_enqueue over limit fq_codel_enqueue maxidx = 797 fq_codel_enqueue over limit fq_codel_enqueue maxidx = 797They seem to show at random times maybe 2 or 3 times a week and will repeat for a few minutes before stopping. Not sure if they represent a problem or how to fix them? Could definitely use any advice.
-
Hi guys, I'm new here and just followed the YouTube video from Netgate on how to set up fq codel with limiters. But then browsing this thread, I see people saying those settings are wrong and to use 'Tail Drop' as the queue management algorithm? I'm confused, why?
Also, any recommendation on settings I should use? I have a gigabit verizon fios connection. It's hit 940/900 on Verizon's test if I recall correctly. Actually the upload has gotten over 900 a few times, the download is usually 500-700 on normal speed test sites except Verizon's where it can hit 900+ (and dslreports which has gotten it to 800+ a few times). I have a Plex Media Server and a gaming machine on the network.
Using the settings from the YouTube vid got my Bufferbloat score to go from A to A+ in dslreports' test.
-
@subzerogts If you are at A+, I don't expect you have much to gain, LOL. I followed the video and also tried the tail drop config. Nothing seems to help me. Bloat is a B/C, Quality is A/B and Speed is D/F, even though the Ookla speedtest client gets me my advertised 300/25 to various local servers (even when limiting the Down limiter to 150. Nonsensical. Completely random speedtest-cli results testing from an inside server through pfSense and at the DSLReports test page. I don't know if maybe it's Comcast, my modem or what.
-
If I want to disable the limiters, I know there are checkboxes for that. But if I just disable the floating rules and reset states, that gets them out of the system anyway, right?
-
@wgstarks said in Playing with fq_codel in 2.4:
I keep seeing these errors in my system log-
fq_codel_enqueue maxidx = 797 fq_codel_enqueue over limit fq_codel_enqueue maxidx = 797 fq_codel_enqueue over limit fq_codel_enqueue maxidx = 797 fq_codel_enqueue over limit fq_codel_enqueue maxidx = 797 fq_codel_enqueue over limit fq_codel_enqueue maxidx = 797They seem to show at random times maybe 2 or 3 times a week and will repeat for a few minutes before stopping. Not sure if they represent a problem or how to fix them? Could definitely use any advice.
@wgstarks - what are your fq-codel parameters set to? One thing you might try is increasing the the value for the limit parameter. Here is a a link to some good documentation on what each parameter does:
http://caia.swin.edu.au/freebsd/aqm/downloads.html
Hope this helps.
-
@provels said in Playing with fq_codel in 2.4:
@subzerogts If you are at A+, I don't expect you have much to gain, LOL. I followed the video and also tried the tail drop config. Nothing seems to help me. Bloat is a B/C, Quality is A/B and Speed is D/F, even though the Ookla speedtest client gets me my advertised 300/25 to various local servers (even when limiting the Down limiter to 150. Nonsensical. Completely random speedtest-cli results testing from an inside server through pfSense and at the DSLReports test page. I don't know if maybe it's Comcast, my modem or what.
@provels - how do the results change if you reduce the number of parallel streams during the DSL Reports speed test? Also, have you tried changing / tuning any of the fq-codel parameters?
Something else I thought of: I admit that I don't know a lot about cable modems, but is buying a DOCSIS 3.1 cable modem an option? As far as I can tell DOCSIS 3.1 includes AQM part of the specification:
https://tools.ietf.org/html/rfc8034
Hope this helps.
-
@subzerogts said in Playing with fq_codel in 2.4:
Hi guys, I'm new here and just followed the YouTube video from Netgate on how to set up fq codel with limiters. But then browsing this thread, I see people saying those settings are wrong and to use 'Tail Drop' as the queue management algorithm? I'm confused, why?
Also, any recommendation on settings I should use? I have a gigabit verizon fios connection. It's hit 940/900 on Verizon's test if I recall correctly. Actually the upload has gotten over 900 a few times, the download is usually 500-700 on normal speed test sites except Verizon's where it can hit 900+ (and dslreports which has gotten it to 800+ a few times). I have a Plex Media Server and a gaming machine on the network.
Using the settings from the YouTube vid got my Bufferbloat score to go from A to A+ in dslreports' test.
When you select FQ_CODEL as scheduler you don't have to worry about setting up queue management algorithms for the queues because fq-codel manages per flow created packet queues automatically as part of fq-codel algorithm. This is why you don't really see any difference in performance if you e.g. select Codel as the queue management algorithm if you have FQ_CODEL selected.
Having said that, fq-codel is a bit unique in this regard (in that it also manages per flow queues vs. just scheduling packets). In a way you can kind of think of fq-codel as a hybrid or all-in one packet scheduler and queue management algorithm. However, this is not the case for some of the other algorithms you have available to choose from as well as schedulers (e.g. QFQ, Round Robin, etc.). These are actually just scheduling algorithms that determine how packets should be dequeued from one more packet queues (e.g. Round Robin, weighted, etc.) When using one of these scheduling-only algorithms, you'll still have to setup packet queues and then choose an algorithm to manage those queues. In such as scenario, choosing a queue management algorithm will be important and will make a difference in performance.
Hope this helps.
-
@tman222 BB goes to A and Speed tanks when I reduced DL streams from 24 to 6 or 12. Already bought the new Netgear modem after Comcast bitched about my Motorola for 2 years. Screw it. I'll monitor thread, but for now, I'm off the program.
N
-
@provels said in Playing with fq_codel in 2.4:
@tman222 BB goes to A and Speed tanks when I reduced DL streams from 24 to 6 or 12. Already bought the new Netgear modem after Comcast bitched about my Motorola for 2 years. Screw it. I'll monitor thread, but for now, I'm off the program.
NThanks @provels for getting back to me. If you don't mind me asking, how are you testing? Are the results consistent across different machines and browsers?
-
@tman222 My pfSense is a VM on Hyper-V and I'm testing from the host through the 10Gb Hyper-V interfaces through an Intel I340 Gb card hosting the v-switches via Cat6 to the modem. It's the only wired machine I have and no diff with other browsers. Results are completely random. Thanks for the help, but I'm not thinking it's worth the effort.
-
@tman222 said in Playing with fq_codel in 2.4:
@wgstarks - what are your fq-codel parameters set to? One thing you might try is increasing the the value for the limit parameter. Here is a a link to some good documentation on what each parameter does:
http://caia.swin.edu.au/freebsd/aqm/downloads.html
Hope this helps.Limiters: 00001: 25.000 Mbit/s 0 ms burst 0 q131073 50 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0 droptail sched 65537 type FIFO flags 0x0 0 buckets 0 active 00002: 400.000 Mbit/s 0 ms burst 0 q131074 50 sl. 0 flows (1 buckets) sched 65538 weight 0 lmax 0 pri 0 droptail sched 65538 type FIFO flags 0x0 0 buckets 0 active Schedulers: 00001: 25.000 Mbit/s 0 ms burst 0 q65537 50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 NoECN Children flowsets: 1 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 ip 0.0.0.0/0 0.0.0.0/0 1 1390 0 0 0 00002: 400.000 Mbit/s 0 ms burst 0 q65538 50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 NoECN Children flowsets: 2 0 ip 0.0.0.0/0 0.0.0.0/0 1 90 0 0 0 Queues: q00001 50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail q00002 50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptailThese settings were based on speedtest results.
Thanks for the link. I'll check it out. -
@tman222 said in Playing with fq_codel in 2.4:
@wgstarks - what are your fq-codel parameters set to? One thing you might try is increasing the the value for the limit parameter.
The limit was set at default 10240 packets. I increased that to 10340, but I'm wondering if that is too small to make any difference. Should I try a larger increase?
-
@wgstarks said in Playing with fq_codel in 2.4:
@tman222 said in Playing with fq_codel in 2.4:
@wgstarks - what are your fq-codel parameters set to? One thing you might try is increasing the the value for the limit parameter.
The limit was set at default 10240 packets. I increased that to 10340, but I'm wondering if that is too small to make any difference. Should I try a larger increase?
When I saw these messages I ended up doubling the limit value from 10240 to 20480. That might be over-compensating somewhat, but thankfully I have not had any issues since. Hope this helps.
-
@jasonraymundo31 I'll give it a try.
You can see what I meant regarding having different limiters per WAN connection, and a single queue inside each limiter. In the second picture you can also see the use of a floating rule per IPv4 or IPv6 version of each WAN connection. In this instance my IPv6 is provided by Hurricane Electric, and is relatively irrelevant in this matter as it's so rarely used.
-
Have applied settings @uptownVagrant described in post of 27Nov. Running a traceroute (on iMac) I get unexpected results as others have posted.
traceroute google.com traceroute to google.com (172.217.5.110), 64 hops max, 52 byte packets 1 pfsense.firewall.localdomain (192.168.10.1) 0.531 ms 0.247 ms 0.224 ms 2 sfo03s07-in-f110.1e100.net (172.217.5.110) 0.942 ms 0.838 ms 0.906 ms 3 sfo03s07-in-f110.1e100.net (172.217.5.110) 5.972 ms 9.392 ms 7.845 ms <snip> 11 sfo03s07-in-f110.1e100.net (172.217.5.110) 9.272 ms 8.283 ms 8.661 msWith Floating Rules disabled it works normally
traceroute google.com traceroute to google.com (172.217.5.110), 64 hops max, 52 byte packets 1 pfsense.firewall.localdomain (192.168.10.1) 0.389 ms 0.156 ms 0.243 ms 2 192.168.1.254 (192.168.1.254) 0.815 ms 0.810 ms 0.733 ms 3 <snip> 9 * * * 10 108.170.237.106 (108.170.237.106) 8.826 ms 72.14.235.2 (72.14.235.2) 9.178 ms 74.125.252.150 (74.125.252.150) 8.790 ms 11 108.170.236.61 (108.170.236.61) 8.752 ms sfo03s07-in-f110.1e100.net (172.217.5.110) 8.728 ms 108.170.236.61 (108.170.236.61) 8.469 msI think my limiters & rules are the same, EXCEPT I use pfBlockerNG and it has rules at the TOP of Floating.
Limiter:
Limiters: 00001: 838.000 Mbit/s 0 ms burst 0 q131073 50 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0 droptail sched 65537 type FIFO flags 0x0 0 buckets 0 active 00002: 910.000 Mbit/s 0 ms burst 0 q131074 50 sl. 0 flows (1 buckets) sched 65538 weight 0 lmax 0 pri 0 droptail sched 65538 type FIFO flags 0x0 0 buckets 0 active Schedulers: 00001: 838.000 Mbit/s 0 ms burst 0 q65537 50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail sched 1 type FQ_CODEL flags 0x0 0 buckets 0 active FQ_CODEL target 5ms interval 100ms quantum 300 limit 10240 flows 4096 NoECN Children flowsets: 1 00002: 910.000 Mbit/s 0 ms burst 0 q65538 50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail sched 2 type FQ_CODEL flags 0x0 0 buckets 0 active FQ_CODEL target 5ms interval 100ms quantum 300 limit 10240 flows 4096 NoECN Children flowsets: 2 Queues: q00001 50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail q00002 50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptailpfctl -vvsr | grep “Codel”
@124(1566879036) pass out quick on igb0 reply-to (igb0 x.x.x.1) inet proto icmp all icmp-type trace keep state label "USER_RULE: work around for fq_Codel limiter"
@125(1566882242) pass quick on igb0 inet proto icmp all icmp-type echorep keep state label "USER_RULE: work around for fq_Codel limiter"
@126(1566882242) pass quick on igb0 inet proto icmp all icmp-type echoreq keep state label "USER_RULE: work around for fq_Codel limiter"
@127(1566882594) match in on igb0 inet all label "USER_RULE: No Improvement in Buffer Bloat: WAN in Codel limi..." dnqueue(1, 2)
@128(1566795208) match out on igb0 inet all label "USER_RULE: No Improvement in Buffer Bloat: WAN out Codel lim..." dnqueue(2, 1)/tmp/rules.limiter
pipe 1 config bw 838Mb droptail sched 1 config pipe 1 type fq_codel target 5ms interval 100ms quantum 300 limit 10240 flows 4096 noecn queue 1 config pipe 1 droptail pipe 2 config bw 910Mb droptail sched 2 config pipe 2 type fq_codel target 5ms interval 100ms quantum 300 limit 10240 flows 4096 noecn queue 2 config pipe 2 droptailAny ideas of why I still have incorrect traceroute?
-
@JonH I have the same problem, and after reading 600 more posts in this topic, I believe I have the answer for you.
You're using a traceroute that uses UDP by default, and you're shaping TCP and UDP, and this is a bug in pfSense.
You can work around it by using ICMP for traceroutes, e.g. (disclaimer: I'm using Linux):
alias traceroute='traceroute -I'Some here have mentioned that you may also be able to work around it by applying the limits on LAN rules, not floating rules, however the alias is good enough for me for now, so I stopped reading at around 600 posts and can't show you what to do there :)
-
@forbiddenlake Thanks for this info. I backed out of fq_codel a couple of months ago but may revisit it using the info you provided.
-
What I don't understand is that with no limiters and QOS disciplines not enabled, my traceroutes are still not working. Where else could there be an issue? I don't need qos now with gigabit fiber.
-
@forbiddenlake said in Playing with fq_codel in 2.4:
@JonH I have the same problem, and after reading 600 more posts in this topic, I believe I have the answer for you.
You're using a traceroute that uses UDP by default, and you're shaping TCP and UDP, and this is a bug in pfSense.
You can work around it by using ICMP for traceroutes, e.g. (disclaimer: I'm using Linux):
alias traceroute='traceroute -I'Some here have mentioned that you may also be able to work around it by applying the limits on LAN rules, not floating rules, however the alias is good enough for me for now, so I stopped reading at around 600 posts and can't show you what to do there :)
Hi, you have details of this bug? thanks.
-
@chrcoluk I believe this "bug" is being referenced. Certain configuration will cause pfSense to not decrease the TTL when forwarding. Policy routing is used with direction=out limiters, so it's a common cause of the behavior folks are seeing in this thread where traceroute doesn't work. If you're using configuration similar to this, there is a provision for ICMP traceroute but if you are using a traceroute program that is using UDP packets then those packets would use policy routing and the TTL would not decrease on those packets at pfSense.
