Playing with fq_codel in 2.4

jasonraymundo31

@Pentangle said in Playing with fq_codel in 2.4:

@ohbobva I'm running dual WAN with different speeds, and all you need to do is duplicate your setup. i.e. create four different limiters - upstream WAN1, upstream WAN2, downstream WAN1, downstream WAN2, and then a queue within each of those limiters. Set up four floating rules for WAN1-IPv4, WAN1-IPv6, WAN2-IPv4, WAN2-IPv6 and it all works as advertised. Be careful not to just create two queues within the same limiter as the limit will be the aggregate bandwidth of the two.

I'm having hard time to follow you, do you mind taking a screenshot of your config ? Thanks in advance, very much appreciated.

provels

@uptownVagrant said in Playing with fq_codel in 2.4:

Thanks very much for your quick reply ;)

@provels without the limiter rules enabled, and your filters reloaded and old states killed, what speeds are you seeing when you run the dslreports.com/speedtest for 60 seconds? It could be that Comcast's burst capacity during short tests may be heavily skewing your results - I seem to remember the default test duration only being 20 seconds. I think you may have to create an account and go into the preferences of the speed test to increase upload and download duration to 60s. There's always the flent project if you're savvy.

Did as you suggested, I get results as follows. Looked at flent, but I'm Windows. My upload is no issue, with or without limiters, but the download bloat is normally in the 60-90ms range.

179/24 Mbps
189/23
169/24

This result is pretty typical.

This is a typical result using the Ookla Speed Test (client version)
>

Another thought, depending on how close your current limiter config is to the Comcast circuit sustained cap, you may be running into some buffering on your cable modem. What DOCSIS version is your cable modem operating?

Modem is a Netgear CM600, DOCSIS 3.0, 24x8, Broadcom

I'm assuming you're testing with a computer attached to your router/switch with an Ethernet cable and not using Wi-Fi.

Yes, testing wired, but my pfSense is a VM running in Hyper-V and I'm testing from the host Desktop session. Maybe I'll wire up my laptop.

You could try halving your advertised values in your limiters and then increasing from there after each test.

I'll start by halving my DL BW and add from there.

Note, if you're using FQ_CoDel then the queue length that you set in the limiter config is not being used. FQ_CoDel manages queue lengths dynamically and works to dequeue packets within the target you set.

Noted and deleted.

Here's a guide that might help although I would suggest leaving target, interval, quantum, limit, and flows set as the pfSense defaults for the initial tests.:
https://forum.netgate.com/topic/112527/playing-with-fq_codel-in-2-4/815

Best of luck

Thanks for the tips. I'll check out the guide and report back. Thanks again! :)

cwagz

I can start a new thread if more appropriate...

What is the best way to prioritize WiFi Calling traffic while using an FQ_CoDel limiter setup?

I have a setup with 30 - 80 WiFi clients. Currently using limiters and FQ_CoDel which seems to share the bandwidth very nicely. We have been having problems with WiFi Calling not being super reliable. We are in a cellular dead zone so people are relying on it.

I was able to improve WiFi calling reliability by changing the Firewall Optimization to Conservative and changing the outbound NAT mode to Automatic rule generation.

I would like to further optimize by making sure all the WiFi calling traffic has top priority.

dtaht

This really is the thread from hell isn't it? I'd do a new thread.

Over on linux (and not bsd as yet), we did this:

https://www.usenix.org/system/files/conference/atc17/atc17-hoiland-jorgensen.pdf

Skip to the MOS score at the end, and work back. There's support now for intel, qca, and mediatek chips. However... if you can get the clients to dscp mark for the VO or VI queue for how your AP defines it, that helps in that direction, and
it is generally possible to build a more complicated qos/sqm setup that explicitly prioritizes voip out of the ipfw tools.

In general I'm a big believer in short (fair) queues and lots of 'em, and not huge on prioritization. sch_cake (also mentioned on this thread), has some built-in optimizations as well.

cwagz

This post is deleted!

cwagz

@dtaht said in Playing with fq_codel in 2.4:

This really is the thread from hell isn't it? I'd do a new thread.

Over on linux (and not bsd as yet), we did this:

https://www.usenix.org/system/files/conference/atc17/atc17-hoiland-jorgensen.pdf

Skip to the MOS score at the end, and work back. There's support now for intel, qca, and mediatek chips. However... if you can get the clients to dscp mark for the VO or VI queue for how your AP defines it, that helps in that direction, and
it is generally possible to build a more complicated qos/sqm setup that explicitly prioritizes voip out of the ipfw tools.

In general I'm a big believer in short (fair) queues and lots of 'em, and not huge on prioritization. sch_cake (also mentioned on this thread), has some built-in optimizations as well.

Thank you, dthat. I will look at this information.

New thread:
https://forum.netgate.com/topic/145924/prioritize-wifi-calling-traffic-and-fq_codel

provels

After limiters and rules are in effect and states have been initially cleared, is it necessary to dump the states every time a modification is made to the limiters during testing? Thanks.

xciter327

@provels said in Playing with fq_codel in 2.4:

After limiters and rules are in effect and states have been initially cleared, is it necessary to dump the states every time a modification is made to the limiters during testing? Thanks.

I clear them every time. Also monitor syslog to check if there are any errors.

wgstarks

I keep seeing these errors in my system log-

fq_codel_enqueue maxidx = 797
fq_codel_enqueue over limit
fq_codel_enqueue maxidx = 797
fq_codel_enqueue over limit
fq_codel_enqueue maxidx = 797
fq_codel_enqueue over limit
fq_codel_enqueue maxidx = 797
fq_codel_enqueue over limit
fq_codel_enqueue maxidx = 797

They seem to show at random times maybe 2 or 3 times a week and will repeat for a few minutes before stopping. Not sure if they represent a problem or how to fix them? Could definitely use any advice.

subzerogts

Hi guys, I'm new here and just followed the YouTube video from Netgate on how to set up fq codel with limiters. But then browsing this thread, I see people saying those settings are wrong and to use 'Tail Drop' as the queue management algorithm? I'm confused, why?

Also, any recommendation on settings I should use? I have a gigabit verizon fios connection. It's hit 940/900 on Verizon's test if I recall correctly. Actually the upload has gotten over 900 a few times, the download is usually 500-700 on normal speed test sites except Verizon's where it can hit 900+ (and dslreports which has gotten it to 800+ a few times). I have a Plex Media Server and a gaming machine on the network.

Using the settings from the YouTube vid got my Bufferbloat score to go from A to A+ in dslreports' test.

provels

@subzerogts If you are at A+, I don't expect you have much to gain, LOL. I followed the video and also tried the tail drop config. Nothing seems to help me. Bloat is a B/C, Quality is A/B and Speed is D/F, even though the Ookla speedtest client gets me my advertised 300/25 to various local servers (even when limiting the Down limiter to 150. Nonsensical. Completely random speedtest-cli results testing from an inside server through pfSense and at the DSLReports test page. I don't know if maybe it's Comcast, my modem or what.

provels

If I want to disable the limiters, I know there are checkboxes for that. But if I just disable the floating rules and reset states, that gets them out of the system anyway, right?

tman222

@wgstarks said in Playing with fq_codel in 2.4:

I keep seeing these errors in my system log-
fq_codel_enqueue maxidx = 797
fq_codel_enqueue over limit
fq_codel_enqueue maxidx = 797
fq_codel_enqueue over limit
fq_codel_enqueue maxidx = 797
fq_codel_enqueue over limit
fq_codel_enqueue maxidx = 797
fq_codel_enqueue over limit
fq_codel_enqueue maxidx = 797
They seem to show at random times maybe 2 or 3 times a week and will repeat for a few minutes before stopping. Not sure if they represent a problem or how to fix them? Could definitely use any advice.

@wgstarks - what are your fq-codel parameters set to? One thing you might try is increasing the the value for the limit parameter. Here is a a link to some good documentation on what each parameter does:

http://caia.swin.edu.au/freebsd/aqm/downloads.html

Hope this helps.

tman222

@provels said in Playing with fq_codel in 2.4:

@subzerogts If you are at A+, I don't expect you have much to gain, LOL. I followed the video and also tried the tail drop config. Nothing seems to help me. Bloat is a B/C, Quality is A/B and Speed is D/F, even though the Ookla speedtest client gets me my advertised 300/25 to various local servers (even when limiting the Down limiter to 150. Nonsensical. Completely random speedtest-cli results testing from an inside server through pfSense and at the DSLReports test page. I don't know if maybe it's Comcast, my modem or what.

@provels - how do the results change if you reduce the number of parallel streams during the DSL Reports speed test? Also, have you tried changing / tuning any of the fq-codel parameters?

Something else I thought of: I admit that I don't know a lot about cable modems, but is buying a DOCSIS 3.1 cable modem an option? As far as I can tell DOCSIS 3.1 includes AQM part of the specification:

https://tools.ietf.org/html/rfc8034

Hope this helps.

tman222

@subzerogts said in Playing with fq_codel in 2.4:

Hi guys, I'm new here and just followed the YouTube video from Netgate on how to set up fq codel with limiters. But then browsing this thread, I see people saying those settings are wrong and to use 'Tail Drop' as the queue management algorithm? I'm confused, why?

Also, any recommendation on settings I should use? I have a gigabit verizon fios connection. It's hit 940/900 on Verizon's test if I recall correctly. Actually the upload has gotten over 900 a few times, the download is usually 500-700 on normal speed test sites except Verizon's where it can hit 900+ (and dslreports which has gotten it to 800+ a few times). I have a Plex Media Server and a gaming machine on the network.

Using the settings from the YouTube vid got my Bufferbloat score to go from A to A+ in dslreports' test.

When you select FQ_CODEL as scheduler you don't have to worry about setting up queue management algorithms for the queues because fq-codel manages per flow created packet queues automatically as part of fq-codel algorithm. This is why you don't really see any difference in performance if you e.g. select Codel as the queue management algorithm if you have FQ_CODEL selected.

Having said that, fq-codel is a bit unique in this regard (in that it also manages per flow queues vs. just scheduling packets). In a way you can kind of think of fq-codel as a hybrid or all-in one packet scheduler and queue management algorithm. However, this is not the case for some of the other algorithms you have available to choose from as well as schedulers (e.g. QFQ, Round Robin, etc.). These are actually just scheduling algorithms that determine how packets should be dequeued from one more packet queues (e.g. Round Robin, weighted, etc.) When using one of these scheduling-only algorithms, you'll still have to setup packet queues and then choose an algorithm to manage those queues. In such as scenario, choosing a queue management algorithm will be important and will make a difference in performance.

Hope this helps.

provels

@tman222 BB goes to A and Speed tanks when I reduced DL streams from 24 to 6 or 12. Already bought the new Netgear modem after Comcast bitched about my Motorola for 2 years. Screw it. I'll monitor thread, but for now, I'm off the program.
N

tman222

@provels said in Playing with fq_codel in 2.4:

@tman222 BB goes to A and Speed tanks when I reduced DL streams from 24 to 6 or 12. Already bought the new Netgear modem after Comcast bitched about my Motorola for 2 years. Screw it. I'll monitor thread, but for now, I'm off the program.
N

Thanks @provels for getting back to me. If you don't mind me asking, how are you testing? Are the results consistent across different machines and browsers?

provels

@tman222 My pfSense is a VM on Hyper-V and I'm testing from the host through the 10Gb Hyper-V interfaces through an Intel I340 Gb card hosting the v-switches via Cat6 to the modem. It's the only wired machine I have and no diff with other browsers. Results are completely random. Thanks for the help, but I'm not thinking it's worth the effort.

wgstarks

@tman222 said in Playing with fq_codel in 2.4:

@wgstarks - what are your fq-codel parameters set to? One thing you might try is increasing the the value for the limit parameter. Here is a a link to some good documentation on what each parameter does:
http://caia.swin.edu.au/freebsd/aqm/downloads.html
Hope this helps.

Limiters:
00001:  25.000 Mbit/s    0 ms burst 0 
q131073  50 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0 droptail
 sched 65537 type FIFO flags 0x0 0 buckets 0 active
00002: 400.000 Mbit/s    0 ms burst 0 
q131074  50 sl. 0 flows (1 buckets) sched 65538 weight 0 lmax 0 pri 0 droptail
 sched 65538 type FIFO flags 0x0 0 buckets 0 active


Schedulers:
00001:  25.000 Mbit/s    0 ms burst 0 
q65537  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 NoECN
   Children flowsets: 1 
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0        1     1390  0    0   0
00002: 400.000 Mbit/s    0 ms burst 0 
q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 NoECN
   Children flowsets: 2 
  0 ip           0.0.0.0/0             0.0.0.0/0        1       90  0    0   0


Queues:
q00001  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
q00002  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail

These settings were based on speedtest results.
Thanks for the link. I'll check it out.

wgstarks

@tman222 said in Playing with fq_codel in 2.4:

@wgstarks - what are your fq-codel parameters set to? One thing you might try is increasing the the value for the limit parameter.

The limit was set at default 10240 packets. I increased that to 10340, but I'm wondering if that is too small to make any difference. Should I try a larger increase?