Playing with fq_codel in 2.4

[A Former User]

@kripz:

Anybody know why dslreports doenst how bufferbloat? It never gives me a score regardless of settings on that website.

It depends on where you are.  If you live in Australia or NZ, it doesn't seem to work for us (it gives an error and doesn't repot anything)

[dennypage]

@w0w:

dennypage
The best way to check is it related to pfSense is to downgrade and re-test it again. You can also try to lower your bandwidth limits or play with KPTI option.
As for me, nothing changed in bufferbloat, but I have over provisioned hardware.

KPTI doesn’t affect it. Wouldn’t expect it to as limiters are kernel code. I’ve tested with half available bandwidth, and still encounter the bloat issue. I don’t think I’m under provisioned with hardware -– I have a 4860 running a 150/20 connection.

I’m assuming that this is the result of a FreeBSD kernel change. I haven’t downgraded and run a test to confirm yet, but hat’s about what I’m down to.

I’m taking it that you haven’t seen any change with a multichannel http test?

[Gatekeeper-ZA]

I have been having endless years of shaping due to out poor LTE service in South Africa.
Just want to Say THANKS to all for this guide. I although did add a little twist. am still running
HSFC traffic shaper with Limiters children in addition to queues from the traffic shaper I can
still manage my ports as before just give them a nice hard piece of codel if needed.
due to speeds in my area fluctuating alot the schedules works wonders to "dynamically"
change the pipe has not brought ping down from 100 to +- 30 - 43ms which is huge.
another thing is that when using the traffic shaper with fq_codel I can still watch netflix / youtube
full hd no buffering while gaming @ acceptable latency.

Tests were done while streaming netflix minimal impact my girlfriend didnt even notice this while
playing CS:GO test added 0.3-4ms to her game.

Note Due to intentional latency being terrible please dont work on the latency results from dslreports.
Note Speeds are currently Set @ 10mb / 10Mb 2Pipes 4Children Each  75 / 50 / 5 / 2
Depending where I have lay them in the sharper determines which children I am using.
Yep I know its weird but for me it works.


[TheNarc]

I just wanted to report that I did run some tests over the weekend using 2.4.3 and don't see any differences (i.e. fq_codel still seems to be working for me).  I tested using the dslreports speed test and also ran flent's rrul and rrul_torrent tests.  I realize that just another report of "works for me" isn't especially helpful, though it does seem to point to some specific configuration issue rather than a wholesale breakdown of fq_codel in 2.4.3.

[dennypage]

@TheNarc:

I just wanted to report that I did run some tests over the weekend using 2.4.3 and don't see any differences (i.e. fq_codel still seems to be working for me).  I tested using the dslreports speed test and also ran flent's rrul and rrul_torrent tests.  I realize that just another report of "works for me" isn't especially helpful, though it does seem to point to some specific configuration issue rather than a wholesale breakdown of fq_codel in 2.4.3.

No, it still helps. Just to be sure, when you test with dslreports, you used http rather than https, yes? My tests with https still show an A most of the time, but http is showing C/D. I haven't run anything with rrul.

[TheNarc]

That's a very interesting point.  I didn't explicitly select HTTP, and I use the HTTPS Everywhere plugin, so I quite possibly was testing with HTTPS.  I'm at work, but when I get home this evening I'll run a test with HTTP and get back to you.  Are you aware of a specific reason why that may make a difference, or just know that it does based on your testing?

[dennypage]

I ran some tests with flent/rrul, and it doesn't show any issue at all. I also ran a rrul with the limiters disabled. The bad results with limiters disabled confirmed that basic fq_codel is working. :)

Of note is that rrul seems to be hard coded to use 4 streams only. Testing with DSLReports restricted to 4 streams (http/websocket) doesn't show an issue either.

DSLReports with 8 streams is marginally okay. Above 12 streams or so, DSLReports starts to tank. By the time it gets to 24 streams, it's digging big holes in the ground.

As it seems to be an issue with the number of simultaneous streams, I'm going to try multiple lan systems running simultaneous tests next.

[w0w]

dennypage
http://www.dslreports.com/speedtest/31667858 plain http

But I found that wifi clients have different situation — bufferbloat is rated like "С" and I remember that last summer I have activated QoS on the wifi access point to eliminate this problem and now looks like it does not work anymore, I don't think it's related to pfSense, but may be to wifi client that have updated against spectre/meltdown android kernel. I

[dennypage]

In general my testing is wired, both Linux (Chrome & Firefox) and Mac (Safari and Chrome). I've tested wireless with Mac and the results appear similar at first blush, but I haven't done a lot of testing there.

[TheNarc]

I would expect that with wireless, all bets are off.  I run all my testing wired as well.  I did run one test last night with 24 streams each up and down and still didn't see degradation on the HTTP dslreports test.  I forget what hardware you were running with.  Have you monitored CPU usage while running the tests?  That seems like a long shot for sure, but if I understand correctly, more flows equals more queues, so queue management becomes a higher burden.

[w0w]

My testing was also wired. 32 streams with Hi-Res BufferBloat.
@dennypage:

In general my testing is wired, both Linux (Chrome & Firefox) and Mac (Safari and Chrome). I've tested wireless with Mac and the results appear similar at first blush, but I haven't done a lot of testing there.

If I were you, I would try to disable all shapers, limiters, leaving just plain firewall and let it run again, to see is it worse or better. I just think that it can be overloaded ISP upstream router, but the simpliest way to check it is to downgrade pfSense or connect some laptop directly to your ISP modem or router and re-test.

[Gatekeeper-ZA]

Hey Guys have been playing around some more. could someone please help with my config it seems that my pipe is not selecting the correct que
I have tried please see attached screenshot. all seems to be working correct except for me not being able to use the queues on the in/out
on firewall rules problem still persists after reboot. thanks in advance for the help.

[dennypage]

@TheNarc:

I forget what hardware you were running with.  Have you monitored CPU usage while running the tests?

It's a 4860. Peak CPU @ ~25%, average under 20%.

[dennypage]

@w0w:

If I were you, I would try to disable all shapers, limiters, leaving just plain firewall and let it run again, to see is it worse or better.

Yes, I ran rrul tests with and without limiters enabled, confirming the basic operation of fq_codel with 4 streams.

[TheNarc]

@Gatekeeper-ZA:

Hey Guys have been playing around some more. could someone please help with my config it seems that my pipe is not selecting the correct que
I have tried please see attached screenshot. all seems to be working correct except for me not being able to use the queues on the in/out
on firewall rules problem still persists after reboot. thanks in advance for the help.

I'm not sure that your output is incorrect.  At least, it's very similar to mine for the output of ipfw pipe show and my configuration appears to be working based on test results.  What's more interesting to me is that your output for the other two commands (ipfw queue show and ipfw sched show) differ significantly from mine.  And I don't know which of ours is correct (if either).  How did you set your pipes and queues up?  I added my two pipes with no masks, then added two child queues to each pipe with masks (source address mask for upload pipe, destination address mask for download pipe).

I did find a rather amusing (albeit unhelpful) posting that validated my impression of the output of these commands as . . . inscrutable:  https://lists.freebsd.org/pipermail/freebsd-net/2017-March/047705.html

[Gatekeeper-ZA]

@TheNarc:

@Gatekeeper-ZA:

Hey Guys have been playing around some more. could someone please help with my config it seems that my pipe is not selecting the correct que
I have tried please see attached screenshot. all seems to be working correct except for me not being able to use the queues on the in/out
on firewall rules problem still persists after reboot. thanks in advance for the help.

I'm not sure that your output is incorrect.  At least, it's very similar to mine for the output of ipfw pipe show and my configuration appears to be working based on test results.  What's more interesting to me is that your output for the other two commands (ipfw queue show and ipfw sched show) differ significantly from mine.  And I don't know which of ours is correct (if either).  How did you set your pipes and queues up?  I added my two pipes with no masks, then added two child queues to each pipe with masks (source address mask for upload pipe, destination address mask for download pipe).

I did find a rather amusing (albeit unhelpful) posting that validated my impression of the output of these commands as . . . inscrutable:  https://lists.freebsd.org/pipermail/freebsd-net/2017-March/047705.html

Hey TheNarc please see screenshots. appologies i cannot give my speeds as they are very unstable currently 5/20 but it ranges from 10/10 5/10 8/10- upload usually way high than download.
Basically Created the 2 Pipes LAN/WAN and their children queues all with masks matching my subnet
The WAN-Pipe Using Source Address and LAN-Pipe Using Destination.
Added A Floating Rule to allow all with DefaultLimiter in/out queues. [ strange thing is if i use my Low-Queue there is not speed difference ] do the queues only work if there is alot of traffic.?
although according to DSLReports there is definitely something happening as im now getting AAA A+++ Depending on conditions as I do not have the most stable connection and speed varies.
I want to use Time Based Queues as there are certain times of the day / evening when the LTE Bands get congested its always the same time so I will be able to time based drop speed to favor latency and avoid bloat. As me saying it is for sure doing something but hell knows what lol there is not much info to go on, my cmdshell cmd is as follows. Note no ECN on Upsteam.
ipfw sched 1 config pipe 1 type fq_codel target 5 noecn quantum 300 limit 600 && ipfw sched 2 config pipe 2 type fq_codel target 5 ecn quantum 300 limit 600

If anyone can think of anything I might have missed or done wrong please post as for others to see thanks very much for the input guys

[TheNarc]

@Gatekeeper-ZA:

@TheNarc:

@Gatekeeper-ZA:

Hey Guys have been playing around some more. could someone please help with my config it seems that my pipe is not selecting the correct que
I have tried please see attached screenshot. all seems to be working correct except for me not being able to use the queues on the in/out
on firewall rules problem still persists after reboot. thanks in advance for the help.

I'm not sure that your output is incorrect.  At least, it's very similar to mine for the output of ipfw pipe show and my configuration appears to be working based on test results.  What's more interesting to me is that your output for the other two commands (ipfw queue show and ipfw sched show) differ significantly from mine.  And I don't know which of ours is correct (if either).  How did you set your pipes and queues up?  I added my two pipes with no masks, then added two child queues to each pipe with masks (source address mask for upload pipe, destination address mask for download pipe).

I did find a rather amusing (albeit unhelpful) posting that validated my impression of the output of these commands as . . . inscrutable:  https://lists.freebsd.org/pipermail/freebsd-net/2017-March/047705.html

Hey TheNarc please see screenshots. appologies i cannot give my speeds as they are very unstable currently 5/20 but it ranges from 10/10 5/10 8/10- upload usually way high than download.
Basically Created the 2 Pipes LAN/WAN and their children queues all with masks matching my subnet
The WAN-Pipe Using Source Address and LAN-Pipe Using Destination.
Added A Floating Rule to allow all with DefaultLimiter in/out queues. [ strange thing is if i use my Low-Queue there is not speed difference ] do the queues only work if there is alot of traffic.?
although according to DSLReports there is definitely something happening as im now getting AAA A+++ Depending on conditions as I do not have the most stable connection and speed varies.
I want to use Time Based Queues as there are certain times of the day / evening when the LTE Bands get congested its always the same time so I will be able to time based drop speed to favor latency and avoid bloat. As me saying it is for sure doing something but hell knows what lol there is not much info to go on, my cmdshell cmd is as follows. Note no ECN on Upsteam.
ipfw sched 1 config pipe 1 type fq_codel target 5 noecn quantum 300 limit 600 && ipfw sched 2 config pipe 2 type fq_codel target 5 ecn quantum 300 limit 600

If anyone can think of anything I might have missed or done wrong please post as for others to see thanks very much for the input guys

So, with the caveat that I am quite new to playing with dummynet and fq_codel, I'm fairly certain that you don't want masks on your pipes (limiters), but only your queues.  As I understand it, the distinction, as gleaned from the FreeBSD man page for ipfw (https://tinyurl.com/jfzok5z), is that masks set of pipes result in dynamic pipes and masks set on queues result in dynamic queues:

when dynamic pipes are used, each flow will get the same bandwidth as defined by the pipe, whereas when dynamic queues are used, each flow will share the parent's pipe bandwidth evenly with other flows generated by the same queue (note that other queues with different weights might be connected to the same pipe).

That said, since you have masks on your pipes that match your entire subnet, maybe you're effectively achieving the same thing (i.e. you end up with only one download limiter and one upload limiter, because your masking only ends up defining one flow which is your entire subnet).  Functionally, though, I would expect that to be identical to just having no masks on your pipes.

On the other hand, I think you want /32 masks on your queues, so that each host on your LAN gets its own queue.

Or I could be entirely wrong  :D  Here's hoping someone more knowledgeable than I will chime in . . .

[Gatekeeper-ZA]

Ok coolies. I tried the suggested it seems to show traffic now passing through the queues although still im not sure if the queue weight is working.
bandwidth still tops off at max the only way is to create a LAN Rule passing everything through the pipes although the damn queues still don't work
even setting weight to 1 sill gives full speed on speedtests. only way to let it limit bandwidth is to actually lower speeds on the pipe itself..
Any Ideas would be greatly appreciated thanks so far guys.

Note :  using the single pipe does work great that is what had my initial thoughts saying its looking good and working its still working but want to assign
the queues to aliases of high bandwidth users on the network as to throttle them. and im not sure if it can even be done this way..

[TheNarc]

Ah okay I think I can help with this.  The weights on child queues dictate the relative share of the parent pipe in the event that the pipe is maxed out.  For example, suppose you have a pipe with a bandwidth limit of 100Mbps and two child queues for it:  one with a weight of 10 and the other with a weight of 90.  If the queue with a weight of 90 is totally idle, the queue with a weight of 10 will be allowed to use all 100Mbps of the parent pipe.  The weight of 10 only means that (in this example) the queue is guaranteed at least 10Mbps of bandwidth; so it's a floor, not a ceiling.

If you want to limit certain hosts on your network to never have more than a certain amount of bandwidth, you'll need to use multiple pipes; they provide a ceiling.

[Gatekeeper-ZA]

Hey TheNarc. thanks for the help. Im stating to sort of understand, lemme try ring this back to you.
so if I have a 10/10 connection and want to "sort of" restrict certain users using aliases, I would do some thing like this:
Create maybe 6 Pipes 3in and 3out pipes each with say 2Queues, with the weight of say 90 and 25 having the pipes being masked to direction/32
–---------------------------------------------------------------------------------------------
---  Wan/Upload              Pipe 1 =  8Mb / Pipe 2 = 4Mb and Pipe 3 = 1.5Mb
-- QueueHigh =  90 and QueueLow 25
---  Lan/Download Pipe  Pipe 1 = 8Mb/ Pipe 2 = 4Mb and Pipe 3 = 512Kb
-- QueueHigh = 90 and QueueLow 25

Aliases : DefaultTraffic // HighTraffic // LowTraffic
Representing the Group of IP's / Computers requiring the rules
Have some rules for services 80 / 443 on Default
Have some rules for NNTP 119/563 on Low

FIREWALL/LAN/RULES
IPv4 * LAN net * * * * none   Default allow LAN to any rule = Have Pipes not Queues assigned.

Floaing Rules :
IPv4 * DefaultTraffic * * * * none   Assigned IN/OUT Pipes = 8Mb Pipe with 2 Children Queues as IN/OUT.
IPv4 * HighTraffic * * * * none   Assigned IN/OUT Pipes = 4Mb Pipe with 2 Children Queues as IN/OUT.
IPv4 * LowTraffic * * * * none   Assigned IN/OUT Pipes = 1.5Mb Pipe with Children Queues as IN/OUT.
–-----------------------------------------------------------------------------------------
ShellCmd :
Either :

ipfw sched 1 config pipe 1 type fq_codel target 5 noecn quantum 300 limit 600 && ipfw sched 2 config pipe 2 type fq_codel target 5 noecn quantum 300 limit 600 && ipfw sched 3 config pipe 3 type fq_codel target 5 noecn quantum 300 limit 600 ipfw sched 4 config pipe 4 type fq_codel target 5 ecn quantum 300 limit 600 && ipfw sched 5 config pipe 5 type fq_codel target 5 ecn quantum 300 limit 600 && ipfw sched 6 config pipe 6 type fq_codel target 5 ecn quantum 300 limit 600

OR:

ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codel && ipfw sched 3 config pipe 3 type fq_codel && ipfw sched 4 config pipe 4 type fq_codel && ipfw sched 5 config pipe 5 type fq_codel && ipfw sched 6 config pipe 6 type fq_codel

would a configuration something like this work with FQ_Codel..? and would I leave the shellcmd as is or strip the parameters out note the top ipcf command has NOECN on WAN/IN Pipes.
and would it be advised to run it like this. please if possible could you maybe provide a guide from start to finish if it is not too much to ask which uses multiple pipes and multiple queues.
I have notices that if I do it this way that the queues type do not get set as FQ_Codel but a dynamic wf2q+ queue if I saw correct..
Thanks in advance for the help ..