Playing with fq_codel in 2.4

uptownVagrant

@chrcoluk just a quick note after some additional testing. I believe I ran into what you were seeing when the number of active states passing through the limiter queues far exceeded the FQ-CoDel flows value and flows were not being separated any longer. Increasing the FQ-CoDel flows value allowed flow separation to be maintained and interactive flow packets were not being sent to sub-queues where non interactive flow packets were also being sent.

csutcliff

@uptownvagrant I'll have a read, thanks.

Regarding the limiters settings, I obviously haven't been able to tune the upstream setting yet (I've got it at just under 95% of the sync speed) but I have tried it set at only 10Mbit/s and it makes no difference.

For the dowstream, my sync is over 65Mbit/s and I've settled on 62Mbit/s after trial an error to see what value eliminated bufferbloat whilst keeping as much bandwidth available as possible. The reason you don't see 65+ on the speed test without the limiters is because my ISP allows me to set the sending rate on the download which I have set to 95% of sync, this is meant to help with bufferbloat and improve VOIP etc because the link is never 100% pegged and in theory stops the wholesale ISP (their supplier) buffers getting involved.

Edit: had a read of the thread you linked. I don't think any of that applies here as I'm using plain PPPoE not PPPoA, I have a full 1500 MTU to the internet thanks to baby jumbo frames (1508 to the modem to account for the PPPoE header) and my upload speed is not restrictive like in the example, in fact it's only a couple of Mbit/s shy of the cable connection that works fine.

uptownVagrant

@csutcliff If you set quantum to 1508 and enable the limiter what is the result? Also, does FQ-CoDel perform as expected over the VDSL2 circuit without dual WAN in the mix?

csutcliff

@uptownvagrant no change in behaviour with 1508 quantum. I'll test just the VDSL limiters enabled later this evening.

robnitro

Limit could be lowered to help out:
https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/#tuning-fq-codel

When running it at 1GigE and lower, today it helps to change a few parameters given limitations in today’s Linux implementation and underlying device drivers.

The default packet limit of 10000 packets is crazy in any other scenario. It is sane to reduce this to a 1000, or less, on anything running at gigE or below. The over-large packet limit leads to bad results during slow start on some benchmarks. Note that, unlike txqueuelen, CoDel derived algorithms can and DO take advantage of larger queues, so reducing it to, say, 100, impacts new flow start, and a variety of other things.

We tend to use ranges of 800-1200 in our testing, and at 10Mbit, currently 600.

csutcliff

@uptownvagrant Sorry for the delay, just got a chance to test it without the dual wan. No change in the result.

csutcliff

@robnitro Thanks for the suggestion, It has seemingly improved my cable connection responsiveness slightly but doesn't make a difference to the upload problem on vdsl.

lightsy

I wanted to give a brief update - I actually am here about the issues with traceroutes showing the default destination addresses for every hop, caused by the Floating Firewall rule.

I fixed this on my side, initially, by only having the rule shape TCP/UDP - but some other clients do traceroute via UDP so I had this same issue. Instead, there's a firewall option to limit/scope it down to particular TCP flags - you can set the ones you wish, but I ultimately ended up targeting it to "Any" protocol, and "Any flags" - this fixed it for me. :)

EDIT: My pfSense randomly crashed... and... it rebooted... and... somehow now it isn't working.
Sigh. Guess I'm back to applying this to a non-floating rule. :(

robnitro

Hi, what rule did you use for non floating?
When I do tracert from windows, it shows only 2 hops. Router and then the destination.

From openwrt dumb access point, all hops show fine like From the router I get all of the hops. Floating rule instant match allow any IPV4 ICMP trace - and another rule icmp any at top of list. So that means it should skip using the codel limiters right? I just dont understand why my desktop client gets bad tracert info?

lightsy

@robnitro

i actually just reverted to using a simple TCP/UDP floating rule, setting up the non-floating was a huge pain (I had it working before) but can't remember how.

I'm going to be creating exemptions / rules that exempt specific clients in the future - now that it's working fine (on my primary use case, Windows!)

Here is the exact rule I have (that is, somehow, letting pfSense UDP traceroute work - I am tired of messing with it / don't care). Anything that is missing means it's default.

Action: Pass
Interface: WAN
Direction: Out
Addr Family: IPv4
Protocol: TCP/UDP
Gateway: WAN_DHCP (my gateway selected)
In/Out Pipe: WANUpQ (my name) & WANDownQ (my name)

Here are the rules I have:
Firewall Rules > Traffic Shaper > "Limiters"

"CODEL_QMDown"
Limiter: Enabled
Bandwidth: 320 Mbit
Queue Management Algorithm: CoDel
Scheduler: FQ_CODEL

Subqueue, "WANDownQ"
Queue algorithm: CoDel

"CODEL_QMUp"
Limiter: Enabled
Bandwidth: 340Mbit (yes, I have higher upload than download)
Queue management algorithm: CoDel
Scheduler: FQ_CODEL

Subqueue, "WANUpQ"
Queue algorithm: CoDel

I know these are basically defaults - I'm not sure why these work? I simply re-created everything, and now everything works great... it's quite odd. I'm curious if there is an issue with pfSense and having the rules edited, or large changes made to the queues (I hit Save/Apply, particularly, after every single change / creation - no going back and editing for me!)

Quite odd. I've used FQ_CoDel in other devices/implementations and have never run into these nuances. It's working great, now, though - but the traceroute/etc was bothering me horribly. With these rules (no floating rule for 'any') it seems to be working great... for... whatever reason.

maverick_slo

Hi all!
Configured limiters and now my traceroute always shows resolved IP address instead of actual hops.
I`ve read here this is common but just to be sure, is this supposed to be like this or is it a bug?

Thanks!

Veldkornet

Hi guys,

I’m also having an issue with my upload speed... So I have a 400Mbit down / 40 Mbit up connection, and I've followed the guide here to enable the limiters, filling in the 400 and 40 as my down and up speeds.

However when I do speedtests, while my bufferbloat is now gone... I no can no longer reach my maximum speeds.
Down I may get around 360Mbit max which I can live with butup is just pathetic, maybe between 2-8Mbit at most....

Is this known? Anything I can do to improve this? I played with the Queue Length value which helped a bit, but in the end didn't really have too much of an impact....

Limiters:
00001: 400.000 Mbit/s    0 ms burst 0 
q131073 10000 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0  AQM CoDel target 5ms interval 100ms ECN
 sched 65537 type FIFO flags 0x0 0 buckets 0 active
00002:  40.000 Mbit/s    0 ms burst 0 
q131074 1000 sl. 0 flows (1 buckets) sched 65538 weight 0 lmax 0 pri 0  AQM CoDel target 5ms interval 100ms ECN
 sched 65538 type FIFO flags 0x0 0 buckets 0 active


Schedulers:
00001: 400.000 Mbit/s    0 ms burst 0 
q65537  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
 sched 1 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1 
00002:  40.000 Mbit/s    0 ms burst 0 
q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
 sched 2 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2 


Queues:
q00001  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0  AQM CoDel target 5ms interval 100ms ECN
q00002  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0  AQM CoDel target 5ms interval 100ms ECN

Disabled:

Enabled:

xciter327

You are supposed to set the limiters lower than your actual connection speed. I set mine to 95% of tested(not ISP advertised) speed.

Also please use the guide posted a little while earlier. The one from the hangouts session is outdated and does not cover a bug workaround.

https://forum.netgate.com/post/807490

P.S - I am also on Ziggo NL 240/24 connection.

Veldkornet

@xciter327 Even at 95% of 400/40 which is 380/38, the upload speed is nowhere near that. The download at least comes in the vicinity.

csutcliff

@Veldkornet I don't suppose it's a PPPoE connection? I had a similar problem.

maverick_slo

Hi how did you resolve pppoe problem?
Thanks!

csutcliff

@maverick_slo I didn't. I am only able to use the Download shaper for my PPPoE WAN. It works perfectly on my other WAN which is DHCP.

Veldkornet

@csutcliff Nope, mine is just a DHCPv4 connection

xciter327

@Veldkornet said in Playing with fq_codel in 2.4:

@xciter327 Even at 95% of 400/40 which is 380/38, the upload speed is nowhere near that. The download at least comes in the vicinity.

Hm. That should not be. Did You follow the guide I linked? Also what hardware is the pfsense running on? Limiters add extra CPU usage. Take a look with "htop -d 1" (need to install it first "pkg install htop") and see if You are peaking the CPU while doing the test.

Veldkornet

@xciter327 ah! I didn't see the guide. I've now re-created everything as per the guide... although it didn't change the results...

I have a PCEngines APU2. I just had a look in "top" while doing the tests. Didn't really see anything climb very high at all. Even then, if it can handle the download, it should be able to manage the upload which is 10% of the download.

If I watch the test, the upload starts strong and climbs to around 20Mbit quickly, for 2 seconds or so, but then drops down to around 4/5 for the remainder of the test where it eventually finishes.