Playing with fq_codel in 2.4

gsmornot

@teh-g said in Playing with fq_codel in 2.4:

@gsmornot I'm running into similar CPU limits when attempting to use fq_codel. This subsequently caused a drop in outbound speeds I think. Running without the floating rules from the video I can hit my ~1000 Mb/s down, when the rules are enabled I only get ~650 Mb/s down and still get a C on bufferbloat.

I'll have to give it a go with CODELQ.

Same here. About 650 and packet loss on gateways.

mattund

@satadru said in Playing with fq_codel in 2.4:

Do any of you have a working setup with both a codel limiter and a failover WAN setup, and if so would you be willing to share your configuration?

On your floating rule, have you tried changing it to a match only rule (not pass), and turning off quick match? I didn't use the pass option, and I also don't use quick match like Jim shows. I do have smart home devices that all work perfectly fine and I can reach them without issue. They are even on a different VLAN + AP.

My floating rule(s):

Action: Match
Direction: out
Interface: WAN
Address Family: IPv4 or IPv6, but not both
Protocol: TCP/UDP (to avoid the ICMP traceroute craziness others have demonstrated)
Destination: Invert match, Single host or alias, RFC_1918 (an alias of mine, to prevent shaping to modem)
Gateway: Self-explanatory; must be matching address family
In / Out pipe: qWANUpload / qWANDownload

What hardware is everyone using? Shaping probably takes a bit of horsepower

@mattund that Protocol: TCP/UDP only avoids it for Windows, which uses ICMP.
Mac (I think) and other unix hosts use UDP so with this config, I still saw these loops.

I'm going to try and find a simple repro for it and log a ticket (not FQ_CODEL related, btw)

w0w

I've searched a bit over internet and so far I've found that fq_codel is limited to use only one cpu core, so it definitely fails to achieve best results on some CPUs that have low horsepower per core. Also if you have igb NIC on WAN configured as PPPoE than you will be limited to one core performance as well.

teh g

@w0w said in Playing with fq_codel in 2.4:

I've searched a bit over internet and so far I've found that fq_codel is limited to use only one cpu core, so it definitely fails to achieve best results on some CPUs that have low horsepower per core. Also if you have igb NIC on WAN configured as PPPoE than you will be limited to one core performance as well.

That likely explains the performance hit I am seeing. I have a quad core with not the fastest speeds per core.

satadru

@mattund said in Playing with fq_codel in 2.4:

On your floating rule, have you tried changing it to a match only rule (not pass), and turning off quick match? I didn't use the pass option, and I also don't use quick match like Jim shows. I do have smart home devices that all work perfectly fine and I can reach them without issue. They are even on a different VLAN + AP.

My floating rule(s):

Thanks for that. I'm going to try match only the next time I'm not likely to disrupt anyone. I didn't know that ICMP was going to be an issue either, so changing the Address Family in the rule to TCP/UDP is also on my list.

For what it is worth my hardware CPU-wise is:

Intel(R) Atom(TM) CPU 330 @ 1.60GHz
4 CPUs: 1 package(s) x 2 core(s) x 2 hardware threads

Even with the limiter I'm able to get 100/10 throughput according to fast.com

zwck

@mattund said in Playing with fq_codel in 2.4:

Personally, I don't use it (ECN) on that side (upload), and I haven't noticed any performance loss. I am not sure where the idea came from to not use ECN on the outgoing queues, however in saying that I don't mean to discredit the idea.

I remember just this comment,

@strangegopher said in Playing with fq_codel in 2.4:

According to openwrt ecn should only be enabled on inbound packets.

which sources the openwrt wiki, maybe @dtaht can chime in :D

dtaht

@zwck What we saw with ecn on in fq_codel low upload speeds < 40mbit, was an occasional lockout issue where ecn'd flows seemed to cause too many drops of non-ecn'd flows. If you have a reasonable amount of upload bandwidth feel free to try enabling ecn there also. Go hog wild about turning it on on your tcp stacks also...

but do so on an informed basis.

The potential problems with ecn are vast... as is the potential benefit. I worry about overuse and mis-applications of it so much that we started a new (sadly unfunded) project and mailing list for it:

https://www.bufferbloat.net/projects/ecn-sane/wiki/

My position statement is here: https://www.bufferbloat.net/projects/ecn-sane/wiki/dtaht_ecn_editorial/ - and the related rant at the bottom one of my best and worth reading before you fiddle with ecn at all.

Our reasoning (in 2012) for only enabling it on inbound shaping was that more bw was available & the packet had already traversed the internet, so why drop it? At the point of congestion on outbound, though, I'd rather clear room immediately.

I note that I am presently in the minority as to grave concern over widespread ecn deployment - but willing to encourage others to try it on an informed basis.

dtaht

@teh-g I can certainly see running out of cheap cpu at these speeds.

I have to note that usually it's the shaper (limiter) that accounts for 80% or more of the cpu cost of a fq_codel based qos system, and thus I don't get why folk are claiming here fairq + codelq eats less than fq_codel does unless that is effectively (?) mult-cored (?).

fq_codel adds a hash calculation and a ptr lookup and is almost immeasurably the same weight as codel alone (as, at least in linux, the hash often occurs elsewhere)

Elsewhere, I started work on a multi-core capable shaped fq_codel instance but ran out of money and time for now.

dtaht

@gsmornot Is there a way to increase your token bucket size on the shaper at these speeds? BSD does not have high resolution timers.

teh g

@dtaht said in Playing with fq_codel in 2.4:

@teh-g I can certainly see running out of cheap cpu at these speeds.

I have to note that usually it's the shaper (limiter) that accounts for 80% or more of the cpu cost of a fq_codel based qos system, and thus I don't get why folk are claiming here fairq + codelq eats less than fq_codel does unless that is effectively (?) mult-cored (?).

fq_codel adds a hash calculation and a ptr lookup and is almost immeasurably the same weight as codel alone (as, at least in linux, the hash often occurs elsewhere)

Elsewhere, I started work on a multi-core capable shaped fq_codel instance but ran out of money and time for now.

Thanks for the info. My little Celeron J3455 just isn't built for fq_codel at gigabit down speeds. Luckily I don't run into bufferbloat issues too much, since download seems to be impacted most by it, and it isn't too frequent that I cap out the line with one gigabit.

A multi-core capable fq_codel would be great, sad to hear that money and time were lacking. I'm only pegging a core, so it is definitely a limitation there.

dtaht

Always have funding problems. Used to it. What happens usually is someone with more time and anger tackles it for us. :) https://github.com/dtaht/fq_codel_fast is where I am with it presently. Single core it's about 5% faster so far. :(

I think this thread has established that 1.6 ghz arm and low end atoms bottleneck on a single core for inbound shaping on this OS at around 500Mbit? Higher end boxes are fine? I do think tuning the shaper might help some ( see for example: https://github.com/tohojo/sqm-scripts/issues/71 and the related pull request - but this is for linux, not bsd)

and the bulk of the bloat problem is usually outbound and at lower speeds, so the bulk of y'all can solve half the problem at least.

mattund

I would expect the traffic would respect the CPU affinity of the packet when it's received at the kernel level, over into the dummynet code. That may be why single cores are pinned -- there's only one TCP stream, thusly one core involved in the transport? Multiple flows may scale better. In this scenario, making FQ_CODEL multi-threaded would work for single flows well of course.

All this thinking gets me anxious to factor CAKE into dummynet, upstream in BSD @dtaht ... I'm no expert on C, but maybe there is a possibility if it's not too buried in Linux only features. Why stop with FQ_CODEL right? :)

mattund

@mattund

As an update to my previous post concerning Unable to configure flowset, flowset busy!, I was able to determine the following from the FreeBSD source with some level of confidence:

The error is related to the re-configuring of the selected AQM.
The error only occurs when you attempt to explicitly configure an AQM on a queue (aka flowset) that is currently passing traffic, hence the intermittent messages.
The error only indicates your AQM setting (and its related tune-ables) failed to change as instructed, but everything else should have saved fine.
As Reddit users have pointed out, the error may recur on its own for no reason due to the filter resync job. Besides the log spam (sorry), it isn't affecting anything since you're not changing settings at that time.

As mentioned in my last post, the patch's commands do explicitly reconfigure the AQM, via the use of the directives: codel, droptail, red, or gred. They are always present/generated by the patch, to ensure consistency. Unfortunately, dummynet has a limitation where it cannot "hot-swap" the AQM as I anticipated it could.

I have tried configuring the assigned sched/pipe # to -1 first prior to the currently bugged queue configuration command, but this produces errors of its own (hah!). My thinking there was to disconnect the parent pipe, so the queue has nothing to complain about. I have also tried explicitly deleting any existing queue prior, but if a pipe does not exist, these commands fail and it breaks the rest of the rules.limiter execution. We could check for the existence of a queue first, and delete it if one already exists. Keep in mind, that these commands get run at a regular interval, so this could cause packet loss every 15 minutes if we're not careful...

This is a tricky one. But, I'll keep trying to get it error-less and working.

luckman212

@mattund said in Playing with fq_codel in 2.4:

[...] these commands get run at a regular interval, so this could cause packet loss every 15 minutes if we're not careful...

I'm trying to wrap my head around this thread (521 posts and growing, it's a real doozy...)

Just to clarify, you are talking theoretically about the filter reload causing packet loss every ~15m right? I took a stab at FQ_CODEL a couple of days ago and it caused me a lot of misery. I think mainly because I have a fairly complex multiwan config with lots of outbound NAT rules, VPNs, and policy-based routes. I think I have a fundamental misunderstanding of how the floating rules interact with everything else.

But, long story short - if we are just using the standard GUI to configure limiters, without any other patches, there is no packet loss issue, right?

mattund

@luckman212

I don't think there is any packet loss issue presently. However, some of the potential solutions I'm imagining for the "Unable to configure flowset, flowset busy!" problem could cause, say, 1 packet to be dropped if we disconnect or delete the receiving pipe on a limiter to avoid the error being printed in console.

As long as the error messages we're seeing are pallet-able, all one needs to do at this time is stop any traffic flowing through a queue (disable its firewall rule), save the new AQM configuration, then plug it all back together. Besides that, I'm not aware of other problems as far as packet delivery or reliability are concerned.

When it comes to floating rules, my rule of thumb (no pun intended...) has been to treat the FQ_CODEL/Limiter stuff as a separate rule, all on its own, and any other floating rules can remain the same. So long as it's matched, it'll pick up the configuration and carry it through the rest of the further rule processing, so as to make it a "drop-in" setup on an existing configuration without messing with too much behavior-wise.

luckman212

Thanks @mattund. I plan to try again this weekend. Is there any workaround for traceroute becoming brain dead when FQ_CODEL is on? Unfortunately I really need working traceroute for $DAYJOB.

mattund

@luckman212

Not a UDP-based traceroute unfortunately, unless you want to avoid shaping UDP which, for me at least, is a deal-breaker (I'd rather deal with a potato'd traceroute). I sacrificed ICMP shaping and avoided shaping that protocol to fix Windows traceroutes. I actually don't know if anyone's identified why this happens yet, that might be the first step to getting a workaround.

Or optionally, we could all settle and choose to live in an imaginary world where hosts are all one hop away, but refuse to accept our packets until an arbitrary TTL is reached. Sounds funny, but doesn't seem so fun...

luckman212

@mattund said in Playing with fq_codel in 2.4:

Not a UDP-based traceroute unfortunately

I tried the other protocols available for traceroute, which on my Mac are: UDP TCP GRE or ICMP. Sadly, none of them besides UDP gave any meaningful results. And this is without any shaping enabled. Just vanilla.

TCP just flat out doesn't work (timeouts across the board).

GRE worked a little, but always got blocked inevitably at one of the hops along the way. So I could never get a trace to complete using GRE.

ICMP always just immediately completed and told me that whatever I was tracing to was 1 hop away.

zwck

@mattund Sorry Matt, that i ask again, is this post from earlier in this thread still accurate https://forum.netgate.com/topic/112527/playing-with-fq_codel-in-2-4/339 if not can you recommend some settings. in a screenshot form. That would be lovely