Playing with fq_codel in 2.4

gsmornot · Sep 26, 2018, 3:24 AM

@w0w said in Playing with fq_codel in 2.4:

@gsmornot
What packages you have installed? What is WAN connection type? Do you have some other features configured on traffic shaper side?

Avahi, pfblocker, open vpn export, and nut. WAN is gigabit fiber with cellular backup.

I have a backup much more powerful system. I will update it tomorrow and test.

w0w · Sep 26, 2018, 3:28 AM

@gsmornot said in Playing with fq_codel in 2.4:

pfblocker

I do think that pfblocker have issues with limiters. That's reported before. Can you uninstall it and try again sometime?

gsmornot · Sep 26, 2018, 3:34 AM

@w0w said in Playing with fq_codel in 2.4:

@gsmornot said in Playing with fq_codel in 2.4:

pfblocker

I do think that pfblocker have issues with limiters. That's reported before. Can you uninstall it and try again sometime?

I won’t on my main system because it is one of the main packages I use but I will on my backup. The other issue is like I said before, CodelQ fixes my bufferbloat perfectly, or at least it passes the dslreports test with an A or A+.

Sep 26, 2018, 3:34 AM

@w0w said in Playing with fq_codel in 2.4:

@muppet said in Playing with fq_codel in 2.4:

config_aqm Unable to configure flowset, flowset busy!

I've seen those messages also, once I've rebooted this morning and before. I am not sure does it impact anything.

Yea, I agree, everything seems to be working fine (and fq_codel is showing up in ipfw sched show) but I thought it worth reporting as I didn't encounter such issues with 2.4.3-p1 and manually applying the following rules.limiter

pipe 1 config  bw 95Mb
sched 1 config pipe 1 type fq_codel
queue 1 config pipe 1 mask dst-ip6 /128 dst-ip 0xffffffff


pipe 2 config  bw 18Mb
sched 2 config pipe 2 type fq_codel
queue 2 config pipe 2 mask src-ip6 /128 src-ip 0xffffffff

areynot · Sep 26, 2018, 8:28 AM

@tman222 thanks for the info mr tman...
I was just wondering one more thing... did you set up masks on child queues? (Source for upload / destination for downlod)

The reason I ask this is because the netgate video does not set a mask

zwck · Sep 26, 2018, 8:40 AM

@areynot good question! I saw the videos as well and directly noticed the differences too, i.e. https://forum.netgate.com/post/790119 maybe @tman222 can let enlighten us :D

mattund · Sep 26, 2018, 1:43 PM

@muppet

Your configuration seems to not set/configure an AQM (explicitly defined with the inline parameter droptail/codel/pie/red/gred). You're just setting up the necessary fq_codel part. I found this:

if (busy) {
	D("Unable to configure flowset, flowset busy!");
	err = EINVAL;
	break;
}

That's the config_aqm function in dummynet/limiters for FreeBSD. My theory right now is, because my patch explicitly supplies one of those 4 aforementioned AQM arguments, dummynet is interpreting that as, "re-configure the AQM". Unfortunately, from what I know dummynet has a limitation where if the queue is "busy" (and I don't understand the specifics of that), you cannot re-configure the AQM only.

I don't think this affects the Scheduler option though, unless I am reading this wrong. Maybe someone can double-check this. So, in summary, if you see these lines it just means that your AQM option didn't save, which most people would be leaving at Drop Tail would be my guess.

EDIT: I'm continuing to dig into this more.

/*
 * Reconfigure AQM as the parameters can be changed.
 * We consider the flowset as busy if it has scheduler
 * instance(s).
 */ 
s = locate_scheduler(nfs->sched_nr);
config_aqm(fs, ep, s != NULL && s->siht != NULL);

s->siht != NULL maps to busy. This may mean that, if my patch were to de-configure the scheduler first, then run the current commands, it may not print this error?

zwck · Sep 26, 2018, 1:43 PM

@mattund said in Playing with fq_codel in 2.4:

@muppet

Your configuration seems to not set/configure an AQM (explicitly defined with the inline parameter droptail/codel/pie/red/gred). You're just setting up the necessary fq_codel part. I found this:
if (busy) {
	D("Unable to configure flowset, flowset busy!");
	err = EINVAL;
	break;
}
That's the config_aqm function in dummynet/limiters for FreeBSD. My theory right now is, because my patch explicitly supplies one of those 4 aforementioned AQM arguments, dummynet is interpreting that as, "re-configure the AQM". Unfortunately, from what I know dummynet has a limitation where if the queue is "busy" (and I don't understand the specifics of that), you cannot re-configure the AQM only.

I don't think this affects the Scheduler option though, unless I am reading this wrong. Maybe someone can double-check this. So, in summary, if you see these lines it just means that your AQM option didn't save, which most people would be leaving at Drop Tail would be my guess.

@tman222 have you watched the video, would you mind sharing your thoughts on ecn, and masks.

I remember that ecn should only be on the up queue, and masks should be appropriaty set downq destination upq source

luckman212 · Sep 26, 2018, 1:49 PM

@mattund said in Playing with fq_codel in 2.4:

EDIT: I'm continuing to dig into this more.

In case you missed it:

https://www.reddit.com/r/PFSENSE/comments/9j1h8u/244_codel_limiter_error/?st=JMJ7GJB0&sh=4db7939a

mattund · Sep 26, 2018, 1:58 PM

@zwck

Personally, I don't use it on that side (upload), and I haven't noticed any performance loss. I am not sure where the idea came from to not use ECN on the outgoing queues, however in saying that I don't mean to discredit the idea. I have a limited understanding of what ECN actually accomplishes besides setting a TCP flag for the channel participants when the queue/link is at capacity, so I'll have to pass on saying much more than that. I will say, at first impression it seems as though it would help to have it set on the upload side. We may need to carefully benchmark it set on and with connections shared with ECN-supported hosts (not all support it).

As for masks, I have played with them a little. I do believe they work still, and you can use them if you choose to. Personally, my setup is extremely basic so I don't need them configured outside of the default. FQ_CODEL will show one flow if you have one mask set up, by the way, usually 0.0.0.0/0 for the source and destination. From my experiences so far, this doesn't mean it's not working, it's just how it seems a dummynet scheduler configured as FQ_CODEL ingests streams. I think the developer of dummynet chose to use the internals of the scheduler type to determine the flow instead of using dummynet's capabilities of identifying unique flows, so maybe it "anonymizes" the traffic before heading into the FQ_CODEL code to save on CPU cycles.

Unrelated to that post, I am looking into why people aren't able to configure their limiters after upgrading to 2.4.4. I had no trouble, however, I was using the 2.4.3 patch. I hope I'm not too late to help there

w0w · Sep 26, 2018, 2:01 PM

You don't need masks if you don't want additional features / filtering like evenly shared bandwidth. Anyway I've followed Netgate guide and even tried to change some settings wrongly, everything I've tried — does not affect bufferbloat nor bandwidth numbers at all.

gsmornot · Sep 26, 2018, 8:14 PM

@w0w said in Playing with fq_codel in 2.4:

@gsmornot said in Playing with fq_codel in 2.4:

pfblocker

I do think that pfblocker have issues with limiters. That's reported before. Can you uninstall it and try again sometime?

pfBlocker is not the issue, it's horsepower. My older router is a multicore PC that I replaced with the SG-3100 for power savings. The SG-3100 is fantastic but in this case is under powered for the needs of fq_codel. The older server is A+ across the board at nearly full gigabit rate without breaking a sweat. So, that answers that for me. I will stick with CODELQ and the SG-3100 because it has some advantages in being lower power and compact.

Just for comparison, my older server peaks at about 10% at full rate without shaping, my SG-3100 needs 95% of the CPU. I bet my old server would be faster for my OpenVPN service as well but I will stick with compact.

Shaping, the old server needs @26% to run the dslreports test. 2.1GHz Intel. The SG-3100 is full out. 1.6GHz ARM.

Following a guide I found, I setup a FAIRQ shaper on the WAN with a CODEL Active Queue child. A+ across the board and less processor needed. I peaked at 60% roughly so sounds good to me.

teh g · Sep 26, 2018, 8:45 PM

@gsmornot I'm running into similar CPU limits when attempting to use fq_codel. This subsequently caused a drop in outbound speeds I think. Running without the floating rules from the video I can hit my ~1000 Mb/s down, when the rules are enabled I only get ~650 Mb/s down and still get a C on bufferbloat.

I'll have to give it a go with CODELQ.

satadru · Sep 26, 2018, 8:53 PM

I setup an upload limiter for my primary WAN connection using the codel settings in the pfsense video from this summer here on 2.4.4: https://www.youtube.com/watch?v=o8nL81DzTlU&t=380

This gave me an "A" on the dslreports speedtest site.

With the floating rule set to send outgoing IPv4 traffic using a quick pass through the limiter, I get all sorts of problems with network connectivity on my LAN side. For instance my Google Home devices on my LAN side refused to connect.

I have a failover WAN setup though, and it turns out that after setting this up for my primary WAN connection, my setup decided my primary WAN connection might be having a problem, and failed over to my backup (more expensive) WAN connection.

Do any of you have a working setup with both a codel limiter and a failover WAN setup, and if so would you be willing to share your configuration?

gsmornot · Sep 27, 2018, 7:28 PM

@teh-g said in Playing with fq_codel in 2.4:

@gsmornot I'm running into similar CPU limits when attempting to use fq_codel. This subsequently caused a drop in outbound speeds I think. Running without the floating rules from the video I can hit my ~1000 Mb/s down, when the rules are enabled I only get ~650 Mb/s down and still get a C on bufferbloat.

I'll have to give it a go with CODELQ.

Same here. About 650 and packet loss on gateways.

mattund · Sep 26, 2018, 11:49 PM

@satadru said in Playing with fq_codel in 2.4:

Do any of you have a working setup with both a codel limiter and a failover WAN setup, and if so would you be willing to share your configuration?

On your floating rule, have you tried changing it to a match only rule (not pass), and turning off quick match? I didn't use the pass option, and I also don't use quick match like Jim shows. I do have smart home devices that all work perfectly fine and I can reach them without issue. They are even on a different VLAN + AP.

My floating rule(s):

Action: Match
Direction: out
Interface: WAN
Address Family: IPv4 or IPv6, but not both
Protocol: TCP/UDP (to avoid the ICMP traceroute craziness others have demonstrated)
Destination: Invert match, Single host or alias, RFC_1918 (an alias of mine, to prevent shaping to modem)
Gateway: Self-explanatory; must be matching address family
In / Out pipe: qWANUpload / qWANDownload

What hardware is everyone using? Shaping probably takes a bit of horsepower

Sep 26, 2018, 11:58 PM

@mattund that Protocol: TCP/UDP only avoids it for Windows, which uses ICMP.
Mac (I think) and other unix hosts use UDP so with this config, I still saw these loops.

I'm going to try and find a simple repro for it and log a ticket (not FQ_CODEL related, btw)

w0w · Sep 27, 2018, 1:26 AM

I've searched a bit over internet and so far I've found that fq_codel is limited to use only one cpu core, so it definitely fails to achieve best results on some CPUs that have low horsepower per core. Also if you have igb NIC on WAN configured as PPPoE than you will be limited to one core performance as well.

teh g · Sep 27, 2018, 1:36 AM

@w0w said in Playing with fq_codel in 2.4:

I've searched a bit over internet and so far I've found that fq_codel is limited to use only one cpu core, so it definitely fails to achieve best results on some CPUs that have low horsepower per core. Also if you have igb NIC on WAN configured as PPPoE than you will be limited to one core performance as well.

That likely explains the performance hit I am seeing. I have a quad core with not the fastest speeds per core.

satadru · Sep 27, 2018, 1:58 AM

@mattund said in Playing with fq_codel in 2.4:

On your floating rule, have you tried changing it to a match only rule (not pass), and turning off quick match? I didn't use the pass option, and I also don't use quick match like Jim shows. I do have smart home devices that all work perfectly fine and I can reach them without issue. They are even on a different VLAN + AP.

My floating rule(s):

Thanks for that. I'm going to try match only the next time I'm not likely to disrupt anyone. I didn't know that ICMP was going to be an issue either, so changing the Address Family in the rule to TCP/UDP is also on my list.

For what it is worth my hardware CPU-wise is:

Intel(R) Atom(TM) CPU 330 @ 1.60GHz
4 CPUs: 1 package(s) x 2 core(s) x 2 hardware threads

Even with the limiter I'm able to get 100/10 throughput according to fast.com