Playing with fq_codel in 2.4
-
Hi all. I'm looking for help.
I am in Australia and we have a National Broadband Network (NBN) that everyone connects to. I have a fibre connection which has 1000mbits down and 400mbits up.
The problem I have is that the NBN have a very strict policy enforcer on uploads that aggressively drops packets and the effective upload speed I get is 250mbps (because it bursts initially and them immediately drops way back). This happens on all devices I've tried - my environment is largely macOS.
So I tried two things:
I borrowed a network switch that had packet shaping enabled for upstream and I easily got 360+mbps when using it. So that proves with packet shaping, I can get ahead of the NBN policer and stopping it from ever impacting me by setting my shaping to kick in at 380mbps. Note that the NBN has 5ms before it kicks in.
I have since removed the network switch and have been playing with pfsense using CODEL and FQ_CODEL. I run a very decently spec'd PC that would in no way bottle neck. The CPU goes to around 4% max.
With CODEL and FQ_CODEL I am getting 300mbps upload. Never higher.
Without CODEL and FQ_CODEL I get 250mbps.
Why can't I get 360+mbps like I did with the network switch? Have I missed something in my config?
Here are my screenshots:
Parent upload shaper (note I don't need a download one as those speeds are fine):
https://imgur.com/a/xFkUVLd
https://imgur.com/a/Xi5g5m0
https://imgur.com/a/00FireZChild upload shaper:
https://imgur.com/a/rSzhiN6
https://imgur.com/a/0aysaXQ -
@Larrikin As far as I recall fq_codel should only be used as scheduler on parent limiter.
Everything else should be tail drop.
Try it and I'll try to find the original post. -
@netblues said in Playing with fq_codel in 2.4:
@Larrikin As far as I recall fq_codel should only be used as scheduler on parent limiter.
Everything else should be tail drop.
Try it and I'll try to find the original post.I've just tried that - that is, changed the child upload to tail drop and got the same result. Thanks for replying!
-
@Larrikin And the parent queue to tail drop too.
However I think that the fq-codel will always overshoot initially
-
@netblues said in Playing with fq_codel in 2.4:
@Larrikin And the parent queue to tail drop too.
However I think that the fq-codel will always overshoot initially
I've just done that and tested.
So to be clear, the parent now has Queue Algorithm Management set to Tail Drop, the scheduler as FQ_CODEL and the child Queue Algorithm Management is also set to Tail Drop.
Again, I get the same result.
-
@Larrikin And just make sure you have the needed floating rule in place too
-
@netblues said in Playing with fq_codel in 2.4:
@Larrikin And just make sure you have the needed floating rule in place too
Yep I do. Without it, I get 250mbps, with it I get 300mbps.
When I used a switch (instead of pfsense) to do the shaping I got over 360mbps. Don't know how that switch implemented shaping though and that switch was a loan.
-
Hey mate,
I had best results with gargoyle based on openwrt with it's superior qos and throttling... But at these speeds only a few routers can handle it's intensive cpu use.
With pfsense you may need to tweak more as the way they handle qos is kind of picky.
I have 1000/1000 these days, so qos is off.Maybe try a limiter without fq codel for the upload.
-
@robnitro said in Playing with fq_codel in 2.4:
Hey mate,
Maybe try a limiter without fq codel for the upload.
Yeah, I tried all the other ones. I get about a 5% to 10% improvement, but still not where it should be. Is it possible that pfsense just doesn't have the capability I'm looking for?
-
It definitely should be capable. Do you know what NICs are in the pfSense machine you're running? If they're Realtek, I would not be shocked if that's about all you'll get from them despite being "gigabit". Also, I can't tell from the screen shots, but I believe that generally you only want to enable ECN on the download limiter, not upload. That shouldn't be causing your issue, just something I thought I'd mention.
-
What was the switch you were using on loan that gave the 360mbps?
-
@TheNarc said in Playing with fq_codel in 2.4:
It definitely should be capable. Do you know what NICs are in the pfSense machine you're running? If they're Realtek, I would not be shocked if that's about all you'll get from them despite being "gigabit". Also, I can't tell from the screen shots, but I believe that generally you only want to enable ECN on the download limiter, not upload. That shouldn't be causing your issue, just something I thought I'd mention.
It's an intel quad card and they have no problem reaching gigabit speeds. The issue is the NBN policer given my upload is limited to 400mbps. It hits hard and drops my speed to 250mbps. I'm trying to just get under the radar of the NBN policer so I can get the near maximum speed of the 400mbps without hitting the NBN policier.
I have disabled ECN and appreciate the advice!
-
@q54e3w said in Playing with fq_codel in 2.4:
What was the switch you were using on loan that gave the 360mbps?
Ubiquity.
-
Here is some additional information from the NBN in terms of how they deal with their policy enforcer on uploads (ignore downloads as that is fine).
*******The PBS defines the length of a burst of Layer 2 traffic (either in bytes or milliseconds as set out below) that may be received at ingress to the NBN Co Network for a burst of traffic that pushes the average Information Rate above the configured bandwidth profile for a PIR traffic class. Traffic in excess of the PBS will be discarded by the NBN Co Network. The PBS is set by NBN Co for each PIR specification, and cannot be modified.
The PBS is used by the policing functions of the NBN Co Network at ingress to the NBN Co Network to determine whether a stream of ingress data complies with the subscribed PIR. Customer is responsible for ensuring that all ingress traffic is shaped to comply with the PIR/PBS as specified for the required traffic class and interface, before presentation to the UNI-D or NNI as relevant.
It goes on to define the PBS as:
Downstream at the NNI: 10ms
Specific PBS setting in Bytes is dependent on the TC-4 PIR (bandwidth profile) selected
Upstream at the UNI-D: 40,000 bytes
The AVC TC-4 PBS is set by NBN Co and cannot be modified by Customer******* -
Apologies in advance if you've already provided this information - I read back through but may have missed it - but are you saying that with the pfSense machine you can never get above ~300Mbps regardless of whether you use limiters at all? Or are you saying that you've tried different limiter settings but never get above that speed? In other words, through your testing have you isolated this apparent speed ceiling specifically to when you use limiters?
Another interesting test (maybe, I'm not sure whether it would really provide useful information) might be to set up a download limiter. I know that you don't need one, and it would only be temporary. But may be interesting to know whether setting up a download limit with the same parameters as your upload limiter would "over-limit" your download in the same manner.
I can say that I haven't seen anything like this and I use a roughly identical shaping configuration, but also with dramatically different bandwidth limits. I only have 10Mbps upstream, so it's nowhere near the same throughput.
One other thought, and this is really off the wall and should make no difference, but it's also trivial to try: maybe specify the bandwidth in different units. For example, 380,000 Kbps.
-
@TheNarc said in Playing with fq_codel in 2.4:
Apologies in advance if you've already provided this information - I read back through but may have missed it - but are you saying that with the pfSense machine you can never get above ~300Mbps regardless of whether you use limiters at all? Or are you saying that you've tried different limiter settings but never get above that speed? In other words, through your testing have you isolated this apparent speed ceiling specifically to when you use limiters?
Yes, I am able to get over 360+mbps when I put a switch between pfsense and the NBN connection (where the switch is doing the shaping). I don't own that switch and reluctant to buy one to fix this when hopefully we can resolve this within pfsense.
Another interesting test (maybe, I'm not sure whether it would really provide useful information) might be to set up a download limiter. I know that you don't need one, and it would only be temporary. But may be interesting to know whether setting up a download limit with the same parameters as your upload limiter would "over-limit" your download in the same manner.
I've tried this - no difference in results.
One other thought, and this is really off the wall and should make no difference, but it's also trivial to try: maybe specify the bandwidth in different units. For example, 380,000 Kbps.
I'll give this a shot and revert.
-
@Larrikin Sorry about that, somehow I missed that the pfSense box was still in the loop when you were using the Ubiquity switch. Only other thought I have offhand is whether you've tried ALTQ shaping instead of limiters? Again, I see nothing wrong with what you're doing now, but if it simply won't work you can try ALTQ as a point of comparison.
I can also say that on my upload limiter, I have a much lower limit value (1024 as opposed to 10240) based on information from the following sources:
https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/
Also based on those sources I have my upload limiter quantum set at 300, but then you have a much higher upload than the 100Mbps for which the setting of 300 is recommended.
-
@TheNarc said in Playing with fq_codel in 2.4:
@Larrikin Sorry about that, somehow I missed that the pfSense box was still in the loop when you were using the Ubiquity switch. Only other thought I have offhand is whether you've tried ALTQ shaping instead of limiters? Again, I see nothing wrong with what you're doing now, but if it simply won't work you can try ALTQ as a point of comparison.
I'm unfamiliar with ALTQ. How do I go about setting that up? I can't find anything in the GUI.
-
@Larrikin Sorry, it's not referred to as such in the GUI. It's set up on the "By Interface" tab from the "Firewall > Traffic Shaper" page. But typically it's easiest to set it up using the wizard (far right tab, Wizards, on the same page). It's been a while since I've set it up myself, but there's good information in the pfSense book:
Must head to bed here but good luck! Will check back in.
-
@TheNarc said in Playing with fq_codel in 2.4:
@Larrikin Sorry about that, somehow I missed that the pfSense box was still in the loop when you were using the Ubiquity switch. Only other thought I have offhand is whether you've tried ALTQ shaping instead of limiters? Again, I see nothing wrong with what you're doing now, but if it simply won't work you can try ALTQ as a point of comparison.
Boom - you are right. Using ALTQ and switching off Explicit Network Congestion on qLink has fixed the problem. I've disabled the limiters and use this shaping and it works! Thank you good sir.