Playing with fq_codel in 2.4
-
the weird thing is that with huge uploads or download (and torrents, iperf, rsync... for example) everything seems ok. RTT etc.
When Mac/Apple processes try to upload photos for example... then RTT goes way up. As if some packets dont go through the pipe.
How can I debug it better?
-
Just getting started with implementing an fq_codel limiter... could use some advise on how to interpret what's going on and how to tune if necessary please.
What might cause Ookla SpeedTest to only get 159 Mbps download when other speed tests are resulting in 320Mbps? Limiter info shows only one bucket, so why limited to ~<50% bandwidth in this particular app but not the web-based speed test?
When running the Ookla SpeedTest app from the same host (with 2x1gbps bonded eth, 24-core, 64GB RAM) as the dslreports web-based test, I'm only getting ~160/20Mbps results.
Limiter Info while running Ookla speedtest download:
Limiters: 00001: 350.000 Mbit/s 0 ms burst 0 q131073 1000 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0 AQM CoDel target 5ms interval 100ms ECN sched 65537 type FIFO flags 0x0 0 buckets 0 active 00002: 25.000 Mbit/s 0 ms burst 0 q131074 1000 sl. 0 flows (1 buckets) sched 65538 weight 0 lmax 0 pri 0 AQM CoDel target 5ms interval 100ms ECN sched 65538 type FIFO flags 0x0 0 buckets 0 active Schedulers: 00001: 350.000 Mbit/s 0 ms burst 0 q65537 50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN Children flowsets: 1 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 ip 0.0.0.0/0 0.0.0.0/0 101 151500 0 0 0 00002: 25.000 Mbit/s 0 ms burst 0 q65538 50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN Children flowsets: 2 0 ip 0.0.0.0/0 0.0.0.0/0 65 3380 0 0 0 Queues: q00001 1000 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 AQM CoDel target 5ms interval 100ms ECN q00002 1000 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 AQM CoDel target 5ms interval 100ms ECNLimiter Info while running dslreports speedtest download:
Limiters: 00001: 350.000 Mbit/s 0 ms burst 0 q131073 1000 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0 AQM CoDel target 5ms interval 100ms ECN sched 65537 type FIFO flags 0x0 0 buckets 0 active 00002: 25.000 Mbit/s 0 ms burst 0 q131074 1000 sl. 0 flows (1 buckets) sched 65538 weight 0 lmax 0 pri 0 AQM CoDel target 5ms interval 100ms ECN sched 65538 type FIFO flags 0x0 0 buckets 0 active Schedulers: 00001: 350.000 Mbit/s 0 ms burst 0 q65537 50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN Children flowsets: 1 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 ip 0.0.0.0/0 0.0.0.0/0 16992 25461976 201 301500 2 00002: 25.000 Mbit/s 0 ms burst 0 q65538 50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN Children flowsets: 2 0 ip 0.0.0.0/0 0.0.0.0/0 57 2983 0 0 0 Queues: q00001 1000 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 AQM CoDel target 5ms interval 100ms ECN q00002 1000 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 AQM CoDel target 5ms interval 100ms ECNHave followed these slides/video advise. Set queue length to 10000 rather than default. No other additional configuration of limiters/queues, etc...
Setup Slides: https://www.slideshare.net/NetgateUSA/pfsense-244-short-topic-miscellany-pfsense-hangout-august-2018
Setup Walkthrough: https://www.youtube.com/watch?v=o8nL81DzTlU&feature=youtu.be&t=377Limits set up for 350/25 Mbps for starters with a single WAN to a Comcast Arris TG1682G in bridge mode.
When running https://dslreports.com/speedtest, I'm seeing the following:
Before configuration:
After configuration:
Floating Rules:
HW:
pfSense 2.4.5-RELEASE-p1 (amd64)
ProtectLI FW6 Vault, i5-7200U CPU @ 2.50GHz, 8GB , AES-NI CPU Crypto: Yes (active)
Packages Installed: pfBlockerNG-devel, ntopng, telegraf
Packages To-Do: suricata (not yet configured)
Outbound IPSec VPN (Oracle private cloud VCN interconnect configured, disabled)
OpenVPN inbound clients: no-connections -
-
@Ricardox Not sure I understand the details in your post... the slide deck clearly says floating rule on WAN, not LAN.
What details / instructions am I missing?
Any ideas about the bandwidth limit difference between a web-based dslreports speedtest versus the Ookla speed test app?
-
This post is deleted! -
Ookla without problems, but in Wan we will have loss of icmp packages.
-
Hi all. I'm looking for help.
I am in Australia and we have a National Broadband Network (NBN) that everyone connects to. I have a fibre connection which has 1000mbits down and 400mbits up.
The problem I have is that the NBN have a very strict policy enforcer on uploads that aggressively drops packets and the effective upload speed I get is 250mbps (because it bursts initially and them immediately drops way back). This happens on all devices I've tried - my environment is largely macOS.
So I tried two things:
I borrowed a network switch that had packet shaping enabled for upstream and I easily got 360+mbps when using it. So that proves with packet shaping, I can get ahead of the NBN policer and stopping it from ever impacting me by setting my shaping to kick in at 380mbps. Note that the NBN has 5ms before it kicks in.
I have since removed the network switch and have been playing with pfsense using CODEL and FQ_CODEL. I run a very decently spec'd PC that would in no way bottle neck. The CPU goes to around 4% max.
With CODEL and FQ_CODEL I am getting 300mbps upload. Never higher.
Without CODEL and FQ_CODEL I get 250mbps.
Why can't I get 360+mbps like I did with the network switch? Have I missed something in my config?
Here are my screenshots:
Parent upload shaper (note I don't need a download one as those speeds are fine):
https://imgur.com/a/xFkUVLd
https://imgur.com/a/Xi5g5m0
https://imgur.com/a/00FireZChild upload shaper:
https://imgur.com/a/rSzhiN6
https://imgur.com/a/0aysaXQ -
@Larrikin As far as I recall fq_codel should only be used as scheduler on parent limiter.
Everything else should be tail drop.
Try it and I'll try to find the original post. -
@netblues said in Playing with fq_codel in 2.4:
@Larrikin As far as I recall fq_codel should only be used as scheduler on parent limiter.
Everything else should be tail drop.
Try it and I'll try to find the original post.I've just tried that - that is, changed the child upload to tail drop and got the same result. Thanks for replying!
-
@Larrikin And the parent queue to tail drop too.
However I think that the fq-codel will always overshoot initially
-
@netblues said in Playing with fq_codel in 2.4:
@Larrikin And the parent queue to tail drop too.
However I think that the fq-codel will always overshoot initially
I've just done that and tested.
So to be clear, the parent now has Queue Algorithm Management set to Tail Drop, the scheduler as FQ_CODEL and the child Queue Algorithm Management is also set to Tail Drop.
Again, I get the same result.
-
@Larrikin And just make sure you have the needed floating rule in place too
-
@netblues said in Playing with fq_codel in 2.4:
@Larrikin And just make sure you have the needed floating rule in place too
Yep I do. Without it, I get 250mbps, with it I get 300mbps.
When I used a switch (instead of pfsense) to do the shaping I got over 360mbps. Don't know how that switch implemented shaping though and that switch was a loan.
-
Hey mate,
I had best results with gargoyle based on openwrt with it's superior qos and throttling... But at these speeds only a few routers can handle it's intensive cpu use.
With pfsense you may need to tweak more as the way they handle qos is kind of picky.
I have 1000/1000 these days, so qos is off.Maybe try a limiter without fq codel for the upload.
-
@robnitro said in Playing with fq_codel in 2.4:
Hey mate,
Maybe try a limiter without fq codel for the upload.
Yeah, I tried all the other ones. I get about a 5% to 10% improvement, but still not where it should be. Is it possible that pfsense just doesn't have the capability I'm looking for?
-
It definitely should be capable. Do you know what NICs are in the pfSense machine you're running? If they're Realtek, I would not be shocked if that's about all you'll get from them despite being "gigabit". Also, I can't tell from the screen shots, but I believe that generally you only want to enable ECN on the download limiter, not upload. That shouldn't be causing your issue, just something I thought I'd mention.
-
What was the switch you were using on loan that gave the 360mbps?
-
@TheNarc said in Playing with fq_codel in 2.4:
It definitely should be capable. Do you know what NICs are in the pfSense machine you're running? If they're Realtek, I would not be shocked if that's about all you'll get from them despite being "gigabit". Also, I can't tell from the screen shots, but I believe that generally you only want to enable ECN on the download limiter, not upload. That shouldn't be causing your issue, just something I thought I'd mention.
It's an intel quad card and they have no problem reaching gigabit speeds. The issue is the NBN policer given my upload is limited to 400mbps. It hits hard and drops my speed to 250mbps. I'm trying to just get under the radar of the NBN policer so I can get the near maximum speed of the 400mbps without hitting the NBN policier.
I have disabled ECN and appreciate the advice!
-
@q54e3w said in Playing with fq_codel in 2.4:
What was the switch you were using on loan that gave the 360mbps?
Ubiquity.
-
Here is some additional information from the NBN in terms of how they deal with their policy enforcer on uploads (ignore downloads as that is fine).
*******The PBS defines the length of a burst of Layer 2 traffic (either in bytes or milliseconds as set out below) that may be received at ingress to the NBN Co Network for a burst of traffic that pushes the average Information Rate above the configured bandwidth profile for a PIR traffic class. Traffic in excess of the PBS will be discarded by the NBN Co Network. The PBS is set by NBN Co for each PIR specification, and cannot be modified.
The PBS is used by the policing functions of the NBN Co Network at ingress to the NBN Co Network to determine whether a stream of ingress data complies with the subscribed PIR. Customer is responsible for ensuring that all ingress traffic is shaped to comply with the PIR/PBS as specified for the required traffic class and interface, before presentation to the UNI-D or NNI as relevant.
It goes on to define the PBS as:
Downstream at the NNI: 10ms
Specific PBS setting in Bytes is dependent on the TC-4 PIR (bandwidth profile) selected
Upstream at the UNI-D: 40,000 bytes
The AVC TC-4 PBS is set by NBN Co and cannot be modified by Customer*******