Playing with fq_codel in 2.4
-
@gsakes @uptownVagrant etc. I've been looking for some coherent recommendations for zoom, especially, as to how to improve your videoconferencing experience. Also curious as to how just generic fq_codel is treating that for y'all?
Oy, what a long thread this has been!
-
Hi, I have a problem, if after enabling the limiter I start to download a game that saturates the band and in the meantime I perform a ping, the ping increases from 10 ms to 50 ms.
Here are the screenshots of the settings, my bandwidth is 120/20.
Should I change something?🔒 Log in to view spoiler🔒 Log in to view spoiler -
@mikekoke Sorry my English, I play without problems with these settings, stable ping + or - 2ms of variation.
-
@mikekoke My result
-
Hi, thanks for the reply, but i have already tried to enable those two settings and the result does not change.
By doing the test on dslreports with the configuration shown in the screenshots i get three A + but if i download a game and at the same time i ping 8.8.8.8 the latency increases from 10 to 50 ms. -
@mikekoke I did the download test and I didn't have this problem.
-
When I did my tests I ran multiple downloads simultaneously to saturate the band.
-
@mikekoke Mystery.
-
when you say "downloading a game" do you mean, steam? Steam really abuses the network, opening 10 or more full rate flows, and even with cake, it's hard to beat them down.
-
Yes, Steam.
I'll try doing normal downloads.
Would the best settings be Tail Drop, FQ_Codel, Queue length at 10000 and ECN both on download and on upload or only on download? I read that the ECN annoys the upload.
The Queues of both on Tail Drop but without ECN and Queue length?
Sorry for the questions but I have read the discussion and there are many opinions on the matter. -
Does it still give the same issue when you reduce the headline bandwidth you have set in the limiters?
-
Yes, even if the bandwidth is reduced, the ping increases the same.
-
This post is deleted! -
@mikekoke Follow this tutorial that works great!🔒 Log in to view 🔒 Log in to view
-
Have you tried changing the quantum to < 536 or 300?, I have had better success at these packet size quantums.
Albeit TCP suffers a bit with a 300 quantum.Also had more success with fq_pie with 300 quantum.
@ dev - fq_pie needs an area in the gui to change the quantum ,pls .
I also run fq_pie lan parents for voip and gaming with greater weights given to the udp in addition to the wan
pipes.I still cant figure out why pfsense wont perform as well with udp packets as a linux router does with QOS/cake on
without quantum changes ?.Edit ;
In testing I have found layer 3 tunnel,gre or openvpn works much better and avoids the tcp overhead,you wont notice as much fluctuation,even with the wan pipes setup as standard for voip and udp games.During full saturation of the pipe the voip and game jitter hardly varies from a no tcp load scenario. This shoudlnt have to be done but works better than cake,pie or fq_codel.Note ;If you try the openvpn method you will need to change the nat and firewall rules for udp to point through the vpn gateway.
- 29 days later
-
Assume that I've correctly implemented everything in @uptownVagrant's post #815. It works (splendidly). Thanks for the good instructions, man.
Then, because I have a second physical WAN interface and public IP for my captive portal (working perfectly for several months), let's say I do everything again in parallel - unique limiters, unique queues, and unique rules #3 & 4, setting rule #4's gateway to the captive portal WAN interface.
The LAN still works, but when tested, the captive portal won't do more than 1/4 of the available upload bandwidth.
Anyone feel like helping a brother out with a W.A.G. as to why this might be happening?
-
Thank you so much to everyone in this thread who has helped in testing, and especially to @uptownVagrant for putting in as much time as they have to help the all of us. This has been working well on my network and I'm very much appreciate this massive thread.
I would like to take this one step further and tie this into bandwidth limits for certain VLAN's of mine. Specifically, I'd like to limit bandwidth in both my IoT network and my guest network.
In an ideal scenario, this FQ CoDel rule would cover the entire WAN interface including the Guest and IoT VLAN (which it already does), but then I can set a second child limiter that only limits the bandwidth for specific interfaces. This way there's a single up queue and a single down queue for the entire WAN, but certain VLAN's have an additional restriction on their bandwidth.
What would be the best way to have a single queue as discussed here, but also have bandwidth limits on specific interfaces/VLAN's?
-
Ive noticed my internally hosted services (matrix synapse, seafile etc) that sit behind Traffic reverse proxy have stopped working since implementing traffic shaping as per the recommendations in this thread.
I narrowed the cause down to the application of queues to my inbound WAN port forwards
Clients connecting inbound via Cloudflare report a '522' error (unable to reach the origin web server and the request times out).
I've checked the in/out limiters are applied to the relevant in/out pipe and they appear correct, as in a mirror of the regular outbound rules on internal subnets
The only noticeable difference is my firewall rule is applied on the WAN interface, not the floating rules as per @uptownVagrant
Would be happy to receive guidance on how to debug this issue.EDIT: With no limiters applied I see traffic hitting my Traefik proxy
176.126.xxx.xxx - - [28/May/2020:14:07:06 +0000] "PUT /_matrix/federation/v1/send/159066 HTTP/1.1" 200 12 "-" "-" 33895 "matrix@file" "http://192.168.90.20:80" 19ms
with limiters applied, these accesses aren't visible so it appears that pfSense isn't actually forwarding the request.
- 11 days later
-
Following the steps below I have a question!
- Whats the difference between Queue Management Algorithm = Tail Drop vs CoDel?
- Even though on the limiter with "Tail Drop" its allowed to have active ECN. On the queue with Tail Drop its not allowed to use ECN. Is generally Tail Drop better? With or without ENC? I could not find anything in the pfsense book.
Thanks!
1.) Create "Out" limiter Tick "Enable" Name: FQ_CODEL_OUT Bandwidth: 96907 Kbit/s Mask: None Queue Management Algorithm: Tail Drop Scheduler: FQ_CODEL target: 5 interval: 100 quantum: 300 limit: 10240 flows: 20480 Click Save/Apply Changes 2.) Add "Out" queue Tick "Enable" Name: fq_codel_out_q Mask: None Queue Management Algorithm: Tail Drop Click Save/Apply Changes 3.) Create "In" limiter Tick "Enable" Name: FQ_CODEL_IN Bandwidth: 83886 Kbit/s Mask: None Queue Management Algorithm: Tail Drop Scheduler: FQ_CODEL target: 5 interval: 100 quantum: 300 limit: 10240 flows: 20480 Click Save/Apply Changes 4.) Add "In" queue Tick "Enable" Name: fq_codel_in_q Mask: None Queue Management Algorithm: Tail Drop Click Save/Apply Changes
-
@daemonix I don't really know but I tried both and neither helped me any so I gave up. So just try and test and try again.
-
@provels for me tail drop seems to give me stable low RRT.... CoDel is a bit erratic... this is weird as reading this paper makes me think different. Evaluation of the Impact of Packet Drops due to AQM over Capacity Limited Paths
Any ideas from the experts?
-
@daemonix that paper has no relation to queue management with fq-codel in Pfsense, it compares different algorithms.
I have no clue what codel does if chosen as queue management under fq-codel, but i would just use drop tail as that is how fq-codel should react when latency increases, i.e. it should drop packets in large queues to keep latency low. -
@bobbenheim fair enough :) that indeed keeps the RRT from moving... so Im happy :)
-
is there a reason why a basic fq_codel is behaving perfect with 20 parallel iperf uploads another 20 down (all together)... but then a "speedtest.net" test or google test sends the RTT up?!
Or a full torrent ubuntu alternative. Full download... no problem.Is like some packets dont follow the rules...
-
Hi, I have the floating rule and quick but Im getting problems with traceroute from Lan (not from the pfsense box). Am I missing something in my rules?!
traceroute www.ntua.gr traceroute to www.ntua.gr (147.102.224.101), 64 hops max, 52 byte packets 1 XX (X) 4.054 ms 1.243 ms 1.275 ms 2 www.ntua.gr (147.102.224.101) 8.807 ms 9.811 ms 10.076 ms 3 www.ntua.gr (147.102.224.101) 15.286 ms 9.946 ms 9.692 ms 4 * * * 5 www.ntua.gr (147.102.224.101) 14.548 ms 13.996 ms 13.006 ms 6 www.ntua.gr (147.102.224.101) 13.220 ms 15.703 ms 13.044 ms 7 * www.ntua.gr (147.102.224.101) 45.182 ms 43.441 ms 8 www.ntua.gr (147.102.224.101) 44.675 ms 44.498 ms 47.619 ms
-
@daemonix what settings have you made within the floating rule?
-
@daemonix keep in mind that the traceroute program is likely using UDP unless you are instructing it to use ICMP.
-
I updated the rule to take all ICMP. I dont have any block rule for UDP or TCP from LAN.
One thing I see in my WAN CODEL IN/OUT rule is that is "matching" very little numbers of packages or data. On the LAN site the main pass rule has gigabytes.
Maybe my problems start elsewhere?Are floating matching rules for CODEL mirroring the LAN stats?
-
@daemonix have you set anything under in/out pipe in advanced options within your ICMP rule?
-
@bobbenheim no no, only the gateway as I have a VPN out for some clients.
-
I read this thread and still I am not sure if floating rule for limiters shoud be "quick pass" or "match". Could someone explain which one is correct or if there is difference in use? Thanks
-
@sikita pass rule is first match wins, match rules is last match wins, other than that you get the same result with both if setup correctly :)
-
the weird thing is that with huge uploads or download (and torrents, iperf, rsync... for example) everything seems ok. RTT etc.
When Mac/Apple processes try to upload photos for example... then RTT goes way up. As if some packets dont go through the pipe.
How can I debug it better?
- about a month later
-
Just getting started with implementing an fq_codel limiter... could use some advise on how to interpret what's going on and how to tune if necessary please.
What might cause Ookla SpeedTest to only get 159 Mbps download when other speed tests are resulting in 320Mbps? Limiter info shows only one bucket, so why limited to ~<50% bandwidth in this particular app but not the web-based speed test?
When running the Ookla SpeedTest app from the same host (with 2x1gbps bonded eth, 24-core, 64GB RAM) as the dslreports web-based test, I'm only getting ~160/20Mbps results.
Limiter Info while running Ookla speedtest download:
Limiters: 00001: 350.000 Mbit/s 0 ms burst 0 q131073 1000 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0 AQM CoDel target 5ms interval 100ms ECN sched 65537 type FIFO flags 0x0 0 buckets 0 active 00002: 25.000 Mbit/s 0 ms burst 0 q131074 1000 sl. 0 flows (1 buckets) sched 65538 weight 0 lmax 0 pri 0 AQM CoDel target 5ms interval 100ms ECN sched 65538 type FIFO flags 0x0 0 buckets 0 active Schedulers: 00001: 350.000 Mbit/s 0 ms burst 0 q65537 50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN Children flowsets: 1 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 ip 0.0.0.0/0 0.0.0.0/0 101 151500 0 0 0 00002: 25.000 Mbit/s 0 ms burst 0 q65538 50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN Children flowsets: 2 0 ip 0.0.0.0/0 0.0.0.0/0 65 3380 0 0 0 Queues: q00001 1000 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 AQM CoDel target 5ms interval 100ms ECN q00002 1000 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 AQM CoDel target 5ms interval 100ms ECN
Limiter Info while running dslreports speedtest download:
Limiters: 00001: 350.000 Mbit/s 0 ms burst 0 q131073 1000 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0 AQM CoDel target 5ms interval 100ms ECN sched 65537 type FIFO flags 0x0 0 buckets 0 active 00002: 25.000 Mbit/s 0 ms burst 0 q131074 1000 sl. 0 flows (1 buckets) sched 65538 weight 0 lmax 0 pri 0 AQM CoDel target 5ms interval 100ms ECN sched 65538 type FIFO flags 0x0 0 buckets 0 active Schedulers: 00001: 350.000 Mbit/s 0 ms burst 0 q65537 50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN Children flowsets: 1 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 ip 0.0.0.0/0 0.0.0.0/0 16992 25461976 201 301500 2 00002: 25.000 Mbit/s 0 ms burst 0 q65538 50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN Children flowsets: 2 0 ip 0.0.0.0/0 0.0.0.0/0 57 2983 0 0 0 Queues: q00001 1000 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 AQM CoDel target 5ms interval 100ms ECN q00002 1000 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 AQM CoDel target 5ms interval 100ms ECN
Have followed these slides/video advise. Set queue length to 10000 rather than default. No other additional configuration of limiters/queues, etc...
Setup Slides: https://www.slideshare.net/NetgateUSA/pfsense-244-short-topic-miscellany-pfsense-hangout-august-2018
Setup Walkthrough: https://www.youtube.com/watch?v=o8nL81DzTlU&feature=youtu.be&t=377Limits set up for 350/25 Mbps for starters with a single WAN to a Comcast Arris TG1682G in bridge mode.
When running https://dslreports.com/speedtest, I'm seeing the following:
Before configuration:
🔒 Log in to viewAfter configuration:
🔒 Log in to viewFloating Rules:
🔒 Log in to viewHW:
pfSense 2.4.5-RELEASE-p1 (amd64)
ProtectLI FW6 Vault, i5-7200U CPU @ 2.50GHz, 8GB , AES-NI CPU Crypto: Yes (active)
Packages Installed: pfBlockerNG-devel, ntopng, telegraf
Packages To-Do: suricata (not yet configured)
Outbound IPSec VPN (Oracle private cloud VCN interconnect configured, disabled)
OpenVPN inbound clients: no-connections -
-
@Ricardox Not sure I understand the details in your post... the slide deck clearly says floating rule on WAN, not LAN.
What details / instructions am I missing?
Any ideas about the bandwidth limit difference between a web-based dslreports speedtest versus the Ookla speed test app?
-
This post is deleted! -
Ookla without problems, but in Wan we will have loss of icmp packages.
- 8 days later
-
Hi all. I'm looking for help.
I am in Australia and we have a National Broadband Network (NBN) that everyone connects to. I have a fibre connection which has 1000mbits down and 400mbits up.
The problem I have is that the NBN have a very strict policy enforcer on uploads that aggressively drops packets and the effective upload speed I get is 250mbps (because it bursts initially and them immediately drops way back). This happens on all devices I've tried - my environment is largely macOS.
So I tried two things:
I borrowed a network switch that had packet shaping enabled for upstream and I easily got 360+mbps when using it. So that proves with packet shaping, I can get ahead of the NBN policer and stopping it from ever impacting me by setting my shaping to kick in at 380mbps. Note that the NBN has 5ms before it kicks in.
I have since removed the network switch and have been playing with pfsense using CODEL and FQ_CODEL. I run a very decently spec'd PC that would in no way bottle neck. The CPU goes to around 4% max.
With CODEL and FQ_CODEL I am getting 300mbps upload. Never higher.
Without CODEL and FQ_CODEL I get 250mbps.
Why can't I get 360+mbps like I did with the network switch? Have I missed something in my config?
Here are my screenshots:
Parent upload shaper (note I don't need a download one as those speeds are fine):
https://imgur.com/a/xFkUVLd
https://imgur.com/a/Xi5g5m0
https://imgur.com/a/00FireZChild upload shaper:
https://imgur.com/a/rSzhiN6
https://imgur.com/a/0aysaXQ -
@Larrikin As far as I recall fq_codel should only be used as scheduler on parent limiter.
Everything else should be tail drop.
Try it and I'll try to find the original post.