CoDel - How to use
-
I tested (ping tests while upload is fully saturated) CoDel in pfSense with proper target(20 for my slow cxn) and interval(100) values, but… it made no difference. I expected otherwise. I tested 5, 20, 30 and 50 as values for target, and for interval I tested 5 and 100. The (incorrect) default in pfSense is target=(50 or 100) and interval=5. If you guys would like to test the stand-alone CoDel with proper target/interval values, in /tmp/rules.debug, replace your current WAN queue lines (they start with "altq" or "queue") with the following line into /tmp/rules.debug and reload with "pfctl -vf /tmp/rules.debug". It is only a temporary change (config.xml, your standard GUI config, creates /tmp/rules.debug on the fly).
Modify the "em0" parts to match your interface, and set the bandwidth to your usual.altq on em0 codelq( qlimit 5, interval 105 ) bandwidth 630Kb queue { root_em0 }
Edit: To clarify, qlimit = target.Another strange thing I encountered during testing was that with CoDel, I had 5-12% of pings unreplied (tcpdump registered them as sent). I tested 3 times with HFSC and CoDelQ… every time, CoDel has ping loss while HFSC never had one dropped packet. By my understanding of CoDel's specifications, it should not have been dropping packets, especially ICMP.
I am being premature, but I am thinking CoDel is not working properly. Here are some tcpdump and ping pastes that tell me the ping request was sent, but never replied, but how can CoDel cause someone to not reply? Any ideas? Am I interpreting tcpdump improperly?
I just noticed that when I switch to CoDel, all my firewall rules are still assigned to non-existing HFSC queues. I wonder if that is causing probs...20:09:03.422774 AF IPv4 (2), length 88: (tos 0x0, ttl 63, id 17061, offset 0, flags [none], proto ICMP (1), length 84)
173.187.x.x > 74.125.22.102: ICMP echo request, id 37395, seq 11, length 64
20:09:03.451752 AF IPv4 (2), length 88: (tos 0x0, ttl 45, id 0, offset 0, flags [none], proto ICMP (1), length 84)
74.125.22.102 > 173.187.x.x: ICMP echo reply, id 37395, seq 11, length 64
20:09:04.424272 AF IPv4 (2), length 88: (tos 0x0, ttl 63, id 32553, offset 0, flags [none], proto ICMP (1), length 84)
173.187.x.x > 74.125.22.102: ICMP echo request, id 37395, seq 12, length 64
20:09:05.424160 AF IPv4 (2), length 88: (tos 0x0, ttl 63, id 45955, offset 0, flags [none], proto ICMP (1), length 84)
173.187.x.x > 74.125.22.102: ICMP echo request, id 37395, seq 13, length 64
20:09:05.480163 AF IPv4 (2), length 88: (tos 0x0, ttl 45, id 0, offset 0, flags [none], proto ICMP (1), length 84)
74.125.22.102 > 173.187.x.x: ICMP echo reply, id 37395, seq 13, length 64PING google.com (74.125.22.102) 56(84) bytes of data.
64 bytes from qh-in-f102.1e100.net (74.125.22.102): icmp_seq=1 ttl=44 time=45.8 ms
64 bytes from qh-in-f102.1e100.net (74.125.22.102): icmp_seq=2 ttl=44 time=46.1 ms
64 bytes from qh-in-f102.1e100.net (74.125.22.102): icmp_seq=3 ttl=44 time=61.8 ms
64 bytes from qh-in-f102.1e100.net (74.125.22.102): icmp_seq=4 ttl=44 time=57.5 ms
64 bytes from qh-in-f102.1e100.net (74.125.22.102): icmp_seq=5 ttl=44 time=55.7 ms
64 bytes from qh-in-f102.1e100.net (74.125.22.102): icmp_seq=6 ttl=44 time=87.6 ms
64 bytes from qh-in-f102.1e100.net (74.125.22.102): icmp_seq=7 ttl=44 time=33.0 ms
64 bytes from qh-in-f102.1e100.net (74.125.22.102): icmp_seq=8 ttl=44 time=28.9 ms
64 bytes from qh-in-f102.1e100.net (74.125.22.102): icmp_seq=9 ttl=44 time=73.3 ms
64 bytes from qh-in-f102.1e100.net (74.125.22.102): icmp_seq=10 ttl=44 time=34.9 ms
64 bytes from qh-in-f102.1e100.net (74.125.22.102): icmp_seq=11 ttl=44 time=29.7 ms
64 bytes from qh-in-f102.1e100.net (74.125.22.102): icmp_seq=13 ttl=44 time=57.0 ms -
Your tcpdump can see the packet but it does not mean that it made it out :)
It certainly means that it was sent to the queue which in this case is Codel.You should have statistics of the codel queue tell you what it dropped and what it passed.
-
@ermal:
Your tcpdump can see the packet but it does not mean that it made it out :)
It certainly means that it was sent to the queue which in this case is Codel.You should have statistics of the codel queue tell you what it dropped and what it passed.
Thanks ermal. Your insight is always welcomed. :)
Quick question or two about CoDel when it is a sub-discipline;
1. are target/interval configurable by the user?
2. are CoDel's stats exposed to the user? -
I have to check the code but normally even when is a sub-discipline all the stats are exported.
When you have Codel as part of HFSC queue the queue stats you see are the ones of Codel book keeping ones. -
Anyone figure out how to apply the patch?
https://redmine.pfsense.org/issues/4692
Keep getting "no file to patch" when testing, so clearly some steps missing on how to apply it.
-
Testing how? You'd need the full pfsense-tools sources from github plus recompile the thing
-
The test button within pfsense>system>patches.
So this is not a directly applicable patch to an installed pfsense which is what I needed to know.
Thanks.
-
The test button within pfsense>system>patches.
So this is not a directly applicable patch to an installed pfsense which is what I needed to know.
Thanks.
Yeah, the patch applies to the source-code. See https://forum.pfsense.org/index.php?topic=76132.0 to get accessto the source-code.
Please be careful. That patch was posted by me and it is completely untested. There are a few other incorrect values I later found, but this bug is not vital so I neglected to create a new (and very possibly wrong) patch. If I remember correctly, all the values needing correction were in ~/patches/stable/10/altq_codel.diff and can found rather simply with grep.
I will leave this to the professionals to patch, until I have some experience building pfSense. I wish pfSense's source-code was more accessible. Already I have had to resort to using doktornotor's "illegitmate" pfsense-tools repository on github to refer to code that the unwashed masses can see.
-
Thanks for the clarification and warning. I tried using Codel with a very functional PRIQ with limiter setup this past weekend at a 50+ person LAN party and Codel seemed to inflict a lot of latency (erratic 200+ ms). Disabled it and we went back to our normal 20-50 ms pings to external game servers.
At home the results seemed to be erratic with a single user and Codel only (different pfSense box). Ended up switching to Barrier Breaker and their SQM fq_codel setup which does actually work very well. Took buffer bloat rating from continuous B-C to a repeatable A+.
-
I wonder if FairQueue in PFSense could be an option for child disciplines also. fq_CoDel, why are you so close yet so far?!
-
the docsis 3.1 standard is a "16ms" target for pie. Pie interpret's target a bit differently than codel. pie is target 20ms in the linux code and pretty different from the docsis-pie version.
the "target" for codel is what it aims for for local queue delay, while trying to keep the pipe filled. The important part is "keeping the pipe filled". It is certainly feasible to achieve a given queue delay while having terrible throughput (tcp's timing out, and so on).
we hit the issue (needing to increase target) with sub 2.5Mbit uplinks after publication of the codel paper.
Codel by itself is very gentle, it gradually reduces queue length to more reasonable amounts and certainly you can briefly see latencies go to 100s, of ms. even thousands of ms on unresponsive traffic.
fair queuing + codel takes most of the latencies for gamer-ish traffic out of the equation, while still moderating (typically tcp) queue length.
I am sorry you are having such difficulty with the fairq + codel emulation in pfsense. testing for codel's correct behavior is somewhat straightforward using things like the tcp_upload test in the flent suite, or merely saturating the link and looking for time of first drop, 2nd drop, etc to follow the 1/sqrt(100) rule. To make your life more difficult, there are several mildly different versions of codel out there and I have not reviewed the one in pfsense.
We tried all sorts of fairq+aqm combining techniques for several years, before incorporating them into one qdisc. two separate modules have flaws - including being difficult to debug.
Another poster here mentioned they had ping packet loss when using codel. Yes, that will happen. Gotta drop packets, some will be pings. hfsc does some clever priorization so pings more rarely drop - but which would you rather drop, pings? or data?
-
I would rather see what my queue is seeing. So HFSC does mess with the order in which packets are sent within a queue? I was under the impression that HFSC is only a scheduler and only messes with inter-queue ordering, not intra-queue ordering. So if you select a dumb FIFO child queue, will HFSC change the order in which packets from that queue leave the interface? I know fq_codel and any fair queuing AQM will do that.
-
I would rather see what my queue is seeing. So HFSC does mess with the order in which packets are sent within a queue? I was under the impression that HFSC is only a scheduler and only messes with inter-queue ordering, not intra-queue ordering. So if you select a dumb FIFO child queue, will HFSC change the order in which packets from that queue leave the interface? I know fq_codel and any fair queuing AQM will do that.
You are seeing what your flow's queue is seeing, which is all that really matters (End-to-End principle).
If you want to know the details of HFSC, read the paper. It is super-fun and the ladies love an HFSC-man.
-
Forgive me if some of this has been repeated already. I just wanted throw in some input on this. I was playing around with CODELQ last night.
Some observations:
I was using the speedtest.net test which tests for bufferbloat.
My link typically achieves 86x5.6Mbps pretty consistently on off peak hours.
I actually used the bandwidth window to get some settings tweak. Initially I had set the bandwidths to 10% below what my actual speeds were but it cutoff WAY too much of my top end bandwidth. When I did that I was getting an A rating on buffer bloat in both directions but speeds were cut to 75x4.2Mbps
I finally settled by testing on going higher. It was a balancing act because if I went too high then the bufferbloat was bad again as originally seen. If I went too low, it would knock off too much off the limits of the speeds.So for the download (LAN) queue, I went for 87.5Mbps and was able to download 83-84Mbps.
For the upload (WAN) queue, I went for 6.1Mbps and was able to achieve 5.2 Mbps.
Both these settings resulted in a "B" rating with approximately 60ms of delay.
Considering the upload delay had been 512ms at one point, I say not bad to giving up ~7% of my top end upload bandwidth, ~3% download bandwidth.
I didn't have too much buffer bloat on my download (LAN) because I guess the ISP managed it very well. I believe it was originally at about 100msSo this whole thing was trial and error but I consistently get the same results which leads me to believe I have achieved the intended goal.
I'm using 2.2.2 4G embedded on a Firebox.
Thanks.
-
Yeah, the whole point of managing bufferbloat is your firewall needs to be the chokepoint, and if your internet connection's rate varies too much, you can get bloat spikes.
My limited understanding of the bufferbloat problem and some technologies is DOCSIS and DSL don't use native Ethernet which cause them to do some strange batching or fragmenting of your packets. This reduces the efficiently and makes for strange bursts in speeds. The bufferbloat forumns are constantly discussing how to properly emulate how DSL and DOCSIS work in an attempt to artificially create the same bottlenecks in the firewall, which allows better management of bufferbloat.
All of that being said, you need to set your bandwidth to be less than your peak, and losing some bandwidth is part of the cost of getting rid of bufferbloat when your ISP doesn't manage it for you.
Personally, I have a 100Mb connection and have my rates set to 99Mb and I get 1ms of bloat on average. But I seem to be a special case.
-
Gazoo - I noticed this too. If I put my full speed in pfsense it is as-if it lowers this amount so it is somewhat less (95%?) of the connection speed you enter for traffic shaping limit.
-
95% is pretty decent. I see closer to about 97%.
-
95% is pretty decent. I see closer to about 97%.
Might be 97%, haven't looked in a while. In any event looks like the pfsense coders deliberately lower the speed you enter. Can anyone confirm?
-
I have turned to the dark-side and chose full bandwidth over best latency on my incoming traffic. The average latency increases to ~70ms (~10ms idle) during a link saturating download, but even with artificial throughput limiting I was reaching ~40ms. Addiction to throughput probably stems from years of dial-up… the experience still haunts me. :)
The difference on uploads is huge though; ~600ms vs ~50ms.
My ISP is the not-so-great-but-improving Windstream. I have wondered what other people experience on other ISPs and CPE.
-
Since I use HFSC even for download traffic shaping, I can make sure Netflix/Youtube buffering is kept to a minimum. I pretty much can't tell the difference, even when using P2P and saturating the downstream.
I have 100Mb with a 2:1 ratio between Normal:Low. P2P will be sitting around 97Mb, then I start of Netflix and I pretty much immediately see P2P drop down to ~32Mb/s and Netflix pushes through ~64Mb/s. Even with DSLReports, I still see an A/A+ during the download test. But if P2P is using 0 and DSLReports is using max, and suddenly P2P jumps up to 33, the bufferbloat can get bad, like C/D, but that's because available bandwidth is decreasing instead of increasing and I'm still stuck with regular CoDel and not fq_CoDel.