CoDel - How to use
-
I wonder if my ISP uses fq_CoDel. This was taken during my recent issue with BitTorrent flooding my connection with up to 103Mb/s, yet my average was 98.6Mb/s as reported by RRD.
Even then, my pings remained low. I cannot think of a way that my pings could remain so low while still maintaining large enough buffers to be practical. Packet loss indicates a full buffer, yet the pings do not reflect such a thing. fq_CoDel is the only algorithm that comes to mind. It was not this way prior to their recent upgrades. Packet loss was typically accompanied by latency, albeit 10-20ms.
edit: Seems Cisco used the idea of CoDel and made PIE. Both are similar. Cisco even has fq_PIE. I assume this is why I see stable latency from my ISP.
-
I removed all my shapers and applied only codel. My downloads were as fast as I've ever seen them and simultaneous pings to my ISP's first hop were unaffected. Uploads, however, resulted in ping latency going from about 12ms to about 175ms. HFSC completely cures that at the expense of a little top-end speed. I did leave the codel checkboxes checked on all my queues though.
While not perfect, it seems CoDel and make things a lot better without any configuration. I would love to see this same test with fq_codel, if we remember by then. ;D
-
He's talking about it is of my opinion that CoDel won't be as effective if you don't set your interface bandwidth. It is logically impossible for CoDel or other forms of traffic shaping or queue management to work without having some means of knowing how quickly the queue should be drained. This is easy for a synchronous interface like plugging a 1Gb WAN into a 1Gb internet connection, but it is not so simple when you plug a 1Gb wan into a 30Mb internet connection. If your upstream does something like sending back pause frames, the WAN port can know to back off, allowing packets to buffer and CoDel to do its magic. Pause frames still mean that buffering is happening on the receiving interface, which is not desirable because you cannot control buffers in other systems.
CoDel doesn't need to know the bandwidth because it's the interface's job to know how fast it's allowed to dequeue. CoDel just monitors the delays on the packets. Without something to limit CoDel, it will dequeue at full interface rate.
I just wondered if inputing a number in the Bandwidth box would actually do anything because CoDel is knobless. It does do something. It does limit the connection. (I tested it by limiting my connection severely) The only reason I wondered if this knob/setting would work is because everything describes CoDel as knobless. But I also understood that it was an evolution of RED. RED having all the settings it did I could see that knobless in relation to CoDel ment, no settings but still takes a Bandwidth setting.
This has, at least, turned into an interesting post with all these result studies. I may mirror some of the methods and post my results just to compare.
-
I use the bandwidth settings to make sure that my upstream never ever receives data from me faster than it can process it. Even the CoDel people have stated in at least one of their many videos that it is desirable to make sure that your upload speed is slightly less than your actual rated speed. I have 100Mb, so I set it to 96Mb. If your immediate upstream buffer already uses CoDel or similar AQM, then you gain little or nothing, but most ISPs do not implement modern AQMs or any AQM.
You want CoDel on the egress interface at any junction of a fast and slow interface pairing. Example, if you have a 10Gb interface that feeds into a 1Gb interface, you want the egress side of the 1Gb interface to have CoDel, as that is where the choke point is located. Another more common example would be a 1Gb uplink with 2 or more 1Gb feeds, then you want CoDel on the egress side of the 1Gb uplink.
-
I just found this on their page. They have a whole section in tuning your rate limiter to work best with CoDel. They even warn against using hardware offloading because stuff like TSO can ignore your rate limiter, causing your interface to send data too quickly.
http://www.bufferbloat.net/projects/codel/wiki/Best_Practices_for_Benchmarking_CoDel_and_FQ_CoDel
Broadband Links
Fq_codel runs extremely well on asymmetric links such as your commonly available 24.5/5.5 service from a cable modem provider like Comcast. (in conjunction with setting a shaper to your providers's rates and htb rate limiting)Lastly, just running fq_codel by itself, does not help you very much when the next device in line is overbuffered (as in a home router next to today's cable modems). (it DOES help break up microbursts, however, and generally "does no harm") In that case, using HTB to rate limit your router to below the next gateway device and then applying fq_codel will work. See the note above about limitations to HTB.
-
Just another cent that keeps me wondering…..
Basically both WAN and LAN are using Scheduler Type: HFSC . ( see attached pf1.jpg)
But all its children (qACK, qDefault,qP2P…etc) are using Scheduler Options: Codel Active Queues (attached pf2.jpg)=========================================
pfctl -sa
…
ALTQ:
queue root_vmx2 on vmx2 bandwidth 1Gb priority 0 {qLink, qInternet}
queue qLink on vmx2 bandwidth 976.50Mb qlimit 500
queue qInternet on vmx2 bandwidth 23.50Mb hfsc( codel upperlimit 23.50Mb ) {qACK, qP2P, qVoIP, qOthersHigh, qOthersLow, qDefault}
queue qACK on vmx2 bandwidth 5.88Mb hfsc( codel )
queue qP2P on vmx2 bandwidth 2.35Mb hfsc( codel )
queue qVoIP on vmx2 bandwidth 96Kb hfsc( codel realtime 96Kb )
queue qOthersHigh on vmx2 bandwidth 8.22Mb hfsc( codel )
queue qOthersLow on vmx2 bandwidth 1.18Mb hfsc( codel )
queue qDefault on vmx2 bandwidth 3.52Mb hfsc( codel default )
queue root_vmx0 on vmx0 bandwidth 1.14Mb priority 0 {qACK, qDefault, qP2P, qVoIP, qOthersHigh, qOthersLow}
queue qACK on vmx0 bandwidth 285Kb hfsc( codel )
queue qDefault on vmx0 bandwidth 171Kb hfsc( codel default )
queue qP2P on vmx0 bandwidth 114Kb hfsc( codel )
queue qVoIP on vmx0 bandwidth 96Kb hfsc( codel realtime 96Kb )
queue qOthersHigh on vmx0 bandwidth 399Kb hfsc( codel )
queue qOthersLow on vmx0 bandwidth 57Kb hfsc( codel )
=========================================Does this best-of-both-worlds "hybrid" setup actually works for more granular control?
-
Yes. CoDel manages queue latency and HFSC manages queue bandwidth. The other thing to remember is that any queue that is not using all of its bandwidth should effectively have "0'" latency. So if I give my qGames 5Mb of bandwidth, but it is only using 4Mb, then the latency should be 0, except for microbursts.
CoDel only really helps when you do not have enough bandwidth. Technically, enabling CoDel on every queue is probably a good thing, but some traffic types tend to be low or fixed bandwidth and I see no reason to use CoDel if I've given them enough real time.
HFSC and CoDel used together really shines when you are using all of your bandwidth and you want to keep all queues low latency, even bulk data queues, like web traffic. You might have web traffic set with lower bandwidth than VPN traffic, but web traffic can be bursty or consume large amounts of bandwidth for long periods of time. CoDel is great for this because it keeps web traffic low latency and responsive for users attempting to browse while allowing good bandwidth utilization for file transfers.
HFSC is not as good as a simple priority queue if your link bandwidth is not stable because HFSC does not dynamically adjust to changing bandwidth conditions.
fq_CoDel will be that much better once available.
-
I am going to put this in place for my configs I use at LAN parties. Going to test at home then apply to the routers used for the LAN's.
I use the following queues for high traffic - qGames and qHTTPSteam so using CODEL on them should have the desired effect as they are commonly the most used queues.
Next LAN isnt' until March but I might be able to squeeze some testing in at a smaller event in Feb.
-
fq_CoDel will be that much better once available.
Have the developers indicated they plan on supporting fq CoDel in pfSense. I have not seen any post that they are. I probably should set up HFSC plus CoDel . Today I just have CoDel applied to the lan and wan ports with no other traffic shaping. This way at least my traffic is dynamically reacting to congestion in the network.
During prime time, I can see the slow down. If I have a lot off file transfers happening, I can see the link speed up as it gets later in the night. By about 10:30, my link is running a full speed and can pull a sustained 100/5 transfer either direction.
Fq CoDel would be a great solution as it is adaptive. I really hope it gets implemented.
-
-
Yes, but no ETA
https://forum.pfsense.org/index.php?topic=87931.msg485936#msg485936
I saw that, but I read that as "yea it would be nice", but not firm "yes we are going to to this", now we just need to pick the release.
-
Time for a bounty!
I wouldn't expect it before 2.2.3 at the earliest. Looks like a good backlog of bugs to stomp. It would be nice if it was added to Redmine as a feature request, just so it's official.
-
I take back what I said about Codel. When I saturate the upload fully for a long period of time my pings still shoot through the roof (200+ ms).
-
I take back what I said about Codel. When I saturate the upload fully for a long period of time my pings still shoot through the roof (200+ ms).
Thats why the developers of CoDel say you should actually deploy FQ-CoDel.
-
I take back what I said about Codel. When I saturate the upload fully for a long period of time my pings still shoot through the roof (200+ ms).
Try settings your WAN rate to about 95% of your stable upload rate and see what happens.
-
-
I take back what I said about Codel. When I saturate the upload fully for a long period of time my pings still shoot through the roof (200+ ms).
Try settings your WAN rate to about 95% of your stable upload rate and see what happens.
That fixed it…
Well, 200ms is a lot less than what I thought it would have been, so just enabling CoDel without setting bandwidth is probably doing something, just not as effective as in conjunction with settings your bandwidth.
Just wondering, what was your ping during saturation when you have your bandwidth set?
-
I take back what I said about Codel. When I saturate the upload fully for a long period of time my pings still shoot through the roof (200+ ms).
Try settings your WAN rate to about 95% of your stable upload rate and see what happens.
That fixed it…
Well, 200ms is a lot less than what I thought it would have been, so just enabling CoDel without setting bandwidth is probably doing something, just not as effective as in conjunction with settings your bandwidth.
Just wondering, what was your ping during saturation when you have your bandwidth set?
I've been all over the place and probably gave some false information (because I am new to this and didn't really understand what I was doing). Anyway, let me try and clear up my findings:
1. No traffic shaping, all bandwidth available - ping 10ms
2. No traffic shaping, download saturated - ping 60ms
3. No traffic shaping, upload saturated - ping 500+ms4. CODEL active with no bandwidth limit set - all pings same as scenarios 1,2 & 3.
5. CODEL active (WAN side only) with bandwidth limit set at 98% of uplink speed, all bandwidth available - ping 10ms
6. CODEL active (WAN side only) with bandwidth limit set at 98% of uplink speed, download saturated - ping 60ms
7. CODEL active (WAN side only) with bandwidth limit set at 98% of uplink speed, upload saturated - ping 50 to 80msSo it does work, although during a saturated upload I notice my VoIP phone going a bit "robotic", but much improved over no traffic shaping (VoIP calls would be largely unusable). Did not have that issue with PRIQ when prioritizing VoIP as first in line. That being said, CODEL is much easier to setup and it looks like it doesn't completely starve other queues like PRIQ. I'm sticking with CODEL.
Interested to see if FQ_CODEL will make my pings even lower. Will test it as soon as it is available in pfsense.
-
Thanks for the info. I've done my fair share of "false" information with good intentions. It happens.
Yes, the whole PRIQ replacement is kind of interesting. The eventual fq_codel will actually do similar things of PRIQ, but better for general usage. Except in the case where you need certain traffic to always get priority, fq_codel will do what most people want, keep latency low without the hassle of floating rules.
-
I take back what I said about Codel. When I saturate the upload fully for a long period of time my pings still shoot through the roof (200+ ms).
Try settings your WAN rate to about 95% of your stable upload rate and see what happens.
That fixed it…
Well, 200ms is a lot less than what I thought it would have been, so just enabling CoDel without setting bandwidth is probably doing something, just not as effective as in conjunction with settings your bandwidth.
Just wondering, what was your ping during saturation when you have your bandwidth set?
I've been all over the place and probably gave some false information (because I am new to this and didn't really understand what I was doing). Anyway, let me try and clear up my findings:
1. No traffic shaping, all bandwidth available - ping 10ms
2. No traffic shaping, download saturated - ping 60ms
3. No traffic shaping, upload saturated - ping 500+ms4. CODEL active with no bandwidth limit set - all pings same as scenarios 1,2 & 3.
5. CODEL active (WAN side only) with bandwidth limit set at 98% of uplink speed, all bandwidth available - ping 10ms
6. CODEL active (WAN side only) with bandwidth limit set at 98% of uplink speed, download saturated - ping 60ms
7. CODEL active (WAN side only) with bandwidth limit set at 98% of uplink speed, upload saturated - ping 50 to 80msSo it does work, although during a saturated upload I notice my VoIP phone going a bit "robotic", but much improved over no traffic shaping (VoIP calls would be largely unusable). Did not have that issue with PRIQ when prioritizing VoIP as first in line. That being said, CODEL is much easier to setup and it looks like it doesn't completely starve other queues like PRIQ. I'm sticking with CODEL.
Interested to see if FQ_CODEL will make my pings even lower. Will test it as soon as it is available in pfsense.
Out of curiosity, which exact places do you have CoDel active for? As the shaper discipline for the interface, or using HFSC with "Codel Active Queue" on specific queues? And if the latter, which queues?
The more good/working examples we can get the better, I can put them up on the wiki for future reference.
