CoDel - How to use



  • Just another cent that keeps me wondering…..

    Basically both WAN and LAN are using Scheduler Type: HFSC . ( see attached pf1.jpg)
    But all its children (qACK, qDefault,qP2P…etc) are using Scheduler Options: Codel Active Queues (attached pf2.jpg)

    =========================================
    pfctl -sa

    ALTQ:
    queue root_vmx2 on vmx2 bandwidth 1Gb priority 0 {qLink, qInternet}
    queue  qLink on vmx2 bandwidth 976.50Mb qlimit 500
    queue  qInternet on vmx2 bandwidth 23.50Mb hfsc( codel upperlimit 23.50Mb ) {qACK, qP2P, qVoIP, qOthersHigh, qOthersLow, qDefault}
    queue  qACK on vmx2 bandwidth 5.88Mb hfsc( codel )
    queue  qP2P on vmx2 bandwidth 2.35Mb hfsc( codel )
    queue  qVoIP on vmx2 bandwidth 96Kb hfsc( codel realtime 96Kb )
    queue  qOthersHigh on vmx2 bandwidth 8.22Mb hfsc( codel )
    queue  qOthersLow on vmx2 bandwidth 1.18Mb hfsc( codel )
    queue  qDefault on vmx2 bandwidth 3.52Mb hfsc( codel default )
    queue root_vmx0 on vmx0 bandwidth 1.14Mb priority 0 {qACK, qDefault, qP2P, qVoIP, qOthersHigh, qOthersLow}
    queue  qACK on vmx0 bandwidth 285Kb hfsc( codel )
    queue  qDefault on vmx0 bandwidth 171Kb hfsc( codel default )
    queue  qP2P on vmx0 bandwidth 114Kb hfsc( codel )
    queue  qVoIP on vmx0 bandwidth 96Kb hfsc( codel realtime 96Kb )
    queue  qOthersHigh on vmx0 bandwidth 399Kb hfsc( codel )
    queue  qOthersLow on vmx0 bandwidth 57Kb hfsc( codel )
    =========================================

    Does this best-of-both-worlds "hybrid" setup actually works for more granular control?






  • @chrislee

    Yes. CoDel manages queue latency and HFSC manages queue bandwidth. The other thing to remember is that any queue that is not using all of its bandwidth should effectively have "0'" latency. So if I give my qGames 5Mb of bandwidth, but it is only using 4Mb, then the latency should be 0, except for microbursts.

    CoDel only really helps when you do not have enough bandwidth. Technically, enabling CoDel on every queue is probably a good thing, but some traffic types tend to be low or fixed bandwidth and I see no reason to use CoDel if I've given them enough real time.

    HFSC and CoDel used together really shines when you are using all of your bandwidth and you want to keep all queues low latency, even bulk data queues, like web traffic. You might have web traffic set with lower bandwidth than VPN traffic, but web traffic can be bursty or consume large amounts of bandwidth for long periods of time. CoDel is great for this because it keeps web traffic low latency and responsive for users attempting to browse while allowing good bandwidth utilization for file transfers.

    HFSC is not as good as a simple priority queue if your link bandwidth is not stable because HFSC does not dynamically adjust to changing bandwidth conditions.

    fq_CoDel will be that much better once available.



  • I am going to put this in place for my configs I use at LAN parties. Going to test at home then apply to the routers used for the LAN's.

    I use the following queues for high traffic - qGames and qHTTPSteam so using CODEL on them should have the desired effect as they are commonly the most used queues.

    Next LAN isnt' until March but I might be able to squeeze some testing in at a smaller event in Feb.



  • @Harvy66:

    fq_CoDel will be that much better once available.

    Have the developers indicated they plan on supporting fq CoDel in pfSense.  I have not seen any post that they are.  I probably should set up HFSC plus CoDel .  Today I just have CoDel applied to the lan and wan ports with no other traffic shaping. This way at least my traffic is dynamically reacting to congestion in the network.

    During prime time, I can see the slow down.  If I have a lot off file transfers happening, I can see the link speed up as it gets later in the night.  By about 10:30, my link is running a full speed and can pull a sustained 100/5 transfer either direction.

    Fq CoDel would be a great solution as it is adaptive.  I really hope it gets implemented.





  • @Harvy66:

    @switchman

    Yes, but no ETA

    https://forum.pfsense.org/index.php?topic=87931.msg485936#msg485936

    I saw that, but I read that as "yea it would be nice", but not firm "yes we are going to to this", now we just need to pick the release.



  • Time for a bounty!

    I wouldn't expect it before 2.2.3 at the earliest. Looks like a good backlog of bugs to stomp. It would be nice if it was added to Redmine as a feature request, just so it's official.



  • I take back what I said about Codel.  When I saturate the upload fully for a long period of time my pings still shoot through the roof (200+ ms).



  • @tuffcalc:

    I take back what I said about Codel.  When I saturate the upload fully for a long period of time my pings still shoot through the roof (200+ ms).

    Thats why the developers of CoDel say you should actually deploy FQ-CoDel.



  • @tuffcalc:

    I take back what I said about Codel.  When I saturate the upload fully for a long period of time my pings still shoot through the roof (200+ ms).

    Try settings your WAN rate to about 95% of your stable upload rate and see what happens.



  • @Harvy66:

    @tuffcalc:

    I take back what I said about Codel.  When I saturate the upload fully for a long period of time my pings still shoot through the roof (200+ ms).

    Try settings your WAN rate to about 95% of your stable upload rate and see what happens.

    That fixed it…



  • @tuffcalc:

    @Harvy66:

    @tuffcalc:

    I take back what I said about Codel.  When I saturate the upload fully for a long period of time my pings still shoot through the roof (200+ ms).

    Try settings your WAN rate to about 95% of your stable upload rate and see what happens.

    That fixed it…

    Well, 200ms is a lot less than what I thought it would have been, so just enabling CoDel without setting bandwidth is probably doing something, just not as effective as in conjunction with settings your bandwidth.

    Just wondering, what was your ping during saturation when you have your bandwidth set?



  • @Harvy66:

    @tuffcalc:

    @Harvy66:

    @tuffcalc:

    I take back what I said about Codel.  When I saturate the upload fully for a long period of time my pings still shoot through the roof (200+ ms).

    Try settings your WAN rate to about 95% of your stable upload rate and see what happens.

    That fixed it…

    Well, 200ms is a lot less than what I thought it would have been, so just enabling CoDel without setting bandwidth is probably doing something, just not as effective as in conjunction with settings your bandwidth.

    Just wondering, what was your ping during saturation when you have your bandwidth set?

    I've been all over the place and probably gave some false information (because I am new to this and didn't really understand what I was doing).  Anyway, let me try and clear up my findings:

    1. No traffic shaping, all bandwidth available - ping 10ms
    2. No traffic shaping, download saturated - ping 60ms
    3. No traffic shaping, upload saturated - ping 500+ms

    4. CODEL active with no bandwidth limit set - all pings same as scenarios 1,2 & 3.

    5. CODEL active (WAN side only) with bandwidth limit set at 98% of uplink speed, all bandwidth available - ping 10ms
    6. CODEL active (WAN side only) with bandwidth limit set at 98% of uplink speed, download saturated - ping 60ms
    7. CODEL active (WAN side only) with bandwidth limit set at 98% of uplink speed, upload saturated - ping 50 to 80ms

    So it does work, although during a saturated upload I notice my VoIP phone going a bit "robotic", but much improved over no traffic shaping (VoIP calls would be largely unusable).  Did not have that issue with PRIQ when prioritizing VoIP as first in line.  That being said, CODEL is much easier to setup and it looks like it doesn't completely starve other queues like PRIQ.  I'm sticking with CODEL.

    Interested to see if FQ_CODEL will make my pings even lower.  Will test it as soon as it is available in pfsense.



  • @tuffcalc

    Thanks for the info. I've done my fair share of "false" information with good intentions. It happens.

    Yes, the whole PRIQ replacement is kind of interesting. The eventual fq_codel will actually do similar things of PRIQ, but better for general usage. Except in the case where you need certain traffic to always get priority, fq_codel will do what most people want, keep latency low without the hassle of floating rules.


  • Rebel Alliance Developer Netgate

    @tuffcalc:

    @Harvy66:

    @tuffcalc:

    @Harvy66:

    @tuffcalc:

    I take back what I said about Codel.  When I saturate the upload fully for a long period of time my pings still shoot through the roof (200+ ms).

    Try settings your WAN rate to about 95% of your stable upload rate and see what happens.

    That fixed it…

    Well, 200ms is a lot less than what I thought it would have been, so just enabling CoDel without setting bandwidth is probably doing something, just not as effective as in conjunction with settings your bandwidth.

    Just wondering, what was your ping during saturation when you have your bandwidth set?

    I've been all over the place and probably gave some false information (because I am new to this and didn't really understand what I was doing).  Anyway, let me try and clear up my findings:

    1. No traffic shaping, all bandwidth available - ping 10ms
    2. No traffic shaping, download saturated - ping 60ms
    3. No traffic shaping, upload saturated - ping 500+ms

    4. CODEL active with no bandwidth limit set - all pings same as scenarios 1,2 & 3.

    5. CODEL active (WAN side only) with bandwidth limit set at 98% of uplink speed, all bandwidth available - ping 10ms
    6. CODEL active (WAN side only) with bandwidth limit set at 98% of uplink speed, download saturated - ping 60ms
    7. CODEL active (WAN side only) with bandwidth limit set at 98% of uplink speed, upload saturated - ping 50 to 80ms

    So it does work, although during a saturated upload I notice my VoIP phone going a bit "robotic", but much improved over no traffic shaping (VoIP calls would be largely unusable).  Did not have that issue with PRIQ when prioritizing VoIP as first in line.  That being said, CODEL is much easier to setup and it looks like it doesn't completely starve other queues like PRIQ.  I'm sticking with CODEL.

    Interested to see if FQ_CODEL will make my pings even lower.  Will test it as soon as it is available in pfsense.

    Out of curiosity, which exact places do you have CoDel active for? As the shaper discipline for the interface, or using HFSC with "Codel Active Queue" on specific queues? And if the latter, which queues?

    The more good/working examples we can get the better, I can put them up on the wiki for future reference.



  • @jimp:

    @tuffcalc:

    @Harvy66:

    @tuffcalc:

    @Harvy66:

    @tuffcalc:

    I take back what I said about Codel.  When I saturate the upload fully for a long period of time my pings still shoot through the roof (200+ ms).

    Try settings your WAN rate to about 95% of your stable upload rate and see what happens.

    That fixed it…

    Well, 200ms is a lot less than what I thought it would have been, so just enabling CoDel without setting bandwidth is probably doing something, just not as effective as in conjunction with settings your bandwidth.

    Just wondering, what was your ping during saturation when you have your bandwidth set?

    I've been all over the place and probably gave some false information (because I am new to this and didn't really understand what I was doing).  Anyway, let me try and clear up my findings:

    1. No traffic shaping, all bandwidth available - ping 10ms
    2. No traffic shaping, download saturated - ping 60ms
    3. No traffic shaping, upload saturated - ping 500+ms

    4. CODEL active with no bandwidth limit set - all pings same as scenarios 1,2 & 3.

    5. CODEL active (WAN side only) with bandwidth limit set at 98% of uplink speed, all bandwidth available - ping 10ms
    6. CODEL active (WAN side only) with bandwidth limit set at 98% of uplink speed, download saturated - ping 60ms
    7. CODEL active (WAN side only) with bandwidth limit set at 98% of uplink speed, upload saturated - ping 50 to 80ms

    So it does work, although during a saturated upload I notice my VoIP phone going a bit "robotic", but much improved over no traffic shaping (VoIP calls would be largely unusable).  Did not have that issue with PRIQ when prioritizing VoIP as first in line.  That being said, CODEL is much easier to setup and it looks like it doesn't completely starve other queues like PRIQ.  I'm sticking with CODEL.

    Interested to see if FQ_CODEL will make my pings even lower.  Will test it as soon as it is available in pfsense.

    Out of curiosity, which exact places do you have CoDel active for? As the shaper discipline for the interface, or using HFSC with "Codel Active Queue" on specific queues? And if the latter, which queues?

    The more good/working examples we can get the better, I can put them up on the wiki for future reference.

    It's active only for the shaper discipline for the WAN interface.


  • LAYER 8 Netgate

    It's active only for the shaper discipline for the WAN interface.

    OK.  Now I'm confused.  If codel has no knobs, what is actually doing the shaping based on the bandwidth value?



  • @Derelict:

    It's active only for the shaper discipline for the WAN interface.

    OK.  Now I'm confused.  If codel has no knobs, what is actually doing the shaping based on the bandwidth value?

    CoDel is not a traffic shaper, it does not do any rate limiting, that is the job of your rate limiter. All CoDel does is figure out which packets to dequeue next or which to drop.

    See my quotes from the makers of CoDel

    https://forum.pfsense.org/index.php?topic=88162.msg488220#msg488220


  • LAYER 8 Netgate

    Exactly.

    The question I have is why tuffcalc sees a difference in behavior when he sets a bandwidth value of 98% of his upload when his scheduler type is CODELQ?

    As I understand codel, it should make no difference, so what is actually doing the shaping?  Is it built into altq itself absent any HFSC, PRIQ, etc?



  • Ahh.. I assumed PFSense/FreeBSD had a built in interface rate limiter that functioned separately of queue management like AQMs or shapers. To me, they are all orthogonal problems. The interface can be rate limited and not care about how queues are implemented, how many queues there are, or which queue to dequeue next. Traffic shapers can decide which queues to pull from and how fast to dequeue those queues, and AQMs can decide which packet to dequeue from a given queue.

    Interface->InterfaceRateLimiter->TrafficShaper(HFSC)->AQM(CoDel)

    Like a pipeline, with none needing to know anything about the others. Not to say this is how it is implemented, but it could be implemented this way as none of them have any overlap or unique dependencies on each other. Some traffic shapers may need to know how fast to expect the interface to dequeue from them.  HFSC might not actually be really "limiting", it might be just "scheduling", while the interface limiter might be doing the limiting. I'm not sure about implementation details, just looking at it from an abstract point of view and what could be done. I am ignoring the fact that HFSC does have the notion of an upper bound, but ignoring that, pretty much all other features of HFSC could be done without any rate limiting, only scheduling.


  • LAYER 8 Netgate

    This is what I get when I set scheduler type to CODELQ and a 900Mbit bandwidth:

    altq on  em0_vlan1003 codelq bandwidth 900Mb queue

    Absent any other requirements, that is probably sufficient to eliminate any buffer bloat one might have at their ISP (assuming bandwidth set to about 90-95% of reasonably-expected uplink.)



  • This is a very informative thread, I've been searching this forum on general traffic shaping methods / tricks and I've bookmarked a couple of threads including this one…I'm yet do to my setup, I like doing the research first so I can have an idea of what I'm dealing with.

    I have a question tho'; from what I read CoDel is only meant to work (well) with high speed connections? 2.5mbit and above?. I'm currently using a very low ADSL connection 1770kbps down 550kbps up (thats what I get from speedtest although the ISP sells as 2mbit down 1mbit up), would CoDel work for me with this speeds?

    I'm looking to upgrade to 10mbit down 2.5mbit up soon but I kindly need to know if CoDel would work for me with my current speeds.

    Another question is; I'm I right to assume that CoDel can work side to side with PRIQ and the Limiter (I intend to use the Limiter to implement share bandwidth evenly on LAN as foxale08 has illustrated here; https://forum.pfsense.org/index.php?topic=63531.0 )

    Regards.



  • @cmutwiwa:

    This is a very informative thread, I've been searching this forum on general traffic shaping methods / tricks and I've bookmarked a couple of threads including this one…I'm yet do to my setup, I like doing the research first so I can have an idea of what I'm dealing with.

    I have a question tho'; from what I read CoDel is only meant to work (well) with high speed connections? 2.5mbit and above?. I'm currently using a very low ADSL connection 1770kbps down 550kbps up (thats what I get from speedtest although the ISP sells as 2mbit down 1mbit up), would CoDel work for me with this speeds?

    I'm looking to upgrade to 10mbit down 2.5mbit up soon but I kindly need to know if CoDel would work for me with my current speeds.

    Another question is; I'm I right to assume that CoDel can work side to side with PRIQ and the Limiter (I intend to use the Limiter to implement share bandwidth evenly on LAN as foxale08 has illustrated here; https://forum.pfsense.org/index.php?topic=63531.0 )

    Regards.

    Answer to your first question: No (but it couldn't hurt to try… :))

    If you check CoDel's official site you would see that there are a few known problems. Here is a quote from  http://www.bufferbloat.net/projects/codel/wiki/Wiki/

    At very low bandwidths (e.g. .5Mbps) on ADSL, we're having to play with the target; Kathie did not have to in her simulations. This is due to inevitable buffering
    in htb or in the device driver. We have a version under development that does bandwidth limiting without buffering an extra packet, called cake. It's looking good so far.

    The quote mentions "playing with the target", but this level of configuration is only available in the Linux kernel's Codel/fq_codel implementation (please correct me if I am wrong). I am assuming they mean target delay. Please refer to to the following link for more information:  http://man7.org/linux/man-pages/man8/tc-fq_codel.8.html

    Regarding your second question; I have only used Codel as a stand-alone queue, never as part of another queueing algorithm like PRIQ or HFSC. I have no reason to think it wouldn't work though. Make sure to reset the firewall states and test your queueing configuration to confirm that it is working as expected.



  • I use Codel in conjunction with HFSC on my 100/100 connection. In theory they're independent of each other. HFSC schedules which queue to dequeue and Codel deiced which packets to dequeue or to drop.

    In practice, I never see any packets dropped according to the queue statistics, but I also never see any measurable latency spikes.

    I'm not sure of a goad way to load test because my connection is "too" stable.

    It is possible that if HFSC isn't "smooth" enough at interleaving the queues, a queue could wait longer than what Codel wants and could create packetloss when there isn't any real congestion, but HFSC seems like freaking magic. I assume bad things could happen with an older machine with 10ms timers or 10/40Gb interfaces that have fq_Codel tuned for 0.5ms, which is recommended for those rates. but right now, we do not have fq_codel nor do we have a way to tweak the target delay via the interface.



  • Here is how I have mine setup for testing right now:
















  • @Nullity:

    Answer to your first question: No (but it couldn't hurt to try… :))

    If you check CoDel's official site you would see that there are a few known problems. Here is a quote from  http://www.bufferbloat.net/projects/codel/wiki/Wiki/

    Thanks, will give it a try.

    I'm hoping it will work smoothly with PRIQ and the Limiter, I would achieve alot!



  • I have been experimenting with my home router which is running DD-WRT and enabled HTB along with FQ_CoDel. My latency under load has improved from 300ms+ to just under 20ms+ as measured by pinging Google DNS. The connection is a Frontier FIOS 25M/25M. My throughput as verified by speedtest.net are good whether I am using a Seattle based server or one in Atlanta. I thought that was great and looked into turning CoDel on in pfSense at work.

    We have two pfSense firewalls both in the Seattle area. One is sitting on a 50M/50M Comcast EDI circuit. The other one is sitting on a collocated burstable gigabit circuit that we shape to 100M/100M. Both use a simple PRIQ shaper with 500 packet queue limit, that's roughly a max queue delay of 114ms for 50M and 57ms for 100M. I have found that the 500 packet queue limit offers the best throughput performance with least drops and this has worked for us for a few years now.

    For most general traffic I really don't care that it queues up and gets delayed, but less intensive flows should not get delayed. Which I see CoDel doing on my home router. pfSsense was able to do the same on our work connections. However, something weird is going on when I do speed tests from the pfSense boxes. When I choose a local Seattle server with an RTT of under 10ms, both connections approach close to their shaped throughput. When I choose a server in Atlanta or Miami where the RTT is around 80ms, suddenly the upload throughput is 25-50% less, and it takes longer to get up to that throughput. Download throughput is never affected even though CoDel is enabled on the LAN queues and I verified that it is working via ping. If I turn off CoDel AQM, the upload for high RTT servers goes back to roughly shaped throughput.

    Can anyone explain why the longer RTT is causing issues for upload on pfSsense using CoDel. Why does DD-WRT not experience this issue?



  • Are you using the same computer when doing speed tests? I've seen large variations in speed tests between computers. Too bad PFSense doesn't have fq_codel yet, but there are some other awesome changes in the pipeline.



  • @Harvy66:

    Are you using the same computer when doing speed tests? I've seen large variations in speed tests between computers. Too bad PFSense doesn't have fq_codel yet, but there are some other awesome changes in the pipeline.

    I am using Windows Server 2008 R2 when testing the work connections. I am using Mac OS X 10.10.2 and Windows 8.1 Pro at home. I'll see if Windows 8.1 Pro makes a difference on the office connection tomorrow, though I doubt it.

    I am thinking that the queue length has something to do with it on pfSense.



  • @ltctech:

    @Harvy66:

    Are you using the same computer when doing speed tests? I've seen large variations in speed tests between computers. Too bad PFSense doesn't have fq_codel yet, but there are some other awesome changes in the pipeline.

    I am using Windows Server 2008 R2 when testing the work connections. I am using Mac OS X 10.10.2 and Windows 8.1 Pro at home. I'll see if Windows 8.1 Pro makes a difference on the office connection tomorrow, though I doubt it.

    I am thinking that the queue length has something to do with it on pfSense.

    If stream fairness is your goal then FAIRQ might be a better choice. There is a picture somewhere showing graphs of fq_codel, codel, SFQ (stochastic fair queue) and the latency changes when each algorithm is dealing with dozens of simultaneous streams. SFQ was a close 2nd behind fq_codel for best latency. FAIRQ is very similar to SFQ (both give each stream a hash then iterate through them round-robin style).

    Codel is lacking "fair queueing" (there are many papers on this topic) so it does poorly with multiple streams, unlike fq_codel.



  • @ltctech:

    @Harvy66:

    Are you using the same computer when doing speed tests? I've seen large variations in speed tests between computers. Too bad PFSense doesn't have fq_codel yet, but there are some other awesome changes in the pipeline.

    I am using Windows Server 2008 R2 when testing the work connections. I am using Mac OS X 10.10.2 and Windows 8.1 Pro at home. I'll see if Windows 8.1 Pro makes a difference on the office connection tomorrow, though I doubt it.

    I am thinking that the queue length has something to do with it on pfSense.

    There are a few things at play.

    1. If your queue is too small, it will drop packets too aggressively. You can look at queue statistics to find out if there are any drops happening.
    2. If your queue is too large and are not using something like Codel with time based dropping, your bandwidth can also be made less efficient
    3. My personal most common reason for poor upload speeds is the TCP stack of the OS I'm using

    Windows 202 R2 is the Win7 kernel, and Win7 defaults to some latency sensitive TCP congestion control. This may not be the same for the server edition, but when I switched to using CTCP, my upload bandwidth to higher latency targets increased substantially. Win8 of all versions default to CTCP.

    And never assume two similar machines would get the same performance. I had two identical computers, exactly the same hardware, both with a fresh install of Win7, and one was over 50% faster than the other for speed tests. The only thing I could think of that would make the difference was heuristics. Win7 tries to be "smart" about certain things, which can cause it to get confused. A quick trip into the registry to change some settings and a reboot and both systems were getting identical speedtests.

    So even freshly installed identical hardware can get large variations. ALWAYS test using the same machine. Or use an OS that doesn't suck. Freaking Windows.



  • @Harvy66:

    @ltctech:

    @Harvy66:

    Are you using the same computer when doing speed tests? I've seen large variations in speed tests between computers. Too bad PFSense doesn't have fq_codel yet, but there are some other awesome changes in the pipeline.

    I am using Windows Server 2008 R2 when testing the work connections. I am using Mac OS X 10.10.2 and Windows 8.1 Pro at home. I'll see if Windows 8.1 Pro makes a difference on the office connection tomorrow, though I doubt it.

    I am thinking that the queue length has something to do with it on pfSense.

    There are a few things at play.

    1. If your queue is too small, it will drop packets too aggressively. You can look at queue statistics to find out if there are any drops happening.
    2. If your queue is too large and are not using something like Codel with time based dropping, your bandwidth can also be made less efficient
    3. My personal most common reason for poor upload speeds is the TCP stack of the OS I'm using

    Windows 202 R2 is the Win7 kernel, and Win7 defaults to some latency sensitive TCP congestion control. This may not be the same for the server edition, but when I switched to using CTCP, my upload bandwidth to higher latency targets increased substantially. Win8 of all versions default to CTCP.

    And never assume two similar machines would get the same performance. I had two identical computers, exactly the same hardware, both with a fresh install of Win7, and one was over 50% faster than the other for speed tests. The only thing I could think of that would make the difference was heuristics. Win7 tries to be "smart" about certain things, which can cause it to get confused. A quick trip into the registry to change some settings and a reboot and both systems were getting identical speedtests.

    So even freshly installed identical hardware can get large variations. ALWAYS test using the same machine. Or use an OS that doesn't suck. Freaking Windows.

    You are right about the TCP stack differences. Windows 2008 R2 copes much worse than Windows 8.1 or Windows 2012 R2. Though even in Windows 8.1 where I checked that CTCP is enabled, I am getting an average upload of 40-45Mbits where it should be 48-49Mbits. So I guess CoDel is just not worth it, at least how its implemented on pfSense. Maybe they'll fix it when they do FQ_CoDel.



  • PFSense only implements the original Codel which has a large buffer length and has a target latency of 5ms. This allows it to do well if lots of small or large packets come through at the same time. One of the big issues with buffer bloat is if you buffer is too small you can drop small packets, but if your buffer is too large, then large packets cause too much back-log.

    fq_Codel extends this to include "fair" queuing which breaks up data flows into hash buckets and does a mixture of prioritizing packets arriving into empty buckets and dequeing back-logged buckets equally. Codel is still pretty much the best option for now. Set and forget.



  • Some have said CoDel is not a traffic shaper. This is confusing because CoDel drops packets to keep the buffers in check. Dropped TCP packets result in a throttling effect.

    Perhaps I am confusing a "traffic shaper" with a "traffic policer".
    http://www.cisco.com/c/en/us/support/docs/quality-of-service-qos/qos-policing/19645-policevsshape.html

    CoDel is one of the 2 though, right?

    I am confused. :o


  • Rebel Alliance Developer Netgate

    To oversimplify it quite a bit:

    Shaping can delay sending traffic (as well as drop) to smooth out usage, whereas policing simply lops off anything over the max rate and chucks it in the bit bucket.

    Shaping typically employs queues as well as the occasional drop, whereas policing just says "nope" and drops it hard if it crosses the high rate.

    Policing is very harsh, if you have ever had to deal with a circuit that had traffic policing, you know that both ends MUST have the same policing set or it's a nightmare of dropped packets. I haven't personally seen a circuit with traffic policing in probably 10 yrs or so, thankfully.



  • @Nullity:

    Some have said CoDel is not a traffic shaper. This is confusing because CoDel drops packets to keep the buffers in check. Dropped TCP packets result in a throttling effect.

    Perhaps I am confusing a "traffic shaper" with a "traffic policer".
    http://www.cisco.com/c/en/us/support/docs/quality-of-service-qos/qos-policing/19645-policevsshape.html

    CoDel is one of the 2 though, right?

    I am confused. :o

    Traffics shapers do not drop packets, they dequeue packets queues at specified rates. It's the queue's drop packets, but the traffic shaper's job to decide which queue and when.



  • @Harvy66:

    @Nullity:

    Some have said CoDel is not a traffic shaper. This is confusing because CoDel drops packets to keep the buffers in check. Dropped TCP packets result in a throttling effect.

    Perhaps I am confusing a "traffic shaper" with a "traffic policer".
    http://www.cisco.com/c/en/us/support/docs/quality-of-service-qos/qos-policing/19645-policevsshape.html

    CoDel is one of the 2 though, right?

    I am confused. :o

    Traffics shapers do not drop packets, they dequeue packets queues at specified rates. It's the queue's drop packets, but the traffic shaper's job to decide which queue and when.

    I think I understand what you are saying, but he post above you and the Cisco link both say that shapers drop packets. :o


  • Rebel Alliance Developer Netgate

    Shaping can drop but only by way of it dropping out of a queue. It still had to be queued, possibly delayed, etc.

    The only action of Policing is to drop, no queue.



  • Perhaps it is my confusion between incoming and outgoing egress. CoDel throttles (shapes?) incoming egress TCP streams based on queueing delay, but this queueing delay is controlled by outgoing egress speeds, which are controlled by the traffic-shaper.

    I should probably just head back to the books… :-X

    :D

    Edit: I am referring to WAN interface.



  • Codel is just a regular queue. Just like when the default queue gets full, it drops packets. The difference is the default queue does tail drops and does abrupt drops once full. Codel does head drops and defines full not as a number of packets but how long a packet was in the queue, even then, it doesn't do abrupt drops does does ever increasing rates of drops.

    It is impossible to have a network interface without a queue, even if it's a queue of one. The whole point of a queue is to buffer packets. Codel does so in a way that reduces buffer bloat while allowing high throughput relative to the default fixed-size tail-drop that has been around for decades.

    When writing multi-threaded code, you use queues a lot because synchronizing threads is expensive and you rarely have two threads that process data at the same rate. You need to buffer that data somewhere. Queues!


Log in to reply