CoDel - How to use



  • Codel uses a target of 5ms, which at 768Kb/s is only 480bytes. This means a single 1500byte packet will cause Codel to want to start dropping packets. On my 100Mb connection, 5ms is 62,500 bytes, which is  nearly 42 1500 byte packets. A single 1500 byte packet is 5ms at 2.4Mb/s. May be best to recommend Codel to be only implemented on 3Mb/s+ connections. fq_Codel probably wouldn't fair much better for bandwidth utilization, but would do better for not dropping small packets that immediately followed a 1500byte packet.

    I think I remember reading that 10Mb+ is recommended for Codel, but I'm not sure if that was an official value or just an easy to remember number given as a rule of thumb.

    Anyway, 1500 bytes is way to large for slow connections when latency is an issue.

    edit: I think the 10Mb comment was in reference that 5ms is not optimal for connections below 10Mb, but not to say it won't work. I assume there is a lower bound where Codel is definitely not good, like the 657Kb someone else said they read or the 2.4Mb/s rate required to transmit 1500 bytes in 5ms.



  • I just tried the "Codel Active Queue" on my outgoing bulk HFSC queue and it worked like a charm. Dropped my average queue size from ~30 to ~1 and dropped my ping from ~600ms to ~50ms during a single stream upload.

    Now I need to test out FAIRQ with Codel check-marked to see how that setup deals with multiple concurrent streams.



  • Hmm… I just ran into an unexpected negative side-effect of CoDel. I knew it had some drawbacks. ;)

    I check-marked Codel on my WAN HFSC qBulk queue, that was configured to have an increased worst-case delay with link-share [0Kb, 25, 300Kb], so other packets would be prioritized. It also had a queue limit of 500, so that if things got bad, it could just queue up packets and let the delay climb, but. CoDel won't allow that, because it keeps the packet queueing delay at 5ms… right?

    Without much consideration, I excitedly chose to decrease the delay of the my greediest queue to resultingly decrease the delay of all other queues. Dumb... I think I should have taken the more direct route of exclusively decreasing the delay of the non-qBulk queues, leaving qBulk to become backlogged and delayed as it increasingly yields to packets with more priority.

    Small queues are not always the answer, apparently. :)



  • I have a 15m/1m DSL line and tried implementing CoDel in pfSense last night.  I was not able to detect any improvement, so I deleted the discipline; which in turn crashed pfSense and I had to reboot, which made me wonder if I did things right or not.



  • @Nonsense:

    I have a 15m/1m DSL line and tried implementing CoDel in pfSense last night.  I was not able to detect any improvement, so I deleted the discipline; which in turn crashed pfSense and I had to reboot, which made me wonder if I did things right or not.

    No, lol… I experience the same thing; Put a traffic-shaper queue on LAN, remove it... crash. Though, pfSense did unfreeze itself once, but it took a good 60+ seconds

    About your poor experience; CoDel or not, you should see latency improvement if you configured your pfSense gateway as the slowest hop in your route. Transferring egress queueing from your modem to pfSense and your ingress queueing from your ISP to pfSense, should always see some sort of improvement, unless you are lucky enough to have a great ISP.

    What were you results exactly?



  • Nullify wrote:

    "About your poor experience; CoDel or not, you should see latency improvement if you configured your pfSense gateway as the slowest hop in your route. Transferring egress queueing from your modem to pfSense and your ingress queueing from your ISP to pfSense, should always see some sort of improvement, unless you are lucky enough to have a great ISP."

    Could you please describe how to do that "for the complete idiot?"  Thanks.

    I tested the changes with the new speed test over at DSL Reports (which has a buffer bloat meter–don't know exactly how it works).



  • Just make sure that your pfSense router is the slowest part of your upload and download traffic.



  • I love that new speed test. Should I have not been seeding 30Mb/s while running this test? lololol



  • Nullify wrote:

    "Just make sure that your pfSense router is the slowest part of your upload and download traffic."

    So, in order to do that, should I use the traffic shaping wizard (where CoDel is not an option) to set my bandwidth (and leave all other settings on default, since I don't need to prioritize traffic), or should I use the CODELQ option under "By Interface?"

    Someone really should write the book, pfSense for Dummies.



  • You shouldn't even need to use the wizard. Just click on  your WAN interface, set your bandwidth, and set it to codel.

    Another thing you could try is to set your interface to fairq, create a child queue, set as the default, then configure that queue to be codel.



  • First run some download/upload tests to get a baseline. Subtract about 5-10%, and use thagt as your bandwidth for your manually setup WAN and LAN queues using CODELQ.

    Run down/upload test to confirm that you are seeing queues registering some packets.

    I was a pfSense/networking noob a few months ago. This forum, the official book, the official wiki, and the interweb in general has about a decade of pfSense knowledge just waiting to be read. :)



  • Thanks all, but as my upload speed seems to vary quite a bit, no changes I make to traffic shaping seem to have much of an effect.  I'll likely have to wait for a faster internet connection before I am able to put these tools to good use–I might have to wait quite a long time, given my ISP's disinterest in improving its service.



  • Set your upload speed 80% of your "normal" max speed. It will help more than not doing anything.



  • @Nonsense:

    Nullify wrote:

    "Just make sure that your pfSense router is the slowest part of your upload and download traffic."

    So, in order to do that, should I use the traffic shaping wizard (where CoDel is not an option) to set my bandwidth (and leave all other settings on default, since I don't need to prioritize traffic), or should I use the CODELQ option under "By Interface?"

    Someone really should write the book, pfSense for Dummies.

    +1 for pfSense for Dummies.

    Sometimes you just want a clear and concise example and not to spend hours, days or even weeks combing the forum for that needle in a haystack answer.



  • Just wait 2-3 days. The same questions get asked over and over.



  • O.K., I set my WAN to use CODELQ with an upload speed of 800 kb/s.  I, however, still get a "D" in relation to buffer bloat on the DSL Reports speed test. :P  Do I need to reboot pfSense for the traffic shaping to take effect (it does not say so)?



  • Can you show us your config? I assume you have something simple. Maybe just a screenshot of your WAN interface under Traffic Shaping would be good.

    I assume you have slightly more than 800Kb/s of upload bandwidth?



  • As per your request:




  • You need to check "Enable/Disable Discipline and its children".



  • Yes, of course, my oversight, thanks.  Now I get a "C" in relation to buffer bloat on the DSL Reports speed test.  It appears that download has the most effect on the score.



  • Do the same thing for your LAN interface, except your LAN bandwidth will be whatever your download speed is.



  • I was wondering if it could be possible for someone to create two new versions of codel in PFSense, one for low bandwidth connections and one for high bandwidth.

    Codel has two parameters. something like a target latency and an interval, which for normal codel is something like 5ms and 100ms. They recommend 0.5ms and 50ms for 40Gb NICs. We may want a less aggressive one for low bandwidth connections, like 1Mb, so maybe 30ms and 150ms. I have no idea what number, just something less aggressive than 5ms on a 1Mb link trying to push a 1500byte packet.



  • In theory, the only thing that needs to get changes are those two numbers.



  • "Do the same thing for your LAN interface, except your LAN bandwidth will be whatever your download speed is."

    Done, thanks, now my score is a "B."



  • You could try switching to fairq instead of codel and see how your milage goes.



  • looks like 20ms target may be better for cable modems with a 150 interval, because of how buffering and scheduling works with DOCSIS. The modem needs to see a certain amount of backlog in the buffer in order to schedule more time, so a small target latency can artificially reduce buffering too much for DOCSIS. This may or may not be an issue for the router but the modem. But it does bring up possibilities where different target latencies for codel would be nice.



  • @Harvy66:

    I was wondering if it could be possible for someone to create two new versions of codel in PFSense, one for low bandwidth connections and one for high bandwidth.

    Codel has two parameters. something like a target latency and an interval, which for normal codel is something like 5ms and 100ms. They recommend 0.5ms and 50ms for 40Gb NICs. We may want a less aggressive one for low bandwidth connections, like 1Mb, so maybe 30ms and 150ms. I have no idea what number, just something less aggressive than 5ms on a 1Mb link trying to push a 1500byte packet.

    If I read the source code properly, pfSense's CoDel implementation supports both "target" ("qlimit" in pfSense) and "interval" but not the GUI. I have no idea what the standard practice is for manually editing the pfSense pf.conf/rules.debug. Here is the pfSense source: https://github.com/doktornotor/pfsense-tools/blob/master/patches/RELENG_10_0/altq_codel.diff#L126
    (Is there no better way to show pfSense source-code than this repo?)

    I will test a 20ms target and see if it changes anything. I thought the problems with CoDel and <768Kbit  connections was caused by more than an undersized target value… cannot hurt to try and see though! :)



  • When I was looking at the CoDel source, I noticed that the default target and interval might be incorrect. It should be target=5ms and interval=100ms, but it is not on my end.

    Please SSH to your pfSense device and run "pfctl -vs queue | grep -i codel".
    I get "altq on em0 codel( target 50 interval 5) bandwidth 300Mb tbrsize 36000". Target should be "5" and interval "100".
    Here is the problem, or one of them; https://github.com/doktornotor/pfsense-tools/blob/master/patches/RELENG_10_0/altq_codel.diff#L1330
    "100" and "5" should be switched. I am unsure why my pfSense is showing values of "50" and "5", since I only see "100" and "5" in the source, but then again pfSense source-code is very confusing to me.

    I wonder if this bug has any affect…

    Anyone who is familiar with submitting bug-fixes, please do so. I will try to figure it out this weekend, if nobody else does.



  • @Nullity:

    When I was looking at the CoDel source, I noticed that the default target and interval might be incorrect. It should be target=5ms and interval=100ms, but it is not on my end.

    Please SSH to your pfSense device and run "pfctl -vs queue | grep -i codel".
    I get "altq on em0 codel( target 50 interval 5) bandwidth 300Mb tbrsize 36000". Target should be "5" and interval "100".
    Here is the problem, or one of them; https://github.com/doktornotor/pfsense-tools/blob/master/patches/RELENG_10_0/altq_codel.diff#L1330
    "100" and "5" should be switched. I am unsure why my pfSense is showing values of "50" and "5", since I only see "100" and "5" in the source, but then again pfSense source-code is very confusing to me.

    I wonder if this bug has any affect…

    Anyone who is familiar with submitting bug-fixes, please do so. I will try to figure it out this weekend, if nobody else does.

    Also, I have only figured out how to display the target/interval options for the stand-alone CODELQ configuration. (pfctl -vsq)
    I do not know how to display the values of target/interval when Codel is being used together with another scheduler like HFSC or PRIQ. (The "Codel Active Queue" check-box.)

    Any help would be dandy.



  • $ pfctl -vs queue | grep -i codel
    queue  qDefault on igb0 bandwidth 14.70Mb qlimit 4096 hfsc( codel default )
    queue  qHigh on igb0 bandwidth 29.40Mb qlimit 4096 hfsc( codel linkshare(48.02Mb 5 29.40Mb) )
    queue  qNormal on igb0 bandwidth 29.40Mb qlimit 4096 hfsc( codel )
    queue  qDefault on igb1 bandwidth 14.25Mb qlimit 4096 hfsc( codel default )
    queue  qHigh on igb1 bandwidth 28.50Mb qlimit 4096 hfsc( codel linkshare(46.55Mb 5 28.50Mb) )
    queue  qNormal on igb1 bandwidth 28.50Mb qlimit 4096 hfsc( codel )

    It doesn't say when I use it as a child queue instead of the scheduler.



  • @Harvy66:

    $ pfctl -vs queue | grep -i codel
    queue  qDefault on igb0 bandwidth 14.70Mb qlimit 4096 hfsc( codel default )
    queue  qHigh on igb0 bandwidth 29.40Mb qlimit 4096 hfsc( codel linkshare(48.02Mb 5 29.40Mb) )
    queue  qNormal on igb0 bandwidth 29.40Mb qlimit 4096 hfsc( codel )
    queue  qDefault on igb1 bandwidth 14.25Mb qlimit 4096 hfsc( codel default )
    queue  qHigh on igb1 bandwidth 28.50Mb qlimit 4096 hfsc( codel linkshare(46.55Mb 5 28.50Mb) )
    queue  qNormal on igb1 bandwidth 28.50Mb qlimit 4096 hfsc( codel )

    It doesn't say when I use it as a child queue instead of the scheduler.

    Yeah, I had to create the stand-alone CODELQ queue to test.I looked through pfctl's man page and some source code to track down where details were accessable to userland, but I only found the stand-alone queue.

    According to my grepping of the pfSense-tools repo, all Codel options are configured backwards. I will continue to verify and then submit a ticket on the bugtracker… I guess...  :o

    I would like to know how to configure and/or display the values of CODELQ when it is used as a sub-discipline.



  • Thanks for your work! More dedicated than I am. /salute



  • Bug tracker ticket has been submitted. Hopefully I interpreted all this crap properly.

    If you want to test some proper target/interval values and you are using stand-alone CODELQ queueing, you could use pfctl to see what your current pf.conf/rules.debug queue line is then switch "target" and "interval", either by a rules.debug modification or directly through pfctl. The change will be temporary, since the GUI will eventually over-write rules.debug.

    As I understand it, pfctl uses system calls (ioctl?) to get super special info from /dev/pf, and that is where CoDel's target/interval values are, but pfctl has no pre-programmed method of extracting the values when CoDel is acting as a secondary algorithm ("Codel Active Queue" checkbox) on top of the primary algorithm.

    Adding code to pfctl would be rather easy (I naively assume), so I may try to do that, just so we can have the ability to view/change CoDel regardless of it being the primary or secondary scheduling algorithm. Then again, most people probably do not care. CoDel is meant to be a self-configuring scheduler where you do not need to be an expert knob twiddler.

    The particulars of how a target of 50 and an interval of 5 were functioning seemingly well is interesting… lol, hopefully the "proper" values are better and not worse.  :P



  • I tested (ping tests while upload is fully saturated) CoDel in pfSense with proper target(20 for my slow cxn) and interval(100) values, but… it made no difference. I expected otherwise. I tested 5, 20, 30 and 50 as values for target, and for interval I tested 5 and 100. The (incorrect) default in pfSense is target=(50 or 100) and interval=5. If you guys would like to test the stand-alone CoDel with proper target/interval values, in /tmp/rules.debug, replace your current WAN queue lines (they start with "altq" or "queue") with the following line into /tmp/rules.debug and reload with "pfctl -vf /tmp/rules.debug". It is only a temporary change (config.xml, your standard GUI config, creates /tmp/rules.debug on the fly).
    Modify the "em0" parts to match your interface, and set the bandwidth to your usual.

    altq on em0 codelq( qlimit 5, interval 105 ) bandwidth 630Kb queue { root_em0 }
    Edit: To clarify, qlimit = target.

    Another strange thing I encountered during testing was that with CoDel, I had 5-12% of pings unreplied (tcpdump registered them as sent). I tested 3 times with HFSC and CoDelQ… every time, CoDel has ping loss while HFSC never had one dropped packet. By my understanding of CoDel's specifications, it should not have been dropping packets, especially ICMP.

    I am being premature, but I am thinking CoDel is not working properly. Here are some tcpdump and ping pastes that tell me the ping request was sent, but never replied, but how can CoDel cause someone to not reply? Any ideas? Am I interpreting tcpdump improperly?
    I just noticed that when I switch to CoDel, all my firewall rules are still assigned to non-existing HFSC queues. I wonder if that is causing probs...

    20:09:03.422774 AF IPv4 (2), length 88: (tos 0x0, ttl 63, id 17061, offset 0, flags [none], proto ICMP (1), length 84)
        173.187.x.x > 74.125.22.102: ICMP echo request, id 37395, seq 11, length 64
    20:09:03.451752 AF IPv4 (2), length 88: (tos 0x0, ttl 45, id 0, offset 0, flags [none], proto ICMP (1), length 84)
        74.125.22.102 > 173.187.x.x: ICMP echo reply, id 37395, seq 11, length 64
    20:09:04.424272 AF IPv4 (2), length 88: (tos 0x0, ttl 63, id 32553, offset 0, flags [none], proto ICMP (1), length 84)
        173.187.x.x > 74.125.22.102: ICMP echo request, id 37395, seq 12, length 64
    20:09:05.424160 AF IPv4 (2), length 88: (tos 0x0, ttl 63, id 45955, offset 0, flags [none], proto ICMP (1), length 84)
        173.187.x.x > 74.125.22.102: ICMP echo request, id 37395, seq 13, length 64
    20:09:05.480163 AF IPv4 (2), length 88: (tos 0x0, ttl 45, id 0, offset 0, flags [none], proto ICMP (1), length 84)
        74.125.22.102 > 173.187.x.x: ICMP echo reply, id 37395, seq 13, length 64

    PING google.com (74.125.22.102) 56(84) bytes of data.
    64 bytes from qh-in-f102.1e100.net (74.125.22.102): icmp_seq=1 ttl=44 time=45.8 ms
    64 bytes from qh-in-f102.1e100.net (74.125.22.102): icmp_seq=2 ttl=44 time=46.1 ms
    64 bytes from qh-in-f102.1e100.net (74.125.22.102): icmp_seq=3 ttl=44 time=61.8 ms
    64 bytes from qh-in-f102.1e100.net (74.125.22.102): icmp_seq=4 ttl=44 time=57.5 ms
    64 bytes from qh-in-f102.1e100.net (74.125.22.102): icmp_seq=5 ttl=44 time=55.7 ms
    64 bytes from qh-in-f102.1e100.net (74.125.22.102): icmp_seq=6 ttl=44 time=87.6 ms
    64 bytes from qh-in-f102.1e100.net (74.125.22.102): icmp_seq=7 ttl=44 time=33.0 ms
    64 bytes from qh-in-f102.1e100.net (74.125.22.102): icmp_seq=8 ttl=44 time=28.9 ms
    64 bytes from qh-in-f102.1e100.net (74.125.22.102): icmp_seq=9 ttl=44 time=73.3 ms
    64 bytes from qh-in-f102.1e100.net (74.125.22.102): icmp_seq=10 ttl=44 time=34.9 ms
    64 bytes from qh-in-f102.1e100.net (74.125.22.102): icmp_seq=11 ttl=44 time=29.7 ms
    64 bytes from qh-in-f102.1e100.net (74.125.22.102): icmp_seq=13 ttl=44 time=57.0 ms



  • Your tcpdump can see the packet but it does not mean that it made it out :)
    It certainly means that it was sent to the queue which in this case is Codel.

    You should have statistics of the codel queue tell you what it dropped and what it passed.



  • @ermal:

    Your tcpdump can see the packet but it does not mean that it made it out :)
    It certainly means that it was sent to the queue which in this case is Codel.

    You should have statistics of the codel queue tell you what it dropped and what it passed.

    Thanks ermal. Your insight is always welcomed. :)

    Quick question or two about CoDel when it is a sub-discipline;
    1. are target/interval configurable by the user?
    2. are CoDel's stats exposed to the user?



  • I have to check the code but normally even when is a sub-discipline all the stats are exported.
    When you have Codel as part of HFSC queue the queue stats you see are the ones of Codel book keeping ones.



  • Anyone figure out how to apply the patch?

    https://redmine.pfsense.org/issues/4692

    Keep getting "no file to patch" when testing, so clearly some steps missing on how to apply it.


  • Banned

    Testing how? You'd need the full pfsense-tools sources from github plus recompile the thing



  • The test button within pfsense>system>patches.

    So this is not a directly applicable patch to an installed pfsense which is what I needed to know.

    Thanks.


Log in to reply