CoDel - How to use



  • Like Nullity mentioned, the default values were chosen because they work best for the bulk of users. They are a one size fits all that are not optimal for all users, but is still better than FIFO.

    The optimal interval should be the lowest value that covers the bulk of the RTTs of your flows. For me the default 100ms is my latency to London and way too large for many of the 10ms-20ms servers that I communicate with, until they add the ability to change the interval, I can't complain that they're choosing the recommended defaults. But I will post some results of before and after once I'm allowed to upgrade.



  • I just realized that I should mention that I am using HFSC with CoDel is a child discipline.

    PFSense 2.2.3 - this is actually a fairly typical graph, so I didn't run it more than once

    PFSense 2.2.4 - I ran this test a few times, they all pretty much showed the same


    I see no real difference.



  • @Harvy66:

    I just realized that I should mention that I am using HFSC with CoDel is a child discipline.

    PFSense 2.2.3 - this is actually a fairly typical graph, so I didn't run it more than once

    PFSense 2.2.4 - I ran this test a few times, they all pretty much showed the same


    I see no real difference.

    Did you make sure that the values were different? I thought we stilll did not know how to display the live values of CoDel's params when it is a sub-discipline.

    Maybe we should attempt some real testing though. Local tests mesuring small time-spans, probably kernel timer debugging level of granularity. That DSLreports test is great for introducing regular folks to bufferbloat, but it is not accurate. HTML5 within a browser just cannot perform well enough, especially with your 10ms-latency connection as the test.

    I tried to set ALTQ to debugging state on pfSense then FreeBSD but I never got far. That is my best guess for a way to measure CoDel with enough accuracy to be useful. Any ideas?



  • @Nullity:

    @Harvy66:

    I just realized that I should mention that I am using HFSC with CoDel is a child discipline.

    PFSense 2.2.3 - this is actually a fairly typical graph, so I didn't run it more than once

    Removed images

    PFSense 2.2.4 - I ran this test a few times, they all pretty much showed the same

    Removed images

    I see no real difference.

    Did you make sure that the values were different? I thought we stilll did not know how to display the live values of CoDel's params when it is a sub-discipline.

    Maybe we should attempt some real testing though. Local tests mesuring small time-spans, probably kernel timer debugging level of granularity. That DSLreports test is great for introducing regular folks to bufferbloat, but it is not accurate. HTML5 within a browser just cannot perform well enough, especially with your 10ms-latency connection as the test.

    I tried to set ALTQ to debugging state on pfSense then FreeBSD but I never got far. That is my best guess for a way to measure CoDel with enough accuracy to be useful. Any ideas?

    Like you said, last time I tried to check the values as a child discipline, it didn't show them. I figured if the defaults were changed for the scheduler, it may have affected these as well, but I have a hard time telling.

    If I bypass PFSense and go strait to the Internet, I get a very distinctive bufferbloat on the DSLReports tests.

    You can also see with this that if I change from 16 streams to 32 streams, it starts to tax the simple CoDel algorithm's ability to smooth out latency.

    This is what my connection looks like when I bypass PFSense, which means no CoDel. Same 16 streams.

    It's hard to see because of the large range, but the base idle ping is 17ms avg, the download is 27ms avg, and the upload is 44ms avg. That is distinctly different than what I get through PFSense with HFSC+CoDel, which was +2ms over idle instead of +10-27ms over idle. A magnitude difference.

    As you can tell, my ISP has horrible bufferbloat /sarc 30ms, most horrible.

    I'm not sure the best way to measure the affects of CoDel as a simple test. You'd probably need to use something to load a rate limited connection(not line rate), like iperf, then doing pings. My expectation is there should be a measurable difference between avg ping, and std-dev of ping.

    You may need to be careful doing this test on a LAN where the latency can be measured in microseconds. All TCP implementations have a minimum 2 segments sent when streaming data. 0.1ms ping at 1500byte segments sizes puts a lower limit of 120Mb/s. All known TCP congestion algorithms will not backoff below this speed for that latency. That's per stream. If you have 8 streams, that's 960Mb/s.

    0.1ms is actually a high latency for a LAN. I measure as low as 0.014ms using a high resolution ping program, but my switch is rated for 2.3 microseconds. It really depends on how often the kernel scheduling.



  • @Harvy66:

    I'm not sure the best way to measure the affects of CoDel as a simple test. You'd probably need to use something to load a rate limited connection(not line rate), like iperf, then doing pings. My expectation is there should be a measurable difference between avg ping, and std-dev of ping.

    You may need to be careful doing this test on a LAN where the latency can be measured in microseconds. All TCP implementations have a minimum 2 segments sent when streaming data. 0.1ms ping at 1500byte segments sizes puts a lower limit of 120Mb/s. All known TCP congestion algorithms will not backoff below this speed for that latency. That's per stream. If you have 8 streams, that's 960Mb/s.

    0.1ms is actually a high latency for a LAN. I measure as low as 0.014ms using a high resolution ping program, but my switch is rated for 2.3 microseconds. It really depends on how often the kernel scheduling.

    Network measurements are not what we want. We only care about the time before packets are on the wire. All CoDel controls is the scheduling of the local buffers, so that is what we would want to measure, right?

    CoDel has relatively zero influence over a packet's latency once the packet hits the wire.



  • I was thinking of a simple test that indirectly measures CoDel by the characteristics of the network. If you want a more direct measurement, one would need to implement some wrapper code that acts like the network stack and OS and simulates the network pushing packets through. More direct, much more work.



  • I just got fiber to my home (pfSense WAN plugged into an optical network terminal/modem) with symmetrical gigabit service.

    With the default pfSense install where there was no traffic shaping, I was receiving an F for BufferBloat and my speeds were only in the 600's Mbps on the DSLReports speed tests.  I found this thread and started to play with the codelq.

    So far, I have found that just enabling codelq without specifying any other values works great.  It got me from an F rating to A for BufferBloat.  I also played with the Queue Limit (target) by specifying 5 (pfSense defaults to 50).  The value of 5 resulted in more packet drops but no performance gain.  I also noticed that I get more BufferBloat on my downloads so I specified a bandwidth of 980Mbps on my LAN interface.  This seems to give me the best results.  In my speed tests, I see spikes above 1000Mbps and so I think my ISP fiber connection is faster than my gigabit LAN and that is why I needed to limit my LAN bandwidth to 980Mbps to get better overall performance.

    I am just running codelq on the WAN and LAN with no sub-queues.

    | | |



  • @mifronte:

    I see spikes above 1000Mbps and so I think my ISP fiber connection is faster than my gigabit LAN

    You shouldn't be able to see bursts above your LAN speed because that would require transferring data faster than your LAN. There are ways things may seem to burst, but it's probably just timing issues. The overall average should be fairly accurate.

    Cake will be awesome ones it comes out. I plan on dropping HFSC+codel and just using Cake.



  • @mifronte:

    I just got fiber to my home (pfSense WAN plugged into an optical network terminal/modem) with symmetrical gigabit service.

    With the default pfSense install where there was no traffic shaping, I was receiving an F for BufferBloat and my speeds were only in the 600's Mbps on the DSLReports speed tests.  I found this thread and started to play with the codelq.

    So far, I have found that just enabling codelq without specifying any other values works great.  It got me from an F rating to A for BufferBloat.  I also played with the Queue Limit (target) by specifying 5 (pfSense defaults to 50).  The value of 5 resulted in more packet drops but no performance gain.  I also noticed that I get more BufferBloat on my downloads so I specified a bandwidth of 980Mbps on my LAN interface.  This seems to give me the best results.  In my speed tests, I see spikes above 1000Mbps and so I think my ISP fiber connection is faster than my gigabit LAN and that is why I needed to limit my LAN bandwidth to 980Mbps to get better overall performance.

    I am just running codelq on the WAN and LAN with no sub-queues.

    | | |

    Based on my (many) tests - I think this is working properly for you because your ports are physically limited to 1Gbps.  If your internet upload speed is lower than your port speed, you need (in my experience) to limit your bandwidth speed in the traffic shaper to 95% of your upload speed for codel to properly work.

    Thanks for posting this!



  • So, 2.3's CODELQ (and assumedly the sub-discipline "CoDel Active queue" check-box version) have a CoDel with proper "interval" & "target" values (rather than reversed and/or wrong);```
    [2.3-RELEASE][admin@pfsense.wlan]/root: pfctl -vsq | grep codel
    altq on pppoe1 codel( target 5 interval 100 ) bandwidth 640Kb tbrsize 1492

    
    Hmm… anyone got some stats to share?


  • What kind of stats? I did a DSLReport speedtest after 2.3 and the bufferbloat part looks the same. Still A+.



  • In 2.3 it seems you have to enter a bandwidth value to enable CODELQ. Just to verify that I've understood correctly the information, I have a DSL connection with theoretical maximum of 24Mbs/2Mbs down/up and that is in actuality something like 20Mbs/1.4Mbs down/up. Based on what I read here I should set the WAN bandwidth to about 95% of that 1.4Mbs, is that right?



  • Correct. Good that it forces you to fill out your bandwidth, because it's useless if you just forward the packets at max rate.



  • I'm trying to enter 5.3 Mbps as the bandwidth value but I get a popup box saying "please enter a valid value. the two nearest values are 5 and 6" please help.



  • @vesikk:

    I'm trying to enter 5.3 Mbps …

    Change the multiplier from Mbit/s to Kbit/s.



  • I have a nominal 30/5 link that runs at ~ 40/6.
    I've tried a variety of bandwidth settings on the WAN link and I still get a C-D grades on the DSLreports "bufferbloat" grade and see high echo RTT when the link is loaded.

    What settings result in a DSLreports A grade?

    My previous experience was with Linux and fq_codel which worked perfectly.



  • Test your upstream speed a few times without any shaping and set the upstream bandwidth in the CODELQ settings to about 90-95% of the average value you got from the tests.



  • @bodosom:

    I have a nominal 30/5 link that runs at ~ 40/6.
    I've tried a variety of bandwidth settings on the WAN link and I still get a C-D grades on the DSLreports "bufferbloat" grade and see high echo RTT when the link is loaded.

    What settings result in a DSLreports A grade?

    My previous experience was with Linux and fq_codel which worked perfectly.

    As the other poster said, 90-95% is a safe choice. You can go even higher if your connection is stable.
    You might try doing your own tests, since they will likely be more accurate. Simply upload a file to an ftp (or any other service) and check the Quality graph in the Monitoring section of the pfSense GUI to see what your latency was during upload saturation.

    (FYI, fq_codel/codel only helps upload.)



  • @bodosom:

    I have a nominal 30/5 link that runs at ~ 40/6.
    I've tried a variety of bandwidth settings on the WAN link and I still get a C-D grades on the DSLreports "bufferbloat" grade and see high echo RTT when the link is loaded.

    What settings result in a DSLreports A grade?

    My previous experience was with Linux and fq_codel which worked perfectly.

    Settings on the WAN link only affect upload, not download. If your download is causing bloat, then you will need to also shape your LAN interface. DSLReports does give separate graphs for up and down, but a single grade that represents both.



  • I have tried using codelq and it works but as soon as I start an upload to google drive I get about 50% packetloss and then the gateways appear offline according to pfsense but ping stays low.



  • CoDel does not work well below 1Mb/s. If you have less than this, you may be better off trying FairQ. The problem is the 1500mtu is too large for such a slow link.  At 1Mb/s, a single 1500 byte packet will take about 12ms, which quickly triggers codel's drop mode.



  • @bodosom:

    I've tried a variety of bandwidth settings on the WAN link and I still get a C-D grades on the DSLreports "bufferbloat" grade and see high echo RTT when the link is loaded.

    What settings result in a DSLreports A grade?

    It seems I was unclear.  Ideally I'd like to get back to getting an A or B on the DSLR test which I suspect means having an average latency of 100-200 ms. in both directions.  I can get a B by carefully tweaking the test parameters after setting the WAN speed to ~90% of my uplink speed.  I guess I was hoping for something both simpler and better.


  • LAYER 8 Netgate

    OK, look.

    pfSense does not have a problem with buffer bloat.

    CoDel locally does practically nothing.

    People are likely seeing buffer bloat improvement by simply setting the bandwidth lower than their expected upload so buffer bloat does not occur in the ISP gear.

    CoDel is for transit, not for the edge.

    If you were to start increasing queue length in local shapers AND enabling CoDel you might derive some benefit. But the only reason you would derive benefit from CoDel is because you increased the queue length so just don't.



  • @bodosom:

    @bodosom:

    I've tried a variety of bandwidth settings on the WAN link and I still get a C-D grades on the DSLreports "bufferbloat" grade and see high echo RTT when the link is loaded.

    What settings result in a DSLreports A grade?

    It seems I was unclear.  Ideally I'd like to get back to getting an A or B on the DSLR test which I suspect means having an average latency of 100-200 ms. in both directions.  I can get a B by carefully tweaking the test parameters after setting the WAN speed to ~90% of my uplink speed.  I guess I was hoping for something both simpler and better.

    Like Harvy66 said, the grading score is a combination of your upload and download bufferbloat. You have not yet told us whether the majority of your bufferbloat is occurring during upload or download saturation. The test should show you specifically what your upload and download bufferbloat is in millisecond measurements.

    Also, like I already mentioned, you will get a more accurate measurement if you look at the pfSense Quality graphs (or you could simply run a ping test yourself…). The dslreports test is a program within a browser, so it is not always accurate. dslreports says my latency is ~5x worse than it actually is...

    We need more info from you. For example, what is your latency is (in milliseconds) during idle, upload saturation, and download saturation, with codel & without?

    @vesikk:

    I have tried using codelq and it works but as soon as I start an upload to google drive I get about 50% packetloss and then the gateways appear offline according to pfsense but ping stays low.

    Where are you recording 50% packet loss? tcpdump? Queue Drops? ?



  • @Derelict:

    OK, look.

    pfSense does not have a problem with buffer bloat.

    CoDel locally does practically nothing.

    People are likely seeing buffer bloat improvement by simply setting the bandwidth lower than their expected upload so buffer bloat does not occur in the ISP gear.

    CoDel is for transit, not for the edge.

    If you were to start increasing queue length in local shapers AND enabling CoDel you might derive some benefit. But the only reason you would derive benefit from CoDel is because you increased the queue length so just don't.

    Hmm?

    Codel is for any network node that receives more traffic than it can immediately send, which is usually at the edge.
    Transit nodes (backbones) have almost no need for Codel, since they keep latency low by using the best possible option… always have excess bandwidth (~50%) available.
    The Codel authors specifically advise that it most useful at gateway/edge routers. They even spear-headed the creation of a firmware (CeroWRT) for consumer routers specifically.

    Without Codel, my ping hit 600ms during upload saturation (50 packet default queue depth).
    With Codel, during upload saturation, my queue is usually 1-4 packets deep with a ~50ms ping.

    I wouldn't say pfSense has a bufferbloat problem (no more than everyone else), but Codel does make a big difference in worst-case latency, that's a fact.


  • LAYER 8 Netgate

    I would disagree. On a high-speed connection with 50 buffers CoDel will do practically nothing. The benefit is being derived from not overloading the upstream (where the bufferbloat occurs) with the bandwidth setting.



  • @Derelict:

    I would disagree. On a high-speed connection with 50 buffers CoDel will do practically nothing. The benefit is being derived from not overloading the upstream (where the bufferbloat occurs) with the bandwidth setting.

    Yes, if you have your buffers set to 50, which is the default. I had the issue with my 100/100 connection that a buffer size of 50 was too small and it was dropped quite a few packets under sustained load. My ISP has a fully uncongested network and allows 1Gb microbursts from datacenters half-way across the USA. I know at least Google, and I think several others, configure their TCP stacks to transfer at full line rate for the first several packets. Their line rate is 10Gb-40Gb. My ISP does traffic shaping in their Cisco core router and it seems to have some anti-buffer bloat features built in, which not only gives me an A on DSLReports without any shaping on my end, but also allows short lived bursts through. When doing a WireShark on my WAN interface, a single TCP stream from Google will give me back-to-back packets at 1Gb/s for the first 100ms-250ms before the Cisco router starts to traffic shape. I'm only connected to my ONT via 1Gb Ethernet, so who knows how fast the actual burst is.

    Since I shaped my LAN to 100Mb/s, this 1Gb burst quickly fills my 50 packet queue and I get bursts of packetloss. Even worse is some of these TCP streams fully transfer in the 100ms-250ms window. Since I will ACK all of the packets, Google's TCP stack thinks I can handle the rate they're sending at. As devices make more requests to Google, I get more and more 1Gb bursts until my average utilization gets too high and the ISPs rate limiting starts to kick in.

    Maybe you're thinking, ohh, he's just getting flooded by an on-premises CDN. No. My traffic comes from a Level 3 to Google PoP 300 miles away. I even configured YouTube to pull from a European CDN in Germany. 130ms ping, less than 1ms of jitter, and still getting 1Gb bursts from them. I guess this is the price I pay for not having an uncongested ISP.



  • @Derelict:

    I would disagree. On a high-speed connection with 50 buffers CoDel will do practically nothing. The benefit is being derived from not overloading the upstream (where the bufferbloat occurs) with the bandwidth setting.

    Rate-limiting is a separate function (the function that avoids overloading upstream nodes) from CoDel. Technically, ALTQ rate-limiting is done by either the Token Bucket Regulator (interface rate-limiting) or the queueing discipline (HFSC, CBQ, FAIRQ).

    CoDel, to keep the network buffers small, simply drops packets depending on the amount of time packets are queued.

    Rate-limiting puts pfSense in control (avoids overloading upstream), then CoDel mitigates bufferbloat (by dropping packets). Two separate functions.



  • @Derelict:

    If you were to start increasing queue length in local shapers AND enabling CoDel you might derive some benefit. But the only reason you would derive benefit from CoDel is because you increased the queue length so just don't.

    I didn't increase the queue length.  Did something change it as a result using codel?
    In any case under some load situations codel has improved my performance.  Under others it doesn't but it doesn't seem to make things worse so it's a net win.



  • So I've read this thread, and given the suggestion in it a try.  However, when I enable Codel, I seem to lose about 20% of download speed (100mbps down to 75-80)… though it does solve the buffer bloat problem.

    Is this expected behaviour?

    ~Spritz



  • @Spritzup:

    So I've read this thread, and given the suggestion in it a try.  However, when I enable Codel, I seem to lose about 20% of download speed (100mbps down to 75-80)… though it does solve the buffer bloat problem.

    Is this expected behaviour?

    ~Spritz

    It should be limited to whatever bitrate you set.



  • @Nullity:

    It should be limited to whatever bitrate you set.

    Thank you for the response Nullity, I appreciate it.  I've read many posts over the last week or so from you and Harvy66.

    So it appears then if I'm seeing a drop in throughput, something isn't configured correctly… though I'm at a loss as to what it could be.  Assuming the kids go down easy tonight, I'll play with it further tonight.

    ~Spritz



  • What rate did you set on your LAN and WAN?



  • @Harvy66:

    What rate did you set on your LAN and WAN?

    105/9.5mbps… After multiple speedtests, as well as monitoring usenet downloads I get 117/11mbps.

    Again, thank you.

    ~Spritz



  • @Spritzup:

    @Harvy66:

    What rate did you set on your LAN and WAN?

    105/9.5mbps… After multiple speedtests, as well as monitoring usenet downloads I get 117/11mbps.

    Again, thank you.

    ~Spritz

    Are you using CODELQ or HFSC/CBQ/FAIRQ with the "Codel Active Queue" toggle enabled?



  • I've tried a few different ways to be honest.  I was hoping to figure it out before posting here with my tail between my legs.

    But here is what I've tried (note that the rates I used were consistent) –>

    1.
    CODELQ - Only

    2.
    HFSC - Parent
    CODELQ - Child

    FAIRQ - Parent
    CODELQ - Child

    CBQ - Parent
    CODELQ - Child

    Now 1-4 gave me very similar results, with only slight fluctuations in the throughput (1-2mbps).  When I tried putting in an artificially lowered rate, it honored it.

    I also changed the advanced settings to enabling net.inet.tcp.inflight.enable=1 which seemed to help smooth out ping spikes, and I disabled Hardware TSO offload as per a link posted by Harvy (see I really did read and try and figure this out on my own!).

    Lastly I ran the wizard using HFSC queue, and it seemed to work as expected (buffer bloat under control, correct rate).  However, two things I discovered that makes this a less than ideal situation -->

    1.  I don't like having to use a full blow QoS implementation without knowing why the simpler one didn't work.  Nor do I love the potential support overhead this introduces.
    2.  When I went into the individual queue's after running the wizard to enable Codel, I was unable to due to the fact that the bandwidth numbers were decimals.  It wouldn't let me save unless I put whole #'s (wherever a % was).  This is despite the fact that the wizard uses decimals... is this a bug?

    Thanks!

    ~Spritz

    PS - If something is unclear, or terminology is incorrect, I apologize.  I'm at work and don't have access to my pfsense box currently, so I'm pulling all this from memory.

    I also tried running the QoS wizard, and this seemed to work quite well.  It honored my set rate, as well as keeping my buffer bloat under control... but I'm not keen on just using the wizard unless I know why the other options didn't work.



  • Very interesting subjet indeed.

    Just a quick question: why, when I try to use sysctl net.inet.tcp.inflight in the console, I get

    sysctl: unknown oid 'net.inet.tcp.inflight': No such file or directory

    shouldn't I see it there?



  • So I get a lot of bufferbloat when DOWNLOADS saturate my link. Uploads don't seem to cause the bloat.

    I have codelq, and only codelq, turned on my WAN and LAN connections. I have a 300/20 connection. for WAN speed I set 17 Mb, for the LAN connections I put 1Gb.

    Is there anything that can really be done about the download bloat? My pfSense is also my central router with 3 VLANs, so I'm not sure I can do much on the LAN side as I don't want to ruin my inter-VLAN routing speeds.

    Jason



  • Set your LAN to 280Mb and see how it goes.



  • I'm also trying to battle buffer bloat by implementing CoDel. While it works well to eliminate the bloat, it also drops my download speed by 100Mbps. Here are the details.

    My link speed is 250/10

    dslreports speed tests without any shaping applied show speeds of 250/11, but I usually get a D or F for buffer bloat.

    When I activate CoDel on the LAN/WAN interfaces with bandwidth set to 9Mbps on the WAN and 230Mbps on the LAN, my buffer bloat grade goes to an A, but my download speed drops to 148Mbps. I've tried raising the LAN interface bandwidth up to 240 and 250Mbps but this doesn't increase the download speeds past 150Mbps.

    Why is CoDel taking such a big bite out of my download bandwidth? Is there better way to eliminate buffer bloat without such a big impact to download speed?

    Hardware:          Netgate SG-8860
    pfSense Version: 2.3.2-RELEASE-p1


Log in to reply