Playing with fq_codel in 2.4



  • If you use limiters on 2.4 and check the system log you may have seen this pop up

    load_dn_sched dn_sched FIFO loaded
    load_dn_sched dn_sched QFQ loaded
    load_dn_sched dn_sched RR loaded
    load_dn_sched dn_sched WF2Q+ loaded
    load_dn_sched dn_sched PRIO loaded
    load_dn_sched dn_sched FQ_CODEL loaded
    load_dn_sched dn_sched FQ_PIE loaded
    load_dn_aqm dn_aqm CODEL loaded
    load_dn_aqm dn_aqm PIE loaded
    

    FQ_CODEL was added to FreeBSD in 11.0 in dummynet/ipfw, and since 2.4 is based on that we can enable it by hand without recompiling anything.

    Note: This doesn't look like it will officially be in 2.4 via the GUI, and may need more testing. Since we're messing around with the command line, bad things may happen so use at your own risk.

    Start with a recent 2.4 snapshot. Create two root limiters, Download and Upload, and put 95% your maximum values in bandwidth. Create two queues under each, say LAN and WAN. For LAN, selection destination addresses for mask and source addresses for WAN. Modify the default outgoing firewall rule to use WAN under "in" pipe and LAN under "out" pipe.

    This generates /tmp/rules.limiter with something like the following:

    pipe 1 config  bw 85Mb
    queue 1 config pipe 1 mask dst-ip6 /128 dst-ip 0xffffffff
    
    pipe 2 config  bw 9Mb
    queue 2 config pipe 2 mask src-ip6 /128 src-ip 0xffffffff
    

    and the firewall rule adds a "dnqueue( 2,1)" in /tmp/rules.debug for the outgoing lan rule.

    Without messing with php we can manually change this to fq_codel and have it persist across reboots and ruleset reloads.

    cp /tmp/rules.limiter to /root/rules.limiter

    I edited /etc/inc/shaper.inc as follows:

    4599c4599,4600
    <               mwexec("/sbin/ipfw {$g['tmp_path']}/rules.limiter");
    ---
    >               #mwexec("/sbin/ipfw {$g['tmp_path']}/rules.limiter");
    >               mwexec("/sbin/ipfw /root/rules.limiter");
    

    replace /root/rules.limiter with:

    pipe 1 config  bw 85Mb
    sched 1 config pipe 1 type fq_codel
    queue 1 config sched 1 mask dst-ip6 /128 dst-ip 0xffffffff
    
    pipe 2 config  bw 9Mb
    sched 2 config pipe 2 type fq_codel
    queue 2 config sched 2 mask src-ip6 /128 src-ip 0xffffffff
    

    replace your bandwidth numbers with your own

    Trigger a rule reload (disable, apply, reenable a rule) and kill states. Might want to run "ipfw pipe flush" before doing that. then verify in command line:

    [2.4.0-BETA][admin@pfsense.lan]/root: ipfw sched show
    00001:  85.000 Mbit/s    0 ms burst 0
    q65537  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
     sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
     FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
       Children flowsets: 1
    BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
      0 ip           0.0.0.0/0             0.0.0.0/0     1450  2175000 31 46500   0
    00002:   9.000 Mbit/s    0 ms burst 0
    q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
     sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
     FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
       Children flowsets: 2
      0 ip           0.0.0.0/0             0.0.0.0/0       21      840  0    0   0
    
    [2.4.0-BETA][admin@pfsense.lan]/root: ipfw queue show
    q00001  50 sl. 0 flows (256 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
        mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
    q00002  50 sl. 0 flows (256 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
        mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000
    

    fq_codel is running, and we're passing traffic. Cool.

    I tried using limiters a long time ago but had to stop due to some problems with dropped traffic probably relating to my hardware and igb. I then just had two interface shapers, WAN and LAN with CODELQ, set to 95% my upload and download. This stopped bufferbloat, but I noticed that most real traffic would actually be half of these values.

    So far this has been far superior to the altq CODELQ with some of the following observations from the top of my head:

    Downloads not randomly halved versus codel.
    Twitch streams don't buffer when under heavy load such as steam
    Two heavy bandwidth, multiple connection programs will share bandwidth evenly.
    No more "sendto: No buffer space available" for unbound
    Slight latency increase versus intermittent packet loss at load

    Works just as good as cake in openwrt/lede from my limited home testing.

    Some points:

    1. Since I haven't been able to use plain limiters until now, this may just be better performance due to dummynet just limiting my bandwidth instead of fq_codel actually shaping. But it seems to perform better than plain limiters with reaching my bandwidth values versus the default WF2Q+.
    2. Traffic isn't shown under queues, but 0.0.0.0/0 will show under ipfw sched, so I guess the traffic is still being shaped. I noticed this in the original dummynet aqm paper on the developers' website, so maybe it's by design.

    Discuss if you've tried this or have any input. If you use limiters I'm interested if you can actually measure a difference since I'm coming from altq.



  • ipfw -a list

    That shows your rules and you can see what is matching to validate you have your rules correctly.



  • Thanks for this post. :)

    PS - "Downloads not randomly halved versus codel." shouldn't be happening.



  • @Animosity022:

    ipfw -a list

    That shows your rules and you can see what is matching to validate you have your rules correctly.

    
    [2.4.0-BETA][admin@pfsense.lan]/root: ipfw -a list
    ipfw: retrieving config failed: Protocol not available
    
    

    dummynet is used with pf via "dnqueue" in pf rules which shows up in firewall rules via pfctl with limiters enabled.

    @Nullity:

    Thanks for this post. :)

    PS - "Downloads not randomly halved versus codel." shouldn't be happening.

    Again probably related to my hardware. speedtests would show the full limited speeds on altq but most downloads wouldn't even reach that. Oddities like: dslreports would max out but fast.com would top out to about 40 megabits, as well as downloads via multiple browsers. Works fine without altq now. I think it was related to the igb driver as on my 2440 all networking would sometimes die and require a reboot if I disable the altq codel. Probably fixed recently by https://github.com/pfsense/FreeBSD-src/commit/42a5f2897e93d1e42833eac551c64c1373119ff9 but I haven't touched it in a while as this setup has been working great.



  • I got all three A+ on dslreports, even having active porn downloading on qbittorent. 8)



  • I found that traffic equalization (share bandwidth evenly) works differently with FQ_CODEL. If I start speedtest without FQ_CODEL (pipe settings remain the same, only 'sched x config pipe x type fq_codel' line removed) on two LAN PCs, then I see full equalization, for 300Mbps link I get 150 on both PCs. If I activate FQ_CODEL it gives different result. I see some fluctuations, but the first PC started download always wins with at least 60% of accumulated bandwidth.



  • @w0w:

    I found that traffic equalization (share bandwidth evenly) works differently with FQ_CODEL. If I start speedtest without FQ_CODEL (pipe settings remain the same, only 'sched x config pipe x type fq_codel' line removed) on two LAN PCs, then I see full equalization, for 300Mbps link I get 150 on both PCs. If I activate FQ_CODEL it gives different result. I see some fluctuations, but the first PC started download always wins with at least 60% of accumulated bandwidth.

    These are the default sysctls which may need tweaking depending on traffic and bandwidth

    net.inet.ip.dummynet.fqcodel.limit: 10240
    net.inet.ip.dummynet.fqcodel.flows: 1024
    net.inet.ip.dummynet.fqcodel.quantum: 1514
    net.inet.ip.dummynet.fqcodel.interval: 100000
    net.inet.ip.dummynet.fqcodel.target: 5000
    

    Technical details can be found here: http://caia.swin.edu.au/freebsd/aqm/papers.html

    So far I found the default ok



  • I've played a bit, but I think that default are really OK.
    Now I am using only IPFW FQ_CODEL shaper and disabled ALTQ, this gives me about +4Mbps on 300Mbps bandwidth if I compare with ALTQ shaper tested maximum.
    So far, so good.



  • Thanks alot.  i have been looking for something like this for a while.  I plan to use it for a while to see how things go. ;D



  • interesting i may try this out at some point thanks for sharing the information.



  • I already had the limiters setup.  i Was looking on how to make it easier and i came up with this. Instead of modifying anything i just input the ipfw command to enable fq_codel and worked.

    Limiters:
    00001:  30.000 Mbit/s    0 ms burst 0
    q131075  50 sl. 0 flows (1 buckets) sched 65539 weight 0 lmax 0 pri 0 droptail
    sched 65539 type FIFO flags 0x0 0 buckets 0 active
    00002:  6.00 Mbit/s    0 ms burst 0
    q131076  50 sl. 0 flows (1 buckets) sched 65540 weight 0 lmax 0 pri 0 droptail
    sched 65540 type FIFO flags 0x0 0 buckets 0 active

    Queues:
    q00001  50 sl. 0 flows (256 buckets) sched 3 weight 1 lmax 0 pri 0 droptail
        mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
    q00002  50 sl. 0 flows (256 buckets) sched 4 weight 1 lmax 0 pri 0 droptail
        mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000

    ipfw sched 1 config pipe 1 type fq_codel
    ipfw sched 2 config pipe 2 type fq_codel
    ipfw sched show

    And Done.  While it wont survive a reboot i am sure i can set something up.



  • Survivng reboot and update also is that what I needed.
    Here is my patch (use System_patches package)

    
    --- shaper.inc	Mon Feb 20 18:14:04 2017
    +++ shaper.inc	Sun Mar 05 07:33:23 2017
    @@ -4596,7 +4596,8 @@
     				"net.inet.ip.dummynet.pipe_slot_limit" => $max_qlimit
     		));
     		file_put_contents("{$g['tmp_path']}/rules.limiter", $dn_rules);
    -		mwexec("/sbin/ipfw {$g['tmp_path']}/rules.limiter");
    +		#mwexec("/sbin/ipfw {$g['tmp_path']}/rules.limiter");
    +		mwexec("/sbin/ipfw /root/rules.limiter");
     	}
     }
    
    

    Also, remember, you need to reboot firewall manually after update is completed or disable/enable rule where you have limiters used, like in OP first post.




  • Regarding the sysctl defaults, this link is likely the most official source for details, particularly the "Parameters" section: https://tools.ietf.org/html/draft-ietf-aqm-fq-codel-06



  • One Sunday morning I have found that bufferbloat rating is B or even C  and no drops on my side. I've tried to play with bandwidth limiting and after changing it to twice smaller I got A rating again, looks like it's a problem on the ISP side. OK, I was thinking there is nothing to do, but why not to try to use delay instead of limiting bandwidth.
    SO I changed limiter config to
    pipe 1 config delay 0ms  for both pipes
    And looks like this did the trick, now I have A+ bufferbloat and A or A+ Quality ratings.
    Certainly, I need to do advanced tests before draw some conclusions, but it looks hopefully.



  • Can't WAIT for this to get into the UI.

    FQ_codel's fair queuing is incredible, and HFSC + CODEL, FAIRQ + CODEL and CODELQ in pfSense can't provide multi-bucket fair queuing nearly as well.

    I tested this using shellcmd so it will persist through reboots: "ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codel" runs on reboot, with limiters and firewall pipes configured in the UI. It performs just as good as Linux's fq_codel that I have running on LEDE, IPFire and a few other boxes. pfSense getting fq_codel and wireguard would let me move entirely to pfSense / BSD on the networking side :)



  • As for GUI I was thinking about building some package, but I am not any kind of php programmer and  the best would be mainstream implementation into pfsense by professionals, core team.
    We can also vote for bounty and see what happens.



  • It's literally an on/off setting, and a kernel module



  • @moscato359:

    It's literally an on/off setting, and a kernel module

    Not so simple. You need to enable limiters at least and use it in pf rule. So it's a lot of GUI and code change if we going to make it on the traffic shaper side. If we going to make it on the limiters side, then yes it's much more simpler, we need scheduler type selection and bandwidth OR delay limiting. Since I use delay limiting for pipe, it's not enough to use only bandwidth limit. 
    BTW delay limiting with 0ms gives me the best result with bufferbloat test, since enabled, I have tested it multiple times per day and it's always A/A+ regarding to ISP mainstream router load.
    The best thing that comes with delay setting is that you don't limit your traffic when it's really don't need to be limited. For example my real bandwidth varies from 250 to 300Mbit and sometimes to make it work without bufferbloat I need to limit bandwidth down to 100. I am not sure why delay limiting helps in this case but it really works at least with my ISP and I have no bandwidth limit on my side.



  • @w0w:

    @moscato359:

    It's literally an on/off setting, and a kernel module

    Not so simple. You need to enable limiters at least and use it in pf rule. So it's a lot of GUI and code change if we going to make it on the traffic shaper side. If we going to make it on the limiters side, then yes it's much more simpler, we need scheduler type selection and bandwidth OR delay limiting. Since I use delay limiting for pipe, it's not enough to use only bandwidth limit. 
    BTW delay limiting with 0ms gives me the best result with bufferbloat test, since enabled, I have tested it multiple times per day and it's always A/A+ regarding to ISP mainstream router load.
    The best thing that comes with delay setting is that you don't limit your traffic when it's really don't need to be limited. For example my real bandwidth varies from 250 to 300Mbit and sometimes to make it work without bufferbloat I need to limit bandwidth down to 100. I am not sure why delay limiting helps in this case but it really works at least with my ISP and I have no bandwidth limit on my side.

    Thanks for trying to explain it. When it comes to traffic-shaping, even from a user perspective (disregarding the developer implementation), rarely is anything as simple as "It's literally an on/off setting, and a kernel module".

    I've been guilty of back-seat driving myself… and I'm totally, fully, absolutely awesome.  ::)



  • Why wouldn't it be a check box next to where we already have codel, random, random in and out, and explicit congestion notification

    All of those things are already implemented.

    It's just a different control algorithm tied in at the same place



  • @moscato359:

    Why wouldn't it be a check box next to where we already have codel, random, random in and out, and explicit congestion notification

    All of those things are already implemented.

    It's just a different control algorithm tied in at the same place

    One big reason is because the area you're referring to is in the queues (ALTQ) section while fq_codel was implemented in limiters (dummynet) section.

    Why don't we "just" send humans to Mars? We already have robots there.

    Like I said, back-seat driving is easy.



  • @Nullity:

    @moscato359:

    Why wouldn't it be a check box next to where we already have codel, random, random in and out, and explicit congestion notification

    All of those things are already implemented.

    It's just a different control algorithm tied in at the same place

    One big reason is because the area you're referring to is in the queues (ALTQ) section while fq_codel was implemented in limiters (dummynet) section.

    Why don't we "just" send humans to Mars? We already have robots there.

    Like I said, back-seat driving is easy.

    Why is it under limiter, when the rest of them are under altq?



  • ALTQ and Limiters are two different systems. My understanding is ALTQ is PF traffic shaping and Limiters are IPFW traffic shaping. Two competing firewall systems that FreeBSD has.



  • @moscato359:

    @Nullity:

    @moscato359:

    Why wouldn't it be a check box next to where we already have codel, random, random in and out, and explicit congestion notification

    All of those things are already implemented.

    It's just a different control algorithm tied in at the same place

    One big reason is because the area you're referring to is in the queues (ALTQ) section while fq_codel was implemented in limiters (dummynet) section.

    Why don't we "just" send humans to Mars? We already have robots there.

    Like I said, back-seat driving is easy.

    Why is it under limiter, when the rest of them are under altq?

    I'm a bit unclear about what you're asking but if you are asking why fq_codel was implemented in dummynet rather than ALTQ you'd need to ask the devs: http://caia.swin.edu.au/freebsd/aqm/

    I'd like to know as well. Maybe they think ipfw/dummynet is more future-proof than ALTQ? I dunno…


  • Banned

    By default, fq_codel uses ECN.

    This often doesn't work properly for upload so you may need to try without it. For my config this meant using:

    ipfw sched 1 config pipe 1 type fq_codel ecn && ipfw sched 2 config pipe 2 type fq_codel noecn

    Swap ecn/noecn as needed depending on the order you created the limiters in.



  • Interestingly, on Linux, fq_codel is in mainstream kernel, and enabled by default now.no settings required.



  • @nallar:

    By default, fq_codel uses ECN.

    This often doesn't work properly for upload so you may need to try without it. For my config this meant using:

    ipfw sched 1 config pipe 1 type fq_codel ecn && ipfw sched 2 config pipe 2 type fq_codel noecn

    Swap ecn/noecn as needed depending on the order you created the limiters in.

    I know what are you talking about.
    https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/
    But FQ_CODEL revision was updated several times since this article was published and no official remarks about ECN and recommended settings in docs.
    I have read a lot and played a bit with ECN option, but in my case it have no effect directly. If anybody suggest some simple way to test ECN I will be much thankful.



  • @w0w:

    @nallar:

    By default, fq_codel uses ECN.

    This often doesn't work properly for upload so you may need to try without it. For my config this meant using:

    ipfw sched 1 config pipe 1 type fq_codel ecn && ipfw sched 2 config pipe 2 type fq_codel noecn

    Swap ecn/noecn as needed depending on the order you created the limiters in.

    I know what are you talking about.
    https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/
    But FQ_CODEL revision was updated several times since this article was published and no official remarks about ECN and recommended settings in docs.
    I have read a lot and played a bit with ECN option, but in my case it have no effect directly. If anybody suggest some simple way to test ECN I will be much thankful.

    You can use tcpdump to see whether ECN has been negotiated/used, then run downloads & uploads with ECN disabled/enabled to see if there's any difference in speeds and/or latencies.

    For me, it improved download (or was it upload? or both?) speeds by a few percent but over a few days of using ECN (Linux client /proc/sys/net/ipv4/tcp_ecn = 1) had a couple of sites completely fail to work so I set tcp_ecn back to it's default (2).

    Whether your pfSense router supports ECN is a separate condition from your client supporting it, so make sure to configure it appropriately on both.

    I only played with ECN very quickly so take my input with a grain of salt… ;)



  • @Nullity:


    For me, it improved download (or was it upload? or both?) speeds by a few percent but over a few days of using ECN (Linux client /proc/sys/net/ipv4/tcp_ecn = 1) had a couple of sites completely fail to work so I set tcp_ecn back to it's default (2).

    Whether your pfSense router supports ECN is a separate condition from your client supporting it, so make sure to configure it appropriately on both.

    I only played with ECN very quickly so take my input with a grain of salt... ;)

    Do you remember URLs of sites failed to work with ECN?
    I've seen some reports like "Measuring the State of ECN Readiness in Servers, Clients" and others too, all of them stated that there is some % of servers that have wrongly configured ECN and this is the real problem, even if percentage of those servers lowered over years, but the real quantity raised up, so the simplest way is to test ECN enabled FQ_CODEL against some of those " ECN-failed" sites.



  • Setting my bandwidth to 95% of my always results in about 20mb off of my total bandwidth in tests. It seems that to use this you have to take a bandwidth hit….



  • I have a 150Mb connection, I set my bandwidth to 99%, or 148.5Mb, and I get about 147.8Mb/s with speed tests. If you're losing more than a small faction of a percentage, it's because something is misconfigured, low quality network equipment, or you're dealing with very small amounts of bandwidth where dropping a single packet results in a sizable bandwidth difference.



  • @Harvy66:

    I have a 150Mb connection, I set my bandwidth to 99%, or 148.5Mb, and I get about 147.8Mb/s with speed tests. If you're losing more than a small faction of a percentage, it's because something is misconfigured, low quality network equipment, or you're dealing with very small amounts of bandwidth where dropping a single packet results in a sizable bandwidth difference.

    This is my experience as well. Only when I was beginning my traffic-shaping journey did I experience strange things like that. My assumption is that I was misconfiguring.

    I suppose it's possible that these algorithms incorrectly calculate bitrates but that is very unlikely since transmitting at the configured bitrate is perhaps the most fundamental aspect of any traffic-shaping algorithm.



  • @Harvy66:

    I have a 150Mb connection, I set my bandwidth to 99%, or 148.5Mb, and I get about 147.8Mb/s with speed tests. If you're losing more than a small faction of a percentage, it's because something is misconfigured, low quality network equipment, or you're dealing with very small amounts of bandwidth where dropping a single packet results in a sizable bandwidth difference.

    I also have 150mb connection and am running an i5 mini PC with PFsense. It seems like a simple configuration so I'm not sure what could actually be misconfigured but I'm not ruling it out. Any ideas?



  • @Harvy66:

    I have a 150Mb connection, I set my bandwidth to 99%, or 148.5Mb, and I get about 147.8Mb/s with speed tests. If you're losing more than a small faction of a percentage, it's because something is misconfigured, low quality network equipment, or you're dealing with very small amounts of bandwidth where dropping a single packet results in a sizable bandwidth difference.

    Full disclosure, I am running a VPN, but it pins at 147mb no matter what….until this config.



  • HeatmiserNYC
    So, with FQ_CODEL you have 130Mbps max, right? You said -20Mbps…
    The misconfiguration can be interference with other limiters or rules if you have used same limiter twice or more — I did not checked but it was possible in certain conditions.
    Also TS mentioned that this FQ_CODEL setup equalizes traffic and with VPN it can be a real problem if you have concurrent or even the same traffic on both.
    Anyway, I did tests some time ago and there was 1-2 Mbps difference with bandwidth limit, if we compare to traditional HFSC this is about twice less. Now I don't use bandwidth limit but delay limit that is set to 0ms, this causes FQ_CODEL scheduler to process all traffic by using only internal parameters, I think. Double check everything and if problem persists, please provide some configuration sample.



  • Cool, thanks for replying.

    Yes, I get about 125-130 down when I set my limiter to 143mb (95%). My connection without the limiter will tend to burst initially to a bit over 200mb according to testmy.net. I have a simple setup following the guide detailed in the first post.

    I use the VPN for all outbound traffic, it's not a separate situation.

    I have tried traffic shaping before and this has been true for any configuration I have ever tried. If I try to shape close to my line speed it takes about 20mb off the top. How do you not use a bandwidth limit? Adding a delay limit in the field doesn't take.

    Just need a successful example of this to get running…

    Again, thanks.



  • @HeatmiserNYC:

    Cool, thanks for replying.

    Yes, I get about 125-130 down when I set my limiter to 143mb (95%). My connection without the limiter will tend to burst initially to a bit over 200mb according to testmy.net. I have a simple setup following the guide detailed in the first post.

    I use the VPN for all outbound traffic, it's not a separate situation.

    I have tried traffic shaping before and this has been true for any configuration I have ever tried. If I try to shape close to my line speed it takes about 20mb off the top. How do you not use a bandwidth limit? Adding a delay limit in the field doesn't take.

    Just need a successful example of this to get running…

    Again, thanks.

    Perhaps your speed drop is related to overhead like VPN, TCP, etc. I assume you are referring to goodput bitrates?

    On downloads you will commonly see below the configured bitrate because each time you hit the limit pfSense will tell the sender to slow down below the limit. Personally, I found very little useful benefit by limiting downloads because my ISP has minimal bufferbloat and allowing them to do the rate-limiting gives me 100% speeds.



  • What about to try to move shaper/limiters from LAN side to VPN side firewall rules?



  • That's an idea, I'll give that a shot!



  • Yea, that didn't work.


 

© Copyright 2002 - 2018 Rubicon Communications, LLC | Privacy Policy