Playing with fq_codel in 2.4



  • @belt9:

    @Animosity022:

    To that point, in OPNSense, you would configure a limiter and "weight" your FW rules to prioritize what you wanted.

    It works the same way in pfSense. I weight my guest Network to 10% of my bandwidth.
    So if there is no lan traffic then guest can use all the bandwidth. When someone on lan starts using bandwidth then it will throttle guest all the way until they get down to 10% as necessary.
    It's great, limits without wasting bandwidth. Of course you can set hard limits as well if you need to.

    Apologies as I don't mean to state the obvious so don't read into other than a statement, there is always traffic going on so if the plan is to share out across a LAN.

    I always see some traffic going on which is specifically why I avoided equal sharing across my LAN and focused more on prioritizing hosts. All those Echos, ATVs and such are chatty :)



  • I don't think you're understanding.

    Example:
    On a 100/100 limiter.
    LAN is weight 90, Guest is weight 10.

    LAN is unused, background traffic only (let's say ~2Kbps) - Guest has up to 99998Kbps of bandwidth available.
    In short, guest is free to use as much of the available bandwidth as they want less whatever LAN is using (Guest can only ever take away 10% of the total available bandwidth from LAN. Likewise, LAN can only ever take away 90% of the total available from Guest).

    So, neither network will be limited at all until the pipe is full. The same principle is true for clients within each individual network.

    Equal sharing does not mean that your bandwidth is automatically divided up between the number of clients on the network and each is given a hard limit.
    I.e., 100Mbps limiter with 10 clients on the network automatically limits those clients to 10Mbps each all the time. That does not happen. That scenario would only ever happen if the pipe was full and ALL 10 clients were asking for >10Mbps simultaneously. The instant even one client backed off, that clients bandwidth would be distributed back out into the pool of available bandwidth.



  • @belt9:

    I don't think you're understanding.

    Example:
    On a 100/100 limiter.
    LAN is weight 90, Guest is weight 10.

    LAN is unused, background traffic only (let's say ~2Kbps) - Guest has up to 99998Kbps of bandwidth available.
    In short, guest is free to use as much of the available bandwidth as they want less whatever LAN is using (Guest can only ever take away 10% of the total available bandwidth from LAN. Likewise, LAN can only ever take away 90% of the total available from Guest).

    So, neither network will be limited at all until the pipe is full. The same principle is true for clients within each individual network.

    I understood what you said. I used the term "equally sharing bandwidth across all LAN connections" in my post and you repeated my example of weighting, which I said I used.



  • Ok I see.

    My point was that you made it sound like only opensense offered this feature, which is incorrect.



  • @belt9:

    Ok I see.

    My point was that you made it sound like only opensense offered this feature, which is incorrect.

    Ah, ok as that wasn't my point. I just wanted to share that both the FQ-Codel and HFSC/Codel work well when configured right and my findings with quite a bit of testing was that FQ-Codel was more efficient but not by much and I had working results with both.



  • My bad, my bad! It was a really late couple of nights haha.



  • Hi guys,

    Have been following the discussion on how to setup weights on the queues.  Wanted to go through an example to make sure I understand correctly:

    Let's assume I have 3 subnets (LAN1 - 3) and one guest network.  I'd like to make sure that when under load, no LAN (or guest network) can hog all the bandwidth.

    To set this up with limiters, I would:

    Create an upload and download limiter and then create under each:

    Download:  Create 4 queues (one for each subnet with weight 30, and one for the guest network with weight 10)
    Upload:  Create 4 queues (one for each subnet with weight 30, and one for the guest network with weight 10).

    Assuming I had a 100/100 connection, this would ensure that:

    With no load, any of the subnets including guest network could consume up to 100Mbit.
    Assuming the connection is maxed out, this will ensure the that LAN1 - 3 are limited to 30Mbit each, and the guest network is limited 10Mbit each.

    In the situation where e.g. only LAN1 and LAN2 are trying to use all the bandwidth, how would it work (i.e. not traffic on LAN 3 and guest network)?  Depending on on which subnet started using the bandwidth first, is either able to go up to 70Mbit as the other is guaranteed at least 30Mbit?

    Thanks in advance for your help and explanation I really appreciate it.



  • You got it, in the lan 1 and 2 only scenario it would go to 50 Mbps for each since they are weighted equally.

    The speeds will certainly have transient periods of assymetric throughput but will balance out.



  • @belt9:

    You got it, in the lan 1 and 2 only scenario it would go to 50 Mbps for each since they are weighted equally.

    The speeds will certainly have transient periods of assymetric throughput but will balance out.

    Thanks!  I configured everything as described and was able to test it out by running a speed test on the three LAN's concurrently.  Was a nice to see speeds adjusting so that every LAN got its faire share as determined by the weights, yet if the other two LAN's are busy the third LAN could still use all the bandwidth.

    Thanks again for the help - I think it's great how with proper traffic shaping one can really get the most out of a lower bandwidth connection, e.g. 50/50 or 75/75 will go a long way with proper shaping vs. spending extra $ to upgrade to more bandwidth to try to solve the problem.



  • The problem I'm having with fq_codel is shaping OpenVPN. It's not clear how best to apply fq_codel to OpenVPN for my setup.
    There are two options here.

    1. Apply fq_codel to the WAN firewall rule for OpenVPN. This works well for site-to-site VPNs. If I send highly-compressable data, then the LZ4 compression works and I get a higher throughput. Uncompressable data is shaped normally and works well. This doesn't work well for a road-warrior connection. When the road-warrior accesses the Internet, that traffic is not handled by fq_codel. If it saturates the link then it's like not have fq_codel at all.

    2. Apply fq_codel to the OpenVPN interface firewall rules. This breaks compression apparently as I couldn't get rates that exceeded the limiter speed.

    With the old codelq applied to WAN, it didn't seem to matter what I did, as it would always do a pretty good job of keeping latency under control with/without OpenVPN, highly-compressable data, etc. fq_codel does a better job but having to apply it to every firewall rule is a bit of configuration tangle.

    *Applying fq_codel to the WAN firewall rule for OpenVPN and sending highly-compressable data does introduce a lot of latency for me but still ok. It's much worse without fq_codel.
    For reference:
    Idle: 8ms, regular upstream saturation with fq_codel: 12-18ms, highly-compressable upstream saturation: 100ms, no fq_codel/codel upstream saturation: 1500ms.



  • I have two queues created under the "download" limiter and they show up in Limiter Info, but when I create the schedule only one queue gets added…

    Does the command "ipfw sched 1 config pipe 1 type fq_codel" need to be modified to tell it to include all queues? I'm trying to add a lower weight to the guest network.

    edit: one other observation, I followed the screenshots from post 121 but I needed to set the mask to match my subnets or multiple clients were clashing and still causing buffer bloat.

    ![Screen Shot 2017-10-28 at 10.40.24 PM.png](/public/imported_attachments/1/Screen Shot 2017-10-28 at 10.40.24 PM.png)
    ![Screen Shot 2017-10-28 at 10.40.24 PM.png_thumb](/public/imported_attachments/1/Screen Shot 2017-10-28 at 10.40.24 PM.png_thumb)
    ![Screen Shot 2017-10-28 at 10.40.08 PM.png](/public/imported_attachments/1/Screen Shot 2017-10-28 at 10.40.08 PM.png)
    ![Screen Shot 2017-10-28 at 10.40.08 PM.png_thumb](/public/imported_attachments/1/Screen Shot 2017-10-28 at 10.40.08 PM.png_thumb)



  • @Animosity022:

    I really don't get much difference. I was using OPNSense and fq_codel prior as it seemed to just work better for me.

    With the new release, I changed back and just use HFSC queues with codel checked and some very basic rules to make sure my gaming traffic is first and my non important (downloads for media and other odd plex related download stuff) is limited. Works like a champ.

    Only thing for me always comes back to making sure my upload and download limits match close to reality what I expect out of my link so I use 940 down and 880 on Verizon's Gigabit FIOS with 1000 queue. No drops and no bufferbloat that I've been able to make happen.

    I have been using ALTQ FAIRQ + Codel Active Queue Management on my 150/150 link along with the queue set to 1024 in the child queue. My question is, does it make more sense to set the queue in the Codel child or the FAIRQ parent? Will I see a performance difference?



  • @tibere86:

    I have been using ALTQ FAIRQ + Codel Active Queue Management on my 150/150 link along with the queue set to 1024 in the child queue. My question is, does it make more sense to set the queue in the Codel child or the FAIRQ parent? Will I see a performance difference?

    I've stuck with HFSC and codel on the child queues with a queue limit of 1000. Works perfect for me. I use a very simplistic setup as I only have a high, default, low queue and gaming/voip is high and my download/sync traffic is low. Everything is just defaults.



  • Hello,
    I try to configure the weight on the queus with fq_codel.
    I have a LAN, a WAN and a VPN (ipsec), I want to put a higher weight on the VPN (weight 90) and on my WAN a lower weight (weight 10).
    I use floating rules on the LAN interface to match my flows.

    My limiters :

    • Download : 1800kbit/s
    • Download_LOW : weight 10 (mask /24 with "destination addresses")
    • Download_HIGH : weight 90 (mask /24 with "destination addresses")
    • Upload :  1800kbit/s
    • Upload_LOW : weight de 10 (mask /24 with "source addresses")
    • Upload_HIGH : weight de 90 (mask /24 with "source addresses")

    Here is my /root/rules.limiter file:

    
    pipe 1 config  bw 1800Kb
    sched 1 config pipe 1 type fq_codel
    queue 1 config pipe 1 weight 10 mask dst-ip6 /128 dst-ip 0xffffff00
    queue 2 config pipe 1 weight 90 mask dst-ip6 /128 dst-ip 0xffffff00
    
    pipe 2 config  bw 1800Kb
    sched 2 config pipe 2 type fq_codel
    queue 3 config pipe 2 weight 10 mask src-ip6 /128 src-ip 0xffffff00
    queue 4 config pipe 2 weight 90 mask src-ip6 /128 src-ip 0xffffff00
    
    

    It works without fq_codel but with fq_codel, limiters work but not weight on queues, my queues are not used.

    When I do a "ipfw sched show", I do not understand why my queue 2 is present in the second limiter.

    
    # ipfw sched show
    00001:   1.800 Mbit/s    0 ms burst 0 
    q65537  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
     sched 1 type FQ_CODEL flags 0x0 0 buckets 0 active
     FQ_CODEL target 7ms interval 100ms quantum 2000 limit 10240 flows 2048 ECN
       Children flowsets: 2 1 
    00002:   1.800 Mbit/s    0 ms burst 0 
    [color]q00002  50 sl. 0 flows (256 buckets) sched 1 weight 90 lmax 0 pri 0 droptail[/color]
        mask:  0x00 0x00000000/0x0000 -> 0xffffff00/0x0000
     sched 2 type FQ_CODEL flags 0x0 0 buckets 0 active
     FQ_CODEL target 7ms interval 100ms quantum 2000 limit 10240 flows 2048 ECN
       Children flowsets: 4 3
    
    # ipfw queue show
    q00001  50 sl. 0 flows (256 buckets) sched 1 weight 10 lmax 0 pri 0 droptail
        mask:  0x00 0x00000000/0x0000 -> 0xffffff00/0x0000
    q00002  50 sl. 0 flows (256 buckets) sched 1 weight 90 lmax 0 pri 0 droptail
        mask:  0x00 0x00000000/0x0000 -> 0xffffff00/0x0000
    q00003  50 sl. 0 flows (256 buckets) sched 2 weight 10 lmax 0 pri 0 droptail
        mask:  0x00 0xffffff00/0x0000 -> 0x00000000/0x0000
    q00004  50 sl. 0 flows (256 buckets) sched 2 weight 90 lmax 0 pri 0 droptail
        mask:  0x00 0xffffff00/0x0000 -> 0x00000000/0x0000
    
    

    What is wrong ? Thanks for your help



  • I believe I have everything setup per post #120 and beyond. My output for ipfw sched show looks correct, shellcmd is all set, router rebooted. I'm testing on a network where no other traffic is going on except my computer. My buffer bloat for downloading on dlsreports has improved greatly (300+ down to 51ms avg) but I can't seem to get rid of the bufferbloat for the upload (value slides between 300-1000ms depending on the bandwidth value I select for the limiter). My connection is slow by comparison to most of the folks I've seen in thread (15 down / 1 up) - I'm just wondering if I will ever be able to completely dial out the bufferbloat on a slow link like mine or do I just need to keep experimenting with different bandwidth values.



  • @Cardnyl:

    I believe I have everything setup per post #120 and beyond. My output for ipfw sched show looks correct, shellcmd is all set, router rebooted. I'm testing on a network where no other traffic is going on except my computer. My buffer bloat for downloading on dlsreports has improved greatly (300+ down to 51ms avg) but I can't seem to get rid of the bufferbloat for the upload (value slides between 300-1000ms depending on the bandwidth value I select for the limiter). My connection is slow by comparison to most of the folks I've seen in thread (15 down / 1 up) - I'm just wondering if I will ever be able to completely dial out the bufferbloat on a slow link like mine or do I just need to keep experimenting with different bandwidth values.

    You should be able to. You are sacrificing some bandwidth cap so you don't get bloat. It's just finding that magical number for your connection. If you can't, your provider might be a little more sporadic than you think.



  • @Cardnyl:

    I believe I have everything setup per post #120 and beyond. My output for ipfw sched show looks correct, shellcmd is all set, router rebooted. I'm testing on a network where no other traffic is going on except my computer. My buffer bloat for downloading on dlsreports has improved greatly (300+ down to 51ms avg) but I can't seem to get rid of the bufferbloat for the upload (value slides between 300-1000ms depending on the bandwidth value I select for the limiter). My connection is slow by comparison to most of the folks I've seen in thread (15 down / 1 up) - I'm just wondering if I will ever be able to completely dial out the bufferbloat on a slow link like mine or do I just need to keep experimenting with different bandwidth values.

    You might have some trouble just getting it to work. I can't find where I read it but the codel algorithm has trouble working on connections at or below 1Mbps because the transmission time of an MTU sized frame is too close to the delay that codel uses for its inner-workings.

    In this link they talk about using codel with what they call really low speeds. I don't know what any of it means so good luck.
    https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/#tuning-codel-for-circumstances-it-wasn-t-designed-for



  • You could get some benefit FairQ.



  • This has to be the best thread on the forum right now. Helped me a lot.



  • Noob question here. I installed the Shellcmd package with hopes of having the following command ran at boot:

    ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codel
    

    I searched this forum and Googled, but could not find out how to use the Shellcmd package. I could not locate any new options in the Web UI once the package was installed. What am I missing?



  • @tibere86:

    Noob question here. I installed the Shellcmd package with hopes of having the following command ran at boot:

    ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codel
    

    I searched this forum and Googled, but could not find out how to use the Shellcmd package. I could not locate any new options in the Web UI once the package was installed. What am I missing?

    Refresh your browser window and under Services->Shell Command



  • @Animosity022:

    @tibere86:

    Noob question here. I installed the Shellcmd package with hopes of having the following command ran at boot:

    ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codel
    

    I searched this forum and Googled, but could not find out how to use the Shellcmd package. I could not locate any new options in the Web UI once the package was installed. What am I missing?

    Refresh your browser window and under Services->Shell Command

    Doh. That's the one thing I didn't try. Many thanks!



  • Hi
    It is possible to use queues (different weights) with fq_codel ?



  • So I have a pretty odd situation with fq_codel.
    I created a floating rule:

    Action - Match
    Interface - WAN
    Direction - out
    Address Family - IPv4
    Protocol - Any
    In / Out pipe - lan wan
    

    Now, when I do traceroute from a LAN machine, all the hops show up as the destination! If I disable this floating rule, then traceroute works normally. If it's enabled and I do traceroute from pfsense itself, that traceroute is fine. ICMP ping itself seems oddly unreliable because every first ping is lost.

    I have a regular firewall rule to allow IPv4 ICMP to the WAN address but disabling/enabling this does nothing. I also do manual outbound NAT but just the regular rules.

    What's causing this? As soon as I disable the out floating rule, traceroute works normally. The bad traceroute even has latency corresponding to what the hops would normally be. traceroute through a site-to-site VPN works normally (no floating rules for VPN).

    IPv6 traceroute is unaffected.

    Does anyone else who uses a floating rule like mine see this? Any known solutions?

    A LAN machine example traceroute:

    # traceroute -I forum.pfsense.org
    traceroute to forum.pfsense.org (208.123.73.70), 30 hops max, 60 byte packets
     1  uac.localdomain (192.168.112.1)  0.163 ms  0.143 ms  0.138 ms
     2  * 208.123.73.70 (208.123.73.70)  14.600 ms  15.180 ms
     3  208.123.73.70 (208.123.73.70)  15.136 ms  15.379 ms  15.528 ms
     4  * * *
     5  208.123.73.70 (208.123.73.70)  16.811 ms  16.837 ms  16.848 ms
     6  208.123.73.70 (208.123.73.70)  17.816 ms  16.261 ms  17.627 ms
     7  208.123.73.70 (208.123.73.70)  17.644 ms  19.906 ms  20.173 ms
     8  208.123.73.70 (208.123.73.70)  20.262 ms  19.962 ms  19.813 ms
     9  208.123.73.70 (208.123.73.70)  19.594 ms  19.867 ms  19.898 ms
    10  208.123.73.70 (208.123.73.70)  60.034 ms  59.998 ms  60.353 ms
    11  208.123.73.70 (208.123.73.70)  59.594 ms  55.994 ms  55.201 ms
    12  208.123.73.70 (208.123.73.70)  55.441 ms  55.469 ms  56.421 ms
    13  208.123.73.70 (208.123.73.70)  55.350 ms  57.401 ms  57.401 ms
    14  208.123.73.70 (208.123.73.70)  54.101 ms  55.283 ms  62.990 ms
    15  208.123.73.70 (208.123.73.70)  62.392 ms  62.218 ms *
    16  * * *
    17  * * *
    18  208.123.73.70 (208.123.73.70)  61.501 ms  62.231 ms  59.980 ms
    19  208.123.73.70 (208.123.73.70)  67.374 ms  68.414 ms  68.897 ms
    20  208.123.73.70 (208.123.73.70)  63.797 ms  74.896 ms  70.074 ms
    

    The same traceroute from pfsense itself:

    traceroute -I forum.pfsense.org
    traceroute to forum.pfsense.org (208.123.73.70), 64 hops max, 48 byte packets
     1  173-228-88-1.dsl.dynamic.fusionbroadband.com (173.228.88.1)  14.262 ms  20.740 ms  17.791 ms
     2  gig1-29.cr1.lsatca11.sonic.net (70.36.243.77)  17.422 ms  21.650 ms  14.505 ms
     3  * * *
     4  50.ae4.gw.pao1.sonic.net (50.0.2.5)  15.070 ms  21.048 ms  19.681 ms
     5  ae6-102.cr1-pao1.ip4.gtt.net (69.22.130.85)  14.544 ms  13.726 ms  21.730 ms
     6  xe-8-1-6.cr0-sjc1.ip4.gtt.net (89.149.142.18)  16.210 ms  14.649 ms  14.841 ms
     7  as6461.ip4.gtt.net (216.221.158.110)  18.493 ms  18.531 ms  20.437 ms
     8  ae16.cr1.sjc2.us.zip.zayo.com (64.125.31.12)  22.334 ms  20.133 ms  20.252 ms
     9  ae27.cs1.sjc2.us.eth.zayo.com (64.125.30.230)  60.536 ms  79.274 ms  54.876 ms
    10  ae2.cs1.lax112.us.eth.zayo.com (64.125.28.145)  55.359 ms  57.712 ms  63.770 ms
    11  ae3.cs1.dfw2.us.eth.zayo.com (64.125.29.52)  59.466 ms  64.721 ms  63.456 ms
    12  ae27.cr1.dfw2.us.zip.zayo.com (64.125.30.181)  60.663 ms  63.960 ms  59.337 ms
    13  ae11.er1.dfw2.us.zip.zayo.com (64.125.20.66)  58.965 ms  64.998 ms  61.659 ms
    14  ae8.er2.dfw2.us.zip.zayo.com (64.125.29.122)  64.904 ms  60.535 ms  59.003 ms
    15  te-6-1-aus-core-11.zip.zayo.com (64.125.32.202)  65.484 ms  63.259 ms  63.947 ms
    16  net64-20-229-170.static-customer.corenap.com (64.20.229.170)  65.048 ms  61.145 ms  59.658 ms
    17  gw1.netgate.com (66.219.34.173)  67.752 ms  63.029 ms  63.543 ms
    18  fw2.pfmechanics.com (208.123.73.4)  66.281 ms  68.793 ms  67.865 ms
    19  208.123.73.70 (208.123.73.70)  66.834 ms  66.319 ms  66.047 ms
    


  • @Elhazgul:

    Hi
    It is possible to use queues (different weights) with fq_codel ?

    I'm unsure about exactly what you are asking but ALTQ (pfSense "queues") only supports codel and dummynet (pfSense "limiters") supports fq_codel.

    You should probably create your own thread, including more details about your setup & intentions, if you need more help.



  • @johnpoz:

    I got asked in a PM to post some screenshots of my settings.. Figured post it here as reference.

    Just apply the in/out pipe to firewall rule on your interface.. So that these do not effect your intervlan traffic if you have any.  Put a rule above to allow access to your other vlans without the pipe's applied.

    These settings changed my bufferbloat tests on dslreports to A..

    Hello to all!

    I have followed these instructions, together with the fq_codel trick on /etc/inc/shaper.inc and /root/rules.limiter . All my tests on dslreports are A or A+ . Also the download speed between my computers is shared evenly but with a serios issue:

    When i download one file at one computer and one file at another computer, both from HTTP servers and a different server per computer, each computer takes the half speed which is what i want.

    When any of the two computers start downloading a second file, from the same server or another, then it gets more speed and the second computer download speed slows down. If it starts a third download the other computers gets even less speed.

    For example, when i download one file at each of the two computers, each one gets about 1.5 MB/s (my line maxes out at about 3 MB/s to 3.5 MB/s) . When any one starts a second download, the other computer gets about 1.2 MB/s and the other with the two downloads about 2.3 MB/s.

    I am searching for a lot of days and have not found a solution. What i need to achieve, is to evenly share the speed between my computers no mater how many simultaneous downloads each of them runs.



  • @tibere86:

    Noob question here. I installed the Shellcmd package with hopes of having the following command ran at boot:

    ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codel
    

    I searched this forum and Googled, but could not find out how to use the Shellcmd package. I could not locate any new options in the Web UI once the package was installed. What am I missing?

    Refresh your web browsers and it's under Services->Shellcmd



  • I figured I'd share my config as I spent some time today with little to do at work on converting over to fq_codel setup for my pfSense setup. I have a 1Gig Verizon FIOS line coming in which is rated at 940 down and 880 up. I have a pretty straight forward setup going as I only split up into 3 queues and basically high prioritize my games and VOIP to high and lower all my p2p / plex download traffic to everything else.

    I have the Shell Command to create the proper queue setup:

    https://i.imgur.com/k08PJQZ.png

    I have an upload and download limiters with 3 buckets at 880Mb/s and 940Mb/s respectively. In those queues, I have a high, default and low at a 75, 25, 5 weight.

    https://i.imgur.com/6JZTEXd.png

    https://i.imgur.com/6cDzTe5.png

    Source and Destination in the config gets a little squirrelly for me as I want to make sure I have a clear break in my upload and download traffic so I didn't select either there as I handle that in the rules config.

    I have a series of match floating rules with logging setup so I can validate. All shaping is selected on my WAN interface:

    https://i.imgur.com/HeMy45B.png

    My rules examples are a bit big so I linked them a little different:

    Default queue
    http://i.imgur.com/CQDQGcf.png

    http://i.imgur.com/CQDQGcf.png

    Low priority rule
    http://i.imgur.com/MDuvFFe.png

    http://i.imgur.com/CADTE77.pn

    For floating rules and pipes, the in and out are switched as noted in the help text. I did check that in my speed test as I can see the speeds are exactly what I expected. I noticed much better performance when compared with the other schedules stock in pfSense.

    My speedtest results made me happy:

    Edit 1: I seem to have a slight problem with matching my internal (Private) IPs properly. I've gotta do a little more testing to figure out why they aren't matching. My WAN rules work perfect though so it's a start. I just want to make sure I can get internal stuff matched as well.



  • @Animosity022:

    I figured I'd share my config as I spent some time today with little to do at work on converting over to fq_codel setup for my pfSense setup. I have a 1Gig Verizon FIOS line coming in which is rated at 940 down and 880 up. I have a pretty straight forward setup going as I only split up into 3 queues and basically high prioritize my games and VOIP to high and lower all my p2p / plex download traffic to everything else.

    I have the Shell Command to create the proper queue setup:

    https://i.imgur.com/k08PJQZ.png

    I have an upload and download limiters with 3 buckets at 880Mb/s and 940Mb/s respectively. In those queues, I have a high, default and low at a 75, 25, 5 weight.

    https://i.imgur.com/6JZTEXd.png

    https://i.imgur.com/6cDzTe5.png

    Source and Destination in the config gets a little squirrelly for me as I want to make sure I have a clear break in my upload and download traffic so I didn't select either there as I handle that in the rules config.

    I have a series of match floating rules with logging setup so I can validate. All shaping is selected on my WAN interface:

    https://i.imgur.com/HeMy45B.png

    My rules examples are a bit big so I linked them a little different:

    Default queue
    http://i.imgur.com/CQDQGcf.png

    http://i.imgur.com/CQDQGcf.png

    Low priority rule
    http://i.imgur.com/MDuvFFe.png

    http://i.imgur.com/CADTE77.pn

    For floating rules and pipes, the in and out are switched as noted in the help text. I did check that in my speed test as I can see the speeds are exactly what I expected. I noticed much better performance when compared with the other schedules stock in pfSense.

    My speedtest results made me happy:

    Edit 1: I seem to have a slight problem with matching my internal (Private) IPs properly. I've gotta do a little more testing to figure out why they aren't matching. My WAN rules work perfect though so it's a start. I just want to make sure I can get internal stuff matched as well.

    Are you sure you need both in and out direction floating rules? Mine seems to work with just outbound floating rule.



  • I'm definitely not sure as I was trying to figure out a way to match my private lan traffic to drop a few hosts specifically into my 'low' queue.

    I tried something a little different and deleted all my floating rules and made a few LAN rules and since they are PASS rules, they logged fine on my private IPs and looks to have dropped them in the proper queue.

    I also changed my WAN NAT rules to have a proper queue for in/out as well.

    I get the same results on speed tests and tried saturating my low traffic and my high/default worked fine.

    Going to give that a try for a few days and see how it works and I might write that up instead as it just seems easier to me than the floating rules.



  • Hi,

    just trying this on Pfsense 2.4.2 and 2.4.1 Release.

    Even i install this with  ipfw sched 1 config pipe 1 type FQ_CODEL && ipfw sched 2 config pipe 2 type FQ_CODEL

    the resulting scheduler type is WF2Q+ …

    ipfw sched show

    00001:  25.000 kbit/s    0 ms burst 0
    sched 1 type WF2Q+ flags 0x0 0 buckets 0 active
      Children flowsets: 1
    00002:  5.000 kbit/s    0 ms burst 0
    sched 2 type WF2Q+ flags 0x0 0 buckets 0 active
      Children flowsets: 2

    What im doing wrong ?



  • @pete35:

    Hi,

    just trying this on Pfsense 2.4.2 and 2.4.1 Release.

    Even i install this with  ipfw sched 1 config pipe 1 type FQ_CODEL && ipfw sched 2 config pipe 2 type FQ_CODEL

    the resulting scheduler type is WF2Q+ …

    ipfw sched show

    00001:  25.000 kbit/s    0 ms burst 0
    sched 1 type WF2Q+ flags 0x0 0 buckets 0 active
      Children flowsets: 1
    00002:  5.000 kbit/s    0 ms burst 0
    sched 2 type WF2Q+ flags 0x0 0 buckets 0 active
      Children flowsets: 2

    What im doing wrong ?

    You should see an error on the command line when you type it in as Unix/Linux is case sensitive. Make sure to use lower case for FQ_Codel:

    
    ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codel
    
    


  • Thank you, that worked!



  • I setup fq_codel using floating rules on another system and the same IPv4 traceroute/ICMP problem I mentioned earlier occurs.

    Anyone else who uses floating rules to match traffic for fq_codel, do you see IPv4 ICMP traceroute working properly?



  • @darkcrucible:

    @Cardnyl:

    I believe I have everything setup per post #120 and beyond. My output for ipfw sched show looks correct, shellcmd is all set, router rebooted. I'm testing on a network where no other traffic is going on except my computer. My buffer bloat for downloading on dlsreports has improved greatly (300+ down to 51ms avg) but I can't seem to get rid of the bufferbloat for the upload (value slides between 300-1000ms depending on the bandwidth value I select for the limiter). My connection is slow by comparison to most of the folks I've seen in thread (15 down / 1 up) - I'm just wondering if I will ever be able to completely dial out the bufferbloat on a slow link like mine or do I just need to keep experimenting with different bandwidth values.

    You might have some trouble just getting it to work. I can't find where I read it but the codel algorithm has trouble working on connections at or below 1Mbps because the transmission time of an MTU sized frame is too close to the delay that codel uses for its inner-workings.

    In this link they talk about using codel with what they call really low speeds. I don't know what any of it means so good luck.
    https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/#tuning-codel-for-circumstances-it-wasn-t-designed-for

    Actually this article does help quite a bit. There was also mention of adjusting some of these tuneables by the OP; I just didn't realize how critical it was until now. Here is the list:

    
    net.inet.ip.dummynet.fqcodel.limit: 10240
    net.inet.ip.dummynet.fqcodel.flows: 1024
    net.inet.ip.dummynet.fqcodel.quantum: 1514
    net.inet.ip.dummynet.fqcodel.interval: 100000
    net.inet.ip.dummynet.fqcodel.target: 5000
    

    The article you linked suggests the following for a slow connection like mine:
    net.inet.ip.dummynet.fqcodel.limit < 1000
    net.inet.ip.dummynet.fqcodel.quantum < 300
    ECN OFF

    Is the right way to make these adjustments by adding/modifying them in the GUI (System->Advanced->System Tunables)?



  • Your target should also be at least 1.5x how long it takes to send an MTU amount of data at your bandwidth. Cake does this automatically as they found general limit works well.

    The reason for this is you don't want a single MTU sized packet to trip the drop logic.



  • @Harvy66:

    Your target should also be at least 1.5x how long it takes to send an MTU amount of data at your bandwidth. Cake does this automatically as they found general limit works well.

    The reason for this is you don't want a single MTU sized packet to trip the drop logic.

    What is the correct way to set these targets?



  • I assume the "net.inet.ip.dummynet.fqcodel" settings you mentioned in your just prior post.



  • @Cardnyl:

    @Harvy66:

    Your target should also be at least 1.5x how long it takes to send an MTU amount of data at your bandwidth. Cake does this automatically as they found general limit works well.

    The reason for this is you don't want a single MTU sized packet to trip the drop logic.

    What is the correct way to set these targets?

    So I experimented with this a little bit, and changing the values below either in the System Tunables section or by editing loader.conf.local didn't seem to work because these fq_codel defaults don't appear to show up until fq_codel is actually loaded.  So if you are loading fq_codel using shellcmd this won't really work.

    
    net.inet.ip.dummynet.fqcodel.limit: 10240
    net.inet.ip.dummynet.fqcodel.flows: 1024
    net.inet.ip.dummynet.fqcodel.quantum: 1514
    net.inet.ip.dummynet.fqcodel.interval: 100000
    net.inet.ip.dummynet.fqcodel.target: 5000
    
    

    You can make it work, but you would have to load fq_codel using ipfw so the defaults show up, change the default values to what you want them to be using sysctl and then load fq_codel again with the new values.

    However, I think there is an easier way as you can change these values just by editing the ipfw command that loads fq_codel.

    From a link posted earlier in this thread:

    http://caia.swin.edu.au/freebsd/aqm/patches/README-0.2.1.txt

    From section 3, the syntax for loading fq_codel via ipfw is:

    
    ipfw sched x config [...] type fq_codel [target t] [interval t] [ecn | noecn] 
            [quantum n] [limit n] [flows n]
    
     where
        t is time in seconds (s), milliseconds (ms) or microseconds (us). The 
            default interpretation is milliseconds.
        n is an integer number
    
    

    The command to load fq_codel with the defaults (from earlier in the thread):

    
    ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codel
    
    

    Using similar syntax as above, in your case all you would have to change is:

    
    ipfw sched 1 config pipe 1 type fq_codel target 15 noecn quantum 300 limit 600 && ipfw sched 2 config pipe 2 type fq_codel target 15 noecn quantum 300 limit 600 
    
    

    This does the following:
    Set target to 15ms (you'll need higher than default of 5ms given the low upload on your 15/1 connection)
    Turn off ECN
    Set fq_codel quantum to 300
    Set queue size limit to 600 packets

    (feel free to tweak these values as necessary)

    Now of course, this all assumes that you have already created two limiters and queues underneath them.

    You can issue the command above from the command line and make it persistent between reboots using shellcmd.

    Hope this helps.



  • I also have a question for everyone following this thread:

    The command to enable fq_codel that has been shown in this thread to be:

    
    ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codel
    
    

    However, looking at the examples in 3.3 here:
    http://caia.swin.edu.au/freebsd/aqm/patches/README-0.2.1.txt

    Doesn't the command have to be appended to include setting the the scheduler on the queue, i.e.:

    
    ipfw sched 1 config pipe 1 type fq_codel && ipfw queue 1 config sched 1 && ipfw sched 2 config pipe 2 type fq_codel ipfw queue 2 config sched 2
    
    

    The above assumes only two queues, one under each limiter.

    Thanks in advance for the help.


Log in to reply