Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Playing with fq_codel in 2.4

    Scheduled Pinned Locked Moved Traffic Shaping
    1.1k Posts 123 Posters 1.5m Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P
      pfsvrb
      last edited by pfsvrb

      Thank you for checking on this, it is very strange that I am seeing these errors.

      Here is my current version:
      2.4.4-DEVELOPMENT (amd64)
      built on Tue Jul 10 06:09:20 EDT 2018
      FreeBSD 11.2-RELEASE

      Here is the output of a "php -i" command on this pfSense install:
      phpinfo()
      PHP Version => 7.2.7

      System => FreeBSD pfSense.pfvm.vbox 11.2-RELEASE FreeBSD 11.2-RELEASE #36 79c8a561b61(RELENG_2_4_4): Tue Jul 10 06:14:32 EDT 2018 root@buildbot3:/builder/ce-master/tmp/obj/builder/ce-master/tmp/FreeBSD-src/sys/pfSense amd64
      Build Date => Jul 4 2018 18:58:13

      And, the output of "php --version":
      PHP 7.2.7 (cli) (built: Jul 4 2018 19:00:07) ( NTS )
      Copyright (c) 1997-2018 The PHP Group
      Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies
      with Zend OPcache v7.2.7, Copyright (c) 1999-2018, by Zend Technologies

      A few other items to note, I had installed this from 2.4.4.a built on June 2nd, 2018. I then have done two upgrades, one to a build on June 30th, 2018, and another upgrade yesterday bringing the build up to the current version displayed here.

      It's also worth noting that I had previously enabled fq_codel using the previous threads listed above and using the ShellCMD function to enable fq_codel schedulers on reboot. I wonder if this somehow conflicted with the install of the newer version that had the GUI fq_codel shaping options? I will try a re-install tonight and do a backup restore and see if I can still duplicate the issue.

      This is a non-prod box on a VM that I use for testing before I roll out to real hardware. If there's anything else I can do to provide more logs/testing, I'm open to it.

      w0wW 1 Reply Last reply Reply Quote 0
      • w0wW
        w0w @pfsvrb
        last edited by

        @pfsvrb said in Playing with fq_codel in 2.4:

        wonder if this somehow conflicted with the install of the newer version that had the GUI fq_codel shaping options?

        I think no, but yes it's the best solution to try clean install...

        P 1 Reply Last reply Reply Quote 0
        • P
          pfsvrb @w0w
          last edited by

          @w0w I tried a clean install in a new VM. This is using a current build of 2.4.4.a downloaded from pfsense.org today.

          Unfortunately, I am seeing identical results. When I go to create a new limiter, it shows the same shaper.inc error.

          This is happening on a box stock install without a backup being restored. I've just installed pfSense, logged in, gone through the basic first time setup config wizard, and then clicked on Firewall/Traffic Shaper/Limiters "New Limiter"

          0_1531351946964_96d74fce-ed6e-4440-90e3-ab7fe55a1925-image.png

          0_1531352011533_d1cb9804-8b81-4a92-a9cf-078fea70fa97-image.png

          w0wW 2 Replies Last reply Reply Quote 0
          • w0wW
            w0w @pfsvrb
            last edited by w0w

            @pfsvrb
            I did the same and no error so far. It's definitely related to VM configuration, I think.

            Yep. Reproduced. Sorry used wrong VM at testing 😀
            Shame on me!

            1 Reply Last reply Reply Quote 0
            • w0wW
              w0w @pfsvrb
              last edited by

              @pfsvrb
              I think it's time to report it on redmine!

              mattundM 1 Reply Last reply Reply Quote 0
              • P
                pfsvrb
                last edited by

                Just as a followup, I downloaded the 7/16 2.4.4.a build and tested in my VM today, and this error is gone.

                The only oddity I noticed is that the speed limits specified seem to be ignored if in mbit/s. I had to use kbit/s values to get my limits enforced. However, this is a minor issue and it's great to be able to finally set FQ_Codel in the GUI. Great work guys!

                1 Reply Last reply Reply Quote 0
                • Z
                  zwck
                  last edited by

                  I just updated to the latest beta and i am struggling to understand how to use the GUI. I want to have a very basic setup.

                  Is this correct?
                  0_1531843865563_28665435-6e14-44da-915a-9b1d55a42e27-image.png
                  0_1531843894730_9c5838a1-c854-4bba-980d-2f0720316a25-image.png
                  0_1531843912112_68ff8a64-439b-497e-8961-8bbae08c6d03-image.png
                  0_1531843925580_29d38e1e-286d-4f98-b619-21b66cd5776d-image.png

                  and Lan rule
                  0_1531843949386_ca829890-4a7e-4233-984b-2d57baa7a985-image.png

                  P 1 Reply Last reply Reply Quote 0
                  • P
                    pfsvrb @zwck
                    last edited by

                    @zwck Yes, those settings look fine for a "set it and forget it" fq_codel implementation. How are your bufferbloat scores on dslreports.com/speedtest using the settings in those screenshots??

                    1 Reply Last reply Reply Quote 0
                    • Z
                      zwck
                      last edited by zwck

                      @pfsvrb
                      Its A all around, however i noticed that when checking ipfw sched show periodically during both phases (up and down) there are quite some dropped packages, maybe my NICs are not that great, they are intel but not the recommended 350s

                      00001: 900.000 Mbit/s    0 ms burst 0
                      q65537  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
                       sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
                       FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 NoECN
                         Children flowsets: 1
                      BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
                        0 ip           0.0.0.0/0             0.0.0.0/0     295104 439672423 275 410213 492
                      00002: 900.000 Mbit/s    0 ms burst 0
                      q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
                       sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
                       FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 NoECN
                         Children flowsets: 2
                        0 ip           0.0.0.0/0             0.0.0.0/0     2405   101107  0    0   0
                      
                      00001: 900.000 Mbit/s    0 ms burst 0
                      q65537  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
                       sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
                       FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 NoECN
                         Children flowsets: 1
                      BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
                        0 ip           0.0.0.0/0             0.0.0.0/0      745    31427  0    0   0
                      00002: 900.000 Mbit/s    0 ms burst 0
                      q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
                       sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
                       FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 NoECN
                         Children flowsets: 2
                        0 ip           0.0.0.0/0             0.0.0.0/0     614966 917451569 207 309100 616
                      

                      i am quite sure there are some errors in my config, i exchanged to the 350 intel nics and bufferbloat is back to F

                      1 Reply Last reply Reply Quote 0
                      • Z
                        zwck
                        last edited by zwck

                        @mattund would you mind checking the post above , because i can get a working fq_codel setup as well without the in and out queue, and just selecting download / upload in the lan rules. So i am kindof confused what a minimal fq_codel set up should look like.

                        mattundM 1 Reply Last reply Reply Quote 0
                        • mattundM
                          mattund @zwck
                          last edited by mattund

                          @zwck

                          I feel like that should work; I personally ended up doing floating rules on the WAN interfaces I need to shape (I have a cable/DSL setup). Now, given you have your shaping set to 900/900, maybe you need to increase the slot size/queue length? That's a pretty fast connection. Then again a while ago I think I refuted this and said that queue sizes literally don't matter... the internal behavior of this scheduler as it is on FreeBSD is not well documented, there are some research papers by a particular organization but besides that there's not much else.

                          Generally seeing drops is actually OK, perhaps even good, because FQ_CoDel should be favoring dropping some traffic over allowing it to sit in queue for a while; in my experience FQ_CoDel is a very "all or nothing" algorithm. This doesn't explain your F's though. I find it odd you're getting that while also seeing activity on the queues. Has to be some sort of parameter issue...

                          Here is my ipfw sched show:

                          00001: 128.000 Mbit/s    0 ms burst 0
                          q65537  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
                           sched 1 type FQ_CODEL flags 0x0 0 buckets 0 active
                           FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 NoECN
                             Children flowsets: 1
                          00002:  11.128 Mbit/s    0 ms burst 0
                          q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
                           sched 2 type FQ_CODEL flags 0x0 0 buckets 0 active
                           FQ_CODEL target 5ms interval 10ms quantum 1514 limit 10240 flows 1024 NoECN
                             Children flowsets: 2
                          00003:  25.000 Mbit/s    0 ms burst 0
                          q65539  50 sl. 0 flows (1 buckets) sched 3 weight 0 lmax 0 pri 0 droptail
                           sched 3 type FQ_CODEL flags 0x0 0 buckets 0 active
                           FQ_CODEL target 10ms interval 100ms quantum 1514 limit 10240 flows 1024 NoECN
                             Children flowsets: 3
                          00004:   2.000 Mbit/s    0 ms burst 0
                          q65540  50 sl. 0 flows (1 buckets) sched 4 weight 0 lmax 0 pri 0 droptail
                           sched 4 type FQ_CODEL flags 0x0 0 buckets 0 active
                           FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 NoECN
                             Children flowsets: 4
                          

                          1 - Cable download
                          2 - Cable upload
                          3 - DSL download
                          4 - DSL upload

                          Z 1 Reply Last reply Reply Quote 0
                          • mattundM
                            mattund @w0w
                            last edited by mattund

                            @w0w Do we have a redmine number? I can help out; I don't see any PR's yet to fix this

                            get_single_sysctl("net.inet.ip.dummynet.codel.target")

                            I call this because the script needs to pull in the default values for several of the algorithms and I didn't want to hard-code it.

                            My first impression is this is returning something funky. Anyone's new VMs result of sysctl -a | grep net.inet.ip.dummynet.codel.target showing as empty or a non-numeric value? I get net.inet.ip.dummynet.codel.target: 5000

                            Or, they changed or renamed get_single_sysctl

                            It's those items or something with my PHP that is screwy in a recent change; I actually don't do PHP very often so this is likely...

                            Thinking this out, it may be that without the dummynet kernel module loaded (or something like that) there is no sysctl ready for the script to use. Maybe I need to load the kernel module first before I start trying to get these values. I bet you it's not loaded at the very first execution of the script.

                            If this is the case, /sbin/kldload dummynet in the command line or in the UI's command executor should fix this.

                            w0wW 1 Reply Last reply Reply Quote 0
                            • Z
                              zwck
                              last edited by

                              Would you mind screenshotting
                              / creating a catch one and all floating rule? I am not sure how I am supposed to set this up.

                              mattundM 1 Reply Last reply Reply Quote 0
                              • mattundM
                                mattund @zwck
                                last edited by

                                @zwck https://forum.netgate.com/post/772130

                                I posted that earlier for reference, I am still using that configuration I believe ☺

                                1 Reply Last reply Reply Quote 0
                                • Z
                                  zwck
                                  last edited by

                                  @mattund perfect I'll give that a try. Thank you so much

                                  1 Reply Last reply Reply Quote 1
                                  • w0wW
                                    w0w @mattund
                                    last edited by

                                    @mattund
                                    No I did not created any redmine issues.

                                    1 Reply Last reply Reply Quote 0
                                    • Z
                                      zwck @mattund
                                      last edited by

                                      @mattund said in Playing with fq_codel in 2.4:

                                      Now, given you have your shaping set to 900/900, maybe you need to increase the slot size/queue length? That's a pretty fast connection.

                                      Hey Matti,

                                      in the mean time i switched to a floating rule setup, and the rule pretty much catches all similarly to what you described. I also played with bucket size and queue length to no avail, when i download or upload most of the time the scheduler uses sub 15 buckets and the default is 256 or so. I doubled the limit as well to 20480 and flows up to 2048 there was no difference. Then i read the fq_codel manual and the standard values put in place to cover basically everything from 1mbit to 1000mbit. Maybe the drop i see is just what i have.

                                      Thanks for your help.

                                      1 Reply Last reply Reply Quote 0
                                      • D
                                        Dazog
                                        last edited by

                                        I am getting these errors in 2.4.4

                                        kernel config_aqm Unable to configure flowset, flowset busy!

                                        Anyone else seeing these?

                                        1 Reply Last reply Reply Quote 0
                                        • B
                                          blinde @mattund
                                          last edited by

                                          @mattund said in Playing with fq_codel in 2.4:

                                          @slowgrind said in Playing with fq_codel in 2.4:

                                          So after applying the patch do you just fill in the settings under limiters?

                                          Here's what I'm doing. This might be a little more than what you need, but I figure I would share my configuration in case others have a crazy Multi-WAN multi-LAN setup like I do. I've constructed a series of limiters, one for download and one for upload, each with its own associated queue (you can make the queue with the "+ Add new Queue" button on the bottom of a Limiter's settings page) :
                                          0_1529000895837_s1.PNG

                                          (I have more for my second ISP following that naming scheme: lINTERFACEDownload/lINTERFACEUpload and qINTRERFACEDownload/qINTERFACEUpload children)

                                          I'm assigning FQ_CoDel to the scheduler on the parent limiter and leaving everything else alone. You can either edit the parameters, or leave them at default if you have a typical connection (FQ_CoDel is supposed to be "knobless" after all).

                                          According to the following diagram, this is how the traffic will flow inside dummynet:

                                          			    (flow_mask|sched_mask)  sched_mask
                                          		    +---------+	  weight Wx  +-------------+
                                          		    |	      |->-[flow]-->--|		   |-+
                                          	       -->--| QUEUE x |	  ...	     |		   | |
                                          		    |	      |->-[flow]-->--| SCHEDuler N | |
                                          		    +---------+		     |		   | |
                                          			...		     |		   +--[LINK N]-->--
                                          		    +---------+	  weight Wy  |		   | +--[LINK N]-->--
                                          		    |	      |->-[flow]-->--|		   | |
                                          	       -->--| QUEUE y |	  ...	     |		   | |
                                          		    |	      |->-[flow]-->--|		   | |
                                          		    +---------+		     +-------------+ |
                                          					       +-------------+
                                          

                                          via: https://www.freebsd.org/cgi/man.cgi?query=ipfw&manpath=FreeBSD+9-current&format=html

                                          Dissection: firewall traffic is assigned to a queue, which then generates flows defined by the mask, which pipe into the scheduler (set to FQ_CoDel), which then outputs to the pipe/link at the specified max bitrate.

                                          To assign your traffic to queues, you could do something like I did, which is to use floating rules. I have two WANs, and I need independent shaping and all that, so if you're on a single WAN it may be different for you/you may have better options.

                                          0_1529001614927_s4.PNG

                                          How I set the rules up:

                                          • Interface: WAN A or B interface
                                          • Direction: out
                                          • Address Family: IPv4 or IPv6; I had to do two rules, one for each IP version
                                          • Gateway: Select the applicable IPv4 or IPv6 gateway consistent with how traffic should be routed on that IP stack
                                          • In / Out pipe: qCHARTERUpload / qCHARTERDownload

                                          I have some filtering rules in play here as you can see in my screenshot, but that's only since I'm testing some issues I mentioned previously. It's up to you if you want to match certain protocols/ports, etc.

                                          Anybody know why when i enable floating rules both upload and download speeds get cut in half? As soon as i disable it, speeds are back to normal.

                                          As far as i can see, traffic both download and upload are getting matched by the interface rules anyway, what is the floating rule for?

                                          1 Reply Last reply Reply Quote 0
                                          • Z
                                            zwck
                                            last edited by zwck

                                            Wrong numbers in the limiters? maybe post some images, so people can help you

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.