Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Playing with fq_codel in 2.4

    Scheduled Pinned Locked Moved Traffic Shaping
    1.1k Posts 123 Posters 1.5m Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A
      atrotter01
      last edited by

      I tried leaving the masks off and that didn't seem to make a difference.  Then I tried disconnecting everything from the firewall except the laptop that I am using to test with, so that I could ensure nothing else was on the network.  That didn't make a difference either.  When the limiters are not enabled I can run a sustained 60 second download test and hit gig speeds 95% of the time, other than occasional dips.  When I enable the limiters I can't get over the 650ish mark.

      I can run top from an SSH session and it looks to be pegging one of the cores both with and without fq_codel so I can't really tell one way or the other there.

      1 Reply Last reply Reply Quote 0
      • johnpozJ
        johnpoz LAYER 8 Global Moderator
        last edited by

        If you have symmetrical gig gig, why are you setting 40232Kb on pipe2

        So 40mb?  Whats the amount of traffic for acks to hit 650 vs 940?  Have to do the math, etc.  But limiting your upload could have an effect on your max download.. Shoot 500mbps down would be like 8mbps or something up just in acks..

        If your on gig/gig why would you want to limit your upload so much?

        Not a qos guy, this was my real first test of any sort of qos in pfsense.. But I got asked to take a look, that is what jumps out at me to look odd..  In a few days when my 4860 is online I would be able to test your settings.  But not going to be ready until the weekend for sure..  But will be able try and duplicate the problem next week say.

        An intelligent man is sometimes forced to be drunk to spend time with his fools
        If you get confused: Listen to the Music Play
        Please don't Chat/PM me for help, unless mod related
        SG-4860 24.11 | Lab VMs 2.8, 24.11

        1 Reply Last reply Reply Quote 0
        • A
          atrotter01
          last edited by

          John thanks, it's asymmetrical, not symmetrical.  :) The connection maxes out at about 940/42.

          1 Reply Last reply Reply Quote 0
          • A
            Animosity022
            last edited by

            What's the CPU as well? I couldn't drive shaping until I did some upgrades. I'm overpowered now with an:

            Intel(R) Core(TM) i7-4500U CPU @ 1.80GHz (4 cores)

            But I wanted to future proof a bit. I can drive gig both directions at the same time without breaking my CPU atm.

            1 Reply Last reply Reply Quote 0
            • A
              atrotter01
              last edited by

              It's a dual core ARM v7 Cortex-A9 @ 1.6 GHz with NEON SIMD and FPU. I think I am either hitting some odd issue with traffic shaping on ARM architecture or a CPU limitation.

              1 Reply Last reply Reply Quote 0
              • T
                tman222
                last edited by

                @atrotter01:

                It's a dual core ARM v7 Cortex-A9 @ 1.6 GHz with NEON SIMD and FPU. I think I am either hitting some odd issue with traffic shaping on ARM architecture or a CPU limitation.

                1)  Do you run any type of IPS/IDS currently (e.g. Snort)?  If so, try disabling it temporarily to see if that helps.
                2)  What's the output of "ipfw sched show"?  I think it would be good to doublecheck your fq_codel parameters.
                3)  As alternative you can try ALTQ traffic shaping (instead of dummynet) to see if you experience similar limitations.  Instead of using limiters, try setting up the ALTQ FAIRQ scheduler and then enable Codel in the default queue.  You'll probably also need to increase the default queue size from 50 to something larger like 512 or 1024.  FAIRQ + Codel is not quite the same as fq_codel, but it it's similar (I have used both and the performance in my case was comparable).  If you try that, do you see a similar speed limitation that you currently see with dummynet/fq_codel?

                Hope this helps.

                1 Reply Last reply Reply Quote 0
                • C
                  chrcoluk
                  last edited by

                  Previously when I last tried this on 2.4.0 dev it at least functioned.

                  Now trying again on 2.4.2 release it just wont work.

                  Following the instructions to the latter and even using my backed up config for this which previously functioned, results in all outbound connections timing out as if blocked.
                  If I set the in/out pipes to use the child queues, connections will work briefly but then timeout after a few seconds.
                  If I set the in/out pipes to none, everything works but of course the shaper isnt been used.

                  Limiter info page shows the correct information as does ipfw pipe show and ipfw sched show.  The issue seems to be PF redirecting traffic properly to ipfw dummynet.

                  Also of interest on the console I am getting notices saying end of ipfw rules hit and denying packet, which is odd as on dmesg bootup, it shows as the default policy for ipfw set to allow.  Which conflicts with the denying packets.

                  pfSense CE 2.7.2

                  1 Reply Last reply Reply Quote 0
                  • C
                    chrcoluk
                    last edited by

                    Ok a bit more information since I was doing this before going bed and getting tired.

                    As mentioned before all traffic was been blocked except on one occasion which I will mention in a moment.

                    I disabled ALTQ, enabled dummynet limiter, disabled the majority of my LAN firewall rules (Since they were to divert to different priority HSFC queues, and so the firewall was vastly simplified, the only LAN rules were pfblockerNG rules, rules to route specific ip's to VPN's and the default outbound LAN rule which I set to use the dummynet in/out pipes.

                    If I viewed the live limiter stats (the enhanced stats with the patch provided in thread), I always seen an active bucket seemingly processing continuous data, even tho there was only idle network activity, all internet connections which were not already established timed out.

                    However on one occasion this bucket didnt appear, and the connectivity was working, but as I started a speedtest, it all went to timing out again and this bucket showing continuous data was back.

                    In addition if I left it like this for a while then the console would start getting flooded with messages like "fq codel enqueue over limit" and maxidx warnings.  If I left it further, the kernel panicked.

                    Running ipfw pipe flush immediately killed the console messages (and also prevented the panic).
                    Changing the default outbound LAN rule to not use the dummynet pipes immediately restored connectivity. However this did not stop that bucket flow of data, so that occurred even when no firewall rules were routing traffic to dummynet pipes.

                    I think HFSC may have been causing me some issues, so what I have done for now is gone back to ALTQ and FAIRQ for the upstream, but there is no downstream ALTQ active so right now I have no downstream shaping.

                    I got no idea myself on how to proceed now so unless I get suggestions I wont be trying it again for a while as I wont get anywhere I think.  I am curious tho if anyone here who is using it is running 2.4.2?

                    pfSense CE 2.7.2

                    1 Reply Last reply Reply Quote 0
                    • T
                      tman222
                      last edited by

                      @chrcoluk:

                      Previously when I last tried this on 2.4.0 dev it at least functioned.

                      Now trying again on 2.4.2 release it just wont work.

                      Following the instructions to the latter and even using my backed up config for this which previously functioned, results in all outbound connections timing out as if blocked.
                      If I set the in/out pipes to use the child queues, connections will work briefly but then timeout after a few seconds.
                      If I set the in/out pipes to none, everything works but of course the shaper isnt been used.

                      Limiter info page shows the correct information as does ipfw pipe show and ipfw sched show.  The issue seems to be PF redirecting traffic properly to ipfw dummynet.

                      Also of interest on the console I am getting notices saying end of ipfw rules hit and denying packet, which is odd as on dmesg bootup, it shows as the default policy for ipfw set to allow.  Which conflicts with the denying packets.

                      I've got fq_codel working fine on the latest 2.4.2 release using two root limiters and four queues under each.  The only algorithm parameters I've tweaked from default are the limit, interval, and target.

                      Can you show us the output of:

                      ipfw sched show
                      ipfw pipe show
                      ipfw queue show

                      Along with a screenshot how your limiters/queues are setup?  That will help us debug things further.

                      Hope this helps.

                      1 Reply Last reply Reply Quote 0
                      • C
                        chrcoluk
                        last edited by

                        Ok I wish I already stored that information, I will go back to this again probably on boxing day and post the information you requested then.

                        pfSense CE 2.7.2

                        1 Reply Last reply Reply Quote 0
                        • w0wW
                          w0w
                          last edited by

                          I don't have any problem with fq_codel also. I think it can be some package or enabled feature, like captive portal.

                          1 Reply Last reply Reply Quote 0
                          • A
                            azuisleet
                            last edited by

                            I'm testing pfSense on my network with an SG-4860. Following the hints here, I've got fq_codel working on my network without anything too unusual happening that could caused by the use of fq_codel (hopefully).

                            However, the guide from the bufferbloat project mentions a couple of the tuning parameters like "target", "quantum" and "limit". They suggest a "limit" of under 1000, but the limit option appears to be a global limit of the sum of queue lengths of all flows managed under the scheduler: https://github.com/freebsd/freebsd/blob/2589d9ccafc21d29deade87a50261657c27c5700/sys/netpfil/ipfw/dn_sched_fq_codel.c#L328

                            If I set the limit on fq_codel to 1000 (eg: "sched 1 config pipe 1 type fq_codel target 15 ecn quantum 300 limit 1000"), I get the "kernel: fq_codel_enqueue over limit" and " kernel: fq_codel_enqueue maxidx = XYZ" that a user reported above. This is happening on a 100/5 connection. It could be a problem with my connection, but based on the fact that it spews to the log it seems to be a failure mode. Which direction it's happening on (upload or download) I'm not sure.

                            1 Reply Last reply Reply Quote 0
                            • H
                              Harvy66
                              last edited by

                              I'm not entirely familiar with fq_Codel or its implementations, but your quantum should not be lower than your MTU, which is why the default is 1514 for the spec, which is the standard MTU for Ethernet.

                              edit: I rescind this

                              1 Reply Last reply Reply Quote 0
                              • T
                                tman222
                                last edited by

                                @azuisleet:

                                I'm testing pfSense on my network with an SG-4860. Following the hints here, I've got fq_codel working on my network without anything too unusual happening that could caused by the use of fq_codel (hopefully).

                                However, the guide from the bufferbloat project mentions a couple of the tuning parameters like "target", "quantum" and "limit". They suggest a "limit" of under 1000, but the limit option appears to be a global limit of the sum of queue lengths of all flows managed under the scheduler: https://github.com/freebsd/freebsd/blob/2589d9ccafc21d29deade87a50261657c27c5700/sys/netpfil/ipfw/dn_sched_fq_codel.c#L328

                                If I set the limit on fq_codel to 1000 (eg: "sched 1 config pipe 1 type fq_codel target 15 ecn quantum 300 limit 1000"), I get the "kernel: fq_codel_enqueue over limit" and " kernel: fq_codel_enqueue maxidx = XYZ" that a user reported above. This is happening on a 100/5 connection. It could be a problem with my connection, but based on the fact that it spews to the log it seems to be a failure mode. Which direction it's happening on (upload or download) I'm not sure.

                                Those errors indicate that your queue size (the "limit" parameter) is too small.  For the connection speed you have, the value should be higher (especially if you use fq_codel on upload and download traffic).  I think the default parameters of the algorithm would work fine in your case, but if you want, you can reduce the queue size a little bit (10240 is quite large), and tweak the quantum (depending on your traffic profile, i.e. smaller vs. larger packets).  If performance is not satisfactory, you can also try increasing the target a little (e.g. to 8 or 10ms), but I think the default value of 5ms should work fine for your connection's upload speed.

                                Here's a link to some documentation:

                                http://caia.swin.edu.au/freebsd/aqm/patches/README-0.2.1.txt

                                quantum:
                                    is number of bytes a queue can be served before being moved it
                                    to the tail of old queues list. Default: 1514 bytes, default can be
                                    changed by the sysctl variable net.inet.ip.dummynet.fqcodel.quantum
                                
                                limit:
                                    is the hard limit of all queues size managed by fq_codel schedular
                                    instance. Default: 10240 packets, default can be changed by the 
                                    sysctl variable net.inet.ip.dummynet.fqcodel.limit
                                

                                Additional info that maybe useful:
                                https://tools.ietf.org/html/draft-ietf-aqm-fq-codel-06
                                https://www.reddit.com/r/openbsd/comments/6ttuhn/fq_codel_scheduling/

                                Hope this helps.

                                1 Reply Last reply Reply Quote 0
                                • H
                                  Harvy66
                                  last edited by

                                  tman222 is correct about the quantum.

                                  https://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel/

                                  We have generally settled on a quantum of 300 for usage below 100mbit as this is a good compromise between SFQ and pure DRR behavior that gives smaller packets a boost over larger ones.

                                  1 Reply Last reply Reply Quote 0
                                  • forbiddenlakeF
                                    forbiddenlake
                                    last edited by

                                    @darkcrucible:

                                    I setup fq_codel using floating rules on another system and the same IPv4 traceroute/ICMP problem I mentioned earlier occurs.

                                    Anyone else who uses floating rules to match traffic for fq_codel, do you see IPv4 ICMP traceroute working properly?

                                    I see the same (2.4.2-RELEASE-p1)

                                    1 Reply Last reply Reply Quote 0
                                    • L
                                      lukezamboni
                                      last edited by

                                      @tman222:

                                      I've got fq_codel working fine on the latest 2.4.2 release using two root limiters and four queues under each.  The only algorithm parameters I've tweaked from default are the limit, interval, and target.

                                      Can you show us the output of:

                                      ipfw sched show
                                      ipfw pipe show
                                      ipfw queue show

                                      Along with a screenshot how your limiters/queues are setup?  That will help us debug things further.

                                      Hope this helps.

                                      I have the exact same issue as chrcoluk.

                                      ipfw sched show

                                      
                                      00001: 450.000 Mbit/s    0 ms burst 0
                                      q65537  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
                                       sched 1 type FQ_CODEL flags 0x0 0 buckets 0 active
                                       FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
                                         Children flowsets: 1
                                      00002: 450.000 Mbit/s    0 ms burst 0
                                      q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
                                       sched 2 type FQ_CODEL flags 0x0 0 buckets 0 active
                                       FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
                                         Children flowsets: 2
                                      
                                      

                                      ipfw pipe show

                                      
                                      00001: 450.000 Mbit/s    0 ms burst 0
                                      q131073  50 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0 droptail
                                       sched 65537 type FIFO flags 0x0 0 buckets 0 active
                                      00002: 450.000 Mbit/s    0 ms burst 0
                                      q131074  50 sl. 0 flows (1 buckets) sched 65538 weight 0 lmax 0 pri 0 droptail
                                       sched 65538 type FIFO flags 0x0 0 buckets 0 active
                                      
                                      

                                      ipfw queue show

                                      q00001  50 sl. 0 flows (256 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
                                          mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
                                      q00002  50 sl. 0 flows (256 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
                                          mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000
                                      
                                      

                                      I also noticed that 'ipfw sched show' has 0 buckets active while it appears the correct would be 1 bucket active. Any idea on what I can do here?

                                      Screenshot_1.png
                                      Screenshot_1.png_thumb
                                      Screenshot_3.png
                                      Screenshot_3.png_thumb
                                      Screenshot_4.png
                                      Screenshot_4.png_thumb
                                      Screenshot_5.png
                                      Screenshot_5.png_thumb
                                      Screenshot_6.png
                                      Screenshot_6.png_thumb
                                      Screenshot_7.png
                                      Screenshot_7.png_thumb

                                      1 Reply Last reply Reply Quote 0
                                      • T
                                        tman222
                                        last edited by

                                        @lukezamboni:

                                        @tman222:

                                        I've got fq_codel working fine on the latest 2.4.2 release using two root limiters and four queues under each.  The only algorithm parameters I've tweaked from default are the limit, interval, and target.

                                        Can you show us the output of:

                                        ipfw sched show
                                        ipfw pipe show
                                        ipfw queue show

                                        Along with a screenshot how your limiters/queues are setup?  That will help us debug things further.

                                        Hope this helps.

                                        I have the exact same issue as chrcoluk.

                                        ipfw sched show

                                        
                                        00001: 450.000 Mbit/s    0 ms burst 0
                                        q65537  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
                                         sched 1 type FQ_CODEL flags 0x0 0 buckets 0 active
                                         FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
                                           Children flowsets: 1
                                        00002: 450.000 Mbit/s    0 ms burst 0
                                        q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
                                         sched 2 type FQ_CODEL flags 0x0 0 buckets 0 active
                                         FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
                                           Children flowsets: 2
                                        
                                        

                                        ipfw pipe show

                                        
                                        00001: 450.000 Mbit/s    0 ms burst 0
                                        q131073  50 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0 droptail
                                         sched 65537 type FIFO flags 0x0 0 buckets 0 active
                                        00002: 450.000 Mbit/s    0 ms burst 0
                                        q131074  50 sl. 0 flows (1 buckets) sched 65538 weight 0 lmax 0 pri 0 droptail
                                         sched 65538 type FIFO flags 0x0 0 buckets 0 active
                                        
                                        

                                        ipfw queue show

                                        q00001  50 sl. 0 flows (256 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
                                            mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
                                        q00002  50 sl. 0 flows (256 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
                                            mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000
                                        
                                        

                                        I also noticed that 'ipfw sched show' has 0 buckets active while it appears the correct would be 1 bucket active. Any idea on what I can do here?

                                        Looking at your setup, can you explain why you chose to have both LAN and WAN queues under each Upload and Download?  Do you use all those queues for separate traffic, subnets, etc.?  A basic setup actually requires just one queue under each limiter.  For example, have a look at post #121:

                                        https://forum.pfsense.org/index.php?topic=126637.msg754199#msg754199

                                        If you try those basic settings such as those, do you experience problems?

                                        Hope this helps.

                                        1 Reply Last reply Reply Quote 0
                                        • C
                                          chrcoluk
                                          last edited by

                                          Mine only has one queue for each limiter, and is basically like the post you linked to except with different limits.

                                          I of course have the script to override using fq_codel meaning the GUI setting stops controlling dummynet, but the GUI settings are the baseline with the only adjustment been the switch to fq_codel.

                                          I have not had time yet to retest this so I still have no live ipfw pipe etc. outputs, I wish you would accept my word without me having to paste it all, but I am now glad someone has repeated the problem.

                                          pfSense CE 2.7.2

                                          1 Reply Last reply Reply Quote 0
                                          • T
                                            tman222
                                            last edited by

                                            @chrcoluk:

                                            Mine only has one queue for each limiter, and is basically like the post you linked to except with different limits.

                                            I of course have the script to override using fq_codel meaning the GUI setting stops controlling dummynet, but the GUI settings are the baseline with the only adjustment been the switch to fq_codel.

                                            I have not had time yet to retest this so I still have no live ipfw pipe etc. outputs, I wish you would accept my word without me having to paste it all, but I am now glad someone has repeated the problem.

                                            Can you remind me of your setup/configuration:

                                            1)  What script are you using that you referenced above?  What does it do?  I don't recall having to setup any scripts - only Shellcmd to make sure the fq_codel starts automatically after reach reboot.
                                            2)  Do you have your queues applied to your outgoing traffic LAN rule?  Or in a traffic matching rule on your WAN interface?  If the latter, could you show us the configuration?

                                            Hope this helps.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.