Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Playing with fq_codel in 2.4

    Scheduled Pinned Locked Moved Traffic Shaping
    1.1k Posts 123 Posters 1.5m Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • w0wW
      w0w @cplmayo
      last edited by w0w

      @cplmayo
      Install System_Patches package, go to System->Patches, Press Add New Patch button and copy-paste URL https://github.com/pfsense/pfsense/compare/RELENG_2_4_3...mattund:RELENG_2_4_3.diff into URL/Commit ID field.
      Also, write something into description field, press SAVE button. After that you will see you newly created patch in the list and Fetch button on the right, press it, then press appeared Test button, you should see

      Patch can be applied cleanly (detail)
      Patch can NOT be reverted cleanly (detail)
      

      If you see that, then you can press Apply button and enjoy your new Limiters

      1 Reply Last reply Reply Quote 3
      • S
        SlowGrind
        last edited by

        So after applying the patch do you just fill in the settings under limiters?

        0_1528934305795_fq-codel.JPG

        New here to traffic shaping so just looking for an easy way to decrease latency. Thanks.

        w0wW mattundM 2 Replies Last reply Reply Quote 0
        • w0wW
          w0w @SlowGrind
          last edited by

          @slowgrind
          Generally yes, but I prefer to share bandwidth between all users evenly.
          Unfortunately looks like foxale08 guide is broken, it was here but now it's completely gone :(
          I'll provide some guide later, if you want it.

          S 1 Reply Last reply Reply Quote 0
          • S
            SlowGrind @w0w
            last edited by

            @w0w

            Thank you and I just have a simple network setup. 1 wan and 1 lan with 1000 Mbps download and 50 Mbps upload.

            1 Reply Last reply Reply Quote 0
            • mattundM
              mattund
              last edited by mattund

              I'm testing my patch, and I am seeing some unexpected behavior (maybe intentional).

              pipe 2 config  bw 122800Kb queue 100 buckets 256 droptail
              sched 2 config pipe 2 type fq_codel target 35ms interval 35ms quantum 1514 limit 10240 flows 1024 noecn
              queue 1 config pipe 2 queue 100 droptail
              
              pipe 1 config  bw 12000Kb queue 100 droptail
              sched 1 config pipe 1 type fq_codel target 5ms interval 10ms quantum 1514 limit 10240 flows 1024 noecn
              queue 2 config pipe 1 queue 100 droptail
              

              Using the above generated ruleset, if I start a speedtest, and have an ICMP ping up at the same time, upon reaching the ceiling of ~120Mbps, pings drop completely. Not bufferbloat, but full drops. Now, of course that would happen, the limiter is supposed to limit the bandwidth. However, is there a way around this? I figured burst might help, but it doesn't appear to -- FQ_CoDel assumes the new burst value as part of the total bandwidth share and doesn't kick in.

              It's almost as though FQ_CoDel isn't active, and the limiter is putting itself into the drop state before CoDel can say, "hey, this teeny tiny little ICMP packet isn't part of a busy flow, so let's let that through". Unless I am mistaken on something.

              What I want:

              • Small flow traffic to not get dropped (think my little ping)
              • Busy flow traffic to be shaped (dropped) appropriately
              • Not allow the limiter pipe drop packets before FQ_CoDel has a chance to process them (?)

              I was under the impression FQ_CoDel was designed around those objectives. Am I doing this right?

              C mattundM w0wW 3 Replies Last reply Reply Quote 0
              • C
                cplmayo @mattund
                last edited by

                @mattund said in Playing with fq_codel in 2.4:

                I'm testing my patch, and I am seeing some unexpected behavior (maybe intentional).

                pipe 2 config  bw 122800Kb queue 100 buckets 256 droptail
                sched 2 config pipe 2 type fq_codel target 35ms interval 35ms quantum 1514 limit 10240 flows 1024 noecn
                queue 1 config pipe 2 queue 100 droptail
                
                pipe 1 config  bw 12000Kb queue 100 droptail
                sched 1 config pipe 1 type fq_codel target 5ms interval 10ms quantum 1514 limit 10240 flows 1024 noecn
                queue 2 config pipe 1 queue 100 droptail
                

                Using the above generated ruleset, if I start a speedtest, and have an ICMP ping up at the same time, upon reaching the ceiling of ~120Mbps, pings drop completely. Not bufferbloat, but full drops. Now, of course that would happen, the limiter is supposed to limit the bandwidth. However, is there a way around this? I figured burst might help, but it doesn't appear to -- FQ_CoDel assumes the new burst value as part of the total bandwidth share and doesn't kick in.

                It's almost as though FQ_CoDel isn't active, and the limiter is putting itself into the drop state before CoDel can say, "hey, this teeny tiny little ICMP packet isn't part of a busy flow, so let's let that through". Unless I am mistaken on something.

                Am I doing this right?

                I applied your patch to my install and so far it is working as intended. I haven't noticed any huge difference between it and manual approach I have been using for months.

                I did noticed that when I tried setting weights for different queues and if I set the scheduler in the queues no traffic would pass. I went back and reread your instructions on GitHub and once I removed the weights and scheduler options from the queue it works.

                I am still doing some testing to see results in dslreports but so far your interface is simple and just seems to work.

                Great job!

                1 Reply Last reply Reply Quote 1
                • mattundM
                  mattund @SlowGrind
                  last edited by mattund

                  @slowgrind said in Playing with fq_codel in 2.4:

                  So after applying the patch do you just fill in the settings under limiters?

                  Here's what I'm doing. This might be a little more than what you need, but I figure I would share my configuration in case others have a crazy Multi-WAN multi-LAN setup like I do. I've constructed a series of limiters, one for download and one for upload, each with its own associated queue (you can make the queue with the "+ Add new Queue" button on the bottom of a Limiter's settings page) :
                  0_1529000895837_s1.PNG

                  (I have more for my second ISP following that naming scheme: lINTERFACEDownload/lINTERFACEUpload and qINTRERFACEDownload/qINTERFACEUpload children)

                  I'm assigning FQ_CoDel to the scheduler on the parent limiter and leaving everything else alone. You can either edit the parameters, or leave them at default if you have a typical connection (FQ_CoDel is supposed to be "knobless" after all).

                  According to the following diagram, this is how the traffic will flow inside dummynet:

                  			    (flow_mask|sched_mask)  sched_mask
                  		    +---------+	  weight Wx  +-------------+
                  		    |	      |->-[flow]-->--|		   |-+
                  	       -->--| QUEUE x |	  ...	     |		   | |
                  		    |	      |->-[flow]-->--| SCHEDuler N | |
                  		    +---------+		     |		   | |
                  			...		     |		   +--[LINK N]-->--
                  		    +---------+	  weight Wy  |		   | +--[LINK N]-->--
                  		    |	      |->-[flow]-->--|		   | |
                  	       -->--| QUEUE y |	  ...	     |		   | |
                  		    |	      |->-[flow]-->--|		   | |
                  		    +---------+		     +-------------+ |
                  					       +-------------+
                  

                  via: https://www.freebsd.org/cgi/man.cgi?query=ipfw&manpath=FreeBSD+9-current&format=html

                  Dissection: firewall traffic is assigned to a queue, which then generates flows defined by the mask, which pipe into the scheduler (set to FQ_CoDel), which then outputs to the pipe/link at the specified max bitrate.

                  To assign your traffic to queues, you could do something like I did, which is to use floating rules. I have two WANs, and I need independent shaping and all that, so if you're on a single WAN it may be different for you/you may have better options.

                  0_1529001614927_s4.PNG

                  How I set the rules up:

                  • Interface: WAN A or B interface
                  • Direction: out
                  • Address Family: IPv4 or IPv6; I had to do two rules, one for each IP version
                  • Gateway: Select the applicable IPv4 or IPv6 gateway consistent with how traffic should be routed on that IP stack
                  • In / Out pipe: qCHARTERUpload / qCHARTERDownload

                  I have some filtering rules in play here as you can see in my screenshot, but that's only since I'm testing some issues I mentioned previously. It's up to you if you want to match certain protocols/ports, etc.

                  C B Z 3 Replies Last reply Reply Quote 0
                  • mattundM
                    mattund @mattund
                    last edited by mattund

                    @mattund said in Playing with fq_codel in 2.4:

                    I'm testing my patch, and I am seeing some unexpected behavior (maybe intentional).

                    This stopped happening as soon as I enabled masks on the offending download queue correlating to the one ICMP traffic was being dropped on. Not sure why.

                    1 Reply Last reply Reply Quote 0
                    • C
                      cplmayo @mattund
                      last edited by

                      @mattund said in Playing with fq_codel in 2.4:

                      @slowgrind said in Playing with fq_codel in 2.4:

                      So after applying the patch do you just fill in the settings under limiters?

                      Here's what I'm doing. This might be a little more than what you need, but I figure I would share my configuration in case others have a crazy Multi-WAN multi-LAN setup like I do. I've constructed a series of limiters, one for download and one for upload, each with its own associated queue (you can make the queue with the "+ Add new Queue" button on the bottom of a Limiter's settings page) :
                      0_1529000895837_s1.PNG

                      (I have more for my second ISP following that naming scheme: lINTERFACEDownload/lINTERFACEUpload and qINTRERFACEDownload/qINTERFACEUpload children)

                      I'm assigning FQ_CoDel to the scheduler on the parent limiter and leaving everything else alone. You can either edit the parameters, or leave them at default if you have a typical connection (FQ_CoDel is supposed to be "knobless" after all).

                      According to the following diagram, this is how the traffic will flow inside dummynet:

                      			    (flow_mask|sched_mask)  sched_mask
                      		    +---------+	  weight Wx  +-------------+
                      		    |	      |->-[flow]-->--|		   |-+
                      	       -->--| QUEUE x |	  ...	     |		   | |
                      		    |	      |->-[flow]-->--| SCHEDuler N | |
                      		    +---------+		     |		   | |
                      			...		     |		   +--[LINK N]-->--
                      		    +---------+	  weight Wy  |		   | +--[LINK N]-->--
                      		    |	      |->-[flow]-->--|		   | |
                      	       -->--| QUEUE y |	  ...	     |		   | |
                      		    |	      |->-[flow]-->--|		   | |
                      		    +---------+		     +-------------+ |
                      					       +-------------+
                      

                      via: https://www.freebsd.org/cgi/man.cgi?query=ipfw&manpath=FreeBSD+9-current&format=html

                      Dissection: firewall traffic is assigned to a queue, which then generates flows defined by the mask, which pipe into the scheduler (set to FQ_CoDel), which then outputs to the pipe/link at the specified max bitrate.

                      To assign your traffic to queues, you could do something like I did, which is to use floating rules. I have two WANs, and I need independent shaping and all that, so if you're on a single WAN it may be different for you/you may have better options.

                      0_1529001614927_s4.PNG

                      How I set the rules up:

                      • Interface: WAN A or B interface
                      • Direction: out
                      • Address Family: IPv4 or IPv6; I had to do two rules, one for each IP version
                      • Gateway: Select the applicable IPv4 or IPv6 gateway consistent with how traffic should be routed on that IP stack
                      • In / Out pipe: qCHARTERUpload / qCHARTERDownload

                      I have some filtering rules in play here as you can see in my screenshot, but that's only since I'm testing some issues I mentioned previously. It's up to you if you want to match certain protocols/ports, etc.

                      When I setup my floating rules I tried using the out direction a few times but have run into problems each and every time.

                      For instance if I set the rule direction to "out" and then set the gateway I get what appears to be a routing loop. Traffic forwards find and I can hit external hosts however when I do a traceroute from an internal host I don't see the hop routers I just see the destination IP over and over. Are you seeing this or have you changed your setup?

                      I have also been considering using the dynamic pipes. Anyone tried this with the fq_codel scheduler?

                      mattundM 1 Reply Last reply Reply Quote 1
                      • mattundM
                        mattund @cplmayo
                        last edited by

                        @cplmayo I also get a routing loop in traceroutes. Are you using the in direction?

                        C 1 Reply Last reply Reply Quote 0
                        • C
                          cplmayo @mattund
                          last edited by cplmayo

                          @mattund I have been, I would prefer to use out but it makes any troubleshooting I may need to do difficult. Not that I do traceroutes a lot but it annoyed me.

                          I have wondered if this was due to how the rule is creating states. Like it is rerouting the packet once the rule is applied; very weird.

                          There are two floating rules both in; one on WAN the other all of my vlans.

                          I wish I could just apply them to the WAN with in & out as the directions.

                          1 Reply Last reply Reply Quote 1
                          • X
                            xRaisen
                            last edited by

                            Hello, I am new here. I have been following this thread since last week. Tried the new patch and it's working as intended just like the manual. I leave everything in default, I just add the Upload/Download bandwidth Queue Management Algorithm
                            CoDel and FQ_Codel for Scheduler.

                            I have tested it, hogging my connections doing 20 torrents and 1 download on IDM. Played Online game to see the ping and CMD ping www.google.com -t, I experienced 1 to 2 packet loss, and 3 to 5 ping dancing and I can still play like as if there were no downloads happening on the background.

                            Some issues on the syslog, but fixed it by this command: ifconfig <nics> -promisc

                            0_1529003640259_4f840f82-1253-4255-a783-eb3ad481a2d3-image.png

                            Just a heads up! This one though, it's just showing on the syslogs everytime.

                            0_1529003711093_e9ff68bc-48a9-450f-8db7-ed6bda851357-image.png

                            By the way, I have Floating rules active and limiters(for LAN scheds, no queues).

                            Overall, thank you. This helps me a lot.

                            C mattundM 2 Replies Last reply Reply Quote 1
                            • C
                              cplmayo @xRaisen
                              last edited by cplmayo

                              @xraisen Your nics shouldn't need to be in promiscuous mode. That is a message that I would ignore.

                              Promiscuous mode is just going to have your NIC accept all packets.

                              Non-Promiscuous mode will have you NIC drop packets not destined for it.

                              Unless you are doing packet captures I can't think of a reason you would want to run in promiscuous mode.

                              X 2 Replies Last reply Reply Quote 0
                              • X
                                xRaisen @cplmayo
                                last edited by

                                @cplmayo This was enabled default because I don't remember enabling this. But thanks for the info!

                                1 Reply Last reply Reply Quote 0
                                • mattundM
                                  mattund @xRaisen
                                  last edited by mattund

                                  @xraisen

                                  I'm also seeing the flowset busy log entries occasionally, I think it has to do with the fact that the rules don't always get deleted (just reconfigured). However, I am not positive on that yet. Others may have gotten around this or have other ideas...

                                  EDIT:

                                  We're a lot like opnsense with this patch, because they are using this (dummynet - not my patch lol) for their shaping I think. Well, regardless, this ticket:

                                  https://github.com/opnsense/core/issues/1279

                                  Due to the fact that the internal variables of each AQM instance is allocated and freed dynamically, re-configuring AQM (e.g. select CoDel for a pipe that currently uses PIE)
                                  can lead the AQM code to access unallocated memory (freed during AQM re-configuration) and kernel panic could happen.
                                  Therefore, I tried to avoid any potential kernel panic by preventing re-configuring AQM for busy pipe/queue.

                                  He suggests:

                                  • Temporarily block the traffic (using IPFW rules) from passing through AQM PIE/queue until you finish your configuration.
                                  • Delete the old pipes/queues that need to be re-configured with AQM and create them again with all the desired settings.
                                  • Reboot the firewall

                                  I think this really would need to be a code-level patch somehow. Somehow "unbusy" it, then apply

                                  EDIT 2:

                                  I find this kind of hilarious and coincidental, but I'm experiencing this right now. I'm going to screw around with my shaper.inc and see what it takes to shake this error.

                                  EDIT 3:

                                  Deleting and recreating the rules does nothing to alleviate the error, however setting the Queue Management Algorithm to on the Parent Limiter droptail worked to get it to stop throwing each apply for me.

                                  C X S 4 Replies Last reply Reply Quote 0
                                  • C
                                    cplmayo @mattund
                                    last edited by

                                    @mattund From my experience with limiters and floating rules; always apply after any single change to a limiter. Each time I make a change I save and apply. I ran into problems at first when I would save several changes and apply at once.

                                    May help may not.

                                    For floating rules always reset states after making a change. This is due to how pfSense handles floating rules. It just helps to ensure that your floating rules are being applied to all of your traffic. If you add a floating rule and do not reset states only new traffic will be evaluated against the rule.

                                    1 Reply Last reply Reply Quote 0
                                    • w0wW
                                      w0w @mattund
                                      last edited by

                                      This post is deleted!
                                      1 Reply Last reply Reply Quote 0
                                      • X
                                        xRaisen @cplmayo
                                        last edited by

                                        @cplmayo said in Playing with fq_codel in 2.4:

                                        @xraisen Your nics shouldn’t need to be in promiscuous mode. That is a message that I would ignore.
                                        Promiscuous mode is just going to have your NIC accept all packets.
                                        Non-Promiscuous mode will have you NIC drop packets not destined for it.
                                        Unless you are doing packet captures I can’t think of a reason you would want to run in promiscuous mode.

                                        I found out why it was enabled. Because I use the NTOPNG package. I have uninstalled it, now it's gone. The package turned on the promiscuous mode to monitor the packets.

                                        1 Reply Last reply Reply Quote 0
                                        • X
                                          xRaisen @mattund
                                          last edited by xRaisen

                                          @mattund said in Playing with fq_codel in 2.4:

                                          He suggests:

                                          Temporarily block the traffic (using IPFW rules) from passing through AQM PIE/queue until you finish your configuration.
                                          Delete the old pipes/queues that need to be re-configured with AQM and create them again with all the desired settings.
                                          Reboot the firewall

                                          I have tried this method, but when doing "ipfw sched show" this will show up

                                          0_1529008156466_c20bb1cc-e2f3-45d8-bce7-1d1bb14456d2-image.png

                                          Notice the "Drop tail" cannot be changed to CoDel, PIE, RED and GRED and It's very persistent.

                                          Please confirm this on your end.

                                          EDIT 1:

                                          Changing the Queue Management Algorithm to Tail Drop, the "config_aqm Unable to configure flowset, flowset busy!" was gone.

                                          EDIT 2:

                                          Trying Tail Drop + FQ_CODEL on scheds and CODEL on queues and use queues instead of scheds, gives me a better results without the syslog nagging. Hogging the bandwidth again for testing, I got zero packet loss and ping on www.google.com and CSGO are dancing to 0ms to 1ms swiftly like nothing happened. This is the best bandwidth management setup I have experienced so far on pfsense. Take note, I have an active traffic shaper (fairq+codel) on floating rules for voice, games and browsing priorities.

                                          Z M 2 Replies Last reply Reply Quote 0
                                          • Z
                                            zwck @xRaisen
                                            last edited by

                                            @xraisen Would you mind sharing some screenshots of your config :D

                                            X 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.