Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Playing with fq_codel in 2.4

    Scheduled Pinned Locked Moved Traffic Shaping
    1.1k Posts 123 Posters 1.7m Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • D
      dtaht
      last edited by

      we probably should publish it because it doesn't vary much any more. We used to have 15, now, it's 7? 6? So far we haven't any major scaling issues.

      Unlike speedtest.net we don't have a revenue model, and the hope was we'd see folk setting up their own internal servers for testing. It's just netperf and irtt.

      w0wW 1 Reply Last reply Reply Quote 0
      • w0wW
        w0w @dtaht
        last edited by w0w

        @dtaht
        I really respect the work that was done by you and people fighting bufferbloat over the world.
        As for me I use AQM everywhere it possible, to eliminate bufferbloat from my networks. The one problem is still to be investigated is those networks where we can't to detect the bottleneck size aka bandwidth limit varies a lot during day usage. I know there is a lot of such ISP networks in Japan, for example.
        Yes we can set bandwidth limit to the minimum one, but it not smart enough, it would be good to detect the current bandwidth automatically and adjust everything. I do know that there are some software and router scripts samples on the net that do it in various ways, but it still need to be studied and developed in many systems, also pfSense.

        D 1 Reply Last reply Reply Quote 0
        • D
          dtaht @w0w
          last edited by

          @w0w - and you. keep fighting the bloat!

          My sadness is that I'd hoped we'd see all the core bufferbloat-fighting tools in DSL/fiber/cablemodems by now. BQL + FQ-codel work well on varying line rates, so long as your underlying driver is "tight". We also showed how to do up wifi right. Also thought we'd see ways (snmp? screen scraping?) of getting more core stats out of more devices, so as to be able to handle sag better.

          with sch_cake we made modifying the bandwidth shaping nondestructive and really easy (can pfsense's shaper be changed in place?) - it's essentially

          while getlinestatssomehow()
          do
          tc change dev yourdevice newbandwidth
          done

          So far the only deployment of that feature is in the evenroute, but I'm pretty sure at least a few devices can be screenscraped.

          As for detecting issues further upstream, there are ways appearing, but they need to be sampling streams to work (see various tools of kathie's: https://github.com/pollere )

          1 Reply Last reply Reply Quote 0
          • T
            tman222
            last edited by

            Hi @dtaht,

            I have a basic question regarding bufferbloat that I never quite understood. I can understand how there can be bufferbloat on the uplink of a WAN connection (e.g. 1Gbit LAN interfaces sending into a e.g. 10Mbit cable modem upload). However, what causes bufferbloat on the download since generally the interface the data is being sent into is larger than the download speed on the WAN interface (e.g. a 250Mbit cable modem download speed sent into a 1Gbit LAN interface)?

            Thanks in advance for the insight and explanation, I really appreciate it.

            1 Reply Last reply Reply Quote 1
            • D
              dtaht
              last edited by

              The buffering in that case builds at the CMTS (on the other side of the cablemodem). CMTS's tend to have outrageous amounts of FIFO buffering (680ms on my 120Mbit comcast connection), so, if you set your inbound shaper to less than their setting, you shift the bottleneck to your device and can control it better. It's not always effective (you can end up in a pathological situation where the CMTS is buffering madly as fast as you are trying to regain control of the link), but setting up an inbound shaper to 85% or so of your provisioned rate generally works, and you end up with zero inbound buffering for sparse packets, and 5-20ms for bigger flows, locally.

              Does that work for you? (It's still an open question for me as to how netgate does inbound shaping).

              It's horribly compute intensive to do it this way, but since we've been after the cablecos for 7+ years now to fix their CMTSes with no progress, shaping inbound is a necessity. In my networks, I drop 30x more packets inbound than outbound but my links stay usable for tons of users, web PLTs are good, voip and videoconferencing "just work", netflix automagically finds the right rate... etc.

              That work for you? The buffering comes from bad shapers on the far side of the link. It's not just CMTSes that are awful. DSL is often horrific. I'm now seeing some 1G GPON networks with several seconds of downlink buffering. I guess they didn't get the memo.

              D T 3 Replies Last reply Reply Quote 1
              • D
                dtaht @dtaht
                last edited by

                I actually kind of wish I hadn't stopped work on "bobbie", a better policer. fq_codel is far too gentle and has the wrong goal for inbound shaping. Yes - it works better than anything we've ever tried, but a better policer would have zero delay for all packets at a similar cost in bandwidth and far less cpu. I think. Haven't got around to it. (basically you substitute achieving a rate in a codel like algorithm instead of a target delay). Tis research for someone else to do, I'm pretty broke after helping get sch_cake out the door.

                1 Reply Last reply Reply Quote 0
                • D
                  dtaht @dtaht
                  last edited by

                  Over here ( https://github.com/dtaht/fq_codel_fast ) I'm trying to speed up fq_codel a bit, and add an integrated multi-cpu inbound shaper.

                  1 Reply Last reply Reply Quote 1
                  • T
                    tman222 @dtaht
                    last edited by

                    @dtaht - thanks for the response - that makes a lot of sense.

                    Event though I used a cable modem in the example in my previous post, the question was really about GPON. I suppose what ends up happening is that there are buffers at at the GPON card (hardware) for both upload and download. If there is too much data being pushed into the link from upstream servers (i.e. a bunch of people downloading), the buffers start to fill up and packet delay (bufferbloat) occurs for downstream users. Since GPON bandwidth is generally shared among several users, I suppose the severity can vary depending on the amount of users, their usage patterns, and level of congestion. Furthermore, I suppose it's likely easier to experience bufferbloat on the uplink direction since GPON is asymmetric (2.4Gbit down / 1.2Gbit up).

                    In my personal experience with a gigabit GPON link I have been fortunate: I can set inbound/outbound shaping at >95% of max bandwidth and still not experience any significant delay/bloat.

                    1 Reply Last reply Reply Quote 0
                    • D
                      dtaht
                      last edited by

                      Yes, the GPON folk did not think hard about buffering, and it won't be much of a problem in their early deployments until they start oversubscribing more links. This is a flaw repeated time after time in this industry - 3g grew to suck, 4g was "better", 5g is going to fix 4g... (and 2g, 'cause so many have exited the band now, can be surprisingly good nowadays).

                      Recently I gave a presentation to broadcom ( http://flent-fremont.bufferbloat.net/~d/broadcom_aug9.pdf ) Another one of my hopes in the bufferbloat project was that someone would solve the over-subscription problem up front for a change - and I thought we'd have a chance with GPON and gfiber, but the team I was on got dissolved about 9 months from being able to deploy.

                      Sigh:

                      http://www.dslreports.com/speedtest/results/isp/r3910-google-fiber

                      1 Reply Last reply Reply Quote 0
                      • H
                        Harvy66
                        last edited by

                        I'm going to have some serious issues attempting to measure the reduction of bufferbloat from fq_Codel

                        This is with shaping disabled. Dear lord, what is my ISP doing? I love fiddling and they're taking that away from me. I just realized I forgot to disable BitTorrent. Explains my low upload speed.

                        0_1536092141802_ISPBloat20180904.PNG
                        0_1536092264272_ISPBloat20180904_updetail.PNG 0_1536092270317_ISPBloat20180904_downdetail.PNG

                        1 Reply Last reply Reply Quote 0
                        • occamsrazorO
                          occamsrazor
                          last edited by

                          I just updated from 2.4.3 to the 2.4.4 release candidate and am keen to try out fq_codel. I haven't read all 434 (!) posts of this thread, but can anyone kindly point me to a basic beginner's guide or instructions to setting it up for a simple home connection?

                          pfSense CE on Qotom Q355G4 8GB RAM/60GB SSD
                          Ubiquiti Unifi wired and wireless network, APC UPSs
                          Mac OSX and IOS devices, QNAP NAS

                          KOMK 1 Reply Last reply Reply Quote 0
                          • KOMK
                            KOM @occamsrazor
                            last edited by

                            @occamsrazor Jimp did a quick bit on codel during the last Hangout, with a config example.

                            https://www.youtube.com/watch?v=o8nL81DzTlU&t=380

                            occamsrazorO 1 Reply Last reply Reply Quote 1
                            • occamsrazorO
                              occamsrazor @KOM
                              last edited by

                              @kom said in Playing with fq_codel in 2.4:

                              @occamsrazor Jimp did a quick bit on codel during the last Hangout, with a config example.

                              https://www.youtube.com/watch?v=o8nL81DzTlU&t=380

                              Thanks. That's a nice and easy guide. I followed all the steps but wondering if I've done something wrong as the results are inferior to without. I'm on a 50MB dsl connection and before this got this result:

                              <a href="http://www.dslreports.com/speedtest/39052963"><img src="http://www.dslreports.com/speedtest/39052963.png"></a>

                              I had an A and A+ score so maybe the traffic shaper is not even needed. After enabling the shaper as per the video with WANDown bandwidth set to 50MB and WanUp set to 40MB, I get this:

                              <a href="http://www.dslreports.com/speedtest/39053743"><img src="http://www.dslreports.com/speedtest/39053743.png"></a>

                              My bufferbloat score is way worse as is my bandwidth. One thing I notice, when I go to PFSense > Status > Queues it says "Traffic shaping is not configured." - is that normal for this traffic shaping method?

                              pfSense CE on Qotom Q355G4 8GB RAM/60GB SSD
                              Ubiquiti Unifi wired and wireless network, APC UPSs
                              Mac OSX and IOS devices, QNAP NAS

                              Z 1 Reply Last reply Reply Quote 0
                              • Z
                                zwck @occamsrazor
                                last edited by zwck

                                @occamsrazor

                                go to command prompt and type "ipfw sched show" and print the output please

                                typo :( sorry

                                occamsrazorO 1 Reply Last reply Reply Quote 0
                                • occamsrazorO
                                  occamsrazor @zwck
                                  last edited by

                                  @zwck said in Playing with fq_codel in 2.4:

                                  ipfw shed show

                                  Thanks, output is:

                                  ipfw: bad command `shed'
                                  

                                  I set up the two limiters as in the video:

                                  0_1537256451183_Screen Shot 2018-09-18 at 10.39.21.jpg

                                  And the rule in the Firewall Floating tab:

                                  0_1537256567789_Screen Shot 2018-09-18 at 10.41.45.jpg

                                  Is there supposed to be something showing in the queues tab?

                                  0_1537256480988_Screen Shot 2018-09-18 at 10.39.28.jpg

                                  pfSense CE on Qotom Q355G4 8GB RAM/60GB SSD
                                  Ubiquiti Unifi wired and wireless network, APC UPSs
                                  Mac OSX and IOS devices, QNAP NAS

                                  1 Reply Last reply Reply Quote 0
                                  • GertjanG
                                    Gertjan
                                    last edited by

                                    ipfw sched show
                                    

                                    No "help me" PM's please. Use the forum, the community will thank you.
                                    Edit : and where are the logs ??

                                    occamsrazorO 1 Reply Last reply Reply Quote 0
                                    • occamsrazorO
                                      occamsrazor @Gertjan
                                      last edited by

                                      @gertjan said in Playing with fq_codel in 2.4:

                                      ipfw sched show
                                      

                                      Thanks, output is:

                                      00001:  50.000 Mbit/s    0 ms burst 0
                                      q65537  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
                                       sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
                                       FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
                                         Children flowsets: 1
                                      BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
                                        0 ip           0.0.0.0/0             0.0.0.0/0        2      738  0    0   0
                                      00002:  40.000 Mbit/s    0 ms burst 0
                                      q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
                                       sched 2 type FQ_CODEL flags 0x0 0 buckets 0 active
                                       FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
                                         Children flowsets: 2
                                      

                                      pfSense CE on Qotom Q355G4 8GB RAM/60GB SSD
                                      Ubiquiti Unifi wired and wireless network, APC UPSs
                                      Mac OSX and IOS devices, QNAP NAS

                                      1 Reply Last reply Reply Quote 0
                                      • Z
                                        zwck
                                        last edited by zwck

                                        @occamsrazor said in Playing with fq_codel in 2.4:

                                        FQ_CODEL

                                        sched shed shit :D

                                        sorry for the typo, when you run the dslspeedtest you should see in this output if fq_codel is active, which from here it seems like it is.

                                        1 Reply Last reply Reply Quote 0
                                        • Z
                                          zwck
                                          last edited by

                                          @mattund I watched the video linked before and end up with more question then before. Do we use masks or not. ? Do we use ECN or not, i remember on the upload side we should not use ECN.

                                          1 Reply Last reply Reply Quote 0
                                          • occamsrazorO
                                            occamsrazor
                                            last edited by occamsrazor

                                            Not sure what changed but now my setup seems to be working OK...

                                            https://www.dslreports.com/speedtest/39073625

                                            This while doing speedtest:

                                            Shell Output - ipfw sched show
                                            
                                            00001:  50.000 Mbit/s    0 ms burst 0
                                            q65537  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
                                             sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
                                             FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
                                               Children flowsets: 1
                                            BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
                                              0 ip           0.0.0.0/0             0.0.0.0/0      653   974276  0    0   0
                                            00002:  40.000 Mbit/s    0 ms burst 0
                                            q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
                                             sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
                                             FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
                                               Children flowsets: 2
                                              0 ip           0.0.0.0/0             0.0.0.0/0       59     3149  0    0   0
                                            

                                            Is fq_codel adaptive to bandwidth in some way? I'm wondering what to put for the WAN bandwidth values in the limiter, as my speeds can sometimes vary during the day.

                                            Also, I am running pfBlocker developer version. Should the fq_codel floating rule be above or below all the pfb rules?

                                            pfSense CE on Qotom Q355G4 8GB RAM/60GB SSD
                                            Ubiquiti Unifi wired and wireless network, APC UPSs
                                            Mac OSX and IOS devices, QNAP NAS

                                            X KOMK 2 Replies Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.