Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Playing with fq_codel in 2.4

    Scheduled Pinned Locked Moved
    Traffic Shaping
    123
    1.1k
    1.2m
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • B
      belt9
      last edited by

      I've got some great results with fq_codel, it makes significant improvements on wifi where latency can be a real problem!

      https://forum.pfsense.org/index.php?topic=135843.msg745944#msg745944

      1 Reply Last reply Reply Quote 0
      • C
        chrcoluk
        last edited by

        I think its interrupt related.

        I enabled polling on both nic's in use and the packet loss when downloading of steam or meganz is gone.

        Before I enabled polling I was observing the interrupts/sec, first the AIM (adaptive interrupt moderation) seems to not be working on my i350 ports as it has no affect, secondly I will generate 8000/sec interrupts for a download of about 70mbit/sec.  I see people on here reporting similar usage but for half a gigabit/sec, so it seems two things pointing to moderation not working.

        altq reports around 2000 pps, assuming thats accurate then some how I have 4 interrupts for every packet.

        I have seen threads on here regarding fake i350's, I dont think its impossible mine is fake which could explain the broken AIM.

        pfSense CE 2.7.2

        1 Reply Last reply Reply Quote 0
        • H
          Harvy66
          last edited by

          You can get the number off of your nic and run it through Intel.com to see if it's valid.
          https://www.intel.com/content/www/us/en/support/network-and-i-o/ethernet-products/000007074.html

          1 Reply Last reply Reply Quote 0
          • w0wW
            w0w
            last edited by

            How did you enable polling?
            I am not quite sure this is good idea anyway, but I can't see this option in GUI  anymore, is it moved somewhere?
            You'd better try to add boot.conf.local "safe" settings for Intel cards
            hw.igb.num_queues=2
            dev.igb.0.fc=0
            dev.igb.1.fc=0
            Also I have disabled hardware TCP segmentation offload and hardware large receive offload.
            P.S. In addition to Harvy66 post
            https://forums.servethehome.com/index.php?threads/chinese-intel-i350-adapters.3746/#post-58686

            1 Reply Last reply Reply Quote 0
            • C
              chrcoluk
              last edited by

              I compiled it into the kernel and then enabled it in the cli via ifconfig, I dont think you can enable it with a module.

              Polling is seen as pointless now days, but only because modern intel and broadcom hardware do interrupt moderation, for some reason my i350 isnt moderating the interrupts.

              I have already disabled flow control and played with the queues, all that made no difference.

              When I can be bothered I will check how harvy said, although I am already pretty sure I had no such label on my card.

              pfSense CE 2.7.2

              1 Reply Last reply Reply Quote 0
              • w0wW
                w0w
                last edited by

                Looks like most of those chinese Intel cards are failing because of cheap fake Delta ethernet transformers used on their boards, but it always come up with packet loss and I don't think that AIM could be affected. I think they all using Intel chipset not some fake remarked crap realtek.

                1 Reply Last reply Reply Quote 0
                • C
                  chrcoluk
                  last edited by

                  Well I am just speculating.

                  There is no difference in the interrupts with aim on or off, and the amount of interrupts generated seems high for the pps/bw used.  So that suggests to me aim isnt working.

                  pfSense CE 2.7.2

                  1 Reply Last reply Reply Quote 0
                  • w0wW
                    w0w
                    last edited by

                    I have not tested AIM in any way you did it, so I can't say why it's working or not.
                    Also I've never seen packet loss on my igb cards, some chinese, not fake, but Winayo branded Intel NICs.
                    https://forums.freebsd.org/threads/62406/#post-360467 I see you are already digging deeper.
                    Did you already read that https://forums.servethehome.com/index.php?threads/comparison-intel-i350-t4-genuine-vs-fake.6917/ ? We must be 100 percent sure which card you have exactly. May be some photos?

                    1 Reply Last reply Reply Quote 0
                    • C
                      chrcoluk
                      last edited by

                      yep read that thread which is why I made that comment about the fake cards.  When I can be bothered I will take closer photos of the card both sides.

                      I also yes have been considering flashing the firmware, but is not much documentation on how to use bootutil, the only guides I found were when using a uefi shell but they do not explain how to boot into a uefi shell.

                      pfSense CE 2.7.2

                      1 Reply Last reply Reply Quote 0
                      • w0wW
                        w0w
                        last edited by

                        Most of the UEFI motherboards will eat the bootmgr.efi placed in EFI\Microsoft\Boot\bootmgr.efi on FAT32 formatted USB flash.
                        Here you can find how to obtain it https://wiki.archlinux.org/index.php/Unified_Extensible_Firmware_Interface#Obtaining_UEFI_Shell
                        I've used binaries from https://github.com/tianocore/edk2/tree/master/ShellBinPkg
                        From what I've learned reading forums the first sign that this is a fake is the price and redrawn "Delta-like" ethernet transformers on it.

                        1 Reply Last reply Reply Quote 0
                        • C
                          chrcoluk
                          last edited by

                          thanks for the help guys I do appreciate it.

                          I havent really got alternative kit to test on, but I managed to get pfsense running in esxi, after spending a couple of hours looking for a second nic (dual port server class intel nic I put in there both ports dead) and using vmx drivers in 2.4-RC is fine without polling.  So I am putting this down to a dodgy hardware setup and I think I am going to replace my unit.

                          But I am now going to leave this VM running so I can take the photos and play around with the firmware in the meantime without worrying about downtime.

                          pfSense CE 2.7.2

                          1 Reply Last reply Reply Quote 0
                          • P
                            pf3000
                            last edited by

                            @chrcoluk:

                            I also yes have been considering flashing the firmware, but is not much documentation on how to use bootutil, the only guides I found were when using a uefi shell but they do not explain how to boot into a uefi shell.

                            I described how to flash via EFI here https://forum.pfsense.org/index.php?topic=112968.msg629211#msg629211

                            1 Reply Last reply Reply Quote 0
                            • C
                              chrcoluk
                              last edited by

                              thanks is flashed, will post pics tomorrow but in a different thread as I am derailing this thread too much. I will edit this post with the link after I posted.

                              thread here https://forum.pfsense.org/index.php?topic=136561.new#new

                              pfSense CE 2.7.2

                              1 Reply Last reply Reply Quote 0
                              • w0wW
                                w0w
                                last edited by

                                Moved to https://forum.pfsense.org/index.php?topic=136561.new#new

                                1 Reply Last reply Reply Quote 0
                                • H
                                  Harvy66
                                  last edited by

                                  If it's an official Intel NIC, it will have a YottaMark sticker

                                  According to Intel, if your NIC does not have a YottaMark, it is defective and should be returned. Without the YottaMark, you have no access to any warranty claims or support.

                                  1 Reply Last reply Reply Quote 0
                                  • T
                                    tman222
                                    last edited by

                                    I recently upgraded to 2.4.0-RC so I could give fq_codel a try.  Up to now I had been using the ALTQ FAIRQ scheduler together with codel managed queues to sort of emulate fq_codel.  I disabled my ALTQ shaping settings and I followed the steps in the original post.  After configuring everything I did a:

                                    ipfw sched show

                                    I could see fq_codel enabled but I could not see any traffic passing through.  However, I then also tried a speed test over at DSL Reports and could see fq_codel working, i.e. I had a A+ on the bufferbloat score.  This left me a little perplexed.  I could not see traffic passing through the fq_codel queues but yet it seemed to be working.  Is there a step I might have missed to make sure I can also see traffic passing through the queues?

                                    Thanks in advance for your help.

                                    1 Reply Last reply Reply Quote 0
                                    • johnpozJ
                                      johnpoz LAYER 8 Global Moderator
                                      last edited by

                                      Did you look at show when you know there was data flowing?

                                      So for example… I did the ipfw sched show.  Then I started the dslreport test and looked at it while it was running

                                      
[2.4.0-RC][root@pfsense.local.lan]/root: ipfw sched show
00001:  85.000 Mbit/s    0 ms burst 0 
q00001  50 sl. 0 flows (256 buckets) sched 1 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
 sched 1 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1 
00002:  11.000 Mbit/s    0 ms burst 0 
q00002  50 sl. 0 flows (256 buckets) sched 2 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2 
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0        1      262  0    0   0
[2.4.0-RC][root@pfsense.local.lan]/root: ipfw sched show
00001:  85.000 Mbit/s    0 ms burst 0 
q00001  50 sl. 0 flows (256 buckets) sched 1 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1 
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0     16133 24150846 81 121500  50
00002:  11.000 Mbit/s    0 ms burst 0 
q00002  50 sl. 0 flows (256 buckets) sched 2 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2 
  0 ip           0.0.0.0/0             0.0.0.0/0     1169    50866  0    0   0
[2.4.0-RC][root@pfsense.local.lan]/root: ipfw sched show
00001:  85.000 Mbit/s    0 ms burst 0 
q00001  50 sl. 0 flows (256 buckets) sched 1 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1 
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0       77     3563  0    0   0
00002:  11.000 Mbit/s    0 ms burst 0 
q00002  50 sl. 0 flows (256 buckets) sched 2 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2 
  0 ip           0.0.0.0/0             0.0.0.0/0     3244  4740145  7 10500  74
[2.4.0-RC][root@pfsense.local.lan]/root:

                                      Got A+ for quality, A for bufferbloat and A+ overall… This really is easy to implement too.. Be nice when can be fully done in the gui though.

                                      An intelligent man is sometimes forced to be drunk to spend time with his fools
                                      If you get confused: Listen to the Music Play
                                      Please don't Chat/PM me for help, unless mod related
                                      SG-4860 24.03 | Lab VMs 2.7.2, 24.03

                                      1 Reply Last reply Reply Quote 0
                                      • T
                                        tman222
                                        last edited by

                                        @johnpoz:

                                        Did you look at show when you know there was data flowing?

                                        So for example… I did the ipfw sched show.  Then I started the dslreport test and looked at it while it was running

                                        
[2.4.0-RC][root@pfsense.local.lan]/root: ipfw sched show
00001:  85.000 Mbit/s    0 ms burst 0 
q00001  50 sl. 0 flows (256 buckets) sched 1 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
 sched 1 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1 
00002:  11.000 Mbit/s    0 ms burst 0 
q00002  50 sl. 0 flows (256 buckets) sched 2 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2 
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0        1      262  0    0   0
[2.4.0-RC][root@pfsense.local.lan]/root: ipfw sched show
00001:  85.000 Mbit/s    0 ms burst 0 
q00001  50 sl. 0 flows (256 buckets) sched 1 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1 
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0     16133 24150846 81 121500  50
00002:  11.000 Mbit/s    0 ms burst 0 
q00002  50 sl. 0 flows (256 buckets) sched 2 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2 
  0 ip           0.0.0.0/0             0.0.0.0/0     1169    50866  0    0   0
[2.4.0-RC][root@pfsense.local.lan]/root: ipfw sched show
00001:  85.000 Mbit/s    0 ms burst 0 
q00001  50 sl. 0 flows (256 buckets) sched 1 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1 
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0       77     3563  0    0   0
00002:  11.000 Mbit/s    0 ms burst 0 
q00002  50 sl. 0 flows (256 buckets) sched 2 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2 
  0 ip           0.0.0.0/0             0.0.0.0/0     3244  4740145  7 10500  74
[2.4.0-RC][root@pfsense.local.lan]/root:

                                        Got A+ for quality, A for bufferbloat and A+ overall… This really is easy to implement too.. Be nice when can be fully done in the gui though.

                                        Thanks John, I really appreciate your help on this.  After I setup everything and ran a test at DSL Reports I actually did not continue to refresh ipfw sched show manually.  I naively thought that it would update automatically when traffic is passing through the queues.  That being said, is this the case though for the "Limiter Information" section under Diagnostics in the Web UI (i.e. does that section refresh automatically and show traffic passing through the queues)?

                                        I think I'm going to try this out again.  However, I think my implementation needs to be a little different than was suggested in the original post.  I actually have more than 1 LAN interface on my pfSense box and each interface handles a different subnet.  In order to not limit the amount of bandwidth between the subnets, I don't think I should put the limiter queues on the default allow all rule on the subnets.  Instead, should I just setup a floating firewall rule that matches on WAN traffic?  If that makes sense, what other settings do I need configure (besides choosing the limiter queues at the bottom)?  For instance, what should be the source and destination?

                                        Thanks again for all your help.

                                        1 Reply Last reply Reply Quote 0
                                        • w0wW
                                          w0w
                                          last edited by

                                          Limiter info GUI do not show anything just by design of current fq_соdel implementation in both FreeBSD and pfSense, so it's normal that you don't see traffic there.

                                          1 Reply Last reply Reply Quote 0
                                          • T
                                            tman222
                                            last edited by

                                            So I tried setting up fq_codel again and confirm that it is working just fine (just had to refresh the "ipfw sched show" command as traffic was passing through the queues).  I did however, notice a lot of dropped packets while running a speed test.  I have symmetric gigabit connection and also had this issue with the ALTQ traffic shappers.  The solution was to increase the queue size sufficiently.  When using dummynet limiters is the default queue size adjusted under Advanced Options for the queue (that was created under the root limiter) or is it somewhere else?

                                            Also, I'm still not quite sure how to configure my firewall rules so I don't accidentally limit traffic between local subnets, i.e. I only want to shape traffic internet bound traffic.  What would be the best way to do this (as using the default allow all rule on the LAN will impact subnet traffic too)?

                                            Thanks again for your help, I really appreciate it.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post