• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

Playing with fq_codel in 2.4

Traffic Shaping
123
1.1k
1.4m
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • H
    Harvy66
    last edited by Sep 6, 2017, 1:12 PM

    Sounds like something is wrong. My i5-3.2ghz Haswell Intel-i350T2 is handling 1.44Mpps with HFSC+Codel just fine. 400Kpps is about 15-20% load and 1.44Mpps is about 20%-25% load. And I still get about 1ms pings while doing this.

    I have some forum posts around about what changes I made, like enabling MSI-X (enable soft interrupts if your NIC properly supports it) and removing the limitation on the number of packets to process per interrupt, which is by default 40 I think.

    I used to process 1.44Mpps at only 600 interrupts per second, but over the years, it is now about 1,200 interrupts per second. All I know is CPU is low, interrupts are low, latency and jitter and loss are low. Even with only 300 interrupts per core per second.

    1 Reply Last reply Reply Quote 0
    • C
      chrcoluk
      last edited by Sep 6, 2017, 2:03 PM

      agree something is wrong, problem is I dont know what at the moment. :)

      Will diagnose some more at some point.

      MSIX is enabled by default on my i350, I actually temporarily disabled it already to try and get to bottom of it but MSIX in this case is not the solution. The packets per interrupt is something I not heard off, you know where that is configured?

      pfSense CE 2.7.2

      1 Reply Last reply Reply Quote 0
      • H
        Harvy66
        last edited by Sep 6, 2017, 3:43 PM Sep 6, 2017, 3:35 PM

        @chrcoluk:

        agree something is wrong, problem is I dont know what at the moment. :)

        Will diagnose some more at some point.

        MSIX is enabled by default on my i350, I actually temporarily disabled it already to try and get to bottom of it but MSIX in this case is not the solution. The packets per interrupt is something I not heard off, you know where that is configured?

        I guess it's not 40. Not sure what "40" is. I definitely remembering something.

        https://calomel.org/freebsd_network_tuning.html

        Intel igb(4): FreeBSD puts an upper limit on the the number of received

        packets a network card can process to 100 packets per interrupt cycle. This

        limit is in place because of inefficiencies in IRQ sharing when the network

        card is using the same IRQ as another device. When the Intel network card is

        assigned a unique IRQ (dmesg) and MSI-X is enabled through the driver

        (hw.igb.enable_msix=1) then interrupt scheduling is significantly more

        efficient and the NIC can be allowed to process packets as fast as they are

        received. A value of "-1" means unlimited packet processing and sets the same

        value to dev.igb.0.rx_processing_limit and dev.igb.1.rx_processing_limit .

        hw.igb.rx_process_limit="-1"  # (default 100 pps, packets per second)

        I want to say that when I echo'd my system settings, msix was disabled and I had to explicitly add it to pfSense to enabled it. I'm also under the impression that many NICs that claim to support MSIX, do not correctly and have odd bugs when msix is enabled, so it was turned off. I'm going off of memory from a long while ago when I was researching.

        1 Reply Last reply Reply Quote 0
        • C
          chrcoluk
          last edited by Sep 6, 2017, 3:53 PM

          yeah mine has definitely been on, checked via dmesg.

          I will turn it back on, I only turned off just to check if somehow it was not working properly.

          pfSense CE 2.7.2

          1 Reply Last reply Reply Quote 0
          • B
            belt9
            last edited by Sep 7, 2017, 9:40 AM

            I've got some great results with fq_codel, it makes significant improvements on wifi where latency can be a real problem!

            https://forum.pfsense.org/index.php?topic=135843.msg745944#msg745944

            1 Reply Last reply Reply Quote 0
            • C
              chrcoluk
              last edited by Sep 9, 2017, 6:17 PM

              I think its interrupt related.

              I enabled polling on both nic's in use and the packet loss when downloading of steam or meganz is gone.

              Before I enabled polling I was observing the interrupts/sec, first the AIM (adaptive interrupt moderation) seems to not be working on my i350 ports as it has no affect, secondly I will generate 8000/sec interrupts for a download of about 70mbit/sec.  I see people on here reporting similar usage but for half a gigabit/sec, so it seems two things pointing to moderation not working.

              altq reports around 2000 pps, assuming thats accurate then some how I have 4 interrupts for every packet.

              I have seen threads on here regarding fake i350's, I dont think its impossible mine is fake which could explain the broken AIM.

              pfSense CE 2.7.2

              1 Reply Last reply Reply Quote 0
              • H
                Harvy66
                last edited by Sep 10, 2017, 6:00 AM

                You can get the number off of your nic and run it through Intel.com to see if it's valid.
                https://www.intel.com/content/www/us/en/support/network-and-i-o/ethernet-products/000007074.html

                1 Reply Last reply Reply Quote 0
                • W
                  w0w
                  last edited by Sep 10, 2017, 2:01 PM Sep 10, 2017, 12:56 PM

                  How did you enable polling?
                  I am not quite sure this is good idea anyway, but I can't see this option in GUI  anymore, is it moved somewhere?
                  You'd better try to add boot.conf.local "safe" settings for Intel cards
                  hw.igb.num_queues=2
                  dev.igb.0.fc=0
                  dev.igb.1.fc=0
                  Also I have disabled hardware TCP segmentation offload and hardware large receive offload.
                  P.S. In addition to Harvy66 post
                  https://forums.servethehome.com/index.php?threads/chinese-intel-i350-adapters.3746/#post-58686

                  1 Reply Last reply Reply Quote 0
                  • C
                    chrcoluk
                    last edited by Sep 10, 2017, 2:00 PM Sep 10, 2017, 1:57 PM

                    I compiled it into the kernel and then enabled it in the cli via ifconfig, I dont think you can enable it with a module.

                    Polling is seen as pointless now days, but only because modern intel and broadcom hardware do interrupt moderation, for some reason my i350 isnt moderating the interrupts.

                    I have already disabled flow control and played with the queues, all that made no difference.

                    When I can be bothered I will check how harvy said, although I am already pretty sure I had no such label on my card.

                    pfSense CE 2.7.2

                    1 Reply Last reply Reply Quote 0
                    • W
                      w0w
                      last edited by Sep 10, 2017, 2:01 PM

                      Looks like most of those chinese Intel cards are failing because of cheap fake Delta ethernet transformers used on their boards, but it always come up with packet loss and I don't think that AIM could be affected. I think they all using Intel chipset not some fake remarked crap realtek.

                      1 Reply Last reply Reply Quote 0
                      • C
                        chrcoluk
                        last edited by Sep 10, 2017, 2:10 PM

                        Well I am just speculating.

                        There is no difference in the interrupts with aim on or off, and the amount of interrupts generated seems high for the pps/bw used.  So that suggests to me aim isnt working.

                        pfSense CE 2.7.2

                        1 Reply Last reply Reply Quote 0
                        • W
                          w0w
                          last edited by Sep 10, 2017, 3:47 PM

                          I have not tested AIM in any way you did it, so I can't say why it's working or not.
                          Also I've never seen packet loss on my igb cards, some chinese, not fake, but Winayo branded Intel NICs.
                          https://forums.freebsd.org/threads/62406/#post-360467 I see you are already digging deeper.
                          Did you already read that https://forums.servethehome.com/index.php?threads/comparison-intel-i350-t4-genuine-vs-fake.6917/ ? We must be 100 percent sure which card you have exactly. May be some photos?

                          1 Reply Last reply Reply Quote 0
                          • C
                            chrcoluk
                            last edited by Sep 10, 2017, 4:25 PM

                            yep read that thread which is why I made that comment about the fake cards.  When I can be bothered I will take closer photos of the card both sides.

                            I also yes have been considering flashing the firmware, but is not much documentation on how to use bootutil, the only guides I found were when using a uefi shell but they do not explain how to boot into a uefi shell.

                            pfSense CE 2.7.2

                            1 Reply Last reply Reply Quote 0
                            • W
                              w0w
                              last edited by Sep 11, 2017, 2:45 AM Sep 10, 2017, 6:38 PM

                              Most of the UEFI motherboards will eat the bootmgr.efi placed in EFI\Microsoft\Boot\bootmgr.efi on FAT32 formatted USB flash.
                              Here you can find how to obtain it https://wiki.archlinux.org/index.php/Unified_Extensible_Firmware_Interface#Obtaining_UEFI_Shell
                              I've used binaries from https://github.com/tianocore/edk2/tree/master/ShellBinPkg
                              From what I've learned reading forums the first sign that this is a fake is the price and redrawn "Delta-like" ethernet transformers on it.

                              1 Reply Last reply Reply Quote 0
                              • C
                                chrcoluk
                                last edited by Sep 11, 2017, 2:55 PM

                                thanks for the help guys I do appreciate it.

                                I havent really got alternative kit to test on, but I managed to get pfsense running in esxi, after spending a couple of hours looking for a second nic (dual port server class intel nic I put in there both ports dead) and using vmx drivers in 2.4-RC is fine without polling.  So I am putting this down to a dodgy hardware setup and I think I am going to replace my unit.

                                But I am now going to leave this VM running so I can take the photos and play around with the firmware in the meantime without worrying about downtime.

                                pfSense CE 2.7.2

                                1 Reply Last reply Reply Quote 0
                                • P
                                  pf3000
                                  last edited by Sep 11, 2017, 8:54 PM

                                  @chrcoluk:

                                  I also yes have been considering flashing the firmware, but is not much documentation on how to use bootutil, the only guides I found were when using a uefi shell but they do not explain how to boot into a uefi shell.

                                  I described how to flash via EFI here https://forum.pfsense.org/index.php?topic=112968.msg629211#msg629211

                                  1 Reply Last reply Reply Quote 0
                                  • C
                                    chrcoluk
                                    last edited by Sep 12, 2017, 6:48 PM Sep 11, 2017, 10:40 PM

                                    thanks is flashed, will post pics tomorrow but in a different thread as I am derailing this thread too much. I will edit this post with the link after I posted.

                                    thread here https://forum.pfsense.org/index.php?topic=136561.new#new

                                    pfSense CE 2.7.2

                                    1 Reply Last reply Reply Quote 0
                                    • W
                                      w0w
                                      last edited by Sep 14, 2017, 4:04 PM Sep 14, 2017, 3:52 AM

                                      Moved to https://forum.pfsense.org/index.php?topic=136561.new#new

                                      1 Reply Last reply Reply Quote 0
                                      • H
                                        Harvy66
                                        last edited by Sep 14, 2017, 3:01 PM

                                        If it's an official Intel NIC, it will have a YottaMark sticker

                                        According to Intel, if your NIC does not have a YottaMark, it is defective and should be returned. Without the YottaMark, you have no access to any warranty claims or support.

                                        1 Reply Last reply Reply Quote 0
                                        • T
                                          tman222
                                          last edited by Sep 21, 2017, 1:01 AM

                                          I recently upgraded to 2.4.0-RC so I could give fq_codel a try.  Up to now I had been using the ALTQ FAIRQ scheduler together with codel managed queues to sort of emulate fq_codel.  I disabled my ALTQ shaping settings and I followed the steps in the original post.  After configuring everything I did a:

                                          ipfw sched show

                                          I could see fq_codel enabled but I could not see any traffic passing through.  However, I then also tried a speed test over at DSL Reports and could see fq_codel working, i.e. I had a A+ on the bufferbloat score.  This left me a little perplexed.  I could not see traffic passing through the fq_codel queues but yet it seemed to be working.  Is there a step I might have missed to make sure I can also see traffic passing through the queues?

                                          Thanks in advance for your help.

                                          1 Reply Last reply Reply Quote 0
                                          88 out of 1108
                                          • First post
                                            88/1108
                                            Last post
                                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.