Playing with fq_codel in 2.4



  • @chrcoluk:

    agree something is wrong, problem is I dont know what at the moment. :)

    Will diagnose some more at some point.

    MSIX is enabled by default on my i350, I actually temporarily disabled it already to try and get to bottom of it but MSIX in this case is not the solution. The packets per interrupt is something I not heard off, you know where that is configured?

    I guess it's not 40. Not sure what "40" is. I definitely remembering something.

    https://calomel.org/freebsd_network_tuning.html

    Intel igb(4): FreeBSD puts an upper limit on the the number of received

    packets a network card can process to 100 packets per interrupt cycle. This

    limit is in place because of inefficiencies in IRQ sharing when the network

    card is using the same IRQ as another device. When the Intel network card is

    assigned a unique IRQ (dmesg) and MSI-X is enabled through the driver

    (hw.igb.enable_msix=1) then interrupt scheduling is significantly more

    efficient and the NIC can be allowed to process packets as fast as they are

    received. A value of "-1" means unlimited packet processing and sets the same

    value to dev.igb.0.rx_processing_limit and dev.igb.1.rx_processing_limit .

    hw.igb.rx_process_limit="-1"  # (default 100 pps, packets per second)

    I want to say that when I echo'd my system settings, msix was disabled and I had to explicitly add it to pfSense to enabled it. I'm also under the impression that many NICs that claim to support MSIX, do not correctly and have odd bugs when msix is enabled, so it was turned off. I'm going off of memory from a long while ago when I was researching.



  • yeah mine has definitely been on, checked via dmesg.

    I will turn it back on, I only turned off just to check if somehow it was not working properly.



  • I've got some great results with fq_codel, it makes significant improvements on wifi where latency can be a real problem!

    https://forum.pfsense.org/index.php?topic=135843.msg745944#msg745944



  • I think its interrupt related.

    I enabled polling on both nic's in use and the packet loss when downloading of steam or meganz is gone.

    Before I enabled polling I was observing the interrupts/sec, first the AIM (adaptive interrupt moderation) seems to not be working on my i350 ports as it has no affect, secondly I will generate 8000/sec interrupts for a download of about 70mbit/sec.  I see people on here reporting similar usage but for half a gigabit/sec, so it seems two things pointing to moderation not working.

    altq reports around 2000 pps, assuming thats accurate then some how I have 4 interrupts for every packet.

    I have seen threads on here regarding fake i350's, I dont think its impossible mine is fake which could explain the broken AIM.



  • You can get the number off of your nic and run it through Intel.com to see if it's valid.
    https://www.intel.com/content/www/us/en/support/network-and-i-o/ethernet-products/000007074.html



  • How did you enable polling?
    I am not quite sure this is good idea anyway, but I can't see this option in GUI  anymore, is it moved somewhere?
    You'd better try to add boot.conf.local "safe" settings for Intel cards
    hw.igb.num_queues=2
    dev.igb.0.fc=0
    dev.igb.1.fc=0
    Also I have disabled hardware TCP segmentation offload and hardware large receive offload.
    P.S. In addition to Harvy66 post
    https://forums.servethehome.com/index.php?threads/chinese-intel-i350-adapters.3746/#post-58686



  • I compiled it into the kernel and then enabled it in the cli via ifconfig, I dont think you can enable it with a module.

    Polling is seen as pointless now days, but only because modern intel and broadcom hardware do interrupt moderation, for some reason my i350 isnt moderating the interrupts.

    I have already disabled flow control and played with the queues, all that made no difference.

    When I can be bothered I will check how harvy said, although I am already pretty sure I had no such label on my card.



  • Looks like most of those chinese Intel cards are failing because of cheap fake Delta ethernet transformers used on their boards, but it always come up with packet loss and I don't think that AIM could be affected. I think they all using Intel chipset not some fake remarked crap realtek.



  • Well I am just speculating.

    There is no difference in the interrupts with aim on or off, and the amount of interrupts generated seems high for the pps/bw used.  So that suggests to me aim isnt working.



  • I have not tested AIM in any way you did it, so I can't say why it's working or not.
    Also I've never seen packet loss on my igb cards, some chinese, not fake, but Winayo branded Intel NICs.
    https://forums.freebsd.org/threads/62406/#post-360467 I see you are already digging deeper.
    Did you already read that https://forums.servethehome.com/index.php?threads/comparison-intel-i350-t4-genuine-vs-fake.6917/ ? We must be 100 percent sure which card you have exactly. May be some photos?



  • yep read that thread which is why I made that comment about the fake cards.  When I can be bothered I will take closer photos of the card both sides.

    I also yes have been considering flashing the firmware, but is not much documentation on how to use bootutil, the only guides I found were when using a uefi shell but they do not explain how to boot into a uefi shell.



  • Most of the UEFI motherboards will eat the bootmgr.efi placed in EFI\Microsoft\Boot\bootmgr.efi on FAT32 formatted USB flash.
    Here you can find how to obtain it https://wiki.archlinux.org/index.php/Unified_Extensible_Firmware_Interface#Obtaining_UEFI_Shell
    I've used binaries from https://github.com/tianocore/edk2/tree/master/ShellBinPkg
    From what I've learned reading forums the first sign that this is a fake is the price and redrawn "Delta-like" ethernet transformers on it.



  • thanks for the help guys I do appreciate it.

    I havent really got alternative kit to test on, but I managed to get pfsense running in esxi, after spending a couple of hours looking for a second nic (dual port server class intel nic I put in there both ports dead) and using vmx drivers in 2.4-RC is fine without polling.  So I am putting this down to a dodgy hardware setup and I think I am going to replace my unit.

    But I am now going to leave this VM running so I can take the photos and play around with the firmware in the meantime without worrying about downtime.



  • @chrcoluk:

    I also yes have been considering flashing the firmware, but is not much documentation on how to use bootutil, the only guides I found were when using a uefi shell but they do not explain how to boot into a uefi shell.

    I described how to flash via EFI here https://forum.pfsense.org/index.php?topic=112968.msg629211#msg629211



  • thanks is flashed, will post pics tomorrow but in a different thread as I am derailing this thread too much. I will edit this post with the link after I posted.

    thread here https://forum.pfsense.org/index.php?topic=136561.new#new





  • If it's an official Intel NIC, it will have a YottaMark sticker

    According to Intel, if your NIC does not have a YottaMark, it is defective and should be returned. Without the YottaMark, you have no access to any warranty claims or support.



  • I recently upgraded to 2.4.0-RC so I could give fq_codel a try.  Up to now I had been using the ALTQ FAIRQ scheduler together with codel managed queues to sort of emulate fq_codel.  I disabled my ALTQ shaping settings and I followed the steps in the original post.  After configuring everything I did a:

    ipfw sched show

    I could see fq_codel enabled but I could not see any traffic passing through.  However, I then also tried a speed test over at DSL Reports and could see fq_codel working, i.e. I had a A+ on the bufferbloat score.  This left me a little perplexed.  I could not see traffic passing through the fq_codel queues but yet it seemed to be working.  Is there a step I might have missed to make sure I can also see traffic passing through the queues?

    Thanks in advance for your help.


  • LAYER 8 Global Moderator

    Did you look at show when you know there was data flowing?

    So for example… I did the ipfw sched show.  Then I started the dslreport test and looked at it while it was running

    
    [2.4.0-RC][root@pfsense.local.lan]/root: ipfw sched show
    00001:  85.000 Mbit/s    0 ms burst 0 
    q00001  50 sl. 0 flows (256 buckets) sched 1 weight 1 lmax 0 pri 0 droptail
        mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
     sched 1 type FQ_CODEL flags 0x0 0 buckets 0 active
     FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
       Children flowsets: 1 
    00002:  11.000 Mbit/s    0 ms burst 0 
    q00002  50 sl. 0 flows (256 buckets) sched 2 weight 1 lmax 0 pri 0 droptail
        mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000
     sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
     FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
       Children flowsets: 2 
    BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
      0 ip           0.0.0.0/0             0.0.0.0/0        1      262  0    0   0
    [2.4.0-RC][root@pfsense.local.lan]/root: ipfw sched show
    00001:  85.000 Mbit/s    0 ms burst 0 
    q00001  50 sl. 0 flows (256 buckets) sched 1 weight 1 lmax 0 pri 0 droptail
        mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
     sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
     FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
       Children flowsets: 1 
    BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
      0 ip           0.0.0.0/0             0.0.0.0/0     16133 24150846 81 121500  50
    00002:  11.000 Mbit/s    0 ms burst 0 
    q00002  50 sl. 0 flows (256 buckets) sched 2 weight 1 lmax 0 pri 0 droptail
        mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000
     sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
     FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
       Children flowsets: 2 
      0 ip           0.0.0.0/0             0.0.0.0/0     1169    50866  0    0   0
    [2.4.0-RC][root@pfsense.local.lan]/root: ipfw sched show
    00001:  85.000 Mbit/s    0 ms burst 0 
    q00001  50 sl. 0 flows (256 buckets) sched 1 weight 1 lmax 0 pri 0 droptail
        mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
     sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
     FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
       Children flowsets: 1 
    BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
      0 ip           0.0.0.0/0             0.0.0.0/0       77     3563  0    0   0
    00002:  11.000 Mbit/s    0 ms burst 0 
    q00002  50 sl. 0 flows (256 buckets) sched 2 weight 1 lmax 0 pri 0 droptail
        mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000
     sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
     FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
       Children flowsets: 2 
      0 ip           0.0.0.0/0             0.0.0.0/0     3244  4740145  7 10500  74
    [2.4.0-RC][root@pfsense.local.lan]/root:
    
    

    Got A+ for quality, A for bufferbloat and A+ overall… This really is easy to implement too.. Be nice when can be fully done in the gui though.



  • @johnpoz:

    Did you look at show when you know there was data flowing?

    So for example… I did the ipfw sched show.  Then I started the dslreport test and looked at it while it was running

    
    [2.4.0-RC][root@pfsense.local.lan]/root: ipfw sched show
    00001:  85.000 Mbit/s    0 ms burst 0 
    q00001  50 sl. 0 flows (256 buckets) sched 1 weight 1 lmax 0 pri 0 droptail
        mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
     sched 1 type FQ_CODEL flags 0x0 0 buckets 0 active
     FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
       Children flowsets: 1 
    00002:  11.000 Mbit/s    0 ms burst 0 
    q00002  50 sl. 0 flows (256 buckets) sched 2 weight 1 lmax 0 pri 0 droptail
        mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000
     sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
     FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
       Children flowsets: 2 
    BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
      0 ip           0.0.0.0/0             0.0.0.0/0        1      262  0    0   0
    [2.4.0-RC][root@pfsense.local.lan]/root: ipfw sched show
    00001:  85.000 Mbit/s    0 ms burst 0 
    q00001  50 sl. 0 flows (256 buckets) sched 1 weight 1 lmax 0 pri 0 droptail
        mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
     sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
     FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
       Children flowsets: 1 
    BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
      0 ip           0.0.0.0/0             0.0.0.0/0     16133 24150846 81 121500  50
    00002:  11.000 Mbit/s    0 ms burst 0 
    q00002  50 sl. 0 flows (256 buckets) sched 2 weight 1 lmax 0 pri 0 droptail
        mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000
     sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
     FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
       Children flowsets: 2 
      0 ip           0.0.0.0/0             0.0.0.0/0     1169    50866  0    0   0
    [2.4.0-RC][root@pfsense.local.lan]/root: ipfw sched show
    00001:  85.000 Mbit/s    0 ms burst 0 
    q00001  50 sl. 0 flows (256 buckets) sched 1 weight 1 lmax 0 pri 0 droptail
        mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
     sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
     FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
       Children flowsets: 1 
    BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
      0 ip           0.0.0.0/0             0.0.0.0/0       77     3563  0    0   0
    00002:  11.000 Mbit/s    0 ms burst 0 
    q00002  50 sl. 0 flows (256 buckets) sched 2 weight 1 lmax 0 pri 0 droptail
        mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000
     sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
     FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
       Children flowsets: 2 
      0 ip           0.0.0.0/0             0.0.0.0/0     3244  4740145  7 10500  74
    [2.4.0-RC][root@pfsense.local.lan]/root:
    
    

    Got A+ for quality, A for bufferbloat and A+ overall… This really is easy to implement too.. Be nice when can be fully done in the gui though.

    Thanks John, I really appreciate your help on this.  After I setup everything and ran a test at DSL Reports I actually did not continue to refresh ipfw sched show manually.  I naively thought that it would update automatically when traffic is passing through the queues.  That being said, is this the case though for the "Limiter Information" section under Diagnostics in the Web UI (i.e. does that section refresh automatically and show traffic passing through the queues)?

    I think I'm going to try this out again.  However, I think my implementation needs to be a little different than was suggested in the original post.  I actually have more than 1 LAN interface on my pfSense box and each interface handles a different subnet.  In order to not limit the amount of bandwidth between the subnets, I don't think I should put the limiter queues on the default allow all rule on the subnets.  Instead, should I just setup a floating firewall rule that matches on WAN traffic?  If that makes sense, what other settings do I need configure (besides choosing the limiter queues at the bottom)?  For instance, what should be the source and destination?

    Thanks again for all your help.



  • Limiter info GUI do not show anything just by design of current fq_соdel implementation in both FreeBSD and pfSense, so it's normal that you don't see traffic there.



  • So I tried setting up fq_codel again and confirm that it is working just fine (just had to refresh the "ipfw sched show" command as traffic was passing through the queues).  I did however, notice a lot of dropped packets while running a speed test.  I have symmetric gigabit connection and also had this issue with the ALTQ traffic shappers.  The solution was to increase the queue size sufficiently.  When using dummynet limiters is the default queue size adjusted under Advanced Options for the queue (that was created under the root limiter) or is it somewhere else?

    Also, I'm still not quite sure how to configure my firewall rules so I don't accidentally limit traffic between local subnets, i.e. I only want to shape traffic internet bound traffic.  What would be the best way to do this (as using the default allow all rule on the LAN will impact subnet traffic too)?

    Thanks again for your help, I really appreciate it.



  • You have to pick your poison, dropped packets or large queue size.

    Dropped packets are no bueno in networking, and so many manufacturers have opted for large queue sizes, which eliminates dropped packets at the expense of (significantly) increased latency, meet bufferbloat.

    fq_codel does an excellent job of eliminating bufferbloat by dropping packets in an intelligent way. For most types of traffic this is preferable to using huge FIFO queues to avoid dropping packets.

    Which is better for you will depend on your network traffic.


  • LAYER 8 Global Moderator

    @tman222

    I run multiple vlans as well..  What is nice that you apply this in your firewall rule.  Just create a rule above the rule that allows your traffic out to the net on whatever interface you want to get to the other segments.  Then on the rule on that interface that allows traffic out to the internet apply in out queues..

    So for example on my lan top rule allows access to rfc1918 space.. So if going to any of my other vlans/segments does not apply..  Then the any any rule below that does apply them, so if going to the internet the allow rule to rfc1918 would be skipped and then the any any rule at the bottom would apply the queues..



  • @belt9:

    You have to pick your poison, dropped packets or large queue size.

    Dropped packets are no bueno in networking, and so many manufacturers have opted for large queue sizes, which eliminates dropped packets at the expense of (significantly) increased latency, meet bufferbloat.

    fq_codel does an excellent job of eliminating bufferbloat by dropping packets in an intelligent way. For most types of traffic this is preferable to using huge FIFO queues to avoid dropping packets.

    Which is better for you will depend on your network traffic.

    Thanks for this info - you are right.  I guess my point was that with very fast (high bandwidth) connections if the queue is too short, packets may drop unnecessarily (irrespective of any AQM such as Codel), which would limit the ability to realize (close to) full bandwidth on the link.  That being said, going too large on the queue size does increase the risk of bufflerbloat, but does this also impact the efficacy of AQM?  In other words if the queue is too large will Codel no longer work effectively?

    @johnpoz:

    @tman222

    I run multiple vlans as well..  What is nice that you apply this in your firewall rule.  Just create a rule above the rule that allows your traffic out to the net on whatever interface you want to get to the other segments.  Then on the rule on that interface that allows traffic out to the internet apply in out queues..

    So for example on my lan top rule allows access to rfc1918 space.. So if going to any of my other vlans/segments does not apply..  Then the any any rule below that does apply them, so if going to the internet the allow rule to rfc1918 would be skipped and then the any any rule at the bottom would apply the queues..

    Thanks John, that makes perfect sense and is probably the best way to ensure that RFC 1918 traffic (or traffic on local subnets) does not get pushed into queues.  With a symmetric gigabit internet connection this is not necessarily a big deal as there are essentially no slow and fast links in the network topology, but in most other cases this configuration is very important so one does not unnecessarily limit bandwidth on local traffic.

    –----------------

    I have now tried both fq_codel on dummynet and FAIRQ with Codel AQM on ALTQ on 2.4.0-RC and the results so far (at least for me) have been similar.  I'm curious if anyone has any ideas for additional testing to better demonstrate the superiority of one traffic shaping solution over the other?

    Thanks again for all your help.



  • I'm not sure one is better than the other.

    It depends on what you're trying to do.

    ALTQ gives you a lot of options and control.

    Dummynet is much less specific, but is very easy to implement. IMO, fq_codel is the only dummynet algorithm worth messing around with for most home users. But there are reasons to use the other algorithms.

    So really it depends on what you are trying to do and why.

    As far as testing your specific network under different types of traffic shaping, I would recommend FLENT.

    https://flent.org/index.html



  • @belt9:

    I'm not sure one is better than the other.

    It depends on what your trying to do.

    ALTQ gives you a lot of options and control.

    Dummynet is much less specific, but is very easy to implement. IMO, fq_codel is the only dummynet algorithm worth messing around with for most home users. But there are reasons to use the other algorithms.

    So really it depends on what you are trying to do and why.

    As far as testing your specific network under different types of traffic shaping, I would recommend FLENT.

    https://flent.org/index.html

    Thanks again for your help and explanation - I will check out that link.



  • @superbree:

    I really wish I could find a way to limit a subnet to say 100Mbs and then limit each ip host address in the subnet to 5 Mbs.  And then have each IP address dynamically shaped if the overall link was approaching the 100Mbs total.

    Take a look at the message in thread: https://forum.pfsense.org/index.php?topic=99503.msg642533#msg642533

    You should be able to tweak the config files from that thread to match your networks and do exactly what you want.



  • fq_Codel is a zero-config AQM. All it needs is to be hooked up to a shaper of some sort and and works magic. You really need to understand how to traffic shape to do better than it. Eve then, it's great.

    I hope they get the kinks worked out of Cake, but they keep trying to add the kitchen sink and has caused some back-and-forth regression.



  • @Harvy66:

    fq_Codel is a zero-config AQM. All it needs is to be hooked up to a shaper of some sort and and works magic. You really need to understand how to traffic shape to do better than it. Eve then, it's great.

    Agreed, it really is very impressive - probably one of the more impressive things I've seen in pfSense.

    It's a huge improvement for very little config, and the config you have to do is not complicated even for a non-tech-savvy home user.

    Netgate should implement some sort of automatic bandwidth limiting, and place that in the UI next to dummynet using fq_codel. Maybe 2.4.2?

    The net result of the above would be that pfSense would dramatically improve the quality of even the crappiest connections from ISP with a  sub 5 minute configuration for even the least experienced user.

    I will grant you that pfSense can already do that (very, very well) with HFSC and limiting your bandwidth manually to below the lowest values you ever see. But HFSC you have to learn how to do, and as Harvy noted - even if you know what you're doing you will have to spend some time getting it as good as fq_codel can be just by turning it on. The result of that is most people either don't use it or don't use it well.

    Also, many WAN connection speeds dip dramatically during peak hours. No one wants to cut their bandwidth down by a large percentage all the time just so their limiter can catch the traffic during peak hours.
    Either an automatic speedtest similar to ubiquiti's, or an automatic latency test similar to gargoyle could be leveraged to automatically keep bandwidth limited just below the current WAN speeds so your limiter is always catching the traffic and you are always making the most of your available bandwidth.

    fq_codel + automatic bandwidth limiter = killer app - huge bullet point for pfSense & Netgate.



  • If anyone has an fq_codel resource they can point me to that demonstrates how to de-priortize traffic to a group of specific subnets, I'd love to see it.

    I'm trying to de-prioritize traffic to Backblaze servers as outlined in this thread.  I don't want to limit it, just make it a lower priority for any other traffic that happens, including starving out Backblaze entirely if there is ANY other traffic. But if there isn't other traffic, let Backblaze consume all the available bandwidth of the internet connection.

    I would have thought not only would this be a simple thing to do, but it would also be fairly common - ha!  I have found precious few examples of how to do ANY traffic shaping to specific subnets - everything I have found so far is all around specific ports or services, which won't work in this instance since all the Backblaze traffic all over SSL.  They do provide a list of all the subnets their servers are on, so I have (one would think!) an easy way to classify their traffic



  • @EricE:

    If anyone has an fq_codel resource they can point me to that demonstrates how to de-priortize traffic to a group of specific subnets, I'd love to see it.

    I'm trying to de-prioritize traffic to Backblaze servers as outlined in this thread.  I don't want to limit it, just make it a lower priority for any other traffic that happens - but if there isn't other traffic, consume all the available bandwidth.

    For this, set your up and down limiters like normal.

    Within each limit set two queues, lets say you call one normal and the other backblaze.

    Set the subnet to match your network (probably /24). Down=destination up=source

    If you wanted to prioritize normal traffic to have 90% bandwidth and backblaze to get 10% when the pipe is full. Then weight normal as 90 and backblaze as 10.

    If the pipe is empty backblaze can still use it all.



  • I'm going to ask what may seem like dumb or trivial questions, just because I have seen so much conflicting information I don't want to leave anything to assumptions.  Thanks in advance.

    @belt9:

    For this, set your up and down limiters like normal.

    So are we talking Limiters in the Firewall/Traffic Shaper/Limiters or Firewall/Traffic Shaper/By Interface?

    @belt9:

    Within each limit set two queues, lets say you call one normal and the other backblaze.

    OK - right now I've got CODELQ queues in Interfaces side and that doesn't support sub-queues, but it was also the only thing that appeared to touch buffer bloat.  Sounds like I need to be in the limiters instead - that might be where I went wrong.

    @belt9:

    Set the subnet to match your network (probably /24). Down=destination up=source

    I'm assuming your talking about a firewall match or pass rule to classify the traffic and assign it to a queue.  If I'm using a floating rule I want the interface to be WAN and the Destination to have the Backblaze networks, right?  I have an Alias with all the subnets for the BackBlaze servers.

    I never did get a floating rule to work, but a Pass rule directly on the LAN interface worked with the Backblaze subnet list alias in the Destination section.  It's just the wizard configs for traffic shaping didn't seem to touch buffer bloat - latency and overall bandwidth was horrible.  But CODELQ only handles buffer bloat wonderfully but I didn't see how to shape the Backblaze traffic.

    It sounds like I really need to play with the limiters instead.  Thanks again for the hints.



  • @EricE:

    I'm going to ask what may seem like dumb or trivial questions, just because I have seen so much conflicting information I don't want to leave anything to assumptions.  Thanks in advance.

    @belt9:

    For this, set your up and down limiters like normal.

    So are we talking Limiters in the Firewall/Traffic Shaper/Limiters or Firewall/Traffic Shaper/By Interface?

    Firewall/Traffic Shaper/Limiters

    @EricE:

    @belt9:

    Within each limit set two queues, lets say you call one normal and the other backblaze.

    OK - right now I've got CODELQ queues in Interfaces side and that doesn't support sub-queues, but it was also the only thing that appeared to touch buffer bloat.  Sounds like I need to be in the limiters instead - that might be where I went wrong.

    CODELQ is under the ALTQ system - which does certainly work, it's just a much more involved setup.

    @EricE:

    @belt9:

    Set the subnet to match your network (probably /24). Down=destination up=source

    I'm assuming your talking about a firewall match or pass rule to classify the traffic and assign it to a queue.  If I'm using a floating rule I want the interface to be WAN and the Destination to have the Backblaze networks, right?  I have an Alias with all the subnets for the BackBlaze servers.

    When you make the queues in dummynet there will be an area to enter your subnet size so that it can share traffic between clients, set that to match which probably means setting it to /24, and the Download limiter is "destination" and Upload Limiter is "source".
    You will also need to apply your queues to firewall rules.


    In order to make sure everything is working and set correctly, I would temporarily set your up and down speeds to something way under your upload speed and set a unique number so that you will easily recognize it on speedtest.
    What I mean by that is, if your normal download/up speeds are 40/10, then on dummynet set download to something like 4200Kbps and set upload to something like 650Kbps.
    The only point of this is so that if you've accidentally reversed the upload and download queues in your firewall rules you will easily recognize that and fix it when you run a speedtest if you see upload at 4.2Mbps and download at 0.65Mbps. If you already know it's all setup correctly then just skip all that stuff.


    @EricE:

    I never did get a floating rule to work, but a Pass rule directly on the LAN interface worked with the Backblaze subnet list alias in the Destination section.  It's just the wizard configs for traffic shaping didn't seem to touch buffer bloat - latency and overall bandwidth was horrible.  But CODELQ only handles buffer bloat wonderfully but I didn't see how to shape the Backblaze traffic.

    It sounds like I really need to play with the limiters instead.  Thanks again for the hints.

    I'm sorry for the convoluted explanation. I'm not near a pfSense box I can access and won't be for awhile. Otherwise I would just give you a screenshot to explain this, it's very easy I'm just trying to explain this from memory of what the config GUI looks like.



  • Patch for Limiter Info page with schedulers information and refresh interval of 500ms

    
    --- diag_limiter_info.php	Wed Sep 07 00:26:47 2016
    +++ diag_limiter_info.php	Sun Oct 01 08:20:33 2017
    @@ -40,5 +40,5 @@
     	echo $text;
    -	$text = `/sbin/ipfw queue show`;
    +	$text = `/sbin/ipfw sched show`;
     	if ($text != "") {
    -		echo "\n\n" . gettext("Queues") . ":\n";
    +		echo "\n\n" . gettext("Shedulers") . ":\n";
     		echo $text;
    @@ -72,3 +76,3 @@
     	events.push(function() {
    -		setInterval('getlimiteractivity()', 2500);
    +		setInterval('getlimiteractivity()', 500);
     		getlimiteractivity();
    
    


  • @belt9:

    @Harvy66:

    fq_Codel is a zero-config AQM. All it needs is to be hooked up to a shaper of some sort and and works magic. You really need to understand how to traffic shape to do better than it. Eve then, it's great.

    Agreed, it really is very impressive - probably one of the more impressive things I've seen in pfSense.

    It's a huge improvement for very little config, and the config you have to do is not complicated even for a non-tech-savvy home user.

    Netgate should implement some sort of automatic bandwidth limiting, and place that in the UI next to dummynet using fq_codel. Maybe 2.4.2?

    The net result of the above would be that pfSense would dramatically improve the quality of even the crappiest connections from ISP with a  sub 5 minute configuration for even the least experienced user.

    I will grant you that pfSense can already do that (very, very well) with HFSC and limiting your bandwidth manually to below the lowest values you ever see. But HFSC you have to learn how to do, and as Harvy noted - even if you know what you're doing you will have to spend some time getting it as good as fq_codel can be just by turning it on. The result of that is most people either don't use it or don't use it well.

    Also, many WAN connection speeds dip dramatically during peak hours. No one wants to cut their bandwidth down by a large percentage all the time just so their limiter can catch the traffic during peak hours.
    Either an automatic speedtest similar to ubiquiti's, or an automatic latency test similar to gargoyle could be leveraged to automatically keep bandwidth limited just below the current WAN speeds so your limiter is always catching the traffic and you are always making the most of your available bandwidth.

    fq_codel + automatic bandwidth limiter = killer app - huge bullet point for pfSense & Netgate.

    Agreed with all you said. They should look into implementing it asap



  • @w0w:

    Patch for Limiter Info page with schedulers information and refresh interval of 500ms

    
    --- diag_limiter_info.php	Wed Sep 07 00:26:47 2016
    +++ diag_limiter_info.php	Sun Oct 01 08:20:33 2017
    @@ -40,5 +40,5 @@
     	echo $text;
    -	$text = `/sbin/ipfw queue show`;
    +	$text = `/sbin/ipfw sched show`;
     	if ($text != "") {
    -		echo "\n\n" . gettext("Queues") . ":\n";
    +		echo "\n\n" . gettext("Shedulers") . ":\n";
     		echo $text;
    @@ -72,3 +76,3 @@
     	events.push(function() {
    -		setInterval('getlimiteractivity()', 2500);
    +		setInterval('getlimiteractivity()', 500);
     		getlimiteractivity();
    
    

    Would love to try this patch out.  This will show fq_codel on the limiter info page?  Is there are kind soul who could explain how to implement this to the lay person?



  • There's a redmine feature request to get an automatic bandwidth limiter added to dummynet.

    If anyone is interested and technically inclined please chime in!

    Check out the links in my signature for more info.

    https://redmine.pfsense.org/issues/7904



  • I finally got fq_codel limiters applied to just my WAN connection via floating rules.

    From what I am seeing I think I like it better than using my vlan's interfaces. From what I am seeing in my own testing the jitter seems lower and I see fewer latency spikes on my upload bandwidth tests. Also since this is queuing all traffic on the WAN interface I feel like it is handling separate flows better than it did before.

    I could be wrong and all of this is anecdotal or a placebo affect from all of my messing around with shappers and limiters.

    If anyone is interested in trying it out the setup is fairly easy.

    Firewall > Rules > Floating

    *Add new rule

    *Change "Action" from "Pass" to "Match"

    *Select "WAN" in Interface

    *Set "Direction" to "Out"

    *Set "Protocol" to "any"

    *Source to "any"

    *Destination to "any"

    Advanced settings

    *Set Gateway (Cannot leave as default; you have to specifically set it to your configured gateway)

    *Set In/Out (Because it is a floating rule and it is set to "Out" it gets a little confusing. It reverses In/Out ie In is for outgoing and Out is for incoming.)



  • dslreports.com has a good bufferbloat test.


Log in to reply