Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login
    Introducing Netgate Nexus: Multi-Instance Management at Your Fingertips.

    This is how i used limiters to fairly share bandwidth among all devices on LAN

    Scheduled Pinned Locked Moved Traffic Shaping
    25 Posts 4 Posters 836 Views 4 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • F Offline
      fsr @TheNarc
      last edited by fsr

      @TheNarc Sorry, i have gathered no evidence, but i remember following this recipe quite some time ago: https://docs.netgate.com/pfsense/en/latest/recipes/codel-limiters.html , and it was not that good for my use case, to be honest, especially because we use OpenVPN for remote access to pfSense, and back in the day it wasn't that clear to me, but that recipe doesn't really do anything about connections initiated from the Internet. The recipe will be useful to keep latency low and prioritize small packets over big packets, however.

      Also, any rules applied on WAN won't know the internal LAN addresses at all. For rules in the IN direction, they will only see Source = Some IP on the Internet, Dest = WAN adresss. For rules in the OUT direction, they will only see Source = WAN adresss, Dest = Some IP on the Internet. So, there is no way fq_codel could use the LAN IP at all on it's queuing.

      So, the rules should be applied on LAN for fq_codel to have any chance to perform any kind of good enough fairness by LAN IP.

      But i understand that fairly sharing the bandwidth among LAN IP is NOT really the purpose of this scheduler. According to: https://www.rfc-editor.org/rfc/rfc8290.html

      ++++++++++++++++++++
      6.1. Fairness between Things Other Than Flows

      In some parts of the network, enforcing flow-level fairness may not
      be desirable, or some other form of fairness may be more important.
      Some examples of this include an ISP that may be more interested in
      ensuring fairness between customers than between flows
      ++++++++++++++++++++

      And as i was already experimenting before with Limiters with Masks, i already had a Limiter implemented with Masked child Queues by the time i decided to switch the scheduler to fq_codel. But maybe the Masked Queues aren't really necessary when using fq_codel, or at least not critical to have, or it's even possible they are being ignored with the fq_codel scheduler (no way to see almost any detail in Diagnostics -> Limiter Info, so there doesn't seems to be any way to tell what it's really doing). I know that if i start two speedtest.net tests on two PCs at the same time to the same server, they get roughly 50% bandwidth each, however.

      T SteveITSS 2 Replies Last reply Reply Quote 0
      • T Offline
        TheNarc @fsr
        last edited by

        @fsr That's a good point about rules applied on the WAN interfaces not knowing about the LAN addresses, which I had not thought of. On the other hand, as you point out, fq_codel provides flow-level fairness, not host (LAN IP level) fairness. So I'm not sure how much that matters? But also because of that, it does seem that your point about achieving that level of fairness - between individual LAN hosts - is valid with respect to enabling masks on the limiter queues.

        I'm definitely intrigued, and will do some testing of my own as well. I feel that every time I think I have something approaching a passable understanding of the traffic shaping nuances, I find I'm not quite there :)

        T 1 Reply Last reply Reply Quote 0
        • T Offline
          TheNarc @TheNarc
          last edited by

          So in my testing, switching my floating rules to apply the limiters on LAN interface traffic instead of WAN interface traffic completely eliminated my bufferbloat mitigation. I want from an A+ on waveform's test to a B or C. But it's interesting if you're seeing good bufferbloat mitigation with rules applied on your LAN interface.

          Now the only other difference is that I was applying the limiters using floating rules with a Match action, because that's how I learned to do it and also because I couldn't have rule processing stop at that point since I have more complex policy routing rules in my preexisting LAN pass rules. But I still can't see an obvious reason why it would be substantively different from what you're doing.

          F 1 Reply Last reply Reply Quote 0
          • SteveITSS Offline
            SteveITS Rebel Alliance @fsr
            last edited by

            @fsr said in This is how i used limiters to fairly share bandwidth among all devices on LAN:

            rules applied on WAN won't know the internal LAN addresses at all

            If I followed, this can be done for floating rules with tagging:
            https://docs.netgate.com/pfsense/en/latest/firewall/floating-rules.html#marking-and-matching
            so, tag=VOIP, match on tag=VOIP

            But if what you have is working fine, don't break it. ;)

            To upgrade, select your branch in System/Update/Update Settings. When upgrading, allow 10-15 minutes to reboot, or more depending on packages, CPU, and/or disk speed.
            Only install packages for your version of pfSense.
            Upvote 👍 helpful posts!

            F 1 Reply Last reply Reply Quote 1
            • F Offline
              fsr @SteveITS
              last edited by fsr

              Last night i made some tests on traffic of the two WANs i have. WAN20 is a 20 Mbps dedicated connection, while WAN600 is a 600 Mbps symmetric, but not dedicated.

              WAN20 has the limiters i described before (on LAN), while WAN600 has the limiters applied like the following recipe https://docs.netgate.com/pfsense/en/latest/recipes/codel-limiters.html

              The test was two PCs performing a speedtest.net simultaneously against the same server.

              WAN20 results

              PC1
              Speedtest 20 Mbps PC1.png

              PC2
              Speedtest 20 Mbps PC2.png

              0.84% difference in download between PCs, and 4.79% difference in upload

              Limiter info showed this (for the curious that at the same time can decipher that wall of numbers):

              NOTE: The limiters have different speeds to be able to recognize them:
              19005 Kbps for the Download Limiter on WAN20
              19006 Kbps for the Upload Limiter on WAN20
              570 Mbps for the Download Limiter on WAN600
              571 Mbps for the Upload Limiter on WAN600

              TWO SIMULTANEOUS SPEEDTESTS (DOWNLOAD PHASE)

Limiter Information

Limiters:
00003:  19.006 Mbit/s    0 ms burst 0 
q131075  50 sl. 0 flows (1 buckets) sched 65539 weight 0 lmax 0 pri 0 droptail
 sched 65539 type FIFO flags 0x0 0 buckets 0 active
00004:  19.005 Mbit/s    0 ms burst 0 
q131076  50 sl. 0 flows (1 buckets) sched 65540 weight 0 lmax 0 pri 0 droptail
 sched 65540 type FIFO flags 0x0 0 buckets 0 active
00005: 570.000 Mbit/s    0 ms burst 0 
q131077  50 sl. 0 flows (1 buckets) sched 65541 weight 0 lmax 0 pri 0 droptail
 sched 65541 type FIFO flags 0x0 0 buckets 0 active
00006: 571.000 Mbit/s    0 ms burst 0 
q131078  50 sl. 0 flows (1 buckets) sched 65542 weight 0 lmax 0 pri 0 droptail
 sched 65542 type FIFO flags 0x0 0 buckets 0 active


Schedulers:
00003:  19.006 Mbit/s    0 ms burst 0 
q00003  50 sl. 0 flows (256 buckets) sched 4 weight 0 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
 sched 3 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 4 
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0       40     2495  0    0   0
00004:  19.005 Mbit/s    0 ms burst 0 
q65540  50 sl. 0 flows (1 buckets) sched 4 weight 0 lmax 0 pri 0 droptail
 sched 4 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 3 
  0 ip           0.0.0.0/0             0.0.0.0/0     11472 13588082 33 48083 205
00005: 570.000 Mbit/s    0 ms burst 0 
q65541  50 sl. 0 flows (1 buckets) sched 5 weight 0 lmax 0 pri 0 droptail
 sched 5 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1 
  0 ip           0.0.0.0/0             0.0.0.0/0        2      184  0    0   0
00006: 571.000 Mbit/s    0 ms burst 0 
q65542  50 sl. 0 flows (1 buckets) sched 6 weight 0 lmax 0 pri 0 droptail
 sched 6 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2 


Queues:
q00001  50 sl. 0 flows (1 buckets) sched 5 weight 0 lmax 0 pri 0 droptail
q00002  50 sl. 0 flows (1 buckets) sched 6 weight 0 lmax 0 pri 0 droptail
q00003  50 sl. 0 flows (256 buckets) sched 4 weight 0 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
q00004  50 sl. 0 flows (256 buckets) sched 3 weight 0 lmax 0 pri 0 droptail
    mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000



--------------------------------


TWO SIMULTANEOUS SPEEDTESTS (UPLOAD PHASE)

Limiter Information

Limiters:
00003:  19.006 Mbit/s    0 ms burst 0 
q131075  50 sl. 0 flows (1 buckets) sched 65539 weight 0 lmax 0 pri 0 droptail
 sched 65539 type FIFO flags 0x0 0 buckets 0 active
00004:  19.005 Mbit/s    0 ms burst 0 
q131076  50 sl. 0 flows (1 buckets) sched 65540 weight 0 lmax 0 pri 0 droptail
 sched 65540 type FIFO flags 0x0 0 buckets 0 active
00005: 570.000 Mbit/s    0 ms burst 0 
q131077  50 sl. 0 flows (1 buckets) sched 65541 weight 0 lmax 0 pri 0 droptail
 sched 65541 type FIFO flags 0x0 0 buckets 0 active
00006: 571.000 Mbit/s    0 ms burst 0 
q131078  50 sl. 0 flows (1 buckets) sched 65542 weight 0 lmax 0 pri 0 droptail
 sched 65542 type FIFO flags 0x0 0 buckets 0 active


Schedulers:
00003:  19.006 Mbit/s    0 ms burst 0 
q00003  50 sl. 0 flows (256 buckets) sched 4 weight 0 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
 sched 3 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 4 
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0     14732 18775095 37 55500 1255
00004:  19.005 Mbit/s    0 ms burst 0 
q65540  50 sl. 0 flows (1 buckets) sched 4 weight 0 lmax 0 pri 0 droptail
 sched 4 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 3 
  0 ip           0.0.0.0/0             0.0.0.0/0      745    49087  0    0   0
00005: 570.000 Mbit/s    0 ms burst 0 
q65541  50 sl. 0 flows (1 buckets) sched 5 weight 0 lmax 0 pri 0 droptail
 sched 5 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1 
  0 ip           0.0.0.0/0             0.0.0.0/0       43     4197  0    0   0
00006: 571.000 Mbit/s    0 ms burst 0 
q65542  50 sl. 0 flows (1 buckets) sched 6 weight 0 lmax 0 pri 0 droptail
 sched 6 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2 
  0 ip           0.0.0.0/0             0.0.0.0/0       48    13792  0    0   0


Queues:
q00001  50 sl. 0 flows (1 buckets) sched 5 weight 0 lmax 0 pri 0 droptail
q00002  50 sl. 0 flows (1 buckets) sched 6 weight 0 lmax 0 pri 0 droptail
q00003  50 sl. 0 flows (256 buckets) sched 4 weight 0 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
q00004  50 sl. 0 flows (256 buckets) sched 3 weight 0 lmax 0 pri 0 droptail
    mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000

              WAN600 results

              PC3
              Speedtest 600 Mbps PC3.png

              PC4
              Speedtest 600 Mbps PC4.png

              57% difference in download between PCs, and 26% difference in upload

              It seems to me, like adding the Masked Queues to the LAN in the WAN20 test introduced "Host Fairness", as the percentages are quite a lot lower in that test.

              I also have the Limiter info for the WAN600 test:

              NOTE: The limiters have different speeds to be able to recognize them:
              19005 Kbps for the Download Limiter on WAN20
              19006 Kbps for the Upload Limiter on WAN20
              570 Mbps for the Download Limiter on WAN600
              571 Mbps for the Upload Limiter on WAN600

              SPEEDTEST FQ_CODEL EN WAN. CONEXION 600 Mbps


TWO SIMULTANEOUS SPEEDTESTS (DOWNLOAD PHASE)

Limiters:
00003:  19.006 Mbit/s    0 ms burst 0 
q131075  50 sl. 0 flows (1 buckets) sched 65539 weight 0 lmax 0 pri 0 droptail
 sched 65539 type FIFO flags 0x0 0 buckets 0 active
00004:  19.005 Mbit/s    0 ms burst 0 
q131076  50 sl. 0 flows (1 buckets) sched 65540 weight 0 lmax 0 pri 0 droptail
 sched 65540 type FIFO flags 0x0 0 buckets 0 active
00005: 570.000 Mbit/s    0 ms burst 0 
q131077  50 sl. 0 flows (1 buckets) sched 65541 weight 0 lmax 0 pri 0 droptail
 sched 65541 type FIFO flags 0x0 0 buckets 0 active
00006: 571.000 Mbit/s    0 ms burst 0 
q131078  50 sl. 0 flows (1 buckets) sched 65542 weight 0 lmax 0 pri 0 droptail
 sched 65542 type FIFO flags 0x0 0 buckets 0 active


Schedulers:
00003:  19.006 Mbit/s    0 ms burst 0 
q00003  50 sl. 0 flows (256 buckets) sched 4 weight 0 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
 sched 3 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 4 
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0       33    10054  0    0   0
00004:  19.005 Mbit/s    0 ms burst 0 
q65540  50 sl. 0 flows (1 buckets) sched 4 weight 0 lmax 0 pri 0 droptail
 sched 4 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 3 
  0 ip           0.0.0.0/0             0.0.0.0/0       26     1996  0    0   0
00005: 570.000 Mbit/s    0 ms burst 0 
q65541  50 sl. 0 flows (1 buckets) sched 5 weight 0 lmax 0 pri 0 droptail
 sched 5 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1 
  0 ip           0.0.0.0/0             0.0.0.0/0     52664 78273224 295 438731  31
00006: 571.000 Mbit/s    0 ms burst 0 
q65542  50 sl. 0 flows (1 buckets) sched 6 weight 0 lmax 0 pri 0 droptail
 sched 6 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2 
  0 ip           0.0.0.0/0             0.0.0.0/0     10123   499009  0    0   0


Queues:
q00001  50 sl. 0 flows (1 buckets) sched 5 weight 0 lmax 0 pri 0 droptail
q00002  50 sl. 0 flows (1 buckets) sched 6 weight 0 lmax 0 pri 0 droptail
q00003  50 sl. 0 flows (256 buckets) sched 4 weight 0 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
q00004  50 sl. 0 flows (256 buckets) sched 3 weight 0 lmax 0 pri 0 droptail
    mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000


---------------------------------------------------------------------------------------------


TWO SIMULTANEOUS SPEEDTESTS (UPLOAD PHASE)


Limiters:
00003:  19.006 Mbit/s    0 ms burst 0 
q131075  50 sl. 0 flows (1 buckets) sched 65539 weight 0 lmax 0 pri 0 droptail
 sched 65539 type FIFO flags 0x0 0 buckets 0 active
00004:  19.005 Mbit/s    0 ms burst 0 
q131076  50 sl. 0 flows (1 buckets) sched 65540 weight 0 lmax 0 pri 0 droptail
 sched 65540 type FIFO flags 0x0 0 buckets 0 active
00005: 570.000 Mbit/s    0 ms burst 0 
q131077  50 sl. 0 flows (1 buckets) sched 65541 weight 0 lmax 0 pri 0 droptail
 sched 65541 type FIFO flags 0x0 0 buckets 0 active
00006: 571.000 Mbit/s    0 ms burst 0 
q131078  50 sl. 0 flows (1 buckets) sched 65542 weight 0 lmax 0 pri 0 droptail
 sched 65542 type FIFO flags 0x0 0 buckets 0 active


Schedulers:
00003:  19.006 Mbit/s    0 ms burst 0 
q00003  50 sl. 0 flows (256 buckets) sched 4 weight 0 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
 sched 3 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 4 
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0       39    18229  0    0   0
00004:  19.005 Mbit/s    0 ms burst 0 
q65540  50 sl. 0 flows (1 buckets) sched 4 weight 0 lmax 0 pri 0 droptail
 sched 4 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 3 
  0 ip           0.0.0.0/0             0.0.0.0/0       25     2168  0    0   0
00005: 570.000 Mbit/s    0 ms burst 0 
q65541  50 sl. 0 flows (1 buckets) sched 5 weight 0 lmax 0 pri 0 droptail
 sched 5 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1 
  0 ip           0.0.0.0/0             0.0.0.0/0     6173   261323  0    0   0
00006: 571.000 Mbit/s    0 ms burst 0 
q65542  50 sl. 0 flows (1 buckets) sched 6 weight 0 lmax 0 pri 0 droptail
 sched 6 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2 
  0 ip           0.0.0.0/0             0.0.0.0/0     17678 26134501  0    0   0


Queues:
q00001  50 sl. 0 flows (1 buckets) sched 5 weight 0 lmax 0 pri 0 droptail
q00002  50 sl. 0 flows (1 buckets) sched 6 weight 0 lmax 0 pri 0 droptail
q00003  50 sl. 0 flows (256 buckets) sched 4 weight 0 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
q00004  50 sl. 0 flows (256 buckets) sched 3 weight 0 lmax 0 pri 0 droptail
    mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000

              I will try to test moving the limiter on WAN600 to LAN if i have the chance, and if possible, because things certainly get complicated with many rules, adapters, policy routing, etc.

              1 Reply Last reply Reply Quote 0
              • F Offline
                fsr
                last edited by fsr

                Those are some bufferbloat tests from https://www.waveform.com/tools/bufferbloat

                Waveform test results for WAN20: https://www.waveform.com/tools/bufferbloat?test-id=317836bf-f9b1-4e10-a677-6a378ec596e2

                ca66a9d0-bf89-43db-bcf4-e1657bb8e344-image.png

                ++++++++++++++++++++++++++++++

                Waveform test results for WAN600: https://www.waveform.com/tools/bufferbloat?test-id=185b5ddb-99fd-4380-bf62-2c918c79233f

                ff9c0f3f-48d9-415f-815e-d3e408b0d15b-image.png

                1 Reply Last reply Reply Quote 0
                • F Offline
                  fsr @TheNarc
                  last edited by fsr

                  @TheNarc said in This is how i used limiters to fairly share bandwidth among all devices on LAN:

                  So in my testing, switching my floating rules to apply the limiters on LAN interface traffic instead of WAN interface traffic completely eliminated my bufferbloat mitigation. I want from an A+ on waveform's test to a B or C. But it's interesting if you're seeing good bufferbloat mitigation with rules applied on your LAN interface.

                  Now the only other difference is that I was applying the limiters using floating rules with a Match action, because that's how I learned to do it and also because I couldn't have rule processing stop at that point since I have more complex policy routing rules in my preexisting LAN pass rules. But I still can't see an obvious reason why it would be substantively different from what you're doing.

                  That's curious. It should work the same if all of the packets are really going thru the limiter. In fact, i have two WAN adapters, and one of them have the limiters applied to LAN, and the other has the limiters applied on WAN (the netgate recipe liked above).

                  In fact, i have several LAN adapters, but every one of them has a rule similar to "Action: Pass, Source: LAN Subnet, Destination: (invert match) LAN Subnet, In Pipe: UploadQueue, Out Pipe: DownloadQueue"

                  There should be no problem to use Floating Rules with a Match Action, but they can get tricky, and the rule must match traffic going to the Internet only, and if you add queues with masks on source and destination addresses, it's critical that the rule matches traffic with LAN addresses as source or destination, according to the case.

                  If you have only outgoing traffic (no open ports for connections originating from the internet), one LAN adapter and one WAN adapter, then a floating rule like "Action: MATCH, Adapter: LAN, Source: LAN Subnet, Destination: (invert match) LAN Subnet, In Pipe: UploadQueue, Out Pipe: DownloadQueue" should match all traffic going to the internet, and frequently other stuff like multicast. I don't use multicast, so it can be ignored in my case. Or you could create a "NotTheInternet" alias, and use it as destination (with "invert match", of course, and the alias could include networks like all the private network space, and other stuff that won't go to the internet). You can fine-tune by looking at Status -> System Logs -> Firewall, if you enabled logging in you rules (i always do).

                  If you used masks and you want to check that the queues are being applied to the correct addresses, you can temporarily select the default scheduler ("Worst-case Weighted fair Queueing"), then you can see in Diagnostics -> Limiter Info the exact IP addresses of all the queues that are active.

                  Things can get complicated, so better to have a saved configuration before starting modifications.

                  In my case, this is used in an office, so there are many different devices trying to get/send some data simultaneously, several different protocols simultaneously, and it's important that if several people are downloading (or other kind of heavy use), every one of them get a fair share of the internet connection. But for other kind of uses, maybe there is no reason to complicate things with masks, and fq_codel by it's own results to be fair enough.

                  @SteveITS said in This is how i used limiters to fairly share bandwidth among all devices on LAN:

                  @fsr said in This is how i used limiters to fairly share bandwidth among all devices on LAN:

                  rules applied on WAN won't know the internal LAN addresses at all

                  If I followed, this can be done for floating rules with tagging:
                  https://docs.netgate.com/pfsense/en/latest/firewall/floating-rules.html#marking-and-matching
                  so, tag=VOIP, match on tag=VOIP

                  But if what you have is working fine, don't break it. ;)

                  Thanks, i use tags for connections originating from the internet. I use a tag to mark which packets come from a specific WAN adapter, so that then i can match the packets on LAN that have that tag, so i can apply the correct limiters to that traffic.

                  I was just pointing to the fact that a rule on WAN doesn't knows anything about the LAN address the packet came from (NAT has already taken place), so fq_codel couldn't possibly apply any kind of host fairness, even if it was designed for that, which it doesn't.

                  T 1 Reply Last reply Reply Quote 0
                  • T Offline
                    TheNarc @fsr
                    last edited by

                    @fsr Thanks for the detailed write-up. I decided to take another shot at it and I seem to have things working well with floating match rules applied to the LAN interfaces now, using limiter sub-queues with masks. I'm a bit confused because I honestly don't know how this configuration is different than my first attempt, but the bufferbloat tests look good.

                    And also to your point, the host-based fairness is probably a bit overkill for my small home network anyway, but conceptually it does seem to make sense. And I figure if it works, why not? So unless someone more knowledgeable stops by and has some compelling reason fir it being a bad idea, I'm going to leave it.

                    Also a neat trick temporarily changing the scheduler algorithm to make the queuing behavior more visible in Diagnostics > Limiter.

                    One other note that may be of interest: I also have an explicit delay of 1ms set on my limiters per the workaround suggested in this bug: https://redmine.pfsense.org/issues/11192 and I believe that it did make a difference. Though obviously there is normal test-to-test variation as well so it can be difficult to be certain except when changes make a very dramatic and obvious impact.

                    F S 2 Replies Last reply Reply Quote 0
                    • F Offline
                      fsr @TheNarc
                      last edited by fsr

                      @TheNarc You're welcome.
                      It's great that it works now. Maybe some parameter was reset to defaults when switching from fq_codel to "Worst-case Weighted fair Queueing" (WF2Q+) and back to fq_codel?
                      I will add some screen captures of my current limiter and queue configuration here. One picture is worth a thousand words.

                      I didn't know about that bug, but it doesn't seems to be happening in my case, because the Gateway Quality delay average graph is between 3 and 10 ms, even when the 20 Mbps link traffic graph is maxed out for like 30 minutes, while several people are using meet, zoom, and the like. And no user complains (that's the most sensitive indicator there is 😂 ). In my case pfSense is running as a VM on Hyper-V in Windows Server, and the pfSense is showing hn network adapters.

                      These are all the limiters i have configured right now:

                      099be3d8-d936-46e2-9574-02e7c1e31aeb-image.png

                      • WanDLCodel and WanULCodel and it's Queues are configured like in the netgate recipe linked above, so i will not show them. Their Queues are applied to the 600 Mbps link on the WAN side, as indicated in the recipe.
                      • ShareLimitByDestIP and ShareLimitbySourceIP and it's Queues are configured like in the first post i made in this topic. I will show a capture of it's configurations next. Their Queues are applied to the 20 Mbps link on the LAN side. I will post screen captures of them.

                      If the names are confusing, "ShareQueueByDestIP" applies the mask by Destination Addresses. As the rule is applied on LAN that means it's the Download limiter queue (it's probably better to name it that way).

                      Likewise, "ShareQueueBySourceIP" applies the mask by Source Addresses. As the rule is applied on LAN that means it's the Upload limiter queue (it's probably better to name it that way).

                      8d15f236-2cf9-4da1-a25d-71042662e09a-image.png

                      ======================================

                      4fe93ee1-0f32-432e-8db5-0310349ed36d-image.png

                      ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

                      b3de2164-7763-4335-9f51-bd689ae5cb3b-image.png

                      ======================================

                      a5ca71af-6999-46c7-9171-7c7a2b46ebec-image.png

                      1 Reply Last reply Reply Quote 0
                      • F Offline
                        fsr
                        last edited by

                        NOTE: just tried to apply the same kind of limiter on LAN for my second internet connection (thus, using a non-default gateway), and i found that there is a bug in pfSense CE 2.8 that prevents doing this. It will be fixed in 2.9. For rules using the default gateway there is no problem. It only affects non-default gateways:

                        https://forum.netgate.com/topic/197993/limiter-source-mask-now-after-nat-when-using-gateway-groups-2-8-change/15

                        https://redmine.pfsense.org/issues/15770

                        https://github.com/pfsense/FreeBSD-src/commit/7ec06143964a949ebf6885ac120fdf839ad29eab

                        So, my second internet connection will continue to use the netgate recipe on WAN at least until pfSense 2.9 is out.

                        T 1 Reply Last reply Reply Quote 0
                        • S Offline
                          strannik @TheNarc
                          last edited by

                          Hello everyone.
                          I'm trying to do the same thing: per-LAN-IP sharing of Internet bandwidth. I built the same Download and Upload CoDel limiters with nested queues with 32-bit masks.CoDel Limiters.png

                          I'm not using any limiters on WAN interface - only on LAN and my buffebloat score improved from B-C to A+
                          But there is no equal bandwidth sharing per-LAN-IP : if two PCs are downloading it is 30-60% not 50-50. I tried removing masking from my queues - same 30-60 problem.
                          I expected FQ_CoDel scheduler to equly devide the available bandwidth.
                          Anybody has any suggestions ?

                          T 1 Reply Last reply Reply Quote 0
                          • SteveITSS SteveITS referenced this topic
                          • T Offline
                            TheNarc @fsr
                            last edited by

                            @fsr It wasn't clear to me from the redmine bug if this may work, but have you tried using a floating "Match" rule on your LAN interface to apply the limiter, and a separate "Pass" LAN Interface rule to policy route to the non-default gateway? May make no difference, but just a thought.

                            F 1 Reply Last reply Reply Quote 0
                            • T Offline
                              TheNarc @strannik
                              last edited by

                              @strannik Not sure . . . you may want to post screen shots of your limiter and queue configurations. Have you double checked that you have the proper source versus destination masking for your upload versus download queues? For applying the limiter on the LAN, it should be masking on source for upload and destination for download.

                              S 1 Reply Last reply Reply Quote 0
                              • S Offline
                                strannik @TheNarc
                                last edited by strannik

                                @TheNarc - Thanks for reply. Here are my screenshots.
                                Download Parent Limiter part-1.png
                                Download Limiter (part-2)
                                Download Parent Limiter part-2.png
                                and finally - the nested Queue with 32-bit masking:
                                Download Child Queue.png

                                I also tried using just Pipes with per-IP masking without nesting Queues - CoDel does not work at all.
                                I was able to get rid of bad bufferbloat (150-300 ms) with this nested configuration but still it is not "fairly" sharing the assigned bandwidth.

                                *** My total WAN bandwidth is approx. 850 Mb/s down and 650-700 Mb/s up. I'm trying to create separate limers per-department so each department could use no more that 100 Mb/s total - depending on how many users are downloading at once: if 1 user needs bandwidth - he has 100%. If 3 users need bandwidth - each gets approx. 100/3=33% of that 100Mb/s department limit.

                                T 1 Reply Last reply Reply Quote 0
                                • T Offline
                                  TheNarc @strannik
                                  last edited by

                                  @strannik Thanks for the update. I definitely don't see anything wrong with how you have the limiters and queues configured, and Tail Drop is definitely what you want for the queue management algorithm on both the limiter and the child queue.

                                  I'll see whether I can do some more of my own testing to determine whether I seem to get per-host fairness, as I'm configured very much the same. Although I'm also only running it on a small home network. The curious thing is that even without the LAN-side application of the limiters and the masks on the child queues, I would expect things to be generally fair. That is to say, if you only apply limiters on the WAN interface, it should enforce per-flow fairness, based on my understanding. So if host A and host B both run a speed test that makes a connection to a single remote host, they would each be initiating one flow, and FQ_CODEL would split the available bandwidth between them fairly (assuming no other network activity). I think the host-based fairness would come into play more if one host is establishing far more flows than another; say, host A is torrenting with connections to who knows how many remote hosts and host B is just trying to download something from a single remote host.

                                  I realize none of that really helps to answer why you're seeing the behavior that you are, more just to express my confusion, and give others reading this thread a chance to point out anything I may have egregiously wrong there.

                                  If I think of any other suggestions, I'll update, but unfortunately I'm not sure what else to suggest at the moment. I trust that both the hosts you're using for your testing are wired, since wireless would throw all kinds of wrenches into attempting to assess bandwidth sharing fairness. I suppose if you have a 3rd host it may be interesting to add it to the mix as well. For example, is it always one host that receives an unfair share, or does the unequal distribution seem to move from host to host seemingly at random?

                                  S 1 Reply Last reply Reply Quote 0
                                  • S Offline
                                    strannik @TheNarc
                                    last edited by

                                    @TheNarc Thank you for reply. I keep testing and I just discovered that Upload FQ_CoDel is dividing almost perfectly 50-50 unlike download.
                                    -- Also I found that having CoDel setup on WAN interface has almost NO effect on LAN per-IP fairness. I disabled my floating limiter rule on WAN and nothing changed. My bufferbloat score is still great (under 10 ms in all modes)
                                    -- Also I discovered that changing the scheduler from Default to FQ_CoDel and back does NOT work. I had to create a brand new limiter from scratch for CoDel to kick in even if I reboot the pfSense. That is definitly a BUG in pfSense 2.7.2
                                    Now I'm testing v2.8.1 in VBox before I upgrade my production firewalls. Downtime is very expensive :)

                                    T 1 Reply Last reply Reply Quote 0
                                    • T Offline
                                      TheNarc @strannik
                                      last edited by

                                      @strannik That's very interesting. I wonder if that implies that even though the limiters are being applied using a rule on the LAN interface, it's still pre-NAT? Because, for download, if the destination always just looks like the WAN IP, then masking on destination is going to have no effect because everything will still go in one queue. And if it's pre-NAT in both directions, then from the upload perspective, you're going to have the LAN hosts as sources, so masking on source will get you one queue per host and therefore the fairness that you're seeing. What about keeping the upload rule on LAN masking by source, and moving the download rule to WAN, also masking on source? Does that make any sense, or am I full of it?

                                      The issue you found switching between schedulers sounds like a pain. I've been running 2.8.1 since it was released since my stakes are much lower with a home network, but I haven't tried going back and forth on the scheduler settings. I can try that, but I'm not sure how to definitively tell whether it's working or not. Did you just find that after setting it to something other than FQ_CODEL and then back that you still had bad bufferbloat?

                                      S 1 Reply Last reply Reply Quote 0
                                      • S Offline
                                        strannik @TheNarc
                                        last edited by strannik

                                        @TheNarc After changing the scheduler and applying it I always go to Diag --> Limiter Info to verify that "scheduler" has changed. Here is where I discovered that it did not - all my pipes and queues kept their old configuration. I rebooted the whole pfSense - same thing. Then I created a new limiters from scratch and it did work.

                                        To bad I can not experiment on production firewalls so I created a Lab setup in Virtual Box to be able to reboot more often. I'm testing every possible way but still I have not found the exact step-by-step procedure that works. Perhaps this FQ_CoDel needs some more development :)

                                        F 1 Reply Last reply Reply Quote 0
                                        • F Offline
                                          fsr @TheNarc
                                          last edited by fsr

                                          Thanks for the idea, but i can't do that, as i'm applying the limiters on the rules of the LAN interface (not in floating rules), because i have multi-WAN with failover, and the only way i found to apply different limiters to different internet links on the LAN side, is to apply the limiters on the same rule that chooses the gateway, and enable [System] [Advanced] [Miscellaneous] [Skip rules when gateway is down]. So, the failover works by using two separate rules for each link on the LAN interface: a rule allowing traffic to go to the internet with the fast gateway and the limiters for the fast link, followed by another rule allowing traffic to go to the internet with the default (slow) gateway and the limiters for the slow link.
                                          So, if the fast gateway is up, the first rule applies. If the gateway is down, the first rule is skipped, and the second rule applies. Each rule applies the correct limiters for each link speed.

                                          I will have to wait for the next version to come out. Meanwhile, applying the limiter on WAN is good enough for the 600 Mbps link, while the slow 20 Mbps link has it's limiter on LAN.

                                          The bug is that the limiters/queues will get the pre-NAT address on rules that set a non-default gateway. So if you use masking, instead of seeing a queue for each LAN address, you will see there is a queue for your WAN address (if you are using a scheduler that shows this, for example the default scheduler "Worst-case Weighted fair Queueing").

                                          T 1 Reply Last reply Reply Quote 0
                                          • F Offline
                                            fsr @strannik
                                            last edited by fsr

                                            @strannik I have readed the messages, and i agree with TheNarc. Your limiter/queue configuration looks ok. Is your rule to assign traffic to queues like the one i wrote on the first message?
                                            I don't see that strange behaviour about changing schedulers with 2.8.1. You should upgrade your test environment to the latest version to avoid those problems.
                                            Using the default "Worst-case Weighted fair Queueing" scheduler is a good way to check that the masking is working ok, and the download/upload behaviour, and fairness (i readed that the default scheduler is designed for "host fairness", but it will do nothing to keep latency low, and that's why schedulers like fq_codel exist which will keep latency low, but won't provide "host fairness", only "flow fainess").
                                            In my tests with speedtest.net, i have seen fair sharing by IP, when running two simultaneous tests against the same server, with low or no traffic from other hosts. That is expected to result in the same amount of connections from each IP. And it showed a clear difference between using masks, and not using them. But in real life use, there could be many reasons why two simultaneous downloads or uploads could get different speeds. The remote server could be the limiting factor, or one of your PCs is opening more connections than the other.
                                            Also, if i understand this correcty, adding queues with masks to the fq_codel limiter will pass packets in a round-robin fashion from each of your active hosts to the fq_codel scheduler, but that only guarantees that no single host will be able to saturate the scheduler by opening a lot of connections (flows) at the same time. The queues with masks will add some "host fairness" to a scheduler that only cares about "flow fairnes", but it's likely that the result won't have a perfect "host fairness".
                                            To get an idea of how little fq_codel cares about "host fairness", consider this: the first thing fq_codel does to assign flows to it's queues is to hash together 5 properties of each packet it receives: source and destination IP addresses, source and destination port numbers, and IP protocol number. Each flow is identified like this. Once this hashing is performed, the scheduler won't know about any of the original properties, only that each flow is a different flow. 50 different flows could be one flow from 50 of your hosts, or 50 flows from one host. No way for it to tell the difference. Host with more flows will get more of the bandwidth. But adding the queues with masks, they will pass packets in a round-robin fashion from each of your active hosts to the fq_codel scheduler, making sure that each active host can send one packet to the scheduler in turn before the hashing begins. That will add some host fairness to it, but when fq_codel has to decide which packets to drop and which ones to prioritize, it will only act according to the information it has, to do what it does: keep latency low, which probably won't result in 100% perfect host fairness. But in most cases it's more important to be able to keep using the link for real-time communications while heavy downloads are performed, than for each of your hosts to get perfect fairness about their downloads.
                                            If you only care about host fairness and not latency, then the default scheduler would probably be a better choice, but the latency will likely suffer a lot.

                                            If you want to read more about fq_codel, this is a very interesting source: https://datatracker.ietf.org/doc/html/draft-ietf-aqm-fq-codel

                                            S 1 Reply Last reply Reply Quote 1
                                            • First post
                                              Last post
                                            Copyright 2026 Rubicon Communications LLC (Netgate). All rights reserved.