Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    [2.8.1.b] Multiple limiter issue

    Scheduled Pinned Locked Moved Development
    13 Posts 3 Posters 868 Views 4 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • stephenw10S Offline
      stephenw10 Netgate Administrator
      last edited by

      Yup we are aware of it.

      Can we see the rules you are using to apply those Limiters?

      Do you see states opened matching those rules?

      P N 2 Replies Last reply Reply Quote 1
      • P Offline
        pst @stephenw10
        last edited by

        @stephenw10 I have updated the post with the details you asked for (I hope...). Yes, there are states opened matching the rules.

        As far as others' reports of limiters not working, they seems to suggests the rules that were working in 2.7.2 no longer works in 2.8.0, so that should be an easy test case to run. I've never run 2.7.2 but remember having limiters enabled at some point in 24.0x (not sure if I still have them in my defunct 24.11 setup)

        1 Reply Last reply Reply Quote 0
        • N Offline
          NWOSwamp @stephenw10
          last edited by

          @stephenw10
          I have a nearly step-by-step, from a fresh install, how I duplicated the issue in my original post. I wiped out my lab, I recreated from that post (hopefully accurately). The testing & results should be pretty close. LMK if you're looking for something different.

          2.7.2
          Rules:

          # User-defined rules follow

anchor "userrules/*"
pass  out  quick  on {  vtnet0  }  $GWWAN_DHCP inet from <WAN IP> to any ridentifier 1752945005 keep state  dnqueue( 1,2) label "USER_RULE: Bufferbloat" label "id:1752945005" label "gw:WAN_DHCP"
pass  in  quick  on $LAN inet from $LAN__NETWORK to any ridentifier 0100000101 keep state  dnpipe ( 3,4) label "USER_RULE: Default allow LAN to any rule" label "id:0100000101"

#

          Limiter Info:

          Limiters:
00001:  20.000 Mbit/s    0 ms burst 0 
q131073  50 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0  AQM CoDel target 1us interval 1us ECN
 sched 65537 type FIFO flags 0x0 0 buckets 0 active
00002: 100.000 Mbit/s    0 ms burst 0 
q131074  50 sl. 0 flows (1 buckets) sched 65538 weight 0 lmax 0 pri 0  AQM CoDel target 1us interval 1us ECN
 sched 65538 type FIFO flags 0x0 0 buckets 0 active
00003:   2.000 Mbit/s    0 ms burst 0 
q131075  50 sl. 0 flows (1 buckets) sched 65539 weight 0 lmax 0 pri 0 droptail
 sched 65539 type FIFO flags 0x0 0 buckets 0 active
00004:   5.000 Mbit/s    0 ms burst 0 
q131076  50 sl. 0 flows (1 buckets) sched 65540 weight 0 lmax 0 pri 0 droptail
 sched 65540 type FIFO flags 0x0 0 buckets 0 active


Schedulers:
00001:  20.000 Mbit/s    0 ms burst 0 
q65537  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
 sched 1 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 1us interval 1us quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1 
00002: 100.000 Mbit/s    0 ms burst 0 
q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
 sched 2 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 1us interval 1us quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2 
00003:   2.000 Mbit/s    0 ms burst 0 
q65539  50 sl. 0 flows (1 buckets) sched 3 weight 0 lmax 0 pri 0 droptail
 sched 3 type FIFO flags 0x0 0 buckets 0 active
00004:   5.000 Mbit/s    0 ms burst 0 
q65540  50 sl. 0 flows (1 buckets) sched 4 weight 0 lmax 0 pri 0 droptail
 sched 4 type FIFO flags 0x0 0 buckets 0 active


Queues:
q00001  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0  AQM CoDel target 1us interval 1us ECN
q00002  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0  AQM CoDel target 1us interval 1us ECN

          Interpreted Rules:

          @84 anchor "userrules/*" all
  [ Evaluations: 73        Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 9606 State Creations: 0     ]
  [ Last Active Time: N/A ]
@85 pass out quick on vtnet0 route-to (vtnet0 <WAN Gateway>) inet from <WAN IP> to any flags S/SA keep state label "USER_RULE: Bufferbloat" label "id:1752945005" label "gw:WAN_DHCP" ridentifier 1752945005 dnqueue(1, 2)
  [ Evaluations: 73        Packets: 14677     Bytes: 15410930    States: 20    ]
  [ Inserted: uid 0 pid 9606 State Creations: 51    ]
  [ Last Active Time: Sat Jul 19 18:14:14 2025 ]
@86 pass in quick on vtnet1 inet from <LAN__NETWORK:1> to any flags S/SA keep state label "USER_RULE: Default allow LAN to any rule" label "id:0100000101" ridentifier 100000101 dnpipe(3, 4)
  [ Evaluations: 22        Packets: 14738     Bytes: 15555983    States: 15    ]
  [ Inserted: uid 0 pid 9606 State Creations: 22    ]
  [ Last Active Time: Sat Jul 19 18:14:14 2025 ]

          Example states:

          all tcp 23.239.29.5:443 <- 192.168.1.100:41090       ESTABLISHED:ESTABLISHED
   [3531538492 + 2147156224] wscale 7  [440337916 + 2147025152] wscale 7
   age 00:00:34, expires in 23:59:27, 15:18 pkts, 2254:10378 bytes, rule 86, dummynet pipe (3 4), log
   id: 09f07b6800000000 creatorid: ae2f1b15
   origif: vtnet1
all tcp <WAN IP>:1291 (192.168.1.100:41090) -> 23.239.29.5:443       ESTABLISHED:ESTABLISHED
   [440337916 + 2147025152] wscale 7  [3531538492 + 2147156224] wscale 7
   age 00:00:34, expires in 23:59:27, 15:18 pkts, 2254:10378 bytes, rule 85, log
   id: 0af07b6800000000 creatorid: ae2f1b15 route-to: <WAN Gateway>@vtnet0
   origif: vtnet0

          2.8.0
          Rules:

          # User-defined rules follow

anchor "userrules/*"
pass  out  quick  on {  vtnet0  }  $GWWAN_DHCP inet from <WAN IP> to any ridentifier 1752945012 keep state  dnqueue( 1,2) label "USER_RULE: Bufferbloat" label "id:1752945012" label "gw:WAN_DHCP"
pass  in  quick  on $LAN inet from $LAN__NETWORK to any ridentifier 0100000101 keep state  dnpipe ( 3,4) label "USER_RULE: Default allow LAN to any rule" label "id:0100000101"

#

          Limiter Info:

          Limiters:
00001:  20.000 Mbit/s    0 ms burst 0 
q131073  50 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0  AQM CoDel target 1us interval 1us ECN
 sched 65537 type FIFO flags 0x0 0 buckets 0 active
00002: 100.000 Mbit/s    0 ms burst 0 
q131074  50 sl. 0 flows (1 buckets) sched 65538 weight 0 lmax 0 pri 0  AQM CoDel target 1us interval 1us ECN
 sched 65538 type FIFO flags 0x0 0 buckets 0 active
00003:   2.000 Mbit/s    0 ms burst 0 
q131075  50 sl. 0 flows (1 buckets) sched 65539 weight 0 lmax 0 pri 0 droptail
 sched 65539 type FIFO flags 0x0 0 buckets 0 active
00004:   5.000 Mbit/s    0 ms burst 0 
q131076  50 sl. 0 flows (1 buckets) sched 65540 weight 0 lmax 0 pri 0 droptail
 sched 65540 type FIFO flags 0x0 0 buckets 0 active


Schedulers:
00001:  20.000 Mbit/s    0 ms burst 0 
q65537  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
 sched 1 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 1us interval 1us quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1 
00002: 100.000 Mbit/s    0 ms burst 0 
q65538  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0 droptail
 sched 2 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 1us interval 1us quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2 
00003:   2.000 Mbit/s    0 ms burst 0 
q65539  50 sl. 0 flows (1 buckets) sched 3 weight 0 lmax 0 pri 0 droptail
 sched 3 type FIFO flags 0x0 0 buckets 0 active
00004:   5.000 Mbit/s    0 ms burst 0 
q65540  50 sl. 0 flows (1 buckets) sched 4 weight 0 lmax 0 pri 0 droptail
 sched 4 type FIFO flags 0x0 0 buckets 0 active


Queues:
q00001  50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0  AQM CoDel target 1us interval 1us ECN
q00002  50 sl. 0 flows (1 buckets) sched 2 weight 0 lmax 0 pri 0  AQM CoDel target 1us interval 1us ECN

          Interpreted Rules:

          @85 anchor "userrules/*" all
  [ Evaluations: 66        Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 0 State Creations: 0     ]
  [ Last Active Time: N/A ]
@86 pass out quick on vtnet0 route-to (vtnet0 <WAN Gateway>) inet from <WAN IP> to any flags S/SA keep state (if-bound) label "USER_RULE: Bufferbloat" label "id:1752945012" label "gw:WAN_DHCP" ridentifier 1752945012 dnqueue(1, 2)
  [ Evaluations: 66        Packets: 163790    Bytes: 206160499   States: 21    ]
  [ Inserted: uid 0 pid 0 State Creations: 41    ]
  [ Last Active Time: Sat Jul 19 18:15:26 2025 ]
@87 pass in quick on vtnet1 inet from <LAN__NETWORK:1> to any flags S/SA keep state (if-bound) label "USER_RULE: Default allow LAN to any rule" label "id:0100000101" ridentifier 100000101 dnpipe(3, 4)
  [ Evaluations: 25        Packets: 154598    Bytes: 192395490   States: 14    ]
  [ Inserted: uid 0 pid 0 State Creations: 23    ]
  [ Last Active Time: Sat Jul 19 18:15:26 2025 ]

          Example states:

          vtnet1 tcp 23.239.29.5:443 <- 192.168.1.100:41256       ESTABLISHED:ESTABLISHED
   [4281932605 + 64128] wscale 7  [3565815079 + 63872] wscale 7
   age 00:00:34, expires in 23:59:27, 15:18 pkts, 2255:10378 bytes, rule 87, dummynet pipe (3 4)
   id: d9f57b6800000000 creatorid: 9d03805d
vtnet0 tcp <WAN IP>:42673 (192.168.1.100:41256) -> 23.239.29.5:443       ESTABLISHED:ESTABLISHED
   [3565815079 + 63872] wscale 7  [4281932605 + 64128] wscale 7
   age 00:00:34, expires in 23:59:27, 15:18 pkts, 2255:10378 bytes, rule 86
   id: daf57b6800000000 creatorid: 9d03805d route-to: <WAN Gateway>@vtnet0
          1 Reply Last reply Reply Quote 0
          • stephenw10S Offline
            stephenw10 Netgate Administrator
            last edited by

            The major difference there is the state binding. Note they are bound to all in 2.7.2 but interface bound in 2.8.0.

            Did you try reverting that change in 2.8.0 to see if that makes any difference?

            The other thing is that I expect to see the limiters in the reverse order on the outbound rule but it could be you're just testing that way? That might explain one of the test failures you saw above.

            Also that's the non-policy routing situation?

            N 1 Reply Last reply Reply Quote 0
            • N Offline
              NWOSwamp @stephenw10
              last edited by

              @stephenw10 said in [2.8.1.b] Multiple limiter issue:

              The major difference there is the state binding. Note they are bound to all in 2.7.2 but interface bound in 2.8.0.

              Did you try reverting that change in 2.8.0 to see if that makes any difference?

              It did not make a difference between Interface Bound States and Floating States on 2.8.0.

              vtnet1 tcp 23.239.29.5:443 <- 192.168.1.100:42446       ESTABLISHED:ESTABLISHED
   [1351492337 + 63360] wscale 7  [4004094591 + 63616] wscale 7
   age 00:00:39, expires in 23:59:58, 26:28 pkts, 3531:19177 bytes, rule 87, dummynet pipe (3 4)
   id: df107d6800000000 creatorid: 9d03805d
vtnet0 tcp <WAN IP>:17951 (192.168.1.100:42446) -> 23.239.29.5:443       ESTABLISHED:ESTABLISHED
   [4004094591 + 63616] wscale 7  [1351492337 + 63360] wscale 7
   age 00:00:39, expires in 23:59:58, 26:28 pkts, 3531:19177 bytes, rule 86
   id: e0107d6800000000 creatorid: 9d03805d route-to: <WAN Gateway>@vtnet0

              and

              all tcp 23.239.29.5:443 <- 192.168.1.100:55138       ESTABLISHED:ESTABLISHED
   [2584166382 + 63360] wscale 7  [3625120434 + 63488] wscale 7
   age 00:00:40, expires in 24:00:00, 40:45 pkts, 4557:37927 bytes, rule 87, dummynet pipe (3 4)
   id: 04137d6800000000 creatorid: 9d03805d
   origif: vtnet1
all tcp <WAN IP>:49985 (192.168.1.100:55138) -> 23.239.29.5:443       ESTABLISHED:ESTABLISHED
   [3625120434 + 63488] wscale 7  [2584166382 + 63360] wscale 7
   age 00:00:40, expires in 24:00:00, 40:45 pkts, 4557:37927 bytes, rule 86
   id: 05137d6800000000 creatorid: 9d03805d route-to: <WAN Gateway>@vtnet0
   origif: vtnet0

              The other thing is that I expect to see the limiters in the reverse order on the outbound rule but it could be you're just testing that way? That might explain one of the test failures you saw above.

              If I understand this, the labeling of my limiter matches the GUI. GUI option is labeled In / Out Pipe so I have the first one labeled WAN-in-q & LAN-in, the second WAN-out-q & LAN-out. I verified bandwidths amounts set in the limiters and the order in the rules are correct and consistent between the two versions.

              Also that's the non-policy routing situation?

              I'm not PBRing in this case. At one site, I have 2 WANs and PBR some devices when the primary fails. There are no limiters on any PBR rule. The floating rule on the primary WAN has the same Bufferbloat limiters. The other site has a single WAN and no PBR.

              stephenw10S 1 Reply Last reply Reply Quote 0
              • stephenw10S Offline
                stephenw10 Netgate Administrator @NWOSwamp
                last edited by

                @NWOSwamp said in [2.8.1.b] Multiple limiter issue:

                It did not make a difference between Interface Bound States and Floating States on 2.8.0.

                Ok good to know.

                @NWOSwamp said in [2.8.1.b] Multiple limiter issue:

                GUI option is labeled In / Out Pipe so I have the first one labeled WAN-in-q & LAN-in, the second WAN-out-q & LAN-out.

                But it also says: If creating a floating rule, if the direction is In then the same rules apply, if the direction is Out the selections are reversed,
                And here the rule is indeed a floating OUT rule so for the WAN side they should be reversed. As unintuitive as that is!

                N 1 Reply Last reply Reply Quote 0
                • N Offline
                  NWOSwamp @stephenw10
                  last edited by

                  @stephenw10 said in [2.8.1.b] Multiple limiter issue:

                  But it also says: If creating a floating rule, if the direction is In then the same rules apply, if the direction is Out the selections are reversed,
                  And here the rule is indeed a floating OUT rule so for the WAN side they should be reversed. As unintuitive as that is!

                  Yep, thanks. Get it and am aware of that. I never remember which way is straight and which is reversed. To make it easier for my feeble memory, if I need to edit or recreate the rule, is to follow the label. The limiter has the correct bandwidth. I donno, makes sense to me but definitely see it confuses anyone else. Maybe there's an easier way...

                  download.png
                  upload.png

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S Offline
                    stephenw10 Netgate Administrator
                    last edited by

                    Ah OK I see, the names threw me!

                    1 Reply Last reply Reply Quote 0
                    • N Offline
                      NWOSwamp
                      last edited by

                      Same behavior is present in the latest RC (2.8.1.r.20250808.1925).

                      1 Reply Last reply Reply Quote 0
                      • stephenw10S Offline
                        stephenw10 Netgate Administrator
                        last edited by

                        I suspect the root cause here is the same as this: https://redmine.pfsense.org/issues/15770

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post
                        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.