Wrong limiter speed



  • interface using em0:em1
    set limiter to 905 Mbit/s
    speedtest can't go over 640 Mbit/s

    if i set my limiter to 965 Mbit/s
    il'l get around 853 Mbit/s

    more the limiter is high more de accuracy is wrong

    is there a broblem with limiter ??



  • I have the same issues. I wind up just increasing the limiter until I can get the actual speeds I'm looking for. At the moment, I have a rule for 350Mbps but I can't scratch above 270Mbps with it. I tell users it's set at 300Mbps and they are within 10% so it's OK, but I don't understand it. It'd be great to get an answer on this, though.



  • I believe that netgate does not worry about 2 idiots who are in the same boat, about a problem which does not exist! do'H



  • @Jimbohello said in Wrong limiter speed:

    I believe that netgate does not worry about 2 idiots who are in the same boat, about a problem which does not exist! do'H

    Well, it must exist since we are both experiencing it. I just kinda gave up on trying to fix it long ago. I remember posting about it but I can't remember if I ever got a response, much less a resolution. I know what settings will give people the numbers they want to see so I just go with it.

    In actuality, I logged into the forums today looking for an answer to this exact same problem. I've tried different algorithms with no success.



  • @Stewart Have you tried setting Powerd to maximum in
    System / Advanced / Miscellaneous?
    I remember encountering something similar once which the above fixed.



  • @bobbenheim Thank's
    did not change noting !
    your help was please ! thanks



  • Hello!

    Never used a limiter in pfsense, but the docs say :

    "Limiters cannot effectively guarantee a minimum bandwidth amount for a pipe or queue, only a maximum."

    I understand this to mean that you wont get more than the limit, but you could get less. Blowing past the limit would be one thing, but never hitting it? I dont know...

    https://docs.netgate.com/pfsense/en/latest/trafficshaper/limiters.html

    John



  • Hey
    Tested it right now
    And hit the limit (not able to get more than the limit)

    Limit 5 on a 100

    The first time
    Then 4,5 then 4,8....

    Not gettin more than the limit an hit it once ;) out of 3



  • @Jimbohello what system are you running pfsense on and are you seeing higher speeds without limiter?



  • @serbus
    first off, i read the documentation already.
    cant’t garranty minimum exacly !
    but suppose to garranty a maximum, that’s the issue.

    if you look a the firts post ! you will understand better ! the situation does appears on a breach of certain point !
    let say max limiter is 300 you will get 270
    let say limiter is set to 425 you will get 390
    let say limiter is set to 905 you will get 670
    and so on ! nothing to do with limiting 5 or 10 or 20! we are talking about more the limiter size increace more de accuracy is wrong

    thank’s



  • @bobbenheim

    lenovo m93p
    core i5 4th gen
    no hardware offloading
    aes-ni
    2 nic intel pro

    useless stuf disable in bios / onboard nic / onboard sound card / etc !



  • @bobbenheim

    and ! yes the maximun reach point is reach without limiter !
    940 -945 approx



  • We've seen in APU1 and APU2 boxes. The disparity gets larger and larger the higher we set it.



  • @Stewart said in Wrong limiter speed:

    We've seen in APU1 and APU2 boxes

    yes same here.. but with some testing
    the limiter is set to a solid work.

    we use it to prevent clients / networks / hosts
    to consume all the bandwidth available



  • What does Diagnostics > Limiter Info show during the testing?

    See post https://forum.netgate.com/topic/112527/playing-with-fq_codel-in-2-4/38 ... "On downloads you will commonly see below the configured bitrate because each time you hit the limit pfSense will tell the sender to slow down below the limit."

    "minimum bandwidth" I think is referring to that limiters can't guarantee a device will get any bandwidth at all, they can only cap the device.



  • @Jimbohello what are your limiter settings?



  • @bobbenheim
    <dnshaper>
    <queue>
    <name>in</name>
    <number>1</number>
    <qlimit></qlimit>
    <plr></plr>
    <description></description>
    <bandwidth>
    <item>
    <bw>799</bw>
    <burst></burst>
    <bwscale>Mb</bwscale>
    <bwsched>none</bwsched>
    </item>
    </bandwidth>
    <enabled>on</enabled>
    <buckets></buckets>
    <mask>none</mask>
    <maskbits></maskbits>
    <maskbitsv6></maskbitsv6>
    <delay>0</delay>
    <sched>wf2q+</sched>
    <aqm>droptail</aqm>
    <ecn></ecn>
    <queue>
    <name>in-queue</name>
    <number>1</number>
    <qlimit></qlimit>
    <description></description>
    <weight></weight>
    <enabled>on</enabled>
    <buckets></buckets>
    <mask>dstaddress</mask>
    <maskbits>32</maskbits>
    <maskbitsv6>128</maskbitsv6>
    <aqm>droptail</aqm>
    <ecn></ecn>
    </queue>
    </queue>
    <queue>
    <name>out</name>
    <number>2</number>
    <qlimit></qlimit>
    <plr></plr>
    <description></description>
    <bandwidth>
    <item>
    <bw>799</bw>
    <burst></burst>
    <bwscale>Mb</bwscale>
    <bwsched>none</bwsched>
    </item>
    </bandwidth>
    <enabled>on</enabled>
    <buckets></buckets>
    <mask>none</mask>
    <maskbits></maskbits>
    <maskbitsv6></maskbitsv6>
    <delay>0</delay>
    <sched>wf2q+</sched>
    <aqm>droptail</aqm>
    <ecn></ecn>
    <queue>
    <name>out-queue</name>
    <number>2</number>
    <qlimit></qlimit>
    <description></description>
    <weight></weight>
    <enabled>on</enabled>
    <buckets></buckets>
    <mask>srcaddress</mask>
    <maskbits>32</maskbits>
    <maskbitsv6>128</maskbitsv6>
    <aqm>droptail</aqm>
    <ecn></ecn>
    </queue>
    </queue>
    </dnshaper>



  • @noplan Certainly we can make it work. It's just that the numbers don't really line up. Oh, and fast.com goes crazy when put through one of the limiters. Through the ones we set up last week fast.com couldn't get more than 120Mbps no matter what the limiter was set to. Without the limiter it would get the full 600+Mbps. I thought that was odd. Speedtest.net is the one that would increase, just at a different rate to what the limiter was being set at.



  • @Jimbohello Can you try enabling ECN in the limiter (not the queues) and see if that makes a difference, else you might also need to increase queue length from the default 50 to something larger.
    Have you made any tweaks in loader.conf.local?



  • @bobbenheim

    what tweak ?

    in loader.conf.local



  • @bobbenheim
    Explicit Congestion Notification is selected, but neither Tail Drop nor Worst-case Weighted fair Queueing (default) support it.



  • @bobbenheim

    this give a better result on lowering de error in the accuracy
    from a limit of 799
    i can reach 735

    before it was close to 500 !
    we are on the right way ! :)

    <dnshaper>
    <queue>
    <name>in</name>
    <number>1</number>
    <qlimit>100</qlimit>
    <plr></plr>
    <description></description>
    <bandwidth>
    <item>
    <bw>799</bw>
    <burst></burst>
    <bwscale>Mb</bwscale>
    <bwsched>none</bwsched>
    </item>
    </bandwidth>
    <enabled>on</enabled>
    <buckets></buckets>
    <mask>none</mask>
    <maskbits></maskbits>
    <maskbitsv6></maskbitsv6>
    <delay>0</delay>
    <sched>fq_codel</sched>
    <aqm>codel</aqm>
    <ecn>on</ecn>
    <queue>
    <name>in-queue</name>
    <number>1</number>
    <qlimit></qlimit>
    <description></description>
    <weight></weight>
    <enabled>on</enabled>
    <buckets></buckets>
    <mask>dstaddress</mask>
    <maskbits>32</maskbits>
    <maskbitsv6>128</maskbitsv6>
    <aqm>codel</aqm>
    <ecn></ecn>
    <param_codel_target>5</param_codel_target>
    <param_codel_interval>100</param_codel_interval>
    </queue>
    <param_fq_codel_target>5</param_fq_codel_target>
    <param_fq_codel_interval>100</param_fq_codel_interval>
    <param_fq_codel_quantum>1514</param_fq_codel_quantum>
    <param_fq_codel_limit>10240</param_fq_codel_limit>
    <param_fq_codel_flows>1024</param_fq_codel_flows>
    <param_codel_target>5</param_codel_target>
    <param_codel_interval>100</param_codel_interval>
    </queue>
    <queue>
    <name>out</name>
    <number>2</number>
    <qlimit>100</qlimit>
    <plr></plr>
    <description></description>
    <bandwidth>
    <item>
    <bw>799</bw>
    <burst></burst>
    <bwscale>Mb</bwscale>
    <bwsched>none</bwsched>
    </item>
    </bandwidth>
    <enabled>on</enabled>
    <buckets></buckets>
    <mask>none</mask>
    <maskbits></maskbits>
    <maskbitsv6></maskbitsv6>
    <delay>0</delay>
    <sched>fq_codel</sched>
    <aqm>codel</aqm>
    <ecn>on</ecn>
    <queue>
    <name>out-queue</name>
    <number>2</number>
    <qlimit></qlimit>
    <description></description>
    <weight></weight>
    <enabled>on</enabled>
    <buckets></buckets>
    <mask>srcaddress</mask>
    <maskbits>32</maskbits>
    <maskbitsv6>128</maskbitsv6>
    <aqm>codel</aqm>
    <ecn></ecn>
    <param_codel_target>5</param_codel_target>
    <param_codel_interval>100</param_codel_interval>
    </queue>
    <param_fq_codel_target>5</param_fq_codel_target>
    <param_fq_codel_interval>100</param_fq_codel_interval>
    <param_fq_codel_quantum>1514</param_fq_codel_quantum>
    <param_fq_codel_limit>10240</param_fq_codel_limit>
    <param_fq_codel_flows>1024</param_fq_codel_flows>
    <param_codel_target>5</param_codel_target>
    <param_codel_interval>100</param_codel_interval>
    </queue>
    </dnshaper>



  • @Jimbohello can you run "top -aSH" in Diagnostics/Command Prompt and paste what it output under load?
    There might also be some tweaks you can do in loader.conf.local
    can you do a sysctl (e.g. "sysctl hw.em.enable_aim") in the command prompt for the following:
    hw.em.enable_aim
    hw.em.flow_control
    hw.em.num_queues
    hw.em.rx_process_limit
    hw.em.tx_process_limit
    hw.em.rxd
    hw.em.txd
    hw.em.max_interrupt_rate
    net.link.ifqmaxlen



  • @bobbenheim

    il get in touch as soon as i can do it ! but honestly
    this is getting far deeper for something that is created on a simple click. i means creating a limiter suppose to be easy as 123.
    IMO this is getthing ridiculous ! if so many thing need to be modify on a single limiter limitation, why when your press APPLY they don’t run a script to set the right adjustement ! imagine you need to forward a port, and then you have to modify this and that TO MAKE IT WORK ! That will be irrelevent or if you prefer useless function !

    anyway ill get back to you
    youre help is please !



  • @Jimbohello said in Wrong limiter speed:

    but honestly
    this is getting far deeper for something that is created on a simple click

    yes true and 4 WHAT

    cuz the limiter does what it is supposed to do !
    here and on some other boxes

    to be honest i can not see the use case for gettin as deep as shown here
    an any set up

    maybe u can help me out here explainin whats the urgent need



  • @noplan

    nothing urgent bro !
    simple !

    let say your 50 on a network, 5 start a download !
    the gateway goes down and packetloss occur on high level !

    with the limiter nothing of that appends because gateway still have room to breathed and nothing goes down on the gateway.

    that the big advantage of the limiter.



  • For us, we were setting it up in a church. They stream several services at the same time. To accommodate this we brought in a second WAN and route specific devices out that interface. To prevent each machine from taking up too much of the bandwidth and affecting the other streams we limit their max usage. In this case it is a 35Mbps upload split between 4 devices, each with a 10Mbps upload limit but only needing 5Mbps for the streams. For example, there are 4 streams going and someone decides to (unwisely and against policy) upload a previous stream to the platform from one of the streaming boxes. Normally that would use up all possible bandwidth but with a limiter in place that particular device is unable to affect the other streams. If the full 10Mbps is used then there is still 25Mbps remaining for the other 3 devices. If each of those 3 devices is using 5Mbps then they are only using 15Mbps out of the 25Mbps, leaving a cushion of 10Mbps. Technically, in this case, 3 users could be offending and the 4th stream should still run fine. Without the limiter just one bad actor could ruin it for everyone else.



  • @Jimbohello said in Wrong limiter speed:

    on

    Could you post this as a screenshot, cropped to just the config section? I'm having a hard time following and duplicating. Thanks.



  • @Jimbohello The problem is that traffic shaping isn't a one size fits all kinda setup. Default settings is more likely to be something that works with bandwidths likely to be 100 Mb/s or below because that is what general use cases were, when the settings were chosen. These settings doesn't take other factors, like different types of hardware setup used or what type of internet connection is used (COAX, xDSL, PTP fiber, GPON fiber), into consideration.
    This also means that the above 900 Mb/s, like in your case, doesn't show the expected performance because the default settings just doesn't work well for that kinda of bandwidth. So this means that tuning is necessary to either obtain the maximum performance which is possible or what is expected as set by the limiter. Once done tuning you can leave it alone and don't bother with it again.



  • @bobbenheim

    thank’s for the explaination!
    verry appreciate
    👍👍👍



  • @bobbenheim said in Wrong limiter speed:

    top -aSH - speddtest.net

    last pid: 33530; load averages: 0.07, 0.06, 0.01 up 19+18:28:36 12:05:35
    181 processes: 5 running, 149 sleeping, 27 waiting

    Mem: 11M Active, 135M Inact, 189M Wired, 19M Buf, 7474M Free
    Swap: 3979M Total, 3979M Free

    PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
    11 root 155 ki31 0K 64K CPU1 1 474.2H 100.00% [idle{idle: cpu1}]
    11 root 155 ki31 0K 64K CPU3 3 474.2H 100.00% [idle{idle: cpu3}]
    11 root 155 ki31 0K 64K CPU2 2 474.2H 100.00% [idle{idle: cpu2}]
    11 root 155 ki31 0K 64K RUN 0 468.6H 100.00% [idle{idle: cpu0}]
    12 root -92 - 0K 432K WAIT 2 1:48 4.59% [intr{irq269: em1:rx0}]
    12 root -92 - 0K 432K WAIT 0 2:44 3.96% [intr{irq264: em0:rx0}]
    346 root 26 0 99704K 39460K piperd 3 0:06 1.17% php-fpm: pool nginx (php-fpm){php-fpm}
    12 root -92 - 0K 432K WAIT 1 0:11 0.20% [intr{irq265: em0:tx0}]
    347 root 52 0 95092K 36704K accept 3 0:05 0.20% php-fpm: pool nginx (php-fpm)
    0 root -92 - 0K 624K - 2 334:57 0.00% [kernel{dummynet}]
    12 root -60 - 0K 432K WAIT 1 10:59 0.00% [intr{swi4: clock (0)}]
    23 root -16 - 0K 16K - 2 1:02 0.00% [rand_harvestq]
    84645 root 52 20 6976K 2752K wait 2 1:00 0.00% /bin/sh /var/db/rrd/updaterrd.sh
    20 root -16 - 0K 16K pftm 2 0:41 0.00% [pf purge]
    46951 unbound 20 0 60368K 37608K kqread 3 0:28 0.00% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}
    28775 root 20 0 6904K 2340K nanslp 2 0:23 0.00% /usr/local/bin/dpinger -S -r 0 -i WAN_PPPOE -B 65.94.26.33 -p /var/run/dpinger_WAN_PPPOE~65.94.26.33~10.11.16.17.pid -u /var/ru
    0 root -16 - 0K 624K swapin 1 0:20 0.00% [kernel{swapper}]
    70389 root 20 0 12464K 5760K select 3 0:16 0.00% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid{ntpd}

    sysctl: unknown oid 'hw.em.enable_aim'
    sysctl: unknown oid hw.em.flow_control
    sysctl: unknown oid hw.em.tx_process_limit
    hw.em.rx_process_limit: 100
    hw.em.rx_process_limit: 100
    hw.em.rxd: 1024
    hw.em.txd: 1024
    sysctl: unknown oid 'hw.em.max_interrupt_rate'
    net.link.ifqmaxlen: 128



  • @Stewart in.zip in-queue.zip

    same for the out queue except sources/adress



  • @Jimbohello can you please run "sysctl hw.em" and "sysctl dev.em" in the command prompt and paste the result.



  • @bobbenheim said in Wrong limiter speed:

    sysctl hw.em

    hw.em.eee_setting: 1
    hw.em.rx_process_limit: 100
    hw.em.enable_msix: 1
    hw.em.sbp: 0
    hw.em.smart_pwr_down: 0
    hw.em.txd: 1024
    hw.em.rxd: 1024
    hw.em.rx_abs_int_delay: 66
    hw.em.tx_abs_int_delay: 66
    hw.em.rx_int_delay: 0
    hw.em.tx_int_delay: 66
    hw.em.disable_crc_stripping: 0



  • OK
    I m Readin and learning

    A Lot
    Thank you!!!



  • @Jimbohello I just assume you haven't tried this before so i will list the following out in simple steps:

    • go to Diagnostics / Edit File

    • press browse and find boot/loader.conf

    • add .local so you get the string "/boot/loader.conf.local" and press save

    • press browse then find and click on loader.conf.local

    • delete the commands within loader.conf.local and add the following:
      hw.em.eee_setting="0"
      hw.em.rx_process_limit="-1"
      hw.em.txd="2048"
      hw.em.rxd="2048"
      net.link.ifqmaxlen="4096"

    • hit save and reboot



  • @bobbenheim

    WOW thank's for your precious help
    on a limiter of 799 i can same as yesterday so close to 735
    so no change !
    up to date the queue lenght 100 was the best tunning
    can you please provide fast explanation on these Tuning you provide !
    i know that *.local usually will load the *.conf and override.
    But what are these tunning ??
    THANK'S



  • @bobbenheim
    1 came close to 760 !
    i can easy live with that.
    that's the best setting so far ! it's almost the limiter setting 799 ! the accuracy bettewn 760/799 is damw better !
    BRAVO !
    explaination on your tunning steel be please ! GRACIAS



  • Try and keep a download running for some minutes and check monitoring you might have the correct limitation but the speedtest are not taking overhead into consideration.
    Need to get back to you on those settings, it's time for some sleep :)



  • Found a couple of links that explains what the following does:

    hw.em.eee_setting
    https://www.freebsd.org/cgi/man.cgi?em(4)

    hw.em.rx_process_limit
    hw.em.rxd
    hw.em.txd
    net.link.ifqmaxlen
    https://calomel.org/freebsd_network_tuning.html
    look under local.conf and just ignore that he uses an Intel nic which uses the igb driver instead of the em driver.


Log in to reply