Wrong limiter speed

Stewart

For us, we were setting it up in a church. They stream several services at the same time. To accommodate this we brought in a second WAN and route specific devices out that interface. To prevent each machine from taking up too much of the bandwidth and affecting the other streams we limit their max usage. In this case it is a 35Mbps upload split between 4 devices, each with a 10Mbps upload limit but only needing 5Mbps for the streams. For example, there are 4 streams going and someone decides to (unwisely and against policy) upload a previous stream to the platform from one of the streaming boxes. Normally that would use up all possible bandwidth but with a limiter in place that particular device is unable to affect the other streams. If the full 10Mbps is used then there is still 25Mbps remaining for the other 3 devices. If each of those 3 devices is using 5Mbps then they are only using 15Mbps out of the 25Mbps, leaving a cushion of 10Mbps. Technically, in this case, 3 users could be offending and the 4th stream should still run fine. Without the limiter just one bad actor could ruin it for everyone else.

Stewart

@Jimbohello said in Wrong limiter speed:

on

Could you post this as a screenshot, cropped to just the config section? I'm having a hard time following and duplicating. Thanks.

bobbenheim

@Jimbohello The problem is that traffic shaping isn't a one size fits all kinda setup. Default settings is more likely to be something that works with bandwidths likely to be 100 Mb/s or below because that is what general use cases were, when the settings were chosen. These settings doesn't take other factors, like different types of hardware setup used or what type of internet connection is used (COAX, xDSL, PTP fiber, GPON fiber), into consideration.
This also means that the above 900 Mb/s, like in your case, doesn't show the expected performance because the default settings just doesn't work well for that kinda of bandwidth. So this means that tuning is necessary to either obtain the maximum performance which is possible or what is expected as set by the limiter. Once done tuning you can leave it alone and don't bother with it again.

Jimbohello

@bobbenheim

thank’s for the explaination!
verry appreciate

Jimbohello

@bobbenheim said in Wrong limiter speed:

top -aSH - speddtest.net

last pid: 33530; load averages: 0.07, 0.06, 0.01 up 19+18:28:36 12:05:35
181 processes: 5 running, 149 sleeping, 27 waiting

Mem: 11M Active, 135M Inact, 189M Wired, 19M Buf, 7474M Free
Swap: 3979M Total, 3979M Free

PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
11 root 155 ki31 0K 64K CPU1 1 474.2H 100.00% [idle{idle: cpu1}]
11 root 155 ki31 0K 64K CPU3 3 474.2H 100.00% [idle{idle: cpu3}]
11 root 155 ki31 0K 64K CPU2 2 474.2H 100.00% [idle{idle: cpu2}]
11 root 155 ki31 0K 64K RUN 0 468.6H 100.00% [idle{idle: cpu0}]
12 root -92 - 0K 432K WAIT 2 1:48 4.59% [intr{irq269: em1:rx0}]
12 root -92 - 0K 432K WAIT 0 2:44 3.96% [intr{irq264: em0:rx0}]
346 root 26 0 99704K 39460K piperd 3 0:06 1.17% php-fpm: pool nginx (php-fpm){php-fpm}
12 root -92 - 0K 432K WAIT 1 0:11 0.20% [intr{irq265: em0:tx0}]
347 root 52 0 95092K 36704K accept 3 0:05 0.20% php-fpm: pool nginx (php-fpm)
0 root -92 - 0K 624K - 2 334:57 0.00% [kernel{dummynet}]
12 root -60 - 0K 432K WAIT 1 10:59 0.00% [intr{swi4: clock (0)}]
23 root -16 - 0K 16K - 2 1:02 0.00% [rand_harvestq]
84645 root 52 20 6976K 2752K wait 2 1:00 0.00% /bin/sh /var/db/rrd/updaterrd.sh
20 root -16 - 0K 16K pftm 2 0:41 0.00% [pf purge]
46951 unbound 20 0 60368K 37608K kqread 3 0:28 0.00% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}
28775 root 20 0 6904K 2340K nanslp 2 0:23 0.00% /usr/local/bin/dpinger -S -r 0 -i WAN_PPPOE -B 65.94.26.33 -p /var/run/dpinger_WAN_PPPOE~65.94.26.33~10.11.16.17.pid -u /var/ru
0 root -16 - 0K 624K swapin 1 0:20 0.00% [kernel{swapper}]
70389 root 20 0 12464K 5760K select 3 0:16 0.00% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid{ntpd}

sysctl: unknown oid 'hw.em.enable_aim'
sysctl: unknown oid hw.em.flow_control
sysctl: unknown oid hw.em.tx_process_limit
hw.em.rx_process_limit: 100
hw.em.rx_process_limit: 100
hw.em.rxd: 1024
hw.em.txd: 1024
sysctl: unknown oid 'hw.em.max_interrupt_rate'
net.link.ifqmaxlen: 128

Jimbohello

@Stewart in.zip in-queue.zip

same for the out queue except sources/adress

bobbenheim

@Jimbohello can you please run "sysctl hw.em" and "sysctl dev.em" in the command prompt and paste the result.

Jimbohello

@bobbenheim said in Wrong limiter speed:

sysctl hw.em

hw.em.eee_setting: 1
hw.em.rx_process_limit: 100
hw.em.enable_msix: 1
hw.em.sbp: 0
hw.em.smart_pwr_down: 0
hw.em.txd: 1024
hw.em.rxd: 1024
hw.em.rx_abs_int_delay: 66
hw.em.tx_abs_int_delay: 66
hw.em.rx_int_delay: 0
hw.em.tx_int_delay: 66
hw.em.disable_crc_stripping: 0

noplan

OK
I m Readin and learning

A Lot
Thank you!!!

bobbenheim

@Jimbohello I just assume you haven't tried this before so i will list the following out in simple steps:

go to Diagnostics / Edit File
press browse and find boot/loader.conf
add .local so you get the string "/boot/loader.conf.local" and press save
press browse then find and click on loader.conf.local
delete the commands within loader.conf.local and add the following:
hw.em.eee_setting="0"
hw.em.rx_process_limit="-1"
hw.em.txd="2048"
hw.em.rxd="2048"
net.link.ifqmaxlen="4096"
hit save and reboot

Jimbohello

@bobbenheim

WOW thank's for your precious help
on a limiter of 799 i can same as yesterday so close to 735
so no change !
up to date the queue lenght 100 was the best tunning
can you please provide fast explanation on these Tuning you provide !
i know that *.local usually will load the *.conf and override.
But what are these tunning ??
THANK'S

Jimbohello

@bobbenheim
1 came close to 760 !
i can easy live with that.
that's the best setting so far ! it's almost the limiter setting 799 ! the accuracy bettewn 760/799 is damw better !
BRAVO !
explaination on your tunning steel be please ! GRACIAS

bobbenheim

Try and keep a download running for some minutes and check monitoring you might have the correct limitation but the speedtest are not taking overhead into consideration.
Need to get back to you on those settings, it's time for some sleep :)

bobbenheim

Found a couple of links that explains what the following does:

hw.em.eee_setting
https://www.freebsd.org/cgi/man.cgi?em(4)

hw.em.rx_process_limit
hw.em.rxd
hw.em.txd
net.link.ifqmaxlen
https://calomel.org/freebsd_network_tuning.html
look under local.conf and just ignore that he uses an Intel nic which uses the igb driver instead of the em driver.

Jimbohello

@bobbenheim
i look at the link you send me
hw.em.eee_setting="0"
mean that energy is enabled
to be at full (no Energy Efficient Ethernet) always should it be disable =1
to force the nic to be at maximum !??

bobbenheim

@Jimbohello Didn't take notice of that, but you can give it a try and change it back to see if it makes a difference. But so far it seems that a lot of people have misunderstood that one if "1" is disable, i wouldn't know since i don't use the em driver and therefore haven't tested it.

Jimbohello

@bobbenheim
because i'm french
correct me if i'm wrong

https://www.freebsd.org/cgi/man.cgi?em(4)

 hw.em.eee_setting
     Disable or	enable Energy Efficient	Ethernet.  Default 1 (dis-
     abled).

so 1 = disabled
wright ??

bobbenheim

@Jimbohello yes i would interpret it that way

Jimbohello

@bobbenheim
What's up BOB !
/boot/loader.conf.local - final - best result ever !
if_em_load="YES"
cc_htcp_load="YES"
hw.em.eee_setting="0"
hw.em.rx_process_limit="-1"
hw.em.txd="2048"
hw.em.rxd="2048"
net.link.ifqmaxlen="4096"

4 computer downloading debian iso 4.4 gig
RTT never moved 0.6ms
RTTsd never moved 0.2ms
Lost 0.0 %

this is far the best config ever
limiter config Download
Codel
FQ_codel
queue lenght 100
ecn enable

limiter Queue Download
Mask / Destination Adresse /32
Codel

limiter config Upload
Codel
FQ_codel
queue lenght 100
ecn enable

limiter Queue Upload
Mask / Source Adresse /32
Codel

limiter on lan rules
in= upload-queue
out=download-queue

with all helps ! all people sharing knoledge ! everything's possible!
Thank you so muck