Best way to reduce game latency
-
Hey tman222,
thanks man for the help, when i up the limit to 930 or 940 the same happens, no real improvement.
The NICS are https://ark.intel.com/products/64404/Intel-Ethernet-Controller-I211-AT if i check what the parameters are the following shows up
These are my current values. maybe i should play around with them.
hw.igb.txd: 1024 hw.igb.rxd: 1024 net.pf.states_hashsize: 32768 net.pf.source_nodes_hashsize: 8192 hw.igb.tx_process_limit: -1 hw.igb.rx_process_limit: 100 net.inet.tcp.syncache.hashsize: 512 net.inet.tcp.syncache.bucketlimit: 30If i would like to change them i most likely have to put them into system tunables, right ?
-
Hey tman222,
thanks man for the help, when i up the limit to 930 or 940 the same happens, no real improvement.
The NICS are https://ark.intel.com/products/64404/Intel-Ethernet-Controller-I211-AT if i check what the parameters are the following shows up
These are my current values. maybe i should play around with them.
hw.igb.txd: 1024 hw.igb.rxd: 1024 net.pf.states_hashsize: 32768 net.pf.source_nodes_hashsize: 8192 hw.igb.tx_process_limit: -1 hw.igb.rx_process_limit: 100 net.inet.tcp.syncache.hashsize: 512 net.inet.tcp.syncache.bucketlimit: 30If i would like to change them i most likely have to put them into system tunables, right ?
Hi again,
Yes, you can change those settings either in the System Tunables section under Advanced Settings, or you can also put them in /boot/loader.conf.local
To begin, I would change the following:
hw.igb.txd: 2048
hw.igb.rxd: 2048hw.igb.tx_process_limit: -1
hw.igb.rx_process_limit: -1 (100 is probably too low for a fast connection like yours).Also, what value did you have for kern.ipc.nmbclusters? If it's less than 131072, I would change it to 131072 to start and see if that offers any improvement as outlined here:
https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards
–-----
Let's see if changing those parameters offers some improvement. Hope this helps.
-
so i completely reinstalled pfsense, from scratch, just set up the traffic shaper. same results as before.
Then i added
hw.igb.txd: 2048
hw.igb.rxd: 2048hw.igb.tx_process_limit: -1
hw.igb.rx_process_limit: -1but besides taking more memory nothing really changed. my kern.ipc.nmbclusters are twice that much. Whats next ? its 3 am and i just restored everything to the before stage… :( Thanks tman for all your help i am really clueless :(
-
Hmmm, this is indeed perplexing and I'm running out of ideas unfortunately. However, there's an alternative we can try. Instead of using dummynet (limiters) and fq_codel, we can emulate the behavior of fq_codel using the ALTQ traffic shaping by using the FAIRQ Scheduler and Codel controlled queues. The performance of this is similar to fq_codel. Would you be willing to try that?
Here's how you would set it up:
- First off remove all your fq_codel limiters and associated queues from both Firewall/Traffic Shaper and from you your firewall rules.
- Next go to Firewall/Traffic Shaper/By Interface tab
- For your WAN interface, choose scheduler type FAIRQ and set bandwidth equal to 900 Mbit/s. Check Enable/disable discipline and its children and hit Save.
- Next go to the bottom and click "Add new Queue".
- In the queue settings choose a name, then choose the default priority of 1. For "Queue Limit", choose either 512 or 1024 (the default is 50, which is too low for your connection speed). For scheduler options check "Default Queue" and "Codel Active Queue". For bandwidth choose 900 Mbit/s. Check "Enable/disable discipline and its children". Click Save to save the queue settings.
- Repeat steps 3-5 for your LAN interface.
Once you have done that, run a speed test again. What does the performance look like?
Hope this helps.
-
Hiii,
this is exactly the way i had it set up before based on this http://www.speedtest.net/insights/blog/maximized-speed-non-gigabit-internet-connection/ article, which lead me to the whole qu_coddle thread here :D
The tests are great i get like ABA mainly,which is better then FCA, however i would really like to know what is off with my system that the qa_cddle thing isnt working, might it be the ram? or similar
-
Thanks for getting back to me. So it's good to know that ALTQ FAIRQ + Codel does work in your case. However, we should be able to get fq_codel working as well using dummynet (limiters).
I have a few more questions for you:
- Is there anything special about your symmetric gigabit connection (e.g. are you using PPoE or something like that)?
- What pfSense add-on's/plug-in's are you running, if any?
- When you installed pfSense from scratch, did you also re-enable are your WAN NAT firewall rules, or did you try shaping with just the defaults (i.e. no special firewall rules on WAN and/or LAN)? I'd be curious to see what results look like with just the system defaults (i.e. no special firewall rules and no add-on's/plug-in's).
- Can you do me a favor and show me screenshots again for your limiter and queue settings, firewall rules, as well as the fq_codel configuration (output) from the command line? I just want to check one more time to make sure we didn't miss anything obvious.
Hope this helps.
-
Hey Tman222,
I am trying to answer this to the best of my ability, i dont think there is anything special about my fiber cable. Its an FTTH setup
Fiber cable -> TP-LINK MC220L, 1x SFP 1000Base-SX/LX/LH, 1x RJ45 1000Base-T (Media converter) + TP-LINK TL-SM321B, SFP, Simplex, LX/LC (Transceiver) -> RJ45 -> PFSENSE
PFSENSE:
Intel(R) Core(TM) i5-5250U CPU @ 1.60GHz
4 1Gbit Intel NIC i211-AT
120GB SSD
4GB RamPFsense Plugins (typically shellcmd103 haproxy0552 nmap1441 ntopng0811 pfblockerng2122) however at the moment only haproxy is on.
after resetting the pfsense i changed the ip of the box created the limiters changed the in/outpipe of the default lan rule allow all,set the traffic shaping to qa_codle through the cmd and run the dslreport test
i did not change anything regarding NAT or other rules, everything should be set to default. such as NAT reflection and so on.
- https://imgur.com/a/5z4zM this is still how i have it.
at the moment i have my download limit to 500 and upload to 890
/tmp/rules.limiterpipe 1 config bw 500Mb queue 1 config pipe 1 mask dst-ip6 /128 dst-ip 0xffffffff pipe 2 config bw 890Mb queue 2 config pipe 2 mask src-ip6 /128 src-ip 0xffffffffand ipfw sched show
00001: 510.000 Mbit/s 0 ms burst 0 q00001 50 sl. 0 flows (256 buckets) sched 1 weight 1 lmax 0 pri 0 droptail mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN Children flowsets: 1 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 ip 0.0.0.0/0 0.0.0.0/0 809 32614 0 0 0 00002: 890.000 Mbit/s 0 ms burst 0 q00002 50 sl. 0 flows (256 buckets) sched 2 weight 1 lmax 0 pri 0 droptail mask: 0x00 0xffffffff/0x0000 -> 0x00000000/0x0000 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN Children flowsets: 2 0 ip 0.0.0.0/0 0.0.0.0/0 1219696 1819422727 416 622680 787when uploading traffic seems to go through it
and upon downloading same thing..
00001: 510.000 Mbit/s 0 ms burst 0 q00001 50 sl. 0 flows (256 buckets) sched 1 weight 1 lmax 0 pri 0 droptail mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN Children flowsets: 1 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 ip 0.0.0.0/0 0.0.0.0/0 209511 312499033 164 245320 377 00002: 890.000 Mbit/s 0 ms burst 0 q00002 50 sl. 0 flows (256 buckets) sched 2 weight 1 lmax 0 pri 0 droptail mask: 0x00 0xffffffff/0x0000 -> 0x00000000/0x0000 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN Children flowsets: 2 0 ip 0.0.0.0/0 0.0.0.0/0 1242 50904 0 0 0In my advanced>interfaces tab after setting up the pfsense disable hardware TCP segmentation offload and Disable hardware large receive offload is ticked. is that alright, or should i be able to untick this?
-
Hi again,
I still feel like there is a bottleneck somewhere and that is why you are seeing poor performance above 500Mbit/s. However, I'm not quite sure yet where that bottleneck is, and while you do have a slower CPU (and it is ultra-low power), I'm not 100% convinced that's it.
So, we can do some troubleshooting to try to find where the bottleneck is occurring on your system. Please see this link:
https://bsdrp.net/documentation/technical_docs/performance
And go down to the section, "Where is the bottleneck?" at the bottom.
Can you try some of the tools suggested there and report back the results? I would try a test at 500Mbit with fq_codel enabled and then one at 900Mbit with fq_codel enabled to see what differences/issues might show up.
Hope this helps.
-
So i walked through the bottle neck part of the link you posted, I basically did as you suggested and ran some of the commands in idle under download and upload conditions for 500-800 and 800-800MBPS, tbh there is not a noticeable difference i can observer during this time. Ill post the results here maybe you/or someone can spot some critical issues.
top -CHIP for limits 500-800
idlelast pid: 95907; load averages: 0.29, 0.20, 0.12 up 0+18:14:12 11:40:29 445 processes: 5 running, 408 sleeping, 32 waiting CPU 0: 0.0% user, 0.0% nice, 1.2% system, 0.0% interrupt, 98.8% idle CPU 1: 0.4% user, 0.0% nice, 0.0% system, 0.0% interrupt, 99.6% idle CPU 2: 0.8% user, 0.0% nice, 0.8% system, 0.8% interrupt, 97.7% idle CPU 3: 0.0% user, 0.0% nice, 0.4% system, 0.0% interrupt, 99.6% idle Mem: 87M Active, 246M Inact, 475M Wired, 152K Buf, 3037M Free ARC: 164M Total, 22M MFU, 135M MRU, 2420K Anon, 751K Header, 4455K Other 77M Compressed, 197M Uncompressed, 2.55:1 Ratio Swap: 2048M Total, 2048M Free PID USERNAME PRI NICE SIZE RES STATE C TIME CPU COMMAND 11 root 155 ki31 0K 64K CPU1 1 18.0H 99.45% idle{idle: cpu1} 11 root 155 ki31 0K 64K RUN 2 17.9H 99.12% idle{idle: cpu2} 11 root 155 ki31 0K 64K CPU3 3 17.9H 98.78% idle{idle: cpu3} 11 root 155 ki31 0K 64K CPU0 0 17.9H 98.59% idle{idle: cpu0} 0 root -92 - 0K 4288K - 0 10:38 0.98% kernel{dummynet} 86267 root 20 0 299M 174M nanslp 2 5:58 0.64% ntopng{ntopng} 12 root -92 - 0K 512K WAIT 2 2:53 0.23% intr{irq269: igb1:que 0} 57848 root 20 0 22116K 5124K CPU2 2 0:00 0.22% top 28709 root 20 0 43056K 15236K kqread 3 6:12 0.17% haproxy 12 root -92 - 0K 512K WAIT 1 1:59 0.15% intr{irq267: igb0:que 1} 86267 root 20 0 299M 174M nanslp 2 0:42 0.11% ntopng{ntopng}download
last pid: 16016; load averages: 0.27, 0.20, 0.13 up 0+18:14:57 11:41:14 445 processes: 5 running, 408 sleeping, 32 waiting CPU 0: 0.8% user, 0.0% nice, 8.3% system, 13.0% interrupt, 78.0% idle CPU 1: 0.4% user, 0.0% nice, 0.4% system, 5.9% interrupt, 93.3% idle CPU 2: 2.4% user, 0.0% nice, 1.6% system, 7.1% interrupt, 89.0% idle CPU 3: 4.3% user, 0.0% nice, 2.8% system, 8.7% interrupt, 84.3% idle Mem: 87M Active, 246M Inact, 475M Wired, 152K Buf, 3037M Free ARC: 164M Total, 22M MFU, 135M MRU, 2424K Anon, 750K Header, 4455K Other 77M Compressed, 197M Uncompressed, 2.55:1 Ratio Swap: 2048M Total, 2048M Free PID USERNAME PRI NICE SIZE RES STATE C TIME CPU COMMAND 11 root 155 ki31 0K 64K CPU1 1 18.0H 89.47% idle{idle: cpu1} 11 root 155 ki31 0K 64K CPU2 2 18.0H 88.66% idle{idle: cpu2} 11 root 155 ki31 0K 64K RUN 3 17.9H 86.68% idle{idle: cpu3} 11 root 155 ki31 0K 64K CPU0 0 17.9H 79.79% idle{idle: cpu0} 0 root -92 - 0K 4288K - 0 10:39 10.64% kernel{dummynet} 12 root -92 - 0K 512K WAIT 0 2:03 9.20% intr{irq266: igb0:que 0} 12 root -92 - 0K 512K WAIT 1 2:00 8.86% intr{irq267: igb0:que 1} 86267 root 22 0 299M 174M bpf 0 2:06 7.04% ntopng{ntopng} 86267 root 22 0 299M 174M bpf 3 1:36 6.80% ntopng{ntopng} 12 root -92 - 0K 512K WAIT 2 2:54 6.42% intr{irq269: igb1:que 0} 12 root -92 - 0K 512K WAIT 3 2:10 5.70% intr{irq270: igb1:upload
last pid: 16016; load averages: 0.43, 0.24, 0.14 up 0+18:15:16 11:41:33 445 processes: 5 running, 408 sleeping, 32 waiting CPU 0: 5.1% user, 0.0% nice, 17.3% system, 11.0% interrupt, 66.7% idle CPU 1: 9.8% user, 0.0% nice, 1.6% system, 11.4% interrupt, 77.3% idle CPU 2: 5.1% user, 0.0% nice, 4.7% system, 8.6% interrupt, 81.6% idle CPU 3: 4.7% user, 0.0% nice, 2.4% system, 9.4% interrupt, 83.5% idle Mem: 87M Active, 246M Inact, 475M Wired, 152K Buf, 3037M Free ARC: 164M Total, 22M MFU, 135M MRU, 2424K Anon, 750K Header, 4455K Other 77M Compressed, 197M Uncompressed, 2.55:1 Ratio Swap: 2048M Total, 2048M Free PID USERNAME PRI NICE SIZE RES STATE C TIME CPU COMMAND 11 root 155 ki31 0K 64K RUN 3 18.0H 83.21% idle{idle: cpu3} 11 root 155 ki31 0K 64K CPU2 2 18.0H 82.15% idle{idle: cpu2} 11 root 155 ki31 0K 64K RUN 1 18.0H 80.52% idle{idle: cpu1} 11 root 155 ki31 0K 64K RUN 0 17.9H 68.66% idle{idle: cpu0} 0 root -92 - 0K 4288K - 3 10:42 21.03% kernel{dummynet} 86267 root 26 0 299M 174M bpf 0 2:08 11.87% ntopng{ntopng} 86267 root 25 0 299M 174M bpf 1 1:38 11.45% ntopng{ntopng}top -CHIP for limits 800-800
idlelast pid: 42737; load averages: 0.17, 0.20, 0.14 up 0+18:16:22 11:42:39 445 processes: 5 running, 408 sleeping, 32 waiting CPU 0: 0.0% user, 0.0% nice, 1.2% system, 0.0% interrupt, 98.8% idle CPU 1: 0.4% user, 0.0% nice, 0.4% system, 0.0% interrupt, 99.2% idle CPU 2: 0.0% user, 0.0% nice, 0.4% system, 0.0% interrupt, 99.6% idle CPU 3: 0.4% user, 0.0% nice, 0.4% system, 0.0% interrupt, 99.2% idle Mem: 88M Active, 247M Inact, 477M Wired, 152K Buf, 3033M Free ARC: 165M Total, 22M MFU, 135M MRU, 2424K Anon, 757K Header, 4457K Other 77M Compressed, 197M Uncompressed, 2.55:1 Ratio Swap: 2048M Total, 2048M Free PID USERNAME PRI NICE SIZE RES STATE C TIME CPU COMMAND 11 root 155 ki31 0K 64K RUN 2 18.0H 99.47% idle{idle: cpu2} 11 root 155 ki31 0K 64K CPU3 3 18.0H 99.10% idle{idle: cpu3} 11 root 155 ki31 0K 64K CPU0 0 17.9H 98.95% idle{idle: cpu0} 11 root 155 ki31 0K 64K CPU1 1 18.1H 97.41% idle{idle: cpu1} 0 root -92 - 0K 4288K - 0 10:46 0.85% kernel{dummynet} 86267 root 20 0 299M 174M nanslp 3 5:58 0.49% ntopng{ntopng}download (when it crashes)
last pid: 61056; load averages: 0.17, 0.19, 0.13 up 0+18:17:00 11:43:17 445 processes: 7 running, 408 sleeping, 30 waiting CPU 0: 4.1% user, 0.0% nice, 13.5% system, 16.5% interrupt, 65.9% idle CPU 1: 2.2% user, 0.0% nice, 0.4% system, 13.9% interrupt, 83.5% idle CPU 2: 4.5% user, 0.0% nice, 5.2% system, 9.4% interrupt, 80.9% idle CPU 3: 7.1% user, 0.0% nice, 4.5% system, 14.2% interrupt, 74.2% idle Mem: 87M Active, 247M Inact, 484M Wired, 152K Buf, 3027M Free ARC: 165M Total, 22M MFU, 135M MRU, 2268K Anon, 756K Header, 4458K Other 77M Compressed, 197M Uncompressed, 2.55:1 Ratio Swap: 2048M Total, 2048M Free PID USERNAME PRI NICE SIZE RES STATE C TIME CPU COMMAND 11 root 155 ki31 0K 64K RUN 1 18.1H 83.40% idle{idle: cpu1} 11 root 155 ki31 0K 64K RUN 2 18.0H 82.88% idle{idle: cpu2} 11 root 155 ki31 0K 64K CPU3 3 18.0H 79.58% idle{idle: cpu3} 11 root 155 ki31 0K 64K CPU0 0 17.9H 69.30% idle{idle: cpu0} 0 root -92 - 0K 4288K - 2 10:47 16.37% kernel{dummynet} 12 root -92 - 0K 512K CPU0 0 2:07 14.98% intr{irq266: igb0:que 0} 12 root -92 - 0K 512K WAIT 1 2:04 12.14% intr{irq267: igb0:que 1}upload totally fine
ast pid: 61192; load averages: 0.20, 0.20, 0.14 up 0+18:17:17 11:43:34 445 processes: 5 running, 408 sleeping, 32 waiting CPU 0: 3.1% user, 0.0% nice, 18.5% system, 14.2% interrupt, 64.2% idle CPU 1: 6.7% user, 0.0% nice, 1.2% system, 11.0% interrupt, 81.1% idle CPU 2: 3.1% user, 0.0% nice, 2.4% system, 7.1% interrupt, 87.4% idle CPU 3: 5.5% user, 0.0% nice, 3.9% system, 8.3% interrupt, 82.3% idle Mem: 87M Active, 247M Inact, 484M Wired, 152K Buf, 3027M Free ARC: 165M Total, 22M MFU, 135M MRU, 2436K Anon, 756K Header, 4458K Other 77M Compressed, 197M Uncompressed, 2.55:1 Ratio Swap: 2048M Total, 2048M Free PID USERNAME PRI NICE SIZE RES STATE C TIME CPU COMMAND 11 root 155 ki31 0K 64K RUN 3 18.0H 82.81% idle{idle: cpu3} 11 root 155 ki31 0K 64K CPU2 2 18.0H 82.50% idle{idle: cpu2} 11 root 155 ki31 0K 64K CPU1 1 18.1H 81.87% idle{idle: cpu1} 11 root 155 ki31 0K 64K CPU0 0 17.9H 67.74% idle{idle: cpu0} 0 root -92 - 0K 4288K - 0 10:49 21.24% kernel{dummynet} 12 root -92 - 0K 512K WAIT 0 2:09 11.88% intr{irq266: igb0:que 0} 86267 root 24 0 299M 174M bpf 2 2:11 11.66% ntopng{ntopng} 86267 root 24 0 299M 174M bpf 2 1:41 11.33% ntopng{ntopng} 12 root -92 - 0K 512K WAIT 1 2:06 9.44% intr{irq267:MBUFF
[2.4.3-RC][admin@inferno.zwck.lan]/root: vmstat -z | head -1 ; vmstat -z | grep -i mbuf #idle ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP mbuf_packet: 256, 1574895, 16790, 6992,74675913, 0, 0 mbuf: 256, 1574895, 412, 2371,77540344, 0, 0 mbuf_cluster: 2048, 1000000, 23782, 6, 23782, 0, 0 mbuf_jumbo_page: 4096, 123038, 0, 698, 4288582, 0, 0 mbuf_jumbo_9k: 9216, 36455, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 20506, 0, 0, 0, 0, 0 download [2.4.3-RC][admin@inferno.zwck.lan]/root: vmstat -z | head -1 ; vmstat -z | grep -i mbuf ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP mbuf_packet: 256, 1574895, 16790, 6992,75836656, 0, 0 mbuf: 256, 1574895, 412, 2371,78304908, 0, 0 mbuf_cluster: 2048, 1000000, 23782, 6, 23782, 0, 0 mbuf_jumbo_page: 4096, 123038, 0, 698, 4288590, 0, 0 mbuf_jumbo_9k: 9216, 36455, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 20506, 0, 0, 0, 0, 0 [2.4.3-RC][admin@inferno.zwck.lan]/root: vmstat -z | head -1 ; vmstat -z | grep -i mbuf ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP mbuf_packet: 256, 1574895, 16790, 6992,75836986, 0, 0 mbuf: 256, 1574895, 412, 2371,78305106, 0, 0 mbuf_cluster: 2048, 1000000, 23782, 6, 23782, 0, 0 mbuf_jumbo_page: 4096, 123038, 0, 698, 4288590, 0, 0 mbuf_jumbo_9k: 9216, 36455, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 20506, 0, 0, 0, 0, 0 upload [2.4.3-RC][admin@inferno.zwck.lan]/root: vmstat -z | head -1 ; vmstat -z | grep -i mbuf ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP mbuf_packet: 256, 1574895, 17065, 6717,76391497, 0, 0 mbuf: 256, 1574895, 412, 2371,78486631, 0, 0 mbuf_cluster: 2048, 1000000, 23782, 6, 23782, 0, 0 mbuf_jumbo_page: 4096, 123038, 0, 698, 4288590, 0, 0 mbuf_jumbo_9k: 9216, 36455, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 20506, 0, 0, 0, 0, 0 [2.4.3-RC][admin@inferno.zwck.lan]/root: vmstat -z | head -1 ; vmstat -z | grep -i mbuf ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP mbuf_packet: 256, 1574895, 17108, 6674,77061386, 0, 0 mbuf: 256, 1574895, 412, 2371,78718590, 0, 0 mbuf_cluster: 2048, 1000000, 23782, 6, 23782, 0, 0 mbuf_jumbo_page: 4096, 123038, 0, 698, 4288590, 0, 0 mbuf_jumbo_9k: 9216, 36455, 0, 0, 0, 0, 0 mbuf_jumbo_16k: 16384, 20506, 0, 0, 0, 0, 0irqs for each que and nic looks good…
[2.4.3-RC][admin@inferno.zwck.lan]/root: vmstat -i interrupt total rate irq4: uart0 224 0 irq23: ehci0 137353 2 cpu0:timer 71932468 1048 cpu1:timer 18262419 266 cpu2:timer 25164107 367 cpu3:timer 19066067 278 irq265: hdac0 35 0 irq266: igb0:que 0 10573218 154 irq267: igb0:que 1 6880452 100 irq268: igb0:link 5 0 irq269: igb1:que 0 18310531 267 irq270: igb1:que 1 6241534 91 irq271: igb1:link 4 0 irq272: igb2:que 0 32 0 irq273: igb2:que 1 32 0 irq274: igb2:link 2 0 irq275: igb3:que 0 68643 1 irq276: igb3:que 1 68643 1 irq277: igb3:link 1 0 irq278: ahci0 819394 12 Total 177525164 2586 [2.4.3-RC][admin@inferno.zwck.lan]/root: vmstat -iNetstat
-
Okay.
What i am about to write is probably going to frustrate you, but i think it might not be the issue of pfsense but instead a browser issue.
I have performed the test with 950 950
chrome (dll crashes upload performs perfect)
firefox (ddl performs perfect upload performs perfect (bufferbloat sure, but not to the extend of what chrome does)
edge (just crashes)tbh, i am not sure if this is normal.
-
Very interesting - looks like we found bottleneck then! :)
I have observed slightly better performance in Firefox than Chrome during speed tests, but things never crashed like they did for you. I have a few questions:
- Are you using the 64-bit version of Chrome?
- How many add-on's/plugins are you using? If you disable them all and run the speed test do the results improve?
- What are the hw specs of the system you are running the tests on?
- On that system, have you tweaked the NIC parameters at all (e.g. RSS queues, TX/RX transmit sizes, etc.)?
-
This is a win 10 64 bit system
CPU is a 6700K at 4.6Ghz, 32gig ram, asus mb that houses an intel 219v and an i211 network card (i tried both).
Chrome Version 65.0.3325.181 (Official Build) (64-bit) i used chrome in incognito which disables all plugins.I am wondering if firefox gives me an accurate results, as i mentioned in one of the earlier post from me, if i have WFQ2+ chrome just deals with the current inflow of data and has 0 problem.
By now i have tuned some parameters of my network cards on the pfsense system following, some articles around the net.
hw.igb.txd="4096" hw.igb.rxd="4096" net.link.ifqmaxlen="8192" hw.igb.num_queues="0" hw.igb.enable_aim="1" hw.igb.max_interrupt_rate="64000" kern.ipc.nmbclusters="1000000" hw.igb.enable_msix="1" hw.pci.enable_msix="1" hw.igb.tx_process_limit="-1" hw.igb.rx_process_limit="-1"Plus some system tuneables
-
I am confused however, if its really a chrome problem why should chrome work when i turn off traffic shaping?
-
Actually, I was curious if you changed the network card parameters at all on the system that you are testing on (i.e. changing the buffer sizes, receive queues, etc.)?
Also, can you try one thing for me: If you re-enable flow control on the NIC's on your pfSense box - do you see any difference in speed / do you still experience issues with Chrome? Going the other way, if you leave flow control disabled on your pfSense box and disable it on your system's NIC, do you see any difference?
Hope this helps.
-
heya tman222,
i tried all possible combination of flowcontrol enabled and disabled everytime with the same results, i.e. the download "crashes" when using dsl reports. When i deactivate qa_codle and use whq+ everything runs fine except with bufferblaot but at most (300ms)
-
heya tman222,
any more suggestions, or things i could try :D
-
I got no solutions but am watching this with interest.
Some may remember I was also hitting some kind of unknown bottleneck when testing with steam downloads, I was getting packetloss rather than increased latency simply from downloading steam games (and I still do but not as bad), no obvious bottleneck in my case either, plenty of free hardware resources during the downloads.
For me dummynet is superior to fairq+codel on ALTQ for upstream latency under load very visibly, its a clear advantage. HSFC on altq for downstream I think beats fq_codel a bit but its way more complex to configure and can misbehave so right now I use fq_codel both ways in the simple one queue configuration, I do have the tuned values for interaction such as smaller quantum.
-
heya tman222,
any more suggestions, or things i could try :D
Hi - I thought about this a little bit more and have a couple more questions for you:
-
On the system your testing the connection speed with (i.e. the PC whose specs you listed earlier in the thread), have you made any changes to the network card settings/parameters? For instance, have you changed the receive or transmit buffers, the number of RSS queues etc.?
-
Have you tried running an iperf3 test between your PC and the router to see how the transfer speed behaves with fq_codel enabled and disabled? I'm actually curious if the slowdown that you experienced is just related to one particular web browser (e.g. Chrome) and online speed test, or if there are any other more general issues with your pfSense box or the PC you are testing with. What I would do is install iperf3 on both the pfSense box and and on your PC and then run a test between them (i.e. make the pfSense box the server and the PC the client). I would run two types of tests:
a. With fq_codel enabled (i.e. with the queues enabled on your LAN firewall rule). Run the test both forward and reverse (i.e. pfSense box receiving traffic and pfSense box sending traffic).
b. With fq_codel disabled (i.e. with the queues not enabled on your LAN firewall rule). Here again, run the test both forward and reverse (i.e. pfSense box receiving traffic and pfSense box sending traffic).
Without fq_codel enabled you should be able to easily max out the transfer speed between both PC and pfSense box at around 945 - 950Mbit/s. However, what happens when you run the test with fq_codel enabled on the LAN interface? Do you see a significant slow down start to occur over time (maybe run the iperf3 test for 30s to 60s to see if any slow down occurs in that time)?
Hope this helps.
-
-
So, just before i went to work i quickly checked my current configuration, here is a quick recap:
fq_codle on download limited to 509Mbit/s and Upload limited to 809Mbit/s
[2.4.3-RELEASE][admin@inferno.zwck.lan]/root: ipfw sched show 00001: 509.000 Mbit/s 0 ms burst 0 q00001 50 sl. 0 flows (256 buckets) sched 1 weight 1 lmax 0 pri 0 droptail mask: 0x00 0x00000000/0x0000 -> 0xffffffff/0x0000 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN Children flowsets: 1 BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp 0 ip 0.0.0.0/0 0.0.0.0/0 1 94 0 0 0 00002: 809.000 Mbit/s 0 ms burst 0 q00002 50 sl. 0 flows (256 buckets) sched 2 weight 1 lmax 0 pri 0 droptail mask: 0x00 0xffffffff/0x0000 -> 0x00000000/0x0000 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN Children flowsets: 2 0 ip 0.0.0.0/0 0.0.0.0/0 1 40 0 0 0I started a iperf3 server on my clients (ubuntu server and windows10 pc) and tested a simple parallel stream. I write it detailed so you can check what my thought process was.
Windows10: iperf3 -s
ubuntu16: iperf3 -sPfsense: iperf3 -c 192.168.0.177 -t 10 -P 10
Results:[SUM] 0.00-10.18 sec 1.08 GBytes 910 Mbits/sec 28947 sender [SUM] 0.00-10.18 sec 1.07 GBytes 907 Mbits/sec receiverPfsense: iperf3 -c 192.168.0.197 -t 10 -P 10
Results:[SUM] 0.00-10.18 sec 1.08 GBytes 910 Mbits/sec 28947 sender [SUM] 0.00-10.18 sec 1.07 GBytes 907 Mbits/sec receiverHowever: when i simply switch the roles it wont work.
Windows10: iperf3 -c 192.168.0.1 -t 10 -P 10
ubuntu16: iperf3 -c 192.168.0.1 -t 10 -P 10Pfsense: iperf3 -s
The test will fail, iperf will establish the connection but nothing happens. I am not sure why this is. if i iperf through the pfsense to a different pc i get full speeds all the directions.
However when i reduce the parallel streams below 10 i.e 9 works and the out put is:
Windows10 to pfsense iperf3 -c 192.168.0.1 -t 10 -P 9
[SUM] 0.00-10.00 sec 933 MBytes 783 Mbits/sec 1520 sender [SUM] 0.00-10.00 sec 929 MBytes 780 Mbits/sec receiverubuntu to pfsense iperf3 -c 192.168.0.1 -t 10 -P 9
[SUM] 0.00-10.00 sec 936 MBytes 786 Mbits/sec sender [SUM] 0.00-10.00 sec 935 MBytes 784 Mbits/sec receiverThis was just the first test, yet it shows some weird behavior that i can not really explain.
-
1) On the system your testing the connection speed with (i.e. the PC whose specs you listed earlier in the thread), have you made any changes to the network card settings/parameters? For instance, have you changed the receive or transmit buffers, the number of RSS queues etc.?
Nope just normal windows settings after installing windows, its an i235, with the standard parameters
2) Have you tried running an iperf3 test between your PC and the router to see how the transfer speed behaves with fq_codel enabled and disabled?
a. With fq_codel enabled (i.e. with the queues enabled on your LAN firewall rule). Run the test both forward and reverse (i.e. pfSense box receiving traffic and pfSense box sending traffic).
I have run the iperf tests between pfsense and two different clients (all with intel nics, one ubuntu and one windows pc)
here are the results https://imgur.com/a/fcITh
Things i have noticed, when i perform the test from the pfsense box to either client the performance is not consistent. i.e. some test perfrom with line/traffic shaping speed, some perform way under between 80Mbits and 200Mbits.this has been fixed after i took out the tuning parameters.b. With fq_codel disabled (i.e. with the queues not enabled on your LAN firewall rule). Here again, run the test both forward and reverse (i.e. pfSense box receiving traffic and pfSense box sending traffic).
https://imgur.com/a/U62rsnot sure how to figure out what is wrongly configured after this point.
I reversed my NIC tuning that i posted and since then the iperf3 test works (expect iperf3 -P>9), as soon as i activate qa_coddle same problems as before.