Best way to reduce game latency

chrcoluk · Apr 10, 2018, 6:53 PM

I got no solutions but am watching this with interest.

Some may remember I was also hitting some kind of unknown bottleneck when testing with steam downloads, I was getting packetloss rather than increased latency simply from downloading steam games (and I still do but not as bad), no obvious bottleneck in my case either, plenty of free hardware resources during the downloads.

For me dummynet is superior to fairq+codel on ALTQ for upstream latency under load very visibly, its a clear advantage. HSFC on altq for downstream I think beats fq_codel a bit but its way more complex to configure and can misbehave so right now I use fq_codel both ways in the simple one queue configuration, I do have the tuned values for interaction such as smaller quantum.

tman222 · Apr 12, 2018, 12:52 AM

@zwck:

heya tman222,

any more suggestions, or things i could try :D

Hi - I thought about this a little bit more and have a couple more questions for you:

On the system your testing the connection speed with (i.e. the PC whose specs you listed earlier in the thread), have you made any changes to the network card settings/parameters? For instance, have you changed the receive or transmit buffers, the number of RSS queues etc.?
Have you tried running an iperf3 test between your PC and the router to see how the transfer speed behaves with fq_codel enabled and disabled? I'm actually curious if the slowdown that you experienced is just related to one particular web browser (e.g. Chrome) and online speed test, or if there are any other more general issues with your pfSense box or the PC you are testing with. What I would do is install iperf3 on both the pfSense box and and on your PC and then run a test between them (i.e. make the pfSense box the server and the PC the client). I would run two types of tests:
a. With fq_codel enabled (i.e. with the queues enabled on your LAN firewall rule). Run the test both forward and reverse (i.e. pfSense box receiving traffic and pfSense box sending traffic).
b. With fq_codel disabled (i.e. with the queues not enabled on your LAN firewall rule). Here again, run the test both forward and reverse (i.e. pfSense box receiving traffic and pfSense box sending traffic).

Without fq_codel enabled you should be able to easily max out the transfer speed between both PC and pfSense box at around 945 - 950Mbit/s. However, what happens when you run the test with fq_codel enabled on the LAN interface? Do you see a significant slow down start to occur over time (maybe run the iperf3 test for 30s to 60s to see if any slow down occurs in that time)?

Hope this helps.

zwck · Apr 12, 2018, 10:48 PM

So, just before i went to work i quickly checked my current configuration, here is a quick recap:

fq_codle on download limited to 509Mbit/s and Upload limited to 809Mbit/s


[2.4.3-RELEASE][admin@inferno.zwck.lan]/root: ipfw sched show
00001: 509.000 Mbit/s    0 ms burst 0
q00001  50 sl. 0 flows (256 buckets) sched 1 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
 sched 1 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 ip           0.0.0.0/0             0.0.0.0/0        1       94  0    0   0
00002: 809.000 Mbit/s    0 ms burst 0
q00002  50 sl. 0 flows (256 buckets) sched 2 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000
 sched 2 type FQ_CODEL flags 0x0 0 buckets 1 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2
  0 ip           0.0.0.0/0             0.0.0.0/0        1       40  0    0   0

I started a iperf3 server on my clients (ubuntu server and windows10 pc) and tested a simple parallel stream. I write it detailed so you can check what my thought process was.

Windows10: iperf3 -s
ubuntu16: iperf3 -s

Pfsense: iperf3 -c 192.168.0.177 -t 10 -P 10
Results:

[SUM]   0.00-10.18  sec  1.08 GBytes   910 Mbits/sec  28947             sender
[SUM]   0.00-10.18  sec  1.07 GBytes   907 Mbits/sec                  receiver

Pfsense: iperf3 -c 192.168.0.197 -t 10 -P 10
Results:

[SUM]   0.00-10.18  sec  1.08 GBytes   910 Mbits/sec  28947             sender
[SUM]   0.00-10.18  sec  1.07 GBytes   907 Mbits/sec                  receiver

However: when i simply switch the roles it wont work.

Windows10: iperf3 -c 192.168.0.1 -t 10 -P 10
ubuntu16: iperf3 -c 192.168.0.1 -t 10 -P 10

Pfsense: iperf3 -s

The test will fail, iperf will establish the connection but nothing happens. I am not sure why this is. if i iperf through the pfsense to a different pc i get full speeds all the directions.

However when i reduce the parallel streams below 10 i.e 9 works and the out put is:

Windows10 to pfsense iperf3 -c 192.168.0.1 -t 10 -P 9


[SUM]   0.00-10.00  sec   933 MBytes   783 Mbits/sec  1520             sender
[SUM]   0.00-10.00  sec   929 MBytes   780 Mbits/sec                  receiver

ubuntu to pfsense iperf3 -c 192.168.0.1 -t 10 -P 9


[SUM]   0.00-10.00  sec   936 MBytes   786 Mbits/sec                  sender
[SUM]   0.00-10.00  sec   935 MBytes   784 Mbits/sec                  receiver

This was just the first test, yet it shows some weird behavior that i can not really explain.

zwck · Apr 17, 2018, 1:08 PM

1) On the system your testing the connection speed with (i.e. the PC whose specs you listed earlier in the thread), have you made any changes to the network card settings/parameters? For instance, have you changed the receive or transmit buffers, the number of RSS queues etc.?

Nope just normal windows settings after installing windows, its an i235, with the standard parameters

2) Have you tried running an iperf3 test between your PC and the router to see how the transfer speed behaves with fq_codel enabled and disabled?
a. With fq_codel enabled (i.e. with the queues enabled on your LAN firewall rule). Run the test both forward and reverse (i.e. pfSense box receiving traffic and pfSense box sending traffic).
I have run the iperf tests between pfsense and two different clients (all with intel nics, one ubuntu and one windows pc)
here are the results https://imgur.com/a/fcITh
Things i have noticed, when i perform the test from the pfsense box to either client the performance is not consistent. i.e. some test perfrom with line/traffic shaping speed, some perform way under between 80Mbits and 200Mbits. this has been fixed after i took out the tuning parameters.

b. With fq_codel disabled (i.e. with the queues not enabled on your LAN firewall rule). Here again, run the test both forward and reverse (i.e. pfSense box receiving traffic and pfSense box sending traffic).
https://imgur.com/a/U62rs

not sure how to figure out what is wrongly configured after this point.

I reversed my NIC tuning that i posted and since then the iperf3 test works (expect iperf3 -P>9), as soon as i activate qa_coddle same problems as before.

tman222 · Apr 17, 2018, 10:11 PM

Hi,

So these are interesting results. However, I'm not quite sure with all the screenshots what is what, so let's simplify this a bit. Can you run these two tests for me:

Test 1: Settings:

Disable FQ_Codel on pfSense
2) Start iperf3 server on pfSense, i.e. iperf3 -s
3) Use the Windows PC as the iperf3 client machine.

Test 1: Test Runs:
a) From Windows PC CLI: iperf3 -c pfSense_IP -P 4 -t 30
b) From Windows PC CLI: iperf3 -c pfSense_IP -P 4 -t 30 -R (-R reverses server and client roles).

I would expect to see the interface closed to maxed out on Test 1, i.e. 940-950Mbit for 1a) and 1b, especially since we are running multiple streams).

–---------------------------------------------------

Test 2: Settings:
1) Enable FQ_Codel with symmetric up down and down limits, e.g. 900/900 or 940/940, etc.
2) Start iperf3 server on pfSense, i.e. iperf3 -s
3) Use the Windows PC as the iperf3 client machine:

Test 2: Test Runs:
a) From Windows PC CLI: iperf3 -c pfSense_IP -P 4 -t 30
b) From Windows PC CLI: iperf3 -c pfSense_IP -P 4 -t 30 -R (-R reverses server and client roles).

What do speeds look like for 2a) and 2b) in this case and are they consistent over the 30 second testing interval?

–----------------------------------------------------

I could not find anything on i235 - are you sure this is the right chipset? Is this an Intel NIC?

Also, are you running any additional security/firewall software on the Windows PC?

EDIT: Something else that came to mind: In pfSense dashboard: System/Advanced/Miscellaneous do you have power savings enabled? If yes, what are your settings?

Hope this helps.

zwck · Apr 18, 2018, 9:36 AM

Heya Tman222,

Thanks again for helping me so much.

The easy questions first:
- I have 3 intel nics, 2 onboard (i219V, i211) and a pcie i210-T1, i have tried all 3 of them with pretty much the same result. (dunno where i got the 235)
- I have Enable PowerD (ticked on) and all 3 to Hiadaptive.

no windows firewall
no tuning parameters

**Test 1: Test Runs:
a) From Windows PC CLI: iperf3 -c pfSense_IP -P 4 -t 30
https://i.imgur.com/SRdKCGH.jpg

b) From Windows PC CLI: iperf3 -c pfSense_IP -P 4 -t 30 -R (-R reverses server and client roles).**
https://i.imgur.com/OJ1Nr82.jpg

Consistent speeds in both direction perfectly fine. (when i can go only up to -P 9 when P>9 the client to pfsense communication can not be established)

After activating qa_codle and selecting in and out pipe for LAN interface Rules:
ipfw sched show

00001: 940.000 Mbit/s    0 ms burst 0
q00001  50 sl. 0 flows (256 buckets) sched 1 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0x00000000/0x0000 -> 0xffffffff/0x0000
 sched 1 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 1
00002: 940.000 Mbit/s    0 ms burst 0
q00002  50 sl. 0 flows (256 buckets) sched 2 weight 1 lmax 0 pri 0 droptail
    mask:  0x00 0xffffffff/0x0000 -> 0x00000000/0x0000
 sched 2 type FQ_CODEL flags 0x0 0 buckets 0 active
 FQ_CODEL target 5ms interval 100ms quantum 1514 limit 10240 flows 1024 ECN
   Children flowsets: 2

**Test 2: Test Runs:
a) From Windows PC CLI: iperf3 -c pfSense_IP -P 4 -t 30
(super consistant ~844)
https://i.imgur.com/VVcniSO.jpg

b) From Windows PC CLI: iperf3 -c pfSense_IP -P 4 -t 30 -R (-R reverses server and client roles).
https://i.imgur.com/oy708eS.jpg (super consistant 914(1))**

all these test were done with no tuning parameters in loader.conf.local i played quite a bit with all sorts of settings but didnt notice any change in any of the test i have performed.

anything else i could check? ram, ssd, or zfs?

tman222 · Apr 21, 2018, 6:49 PM

Hi again,

Sorry for the slight delay.

These iperf3 results don't look too bad at all - so maybe this is pointing back to a software related issue (Chrome?) on the machine you are testing with. I have a couple other questions for you:

Keeping your limits at 940/940:

Do you only see the issue with the download instability on just the DSL Reports speed test or does it occur on other speedtests (e.g. speedtest.net) as well?
Do you only see the issue with the download instability in Windows, or can you duplicate in Linux box as well (e.g. Chrome/Firefox in Windows vs. Chrome/Firefox in Linux)?

Also, I'm starting to wonder if you even need download shaping since that's usually not source of bufferbloat. If you enable fq_codel for just the upload side, what do your speed test and bufferbloat results look like?

Hope this helps.

zwck · Apr 21, 2018, 8:43 PM

hi hi, no worries,

These iperf3 results don't look too bad at all - so maybe this is pointing back to a software related issue (Chrome?) on the machine you are testing with. I have a couple other questions for you:

The problem i have with believing that it is a chrome issue is that, when i just turn qa coddle off, i have full dll and up speed, sure there is bufferbloat, but it works totaly fine. all the other speed tests are always showing full up and download, with and without qacodle.

1) Do you only see the issue with the download instability on just the DSL Reports speed test or does it occur on other speedtests (e.g. speedtest.net) as well?

all the other speedtests show full download speed.

2) Do you only see the issue with the download instability in Windows, or can you duplicate in Linux box as well (e.g. Chrome/Firefox in Windows vs. Chrome/Firefox in Linux)?

i tried the same test with a ubuntu desktop same thing.

Also, I'm starting to wonder if you even need download shaping since that's usually not source of bufferbloat. If you enable fq_codel for just the upload side, what do your speed test and bufferbloat results look like?

at the moment i just have fq_codel on the in pipe on the lan rule and leave the out rule on none. so i get like decent results.

best results for me is with

Download
ipfw sched 1 config pipe 1 type WF2Q+

Upload
ipfw sched 2 config pipe 2 type fq_codel target 3 interval 60

Thanks again for your help.

zwck · May 30, 2018, 8:38 AM

could this be a ram problem ? that i can not handle download with fq_codle?

zwck · Jun 14, 2018, 8:41 PM

So after reading the latest posts in the other coddle thread, and if someone else has the same/similar issues. My reason for having shaping issues with fq_codle (downloads crashing) was promiscuous mode on the NIC, this automatically activates when ntopng package is installed (duh). Turning that package off leads to everything running as expected.

@tman222 Thanks again for your help.

tman222 · Jun 30, 2018, 6:17 PM

@zwck said in Best way to reduce game latency:

So after reading the latest posts in the other coddle thread, and if someone else has the same/similar issues. My reason for having shaping issues with fq_codle (downloads crashing) was promiscuous mode on the NIC, this automatically activates when ntopng package is installed (duh). Turning that package off leads to everything running as expected.

@tman222 Thanks again for your help.

Hi @zwck - great to hear you got things up and running! I'm a little surprised that this was the problem. I also use ntopng on my pfSense box and have not experienced these issues. Maybe it's a difference in the hardware we are using? In any case, thanks for following up and sharing what solved the issues for you.

tyluerdisy · May 12, 2021, 3:40 AM

This post is deleted!