Best way to reduce game latency

necroxix

I have similar needs, except I'm trying to make a second set of queues for my twitch stream traffic so that I don't drop RTMP packets but the weight settings don't seem to do anything different. If I stream and do a speedtest, a consistent number of stream packets get dropped. Chances are, I got something misconfigured but it's at least easy for me to reproduce the test to know if I get it working, so if I figure it out I'll chime in because if I did the same with my game traffic, it wouldn't work any differently.

zwck

I have some "issues" with the download queue. So i'd like to tell you what i have done so far.

I am on PfSense 2.4.2_1 and i have a symetrical 1000Mbit line and DSL reports
image before is attached. And My dsl report looks like this, as expected (image_1…)

Creating Limiters (screenshots attached for the upload Part, for the download part its the same but with a different name)

Upload (limited to 900Mbit)
highUp 75
defaultUp 25
lowUp 5
Download (limited to 900Mbit)
HighDown
defaultDown
lowDown

Creating Floating rules Rules
I created in total 6 Floating rules but only going to show the default ones in the screenshots
the other ones are basically clones anyway
Installing the shellcmd package and adding
ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codel
horrible results, something is not working right on the download side, dunno what it is :D

I added an imgur album to just take a look at all the screenshots. https://imgur.com/a/bkIuA maybe @tman has an idea what i am doing :(

tman222

@zwck:

I have some "issues" with the download queue. So i'd like to tell you what i have done so far.

I am on PfSense 2.4.2_1 and i have a symetrical 1000Mbit line and DSL reports
image before is attached. And My dsl report looks like this, as expected (image_1…)

Creating Limiters (screenshots attached for the upload Part, for the download part its the same but with a different name)

Upload (limited to 900Mbit)

highUp 75

defaultUp 25

lowUp 5

Download (limited to 900Mbit)

HighDown

defaultDown

lowDown

Creating Floating rules Rules
I created in total 6 Floating rules but only going to show the default ones in the screenshots
the other ones are basically clones anyway

Installing the shellcmd package and adding
ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codel

horrible results, something is not working right on the download side, dunno what it is :D

I added an imgur album to just take a look at all the screenshots. https://imgur.com/a/bkIuA maybe @tman has an idea what i am doing :(

I have not set this up with matching floating rules before, but one thing I noticed right away looking at your screenshots is that you are missing the source and destination masks in your upload and download queues.

For each of your download queues, choose "Destination addresses" for the Mask. For each of your upload queues, choose "Source addresses" for the Mask.

Hope this helps.

Harvy66

fq_codel is great at reducing latency on its own. Adding complexity by having more queues may actually make it worse. Of course not in relation to the issue you're seeing.

zwck

@tman222:

stuff

not sure what you mean here, would you mind sending me some screenshots or uploading them here, i thought the floating rules were necessary. I just added for my upload limiters source and for my download limiters destination with the same results :(

zwck

@Harvy66:

fq_codel is great at reducing latency on its own. Adding complexity by having more queues may actually make it worse. Of course not in relation to the issue you're seeing.

What would be the easiest setup here? i dont mind not dealing with queues :D

tman222

@zwck:

@tman222:

stuff

not sure what you mean here, would you mind sending me some screenshots or uploading them here, i thought the floating rules were necessary. I just added for my upload limiters source and for my download limiters destination with the same results :(

Actually the most basic setup requires only an upload and download limiter with one queue under each, and no matching firewall rules.

Here's how you would set that up:

First, remove your existing settings including your matching firewall rules you created for fq_codel.

Next:
1) Create a upload and download limiter and set their bandwidth limits
2) Create one queue under the Upload limiter, i.e. in your case let's call this "in" and make sure the Mask field is set to "Source Addresses". Leave the Weight field empty.
3) Create one queue under the Download limiter, i.e. in your case let's call this "out" and make sure the Mask field is set to "Destination Addresses". Leave the Weight field empty.
4) Next go to your LAN interface and find the rule that allows outbound traffic to the internet (e.g. your default allow all rule). Under that rules' settings, go to Advanced Options, In/Out Pipe.
5) For the In Pipe use the queue you created under the upload limiter, in your case the "in" queue.
6) For the Out Pipe use the queue you created under the download limiter, in your case the "out" queue.
7) Enable fq_codel with this command: ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codel
8 ) Speed test and check for buffer bloat.

Harvy66 is right that fq_codel is pretty good at reducing latency without having to filter traffic into different queues first and then applying fq_codel. In my case I'm only using multiple weighted queues to control the total amount of bandwidth available to different VLAN's instead of controlling the amount of bandwidth available to different traffic on the same interface/VLAN. That may still be possible to do (e.g. with matching firewall rules), but unfortunately I have don't have any specific experience with such a setup.

Hope this helps.

zwck

I must be doing something wrong.

uploadin.JPG_thumb

uploadout.JPG_thumb

lanrule.JPG_thumb

new.JPG_thumb

tman222

@zwck:

I must be doing something wrong.

Try this:

On the command line issue this command: ipfw pipe flush
Then go ahead and reset your firewall states.
Then issue this command on the command line: ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codel
Try another speed test.

What do the results look like now?

Hope this helps.

zwck

@tman222:

@zwck:

I must be doing something wrong.

Try this:

On the command line issue this command: ipfw pipe flush

Then go ahead and reset your firewall states.

Then issue this command on the command line: ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codel

Try another speed test.

What do the results look like now?

Hope this helps.

First off thank you for helping me! That's really great! Unfortunately this did change the outcome significantely, or at least i get the same result.

Could there be anything else besides the trafficshaper that influences this, whats surprising to me is that the upload part of the speedtest just works flawlessly, no bufferbloat and constant high throughput, its only that the download really is not working well, when i remove the traffic shaper its the opposite ?

zwck

after updating to 2.4.3 no change

tman222

@zwck:

after updating to 2.4.3 no change

Something still seems off here. Do you have any other firewall rules (floating or otherwise) or traffic shaping settings enabled that are impacting traffic coming to or from your LAN and/or WAN? Besides setting up the limiters and queues, are there any other changes you made to try to implement fq_codel that you might have forgotten to undo? Can you provide screenshots again so we can see if anything does not look correct? Also, what happens if you raise the limiters to 930 or 940Mbit? Any difference?

Hope this helps.

zwck

Hey tman222,

So, i have some port forwarding rules activated for some services on some other machines, but other than that nothing really. I put as you suggested the in and out pipe on the lan rule instead of creating floating rules, and deactivated/deleted all the other rules I had on. When i'll come home from work i'll upload some screenshots/ or some video. Maybe there is something obviously wrong and i am just too much of a beginner. Thanks again for all the help and effort you put into my problems.

zwck

Hey tman222,

So i basically here are all my settings regarding firewalling and limiters. Could i have messed something up with nat or dns, that could cause a problem like this?

https://imgur.com/a/5z4zM

Edit: Update:
When i limit the download to 500Mbit, i dont get any buffer-bloat as soon as I go above if feels like the download just crashes… any suggestions are welcome.

tman222

@zwck:

Hey tman222,

So i basically here are all my settings regarding firewalling and limiters. Could i have messed something up with nat or dns, that could cause a problem like this?

https://imgur.com/a/5z4zM

Edit: Update:
When i limit the download to 500Mbit, i dont get any buffer-bloat as soon as I go above if feels like the download just crashes… any suggestions are welcome.

The only thing I see right now in those WAN rules that I'm a little suspicious of are the two haproxy rules that pass HTTP/HTTPS traffic on port 80 and 443. What does this NAT redirect do exactly? If you disable those two rules temporarily does it make a difference?

Also, are you running any IDS/IPS (e.g. Snort) on your interfaces? If so, if you disable that, do you see any improvement?

What are the hardware specs of your pfSense box?

Hope this helps.

zwck

Hi,

the ha proxy rules direct incoming traffic on port 80 and 443 to the internal haproxy, to direct to my personal blog and a speed test, https://speed.zwck.de so nothing critical. However, if i disable the haproxy rules the results are the same. I also dont have a snort running.

My system is an older i5 system with 4GB ram and 4 intel nics, i am thinking maybe something is setup wrongly in the general setup. maybe dns ? i really have no idea.

The thing is if i flush the pipe ;) (ipfw pipe flush and reload the filters) the sched resetsto WF2Q+ of course, when i now perform the dlsreport speed tests the speeds are to be expected 900Mbits, quite constant, and with limited bufferbloat. However, when i have qa_coddle on the download just crashes hardcore, it goes up to 900 then stops (bufferbloat 35 seconds) then drops to 40Mbit and avg of 350 or so. its really weird. I checked my cpu performance and states and all, but nothing seems to bottle neck this.

tman222

@zwck:

Hi,

the ha proxy rules direct incoming traffic on port 80 and 443 to the internal haproxy, to direct to my personal blog and a speed test, https://speed.zwck.de so nothing critical. However, if i disable the haproxy rules the results are the same. I also dont have a snort running.

My system is an older i5 system with 4GB ram and 4 intel nics, i am thinking maybe something is setup wrongly in the general setup. maybe dns ? i really have no idea.

The thing is if i flush the pipe ;) (ipfw pipe flush and reload the filters) the sched resetsto WF2Q+ of course, when i now perform the dlsreport speed tests the speeds are to be expected 900Mbits, quite constant, and with limited bufferbloat. However, when i have qa_coddle on the download just crashes hardcore, it goes up to 900 then stops (bufferbloat 35 seconds) then drops to 40Mbit and avg of 350 or so. its really weird. I checked my cpu performance and states and all, but nothing seems to bottle neck this.

Thanks for the additional information. Your particular case is indeed interesting because fq_codel looks like it's working fine on the upload side, but not on the download for some reason. It seems like it there is a constraint somewhere, whether it's physical or some type of processing constraint.

In any case, there are a few more things we can try:

If you increase the limiters from 900Mbit to 930Mbit or 940Mbit, do you see any difference?
Regarding your system specs, what make and model Intel NIC's do you have in your system?
Given that yours is a very fast connection (symmetric gigabit), we might want to try tuning the NIC parameters a bit to see if it will help:

For example, see these two threads and pfSense wiki entry:

https://forum.pfsense.org/index.php?topic=113496.0
https://forum.pfsense.org/index.php?topic=132345
https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards

In particular, I would be curious about, the rx and tx descriptors (rxd, txd), rx and tx process limit, number of queues, and the nmbclusters settings on your system.

You can easily access these values from the command line using, e.g. : sysctl -a | grep hw.igb.txd and so on. Do note that depending on the type of Intel NIC's you have, you may need to "em" instead of "igb".

I actually also have a symmetric gigabit fiber connection and was able to improve performance some after tuning some of these parameters.

Hope this helps.

zwck

Hey tman222,

thanks man for the help, when i up the limit to 930 or 940 the same happens, no real improvement.

The NICS are https://ark.intel.com/products/64404/Intel-Ethernet-Controller-I211-AT if i check what the parameters are the following shows up

These are my current values. maybe i should play around with them.

hw.igb.txd: 1024
hw.igb.rxd: 1024

net.pf.states_hashsize: 32768
net.pf.source_nodes_hashsize: 8192

hw.igb.tx_process_limit: -1
hw.igb.rx_process_limit: 100  

net.inet.tcp.syncache.hashsize: 512
net.inet.tcp.syncache.bucketlimit: 30

If i would like to change them i most likely have to put them into system tunables, right ?

tman222

@zwck:

Hey tman222,

thanks man for the help, when i up the limit to 930 or 940 the same happens, no real improvement.

The NICS are https://ark.intel.com/products/64404/Intel-Ethernet-Controller-I211-AT if i check what the parameters are the following shows up

These are my current values. maybe i should play around with them.
hw.igb.txd: 1024
hw.igb.rxd: 1024

net.pf.states_hashsize: 32768
net.pf.source_nodes_hashsize: 8192

hw.igb.tx_process_limit: -1
hw.igb.rx_process_limit: 100  

net.inet.tcp.syncache.hashsize: 512
net.inet.tcp.syncache.bucketlimit: 30
If i would like to change them i most likely have to put them into system tunables, right ?

Hi again,

Yes, you can change those settings either in the System Tunables section under Advanced Settings, or you can also put them in /boot/loader.conf.local

To begin, I would change the following:

hw.igb.txd: 2048
hw.igb.rxd: 2048

hw.igb.tx_process_limit: -1
hw.igb.rx_process_limit: -1 (100 is probably too low for a fast connection like yours).

Also, what value did you have for kern.ipc.nmbclusters? If it's less than 131072, I would change it to 131072 to start and see if that offers any improvement as outlined here:

https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards

–-----

Let's see if changing those parameters offers some improvement. Hope this helps.

zwck

so i completely reinstalled pfsense, from scratch, just set up the traffic shaper. same results as before.

Then i added

hw.igb.txd: 2048
hw.igb.rxd: 2048

hw.igb.tx_process_limit: -1
hw.igb.rx_process_limit: -1

but besides taking more memory nothing really changed. my kern.ipc.nmbclusters are twice that much. Whats next ? its 3 am and i just restored everything to the before stage… :( Thanks tman for all your help i am really clueless :(