Shaping question, how to keep latency low
-
Hello,
I am running a multi wan router with 3 internet connections on a lan gaming center. Everything is working quite great, but there is one improvement I am trying to do, and I still have not been able to figure out the best way of fixinf my problem.
I have different rules for outbound traffic, so the outgoing traffic will go out on one of my connections depending on a given rule. Basically one connection is a high bandwidth connection for web, IM, downloads, and all unknown traffic, and the other 2 connections are for games, where it is important to keep ping times low.
I am having problems with Steam, Valve's gaming platform, which is used for many games like Counterstrike, Team Fortress 2, and it is becoming a software delivery platform. The problem is that I direct all Steam traffic (ports 27000-27050) to one of our gaming connections. This way I will be able to play TF2 without anyone visiting myspace making my ping go from 20ms to 200ms :D
But, my problem comes now as Steam is used more and more everyday for delivering new games, or just updating the already installed games. TF2 update? No problem, it is just a 250Mb automatic download… that while is being done will use the 5mbps bandwidth on this connection, making everyone's ping sky rocket until it finishes.
I tried to use traffic shapper for solving my problem, but I have been unable to do so 100%. Why? Because I can not tell the difference on Steam traffic if someone is just playing, or if someone is downloading/updating content. That means that if I create a queue that will only use 3mbps (out of 5mbps) and redirect all Steam traffic on that queue, ping times won't go up for those playing other games on that connection, but just one person updating a Steam game will raise everyone else's Steam ping.
I have also tried to use queue's by IP. I create a 1mbps max queue, and redirect Steam traffic from a given IP to that queue. When I am updating Steam on one computer, ping times on another computer will be good. Great! Wait, but it is not so great... Because I have 42 computers, so I would need to make 42 different 1mbps queues, one for each computer. But then again, this wouldn't be such a good solution, since 5 people updating Steam at the same time (it does happen) would also bring the connection to its knees.
Still trying to find out what would the best solution be on my scenario, but I am not so sure on how to handle this. Ideally I would like my Steam alias (ports 27000-27050) to use no more than say 3mbps on the 5mbps connection. But that solution doesn't work for me, I need to make something by IP/computer, since otherwise one single person updating Steam will mess up everyone else's Steam ping times like it happens now. Anyone can see a possible solution here...?
Thanks for reading!
-
The traffic shaper currently does not work with MultiWAN.
(Take a look at the bounty forum –> There's a thread to change that)But to your problem.
Doesnt Steam itself use different ports than the games?
I think i remember something that the Games run on UDP/TCP 27015 whilst the rest of steam uses the ports UDP 27000 to 27020 and TCP 27020 to 27050.Couldnt you set the rules up so that the games go over one line and the updates over another?
-
The traffic shaper currently does not work with MultiWAN.
(Take a look at the bounty forum –> There's a thread to change that)I do not really want it to work on Multiwan, but just on one of the interfaces… I have actually done the tests on my OPT2 interface, and it did work ok. But I wasn't doing any shaping on WAN, just on OPT2.
But to your problem.
Doesnt Steam itself use different ports than the games?
I think i remember something that the Games run on UDP/TCP 27015 whilst the rest of steam uses the ports UDP 27000 to 27020 and TCP 27020 to 27050.Couldnt you set the rules up so that the games go over one line and the updates over another?
I do not think there is just one standard port for each thing Steam does. While I am sure that the default port for Steam servers is 27015, it is very easy to see servers with ports other than that. It is actually pretty common for a single machin to be running several servers, which they will usually start at 27015 and go up from there (27016, 27017, 27018, …) It is not strange either to see servers on ports that will even fall out of my port range, which on my setup will just be played on the other internet connection.
I just launched Steam now on one of the computers, and without even opening one game I have already seen Steam related connections on pfSense to the following IPs and ports:
63.145.202.3:27030 63-145-202-3.dia.static.qwest.net
193.34.49.2:27031 Valve74.milan.clanserver4u.net
69.28.140.245:27031
68.142.72.250:27038
207.173.177.12:27013 steam2.steampowered.com
72.165.61.190:27030 72-165-61-190.dia.static.qwest.net
207.173.176.131:27046 cafe.steampowered.com
208.111.133.85:27017
87.248.196.199:27030Login servers, content servers, who knows... But I do not think there is an easy way to tell the differece amongst all of them. I tried updating one game, and the connections I see are:
63.145.202.3:27030
87.248.196.114:27030
207.173.176.131:27046
87.248.196.195:27030
208.111.133.85:27017
87.248.208.114:27031Maybe I could create a penalised queue for all ports except 27015, and just leave the 27015 outside. I would say that 27015 is the most used server port number (wouldn't be able to give out a %), and hopefully it is never used for any downloading. Guess I will dig some more on the steampowered forums (already did many times, I still do not understand how they do not have an option for limiting bandwidth on the Steam client!), since the option of creating a queue for each computer on the network doesn't seem doable.
-
A quick question on the Steam forums gave me what I think weas the best answer (which I should have known myself), game traffic is UDP, download/update traffic is TCP. While I do not know 100% sure if that is always the case, the tests I have done so far seem to say so.
So my solution has been to "shape" TCP traffic on the Steam ports, while leaving everything else untouched.
From the tests I have done all this morning, there is a noticeable difference. The connection has 6mbps download bandwidth, so I have limited it to 3mbps (incoming) on TCP traffic on the Steam port range. After doing this, doing some ping tests (to my ISP DNS) over a 5 minute period of time gave me the following results:
20-21ms - Nothing being downloaded
33-34ms - Traffic shaping enabled, TCP limit at 3mbps. Traffic graph shows a little bit over 3mbps inbound being used.
64-65ms - Traffic shaping disabled. Traffic graph shows a usage of 5.5-6mbpsI understand that no shapping on the customer side will be perfect, so this results are more than good enough for me. I might do some tests later on to see if I can raise a bit those 3mbps without ping times being affected more than those 13ms, but I am already happy with that :D
Now I will only have to keep an eye on the RRD system graphs for a while to see how my setup handles traffic shaping, I wonder if it would take a big hit on CPU usage if maybe 30 or 40 computers are on Steam at the same time…
Thanks!
-
MindTwist -
I've been running PFSENSE on a 10/1 cable connection with 10-15 gaming rigs every weekend AND we have bittorrent and other traffic on the network. First off I upp'd the max number of connections to 75k, then I put in a rule to give ALL UDP (because most all time sensitive game traffic is UDP) highest priority with web medium and everything else default. I also specified the max bandwidth up/down on the traffic shaper config to be just slightly less than the actual connection. This combination works great for us.
I run a constant ICMP ping to Seattle (~15ms) and then load a few people into a game (normally COD4 or TF2). Pings would hang in the 20s for the gamers. Then I'd run a Speakeasy.net bandwidth test with nothing configured on the traffic shaper and while the downstream was saturated pings would jump to ~100ms however when uploading it would jump to ~500ms (with players complaining).
Then I enabled the traffic shaper as I described above, and WOW the results were amazing: I ran the exact same test, and this time when the download started the constant ICMP still showed poor results HOWEVER the in-game pings held at a constant 20-30ms range max - absolutely amazing! Then we repeated the same test with a couple computers saturating the connection with Bittorrent, running a speed test, AND refreshing COD4 servers (note uses about 6K simultaneous connections for 1 PC to refresh all COD4 servers!) at the same time all while playing: The result? I think 46ms was the maximum ping anyone reported - absolutely stunning performance considering the connection was completely PEGGED!
-
jerm,
Thanks a lot on your input. My problem is mainly with incoming traffic, since I can decide what goes out or doesn't go out on this connection. That is, if my outbound bandwidth is saturated, I could easily fix it. If my inbound bandwidth is all used, I can hardly do anything on my side, since gaming traffic is already coming to me delayed. I can shape it on my side, but whatever I do will never be 100% as efficient as it would be on my ISP side.
So as I have it right now I am more than happy. A 12ms on average ping increase while someone is updating TF2 is good enough for me, way better than with no shapping at all, when Steam was usiong all available bandwidth as soon as someone needed to update something (even the update of the Steam client would give everyone several seconds of lag!)