Rocket League Ping Spikes and packet loss
-
Hi there,
So I've been trying to decrease my bufferbloat on my connection, using Limiters as instructed in the netgate YouTube video. However, in both PFSense and OPNSense I'm experiencing situations where I'm in a game (however it's worse and more often when there's another player - in comparison to a private match still hosted on a dedicated server) and experience high ping spikes lasting upto 5-10s, of even packet Lisa again for the same period of time. However despite this my buffer lost has improved from a C to an A however the more misterious thing is that this latency/packet loss still occurs, and the link isn't saturated (traffic graphs). I've got a 200/20 connection with Virgin Media typical speeds 210-220mbps down and 21mbps up. This is limited to 165 down and 16 up with tbr size at 4000. My queue is FQ Codel with Codel as the aqm.
Also when testing with fast.com, loaded ping is 55ms whereas unloaded is 24ms. Furthermore, with fast.com running and a ping to cloudflare in the background I see an average of a 20ms increase.
All of the disable overload options in advanced networking have been disabled.
I hope this helps,
Many thanks in advanced
Ryan
-
@RGK1 If you go in to settings at fast.com and tick off all the boxes and run the test again.
What is the loaded ping for upload and download? -
Hi there,
So just completed a couple of tests, the download loaded is 43ms the unloaded is 11ms and the loaded upload is 14ms. This by the way isn't through pfsense and ill try that later today.
Edit:
After testing with PFSense using limiters set to 170mbps and 17mbps respectively i'm getting 48ms loaded on download and 21ms loaded on upload with 13ms on unloaded using fast.com. This is with fq_codel as the scheduler and Codel as the AQM but as i've read this doesn't matter as setting the scheduler to fqcodel means it also acts as the aqm too.
Many thanks
Ryan
-
I would recommend to set queue management algorithm to tail drop in the limiter and the queue.
Furthermore you might need to lower limit to improve latency, a starting point could be 700 for download and 400 for upload. Quantum can be set to 300 to improve latency for small packets used for voip and gaming. -
Hi @bobbenheim,
Ive tried this and it seems better in games however, when using dlsreports.com i get what seems worse bufferbloat performance. PFSense : http://www.dslreports.com/speedtest/64736867 Stock: http://www.dslreports.com/speedtest/64736800
The download buferbloat is slightly better than stock however there is almost 600ms more bufferbloat when the upload is tested - this is using the high res bufferblaot setting.
I have tried increasing both the quantum and limit and this was the 'best' result using 400 quantum and 800 limit for the download and 400 quantum and 700 limit for the upload. Also my download is set to 150mbps down and 17 up.
Many thanks and i hope this helps
Ryan
-
@RGK1 with that kind of latency you get with dslreports there is little fq-codel can do to help you.
I would suggest you use fast.com or something else where your latency is well below 100 ms to tune in fq-codel or just find a torrent and do a simple ping to google.
You shouldn't need to set your download limit that low to obtain improved latency, but if your connection varies a lot during the day you would need to set the limit below saturation, when congestion occurs at your ISP, for fq-codel to work at all times. You mentioned packet loss which i assume you see in monitoring have you tried setting another monitoring IP in System > Routing > Gateway?
The reason for monitoring another address is that the ISP gateway isn't prioritizing answering ping and might be dropping ping requests giving you a false report on packet loss in pfsense. -
You might need a better ratio for upload:download. Your upload should not be less than 15% of your download. 150Mb down needs a minimum of about 22Mb upload. If you have your upload set to 17Mb, your download should not be more than ~110Mb.
Most TCP stacks do not respect loss for ACKs. If you're downloading 150Mb/s, the device is going to send about 22.5Mb/s of ACKs no matter what. fq-codel will try to drop packets to make the stream back-off, but TCP does not back off ACKs.
Of course this can vary for many reasons, but it's a good rule of thumb to start at.
-
@Harvy66 i am not sure where you get those numbers, but an Iperf test with 20 connections and 100 Mb/s download gives me about 3.7 Mb/s acknowledge traffic and that is when every TCP packet is acknowledged.
This would be less if i used the default setting, of two packets and an acknowledge, in Windows. -
Seems I mixed up ~5%(3.5% if you assume 54byte ack / 1500 byte payload) and 15%. I was recently working on a free space issue and was about 15%. I guess at 150Mb down, that's only ~7.5Mb up, which could not solely explain latency issues with ~15Mb up.
And it seems the first time I didn't fully comprehend the issue you described. The pfSense config clearly shows a higher spike on the upload, indicating it's probably pfSense causing it, and at a much lower total bandwidth.
-
@bobbenheim When looking closer at your speed tests, one thing I do see is your pfSense test shows this
Server Nett Speed RTT / Jitter Avg Re-xmit Avg Cwnd Avg
Los Angeles 2, CA, USA (webnx) d6 5.89 Mb/s 168.4±6.3ms 0.5% 96
Dallas, USA (softlayer) d3 6.48 Mb/s 145.3±7.3ms 0.1% 89
Dallas, USA (Foremost Communica) d6 6.34 Mb/s 199.6±4.7ms 0.1% 113
Los Angeles 2, CA, USA (webnx) u1 1.33 Mb/s 160±80ms - 10And your non-pfSense test shows this
Los Angeles 2, CA, USA (webnx) d6 9.14 Mb/s 204.1±5.2ms 0.6% 159
Dallas, USA (softlayer) d6 16.14 Mb/s 179.7±7.6ms 0.6% 169Since you're uploading to 2x more servers, tcp synchronization could be making it worse and the long RTTs don't help for response to dropped packets.
I would recommend doing a few tests to see if there's a pattern or not when it comes to the highres graphs.
-
Hi @Harvy66,
Im in the UK and only the amsterdam server is available to me (there is only one europe online server and fails to push 24 streams - i can only get 16) - this means i have to resort to further servers, however, on both tests i ran i manually selected these servers, but strangely also the amsterdam server which you say doesnt seem to appear, so im not sure why the PfSense test is showing more servers than the other.
Ryan
-
Hi @RGK1 - one thing you might try is running fewer parallel streams to see if that helps reduce some of the bufferbloat induced latency you are seeing without compromising on the speed (in particular on the upload). Even though you have Cable, you might try the DSL test option which defaults to 8 streams down and 8 streams up I believe. Otherwise, if you have an account over at DSLReports, you can also customize your test settings further once you login.
One other question I had: What cable modem (brand and model) are you using? Thanks in advance.
-
Hi @tman222 ,
Its a virgin super hub 3 (Arris TG2492), i ran the test with 8 streams: pfsense - http://www.dslreports.com/speedtest/64769252 and stock - http://www.dslreports.com/speedtest/64769085 again similar but the idle again higher on the pfsense box. However, whats more interesting (or maybe expected), but when i dont use hi-res bufferbloat the results show that the bufferbloat is ideal however it doesnt represent what the graph shows during the test and the results of the non hi-res bufferbloat test are completely on another level (in a good way). HERE
Ryan
-
Hi @RGK1 - I'm glad I asked about your modem. Unless I'm mistaking, your cable modem uses the Intel Puma 6 chipset, which is well-known to have latency related issues:
https://www.speedguide.net/routers/arris-touchstone-tg2492-24x8-eurodocsis-voip-gateway-3940
https://www.theregister.com/2016/12/03/intel_puma_chipset_firmware_fix/
https://approvedmodemlist.com/intel-puma-6-modem-list-chipset-defects/I would first try to see if there's a firmware update available that fixes these issues. If not, I would recommend purchasing or renting another modem that's also supported by Virgin but doesn't have the Intel Puma 6 chipset. That should help clear up your latency issues.
-
hi @tman222 ,
they have released a pacth that 'claims' to fix bufferbloat so take it with a grain of salt however, virgin media strictly wont allow you to use your own modem with there services even with mac spoofing, they do have the newer hub 4 with puma 7 (facepalm) available that is supposed to available for those with issues with speed, so it may be possible for an upgrade but the verdict is that there is no real improvement. This was the reason behind pfsense for me, putting the hub into modem mode and then using ubiquiti aps, but im trialling the pfsense part of this first.
-
@RGK1 - I'm not sure that setting up pfSense with traffic shaping (e.g. fq-codel) will be able to provide any relief since sounds like that the latency spikes with the Puma 6 chipset can also occur when the connection is just lightly used.
-
@tman222 ,
Okay. Well and that aside im now experiencing issues with pfsense where its not applying my limter settings for the download and now is reset to deefaults and cant be changed. So i restarted and then im now left with this an no internet access, on the separate subnet of course.
Limiters:
00001: 13.000 Mbit/s 0 ms burst 0
q131073 50 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0 droptail
sched 65537 type FIFO flags 0x0 4000 buckets 0 activeSchedulers:
00001: 13.000 Mbit/s 0 ms burst 0
q65537 50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
sched 1 type FQ_CODEL flags 0x0 0 buckets 0 active
FQ_CODEL target 5ms interval 100ms quantum 1 limit 650 flows 1024 NoECNAre there any other alternatives or anything anybody reccommends running instead of pfsense for running in a VM also i find OPNSense doesnt have the features and customizability that pfsense does - i.e not the same customisable options for schedulers and aqms - no tail drop or cake - not that i need it but its good to try different things. Not to mention when i change subnet on my pc (my vm is on a separate machiene) the whole pfsense box crashes which is completely over my head as to why it will do that.
Many Thanks
Ryan
-
@RGK1 tried IPFire, Opnsense and fq-cake in openwrt but return to PFsense everytime as i get the best performance with fq-codel in it, when testing with flent. Probably could tweak it somewhere in some of the other distributions but tweaking just isn't as user friendly in them in my opinion.
Might just have gotten used to PFsense over the years :)
For testing your modem if it is affected by the PUMA bug you can use pingplotter and set it to ping e.g. google.com in intervals of 0.05-0.02 seconds, always nice with a graph to look at and it should be quite obvious if you are affected. -
How much deviation in the ping are we talking about, because after the initial spike, the ping seems okay, however personally i have to say i dont have much experience with ping plotter so couldnt say.
Ryan
Edit: Okay scrap that after a rerun for some reason it got completely worse.
But apparently putting the router into modem mode helps this issue as i mentioned before allowing pfsense to have the external ip address and handle all the routing etc - making the virgin hub a bridge between the ethernet and docsis. However, when i try to set this up with both pfsense and opnsense i can ping from both lan and wan interfaces - however i cannot ping or connect to the internet with any clients on the lan interface with rules on the firewall allowing traffic both in and out from any source. I'd really like to try this and get this to work to shift the work to the pfsense box but it just wont budge. Also when changing back to router mode (re-enabling wifi and dhcp etc) the pfsense and opnsense both work fine once they gain the correct ip, and no other changes made.
Ryan
-
@RGK1 tman222 posted https://www.theregister.com/2016/12/03/intel_puma_chipset_firmware_fix/ where there is a comparison in ping plotter of a working modem and an Intel PUMA modem you can compare your own results to.
I would keep the comparison to the endpoint as intermittent hops isn't prioritizing ICMP traffic, and there can be large deviations in response time, and is only really useful if the endpoint has the same deviations which your plot doesn't.
So if i understood you correctly you can ping hosts on the internet and on LAN from Pfsense but you can not connect to the internet from anything connected to LAN when your modem is in bridge mode?
If that is the case have you made any changes to Pfsense?