Rocket League Ping Spikes and packet loss
So I've been trying to decrease my bufferbloat on my connection, using Limiters as instructed in the netgate YouTube video. However, in both PFSense and OPNSense I'm experiencing situations where I'm in a game (however it's worse and more often when there's another player - in comparison to a private match still hosted on a dedicated server) and experience high ping spikes lasting upto 5-10s, of even packet Lisa again for the same period of time. However despite this my buffer lost has improved from a C to an A however the more misterious thing is that this latency/packet loss still occurs, and the link isn't saturated (traffic graphs). I've got a 200/20 connection with Virgin Media typical speeds 210-220mbps down and 21mbps up. This is limited to 165 down and 16 up with tbr size at 4000. My queue is FQ Codel with Codel as the aqm.
Also when testing with fast.com, loaded ping is 55ms whereas unloaded is 24ms. Furthermore, with fast.com running and a ping to cloudflare in the background I see an average of a 20ms increase.
All of the disable overload options in advanced networking have been disabled.
I hope this helps,
Many thanks in advanced
@RGK1 If you go in to settings at fast.com and tick off all the boxes and run the test again.
What is the loaded ping for upload and download?
So just completed a couple of tests, the download loaded is 43ms the unloaded is 11ms and the loaded upload is 14ms. This by the way isn't through pfsense and ill try that later today.
After testing with PFSense using limiters set to 170mbps and 17mbps respectively i'm getting 48ms loaded on download and 21ms loaded on upload with 13ms on unloaded using fast.com. This is with fq_codel as the scheduler and Codel as the AQM but as i've read this doesn't matter as setting the scheduler to fqcodel means it also acts as the aqm too.
I would recommend to set queue management algorithm to tail drop in the limiter and the queue.
Furthermore you might need to lower limit to improve latency, a starting point could be 700 for download and 400 for upload. Quantum can be set to 300 to improve latency for small packets used for voip and gaming.
Ive tried this and it seems better in games however, when using dlsreports.com i get what seems worse bufferbloat performance. PFSense : http://www.dslreports.com/speedtest/64736867 Stock: http://www.dslreports.com/speedtest/64736800
The download buferbloat is slightly better than stock however there is almost 600ms more bufferbloat when the upload is tested - this is using the high res bufferblaot setting.
I have tried increasing both the quantum and limit and this was the 'best' result using 400 quantum and 800 limit for the download and 400 quantum and 700 limit for the upload. Also my download is set to 150mbps down and 17 up.
Many thanks and i hope this helps
@RGK1 with that kind of latency you get with dslreports there is little fq-codel can do to help you.
I would suggest you use fast.com or something else where your latency is well below 100 ms to tune in fq-codel or just find a torrent and do a simple ping to google.
You shouldn't need to set your download limit that low to obtain improved latency, but if your connection varies a lot during the day you would need to set the limit below saturation, when congestion occurs at your ISP, for fq-codel to work at all times. You mentioned packet loss which i assume you see in monitoring have you tried setting another monitoring IP in System > Routing > Gateway?
The reason for monitoring another address is that the ISP gateway isn't prioritizing answering ping and might be dropping ping requests giving you a false report on packet loss in pfsense.
You might need a better ratio for upload:download. Your upload should not be less than 15% of your download. 150Mb down needs a minimum of about 22Mb upload. If you have your upload set to 17Mb, your download should not be more than ~110Mb.
Most TCP stacks do not respect loss for ACKs. If you're downloading 150Mb/s, the device is going to send about 22.5Mb/s of ACKs no matter what. fq-codel will try to drop packets to make the stream back-off, but TCP does not back off ACKs.
Of course this can vary for many reasons, but it's a good rule of thumb to start at.
@Harvy66 i am not sure where you get those numbers, but an Iperf test with 20 connections and 100 Mb/s download gives me about 3.7 Mb/s acknowledge traffic and that is when every TCP packet is acknowledged.
This would be less if i used the default setting, of two packets and an acknowledge, in Windows.
Seems I mixed up ~5%(3.5% if you assume 54byte ack / 1500 byte payload) and 15%. I was recently working on a free space issue and was about 15%. I guess at 150Mb down, that's only ~7.5Mb up, which could not solely explain latency issues with ~15Mb up.
And it seems the first time I didn't fully comprehend the issue you described. The pfSense config clearly shows a higher spike on the upload, indicating it's probably pfSense causing it, and at a much lower total bandwidth.
@bobbenheim When looking closer at your speed tests, one thing I do see is your pfSense test shows this
Server Nett Speed RTT / Jitter Avg Re-xmit Avg Cwnd Avg
Los Angeles 2, CA, USA (webnx) d6 5.89 Mb/s 168.4±6.3ms 0.5% 96
Dallas, USA (softlayer) d3 6.48 Mb/s 145.3±7.3ms 0.1% 89
Dallas, USA (Foremost Communica) d6 6.34 Mb/s 199.6±4.7ms 0.1% 113
Los Angeles 2, CA, USA (webnx) u1 1.33 Mb/s 160±80ms - 10
And your non-pfSense test shows this
Los Angeles 2, CA, USA (webnx) d6 9.14 Mb/s 204.1±5.2ms 0.6% 159
Dallas, USA (softlayer) d6 16.14 Mb/s 179.7±7.6ms 0.6% 169
Since you're uploading to 2x more servers, tcp synchronization could be making it worse and the long RTTs don't help for response to dropped packets.
I would recommend doing a few tests to see if there's a pattern or not when it comes to the highres graphs.
Im in the UK and only the amsterdam server is available to me (there is only one europe online server and fails to push 24 streams - i can only get 16) - this means i have to resort to further servers, however, on both tests i ran i manually selected these servers, but strangely also the amsterdam server which you say doesnt seem to appear, so im not sure why the PfSense test is showing more servers than the other.
Hi @RGK1 - one thing you might try is running fewer parallel streams to see if that helps reduce some of the bufferbloat induced latency you are seeing without compromising on the speed (in particular on the upload). Even though you have Cable, you might try the DSL test option which defaults to 8 streams down and 8 streams up I believe. Otherwise, if you have an account over at DSLReports, you can also customize your test settings further once you login.
One other question I had: What cable modem (brand and model) are you using? Thanks in advance.
Hi @tman222 ,
Its a virgin super hub 3 (Arris TG2492), i ran the test with 8 streams: pfsense - http://www.dslreports.com/speedtest/64769252 and stock - http://www.dslreports.com/speedtest/64769085 again similar but the idle again higher on the pfsense box. However, whats more interesting (or maybe expected), but when i dont use hi-res bufferbloat the results show that the bufferbloat is ideal however it doesnt represent what the graph shows during the test and the results of the non hi-res bufferbloat test are completely on another level (in a good way). HERE
Hi @RGK1 - I'm glad I asked about your modem. Unless I'm mistaking, your cable modem uses the Intel Puma 6 chipset, which is well-known to have latency related issues:
I would first try to see if there's a firmware update available that fixes these issues. If not, I would recommend purchasing or renting another modem that's also supported by Virgin but doesn't have the Intel Puma 6 chipset. That should help clear up your latency issues.
hi @tman222 ,
they have released a pacth that 'claims' to fix bufferbloat so take it with a grain of salt however, virgin media strictly wont allow you to use your own modem with there services even with mac spoofing, they do have the newer hub 4 with puma 7 (facepalm) available that is supposed to available for those with issues with speed, so it may be possible for an upgrade but the verdict is that there is no real improvement. This was the reason behind pfsense for me, putting the hub into modem mode and then using ubiquiti aps, but im trialling the pfsense part of this first.
@RGK1 - I'm not sure that setting up pfSense with traffic shaping (e.g. fq-codel) will be able to provide any relief since sounds like that the latency spikes with the Puma 6 chipset can also occur when the connection is just lightly used.
Okay. Well and that aside im now experiencing issues with pfsense where its not applying my limter settings for the download and now is reset to deefaults and cant be changed. So i restarted and then im now left with this an no internet access, on the separate subnet of course.
00001: 13.000 Mbit/s 0 ms burst 0
q131073 50 sl. 0 flows (1 buckets) sched 65537 weight 0 lmax 0 pri 0 droptail
sched 65537 type FIFO flags 0x0 4000 buckets 0 active
00001: 13.000 Mbit/s 0 ms burst 0
q65537 50 sl. 0 flows (1 buckets) sched 1 weight 0 lmax 0 pri 0 droptail
sched 1 type FQ_CODEL flags 0x0 0 buckets 0 active
FQ_CODEL target 5ms interval 100ms quantum 1 limit 650 flows 1024 NoECN
Are there any other alternatives or anything anybody reccommends running instead of pfsense for running in a VM also i find OPNSense doesnt have the features and customizability that pfsense does - i.e not the same customisable options for schedulers and aqms - no tail drop or cake - not that i need it but its good to try different things. Not to mention when i change subnet on my pc (my vm is on a separate machiene) the whole pfsense box crashes which is completely over my head as to why it will do that.
@RGK1 tried IPFire, Opnsense and fq-cake in openwrt but return to PFsense everytime as i get the best performance with fq-codel in it, when testing with flent. Probably could tweak it somewhere in some of the other distributions but tweaking just isn't as user friendly in them in my opinion.
Might just have gotten used to PFsense over the years :)
For testing your modem if it is affected by the PUMA bug you can use pingplotter and set it to ping e.g. google.com in intervals of 0.05-0.02 seconds, always nice with a graph to look at and it should be quite obvious if you are affected.
How much deviation in the ping are we talking about, because after the initial spike, the ping seems okay, however personally i have to say i dont have much experience with ping plotter so couldnt say.
Edit: Okay scrap that after a rerun for some reason it got completely worse.
But apparently putting the router into modem mode helps this issue as i mentioned before allowing pfsense to have the external ip address and handle all the routing etc - making the virgin hub a bridge between the ethernet and docsis. However, when i try to set this up with both pfsense and opnsense i can ping from both lan and wan interfaces - however i cannot ping or connect to the internet with any clients on the lan interface with rules on the firewall allowing traffic both in and out from any source. I'd really like to try this and get this to work to shift the work to the pfsense box but it just wont budge. Also when changing back to router mode (re-enabling wifi and dhcp etc) the pfsense and opnsense both work fine once they gain the correct ip, and no other changes made.
@RGK1 tman222 posted https://www.theregister.com/2016/12/03/intel_puma_chipset_firmware_fix/ where there is a comparison in ping plotter of a working modem and an Intel PUMA modem you can compare your own results to.
I would keep the comparison to the endpoint as intermittent hops isn't prioritizing ICMP traffic, and there can be large deviations in response time, and is only really useful if the endpoint has the same deviations which your plot doesn't.
So if i understood you correctly you can ping hosts on the internet and on LAN from Pfsense but you can not connect to the internet from anything connected to LAN when your modem is in bridge mode?
If that is the case have you made any changes to Pfsense?
Pretty much correct, ive tried but when the router is in modem mode the pfsense box becomes the router gaining the correct external ip address on the wan ip (directing traffic and assigns the routes to external and internal ips) even after resetting to defaults it can get the external ip and can ping external networks (22.214.171.124) but my client a pc directly attatched can ping the local gateway (lan ip) 192.168.1.1 and the wan ip address (beggining with 86) however it cannot ping the external gateway nor any other networks. And when resetting to defaults and having the router in router mode the system works fine first time. Also when directly attatching only my pc to the rouuter in modem mode i can gain the correct external ip address and browse the internet just fine - i also did a dslreports test too - http://www.dslreports.com/speedtest/64771697
I guess from the comparison to the ping plotter graph from the article linked the patch that they applied may have actually done something?
Hope this helps
Hi @RGK1 - after doing a little bit more research, it looks like Virgin did address the Puma 6 issue, although some people are still reporting problems:
Do you mind sharing with us the specs of your pfSense box as well, including brand and type of network interfaces you are using?
I was looking at that post this morning in fact - I have a amd a8 7600 4 core cpu 3.1ghz 16gb of ram with 1 intel gb nic and 1 realtek gigabit nic with all hardware offloading disabled
Have you done any manual IP settings, on the host that can't connect to the internet, when using Pfsense with your Super Hub in bridge mode?
Also, when your trying to ping outside your LAN from the host are you pinging an FQDN address or an IP address?
Yes - is manually set the ip and turned of dhcp on the server so the other clients are affected. The lan ip on the server is 192.168.1.1 and the client directly attached to the lan port was 192.168.1.2 subnet 24 and gateway 192.168.1.1 and dns 126.96.36.199. For either the pc and pfsense as the 'host' i used 188.8.131.52 to ping to and to make sure dns look ups were working on the pfsense box i pinged google.co.uk - only on pfsense box because again no connectivity on pc.
Many Thanks Again :)
Hi @RGK1 - One thing to doublecheck is to make sure you are using the right default gateway under System > Routing > Gateways. Hope this helps.
I did check this and its basically as it should be (external ip but drop last 2 digits for a 1) :).
@RGK1 - how did you setup fq-codel? Did you use floating rules on your WAN interface or limiters on an outbound traffic rule on your LAN interface(s)? Do you mind showing a screenshot of the relevant firewall rules to us? Perhaps we can see if something misconfigured from there. Thanks again.
Just as a note, but when i reset to defaults it works for having the router in router mode straight out of the box after automatic assigning of interfaces however if i again reset to defaults and put the router to modem mode i can get the external ip but cant get a connection.
I will try and get a screeshot o the firewalls but i think this may be irrelevant as everything should be stock (in theory).
Hi so i was able to get it working - after a couple days break and restart it seemed to work, but my problem is im getting quite varied results this is with modem mode enabled. With the same settings i get varying results. Im currently getting and A-B from C originally however even with fq codel im experiencing ping spikes in rocket league again with over 900ms and packet loss. These are my settings.
Top is download at 140mbits on a 200mbit connection but mainly stays around 220 and upload 12mbit on 20mbit normally 22 mbit
Is there anything that i should change or am missing here causing these sudden ping spikes?
Edit: according to the in-game stats im mainly loosing packets on the outbound roughly 3-9 but sometime this goes upto 71 the servers are 60hz tickrate.
Edit2:may have found my problem set the limit too low and quantum too high possibly explaining the packet loss as it won't be prioritising the smaller packets :D
I'll test this tommorow.
@RGK1 are you or others generating any traffic on your connection that would cause the ping spikes?
If you know the IP of the server in rocket league you could try and ping it with pingplotter and you should be able to see if the latency spikes happens on your connection or further onto the internet.
Furthermore, recommended setting for target is 5-10% of interval.
This post is deleted!
@JammEr21 What kind of connection do you have?
For those wondering, there is a bug with the SH3 and FW 9.1.1811.401 when specifically the router is modem only mode and the firewall is PfSense, it doesn't seem to affect other hardware routers, but PfSense and OPNsense.
I've put a limiter on my download and uploads speeds and get A, A, A on dlsreports, bufferbloat stays low, but give or take every hour or so i do get spikes, which i am filtering slowly with trafic shaping.
Not a fix, but does mask the issue VMs router is giving.
In case it helps, here is my last days chart
Sorry guys about my recent inactivity, i'm a 16 year old student that just gone back into education. I haven't really recently had time to investigate further as i ended up hitting a brick wall. Either it was slow speeds A buffer-bloat or viceversa. was able to get better results using OPNsense achieving i believe between an A (i think - sometimes may have dipped to B) and this was the best i was able to consistently achieve with pf-sense going from a C (through pf-sense with router in modem mode and no limiters) which i believe which is an improvement. I believe i used 13mbps up and 150mbps down on a 200mbps down connection. Sometime when i set the limiter to a higher speed i get worse bufferbloat scores than stock say 180 mbps and im getting only a C. However, recently i've stuck with stock settings without pf-sense or opnsense works better as i just ran a test achieving (220.6mbps down and 21.6mbps up as virgin over-prevision speeds) and got A+ connection quality and B rating buffer-bloat and honestly i haven't thought it to be worth the loss in speed - unless there's anything else i can try...
Here's the result if anybody wants to see as it shows the buffer-bloat in more detail - idle vs downloading and uploading - http://www.dslreports.com/speedtest/65616525
This was my last result where i got A buffer-bloat rating using OPNsense - http://www.dslreports.com/speedtest/64830336
Hopefully this helps in diagnosing - and i'm down for tinkering ;)
Use limiters as per this guide
Be sure you flush your states before you try otherwise you may have inaccurate results
Do note with 200Mbps you may want to change your root limiter queue length to ~2000, I have 350-380Mbps down and 3000 works best for me.
Same on the upload queue, but play with those settings, less than 3000 for me still gave me high buffer bloat, the default settings may not be enough, so try somewhere between 1500 and 2500 for your speed
If you are not using limiters, then note this from the guide;
The ALTQ framework is handled through pf and is closely tied to network card drivers. ALTQ can handle several types of schedulers and queue layouts. The traffic shaper wizard configures ALTQ and gives firewall administrators the ability to quickly configure QoS for common scenarios, and it allows custom rules for more complex tasks. ALTQ is inefficient, however, so the maximum potential throughput of a firewall is lowered significantly when it is active.
pfSense software also supports a separate shaper concept called Limiters. Limiters enforce hard bandwidth limits for a group or on a per-IP address or network basis. Inside of those bandwidth limits, limiters can also manage traffic priorities.
This post is deleted!
This post is deleted!
This post is deleted!