High RTT latency on wan [SOLVED]
-
why am i getting such a high latency on my wan line up to 2,000ms and on my wan two line its very low when both are under load. Also my wan line is my more stable line and is a cable isp provider. Because wan two is a wireless isp provider line both are 20mbps.
-
Any latency above propagation latency is caused by buffer bloat. Your cable ISP may have worse bloat, and by "may" I mean "very likely because it's the norm". Don't send data faster than it can handle, add some shaping.
-
Try setting up simple dummynet limiters.
Set one for upload, one for download. Set them for 85% of your subscribed bandwidth.
Apply those limits to your firewall rules in the advanced settings. The order is upload/download limiters left to right in the GUI.
I would recommend setting both pipes to fq_codel as it is very good at eliminating bufferbloat.
Run this command.ipfw sched 1 config pipe 1 type fq_codel && ipfw sched 2 config pipe 2 type fq_codelYou can make that setting permanent on reboot by adding that to shellcommand.
You might need to reset your states after you implement this.
If your latency doesn't go significantly down then the problem.is not bufferbloat.
-
when i set the port speed in pfsense to 100mbps full it doesn't have the high RTTsd instead it has alot of packet loss.
The arris modem port is set to auto negotiate speed and is capable of 1gbps.
I am wondering if its the HP NC364T PCIe intel network card or something in the system.
-
You can't hard-set one side only if the other side is auto-negotiate. You want to set that back to Default (no preference, typically autoselect).
As I understand it, codel doesn't do much unless it is implemented on the device that is actually doing the buffering. If that is your ISP then it won't do any good.
I would try implementing a quick way to limit how fast you send to the ISP. If you can upload 20Mbit/sec reliably but you get buffer bloat and high-latency, I would try a PRIQ like this on WAN.
If you are running 2.4 you can run a Limiter (dummynet) but you will probably tickle a bug with Limiters+NAT on 2.3 so I'd stick to ALTQ/PRIQ since it's the simplest ALTQ shaper and all you have to do is restrict how fast you send.
Adjust the 10Mbit/sec in the interface scheduler down until you can upload at will and not experience any buffer bloat.
 -
what bugs are there with limiters + NAT on 2.3? are there any remaining on 2.4? I've been using both on 2.4 with no issues.
Codel / fq + codel won't do anything if it isn't the system providing the buffering. But you can ensure it does provide the buffering.
If you pay for 100Mbps and your ISP gets you speeds of on average of 100Mbps @ night, 90 Mbps during the day, 85Mbps during peak hours, then you would take a percentage of the lowest value (85Mbps) unless you are never home during peak hours (at work, etc.( then use a percentage of the lowest value when you are home.If you have something like really reliable fiber, then probably 95% is good, if you have good cable then 90-95% is probably good. If you have average cable reliability I'd use 85%, and if your cable provider sucks then 80%. These are just ballpark figures, you'll have to evaluate your own connection to get a good number.
But let's say you have crappy cable so you use 80% of 85Mbps = 68Mbps / 68000Kbps. This is what you would set your download limiter to, apply the same method to your upload.
This will keep your router and therefore fq_codel queueing your internet, eliminating bufferbloat.
It's up to you whether you'd rather have the throughput or the latency. I perosnally prefer to sacrifice a little throughput for snappy internet. You'll really be able to tell the difference in most connections.
On DSLreports I go from a really bad "F" (2000+ ms bufferbloat) to an A or A+ with this method. I don't game, but things like facetime / skype are notably improved. Keep in mind that this really only kicks in when the network is saturated, but that can happen pretty easily for most residential connections.
-
codel still does nothing for you.
Limiting the download speed might but unless your system is doing the buffering, which is almost never the case since it is the (slower) ISP sending to you and out gigabit so there should be minimal if any queuing there, codel won't help.
-
Thanks for the reponse i am going to try this as soon as i get home.
-
codel still does nothing for you.
Limiting the download speed might but unless your system is doing the buffering, which is almost never the case since it is the (slower) ISP sending to you and out gigabit so there should be minimal if any queuing there, codel won't help.
hmmmm, no you're definitely wrong. by limiting the speed at the router, you force the router to be the slowest link not the ISP - therefore codel kicks in.
it's pretty easy to demonstrate this for yourself.
plug into the modem directly and go to DSLreports.com, run a few tests. or just remove all limiters and run the test through pfsense.
now plug back in pfsense and apply a limiter as specified above, making your router the slowest link. run the test again. see the difference?
The difference is dramatic, i've used this on multiple machines at multiple locations.
you can get a more thorough test than dslreports by running an RRUL test on FLENT (linux), if you're so inclined.
-
It really comes down to the stability of the bandwidth the ISP provides. I can set my upload bandwidth to 99% and my download to ~98% and get effective traffic shaping for both in/egress on the WAN. One may need to dramatically reduce their ingress rate limiting to get effective traffic shaping.
-
You can't hard-set one side only if the other side is auto-negotiate. You want to set that back to Default (no preference, typically autoselect).
As I understand it, codel doesn't do much unless it is implemented on the device that is actually doing the buffering. If that is your ISP then it won't do any good.
I would try implementing a quick way to limit how fast you send to the ISP. If you can upload 20Mbit/sec reliably but you get buffer bloat and high-latency, I would try a PRIQ like this on WAN.
If you are running 2.4 you can run a Limiter (dummynet) but you will probably tickle a bug with Limiters+NAT on 2.3 so I'd stick to ALTQ/PRIQ since it's the simplest ALTQ shaper and all you have to do is restrict how fast you send.
Adjust the 10Mbit/sec in the interface scheduler down until you can upload at will and not experience any buffer bloat.
I applied the adjustments to the wan interface and reduce it to PRIQ to14 mbits for testing but when i run a download test it stills get high latency then packet loss. In addition, the bandwidth is still reaching its peak at 20mbit. Nothing changed. It doenst get high latency when i reduce the arris modem to 100mbits full rather than autoselect, which brings it to 1gbps.
-
If your problem is on download:
1: Your ISP sucks - they should not be buffer-bloating downloads like that.
2: You need to limit downloads on LAN as well.
-
Download bufferbloat is a common problem. Most major ISPs do not do anything (useful) to prevent bufferbloat.
AQM in the form of PIE will come to cable modems, it's mandated in DOCSIS 3.1 because bufferbloat is such a ubiquitous problem.
A better statement would be, if you don't have bufferbloat problems on download with sub gigabit WAN, your ISP is exceptional!
-
If your problem is on download:
1: Your ISP sucks - they should not be buffer-bloating downloads like that.
2: You need to limit downloads on LAN as well.
Yes they do suck. Right now i am trying to setup openvpn to work and they have blocked the ports i am trying to use so i had to call them and request they remove the block. They said they would remove it within 24-48 hours.
-
Bullshit.
"Real" ISPs all understand that their customer bandwidth is less than their backbone bandwidth and it is incumbent upon them to eliminate bloat there.
It is preposterous to think that a customer with a 100Mbit/sec down needs to shape to make the connection work properly. A customer cannot affect how fast traffic is sent to him.
-
I am here trying to figure out why it gets high latency only when i set the wan interface to 1gbps. This doesn't happen when i set to 100mbits full. but the problem is i can't set the duplex on the other end which is the Arris modem. My ISP locks the duplex port settings.
-
Well, the modem is probably auto-neg so set the WAN interface to Default (no preference, typically autoselect) and stop screwing with it.
Give a grunt a knob to turn and he'll turn it.
Chattanooga, Tennessee, USA
A comprehensive network diagram is worth 10,000 words and 15 conference calls.
DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
Do Not Chat For Help! NO_WAN_EGRESS(TM) -
Double Bullshit, haha! Just look out there on the internets. ISP's are not eliminating bufferbloat for residential customers, up or down.
If "Real" ISP's don't include say - Cox, Comcast, TWC, AT&T…. then sure, maybe not. But if those are real ISP's, then yeah bufferbloat is a problem and it won't be solved by them.
"Get it to work properly" is pretty subjective. Most customers haven't a clue what bufferbloat is. It doesn't rear it's head on wired connections unless the link is saturated - most people don't do that often. When bufferbloat does arise - it's just (in some cases significantly) increased latency, it's not as if the connection goes down. Most people using most kinds of traffic won't even know it's taking place.
So, a problem that only fringe customers even know exists is exactly the kind of problem most ISP's aren't going to spend money on fixing.
Also, it isn't exactly difficult to make the problem go away on your end for those fringe customers who are aware of it.
-
I am here trying to figure out why it gets high latency only when i set the wan interface to 1gbps. This doesn't happen when i set to 100mbits full. but the problem is i can't set the duplex on the other end which is the Arris modem. My ISP locks the duplex port settings.
Yeah, +1 to Derelict just set it to auto negotiate and leave it alone.
-
the problem is when i set it to autoselect when i have alot of users downloading alot or if i start a downloading test the loss goes to 100% then the gateway goes offline. If that wasn't the problem then i would leave it alone.
I now have it set to autoselect but i have to disable gateway monitoring to let it stay up when the load is high.
