Danger, Latency on WAN mildly loaded

stephenw10

Could be a crappy modem. Could just be buffer bloat, though usually that doesn't drop packets like that.

Try an online buffer-bloat test first.

Steve

jim82

@stephenw10 said in Danger, Latency on WAN mildly loaded:

Could be a crappy modem. Could just be buffer bloat, though usually that doesn't drop packets like that.

Try an online buffer-bloat test first.

Steve

Hi Steve,

Thanks for your reply. I have tried the bufferbloat test, and it seems like that could be my culprit. Here's the result

How do I go about mitigating this issue in pfSense?

Thanks
Jim

stephenw10

The only thing you can do in pfSense is to apply traffic shaping so that any buffering is done there where it can be controlled.

In situations where it's bad using CoDelQ can be quite effective:
https://docs.netgate.com/pfsense/en/latest/trafficshaper/altq-scheduler-types.html#codel-active-queue-management

I will point out that the 100ms shown in that test is far from the 2500ms seen by dpinger though.

jim82

@stephenw10 said in Danger, Latency on WAN mildly loaded:

The only thing you can do in pfSense is to apply traffic shaping so that any buffering is done there where it can be controlled.

In situations where it's bad using CoDelQ can be quite effective:
https://docs.netgate.com/pfsense/en/latest/trafficshaper/altq-scheduler-types.html#codel-active-queue-management

I will point out that the 100ms shown in that test is far from the 2500ms seen by dpinger though.

Thanks again, Steve. I will look into the traffic shaping, although it's never been needed before.

I'm also confused about the latency of the test, as it's nowhere near the problem depicted. Kinda lost as to where to start troubleshooting now.

/Jim

stephenw10

The route to the nearest 8.8.8.8 anycast host could be different to the test server so you might not be hitting the worst bloat there. I would try a few different test servers if you can.

jrey

@jim82

Could be a crappy modem.

or how about just a crappy cable?
or wrong duplex being negotiated?
or wrong speed being negotiated?

You won't potentially see the issue until you load it up.

jim82

@stephenw10 said in Danger, Latency on WAN mildly loaded:

The route to the nearest 8.8.8.8 anycast host could be different to the test server so you might not be hitting the worst bloat there. I would try a few different test servers if you can.

You mean changing my monitor IP for the gateway or doing bufferbloat test to another location, which I don't see as an option using the Waveform site?

@jrey said in Danger, Latency on WAN mildly loaded:

@jim82

Could be a crappy modem.

or how about just a crappy cable?
or wrong duplex being negotiated?
or wrong speed being negotiated?

You won't potentially see the issue until you load it up.

Thanks for your insight, jrey. You're right, sometimes it's the simplest things. I have checked the negotiated speeds and they're 1000Mbit full-duplex.

By cable I assume you mean the one going from the modem to the WAN port of pfSense, right? I understand that it could in theory be all cables, ranging from the one in my PC, to the installations in the attic and of course panels, switches, etc. OOF :)

jrey

@jim82 said in Danger, Latency on WAN mildly loaded:

from the modem to the WAN port of pfSense

If you are seeing the problem on all clients (which it appears you are based on the gateway screen showing the packet loss) then yes, that cable.

might have been pinched, crimped whatever. doesn't hurt to just try a different one.

I had a streaming device at the house once upon a time that when I was on a cable provider never showed a problem. (the cable speed wasn't fast enough to push the limit anyway) However. when I switch to fibre at the house, the streaming device lost it, couldn't attain any real speed even though it should. J Just a speed test, it was limping bad.
Pull out the trusty cable tester, well darn a twisted / broken wire. Chopped off the end, crimped on a new one - issue resolved. No throttle required the device can now hit the limit of the network card with no jitter / packet drop etc -
so yup sometimes just a simple thing, like a cable.

stephenw10

@jim82 said in Danger, Latency on WAN mildly loaded:

You mean changing my monitor IP for the gateway or doing bufferbloat test to another location

I meant test against some other speedtest server that reports latency under load. But changing the monitoring IP is also a good test here.

jim82

@stephenw10

I have attached a "flent" graph I made from an ubuntu host. I'm not really sure if it's good, or bad?

dennypage

@jim82 Google is no longer a reliable target for latency/loss monitoring. They seem to have become cranky about people doing this, and have started dropping packets.

FWIW, I personally use a regional router inside my ISP for dpinger monitoring. I suggest using

traceroute -I 1.1.1.1

to discover a suitable target. If you have multiple WAN interfaces, add the -s option to traceroute to ensure you are looking at the correct interface

For people who absolutely need a target beyond their ISP, I generally recommend Cloudflare (1.1.1.1) instead of Google.

jrey

@jim82

go with stephenw10 suggestion

But changing the monitoring IP is also a good test here.

The trace route might reveal a lot.

I have never had a problem with 8.8.8.8 and would likely not pick 1.1.1.1 based on what is normal at my location.

of course in both cases, hops 1-3 are my gateway and my ISP - so hop 4 is where the fun begins.

so in my case pick up the trace route at hop4 first to 1.1.1.1

 4  209.249.67.221.IDIA-251658-ZYO.zip.zayo.com (209.249.67.221)  14.524 ms  14.654 ms  14.438 ms
 5  * * *
 6  * * *
 7  208.184.122.49.IPYX-330228-005-ZYO.zip.zayo.com (208.184.122.49)  15.811 ms  16.006 ms *
 8  172.70.124.4 (172.70.124.4)  18.895 ms  16.237 ms  16.229 ms
 9  one.one.one.one (1.1.1.1)  15.829 ms  15.918 ms  15.884 ms

the above is pretty typical of things coming out of yayo going anywhere on their path 5 and 6 are pretty much always stars with no meat on the bones.

now pick up from hop 4 to 8.8.8.8 the ping time basically doubles from hop 3 to hop 4, pretty consistent typically anything inside my ISP is <2ms and jumps to 4ish at hop 4. "so I blame google"

 4  google.ip4.torontointernetxchange.net (206.108.34.6)  4.254 ms  4.606 ms  4.395 ms
 5  108.170.250.241 (108.170.250.241)  5.659 ms  5.508 ms  5.254 ms
 6  216.239.35.233 (216.239.35.233)  5.358 ms  5.430 ms  5.000 ms
 7  dns.google (8.8.8.8)  4.239 ms  4.606 ms  4.458 ms

I've not really noticed any specific throttle by google and have used them as the monitor IP for months. Of course this will vary by location. Drop a trace or two.

Day over day, month over month these are pretty typical results here.

dennypage

@jrey said in Danger, Latency on WAN mildly loaded:

I have never had a problem with 8.8.8.8

I tend to hear about these issues. While you may not have experienced or noticed it, a number of people have run into issues with Google recently. I do not recommend using it as a target for monitoring.

jrey

@dennypage said in Danger, Latency on WAN mildly loaded:

While you may not have experienced or noticed it

So are you saying that if that is the gateway monitor is using 8.8.8.8 and has run into a problem, it wouldn't be logged.. That seems wrong.

that would imply that dpinger isn't noticing it, and we both know that's not the case.

Not saying I have never seen a drop or issue with google, just not in a very long time.

my last loss record in the log is

Aug 30 16:34:03	dpinger	29452	WANGWE 8.8.8.8: Clear latency 7182us stddev 19975us loss 5%

and that wasn't really a "google" issue, that was the ISP tech guy playing outside LOL

My gateway hasn't dropped in months, and oh, it would be noticed 24/7 if not by dpinger as you seemed to suggest (JK) then numerous others would. On the other hand if I switch to 1.1.1.1 the gateway monitor will show lag and drop problems (mostly packet loss) constantly (even though the gateway is actually still up). Just look at the packet trace, not hard to figure out why.

Since the for IP addresses google uses 4 of them (2 ipv4 and 2 ipv6) for public are mapped to the nearest operational server by anycast routing, maybe the problem as you are hearing it, is just specific to locations experiencing latency.

The best recommendation is not one or the other, but rather something that works best at a given location. As you suggest something inside the local ISP would be best for a lot of users, I have two options hops 2 and 3, but displaying sub 2ms response times gets boring, so I opt for the 4ms choice inside google.

Google is not to my knowledge specifically throttling or blocking. I could ask. I'm well connected to a bunch of folks there. I can start the conversation with "So I hate google" Those friends always like that and then I end up buying a round. So on second thought - who cares right.

Pick what's best for your own situation and move on.

I'd like to comment on how simple it would be for dpinger to support a group monitor as has been suggested elsewhere today. But clearly there is no pressing need seen to add the feature and appears that it is already dead in the water. So moving on.

dennypage

@jrey said in Danger, Latency on WAN mildly loaded:

So are you saying that if that is the gateway monitor is using 8.8.8.8 and has run into a problem, it wouldn't be logged.. That seems wrong.

I think it's pretty clear what I am saying: A a number of people have run into problems using Google as a target, and I do not recommend its use. That is all. Your mileage may vary.

coxhaus

@dennypage
I would think your local ISP would be your best test for your local (ISP) LAN leg all else you have no control over. I use and like QUAD9 for DNS. It is maybe a little slower, but it serves a purpose.
I would not use Cloudflare for anything.

dennypage

@coxhaus said in Danger, Latency on WAN mildly loaded:

I would not use Cloudflare for anything.

Just out of curiosity, why is that? While I strongly recommend against using any of the general public services for latency/loss monitoring, I have found Cloudflare to be the most reliable among the public services for small ICMP packets. Have you had a bad experience with Cloudflare and ICMP packets?

jrey

@jrey said in Danger, Latency on WAN mildly loaded:

Google is not to my knowledge specifically throttling or blocking. I could ask.

@dennypage

Just FYI, Google will throttle DNS queries when the QPS goes above 1500 QPS for a sustained period of time (multiple seconds in a row). Short bursts above that are permitted. A ping is not considered as a DNS query.

Of course the choice of which monitor IP to use always depends on location.

Out of curiosity do you happen to know which anycast addresses 1.1.1.1 uses by location?

I'd actually like to see if I can find out why 1.1.1.1 is such a flaming pile of .. from this location, short of the zayo.com hole noted above (which is consistent) we can't use it for anything (way to slow). It is not something I need to fix, just curious. the 8.8.8.8 anycast addresses (and there are 4 of those that apply to my location) is less than 100km on the end of the fibre.

The wan never actually dropped through any of this, however the alarm is pretty much constantly spewing this for 1.1.1.1

Screen Shot 2023-11-29 at 8.39.23 AM.png

Thanks

dennypage

@jrey Can you show traceroute info for 1.1.1.1 from your location? Both with and without -I.

jrey

@dennypage

Thanks

No need, the 1.1.1.1 has been identified after a long discussion with the tech folks there.

in the trace above, the issue is actually zayo (which I already knew)

Background, google anycast for here is located yyz (Toronto) a short hop from here about 110 km
on the other hand, the nearest 1.1.1.1 datacenter that is anycast from here is in Chicago. Not that is should matter at the speed of light ;-) However, the pipe between is those yayo 4 hops, and the pipe is too small, thus the delays spike at that point. Their words not mine ;-) They seem to know all about it or at least where they are pointing the finger, so to speak.

That said there is a CloudFlare data center scheduled to be completed in yyz q4 23, so end of year. However, I couldn't get clarity on which services were coming online in what order. But clearly by the end of Q1 24 I'd expect the latency issue from here should be gone.

The only point I was trying to make was that the choice of which one to use is something that can and will vary by location and people need to choose whichever is best suited for them, not just one or the other. Or as we have both stated something that is outside the gateway, but still inside the ISP if that is appropriate. Some folks will have smaller ISPs that don't like the constant pinging so people have to be cautious of that as well.

Currently I'll pick the one that gives consistent 4-5ms response
vs the one that is 15-20ms or not at all at peek times.

My recommendation to anyone, is always pick the one best for you by testing.

With that said, wouldn't it be nice if dpinger, could sample 1, 2 or 3 outside on a cycle instead of just 1. Then take any one of them responding as a good thing. The system would stop whining about being offline based on a single sample monitor location.
Sampling could for example, be 1-2-3 result, 1-2-3 result, if any of them respond the gateway is good.
Same cadence of ping as now, just in a cycle, the solo site that was used would also see less traffic as 2 other locations would be picking up cycle 2 3 in the cadence.

1.1.1.1, 8.8.8.8, w.x.y.z
fail , pass , pass = gateway up (1/3f or 2/3p)
pass, pass, fail = gateway up
pass, fail, fail = g.u. (2/3f 1/3p)
fail, fail, fail = sound the alarm (3f 0p), you toast)
etc.

given the current default cadence, even 1,2,3,4 would not be a bad option or even just 1,2 is better than a single. The number of options could be based on the number of ip addys in the monitor option field, allowing (none, 1-4 delimited list) instead of just (none or 1)

or as you said above

Your mileage may vary