Danger, Latency on WAN mildly loaded

[stephenw10]

The route to the nearest 8.8.8.8 anycast host could be different to the test server so you might not be hitting the worst bloat there. I would try a few different test servers if you can.

[jrey]

@jim82

Could be a crappy modem.

or how about just a crappy cable?
or wrong duplex being negotiated?
or wrong speed being negotiated?

You won't potentially see the issue until you load it up.

[jim82]

@stephenw10 said in Danger, Latency on WAN mildly loaded:

The route to the nearest 8.8.8.8 anycast host could be different to the test server so you might not be hitting the worst bloat there. I would try a few different test servers if you can.

You mean changing my monitor IP for the gateway or doing bufferbloat test to another location, which I don't see as an option using the Waveform site?

@jrey said in Danger, Latency on WAN mildly loaded:

@jim82

Could be a crappy modem.

or how about just a crappy cable?
or wrong duplex being negotiated?
or wrong speed being negotiated?

You won't potentially see the issue until you load it up.

Thanks for your insight, jrey. You're right, sometimes it's the simplest things. I have checked the negotiated speeds and they're 1000Mbit full-duplex.

By cable I assume you mean the one going from the modem to the WAN port of pfSense, right? I understand that it could in theory be all cables, ranging from the one in my PC, to the installations in the attic and of course panels, switches, etc. OOF :)

[jrey]

@jim82 said in Danger, Latency on WAN mildly loaded:

from the modem to the WAN port of pfSense

If you are seeing the problem on all clients (which it appears you are based on the gateway screen showing the packet loss) then yes, that cable.

might have been pinched, crimped whatever. doesn't hurt to just try a different one.

I had a streaming device at the house once upon a time that when I was on a cable provider never showed a problem. (the cable speed wasn't fast enough to push the limit anyway) However. when I switch to fibre at the house, the streaming device lost it, couldn't attain any real speed even though it should. J Just a speed test, it was limping bad.
Pull out the trusty cable tester, well darn a twisted / broken wire. Chopped off the end, crimped on a new one - issue resolved. No throttle required the device can now hit the limit of the network card with no jitter / packet drop etc -
so yup sometimes just a simple thing, like a cable.

[stephenw10]

@jim82 said in Danger, Latency on WAN mildly loaded:

You mean changing my monitor IP for the gateway or doing bufferbloat test to another location

I meant test against some other speedtest server that reports latency under load. But changing the monitoring IP is also a good test here.

[jim82]

@stephenw10

I have attached a "flent" graph I made from an ubuntu host. I'm not really sure if it's good, or bad?

[dennypage]

@jim82 Google is no longer a reliable target for latency/loss monitoring. They seem to have become cranky about people doing this, and have started dropping packets.

FWIW, I personally use a regional router inside my ISP for dpinger monitoring. I suggest using

traceroute -I 1.1.1.1

to discover a suitable target. If you have multiple WAN interfaces, add the -s option to traceroute to ensure you are looking at the correct interface

For people who absolutely need a target beyond their ISP, I generally recommend Cloudflare (1.1.1.1) instead of Google.

[jrey]

@jim82

go with stephenw10 suggestion

But changing the monitoring IP is also a good test here.

The trace route might reveal a lot.

I have never had a problem with 8.8.8.8 and would likely not pick 1.1.1.1 based on what is normal at my location.

of course in both cases, hops 1-3 are my gateway and my ISP - so hop 4 is where the fun begins.

so in my case pick up the trace route at hop4 first to 1.1.1.1

 4  209.249.67.221.IDIA-251658-ZYO.zip.zayo.com (209.249.67.221)  14.524 ms  14.654 ms  14.438 ms
 5  * * *
 6  * * *
 7  208.184.122.49.IPYX-330228-005-ZYO.zip.zayo.com (208.184.122.49)  15.811 ms  16.006 ms *
 8  172.70.124.4 (172.70.124.4)  18.895 ms  16.237 ms  16.229 ms
 9  one.one.one.one (1.1.1.1)  15.829 ms  15.918 ms  15.884 ms

the above is pretty typical of things coming out of yayo going anywhere on their path 5 and 6 are pretty much always stars with no meat on the bones.

now pick up from hop 4 to 8.8.8.8 the ping time basically doubles from hop 3 to hop 4, pretty consistent typically anything inside my ISP is <2ms and jumps to 4ish at hop 4. "so I blame google"

 4  google.ip4.torontointernetxchange.net (206.108.34.6)  4.254 ms  4.606 ms  4.395 ms
 5  108.170.250.241 (108.170.250.241)  5.659 ms  5.508 ms  5.254 ms
 6  216.239.35.233 (216.239.35.233)  5.358 ms  5.430 ms  5.000 ms
 7  dns.google (8.8.8.8)  4.239 ms  4.606 ms  4.458 ms

I've not really noticed any specific throttle by google and have used them as the monitor IP for months. Of course this will vary by location. Drop a trace or two.

Day over day, month over month these are pretty typical results here.

[dennypage]

@jrey said in Danger, Latency on WAN mildly loaded:

I have never had a problem with 8.8.8.8

I tend to hear about these issues. While you may not have experienced or noticed it, a number of people have run into issues with Google recently. I do not recommend using it as a target for monitoring.

[jrey]

@dennypage said in Danger, Latency on WAN mildly loaded:

While you may not have experienced or noticed it

So are you saying that if that is the gateway monitor is using 8.8.8.8 and has run into a problem, it wouldn't be logged.. That seems wrong.

that would imply that dpinger isn't noticing it, and we both know that's not the case.

Not saying I have never seen a drop or issue with google, just not in a very long time.

my last loss record in the log is

Aug 30 16:34:03	dpinger	29452	WANGWE 8.8.8.8: Clear latency 7182us stddev 19975us loss 5%

and that wasn't really a "google" issue, that was the ISP tech guy playing outside LOL

My gateway hasn't dropped in months, and oh, it would be noticed 24/7 if not by dpinger as you seemed to suggest (JK) then numerous others would. On the other hand if I switch to 1.1.1.1 the gateway monitor will show lag and drop problems (mostly packet loss) constantly (even though the gateway is actually still up). Just look at the packet trace, not hard to figure out why.

Since the for IP addresses google uses 4 of them (2 ipv4 and 2 ipv6) for public are mapped to the nearest operational server by anycast routing, maybe the problem as you are hearing it, is just specific to locations experiencing latency.

The best recommendation is not one or the other, but rather something that works best at a given location. As you suggest something inside the local ISP would be best for a lot of users, I have two options hops 2 and 3, but displaying sub 2ms response times gets boring, so I opt for the 4ms choice inside google.

Google is not to my knowledge specifically throttling or blocking. I could ask. I'm well connected to a bunch of folks there. I can start the conversation with "So I hate google" Those friends always like that and then I end up buying a round. So on second thought - who cares right.

Pick what's best for your own situation and move on.

I'd like to comment on how simple it would be for dpinger to support a group monitor as has been suggested elsewhere today. But clearly there is no pressing need seen to add the feature and appears that it is already dead in the water. So moving on.

[dennypage]

@jrey said in Danger, Latency on WAN mildly loaded:

So are you saying that if that is the gateway monitor is using 8.8.8.8 and has run into a problem, it wouldn't be logged.. That seems wrong.

I think it's pretty clear what I am saying: A a number of people have run into problems using Google as a target, and I do not recommend its use. That is all. Your mileage may vary.

[coxhaus]

@dennypage
I would think your local ISP would be your best test for your local (ISP) LAN leg all else you have no control over. I use and like QUAD9 for DNS. It is maybe a little slower, but it serves a purpose.
I would not use Cloudflare for anything.

[dennypage]

@coxhaus said in Danger, Latency on WAN mildly loaded:

I would not use Cloudflare for anything.

Just out of curiosity, why is that? While I strongly recommend against using any of the general public services for latency/loss monitoring, I have found Cloudflare to be the most reliable among the public services for small ICMP packets. Have you had a bad experience with Cloudflare and ICMP packets?

[jrey]

@jrey said in Danger, Latency on WAN mildly loaded:

Google is not to my knowledge specifically throttling or blocking. I could ask.

@dennypage

Just FYI, Google will throttle DNS queries when the QPS goes above 1500 QPS for a sustained period of time (multiple seconds in a row). Short bursts above that are permitted. A ping is not considered as a DNS query.

Of course the choice of which monitor IP to use always depends on location.

Out of curiosity do you happen to know which anycast addresses 1.1.1.1 uses by location?

I'd actually like to see if I can find out why 1.1.1.1 is such a flaming pile of .. from this location, short of the zayo.com hole noted above (which is consistent) we can't use it for anything (way to slow). It is not something I need to fix, just curious. the 8.8.8.8 anycast addresses (and there are 4 of those that apply to my location) is less than 100km on the end of the fibre.

The wan never actually dropped through any of this, however the alarm is pretty much constantly spewing this for 1.1.1.1

Thanks

[dennypage]

@jrey Can you show traceroute info for 1.1.1.1 from your location? Both with and without -I.

[jrey]

@dennypage

Thanks

No need, the 1.1.1.1 has been identified after a long discussion with the tech folks there.

in the trace above, the issue is actually zayo (which I already knew)

Background, google anycast for here is located yyz (Toronto) a short hop from here about 110 km
on the other hand, the nearest 1.1.1.1 datacenter that is anycast from here is in Chicago. Not that is should matter at the speed of light ;-) However, the pipe between is those yayo 4 hops, and the pipe is too small, thus the delays spike at that point. Their words not mine ;-) They seem to know all about it or at least where they are pointing the finger, so to speak.

That said there is a CloudFlare data center scheduled to be completed in yyz q4 23, so end of year. However, I couldn't get clarity on which services were coming online in what order. But clearly by the end of Q1 24 I'd expect the latency issue from here should be gone.

The only point I was trying to make was that the choice of which one to use is something that can and will vary by location and people need to choose whichever is best suited for them, not just one or the other. Or as we have both stated something that is outside the gateway, but still inside the ISP if that is appropriate. Some folks will have smaller ISPs that don't like the constant pinging so people have to be cautious of that as well.

Currently I'll pick the one that gives consistent 4-5ms response
vs the one that is 15-20ms or not at all at peek times.

My recommendation to anyone, is always pick the one best for you by testing.

With that said, wouldn't it be nice if dpinger, could sample 1, 2 or 3 outside on a cycle instead of just 1. Then take any one of them responding as a good thing. The system would stop whining about being offline based on a single sample monitor location.
Sampling could for example, be 1-2-3 result, 1-2-3 result, if any of them respond the gateway is good.
Same cadence of ping as now, just in a cycle, the solo site that was used would also see less traffic as 2 other locations would be picking up cycle 2 3 in the cadence.

1.1.1.1, 8.8.8.8, w.x.y.z
fail , pass , pass = gateway up (1/3f or 2/3p)
pass, pass, fail = gateway up
pass, fail, fail = g.u. (2/3f 1/3p)
fail, fail, fail = sound the alarm (3f 0p), you toast)
etc.

given the current default cadence, even 1,2,3,4 would not be a bad option or even just 1,2 is better than a single. The number of options could be based on the number of ip addys in the monitor option field, allowing (none, 1-4 delimited list) instead of just (none or 1)

or as you said above

Your mileage may vary

[dennypage]

@jrey said in Danger, Latency on WAN mildly loaded:

With that said, wouldn't it be nice if dpinger, could sample 1, 2 or 3 outside on a cycle instead of just 1. Then take any one of them responding as a good thing. The system would stop whining about being offline based on a single sample monitor location.

Multi-target monitoring for dpinger has been discussed previously. It really isn't appropriate as part of dpinger itself, but is something that could possibly be implemented at the pfSense level.

There has been a bounty for pfSense multi-target gateway monitoring outstanding for years, which no one has picked up. A couple of folk tinkered with it for a while, but gave up on it due to the complexities involved. I think mostly the reason it hasn't come about is because single target monitoring is sufficient for 99% of real-world situations, so no one has felt sufficient pain to come up with an implementation.

[jim82]

Hi again everyone,

Thanks a lot for your great input. Now after running for some time with the Limiters on the WAN side, I have not really experienced any difference. I daily see the line completely dying or being swamped with high latency towards the DOCSIS modem.

Today I tried to reset the modem completely, but without luck. What I have figured out so far:

1. When the latency occurs, the modem is completely unavailable (web, icmp, etc.) Which leads me to thinks that this is the culprit.
2. Traffic limiters didn't seem to help, although bufferbloat measured to be more controlled

I will probably try to switch out the modem with my ISP, since I really can't see this being a pfSense issue at all.

Any thoughts?

Thanks
Jim

[jrey]

@jim82

Ah DOCSIS -

when I was on cable a while back, under load this is exactly the type of thing that would happen. Was never able to get answers reliable results with the default settings on the monitor

on the routing / Gateway
Edit - Display Advanced button

The settings you want to look at are these: (you'll need to find a balance) so adjust in small increments.

[jim82]

@jrey

Thanks a lot for pointing me in that direction. Am I right in assuming that getting these numbers just right, would fix the monitor issue with the gateway, but the root cause would still be there, ie. poor connection?

Because I also assume that I could mark the gateway as always being up as per below, to not invoke any actions.

So short and sweet, my issues will not be solved with gateway settings, but probably by hopefully being able to switch to fiber soon and/or a newer better modem.

Thanks
Jim