Danger, Latency on WAN mildly loaded

[jrey]

@dennypage said in Danger, Latency on WAN mildly loaded:

While you may not have experienced or noticed it

So are you saying that if that is the gateway monitor is using 8.8.8.8 and has run into a problem, it wouldn't be logged.. That seems wrong.

that would imply that dpinger isn't noticing it, and we both know that's not the case.

Not saying I have never seen a drop or issue with google, just not in a very long time.

my last loss record in the log is

Aug 30 16:34:03	dpinger	29452	WANGWE 8.8.8.8: Clear latency 7182us stddev 19975us loss 5%

and that wasn't really a "google" issue, that was the ISP tech guy playing outside LOL

My gateway hasn't dropped in months, and oh, it would be noticed 24/7 if not by dpinger as you seemed to suggest (JK) then numerous others would. On the other hand if I switch to 1.1.1.1 the gateway monitor will show lag and drop problems (mostly packet loss) constantly (even though the gateway is actually still up). Just look at the packet trace, not hard to figure out why.

Since the for IP addresses google uses 4 of them (2 ipv4 and 2 ipv6) for public are mapped to the nearest operational server by anycast routing, maybe the problem as you are hearing it, is just specific to locations experiencing latency.

The best recommendation is not one or the other, but rather something that works best at a given location. As you suggest something inside the local ISP would be best for a lot of users, I have two options hops 2 and 3, but displaying sub 2ms response times gets boring, so I opt for the 4ms choice inside google.

Google is not to my knowledge specifically throttling or blocking. I could ask. I'm well connected to a bunch of folks there. I can start the conversation with "So I hate google" Those friends always like that and then I end up buying a round. So on second thought - who cares right.

Pick what's best for your own situation and move on.

I'd like to comment on how simple it would be for dpinger to support a group monitor as has been suggested elsewhere today. But clearly there is no pressing need seen to add the feature and appears that it is already dead in the water. So moving on.

[dennypage]

@jrey said in Danger, Latency on WAN mildly loaded:

So are you saying that if that is the gateway monitor is using 8.8.8.8 and has run into a problem, it wouldn't be logged.. That seems wrong.

I think it's pretty clear what I am saying: A a number of people have run into problems using Google as a target, and I do not recommend its use. That is all. Your mileage may vary.

[coxhaus]

@dennypage
I would think your local ISP would be your best test for your local (ISP) LAN leg all else you have no control over. I use and like QUAD9 for DNS. It is maybe a little slower, but it serves a purpose.
I would not use Cloudflare for anything.

[dennypage]

@coxhaus said in Danger, Latency on WAN mildly loaded:

I would not use Cloudflare for anything.

Just out of curiosity, why is that? While I strongly recommend against using any of the general public services for latency/loss monitoring, I have found Cloudflare to be the most reliable among the public services for small ICMP packets. Have you had a bad experience with Cloudflare and ICMP packets?

[jrey]

@jrey said in Danger, Latency on WAN mildly loaded:

Google is not to my knowledge specifically throttling or blocking. I could ask.

@dennypage

Just FYI, Google will throttle DNS queries when the QPS goes above 1500 QPS for a sustained period of time (multiple seconds in a row). Short bursts above that are permitted. A ping is not considered as a DNS query.

Of course the choice of which monitor IP to use always depends on location.

Out of curiosity do you happen to know which anycast addresses 1.1.1.1 uses by location?

I'd actually like to see if I can find out why 1.1.1.1 is such a flaming pile of .. from this location, short of the zayo.com hole noted above (which is consistent) we can't use it for anything (way to slow). It is not something I need to fix, just curious. the 8.8.8.8 anycast addresses (and there are 4 of those that apply to my location) is less than 100km on the end of the fibre.

The wan never actually dropped through any of this, however the alarm is pretty much constantly spewing this for 1.1.1.1

Thanks

[dennypage]

@jrey Can you show traceroute info for 1.1.1.1 from your location? Both with and without -I.

[jrey]

@dennypage

Thanks

No need, the 1.1.1.1 has been identified after a long discussion with the tech folks there.

in the trace above, the issue is actually zayo (which I already knew)

Background, google anycast for here is located yyz (Toronto) a short hop from here about 110 km
on the other hand, the nearest 1.1.1.1 datacenter that is anycast from here is in Chicago. Not that is should matter at the speed of light ;-) However, the pipe between is those yayo 4 hops, and the pipe is too small, thus the delays spike at that point. Their words not mine ;-) They seem to know all about it or at least where they are pointing the finger, so to speak.

That said there is a CloudFlare data center scheduled to be completed in yyz q4 23, so end of year. However, I couldn't get clarity on which services were coming online in what order. But clearly by the end of Q1 24 I'd expect the latency issue from here should be gone.

The only point I was trying to make was that the choice of which one to use is something that can and will vary by location and people need to choose whichever is best suited for them, not just one or the other. Or as we have both stated something that is outside the gateway, but still inside the ISP if that is appropriate. Some folks will have smaller ISPs that don't like the constant pinging so people have to be cautious of that as well.

Currently I'll pick the one that gives consistent 4-5ms response
vs the one that is 15-20ms or not at all at peek times.

My recommendation to anyone, is always pick the one best for you by testing.

With that said, wouldn't it be nice if dpinger, could sample 1, 2 or 3 outside on a cycle instead of just 1. Then take any one of them responding as a good thing. The system would stop whining about being offline based on a single sample monitor location.
Sampling could for example, be 1-2-3 result, 1-2-3 result, if any of them respond the gateway is good.
Same cadence of ping as now, just in a cycle, the solo site that was used would also see less traffic as 2 other locations would be picking up cycle 2 3 in the cadence.

1.1.1.1, 8.8.8.8, w.x.y.z
fail , pass , pass = gateway up (1/3f or 2/3p)
pass, pass, fail = gateway up
pass, fail, fail = g.u. (2/3f 1/3p)
fail, fail, fail = sound the alarm (3f 0p), you toast)
etc.

given the current default cadence, even 1,2,3,4 would not be a bad option or even just 1,2 is better than a single. The number of options could be based on the number of ip addys in the monitor option field, allowing (none, 1-4 delimited list) instead of just (none or 1)

or as you said above

Your mileage may vary

[dennypage]

@jrey said in Danger, Latency on WAN mildly loaded:

With that said, wouldn't it be nice if dpinger, could sample 1, 2 or 3 outside on a cycle instead of just 1. Then take any one of them responding as a good thing. The system would stop whining about being offline based on a single sample monitor location.

Multi-target monitoring for dpinger has been discussed previously. It really isn't appropriate as part of dpinger itself, but is something that could possibly be implemented at the pfSense level.

There has been a bounty for pfSense multi-target gateway monitoring outstanding for years, which no one has picked up. A couple of folk tinkered with it for a while, but gave up on it due to the complexities involved. I think mostly the reason it hasn't come about is because single target monitoring is sufficient for 99% of real-world situations, so no one has felt sufficient pain to come up with an implementation.

[jim82]

Hi again everyone,

Thanks a lot for your great input. Now after running for some time with the Limiters on the WAN side, I have not really experienced any difference. I daily see the line completely dying or being swamped with high latency towards the DOCSIS modem.

Today I tried to reset the modem completely, but without luck. What I have figured out so far:

1. When the latency occurs, the modem is completely unavailable (web, icmp, etc.) Which leads me to thinks that this is the culprit.
2. Traffic limiters didn't seem to help, although bufferbloat measured to be more controlled

I will probably try to switch out the modem with my ISP, since I really can't see this being a pfSense issue at all.

Any thoughts?

Thanks
Jim

[jrey]

@jim82

Ah DOCSIS -

when I was on cable a while back, under load this is exactly the type of thing that would happen. Was never able to get answers reliable results with the default settings on the monitor

on the routing / Gateway
Edit - Display Advanced button

The settings you want to look at are these: (you'll need to find a balance) so adjust in small increments.

[jim82]

@jrey

Thanks a lot for pointing me in that direction. Am I right in assuming that getting these numbers just right, would fix the monitor issue with the gateway, but the root cause would still be there, ie. poor connection?

Because I also assume that I could mark the gateway as always being up as per below, to not invoke any actions.

So short and sweet, my issues will not be solved with gateway settings, but probably by hopefully being able to switch to fiber soon and/or a newer better modem.

Thanks
Jim

[jrey]

@jim82

Marking it as always up hides the when it really down. Although you'd likely know that without looking at the Dashboard.

Adjusting the monitor parameters a bit would allow you to continue monitoring, while at the same time not getting as many false latency/offline messages. It's not really down after all, it just the pings are delayed more under load and can't meet the threshold. Very common on a DOCSIS under load.

Edit: the description of the settings is here
https://docs.netgate.com/pfsense/en/latest/routing/gateway-configure.html

[dennypage]

@jrey said in Danger, Latency on WAN mildly loaded:

Marking it as always up hides the when it really down.

Before doing that, I would make sure shaping is properly set up. Follow this guide to set up CoDel.

NB: Waveform can be inaccurate with high bandwidth connections. Balance it against another test like Speedtest. Speedtest doesn't give you a grade like Waveform, but it does give averages under load.