Monitor IP Discussion

[Raffi_]

I have seen different opinions on the topic of which IP to monitor. For the sake of this discussion, this is mainly important for the failover scenario.

The main objective,
Monitor IP for the main link will only mark the gateway down when it is actually down and the link to the web is broken.

Some of the most common suggestions I have seen are,

1. The default setting of pfSense to monitor the ISP local device. This will not necessarily satisfy the main objective.
2. Monitoring the second or third hop on the ISP's network. This could potentially satisfy the main objective.
3. Monitor an anycast server such as 8.8.8.8. This could potentially satisfy the main objective.
4. Monitor a server on the web you have control over. This could potentially satisfy the main objective.

In my personal experience, I tried doing #2 and it worked fine for a short while. The ISP then decided to remove that router from my path and the link was permanently marked down. I am now currently doing #3 with google DNS and it has been working fine. Occasionally, there is delay or lost packets, but generally not long enough to mark the gateway down after minor tweaks to the thresholds.

Please let us know what you personally do, and more importantly why.
For example, did you try monitoring 8.8.8.8 or 1.1.1.1 and have trouble with that setup? Was it a brief issue or enough to take the link down for several minutes?
Any suggestions like is there a better public server to monitor other than 8.8.8.8, etc., for those that don't have control over a server out on the web.

Thanks,
Raffi

[netblues]

A fast dpinger, at 500msec can detect very small outages and issues before they become problems.
This means pinging as close as possible to the other end of the physical line. (the one that you need to report, when problems arise).
Now, in a failover scenario, if you are multihomed to different isp's, and isp looses connectivity all together, then you need to ping something reliable but not on your isp network.

In practice, isp's are already multihomed and run bgp so they can handle such situations gracefully. Trying to do the same on your own is a bit far fetched.
If you don't trust your isp on its core business, then you shouldn't be a customer in the first place.

[johnpoz]

I just monitor isp gateway.. I don't see a reason to do it any different. If isp goes offline for some or all of the internet.. I would know from my monitoring of my connection from the outside.

I sure and the hell do not want pfsense to go offline or think its offline because some 1 specific remote site goes offline, etc.

[dotdash]

I'll don the nomex and join in. Let's address a couple of the points brought up.
1- Monitor your ISP, if you don't trust them get someone else.
I don't trust my ISP. That's why I have two and need to monitor when they go down. There is often not a lot of choice of providers, so you can't just tell people to get a trusted ISP (is there such a thing).
2- If the ISP goes offline I'll know, because I'm monitoring.
Ok, fine. I personally like to have customers gracefully fail over without intervention.
3- I don't want the firewall to go offline because some remote site went offline.
Fair enough. Based on my own experiences, public DNS servers do not go offline nearly as often as an ISP has a regional outage or somesuch. As I said in the other thread- see what works for you, make your own decisions. I don't think blanket statements like 'never use public DNS for a monitor' are useful. Configurations and needs vary.

[Raffi_]

Thanks all for responses.

I have the same train of thought as @dotdash due to my own experiences.

@netblues said in Monitor IP Discussion:

In practice, isp's are already multihomed and run bgp so they can handle such situations gracefully. Trying to do the same on your own is a bit far fetched.
If you don't trust your isp on its core business, then you shouldn't be a customer in the first place.

Is this assuming your ISP can offer redundant connections? For example, in our scenario we have a cable link and a 4G LTE modem for emergency failover.

@johnpoz said in Monitor IP Discussion:

I just monitor isp gateway.. I don't see a reason to do it any different. If isp goes offline for some or all of the internet.. I would know from my monitoring of my connection from the outside.

When you say you monitor the ISP gateway I'm assuming you mean e.g. the modem on your side? This is the default setting of pfSense.

[chpalmer]

@johnpoz said in Monitor IP Discussion:

I don't see a reason to do it any different

Verizon as an example does not respond to pings anywhere in their network when using my Cradlepoint.. The first response is outside their network over 100ms away for me. I imagine there are other examples such as this.

[johnpoz]

@Raffi_ said in Monitor IP Discussion:

the modem on your side?

No I mean the isp device.. I am on cable, I get a public IP the first hop is isp router.. about 10ms away.. Cable modems do nothing but bridge the connection from coax to ethernet.

@chpalmer
Well if you can not ping your entry into your isp network, ie their router - then yeah you would need to adjust ;) The first thing you can ping is 100ms away - where are you? The middle of the pacific ocean ? ;)

[chpalmer]

@johnpoz said in Monitor IP Discussion:

@Raffi_ said in Monitor IP Discussion:
;) The first thing you can ping is 100ms away - where are you?

Our primary stuff is actually on cable.. (and no problem pinging first gateway there) MB8600 modem that I use with a bonded. As a commercial customer I could get multiple IP's for multiple devices. But since I use it bonded I can not seem to use the other two ports on the modem for my testbox. (4 port switch on this model.) Therefore the Cradlepoint for that.

This is the results from Verizon.. Im glad this got me testing today as the results are different than in the past.. Im pinging 9.9.9.9 as my gateway as I type this. If I try to ping 69.83.157.179 (5th line second traceroute) then my gateway shows as down. Same result pinging the last router in VZW's network on that traceroute. Im not sure how I can ping them here but not monitor them but alas..

Pinging 140.222.238.81 from the results below does work for now.. but seems to me from memory that it will eventually begin to time out. Ill try it for a while. At least Im closer to 50ms that way. Pinging 108.170.233.159 again shows offline.

Traceroute
1 * * *
2 * * *
3 * * *
4 179.sub-69-83-157.myvzw.com (69.83.157.179) 41.965 ms 45.834 ms 50.149 ms
5 * * *
6 116.sub-69-83-144.myvzw.com (69.83.144.116) 41.193 ms 39.695 ms 72.236 ms
7 130.sub-69-83-157.myvzw.com (69.83.157.130) 47.740 ms 63.426 ms 58.698 ms
8 0.ae5.GW9.SEA1.ALTER.NET (140.222.238.81) 51.428 ms
0.ae6.GW9.SEA1.ALTER.NET (140.222.238.83) 53.393 ms
0.ae5.GW9.SEA1.ALTER.NET (140.222.238.81) 38.071 ms
9 72.14.212.6 (72.14.212.6) 41.645 ms 46.284 ms 45.242 ms
10 108.170.245.113 (108.170.245.113) 27.957 ms
108.170.245.97 (108.170.245.97) 31.504 ms
108.170.245.113 (108.170.245.113) 39.907 ms
11 108.170.233.159 (108.170.233.159) 37.615 ms 40.094 ms 36.394 ms
12 sea15s11-in-f174.1e100.net (172.217.3.174) 37.369 ms 33.187 ms 35.628 ms

Traceroute using ICMP

1 * * *
2 * * *
3 * * *
4 * * *
5 179.sub-69-83-157.myvzw.com (69.83.157.179) 29.558 ms 39.744 ms 37.850 ms
6 * * *
7 116.sub-69-83-144.myvzw.com (69.83.144.116) 42.580 ms 37.752 ms 39.988 ms
8 130.sub-69-83-157.myvzw.com (69.83.157.130) 44.793 ms 36.815 ms 50.387 ms
9 0.ae6.GW9.SEA1.ALTER.NET (140.222.238.83) 48.724 ms 40.587 ms 40.182 ms
10 72.14.212.6 (72.14.212.6) 41.785 ms 32.608 ms 43.139 ms
11 108.170.245.113 (108.170.245.113) 36.703 ms 34.020 ms 41.968 ms
12 108.170.233.159 (108.170.233.159) 38.076 ms 32.065 ms 39.785 ms
13 sea15s11-in-f174.1e100.net (172.217.3.174) 38.072 ms 41.824 ms 41.835 ms

[chpalmer]

And actually- Ive been wanting to experiment a little with the cable modem and have now figured out that issue as well..

Thanks Raffi_ You got me working on this and Ive solved an issue now.