Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Monitor is FALSE detecting one of my WANs as DOWN and another WAN as UP

    Scheduled Pinned Locked Moved Routing and Multi WAN
    39 Posts 7 Posters 3.1k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • D
      dims
      last edited by

      I have 3 WANs, each WAN has static outer IP address, given by provider.

      The topology is following:

      When I have no money on provider2 or I have any other problems with it, I can't ping IP2 from inside my LAN and from AWS hosting.

      Nevertheless, monitoring of this gateway indicates green.

      I.e. I have FALSE POSITIVE detection.

      An opposite is happening with provoder3. If I have everything OK with it, and I can ping IP3 from inside my LAN and from AWS hosting, apinger reports it is down.

      I.e. I have FALSE NEGATIVE detection.

      All gateways are cofigured as parts of one Load_Balancing_Group at tier 1.

      provider1 is configured as default and works ok.

      How it can be?

      1 Reply Last reply Reply Quote 0
      • C
        ccmks
        last edited by

        What are the monitor IP you use?

        1 Reply Last reply Reply Quote 0
        • D
          dims
          last edited by

          Outer IP of my provider. I can ping it from outside at the very same moment when apinger reporting it's down.

          1 Reply Last reply Reply Quote 0
          • R
            rudger_wolvram
            last edited by

            I had as similar problem with ATT, you couldn't ping their gateway IP from the same network (WAN interface IP) but you could from outside that subnet.
            I had to set up my monitor IP to 8.8.8.8 because it's google, they can handle the traffic, it's up pretty much all the time, and it monitors through the ISP gateway.
            Downside to this method, your interface response time reporting is skewed higher because you're hitting an actual internet host instead of the first hop.

            1 Reply Last reply Reply Quote 0
            • D
              dims
              last edited by

              I can ping IP3 from everywhere. Their router has WEB-interface, and it has PING page there. So, I tried to ping from:

              1. workstation inside LAN

              2. pfSense command line

              3. provider's router.

              Ping works from everywhere.

              Only apinger thinks interface is down, by unknown reason, probably BUG.

              1 Reply Last reply Reply Quote 0
              • DerelictD
                Derelict LAYER 8 Netgate
                last edited by

                Probably NOT a bug. Try pinging the monitor IP address from the firewall itself. Diagnostics > Ping or ping from the ssh/console shell.

                Chattanooga, Tennessee, USA
                A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                Do Not Chat For Help! NO_WAN_EGRESS(TM)

                1 Reply Last reply Reply Quote 0
                • D
                  dims
                  last edited by

                  If by "firewall" you mean pfSense box, that I can ping from it. See above.

                  1 Reply Last reply Reply Quote 0
                  • R
                    rudger_wolvram
                    last edited by

                    Wait, what version of pfsense are you on?
                    apinger was removed from pfsense somewhere around 2.3.x and replaced with dpinger. I'm on 2.4.2 and can't find apinger or dpinger (unless dpinger is the underlying pinger for gateways) packages.
                    You may be using a package that shouldn't even be there.

                    1 Reply Last reply Reply Quote 0
                    • DerelictD
                      Derelict LAYER 8 Netgate
                      last edited by

                      Yeah if you have a version that uses apinger, the solution is to upgrade. 2.4.2_1 is current.

                      Chattanooga, Tennessee, USA
                      A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                      DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                      Do Not Chat For Help! NO_WAN_EGRESS(TM)

                      1 Reply Last reply Reply Quote 0
                      • D
                        dims
                        last edited by

                        I am using 2.3.2-RELEASE (amd64)

                        I don't see 2.4.2_1 as upgrade option. It writes Latest Base System 2.3.3_1

                        If I enable unstable and experimental releases, it writes 2.3.6.a.20180103.1249

                        The date is yesterday.

                        Are you really pfSense guys, people?

                        1 Reply Last reply Reply Quote 0
                        • DerelictD
                          Derelict LAYER 8 Netgate
                          last edited by

                          2.3.2 does not have apinger, it has dpinger. I don't recall any issues with it since then.

                          You should upgrade anyway. Take a configuration backup and give it a go. The reported version from there does not always match what you end up with, unfortunately.

                          Chattanooga, Tennessee, USA
                          A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                          DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                          Do Not Chat For Help! NO_WAN_EGRESS(TM)

                          1 Reply Last reply Reply Quote 0
                          • D
                            dims
                            last edited by

                            Ah, I found newer version on site. Updater just doesn't see it…

                            1 Reply Last reply Reply Quote 0
                            • D
                              dims
                              last edited by

                              I don't beleive it will work. If this is not recognized as a bug or problem, then unprobably it was solved…

                              1 Reply Last reply Reply Quote 0
                              • DerelictD
                                Derelict LAYER 8 Netgate
                                last edited by

                                Are you really pfSense guys, people?

                                Insults? Really?

                                Chattanooga, Tennessee, USA
                                A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                1 Reply Last reply Reply Quote 0
                                • DerelictD
                                  Derelict LAYER 8 Netgate
                                  last edited by

                                  @dims:

                                  I don't beleive it will work. If this is not recognized as a bug or problem, then unprobably it was solved…

                                  That is because it is probably not a bug or a problem. You have a unique situation and you need to figure out what to monitor so you get the results you are looking for.

                                  Sometimes when an ISP administratively shuts down a circuit for things like "no more money" they still respond to pings for some close addresses, sometimes they hijack DNS or forward all port 80 "you're out of money" page, etc.

                                  Chattanooga, Tennessee, USA
                                  A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                  DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                  Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                  1 Reply Last reply Reply Quote 0
                                  • DerelictD
                                    Derelict LAYER 8 Netgate
                                    last edited by

                                    1. workstation inside LAN

                                    2. pfSense command line

                                    3. provider's router.

                                    And, to add some clarity. NOTHING but pinging from the firewall itself matters for gateway monitoring. That is the only case that has any impact on the monitoring process. It does not care if you can or cannot ping the target from AWS or LAN or the "provider's router."

                                    What do you have for DNS servers in System > General? Do you have gateways set on those?

                                    What do you have for monitor IP addresses on each gateway? Are they the same or different than the DNS servers and gateways?

                                    Are you trying to use any VPN endpoints as monitor addresses?

                                    Chattanooga, Tennessee, USA
                                    A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                    DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                    Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                    1 Reply Last reply Reply Quote 0
                                    • D
                                      dims
                                      last edited by

                                      @Derelict:

                                      Sometimes when an ISP administratively shuts down a circuit for things like "no more money" they still respond to pings

                                      This is not the case since I tried to ping by ping command line command.

                                      1 Reply Last reply Reply Quote 0
                                      • D
                                        dims
                                        last edited by

                                        @Derelict:

                                        NOTHING but pinging from the firewall itself matters for gateway monitoring.

                                        Okay. So ping from firewall works, while monitor says gateway is down. How can it be?

                                        What do you have for DNS servers in System > General?

                                        How DNS can affect pinging?

                                        Do you have gateways set on those?

                                        Of course not. DNS servers are given by each provider and the matter of change without notice. So I can't set static DNS addresses on General page. This is design error of pfSense (aka deliberate bug).

                                        What do you have for monitor IP addresses on each gateway?

                                        I am pinging my outer IPs for each provider. This is the only thing I can know, because I pay for them.

                                        Are they the same or different than the DNS servers and gateways?

                                        Of course they are different. I can't set DNS server to ping, because DNS server is not obliged to respong to pings.

                                        Are you trying to use any VPN endpoints as monitor addresses?

                                        I would write this, if I did.

                                        1 Reply Last reply Reply Quote 0
                                        • DerelictD
                                          Derelict LAYER 8 Netgate
                                          last edited by

                                          Because there are a lot of things that have to happen to make Multi-WAN and gateway monitoring work.

                                          All of the things I mentioned are because those are instances where the firewall has no choice but to install host routes out a specific interface.

                                          When you set a gateway monitor IP address, a host route is created to steer all traffic to that address out a specific interface.

                                          When you set a gateway on a DNS server in System > General the same thing happens a host route for that DNS server out that interface.

                                          When you set an interface on an IPsec configuration, the same thing happens.

                                          I know you think this should all "just work" but in your (uncommon, complicated) situation you have to have everything just right.

                                          So you can either listen and answer questions without all the snark, or don't. Completely up to you.

                                          Okay. So ping from firewall works, while monitor says gateway is down. How can it be?

                                          Show me a packet capture on that interface where the dpinger echo requests are being sent and replies are reliably being received and the gateway is still showing as down.

                                          Chattanooga, Tennessee, USA
                                          A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                          DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                          Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                          1 Reply Last reply Reply Quote 0
                                          • D
                                            dims
                                            last edited by

                                            Currently I tried to monitor the "gateway" host for the provider's modem. Situation with this address is the same: I can ping it from pfSense, but Monitor says it is down.

                                            Here is the screenshot of Monitor:

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.