Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Multiwan setup is constantly swapping my gateways

    Scheduled Pinned Locked Moved Routing and Multi WAN
    6 Posts 2 Posters 1.8k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A
      arengifo
      last edited by

      Hello guys:

      I'm running pfSense 2.3.1 in a Multi WAN scenario with 2 ISPs. I can barely describe my pfsense configuration:

      • GW_WAN: Gateway for ISP1 (this is my default gateway)
      • GW_WAN2: Gateway for ISP2 (backup)
      • GW_WAN is using 8.8.8.8 as monitoring IP, while GW_WAN2 is using 8.8.4.4 as monitoring IP.
      • Both gateways belong to a gateway group called "GW_HA" in which GW_WAN is Tier1 and GW_WAN2 is Tier2 (Trigger level: member down)
      • I have the "Enable default gateway switching" feature enabled.
      • I have the "State killing on gateway failure" feature disabled.

      Some days ago when both ISPs were working fine I decided to make a simple and short test. So I forced my ISP1 (through GW_WAN1) to fail and I saw how pfSense automatically switched the default gateway to ISP2 (through GW_WAN2) by using "netstat -nr | grep default". I thought this worked successfully as expected so I felt happy :). Also, all my firewall rules (previously configured to use GW_HA as gateway) worked fine too which allowed my clients to reach the Internet without issues (i.e. POP3/IMAP or SMTP connections continued working fine).
      IMPORTANT: This test was really fast. It took me no more than 1 or 2 minutes at most.

      But just yesterday my ISP1 had a real failure scenario (someone forgot to pay the bill, duh!) and I noticed that…

      • My firewall rules with GW_HA as gateway continued working fine, so Mail (IMAP, POP, SMTP) and other traffic switched fine through ISP2. This worked fine as expected.
      • My pfSense box started to swap the default gateway every 3 minutes. Most of the time I was able to see that even when ISP1 was down (according to Status->Gateways), the default gateway was yet pointing to GW_WAN instead of GW_WAN2.

      I've found these suspicious logs:

      Aug 12 08:11:36 fw check_reload_status: updating dyndns GW_WAN
      Aug 12 08:11:36 fw check_reload_status: Restarting ipsec tunnels
      Aug 12 08:11:36 fw check_reload_status: Restarting OpenVPN tunnels/interfaces
      Aug 12 08:11:36 fw check_reload_status: Reloading filter
      Aug 12 08:11:37 fw php-fpm[2693]: /rc.dyndns.update: MONITOR: GW_WAN2 is down, omitting from routing group GW_HA
      Aug 12 08:11:37 fw php-fpm[2693]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use GW_WAN.2 08:11:38 fw php-fpm[2693]: /rc.filter_configure_sync: MONITOR: GW_WAN2 is down, omitting from routing group GW_HA
      Aug 12 08:11:38 fw xinetd[14577]: Starting reconfiguration
      Aug 12 08:11:38 fw xinetd[14577]: Swapping defaults
      Aug 12 08:11:38 fw xinetd[14577]: readjusting service 6969-udp
      Aug 12 08:11:38 fw xinetd[14577]: Reconfigured: new=0 old=1 dropped=0 (services)
      Aug 12 08:12:06 fw check_reload_status: updating dyndns GW_WAN
      Aug 12 08:12:06 fw check_reload_status: Restarting ipsec tunnels
      Aug 12 08:12:06 fw check_reload_status: Restarting OpenVPN tunnels/interfaces
      Aug 12 08:12:06 fw check_reload_status: Reloading filter
      Aug 12 08:12:07 fw php-fpm[37361]: /rc.dyndns.update: MONITOR: GW_WAN is down, omitting from routing group GW_HA
      Aug 12 08:12:07 fw php-fpm[37361]: /rc.dyndns.update: MONITOR: GW_WAN2 is down, omitting from routing group GW_HA
      Aug 12 08:12:07 fw php-fpm[37361]: /rc.dyndns.update: Gateways status could not be determined, considering all as up/active. (Group: GW_HA)
      Aug 12 08:12:08 fw php-fpm[37361]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel CLOGmined, considering all as up/active. (Group: GW_HA)

      The "swapping defaults" and "Gateways status could not be determined, considering all as up/active" messages confused me and made think pfSense wasn't working fine on the "Default gateway switching" feature.

      I'd like someone can let me know if I'm missing an important configuration to fix this or it's maybe a bug.

      Thanks in advance for your time.

      1 Reply Last reply Reply Quote 0
      • A
        arengifo
        last edited by

        nobody? any ideas? am I facing a bug, maybe?

        1 Reply Last reply Reply Quote 0
        • jimpJ
          jimp Rebel Alliance Developer Netgate
          last edited by

          Check System > General Setup. make sure you don't have 8.8.8.8 / 8.8.4.4 there set to the opposite WANs as the monitor IP addresses. They have to match in both places.

          Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

          Need help fast? Netgate Global Support!

          Do not Chat/PM for help!

          1 Reply Last reply Reply Quote 0
          • A
            arengifo
            last edited by

            Sorry for the long delay. I'm not sure I got your point… Do you mean I should use the same address (e.g. 8.8.8.8) as monitoring IP for both gateways? I thought this would create a routing issue due to have the same route using different gateways... how is it supposed to monitor Internet availability if pfSense doesn't know which gateway to use for reaching 8.8.8.8?

            Thanks in advance.

            1 Reply Last reply Reply Quote 0
            • jimpJ
              jimp Rebel Alliance Developer Netgate
              last edited by

              No I mean just the opposite: Make sure you don't have anything that might be causing the same IP address to be used as a monitor for both WANs, which can confuse routing.

              Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

              Need help fast? Netgate Global Support!

              Do not Chat/PM for help!

              1 Reply Last reply Reply Quote 0
              • A
                arengifo
                last edited by

                Oh thanks. That's exactly what I did: I used different monitoring IPs for both gateways.

                1 Reply Last reply Reply Quote 0
                • First post
                  Last post
                Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.