Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    CARP Failover not Working on Manual Outbound NAT

    Scheduled Pinned Locked Moved HA/CARP/VIPs
    20 Posts 2 Posters 9.2k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A
      anthonyg
      last edited by

      Firewall1's Logs

      Dec 27 16:36:06 php: : Processing start -
      Dec 27 16:36:06 php: : Processing -
      Dec 27 16:38:26 check_reload_status: reloading filter
      Dec 27 16:38:28 php: : Beginning XMLRPC sync to http://172.16.1.3:80.
      Dec 27 16:38:29 php: : XMLRPC sync successfully completed with http://172.16.1.3:80.
      Dec 28 08:53:34 check_reload_status: reloading filter
      Dec 28 08:53:36 php: : Beginning XMLRPC sync to http://172.16.1.3:80.
      Dec 28 08:53:36 php: : XMLRPC sync successfully completed with http://172.16.1.3:80.
      Dec 28 08:55:56 check_reload_status: reloading filter
      Dec 28 08:55:58 php: : Beginning XMLRPC sync to http://172.16.1.3:80.
      Dec 28 08:55:59 php: : XMLRPC sync successfully completed with http://172.16.1.3:80.
      Dec 28 09:00:00 check_reload_status: check_reload_status is starting
      Dec 28 09:01:27 syslogd: exiting on signal 15
      Dec 28 09:01:27 syslogd: kernel boot file is /boot/kernel/kernel
      Dec 28 09:02:39 kernel: carp0: incorrect hash
      Dec 28 09:02:39 kernel: carp1: incorrect hash
      Dec 28 09:05:45 dnsmasq[633]: reading /etc/resolv.conf
      Dec 28 09:05:45 dnsmasq[633]: using nameserver 8.8.4.4#53
      Dec 28 09:05:45 dnsmasq[633]: using nameserver 8.8.8.8#53
      Dec 28 09:05:45 dnsmasq[633]: exiting on receipt of SIGTERM
      Dec 28 09:05:46 dnsmasq[37190]: started, version 2.45 cachesize 150
      Dec 28 09:05:46 dnsmasq[37190]: compile time options: IPv6 GNU-getopt BSD-bridge ISC-leasefile no-DBus no-I18N TFTP
      Dec 28 09:05:46 dnsmasq[37190]: reading /etc/resolv.conf
      Dec 28 09:05:46 dnsmasq[37190]: using nameserver 8.8.4.4#53
      Dec 28 09:05:46 dnsmasq[37190]: using nameserver 8.8.8.8#53
      Dec 28 09:05:46 dnsmasq[37190]: read /etc/hosts - 2 addresses
      Dec 28 09:05:51 check_reload_status: webConfigurator restart in progress
      Dec 28 09:05:58 php: : Creating rrd update script
      Dec 28 09:05:58 check_reload_status: reloading filter
      Dec 28 09:06:01 php: : Beginning XMLRPC sync to https://172.16.1.3:443.
      Dec 28 09:07:16 php: : A communications error occured while attempting XMLRPC sync with username admin https://172.16.1.3:443.
      Dec 28 09:07:16 php: : New alert found: A communications error occured while attempting XMLRPC sync with username admin https://172.16.1.3:443.
      Dec 28 09:07:16 php: : Beginning XMLRPC sync to https://172.16.1.3:443.
      Dec 28 09:08:31 php: : A communications error occured while attempting XMLRPC sync with username admin https://172.16.1.3:443.
      Dec 28 09:08:31 php: : New alert found: A communications error occured while attempting XMLRPC sync with username admin https://172.16.1.3:443.
      Dec 28 09:14:56 dnsmasq[37190]: reading /etc/resolv.conf
      Dec 28 09:14:56 dnsmasq[37190]: using nameserver 8.8.4.4#53
      Dec 28 09:14:56 dnsmasq[37190]: using nameserver 8.8.8.8#53
      Dec 28 09:14:56 dnsmasq[37190]: exiting on receipt of SIGTERM
      Dec 28 09:14:57 dnsmasq[38400]: started, version 2.45 cachesize 150
      Dec 28 09:14:57 dnsmasq[38400]: compile time options: IPv6 GNU-getopt BSD-bridge ISC-leasefile no-DBus no-I18N TFTP
      Dec 28 09:14:57 dnsmasq[38400]: reading /etc/resolv.conf
      Dec 28 09:14:57 dnsmasq[38400]: using nameserver 8.8.4.4#53
      Dec 28 09:14:57 dnsmasq[38400]: using nameserver 8.8.8.8#53
      Dec 28 09:14:57 dnsmasq[38400]: read /etc/hosts - 2 addresses
      Dec 28 09:15:02 check_reload_status: webConfigurator restart in progress
      Dec 28 09:15:10 php: : Creating rrd update script
      Dec 28 09:15:10 check_reload_status: reloading filter
      Dec 28 09:15:12 php: : Beginning XMLRPC sync to http://172.16.1.3:80.
      Dec 28 09:15:12 php: : XMLRPC sync successfully completed with http://172.16.1.3:80.

      The information displayed in RED is only an error because I had changed the GUI to be HTTPS. Once switched back it went back to normal, I was just making sure I could switch to HTTPS.

      Firewall2's Logs

      Dec 27 16:33:58 check_reload_status: reloading filter
      Dec 27 16:35:49 kernel: carp0: BACKUP -> MASTER (preempting a slower master)
      Dec 27 16:35:49 kernel: carp0: link state changed to UP
      Dec 27 16:35:51 kernel: em0: link state changed to DOWN
      Dec 27 16:35:51 kernel: carp1: link state changed to DOWN
      Dec 27 16:35:59 kernel: carp1: INIT -> BACKUP
      Dec 27 16:35:59 kernel: em0: link state changed to UP
      Dec 27 16:35:59 kernel: carp1: link state changed to DOWN
      Dec 27 16:36:01 check_reload_status: rc.linkup starting
      Dec 27 16:36:02 kernel: carp0: MASTER -> BACKUP (more frequent advertisement received)
      Dec 27 16:36:02 kernel: carp0: link state changed to DOWN
      Dec 27 16:36:02 php: : Processing em0 - start
      Dec 27 16:36:02 php: : Processing start -
      Dec 27 16:36:02 php: : Processing -
      Dec 27 16:36:02 kernel: carp1: link state changed to UP
      Dec 27 16:36:04 kernel: carp1: MASTER -> BACKUP (more frequent advertisement received)
      Dec 27 16:36:04 kernel: carp1: link state changed to DOWN
      Dec 27 16:38:32 check_reload_status: reloading filter
      Dec 28 01:10:00 check_reload_status: check_reload_status is starting
      Dec 28 08:53:40 check_reload_status: reloading filter
      Dec 28 08:56:03 check_reload_status: reloading filter
      Dec 28 09:01:25 syslogd: exiting on signal 15
      Dec 28 09:01:25 syslogd: kernel boot file is /boot/kernel/kernel
      Dec 28 09:01:49 kernel: carp0: link state changed to DOWN
      Dec 28 09:01:49 kernel: carp1: link state changed to DOWN
      Dec 28 09:02:29 syslogd: exiting on signal 15
      Dec 28 09:02:29 syslogd: kernel boot file is /boot/kernel/kernel
      Dec 28 09:02:39 kernel: carp0: INIT -> MASTER (preempting)
      Dec 28 09:02:39 kernel: carp0: link state changed to UP
      Dec 28 09:02:39 kernel: carp1: INIT -> MASTER (preempting)
      Dec 28 09:02:39 kernel: carp1: link state changed to UP
      Dec 28 09:02:39 kernel: carp0: link state changed to DOWN
      Dec 28 09:02:39 kernel: carp0: INIT -> MASTER (preempting)
      Dec 28 09:02:39 kernel: carp0: link state changed to UP
      Dec 28 09:02:39 kernel: carp1: link state changed to DOWN
      Dec 28 09:02:39 kernel: carp1: INIT -> MASTER (preempting)
      Dec 28 09:02:39 kernel: carp1: link state changed to UP
      Dec 28 09:02:39 kernel: carp0: MASTER -> BACKUP (more frequent advertisement received)
      Dec 28 09:02:39 kernel: carp0: link state changed to DOWN
      Dec 28 09:02:39 kernel: carp0: link state changed to DOWN
      Dec 28 09:02:39 kernel: carp0: INIT -> MASTER (preempting)
      Dec 28 09:02:39 kernel: carp0: link state changed to UP
      Dec 28 09:02:39 kernel: carp1: link state changed to DOWN
      Dec 28 09:02:39 kernel: carp1: INIT -> MASTER (preempting)
      Dec 28 09:02:39 kernel: carp1: link state changed to UP
      Dec 28 09:02:40 kernel: carp1: MASTER -> BACKUP (more frequent advertisement received)
      Dec 28 09:02:40 kernel: carp1: link state changed to DOWN
      Dec 28 09:02:40 kernel: carp0: MASTER -> BACKUP (more frequent advertisement received)
      Dec 28 09:02:40 kernel: carp0: link state changed to DOWN
      Dec 28 09:15:16 check_reload_status: reloading filter

      1 Reply Last reply Reply Quote 0
      • jimpJ
        jimp Rebel Alliance Developer Netgate
        last edited by

        You can switch to https as long as you do it on both boxes. The protocol just has to match between the two.

        Logs look ok, a couple flips like that are normal at boot time.

        Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

        Need help fast? Netgate Global Support!

        Do not Chat/PM for help!

        1 Reply Last reply Reply Quote 0
        • A
          anthonyg
          last edited by

          I just rebooted both firewalls for fresh startup. What should we do from here?

          1 Reply Last reply Reply Quote 0
          • jimpJ
            jimp Rebel Alliance Developer Netgate
            last edited by

            I'd run a packet capture on both master and backup looking for traffic for that CARP VIP, then run the external ping again, confirm which box is actually getting the traffic.

            If that works and the master is getting the traffic, you may have errors in your manual outbound NAT rules, so post screenshots of those.

            Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

            Need help fast? Netgate Global Support!

            Do not Chat/PM for help!

            1 Reply Last reply Reply Quote 0
            • A
              anthonyg
              last edited by

              Packet capture shows that all traffic is being received on the second firewall. Currently NAT is set to Automatic, however that shouldn't effect receiving pings on the correct firewall.

              1 Reply Last reply Reply Quote 0
              • A
                anthonyg
                last edited by

                I have just confirmed with packet capture that firewall1 is receiving all the LAN CARP traffic, just not the WAN.

                1 Reply Last reply Reply Quote 0
                • jimpJ
                  jimp Rebel Alliance Developer Netgate
                  last edited by

                  That is rather odd, especially if it is showing as backup. Are these devices plugged into the rear of your cable modem/router? If so you might try plugging them into a separate switch and then uplinking to that, just to eliminate that as a possible cause.

                  Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                  Need help fast? Netgate Global Support!

                  Do not Chat/PM for help!

                  1 Reply Last reply Reply Quote 0
                  • A
                    anthonyg
                    last edited by

                    Currently I have a connection going from my Comcast Modem to a 24 port Netgear Switch, then from the switch into the two firewalls. For my LAN I have that also plugged into a seperate 24 port switch.

                    1 Reply Last reply Reply Quote 0
                    • A
                      anthonyg
                      last edited by

                      Alright I have deleted the WAN CARP IP and re-added and it looks like the primary firewall is now receiving its traffic.

                      1 Reply Last reply Reply Quote 0
                      • A
                        anthonyg
                        last edited by

                        Nevermind it has flopped back over to firewall2. Why is it that firewall2 would be receiving all the traffic?

                        1 Reply Last reply Reply Quote 0
                        • jimpJ
                          jimp Rebel Alliance Developer Netgate
                          last edited by

                          Typically that only happens when it takes over as a CARP master.

                          The only other times I've seen similar CARP craziness was with some really broken switches.

                          Any way you can try another (perhaps different brand) switch on the WAN side as a test to see if the problem goes away?

                          Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                          Need help fast? Netgate Global Support!

                          Do not Chat/PM for help!

                          1 Reply Last reply Reply Quote 0
                          • A
                            anthonyg
                            last edited by

                            I have switched from a netgear switch to a linksys switch. I have also changed around the interface IPs and CARP IP. We will see what this delivers.

                            1 Reply Last reply Reply Quote 0
                            • A
                              anthonyg
                              last edited by

                              Also could it be a problem with CARP not syncing right? I have changed the virtual IP's on firewall1 and they have yet to reflect on firewall2.

                              1 Reply Last reply Reply Quote 0
                              • jimpJ
                                jimp Rebel Alliance Developer Netgate
                                last edited by

                                No, it wouldn't have much to do with a sync failure.

                                CARP heartbeats happen on the interfaces where the VIPs reside, i.e. a CARP VIP on WAN sends its heartbeats on WAN.
                                XMLRPC sync happens over the sync interface, it only handles configuration.
                                pfsync only happens over the sync interface, it only synchronizes states (insertions, deletions, etc)

                                So a problem with CARP on WAN is nearly always a problem with the switch or connectivity on WAN.

                                Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                                Need help fast? Netgate Global Support!

                                Do not Chat/PM for help!

                                1 Reply Last reply Reply Quote 0
                                • First post
                                  Last post
                                Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.