Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    CARP Failover not Working on Manual Outbound NAT

    Scheduled Pinned Locked Moved
    HA/CARP/VIPs
    2
    20
    9.0k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • jimpJ
      jimp Rebel Alliance Developer Netgate
      last edited by

      Are there any CARP transitions logged under Status > System Logs?

      Any bridging going on?

      Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

      Need help fast? Netgate Global Support!

      Do not Chat/PM for help!

      1 Reply Last reply Reply Quote 0
      • A
        anthonyg
        last edited by

        Firewall1's Logs

        Dec 27 16:36:06 php: : Processing start -
        Dec 27 16:36:06 php: : Processing -
        Dec 27 16:38:26 check_reload_status: reloading filter
        Dec 27 16:38:28 php: : Beginning XMLRPC sync to http://172.16.1.3:80.
        Dec 27 16:38:29 php: : XMLRPC sync successfully completed with http://172.16.1.3:80.
        Dec 28 08:53:34 check_reload_status: reloading filter
        Dec 28 08:53:36 php: : Beginning XMLRPC sync to http://172.16.1.3:80.
        Dec 28 08:53:36 php: : XMLRPC sync successfully completed with http://172.16.1.3:80.
        Dec 28 08:55:56 check_reload_status: reloading filter
        Dec 28 08:55:58 php: : Beginning XMLRPC sync to http://172.16.1.3:80.
        Dec 28 08:55:59 php: : XMLRPC sync successfully completed with http://172.16.1.3:80.
        Dec 28 09:00:00 check_reload_status: check_reload_status is starting
        Dec 28 09:01:27 syslogd: exiting on signal 15
        Dec 28 09:01:27 syslogd: kernel boot file is /boot/kernel/kernel
        Dec 28 09:02:39 kernel: carp0: incorrect hash
        Dec 28 09:02:39 kernel: carp1: incorrect hash
        Dec 28 09:05:45 dnsmasq[633]: reading /etc/resolv.conf
        Dec 28 09:05:45 dnsmasq[633]: using nameserver 8.8.4.4#53
        Dec 28 09:05:45 dnsmasq[633]: using nameserver 8.8.8.8#53
        Dec 28 09:05:45 dnsmasq[633]: exiting on receipt of SIGTERM
        Dec 28 09:05:46 dnsmasq[37190]: started, version 2.45 cachesize 150
        Dec 28 09:05:46 dnsmasq[37190]: compile time options: IPv6 GNU-getopt BSD-bridge ISC-leasefile no-DBus no-I18N TFTP
        Dec 28 09:05:46 dnsmasq[37190]: reading /etc/resolv.conf
        Dec 28 09:05:46 dnsmasq[37190]: using nameserver 8.8.4.4#53
        Dec 28 09:05:46 dnsmasq[37190]: using nameserver 8.8.8.8#53
        Dec 28 09:05:46 dnsmasq[37190]: read /etc/hosts - 2 addresses
        Dec 28 09:05:51 check_reload_status: webConfigurator restart in progress
        Dec 28 09:05:58 php: : Creating rrd update script
        Dec 28 09:05:58 check_reload_status: reloading filter
        Dec 28 09:06:01 php: : Beginning XMLRPC sync to https://172.16.1.3:443.
        Dec 28 09:07:16 php: : A communications error occured while attempting XMLRPC sync with username admin https://172.16.1.3:443.
        Dec 28 09:07:16 php: : New alert found: A communications error occured while attempting XMLRPC sync with username admin https://172.16.1.3:443.
        Dec 28 09:07:16 php: : Beginning XMLRPC sync to https://172.16.1.3:443.
        Dec 28 09:08:31 php: : A communications error occured while attempting XMLRPC sync with username admin https://172.16.1.3:443.
        Dec 28 09:08:31 php: : New alert found: A communications error occured while attempting XMLRPC sync with username admin https://172.16.1.3:443.
        Dec 28 09:14:56 dnsmasq[37190]: reading /etc/resolv.conf
        Dec 28 09:14:56 dnsmasq[37190]: using nameserver 8.8.4.4#53
        Dec 28 09:14:56 dnsmasq[37190]: using nameserver 8.8.8.8#53
        Dec 28 09:14:56 dnsmasq[37190]: exiting on receipt of SIGTERM
        Dec 28 09:14:57 dnsmasq[38400]: started, version 2.45 cachesize 150
        Dec 28 09:14:57 dnsmasq[38400]: compile time options: IPv6 GNU-getopt BSD-bridge ISC-leasefile no-DBus no-I18N TFTP
        Dec 28 09:14:57 dnsmasq[38400]: reading /etc/resolv.conf
        Dec 28 09:14:57 dnsmasq[38400]: using nameserver 8.8.4.4#53
        Dec 28 09:14:57 dnsmasq[38400]: using nameserver 8.8.8.8#53
        Dec 28 09:14:57 dnsmasq[38400]: read /etc/hosts - 2 addresses
        Dec 28 09:15:02 check_reload_status: webConfigurator restart in progress
        Dec 28 09:15:10 php: : Creating rrd update script
        Dec 28 09:15:10 check_reload_status: reloading filter
        Dec 28 09:15:12 php: : Beginning XMLRPC sync to http://172.16.1.3:80.
        Dec 28 09:15:12 php: : XMLRPC sync successfully completed with http://172.16.1.3:80.

        The information displayed in RED is only an error because I had changed the GUI to be HTTPS. Once switched back it went back to normal, I was just making sure I could switch to HTTPS.

        Firewall2's Logs

        Dec 27 16:33:58 check_reload_status: reloading filter
        Dec 27 16:35:49 kernel: carp0: BACKUP -> MASTER (preempting a slower master)
        Dec 27 16:35:49 kernel: carp0: link state changed to UP
        Dec 27 16:35:51 kernel: em0: link state changed to DOWN
        Dec 27 16:35:51 kernel: carp1: link state changed to DOWN
        Dec 27 16:35:59 kernel: carp1: INIT -> BACKUP
        Dec 27 16:35:59 kernel: em0: link state changed to UP
        Dec 27 16:35:59 kernel: carp1: link state changed to DOWN
        Dec 27 16:36:01 check_reload_status: rc.linkup starting
        Dec 27 16:36:02 kernel: carp0: MASTER -> BACKUP (more frequent advertisement received)
        Dec 27 16:36:02 kernel: carp0: link state changed to DOWN
        Dec 27 16:36:02 php: : Processing em0 - start
        Dec 27 16:36:02 php: : Processing start -
        Dec 27 16:36:02 php: : Processing -
        Dec 27 16:36:02 kernel: carp1: link state changed to UP
        Dec 27 16:36:04 kernel: carp1: MASTER -> BACKUP (more frequent advertisement received)
        Dec 27 16:36:04 kernel: carp1: link state changed to DOWN
        Dec 27 16:38:32 check_reload_status: reloading filter
        Dec 28 01:10:00 check_reload_status: check_reload_status is starting
        Dec 28 08:53:40 check_reload_status: reloading filter
        Dec 28 08:56:03 check_reload_status: reloading filter
        Dec 28 09:01:25 syslogd: exiting on signal 15
        Dec 28 09:01:25 syslogd: kernel boot file is /boot/kernel/kernel
        Dec 28 09:01:49 kernel: carp0: link state changed to DOWN
        Dec 28 09:01:49 kernel: carp1: link state changed to DOWN
        Dec 28 09:02:29 syslogd: exiting on signal 15
        Dec 28 09:02:29 syslogd: kernel boot file is /boot/kernel/kernel
        Dec 28 09:02:39 kernel: carp0: INIT -> MASTER (preempting)
        Dec 28 09:02:39 kernel: carp0: link state changed to UP
        Dec 28 09:02:39 kernel: carp1: INIT -> MASTER (preempting)
        Dec 28 09:02:39 kernel: carp1: link state changed to UP
        Dec 28 09:02:39 kernel: carp0: link state changed to DOWN
        Dec 28 09:02:39 kernel: carp0: INIT -> MASTER (preempting)
        Dec 28 09:02:39 kernel: carp0: link state changed to UP
        Dec 28 09:02:39 kernel: carp1: link state changed to DOWN
        Dec 28 09:02:39 kernel: carp1: INIT -> MASTER (preempting)
        Dec 28 09:02:39 kernel: carp1: link state changed to UP
        Dec 28 09:02:39 kernel: carp0: MASTER -> BACKUP (more frequent advertisement received)
        Dec 28 09:02:39 kernel: carp0: link state changed to DOWN
        Dec 28 09:02:39 kernel: carp0: link state changed to DOWN
        Dec 28 09:02:39 kernel: carp0: INIT -> MASTER (preempting)
        Dec 28 09:02:39 kernel: carp0: link state changed to UP
        Dec 28 09:02:39 kernel: carp1: link state changed to DOWN
        Dec 28 09:02:39 kernel: carp1: INIT -> MASTER (preempting)
        Dec 28 09:02:39 kernel: carp1: link state changed to UP
        Dec 28 09:02:40 kernel: carp1: MASTER -> BACKUP (more frequent advertisement received)
        Dec 28 09:02:40 kernel: carp1: link state changed to DOWN
        Dec 28 09:02:40 kernel: carp0: MASTER -> BACKUP (more frequent advertisement received)
        Dec 28 09:02:40 kernel: carp0: link state changed to DOWN
        Dec 28 09:15:16 check_reload_status: reloading filter

        1 Reply Last reply Reply Quote 0
        • jimpJ
          jimp Rebel Alliance Developer Netgate
          last edited by

          You can switch to https as long as you do it on both boxes. The protocol just has to match between the two.

          Logs look ok, a couple flips like that are normal at boot time.

          Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

          Need help fast? Netgate Global Support!

          Do not Chat/PM for help!

          1 Reply Last reply Reply Quote 0
          • A
            anthonyg
            last edited by

            I just rebooted both firewalls for fresh startup. What should we do from here?

            1 Reply Last reply Reply Quote 0
            • jimpJ
              jimp Rebel Alliance Developer Netgate
              last edited by

              I'd run a packet capture on both master and backup looking for traffic for that CARP VIP, then run the external ping again, confirm which box is actually getting the traffic.

              If that works and the master is getting the traffic, you may have errors in your manual outbound NAT rules, so post screenshots of those.

              Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

              Need help fast? Netgate Global Support!

              Do not Chat/PM for help!

              1 Reply Last reply Reply Quote 0
              • A
                anthonyg
                last edited by

                Packet capture shows that all traffic is being received on the second firewall. Currently NAT is set to Automatic, however that shouldn't effect receiving pings on the correct firewall.

                1 Reply Last reply Reply Quote 0
                • A
                  anthonyg
                  last edited by

                  I have just confirmed with packet capture that firewall1 is receiving all the LAN CARP traffic, just not the WAN.

                  1 Reply Last reply Reply Quote 0
                  • jimpJ
                    jimp Rebel Alliance Developer Netgate
                    last edited by

                    That is rather odd, especially if it is showing as backup. Are these devices plugged into the rear of your cable modem/router? If so you might try plugging them into a separate switch and then uplinking to that, just to eliminate that as a possible cause.

                    Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                    Need help fast? Netgate Global Support!

                    Do not Chat/PM for help!

                    1 Reply Last reply Reply Quote 0
                    • A
                      anthonyg
                      last edited by

                      Currently I have a connection going from my Comcast Modem to a 24 port Netgear Switch, then from the switch into the two firewalls. For my LAN I have that also plugged into a seperate 24 port switch.

                      1 Reply Last reply Reply Quote 0
                      • A
                        anthonyg
                        last edited by

                        Alright I have deleted the WAN CARP IP and re-added and it looks like the primary firewall is now receiving its traffic.

                        1 Reply Last reply Reply Quote 0
                        • A
                          anthonyg
                          last edited by

                          Nevermind it has flopped back over to firewall2. Why is it that firewall2 would be receiving all the traffic?

                          1 Reply Last reply Reply Quote 0
                          • jimpJ
                            jimp Rebel Alliance Developer Netgate
                            last edited by

                            Typically that only happens when it takes over as a CARP master.

                            The only other times I've seen similar CARP craziness was with some really broken switches.

                            Any way you can try another (perhaps different brand) switch on the WAN side as a test to see if the problem goes away?

                            Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                            Need help fast? Netgate Global Support!

                            Do not Chat/PM for help!

                            1 Reply Last reply Reply Quote 0
                            • A
                              anthonyg
                              last edited by

                              I have switched from a netgear switch to a linksys switch. I have also changed around the interface IPs and CARP IP. We will see what this delivers.

                              1 Reply Last reply Reply Quote 0
                              • A
                                anthonyg
                                last edited by

                                Also could it be a problem with CARP not syncing right? I have changed the virtual IP's on firewall1 and they have yet to reflect on firewall2.

                                1 Reply Last reply Reply Quote 0
                                • jimpJ
                                  jimp Rebel Alliance Developer Netgate
                                  last edited by

                                  No, it wouldn't have much to do with a sync failure.

                                  CARP heartbeats happen on the interfaces where the VIPs reside, i.e. a CARP VIP on WAN sends its heartbeats on WAN.
                                  XMLRPC sync happens over the sync interface, it only handles configuration.
                                  pfsync only happens over the sync interface, it only synchronizes states (insertions, deletions, etc)

                                  So a problem with CARP on WAN is nearly always a problem with the switch or connectivity on WAN.

                                  Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                                  Need help fast? Netgate Global Support!

                                  Do not Chat/PM for help!

                                  1 Reply Last reply Reply Quote 0
                                  • First post
                                    Last post