Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    CARP Failover not Working on Manual Outbound NAT

    Scheduled Pinned Locked Moved
    HA/CARP/VIPs
    2
    20
    9.0k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A
      anthonyg
      last edited by

      Yes sir. Here is the screenshots. No offense for the blur outs, but since this is a medical facility I cannot publish our public IP's out.

      Firewall1.jpg
      Firewall1.jpg_thumb
      Firewall2.jpg
      Firewall2.jpg_thumb

      1 Reply Last reply Reply Quote 0
      • jimpJ
        jimp Rebel Alliance Developer Netgate
        last edited by

        Are there any CARP transitions logged under Status > System Logs?

        Any bridging going on?

        Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

        Need help fast? Netgate Global Support!

        Do not Chat/PM for help!

        1 Reply Last reply Reply Quote 0
        • A
          anthonyg
          last edited by

          Firewall1's Logs

          Dec 27 16:36:06 php: : Processing start -
          Dec 27 16:36:06 php: : Processing -
          Dec 27 16:38:26 check_reload_status: reloading filter
          Dec 27 16:38:28 php: : Beginning XMLRPC sync to http://172.16.1.3:80.
          Dec 27 16:38:29 php: : XMLRPC sync successfully completed with http://172.16.1.3:80.
          Dec 28 08:53:34 check_reload_status: reloading filter
          Dec 28 08:53:36 php: : Beginning XMLRPC sync to http://172.16.1.3:80.
          Dec 28 08:53:36 php: : XMLRPC sync successfully completed with http://172.16.1.3:80.
          Dec 28 08:55:56 check_reload_status: reloading filter
          Dec 28 08:55:58 php: : Beginning XMLRPC sync to http://172.16.1.3:80.
          Dec 28 08:55:59 php: : XMLRPC sync successfully completed with http://172.16.1.3:80.
          Dec 28 09:00:00 check_reload_status: check_reload_status is starting
          Dec 28 09:01:27 syslogd: exiting on signal 15
          Dec 28 09:01:27 syslogd: kernel boot file is /boot/kernel/kernel
          Dec 28 09:02:39 kernel: carp0: incorrect hash
          Dec 28 09:02:39 kernel: carp1: incorrect hash
          Dec 28 09:05:45 dnsmasq[633]: reading /etc/resolv.conf
          Dec 28 09:05:45 dnsmasq[633]: using nameserver 8.8.4.4#53
          Dec 28 09:05:45 dnsmasq[633]: using nameserver 8.8.8.8#53
          Dec 28 09:05:45 dnsmasq[633]: exiting on receipt of SIGTERM
          Dec 28 09:05:46 dnsmasq[37190]: started, version 2.45 cachesize 150
          Dec 28 09:05:46 dnsmasq[37190]: compile time options: IPv6 GNU-getopt BSD-bridge ISC-leasefile no-DBus no-I18N TFTP
          Dec 28 09:05:46 dnsmasq[37190]: reading /etc/resolv.conf
          Dec 28 09:05:46 dnsmasq[37190]: using nameserver 8.8.4.4#53
          Dec 28 09:05:46 dnsmasq[37190]: using nameserver 8.8.8.8#53
          Dec 28 09:05:46 dnsmasq[37190]: read /etc/hosts - 2 addresses
          Dec 28 09:05:51 check_reload_status: webConfigurator restart in progress
          Dec 28 09:05:58 php: : Creating rrd update script
          Dec 28 09:05:58 check_reload_status: reloading filter
          Dec 28 09:06:01 php: : Beginning XMLRPC sync to https://172.16.1.3:443.
          Dec 28 09:07:16 php: : A communications error occured while attempting XMLRPC sync with username admin https://172.16.1.3:443.
          Dec 28 09:07:16 php: : New alert found: A communications error occured while attempting XMLRPC sync with username admin https://172.16.1.3:443.
          Dec 28 09:07:16 php: : Beginning XMLRPC sync to https://172.16.1.3:443.
          Dec 28 09:08:31 php: : A communications error occured while attempting XMLRPC sync with username admin https://172.16.1.3:443.
          Dec 28 09:08:31 php: : New alert found: A communications error occured while attempting XMLRPC sync with username admin https://172.16.1.3:443.
          Dec 28 09:14:56 dnsmasq[37190]: reading /etc/resolv.conf
          Dec 28 09:14:56 dnsmasq[37190]: using nameserver 8.8.4.4#53
          Dec 28 09:14:56 dnsmasq[37190]: using nameserver 8.8.8.8#53
          Dec 28 09:14:56 dnsmasq[37190]: exiting on receipt of SIGTERM
          Dec 28 09:14:57 dnsmasq[38400]: started, version 2.45 cachesize 150
          Dec 28 09:14:57 dnsmasq[38400]: compile time options: IPv6 GNU-getopt BSD-bridge ISC-leasefile no-DBus no-I18N TFTP
          Dec 28 09:14:57 dnsmasq[38400]: reading /etc/resolv.conf
          Dec 28 09:14:57 dnsmasq[38400]: using nameserver 8.8.4.4#53
          Dec 28 09:14:57 dnsmasq[38400]: using nameserver 8.8.8.8#53
          Dec 28 09:14:57 dnsmasq[38400]: read /etc/hosts - 2 addresses
          Dec 28 09:15:02 check_reload_status: webConfigurator restart in progress
          Dec 28 09:15:10 php: : Creating rrd update script
          Dec 28 09:15:10 check_reload_status: reloading filter
          Dec 28 09:15:12 php: : Beginning XMLRPC sync to http://172.16.1.3:80.
          Dec 28 09:15:12 php: : XMLRPC sync successfully completed with http://172.16.1.3:80.

          The information displayed in RED is only an error because I had changed the GUI to be HTTPS. Once switched back it went back to normal, I was just making sure I could switch to HTTPS.

          Firewall2's Logs

          Dec 27 16:33:58 check_reload_status: reloading filter
          Dec 27 16:35:49 kernel: carp0: BACKUP -> MASTER (preempting a slower master)
          Dec 27 16:35:49 kernel: carp0: link state changed to UP
          Dec 27 16:35:51 kernel: em0: link state changed to DOWN
          Dec 27 16:35:51 kernel: carp1: link state changed to DOWN
          Dec 27 16:35:59 kernel: carp1: INIT -> BACKUP
          Dec 27 16:35:59 kernel: em0: link state changed to UP
          Dec 27 16:35:59 kernel: carp1: link state changed to DOWN
          Dec 27 16:36:01 check_reload_status: rc.linkup starting
          Dec 27 16:36:02 kernel: carp0: MASTER -> BACKUP (more frequent advertisement received)
          Dec 27 16:36:02 kernel: carp0: link state changed to DOWN
          Dec 27 16:36:02 php: : Processing em0 - start
          Dec 27 16:36:02 php: : Processing start -
          Dec 27 16:36:02 php: : Processing -
          Dec 27 16:36:02 kernel: carp1: link state changed to UP
          Dec 27 16:36:04 kernel: carp1: MASTER -> BACKUP (more frequent advertisement received)
          Dec 27 16:36:04 kernel: carp1: link state changed to DOWN
          Dec 27 16:38:32 check_reload_status: reloading filter
          Dec 28 01:10:00 check_reload_status: check_reload_status is starting
          Dec 28 08:53:40 check_reload_status: reloading filter
          Dec 28 08:56:03 check_reload_status: reloading filter
          Dec 28 09:01:25 syslogd: exiting on signal 15
          Dec 28 09:01:25 syslogd: kernel boot file is /boot/kernel/kernel
          Dec 28 09:01:49 kernel: carp0: link state changed to DOWN
          Dec 28 09:01:49 kernel: carp1: link state changed to DOWN
          Dec 28 09:02:29 syslogd: exiting on signal 15
          Dec 28 09:02:29 syslogd: kernel boot file is /boot/kernel/kernel
          Dec 28 09:02:39 kernel: carp0: INIT -> MASTER (preempting)
          Dec 28 09:02:39 kernel: carp0: link state changed to UP
          Dec 28 09:02:39 kernel: carp1: INIT -> MASTER (preempting)
          Dec 28 09:02:39 kernel: carp1: link state changed to UP
          Dec 28 09:02:39 kernel: carp0: link state changed to DOWN
          Dec 28 09:02:39 kernel: carp0: INIT -> MASTER (preempting)
          Dec 28 09:02:39 kernel: carp0: link state changed to UP
          Dec 28 09:02:39 kernel: carp1: link state changed to DOWN
          Dec 28 09:02:39 kernel: carp1: INIT -> MASTER (preempting)
          Dec 28 09:02:39 kernel: carp1: link state changed to UP
          Dec 28 09:02:39 kernel: carp0: MASTER -> BACKUP (more frequent advertisement received)
          Dec 28 09:02:39 kernel: carp0: link state changed to DOWN
          Dec 28 09:02:39 kernel: carp0: link state changed to DOWN
          Dec 28 09:02:39 kernel: carp0: INIT -> MASTER (preempting)
          Dec 28 09:02:39 kernel: carp0: link state changed to UP
          Dec 28 09:02:39 kernel: carp1: link state changed to DOWN
          Dec 28 09:02:39 kernel: carp1: INIT -> MASTER (preempting)
          Dec 28 09:02:39 kernel: carp1: link state changed to UP
          Dec 28 09:02:40 kernel: carp1: MASTER -> BACKUP (more frequent advertisement received)
          Dec 28 09:02:40 kernel: carp1: link state changed to DOWN
          Dec 28 09:02:40 kernel: carp0: MASTER -> BACKUP (more frequent advertisement received)
          Dec 28 09:02:40 kernel: carp0: link state changed to DOWN
          Dec 28 09:15:16 check_reload_status: reloading filter

          1 Reply Last reply Reply Quote 0
          • jimpJ
            jimp Rebel Alliance Developer Netgate
            last edited by

            You can switch to https as long as you do it on both boxes. The protocol just has to match between the two.

            Logs look ok, a couple flips like that are normal at boot time.

            Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

            Need help fast? Netgate Global Support!

            Do not Chat/PM for help!

            1 Reply Last reply Reply Quote 0
            • A
              anthonyg
              last edited by

              I just rebooted both firewalls for fresh startup. What should we do from here?

              1 Reply Last reply Reply Quote 0
              • jimpJ
                jimp Rebel Alliance Developer Netgate
                last edited by

                I'd run a packet capture on both master and backup looking for traffic for that CARP VIP, then run the external ping again, confirm which box is actually getting the traffic.

                If that works and the master is getting the traffic, you may have errors in your manual outbound NAT rules, so post screenshots of those.

                Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                Need help fast? Netgate Global Support!

                Do not Chat/PM for help!

                1 Reply Last reply Reply Quote 0
                • A
                  anthonyg
                  last edited by

                  Packet capture shows that all traffic is being received on the second firewall. Currently NAT is set to Automatic, however that shouldn't effect receiving pings on the correct firewall.

                  1 Reply Last reply Reply Quote 0
                  • A
                    anthonyg
                    last edited by

                    I have just confirmed with packet capture that firewall1 is receiving all the LAN CARP traffic, just not the WAN.

                    1 Reply Last reply Reply Quote 0
                    • jimpJ
                      jimp Rebel Alliance Developer Netgate
                      last edited by

                      That is rather odd, especially if it is showing as backup. Are these devices plugged into the rear of your cable modem/router? If so you might try plugging them into a separate switch and then uplinking to that, just to eliminate that as a possible cause.

                      Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                      Need help fast? Netgate Global Support!

                      Do not Chat/PM for help!

                      1 Reply Last reply Reply Quote 0
                      • A
                        anthonyg
                        last edited by

                        Currently I have a connection going from my Comcast Modem to a 24 port Netgear Switch, then from the switch into the two firewalls. For my LAN I have that also plugged into a seperate 24 port switch.

                        1 Reply Last reply Reply Quote 0
                        • A
                          anthonyg
                          last edited by

                          Alright I have deleted the WAN CARP IP and re-added and it looks like the primary firewall is now receiving its traffic.

                          1 Reply Last reply Reply Quote 0
                          • A
                            anthonyg
                            last edited by

                            Nevermind it has flopped back over to firewall2. Why is it that firewall2 would be receiving all the traffic?

                            1 Reply Last reply Reply Quote 0
                            • jimpJ
                              jimp Rebel Alliance Developer Netgate
                              last edited by

                              Typically that only happens when it takes over as a CARP master.

                              The only other times I've seen similar CARP craziness was with some really broken switches.

                              Any way you can try another (perhaps different brand) switch on the WAN side as a test to see if the problem goes away?

                              Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                              Need help fast? Netgate Global Support!

                              Do not Chat/PM for help!

                              1 Reply Last reply Reply Quote 0
                              • A
                                anthonyg
                                last edited by

                                I have switched from a netgear switch to a linksys switch. I have also changed around the interface IPs and CARP IP. We will see what this delivers.

                                1 Reply Last reply Reply Quote 0
                                • A
                                  anthonyg
                                  last edited by

                                  Also could it be a problem with CARP not syncing right? I have changed the virtual IP's on firewall1 and they have yet to reflect on firewall2.

                                  1 Reply Last reply Reply Quote 0
                                  • jimpJ
                                    jimp Rebel Alliance Developer Netgate
                                    last edited by

                                    No, it wouldn't have much to do with a sync failure.

                                    CARP heartbeats happen on the interfaces where the VIPs reside, i.e. a CARP VIP on WAN sends its heartbeats on WAN.
                                    XMLRPC sync happens over the sync interface, it only handles configuration.
                                    pfsync only happens over the sync interface, it only synchronizes states (insertions, deletions, etc)

                                    So a problem with CARP on WAN is nearly always a problem with the switch or connectivity on WAN.

                                    Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                                    Need help fast? Netgate Global Support!

                                    Do not Chat/PM for help!

                                    1 Reply Last reply Reply Quote 0
                                    • First post
                                      Last post