Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    how to debug CARP ?

    Scheduled Pinned Locked Moved HA/CARP/VIPs
    9 Posts 3 Posters 1.0k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Y
      Yathus
      last edited by Yathus

      Hello,

      For many years (First with pfsense 2.1) we have been using two PFSENSE in CARP as firewalls on a Vmware private cloud.

      Today with the latest version (23.09.1 in Pfsense Plus) we're having random problems

      • a master/slave toggle works correctly, but going backwards doesn't work.
      • IPSEC tunnel won't come up
      • some VIP IP doesn't work

      Only solution is to "shutdown" / "start" both nodes...

      Our pfsense have 1 WAN (with 2 range /26), 5 LAN and 1 LAN dedicated to PFSYNC.

      We have 1 CARP IP for each LAN (used as gateway by VM), 1 CARP IP on each range and a lot of IP Alias on CARP).

      Our vSwitch have security enable :
      Promiscuous mode Accept
      MAC address changes Accept
      Forged transmits Accept

      So we're looking for logs or anything else that might help us diagnose problems during our failover tests.

      From a purely practical point of view over all these years, we have the impression of a regression, which may come from Vmware (5.5 to 7.0) or PFSENSE, but we can't understand the causes, nor can our supplier.

      Thanks

      Yathus

      S 1 Reply Last reply Reply Quote 0
      • S
        SteveITS Galactic Empire @Yathus
        last edited by

        @Yathus have you found this list?
        https://docs.netgate.com/pfsense/en/latest/troubleshooting/index.html#high-availability

        Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
        When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
        Upvote 👍 helpful posts!

        Y 1 Reply Last reply Reply Quote 0
        • Y
          Yathus @SteveITS
          last edited by

          Hello @SteveITS thanks for the links.

          I was familiar with these pages (the links are coloured "already consulted" in my browser) but I'm going to start from scratch anyway.

          Do you know if there is a log because on my side in "System Logs" I don't see the CARP messages, I only see them on the console.

          S 1 Reply Last reply Reply Quote 0
          • S
            SteveITS Galactic Empire @Yathus
            last edited by

            @Yathus So you have both showing as Master? And you looked at the link about VMWare there?

            It should log the "resumed as backup" notes in the system log as I recall. They are sent via the email notification.

            Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
            When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
            Upvote 👍 helpful posts!

            Y 1 Reply Last reply Reply Quote 0
            • Y
              Yathus @SteveITS
              last edited by

              Hello @SteveITS

              I found some information in /var/log/system.log

              When i do a vmotion (moving a VM from one host to another) of PFSENSE BACKUP there is short time :

              Feb 21 13:10:10 pfsense04 check_reload_status[451]: Carp master event
Feb 21 13:10:10 pfsense04 kernel: carp: 3@vmx3: BACKUP -> MASTER (master timed out)
Feb 21 13:10:10 pfsense04 kernel: carp: 2@vmx0: BACKUP -> MASTER (master timed out)
Feb 21 13:10:10 pfsense04 kernel: carp: 9@vmx0: BACKUP -> MASTER (master timed out)
Feb 21 13:10:10 pfsense04 kernel: carp: 4@vmx1: BACKUP -> MASTER (master timed out)
Feb 21 13:10:10 pfsense04 kernel: carp: 8@vmx7: BACKUP -> MASTER (master timed out)
Feb 21 13:10:10 pfsense04 kernel: carp: 5@vmx4: BACKUP -> MASTER (master timed out)
Feb 21 13:10:10 pfsense04 check_reload_status[451]: Carp master event
Feb 21 13:10:10 pfsense04 check_reload_status[451]: Carp master event
Feb 21 13:10:10 pfsense04 check_reload_status[451]: Carp master event
Feb 21 13:10:10 pfsense04 check_reload_status[451]: Carp master event
Feb 21 13:10:10 pfsense04 check_reload_status[451]: Carp master event
Feb 21 13:10:11 pfsense04 kernel: carp: 2@vmx0: MASTER -> BACKUP (more frequent advertisement received)
Feb 21 13:10:11 pfsense04 kernel: carp: 8@vmx7: MASTER -> BACKUP (more frequent advertisement received)
Feb 21 13:10:11 pfsense04 kernel: carp: 3@vmx3: MASTER -> BACKUP (more frequent advertisement received)
Feb 21 13:10:11 pfsense04 kernel: carp: 9@vmx0: MASTER -> BACKUP (more frequent advertisement received)
Feb 21 13:10:11 pfsense04 kernel: in_scrubprefix: err=65, prefix delete failed
Feb 21 13:10:11 pfsense04 kernel: in_scrubprefix: err=65, prefix delete failed
Feb 21 13:10:11 pfsense04 kernel: in_scrubprefix: err=65, prefix delete failed
Feb 21 13:10:11 pfsense04 kernel: in_scrubprefix: err=65, prefix delete failed
Feb 21 13:10:11 pfsense04 kernel: in_scrubprefix: err=65, prefix delete failed
Feb 21 13:10:11 pfsense04 kernel: in_scrubprefix: err=65, prefix delete failed
Feb 21 13:10:11 pfsense04 kernel: carp: 4@vmx1: MASTER -> BACKUP (more frequent advertisement received)
Feb 21 13:10:11 pfsense04 kernel: carp: 5@vmx4: MASTER -> BACKUP (more frequent advertisement received)
Feb 21 13:10:11 pfsense04 check_reload_status[451]: Carp backup event
Feb 21 13:10:11 pfsense04 check_reload_status[451]: Carp backup event
Feb 21 13:10:11 pfsense04 check_reload_status[451]: Carp backup event
Feb 21 13:10:11 pfsense04 check_reload_status[451]: Carp backup event
Feb 21 13:10:11 pfsense04 check_reload_status[451]: Carp backup event
Feb 21 13:10:11 pfsense04 check_reload_status[451]: Carp backup event
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.195@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.195@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[73066]: /rc.carpmaster: HA cluster member "(192.168.112.254@vmx4): (PCC2_5002)" has resumed CARP state "MASTER" for vhid 5
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.233@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.236@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.234@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.196@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.225@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.233@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.236@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.234@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.196@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.225@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.205@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.198@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.197@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.209@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.202@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.216@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.210@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.201@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.240@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.235@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[412]: /rc.carpmaster: HA cluster member "(192.168.111.254@vmx3): (CRN5001)" has resumed CARP state "MASTER" for vhid 3
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.218@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.203@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.222@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.199@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.228@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.220@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.229@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.230@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.213@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.242@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.246@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.200@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.201@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.202@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.203@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
[...]
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.206@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.217@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.210@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.219@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.218@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.238@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.247@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.251@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.226@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.227@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.221@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.222@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.234@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.235@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.249@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.236@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.237@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.233@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.224@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 2
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.205@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.198@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.197@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.209@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.202@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.216@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.210@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.201@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.199@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.232@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.208@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.197@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.198@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.231@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.223@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.227@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.215@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.240@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.235@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.218@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
[...]
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.209@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.211@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.212@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.213@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.214@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.204@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.224@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.215@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.216@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.247@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.219@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: HA cluster member "(XXX.XX.XX.244@vmx0): (VMNETWORK)" has resumed CARP state "MASTER" for vhid 9
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: Starting OpenVPN server instance on VMNETWORK because of transition to CARP master.
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: Starting OpenVPN server instance on VMNETWORK because of transition to CARP master.
Feb 21 13:10:11 pfsense04 check_reload_status[451]: Reloading filter
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: Starting OpenVPN server instance on VMNETWORK because of transition to CARP master.
Feb 21 13:10:11 pfsense04 check_reload_status[451]: Reloading filter
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: Starting OpenVPN server instance on VMNETWORK because of transition to CARP master.
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: Starting OpenVPN server instance on VMNETWORK because of transition to CARP master.
Feb 21 13:10:11 pfsense04 php-fpm[411]: /rc.carpmaster: Reconfiguring IPsec because of transition to CARP master.
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: Starting OpenVPN server instance on VMNETWORK because of transition to CARP master.
Feb 21 13:10:11 pfsense04 php-fpm[99256]: /rc.carpmaster: Reconfiguring IPsec because of transition to CARP master.
Feb 21 13:10:11 pfsense04 php-fpm[73066]: /rc.carpmaster: HA cluster member "(192.168.115.254@vmx7): (HAPROXY_5005)" has resumed CARP state "MASTER" for vhid 8
Feb 21 13:10:11 pfsense04 php-fpm[412]: /rc.carpmaster: HA cluster member "(192.168.110.254@vmx1): (VLAN5000)" has resumed CARP state "MASTER" for vhid 4
Feb 21 13:10:12 pfsense04 php-fpm[412]: /rc.carpbackup: HA cluster member "(192.168.115.254@vmx7): (HAPROXY_5005)" has resumed CARP state "BACKUP" for vhid 8
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.195@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.233@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.236@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.234@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.196@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.225@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.205@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.198@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.197@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.209@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.
[...]
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.220@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.229@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.230@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.213@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.242@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.246@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[99256]: /rc.carpbackup: HA cluster member "(192.168.111.254@vmx3): (CRN5001)" has resumed CARP state "BACKUP" for vhid 3
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.200@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.201@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.202@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.203@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.204@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.205@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.206@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.207@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.214@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.209@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.211@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.212@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.213@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.214@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
[...]
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.221@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.222@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.234@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.235@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.249@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.236@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.237@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.233@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.224@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 2
Feb 21 13:10:12 pfsense04 php-fpm[411]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.195@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 9
Feb 21 13:10:12 pfsense04 php-fpm[411]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.233@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 9
Feb 21 13:10:12 pfsense04 php-fpm[411]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.236@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 9
Feb 21 13:10:12 pfsense04 php-fpm[411]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.234@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 9
Feb 21 13:10:12 pfsense04 php-fpm[411]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.196@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 9
Feb 21 13:10:12 pfsense04 php-fpm[411]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.225@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 9
Feb 21 13:10:12 pfsense04 php-fpm[411]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.205@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 9
Feb 21 13:10:12 pfsense04 php-fpm[411]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.198@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 9
[...]
Feb 21 13:10:12 pfsense04 php-fpm[411]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.202@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 9
Feb 21 13:10:12 pfsense04 php-fpm[411]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.203@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 9
Feb 21 13:10:12 pfsense04 php-fpm[411]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.204@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 9
Feb 21 13:10:12 pfsense04 php-fpm[411]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.205@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 9
Feb 21 13:10:12 pfsense04 php-fpm[411]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.206@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 9
Feb 21 13:10:12 pfsense04 php-fpm[411]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.207@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 9
Feb 21 13:10:12 pfsense04 php-fpm[411]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.214@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 9
Feb 21 13:10:12 pfsense04 php-fpm[411]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.209@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 9
Feb 21 13:10:12 pfsense04 php-fpm[411]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.211@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 9
Feb 21 13:10:12 pfsense04 php-fpm[411]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.212@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 9
Feb 21 13:10:12 pfsense04 php-fpm[411]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.213@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 9
Feb 21 13:10:12 pfsense04 php-fpm[411]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.214@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 9
Feb 21 13:10:12 pfsense04 php-fpm[411]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.204@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 9
Feb 21 13:10:12 pfsense04 php-fpm[411]: /rc.carpbackup: HA cluster member "(XXX.XX.XX.233@vmx0): (VMNETWORK)" has resumed CARP state "BACKUP" for vhid 9
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: Stopping OpenVPN server instance on VMNETWORK because of transition to CARP backup.
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: Stopping OpenVPN server instance on VMNETWORK because of transition to CARP backup.
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: Stopping OpenVPN server instance on VMNETWORK because of transition to CARP backup.
Feb 21 13:10:12 pfsense04 php-fpm[73066]: /rc.carpbackup: Reconfiguring IPsec because of transition to CARP backup.
Feb 21 13:10:12 pfsense04 php-fpm[411]: /rc.carpbackup: Stopping OpenVPN server instance on VMNETWORK because of transition to CARP backup.
Feb 21 13:10:12 pfsense04 php-fpm[411]: /rc.carpbackup: Stopping OpenVPN server instance on VMNETWORK because of transition to CARP backup.
Feb 21 13:10:12 pfsense04 php-fpm[411]: /rc.carpbackup: Stopping OpenVPN server instance on VMNETWORK because of transition to CARP backup.
Feb 21 13:10:12 pfsense04 php-fpm[411]: /rc.carpbackup: Reconfiguring IPsec because of transition to CARP backup.
Feb 21 13:10:12 pfsense04 php-fpm[412]: /rc.carpbackup: HA cluster member "(192.168.110.254@vmx1): (VLAN5000)" has resumed CARP state "BACKUP" for vhid 4
Feb 21 13:10:12 pfsense04 php-fpm[99256]: /rc.carpbackup: HA cluster member "(192.168.112.254@vmx4): (PCC2_5002)" has resumed CARP state "BACKUP" for vhid 5

              At the same time, there's nothing in the logs on the real "MASTER".

              We're using DRS in automatic mode, so it's also causing some vmotions...

              I think I need to find a way of increasing the delay for the "BACKUP" before it considers that there is a timeout.

              S 1 Reply Last reply Reply Quote 0
              • S
                SteveITS Galactic Empire @Yathus
                last edited by

                @Yathus said in how to debug CARP ?:

                using DRS in automatic mode, so it's also causing some vmotions

                We don't use VMWare but I would guess it is pausing the VM for a couple seconds while moving it, which is causing the drop, and if the master is paused it doesn't log anything (because it never sees a drop).

                possibly:
                https://docs.netgate.com/pfsense/en/latest/troubleshooting/high-availability.html
                https://docs.netgate.com/pfsense/en/latest/firewall/virtual-ip-addresses.html#carp

                "Advertising Frequency

                Controls how often the master node sends out CARP heartbeat advertisements. Hosts transmitting faster than their peers assume control of the VIP."

                Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                Upvote 👍 helpful posts!

                Y 1 Reply Last reply Reply Quote 0
                • Y
                  Yathus @SteveITS
                  last edited by

                  Hello @SteveITS

                  I follow all the recommendation on the documentation, specially this one : https://docs.netgate.com/pfsense/en/latest/troubleshooting/high-availability-virtual.html

                  Today we got another "crash", another move between host and losing connectivity for 3 min after the vmotion.

                  So we decide to shutdown the backup node, we'll see if problem occur again.

                  Y 1 Reply Last reply Reply Quote 0
                  • Y
                    Yathus @Yathus
                    last edited by Yathus

                    Problem occured again afternoon... Without backup node, it was a true downtime... Sometimes this job can drive you mad...

                    V 1 Reply Last reply Reply Quote 0
                    • V
                      viragomann @Yathus
                      last edited by

                      @Yathus
                      We are also running two pfSense in HA mode on ESXi7. But we have no issues like this.

                      A view day ago the hosts were taken down for maintenance and started up again, one by the other. The master moved over to the secondary with all services, IPSec, OpenVPN, HAproxy, and back again flawlessly.

                      However, we don't move around the VMs from one host to another.

                      1 Reply Last reply Reply Quote 0
                      • First post
                        Last post
                      Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.