Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Secondary pfSense rendomly setting itself as CARP MASTER

    Scheduled Pinned Locked Moved HA/CARP/VIPs
    21 Posts 2 Posters 3.8k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • DerelictD
      Derelict LAYER 8 Netgate
      last edited by

      There is no need to just change things unless evidence indicates it is a problem.

      Both CARP and VRRP derive the virtual MAC address from configured settings. The Virtual Host ID in the case of CARP and the VRID in the case of VRRP. 00-00-5E-00-01-XX, where XX is the ID in hex.

      They need to be unique on the broadcast domain or, like any case where you have two devices on the same broadcast domain using the same MAC address, there will be problems. If there is not a known collision there is no reason to change anything.

      I would avoid being clicky-clicky here and make changes based on evidence.

      Chattanooga, Tennessee, USA
      A comprehensive network diagram is worth 10,000 words and 15 conference calls.
      DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
      Do Not Chat For Help! NO_WAN_EGRESS(TM)

      1 Reply Last reply Reply Quote 0
      • DerelictD
        Derelict LAYER 8 Netgate
        last edited by

        What shows on Status > CARP on the secondary?

        Chattanooga, Tennessee, USA
        A comprehensive network diagram is worth 10,000 words and 15 conference calls.
        DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
        Do Not Chat For Help! NO_WAN_EGRESS(TM)

        C 1 Reply Last reply Reply Quote 0
        • C
          CPrat @Derelict
          last edited by

          @Derelict Okay, then since the VRRP and CARP ID seem to be different, there is no apparent need for me to change anything.

          I agree that I don't need to change anything if there is no evidence.

          The CARP status on the secondary shows now all Backup, and when the problem happens, it shows the WAN as Master. I have 19 VLANs configured with CARP

          Here is the current status:

          Screen Shot 2019-06-20 at 2.22.59 PM.png

          1 Reply Last reply Reply Quote 0
          • DerelictD
            Derelict LAYER 8 Netgate
            last edited by

            Any weird messages like a demotion set on that page or anything?

            When a secondary that should be BACKUP goes to MASTER it is prettty much invariable that it has stopped receiving heartbeats on that network from the master.

            What does this show on both primary and secondary:

            sysctl -a | grep carp

            Chattanooga, Tennessee, USA
            A comprehensive network diagram is worth 10,000 words and 15 conference calls.
            DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
            Do Not Chat For Help! NO_WAN_EGRESS(TM)

            C 1 Reply Last reply Reply Quote 0
            • C
              CPrat @Derelict
              last edited by

              @Derelict No weird messages, and I don't recall seeing any of such messages when the secondary is acting up. Nonetheless, I will reproduce the problem again tonight and show you if any message like that shows up.

              This is why I was surprised to see the secondary is picking up the messages (from a packet capture) from the primary when they are both showing up as master, and yet, it seems for some reason is not going back to be Backup.

              Here is the result of the commands on both. The primary is pfSense01 and the secondary is pfSense02

              [2.4.4-RELEASE][admin@pfSense01.localdomain]/root: sysctl -a | grep carp
              device carp
              net.inet.carp.ifdown_demotion_factor: 240
              net.inet.carp.senderr_demotion_factor: 0
              net.inet.carp.demotion: 0
              net.inet.carp.log: 1
              net.inet.carp.preempt: 1
              net.inet.carp.allow: 1
              net.pfsync.carp_demotion_factor: 0

              [2.4.4-RELEASE][admin@pfSense02.localdomain]/root: sysctl -a | grep carp
              device carp
              net.inet.carp.ifdown_demotion_factor: 240
              net.inet.carp.senderr_demotion_factor: 0
              net.inet.carp.demotion: 0
              net.inet.carp.log: 1
              net.inet.carp.preempt: 1
              net.inet.carp.allow: 1
              net.pfsync.carp_demotion_factor: 0

              1 Reply Last reply Reply Quote 0
              • DerelictD
                Derelict LAYER 8 Netgate
                last edited by

                That all looks perfectly normal.

                The capture posted clearly shows the secondary using an advskew of 240, yet you say the advskews are all set at 100 as they should be.

                Another interesting data point would be the output of ifconfig -a at the time.

                Chattanooga, Tennessee, USA
                A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                Do Not Chat For Help! NO_WAN_EGRESS(TM)

                C 1 Reply Last reply Reply Quote 0
                • C
                  CPrat @Derelict
                  last edited by

                  @Derelict If you see the other captures posted, it shows a skew of 100 configured on the secondary's GUI, it's strange that it says 240 through the console.

                  Screen Shot 2019-06-20 at 2.57.29 PM.png

                  Also, I logged into the primary, and it triggered the secondary picking up as Master, so I had to reboot it, but I got you a screenshot first, and the capture of the commands you asked, while the secondary was acting as Master.

                  Screen Shot 2019-06-20 at 2.51.59 PM.png

                  This was while Both, the primary and the secondary were showing up as Master for the WAN:

                  [2.4.4-RELEASE][admin@pfSense01.localdomain]/root: sysctl -a | grep carp
                  device carp
                  net.inet.carp.ifdown_demotion_factor: 240
                  net.inet.carp.senderr_demotion_factor: 0
                  net.inet.carp.demotion: 0
                  net.inet.carp.log: 1
                  net.inet.carp.preempt: 1
                  net.inet.carp.allow: 1
                  net.pfsync.carp_demotion_factor: 0

                  [2.4.4-RELEASE][admin@pfSense02.localdomain]/root: sysctl -a | grep carp
                  device carp
                  net.inet.carp.ifdown_demotion_factor: 240
                  net.inet.carp.senderr_demotion_factor: 0
                  net.inet.carp.demotion: 0
                  net.inet.carp.log: 1
                  net.inet.carp.preempt: 1
                  net.inet.carp.allow: 1
                  net.pfsync.carp_demotion_factor: 0

                  For the interface settings, I created a pastebin since the output is giant, and to not completely clog this thread:
                  Again, While the secondary was acting as Master

                  https://pastebin.com/4dek4qGJ

                  1 Reply Last reply Reply Quote 0
                  • C
                    CPrat
                    last edited by

                    The most relevant that I see is with the WAN interface:

                    The primary:

                    hn1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
                    options=48001b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,LINKSTATE,TXCSUM_IPV6>
                    ether 00:15:5d:34:44:15
                    hwaddr 00:15:5d:34:44:15
                    inet6 fe80::215:5dff:fe34:4415%hn1 prefixlen 64 scopeid 0x6
                    inet 205.251.108.165 netmask 0xffffffe0 broadcast 205.251.108.191
                    inet 205.251.108.169 netmask 0xffffffe0 broadcast 205.251.108.191 vhid 22
                    inet 205.251.108.170 netmask 0xffffffe0 broadcast 205.251.108.191 vhid 22
                    inet 205.251.108.171 netmask 0xffffffe0 broadcast 205.251.108.191 vhid 22
                    inet 205.251.108.172 netmask 0xffffffe0 broadcast 205.251.108.191 vhid 22
                    inet 205.251.108.173 netmask 0xffffffe0 broadcast 205.251.108.191 vhid 22
                    inet 205.251.108.174 netmask 0xffffffe0 broadcast 205.251.108.191 vhid 22
                    inet 205.251.108.175 netmask 0xffffffe0 broadcast 205.251.108.191 vhid 22
                    inet 205.251.108.176 netmask 0xffffffe0 broadcast 205.251.108.191 vhid 22
                    inet 205.251.108.177 netmask 0xffffffe0 broadcast 205.251.108.191 vhid 22
                    inet 205.251.108.178 netmask 0xffffffe0 broadcast 205.251.108.191 vhid 22
                    inet 205.251.108.179 netmask 0xffffffe0 broadcast 205.251.108.191 vhid 22
                    inet 205.251.108.180 netmask 0xffffffe0 broadcast 205.251.108.191 vhid 22
                    inet 205.251.108.181 netmask 0xffffffe0 broadcast 205.251.108.191 vhid 22
                    inet 205.251.108.182 netmask 0xffffffe0 broadcast 205.251.108.191 vhid 22
                    inet 205.251.108.183 netmask 0xffffffe0 broadcast 205.251.108.191 vhid 22
                    inet 205.251.108.184 netmask 0xffffffe0 broadcast 205.251.108.191 vhid 22
                    inet 205.251.108.164 netmask 0xffffffe0 broadcast 205.251.108.191 vhid 22
                    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
                    media: Ethernet autoselect (10Gbase-T <full-duplex>)
                    status: active
                    carp: MASTER vhid 22 advbase 1 advskew 0

                    The secondary:

                    hn1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
                    options=48001b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,LINKSTATE,TXCSUM_IPV6>
                    ether 00:15:5d:b5:91:0f
                    hwaddr 00:15:5d:b5:91:0f
                    inet6 fe80::215:5dff:feb5:910f%hn1 prefixlen 64 scopeid 0x6
                    inet 205.251.108.166 netmask 0xffffffe0 broadcast 205.251.108.191
                    inet 205.251.108.169 netmask 0xffffffe0 broadcast 205.251.108.191 vhid 22
                    inet 205.251.108.170 netmask 0xffffffe0 broadcast 205.251.108.191 vhid 22
                    inet 205.251.108.172 netmask 0xffffffe0 broadcast 205.251.108.191 vhid 22
                    inet 205.251.108.173 netmask 0xffffffe0 broadcast 205.251.108.191 vhid 22
                    inet 205.251.108.174 netmask 0xffffffe0 broadcast 205.251.108.191 vhid 22
                    inet 205.251.108.175 netmask 0xffffffe0 broadcast 205.251.108.191 vhid 22
                    inet 205.251.108.176 netmask 0xffffffe0 broadcast 205.251.108.191 vhid 22
                    inet 205.251.108.177 netmask 0xffffffe0 broadcast 205.251.108.191 vhid 22
                    inet 205.251.108.178 netmask 0xffffffe0 broadcast 205.251.108.191 vhid 22
                    inet 205.251.108.179 netmask 0xffffffe0 broadcast 205.251.108.191 vhid 22
                    inet 205.251.108.180 netmask 0xffffffe0 broadcast 205.251.108.191 vhid 22
                    inet 205.251.108.181 netmask 0xffffffe0 broadcast 205.251.108.191 vhid 22
                    inet 205.251.108.182 netmask 0xffffffe0 broadcast 205.251.108.191 vhid 22
                    inet 205.251.108.183 netmask 0xffffffe0 broadcast 205.251.108.191 vhid 22
                    inet 205.251.108.184 netmask 0xffffffe0 broadcast 205.251.108.191 vhid 22
                    inet 205.251.108.164 netmask 0xffffffe0 broadcast 205.251.108.191 vhid 22
                    inet 205.251.108.171 netmask 0xffffffe0 broadcast 205.251.108.191
                    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
                    media: Ethernet autoselect (10Gbase-T <full-duplex>)
                    status: active
                    carp: MASTER vhid 22 advbase 1 advskew 254

                    For some reason, the last inet (205.251.108.171) does not show with a vhid on the secondary, and the first one is the interface IP address, so I think it's normal it doesn't show a vhid

                    1 Reply Last reply Reply Quote 0
                    • DerelictD
                      Derelict LAYER 8 Netgate
                      last edited by

                      The secondary thinks it has an interface down and has been demoted (hence advskew 254)

                      You have something screwed up somewhere. Sorry but with what we have that is the best I can do here.

                      My guess is something in Hyper-V. Hard to say. But these problems are almost always Layer 2 problems.

                      Chattanooga, Tennessee, USA
                      A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                      DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                      Do Not Chat For Help! NO_WAN_EGRESS(TM)

                      C 1 Reply Last reply Reply Quote 0
                      • C
                        CPrat @Derelict
                        last edited by CPrat

                        @Derelict Just so I understand it, this interface down would be the virtual one ended in 171 since it's not showing the vhid? Because there is only one interface, the WAN, and it's showing as active. The others are IP Alias of that interface made in pfSense

                        The only reason I'm doubting it could anything in Hyper-v is because this same machines were all working fine until I switched to a different datacenter provider, so there's got to be a change somewhere or they are messing something up with some traffic, or I configured something wrong.

                        Even now that it's showing as backup, it's showing an advskew of 254

                        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect (10Gbase-T <full-duplex>)
status: active
carp: BACKUP vhid 22 advbase 1 advskew 254
                        1 Reply Last reply Reply Quote 0
                        • DerelictD
                          Derelict LAYER 8 Netgate
                          last edited by

                          On a healthy system the primary would be showing skew 0, the secondary skew 100.

                          Check the system log for entries related to why it is changing the skew to 254.

                          Chattanooga, Tennessee, USA
                          A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                          DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                          Do Not Chat For Help! NO_WAN_EGRESS(TM)

                          1 Reply Last reply Reply Quote 0
                          • First post
                            Last post
                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.