Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    HA CARP - IPv6 Two masters

    Scheduled Pinned Locked Moved HA/CARP/VIPs
    56 Posts 11 Posters 14.9k Views 6 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • DerelictD Offline
      Derelict LAYER 8 Netgate
      last edited by

      Doesn't sound like a bug because far too many people are NOT seeing the issue. It is something specific to the way you have it configured or something in your environment.

      Chattanooga, Tennessee, USA
      A comprehensive network diagram is worth 10,000 words and 15 conference calls.
      DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
      Do Not Chat For Help! NO_WAN_EGRESS(TM)

      1 Reply Last reply Reply Quote 0
      • R Offline
        rhwendt
        last edited by

        Thank you all for helping.

        I just factory rest both devices today and set them up from scratch again.
        All the carp interfaces were working as expected except the IPV6 ULA CARP for the LAN (fd57:187e:523f:715::f/64)
        It was exhibiting the same issues i was seeing prior to the factory reset, both primary and secondary both showing master.
        The IPV6 GUA on the wan was working as expected
        If I rebooted the secondary firewall all the carp interfaces would be in backup status. Anytime I synced the config from the primary it would cause the double master status.

        I was able to find a solution based off what awebster said about unchecking the virtual ip in the HA sync.
        I unchecked this option and rebooted the secondary firewall and now all the carp interfaces are showing the correct status and config syncing doesnt affect them.

        1 Reply Last reply Reply Quote 0
        • DerelictD Offline
          Derelict LAYER 8 Netgate
          last edited by

          Just to be sure there wasn't something somewhere that misbehaved with ULA and CARP:

          Primary:

          xn5: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
          options=3 <rxcsum,txcsum>ether ee:c2:d9:d8:55:46
          hwaddr ee:c2:d9:d8:55:46
          inet6 fe80::ecc2:d9ff:fed8:5546%xn5 prefixlen 64 scopeid 0xd
          inet6 fda9:cfd8:f9f:1000::2 prefixlen 64
          inet6 fda9:cfd8:f9f:1000::1 prefixlen 64 vhid 243
          inet 192.168.123.2 netmask 0xffffff00 broadcast 192.168.123.255
          inet 192.168.123.1 netmask 0xffffff00 broadcast 192.168.123.255 vhid 242
          nd6 options=21 <performnud,auto_linklocal>media: Ethernet manual
          status: active
          carp: MASTER vhid 242 advbase 1 advskew 0
          carp: MASTER vhid 243 advbase 1 advskew 0

          Secondary:

          xn5: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
          ether 6e:24:e4:84:f5:f9
          hwaddr 6e:24:e4:84:f5:f9
          inet6 fe80::6c24:e4ff:fe84:f5f9%xn5 prefixlen 64 scopeid 0xa
          inet6 fda9:cfd8:f9f:1000::3 prefixlen 64
          inet6 fda9:cfd8:f9f:1000::1 prefixlen 64 vhid 243
          inet 192.168.123.3 netmask 0xffffff00 broadcast 192.168.123.255
          inet 192.168.123.1 netmask 0xffffff00 broadcast 192.168.123.255 vhid 242
          nd6 options=21 <performnud,auto_linklocal>media: Ethernet manual
          status: active
          carp: BACKUP vhid 242 advbase 1 advskew 100
          carp: BACKUP vhid 243 advbase 1 advskew 100

          Enter CARP Maintenance mode on Primary:

          Primary:

          xn5: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
          options=3 <rxcsum,txcsum>ether ee:c2:d9:d8:55:46
          hwaddr ee:c2:d9:d8:55:46
          inet6 fe80::ecc2:d9ff:fed8:5546%xn5 prefixlen 64 scopeid 0xd
          inet6 fda9:cfd8:f9f:1000::2 prefixlen 64
          inet6 fda9:cfd8:f9f:1000::1 prefixlen 64 vhid 243
          inet 192.168.123.2 netmask 0xffffff00 broadcast 192.168.123.255
          inet 192.168.123.1 netmask 0xffffff00 broadcast 192.168.123.255 vhid 242
          nd6 options=21 <performnud,auto_linklocal>media: Ethernet manual
          status: active
          carp: BACKUP vhid 242 advbase 1 advskew 254
          carp: BACKUP vhid 243 advbase 1 advskew 254

          Secondary:

          xn5: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
          ether 6e:24:e4:84:f5:f9
          hwaddr 6e:24:e4:84:f5:f9
          inet6 fe80::6c24:e4ff:fe84:f5f9%xn5 prefixlen 64 scopeid 0xa
          inet6 fda9:cfd8:f9f:1000::3 prefixlen 64
          inet6 fda9:cfd8:f9f:1000::1 prefixlen 64 vhid 243
          inet 192.168.123.3 netmask 0xffffff00 broadcast 192.168.123.255
          inet 192.168.123.1 netmask 0xffffff00 broadcast 192.168.123.255 vhid 242
          nd6 options=21 <performnud,auto_linklocal>media: Ethernet manual
          status: active
          carp: MASTER vhid 242 advbase 1 advskew 100
          carp: MASTER vhid 243 advbase 1 advskew 100

          Fails back fine, too.</performnud,auto_linklocal></up,broadcast,running,promisc,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum></up,broadcast,running,promisc,simplex,multicast></performnud,auto_linklocal></up,broadcast,running,promisc,simplex,multicast></performnud,auto_linklocal></rxcsum,txcsum></up,broadcast,running,promisc,simplex,multicast>

          Chattanooga, Tennessee, USA
          A comprehensive network diagram is worth 10,000 words and 15 conference calls.
          DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
          Do Not Chat For Help! NO_WAN_EGRESS(TM)

          1 Reply Last reply Reply Quote 0
          • A Offline
            aeburriel
            last edited by

            I've been dealing with the same problem in my HA setup and it turned to be related to bug #6579
            https://redmine.pfsense.org/issues/6579

            The affected CARP IPv6 address was something like:
            2001:aaaa:bbb:ccc:0d00:ffff:ffff:ffff
            After removing leading zeros:
            2001:aaaa:bbb:ccc:d00:ffff:ffff:ffff

            CARP started to work reliably on that interface

            1 Reply Last reply Reply Quote 0
            • DerelictD Offline
              Derelict LAYER 8 Netgate
              last edited by

              Nice catch.

              LAN@213  fd57:187e:523f:0715::f    MASTER

              Chattanooga, Tennessee, USA
              A comprehensive network diagram is worth 10,000 words and 15 conference calls.
              DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
              Do Not Chat For Help! NO_WAN_EGRESS(TM)

              1 Reply Last reply Reply Quote 0
              • I Offline
                IcePick
                last edited by

                I am having exactly the same as this since moving to 2.4 from 2.3.5
                interesting only on 2 of the 4 IPv6 CARPs
                they were the only 2 that could use :: in their address
                I tried expanding to 0:0:0:
                it did not help

                I have confirmed by packet capture that packets to ff02::12 are seen on both systems

                –------------
                Ok I figured out how to get it to a normal state (all master on primary and all backup on secondary).
                You need to reboot the backup firewall, and while its rebooting clear the firewall states on the primary.
                Carp failover works perfectly when its like this but there is still an issue.

                ANY configuration sync (manual/auto) from the primary to the backup causes the backup to become master on the two IPV6 carps.

                1 Reply Last reply Reply Quote 0
                • I Offline
                  IcePick
                  last edited by

                  To reiterate I did not have this issue until upgrading from 2.3.5 to 2.4.2-RELEASE-p1, or at least it seemed to have gotten worse.

                  More testing:
                  changed from x::1 (ie X:0:0:0:1)  to x:1:1:1:1
                  on one of the CARP interfaces and the problem went away
                  Did not change the real interface IP

                  UPDATE: it worked for the first one, but broke both after I changed the second one.
                  Why are these 2 different then the other 2?
                  they connect to the same switch
                  I found a difference, one set of addresses used all lower case for the hex in the address, the none working ones had capitals.
                  I have changed all to lower and rebooted B unit and it came up all in backup, did not have to reset states on A firewall.
                  I'm not saying this is the issue - but giving people ideas of what I found
                  So in summary: using all lower case for hex and changed the addresses to ones that can not condense to ::

                  1 Reply Last reply Reply Quote 0
                  • A Offline
                    anthonysomerset
                    last edited by

                    Just want to add i appear to have hit this "bug" in one of our SG-4860 clusters

                    our IPv6 addresses are in their shortened form with no leading zeros, had to reboot secondary to clear this out, will keep an eye on things

                    1 Reply Last reply Reply Quote 0
                    • X Offline
                      xciter327
                      last edited by

                      I am also hitting something similar this in our office/test system.

                      Both devices are connected to a Cisco 3560G switch. IGMP snooping and ipv6 mld snooping are disabled. All ports are set to "portfast". There are no "loops" in the network. There are no topology changes.

                      You will notice that each one sees the others advertisements and their own.

                      Primary:
                      16:42:40.428976 IP6 fe80::ec4:7aff:feab:3724 > ff02::12: ip-proto-112 36
                      16:42:42.597228 IP6 fe80::ec4:7aff:feac:821a > ff02::12: ip-proto-112 36
                      16:42:50.886692 IP6 fe80::ec4:7aff:feab:3724 > ff02::12: ip-proto-112 36
                      16:42:52.607533 IP6 fe80::ec4:7aff:feac:821a > ff02::12: ip-proto-112 36
                      16:43:01.382988 IP6 fe80::ec4:7aff:feab:3724 > ff02::12: ip-proto-112 36
                      16:43:02.612549 IP6 fe80::ec4:7aff:feac:821a > ff02::12: ip-proto-112 36

                      Backup:
                      16:42:09.212760 IP6 fe80::ec4:7aff:feab:3724 > ff02::12: ip-proto-112 36
                      16:42:12.573960 IP6 fe80::ec4:7aff:feac:821a > ff02::12: ip-proto-112 36
                      16:42:19.608720 IP6 fe80::ec4:7aff:feab:3724 > ff02::12: ip-proto-112 36
                      16:42:22.578900 IP6 fe80::ec4:7aff:feac:821a > ff02::12: ip-proto-112 36
                      16:42:30.015028 IP6 fe80::ec4:7aff:feab:3724 > ff02::12: ip-proto-112 36
                      16:42:32.585911 IP6 fe80::ec4:7aff:feac:821a > ff02::12: ip-proto-112 36

                      This only happens for IPv6 CARP IPs.

                      Here are the interfaces, just to confirm the vhid:

                      Primary:
                      igb0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
                      options=6400bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,rxcsum_ipv6,txcsum_ipv6>ether 0c:c4:7a:ac:82:1a
                      hwaddr 0c:c4:7a:ac:82:1a
                      inet6 fe80::ec4:7aff:feac:821a%igb0 prefixlen 64 scopeid 0x1
                      inet6 xxxx:xxxx:1:2::3 prefixlen 124
                      inet6 xxxx:xxxx:1:2::2 prefixlen 124 vhid 4
                      inet yyy.yyy.233.108 netmask 0xfffffff0 broadcast yyy.yyy.233.111
                      inet yyy.yyy.233.110 netmask 0xfffffff0 broadcast yyy.yyy.233.111 vhid 1
                      nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
                      status: active
                      carp: MASTER vhid 1 advbase 10 advskew 1
                      carp: MASTER vhid 4 advbase 10 advskew 1

                      Backup:
                      igb0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
                      options=6400bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,rxcsum_ipv6,txcsum_ipv6>ether 0c:c4:7a🆎37:24
                      hwaddr 0c:c4:7a🆎37:24
                      inet6 fe80::ec4:7aff:feab:3724%igb0 prefixlen 64 scopeid 0x1
                      inet6 xxxx:xxxx:1:2::4 prefixlen 124
                      inet yyy.yyy.233.109 netmask 0xfffffff0 broadcast yyy.yyy.233.111
                      inet yyy.yyy.233.110 netmask 0xfffffff0 broadcast yyy.yyy.233.111 vhid 1
                      nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
                      status: active
                      carp: MASTER vhid 4 advbase 10 advskew 101
                      carp: BACKUP vhid 1 advbase 10 advskew 101

                      So the CARP interface is correctly assigned to the primary node, but the backup one still claims its master in the dashboard and with "ifconfig igb0".</full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,rxcsum_ipv6,txcsum_ipv6></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,rxcsum_ipv6,txcsum_ipv6></up,broadcast,running,promisc,simplex,multicast>

                      1 Reply Last reply Reply Quote 0
                      • DerelictD Offline
                        Derelict LAYER 8 Netgate
                        last edited by Derelict

                        Why did you play with advbase/advskew?

                        Use 1/0 on the primary that will sync 1/100 to the secondary. Then just leave it alone.

                        Chattanooga, Tennessee, USA
                        A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                        DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                        Do Not Chat For Help! NO_WAN_EGRESS(TM)

                        1 Reply Last reply Reply Quote 0
                        • X Offline
                          xciter327
                          last edited by

                          Yes. I did try multiple base values between 0 - 20 for the base and 0 and 1 for skew. The settings are correctly(+100 for skew) transferred to the backup unit. Still backup thinks it's primary for IPv6.

                          1 Reply Last reply Reply Quote 0
                          • DerelictD Offline
                            Derelict LAYER 8 Netgate
                            last edited by Derelict

                            Are you 100% certain the case described in reply #15 ^ is not present?

                            Use 1/0 on the primary that will sync 1/100 to the secondary. Then just leave it alone.

                            Just do that. If changing it didn't correct it it is not the problem.

                            Packet capture on both nodes and see if you see the CARP going out the interface or in the interface. You can filter on CARP only in Diagnostics > Packet Capture.

                            Chattanooga, Tennessee, USA
                            A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                            DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                            Do Not Chat For Help! NO_WAN_EGRESS(TM)

                            1 Reply Last reply Reply Quote 0
                            • X Offline
                              xciter327
                              last edited by

                              1. Regarding post #15 solution. I tried both shorthand(no leading zeroes) and full notation with nothing omitted.
                              2. I included a tcpdump in my first post. It looks to me that they are both receiving each other's updates.

                              1 Reply Last reply Reply Quote 0
                              • I Offline
                                IcePick
                                last edited by

                                Have you tried changing to addresses that CAN NOT be shortened to have a :: ?

                                1 Reply Last reply Reply Quote 0
                                • X Offline
                                  xciter327
                                  last edited by

                                  @IcePick:

                                  Have you tried changing to addresses that CAN NOT be shortened to have a :: ?

                                  Yes I did. No difference.

                                  1 Reply Last reply Reply Quote 0
                                  • DerelictD Offline
                                    Derelict LAYER 8 Netgate
                                    last edited by

                                    Did you put base/skew back to the default or not?

                                    Chattanooga, Tennessee, USA
                                    A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                    DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                    Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                    1 Reply Last reply Reply Quote 0
                                    • X Offline
                                      xciter327
                                      last edited by

                                      @Derelict:

                                      Did you put base/skew back to the default or not?

                                      Yes, I did.

                                      1 Reply Last reply Reply Quote 0
                                      • DerelictD Offline
                                        Derelict LAYER 8 Netgate
                                        last edited by

                                        Well, cut loose with more. Screen shots, pcaps, whatever. IPv6 CARP works.

                                        Chattanooga, Tennessee, USA
                                        A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                        DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                        Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                        1 Reply Last reply Reply Quote 0
                                        • X Offline
                                          xciter327
                                          last edited by

                                          I disabled "DHCP Snooping" on the directly connected switch. That was somehow blocking stuff. Seems to be working OK now. I can no longer reproduce the issue. Will post if I can.

                                          1 Reply Last reply Reply Quote 0
                                          • DerelictD Offline
                                            Derelict LAYER 8 Netgate
                                            last edited by

                                            Amazing. It was a setting on the switch. Simply amazing.

                                            Glad you found it.

                                            Chattanooga, Tennessee, USA
                                            A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                            DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                            Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.