Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    HA CARP - IPv6 Two masters

    Scheduled Pinned Locked Moved HA/CARP/VIPs
    56 Posts 11 Posters 14.9k Views 6 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • X Offline
      xciter327
      last edited by

      I am also hitting something similar this in our office/test system.

      Both devices are connected to a Cisco 3560G switch. IGMP snooping and ipv6 mld snooping are disabled. All ports are set to "portfast". There are no "loops" in the network. There are no topology changes.

      You will notice that each one sees the others advertisements and their own.

      Primary:
      16:42:40.428976 IP6 fe80::ec4:7aff:feab:3724 > ff02::12: ip-proto-112 36
      16:42:42.597228 IP6 fe80::ec4:7aff:feac:821a > ff02::12: ip-proto-112 36
      16:42:50.886692 IP6 fe80::ec4:7aff:feab:3724 > ff02::12: ip-proto-112 36
      16:42:52.607533 IP6 fe80::ec4:7aff:feac:821a > ff02::12: ip-proto-112 36
      16:43:01.382988 IP6 fe80::ec4:7aff:feab:3724 > ff02::12: ip-proto-112 36
      16:43:02.612549 IP6 fe80::ec4:7aff:feac:821a > ff02::12: ip-proto-112 36

      Backup:
      16:42:09.212760 IP6 fe80::ec4:7aff:feab:3724 > ff02::12: ip-proto-112 36
      16:42:12.573960 IP6 fe80::ec4:7aff:feac:821a > ff02::12: ip-proto-112 36
      16:42:19.608720 IP6 fe80::ec4:7aff:feab:3724 > ff02::12: ip-proto-112 36
      16:42:22.578900 IP6 fe80::ec4:7aff:feac:821a > ff02::12: ip-proto-112 36
      16:42:30.015028 IP6 fe80::ec4:7aff:feab:3724 > ff02::12: ip-proto-112 36
      16:42:32.585911 IP6 fe80::ec4:7aff:feac:821a > ff02::12: ip-proto-112 36

      This only happens for IPv6 CARP IPs.

      Here are the interfaces, just to confirm the vhid:

      Primary:
      igb0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
      options=6400bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,rxcsum_ipv6,txcsum_ipv6>ether 0c:c4:7a:ac:82:1a
      hwaddr 0c:c4:7a:ac:82:1a
      inet6 fe80::ec4:7aff:feac:821a%igb0 prefixlen 64 scopeid 0x1
      inet6 xxxx:xxxx:1:2::3 prefixlen 124
      inet6 xxxx:xxxx:1:2::2 prefixlen 124 vhid 4
      inet yyy.yyy.233.108 netmask 0xfffffff0 broadcast yyy.yyy.233.111
      inet yyy.yyy.233.110 netmask 0xfffffff0 broadcast yyy.yyy.233.111 vhid 1
      nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
      status: active
      carp: MASTER vhid 1 advbase 10 advskew 1
      carp: MASTER vhid 4 advbase 10 advskew 1

      Backup:
      igb0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
      options=6400bb <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,rxcsum_ipv6,txcsum_ipv6>ether 0c:c4:7a🆎37:24
      hwaddr 0c:c4:7a🆎37:24
      inet6 fe80::ec4:7aff:feab:3724%igb0 prefixlen 64 scopeid 0x1
      inet6 xxxx:xxxx:1:2::4 prefixlen 124
      inet yyy.yyy.233.109 netmask 0xfffffff0 broadcast yyy.yyy.233.111
      inet yyy.yyy.233.110 netmask 0xfffffff0 broadcast yyy.yyy.233.111 vhid 1
      nd6 options=21 <performnud,auto_linklocal>media: Ethernet autoselect (1000baseT <full-duplex>)
      status: active
      carp: MASTER vhid 4 advbase 10 advskew 101
      carp: BACKUP vhid 1 advbase 10 advskew 101

      So the CARP interface is correctly assigned to the primary node, but the backup one still claims its master in the dashboard and with "ifconfig igb0".</full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,rxcsum_ipv6,txcsum_ipv6></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,auto_linklocal></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,jumbo_mtu,vlan_hwcsum,vlan_hwtso,rxcsum_ipv6,txcsum_ipv6></up,broadcast,running,promisc,simplex,multicast>

      1 Reply Last reply Reply Quote 0
      • DerelictD Offline
        Derelict LAYER 8 Netgate
        last edited by Derelict

        Why did you play with advbase/advskew?

        Use 1/0 on the primary that will sync 1/100 to the secondary. Then just leave it alone.

        Chattanooga, Tennessee, USA
        A comprehensive network diagram is worth 10,000 words and 15 conference calls.
        DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
        Do Not Chat For Help! NO_WAN_EGRESS(TM)

        1 Reply Last reply Reply Quote 0
        • X Offline
          xciter327
          last edited by

          Yes. I did try multiple base values between 0 - 20 for the base and 0 and 1 for skew. The settings are correctly(+100 for skew) transferred to the backup unit. Still backup thinks it's primary for IPv6.

          1 Reply Last reply Reply Quote 0
          • DerelictD Offline
            Derelict LAYER 8 Netgate
            last edited by Derelict

            Are you 100% certain the case described in reply #15 ^ is not present?

            Use 1/0 on the primary that will sync 1/100 to the secondary. Then just leave it alone.

            Just do that. If changing it didn't correct it it is not the problem.

            Packet capture on both nodes and see if you see the CARP going out the interface or in the interface. You can filter on CARP only in Diagnostics > Packet Capture.

            Chattanooga, Tennessee, USA
            A comprehensive network diagram is worth 10,000 words and 15 conference calls.
            DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
            Do Not Chat For Help! NO_WAN_EGRESS(TM)

            1 Reply Last reply Reply Quote 0
            • X Offline
              xciter327
              last edited by

              1. Regarding post #15 solution. I tried both shorthand(no leading zeroes) and full notation with nothing omitted.
              2. I included a tcpdump in my first post. It looks to me that they are both receiving each other's updates.

              1 Reply Last reply Reply Quote 0
              • I Offline
                IcePick
                last edited by

                Have you tried changing to addresses that CAN NOT be shortened to have a :: ?

                1 Reply Last reply Reply Quote 0
                • X Offline
                  xciter327
                  last edited by

                  @IcePick:

                  Have you tried changing to addresses that CAN NOT be shortened to have a :: ?

                  Yes I did. No difference.

                  1 Reply Last reply Reply Quote 0
                  • DerelictD Offline
                    Derelict LAYER 8 Netgate
                    last edited by

                    Did you put base/skew back to the default or not?

                    Chattanooga, Tennessee, USA
                    A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                    DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                    Do Not Chat For Help! NO_WAN_EGRESS(TM)

                    1 Reply Last reply Reply Quote 0
                    • X Offline
                      xciter327
                      last edited by

                      @Derelict:

                      Did you put base/skew back to the default or not?

                      Yes, I did.

                      1 Reply Last reply Reply Quote 0
                      • DerelictD Offline
                        Derelict LAYER 8 Netgate
                        last edited by

                        Well, cut loose with more. Screen shots, pcaps, whatever. IPv6 CARP works.

                        Chattanooga, Tennessee, USA
                        A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                        DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                        Do Not Chat For Help! NO_WAN_EGRESS(TM)

                        1 Reply Last reply Reply Quote 0
                        • X Offline
                          xciter327
                          last edited by

                          I disabled "DHCP Snooping" on the directly connected switch. That was somehow blocking stuff. Seems to be working OK now. I can no longer reproduce the issue. Will post if I can.

                          1 Reply Last reply Reply Quote 0
                          • DerelictD Offline
                            Derelict LAYER 8 Netgate
                            last edited by

                            Amazing. It was a setting on the switch. Simply amazing.

                            Glad you found it.

                            Chattanooga, Tennessee, USA
                            A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                            DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                            Do Not Chat For Help! NO_WAN_EGRESS(TM)

                            1 Reply Last reply Reply Quote 0
                            • X Offline
                              xciter327
                              last edited by

                              So the problem is kind of back.

                              Same situation. Secondary pfsense become master for both IPv6 CARP groups, both report as master. The weird thing now is that if I shut down the secondary pfsense box IPv6 stops working completely. The primary box reports CARP status "Master"(as it always does), but the address is not reachable on the local LAN.

                              IGMP / DHCP snooping is disabled on the two switches between test PC and firewalls. The IPv4 CARP works fine.

                              1 Reply Last reply Reply Quote 0
                              • DerelictD Offline
                                Derelict LAYER 8 Netgate
                                last edited by

                                Again, it sounds like something at layer 2.

                                Either of the nodes will show MASTER if it does not receive the heartbeats from the other node. Solving dual MASTER is generally as simple as fixing the reason(s) that one node is not seeing the heartbeats from the other node.

                                Chattanooga, Tennessee, USA
                                A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                1 Reply Last reply Reply Quote 0
                                • X Offline
                                  xciter327
                                  last edited by xciter327

                                  It most definitely not L2 issue. The devices can see each other. Confirmed with tcpdump.( tcpdump -i igb0 -ttt -n proto CARP). There are 2 VHIDs on this interface.

                                  Master:
                                  00:00:00.000094 IP 217.117.yyy.xxx > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 1, authtype none, intvl 1s, length 36
                                  00:00:01.004961 IP 217.117.yyy.xxx > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 1, authtype none, intvl 1s, length 36
                                  00:00:00.000103 IP6 fe80::ec4:7aff:feac:821a > ff02::12: ip-proto-112 36
                                  00:00:01.005069 IP6 fe80::ec4:7aff:feac:821a > ff02::12: ip-proto-112 36

                                  Backup:
                                  00:00:00.000064 IP 217.117.yyy.xxx > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 1, authtype none, intvl 1s, length 36
                                  00:00:01.004781 IP 217.117.yyy.xxx > 224.0.0.18: VRRPv2, Advertisement, vrid 1, prio 1, authtype none, intvl 1s, length 36
                                  00:00:00.000062 IP6 fe80::ec4:7aff:feac:821a > ff02::12: ip-proto-112 36
                                  00:00:01.004907 IP6 fe80::ec4:7aff:feac:821a > ff02::12: ip-proto-112 36

                                  What I did is to download a config backup from each unit and a do a restore from config. That fixed the issue for me. There were no changes made to the underlying switching network.

                                  I also hit this bug with 2.4.3-Release-P1. That left me with one extra VHID on each interface stuck in "INIT" state. Rebooting the firewall is the only way I found to fix it.

                                  P.S. - Yesterday I also tried shutting down the "Backup" unit and fully un-plugging it from the network. While the IPv6 CARP interface on the LAN was showing as "up" and "master" on the only firewall left, IPv6 connectivity was not working until I rebooted the firewall.

                                  1 Reply Last reply Reply Quote 0
                                  • X Offline
                                    xciter327
                                    last edited by

                                    This post is deleted!
                                    1 Reply Last reply Reply Quote 0
                                    • X Offline
                                      xciter327
                                      last edited by xciter327

                                      Happens on other interfaces too(igb2.12):

                                      Master:
                                      00:00:00.000000 IP 172.28.0.1 > 224.0.0.18: VRRPv2, Advertisement, vrid 6, prio 1, authtype none, intvl 1s, length 36
                                      00:00:01.009252 IP 172.28.0.1 > 224.0.0.18: VRRPv2, Advertisement, vrid 6, prio 1, authtype none, intvl 1s, length 36

                                      Backup:
                                      00:00:00.000000 IP 172.28.0.1 > 224.0.0.18: VRRPv2, Advertisement, vrid 6, prio 1, authtype none, intvl 1s, length 36
                                      00:00:01.010086 IP 172.28.0.1 > 224.0.0.18: VRRPv2, Advertisement, vrid 6, prio 1, authtype none, intvl 1s, length 36

                                      Interface shows "Master" on both devices.

                                      igb1:
                                      Master:
                                      00:00:00.431700 IP 172.29.100.1 > 224.0.0.18: VRRPv2, Advertisement, vrid 2, prio 1, authtype none, intvl 1s, length 36
                                      00:00:00.000072 IP6 fe80::ec4:7aff:feac:821b > ff02::12: ip-proto-112 36
                                      00:00:00.964265 IP6 fe80::ec4:7aff:feab:3725 > ff02::12: ip-proto-112 36
                                      00:00:00.040245 IP6 fe80::ec4:7aff:feac:821b > ff02::12: ip-proto-112 36
                                      00:00:00.000067 IP 172.29.100.1 > 224.0.0.18: VRRPv2, Advertisement, vrid 2, prio 1, authtype none, intvl 1s, length 36

                                      Backup:
                                      00:00:01.004330 IP 172.29.100.1 > 224.0.0.18: VRRPv2, Advertisement, vrid 2, prio 1, authtype none, intvl 1s, length 36
                                      00:00:00.000053 IP6 fe80::ec4:7aff:feac:821b > ff02::12: ip-proto-112 36
                                      00:00:00.185346 IP6 fe80::ec4:7aff:feab:3725 > ff02::12: ip-proto-112 36
                                      00:00:00.819555 IP6 fe80::ec4:7aff:feac:821b > ff02::12: ip-proto-112 36
                                      00:00:00.000135 IP 172.29.100.1 > 224.0.0.18: VRRPv2, Advertisement, vrid 2, prio 1, authtype none, intvl 1s, length 36

                                      For some reason the "Backup" unit is also receiving it's own advertisements, but only on IPv6.

                                      Seems setting "advskew" to 100 on primary one, waiting for backup unit to get the config and rebooting the backup unit fixes the issue with the advertisements. Pending further testing of course.

                                      1 Reply Last reply Reply Quote 0
                                      • DerelictD Offline
                                        Derelict LAYER 8 Netgate
                                        last edited by

                                        You should be decoding those as CARP, not VRRP so we can see what is going on in a more clear fashion. You can:

                                        • Set the protocol to CARP then view the capture in Diagnostics > Packet Capture. That will result in tcpdump decoding as CARP.

                                        • Set wireshark to decode as CARP by right-clicking a VRRP packet and using Decode As to decode protocol 112 as CARP instead of VRRP.

                                        0_1530203621154_Screen Shot 2018-06-28 at 9.33.07 AM.png

                                        You should never have to touch advbase/advskew. They should be 1/0 on the primary which should sync to 1/100 on the secondary.

                                        I do recall one issue with IPv6 CARP and the way the VIPs are defined. I cannot remember if it was leading zeroes, capital hex digits or what. How are you specifying your CARP VIPs?

                                        For some reason the “Backup” unit is also receiving it’s own advertisements, but only on IPv6.

                                        If they both think they are MASTER on a VIP they will both be advertising. If you look at the MAC addresses in the capture, you will likely see that the secondary is not receiving its own advertisements, but that it is sending them along with the primary.

                                        Chattanooga, Tennessee, USA
                                        A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                        DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                        Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                        1 Reply Last reply Reply Quote 0
                                        • X Offline
                                          xciter327
                                          last edited by xciter327

                                          Yes, I agree. I saw a suggestion about that. Need to add "-T carp" to the tcpdump command for it work:

                                          tcpdump -npi igb1 -T carp -e | egrep "224.0.0.18|ff02::12:"

                                          for example. The "egrep" is there because if I just use "expression" carp, it does not dump the IPv6 traffic.

                                          Back to my problem. I've reverted to 1/0(and via config sync 1/100 on backup).

                                          How I am able to reproduce the problem:

                                          • Make a change on any CARP VIP
                                          • Reboot primary

                                          However if I reboot the backup unit after I've made a change on CARP, everything works as advertised. Going to do some more dumping to try to figure it out.

                                          1 Reply Last reply Reply Quote 0
                                          • DerelictD Offline
                                            Derelict LAYER 8 Netgate
                                            last edited by

                                            I don't know of any fixes regarding this, but if you are rebooting these units you should be on 2.4.3_1.

                                            https://www.netgate.com/docs/pfsense/highavailability/redundant-firewalls-upgrade-guide.html

                                            Chattanooga, Tennessee, USA
                                            A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                            DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                            Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.