CARP preempt does't work



  • Could you please confirm me that the preempt functionality still works on version 2.4.4-RELEASE-p2?
    I've configured two firewalls as master and slave with 2 VIP on both WAN and LAN and a SYNC interface to connect the two. If only one interface fails, CARP will switch only that interface to the slave and not both.
    I the device fails, both interface goes to the slave and switch back as soon as the master is online again.
    I've tried to reconfigure everything twice and also tried with two VMs but with the same result.
    Thank you for your input.

    net.inet.carp.ifdown_demotion_factor: 240
    net.inet.carp.senderr_demotion_factor: 0
    net.inet.carp.demotion: 0
    net.inet.carp.log: 1
    net.inet.carp.preempt: 1
    net.inet.carp.allow: 1

    mvneta0: flags=8a43<UP,BROADCAST,RUNNING,ALLMULTI,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=800bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
    ether 00:08:a2:0e:10:ef
    hwaddr 00:08:a2:0e:10:ef
    inet6 fe80::208:a2ff:fe0e:10ef%mvneta0 prefixlen 64 scopeid 0x1
    inet 192.168.2.2 netmask 0xfffffffc broadcast 192.168.2.3
    media: Ethernet autoselect (1000baseT <full-duplex,master>)
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    mvneta1: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM>
    ether 00:08:a2:0e:10:f0
    hwaddr 00:08:a2:0e:10:f0
    inet 10.0.0.3 netmask 0xffffff00 broadcast 10.0.0.255
    inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255 vhid 2
    inet6 fe80::1:1%mvneta1 prefixlen 64 scopeid 0x2
    carp: BACKUP vhid 2 advbase 1 advskew 100
    media: Ethernet 2500Base-KX <full-duplex>
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    mvneta2: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=800bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
    ether 00:08:a2:0e:10:f1
    hwaddr 00:08:a2:0e:10:f1
    inet6 fe80::208:a2ff:fe0e:10f1%mvneta2 prefixlen 64 scopeid 0x8
    inet 192.168.1.2 netmask 0xfffffffc broadcast 192.168.1.3
    inet 10.0.5.82 netmask 0xffffff00 broadcast 10.0.5.255 vhid 1
    carp: BACKUP vhid 1 advbase 1 advskew 100
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
    enc0: flags=0<> metric 0 mtu 1536
    groups: enc
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0xa
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    pflog0: flags=100<PROMISC> metric 0 mtu 33184
    groups: pflog
    pfsync0: flags=0<> metric 0 mtu 1500
    syncpeer: 224.0.0.240 maxupd: 128 defer: on
    syncok: 1
    groups: pfsync


  • LAYER 8 Netgate

    What hardware is that?



  • Netgear SG-3100.
    I don't think it is an hardware problem, I get the exact same results with two virtual machines.


  • LAYER 8 Netgate

    How are you taking an interface physically down on a VM?



  • On the SG-3100 I've tried unplugging the cables or taking the interface down.
    On the VM I've taken the interface down.
    Always the same result, master fails to slave correctly but only for the failed interface and not both (LAN and WAN).
    It doesn't matter which interface I let fail (LAN or WAN), the other never switch together (WAN or respectively LAN)


  • LAYER 8 Netgate

    What is being logged by the node on which you are unplugging the device?

    It will be in the General system log.

    No, preemption is not broken so we need to figure out why yours is not working. Note that pfSense needs to see actual carrier down. I expect this might be problematic using the SG-3100 LAN switch ports and VM interfaces.

    The logs you are looking for will look similar to this:

    Mar 12 15:53:40 pfSense check_reload_status: Linkup starting igb2
    Mar 12 15:53:40 pfSense kernel: igb2: link state changed to DOWN
    Mar 12 15:53:41 pfSense check_reload_status: Reloading filter
    Mar 12 15:54:10 pfSense kernel: igb3: link state changed to DOWN
    Mar 12 15:54:10 pfSense kernel: carp: 13@lagg0: MASTER -> INIT (hardware interface down)
    Mar 12 15:54:10 pfSense kernel: carp: demoted by 240 to 240 (interface down)
    Mar 12 15:54:10 pfSense kernel: lagg0: link state changed to DOWN
    Mar 12 15:54:10 pfSense kernel: carp: 14@lagg1: MASTER -> BACKUP (more frequent advertisement received)
    

    Note the CARP interface going down demotes the node to advskew 240 which makes the other CARP VIP "slower" than the secondary at 100 so it assumes BACKUP state too. (That is an LACP lagg so it doesn't actually go DOWN until igb2 and igb3 lose link.)


  • Rebel Alliance Developer Netgate



  • Hier is the log for the Master when unplugging the cable

    Mar 12 17:08:12 syslogd kernel boot file is /boot/kernel/kernel
    Mar 12 17:08:17 firewall1.localdomain nginx: 2019/03/12 17:08:17 [error] 87042#100091: send() failed (54: Connection reset by peer)
    Mar 12 17:08:27 check_reload_status Carp backup event
    Mar 12 17:08:27 kernel carp: 1@mvneta2: MASTER -> INIT (hardware interface down)
    Mar 12 17:08:27 kernel carp: demoted by 240 to 240 (interface down)
    Mar 12 17:08:27 kernel mvneta2: link state changed to DOWN
    Mar 12 17:08:27 kernel carp: 2@mvneta1: MASTER -> BACKUP (more frequent advertisement received)
    Mar 12 17:08:27 kernel ifa_maintain_loopback_route: deletion failed for interface mvneta1: 3
    Mar 12 17:08:27 check_reload_status Linkup starting mvneta2
    Mar 12 17:08:27 check_reload_status Carp backup event
    Mar 12 17:08:28 php-fpm 248 /rc.linkup: DEVD Ethernet detached event for wan
    Mar 12 17:08:28 php-fpm 42160 /rc.carpbackup: HA cluster member "(10.0.5.82@mvneta2): (WAN)" has resumed CARP state "BACKUP" for vhid 1
    Mar 12 17:08:28 kernel ifa_maintain_loopback_route: deletion failed for interface mvneta2: 3
    Mar 12 17:08:28 kernel ifa_maintain_loopback_route: deletion failed for interface mvneta2: 3
    Mar 12 17:08:28 kernel carp: demoted by -240 to 0 (vhid removed)
    Mar 12 17:08:28 kernel mvneta2: promiscuous mode disabled
    Mar 12 17:08:28 php-fpm 247 /rc.carpbackup: HA cluster member "(10.0.0.1@mvneta1): (LAN)" has resumed CARP state "BACKUP" for vhid 2
    Mar 12 17:08:28 check_reload_status Carp master event
    Mar 12 17:08:28 kernel carp: 2@mvneta1: BACKUP -> MASTER (preempting a slower master)
    Mar 12 17:08:28 kernel arp: 10.0.0.1 moved from 00:00:5e:00:01:02 to 00:08:a2:0e:10:f0 on mvneta1
    Mar 12 17:08:29 php-cgi notify_monitor.php: Could not send the message to xxxxxx-- Error: Failed to connect to xxxxxxx:25 [SMTP: Failed to connect socket: php_network_getaddresses: getaddrinfo failed: hostname nor servname provided, or not known (code: -1, response: )]
    Mar 12 17:08:29 php-fpm 32006 /rc.carpmaster: HA cluster member "(10.0.0.1@mvneta1): (LAN)" has resumed CARP state "MASTER" for vhid 2
    Mar 12 17:08:30 php-fpm 248 /rc.linkup: Shutting down Router Advertisment daemon cleanly
    Mar 12 17:08:30 check_reload_status Reloading filter

    Hier the log for the slave

    Mar 12 17:07:55 syslogd kernel boot file is /boot/kernel/kernel
    Mar 12 17:08:25 kernel carp: 2@mvneta1: BACKUP -> MASTER (preempting a slower master)
    Mar 12 17:08:25 check_reload_status Carp master event
    Mar 12 17:08:26 php-fpm 248 /rc.carpmaster: HA cluster member "(10.0.0.1@mvneta1): (LAN)" has resumed CARP state "MASTER" for vhid 2
    Mar 12 17:08:26 check_reload_status Carp backup event
    Mar 12 17:08:26 kernel carp: 2@mvneta1: MASTER -> BACKUP (more frequent advertisement received)
    Mar 12 17:08:26 kernel ifa_maintain_loopback_route: deletion failed for interface mvneta1: 3
    Mar 12 17:08:27 check_reload_status Carp master event
    Mar 12 17:08:27 kernel carp: 1@mvneta2: BACKUP -> MASTER (master timed out)
    Mar 12 17:08:27 php-fpm 91794 /rc.carpbackup: HA cluster member "(10.0.0.1@mvneta1): (LAN)" has resumed CARP state "BACKUP" for vhid 2
    Mar 12 17:08:28 php-fpm 52174 /rc.carpmaster: HA cluster member "(10.0.5.82@mvneta2): (WAN)" has resumed CARP state "MASTER" for vhid 1
    Mar 12 17:08:29 php-cgi notify_monitor.php: Message sent to xxxxxx OK


  • LAYER 8 Netgate

    You'll have to see if the CARP advertisements are actually arriving on the secondary in that case.

    Mar 12 17:08:28 kernel arp: 10.0.0.1 moved from 00:00:5e:00:01:02 to 00:08:a2:0e:10:f0 on mvneta1

    That should never happen. There should never be anything responding to ARP on a CARP VIP with an IS AT anything but the CARP MAC. What is at MAC Address 00:08:a2:0e:10:f0



  • WAP VIP
    10.0.5.82 at 0:0:5e:0:1:1
    WAN Master
    10.0.5.83 at 0:8:a2:0e:10:4f
    WAN Slave
    10.0.5.84 at 0:8:a2:e:10:f1
    LAN VIP
    10.0.0.1 at 0:0:5e:0:1:2
    LAN Master
    10.0.0.2 at 0:8:a2:e:10:4e
    LAN Slave
    10.0.0.3 at 0:8:a2:e:10:f0


  • LAYER 8 Netgate

    Take a really good look at the configuration of the secondary.

    Packet capture a bunch of ARP during failover tests and take a good look at that. There should never be an ARP IS AT response for a CARP VIP that contains an interface MAC as the payload. It will be sourced from the interface MAC but contain the CARP Mac. If there is I would like to see the capture.



  • Operation: unplug the WAN cable on the Master

    Packet capture on LAN on slave
    17:34:48.781554 ARP, Request who-has 10.0.0.1 tell 10.0.0.1, length 28
    17:34:48.781952 ARP, Reply 10.0.0.1 is-at 00:08:a2:0e:10:4e, length 46
    17:34:50.173825 ARP, Request who-has 10.0.0.1 tell 10.0.0.1, length 46
    17:34:50.173860 ARP, Reply 10.0.0.1 is-at 00:08:a2:0e:10:f0, length 28

    Packet capture on LAN on master
    17:34:48.783981 ARP, Request who-has 10.0.0.1 tell 10.0.0.1, length 46
    17:34:48.784012 ARP, Reply 10.0.0.1 is-at 00:08:a2:0e:10:4e, length 28
    17:34:50.175740 ARP, Request who-has 10.0.0.1 tell 10.0.0.1, length 28
    17:34:50.176185 ARP, Reply 10.0.0.1 is-at 00:08:a2:0e:10:f0, length 46

    Slave before failover
    mvneta1: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM>
    ether 00:08:a2:0e:10:f0
    hwaddr 00:08:a2:0e:10:f0
    inet 10.0.0.3 netmask 0xffffff00 broadcast 10.0.0.255
    inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255 vhid 2
    inet6 fe80::1:1%mvneta1 prefixlen 64 scopeid 0x2
    carp: BACKUP vhid 2 advbase 1 advskew 100
    media: Ethernet 2500Base-KX <full-duplex>
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    mvneta2: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=800bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
    ether 00:08:a2:0e:10:f1
    hwaddr 00:08:a2:0e:10:f1
    inet6 fe80::208:a2ff:fe0e:10f1%mvneta2 prefixlen 64 scopeid 0x8
    inet 10.0.5.84 netmask 0xffffff00 broadcast 10.0.5.255
    inet 10.0.5.82 netmask 0xffffff00 broadcast 10.0.5.255 vhid 1
    carp: BACKUP vhid 1 advbase 1 advskew 100
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

    Master before failover
    mvneta1: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM>
    ether 00:08:a2:0e:10:4e
    hwaddr 00:08:a2:0e:10:4e
    inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255
    inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255 vhid 2
    inet6 fe80::1:1%mvneta1 prefixlen 64 scopeid 0x2
    carp: MASTER vhid 2 advbase 1 advskew 0
    media: Ethernet 2500Base-KX <full-duplex>
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    mvneta2: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=800bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
    ether 00:08:a2:0e:10:4f
    hwaddr 00:08:a2:0e:10:4f
    inet6 fe80::208:a2ff:fe0e:104f%mvneta2 prefixlen 64 scopeid 0x8
    inet 10.0.5.83 netmask 0xffffff00 broadcast 10.0.5.255
    inet 10.0.5.82 netmask 0xffffff00 broadcast 10.0.5.255 vhid 1
    carp: MASTER vhid 1 advbase 1 advskew 0
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

    slave after failover
    mvneta1: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM>
    ether 00:08:a2:0e:10:f0
    hwaddr 00:08:a2:0e:10:f0
    inet 10.0.0.3 netmask 0xffffff00 broadcast 10.0.0.255
    inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255 vhid 2
    inet6 fe80::1:1%mvneta1 prefixlen 64 scopeid 0x2
    carp: BACKUP vhid 2 advbase 1 advskew 100
    media: Ethernet 2500Base-KX <full-duplex>
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    mvneta2: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=800bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
    ether 00:08:a2:0e:10:f1
    hwaddr 00:08:a2:0e:10:f1
    inet6 fe80::208:a2ff:fe0e:10f1%mvneta2 prefixlen 64 scopeid 0x8
    inet 10.0.5.84 netmask 0xffffff00 broadcast 10.0.5.255
    inet 10.0.5.82 netmask 0xffffff00 broadcast 10.0.5.255 vhid 1
    carp: MASTER vhid 1 advbase 1 advskew 100
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>

    master after failover
    mvneta1: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM>
    ether 00:08:a2:0e:10:4e
    hwaddr 00:08:a2:0e:10:4e
    inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255
    inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255 vhid 2
    inet6 fe80::1:1%mvneta1 prefixlen 64 scopeid 0x2
    carp: MASTER vhid 2 advbase 1 advskew 0
    media: Ethernet 2500Base-KX <full-duplex>
    status: active
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
    mvneta2: flags=8a43<UP,BROADCAST,RUNNING,ALLMULTI,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=800bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
    ether 00:08:a2:0e:10:4f
    hwaddr 00:08:a2:0e:10:4f
    inet6 fe80::208:a2ff:fe0e:104f%mvneta2 prefixlen 64 scopeid 0x8
    media: Ethernet autoselect (none)
    status: no carrier
    nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>


  • LAYER 8 Netgate

    @mylos said in CARP preempt does't work:

    17:34:48.781554 ARP, Request who-has 10.0.0.1 tell 10.0.0.1, length 28

    That also makes no sense. Does it make sense to you?

    Please download and post the raw pcap.



  • Nope does not make any sense for me as well

    d4c3 b2a1 0200 0400 0000 0000 0000 0000
    0000 0400 0100 0000 1998 8a5c 2f42 0a00
    2a00 0000 2a00 0000 ffff ffff ffff 0008
    a20e 10f0 0806 0001 0800 0604 0001 0000
    5e00 0102 0a00 0001 0000 0000 0000 0a00
    0001 1998 8a5c ad43 0a00 3c00 0000 3c00
    0000 0000 5e00 0102 0008 a20e 104e 0806
    0001 0800 0604 0002 0008 a20e 104e 0a00
    0001 0000 5e00 0102 0a00 0001 0000 0000
    0000 0000 0000 0000 0000 0000 0000 1b98
    8a5c 06fc 0000 3c00 0000 3c00 0000 ffff
    ffff ffff 0008 a20e 104e 0806 0001 0800
    0604 0001 0000 5e00 0102 0a00 0001 0000
    0000 0000 0a00 0001 0000 0000 0000 0000
    0000 0000 0000 0000 0000 1b98 8a5c 26fc
    0000 2a00 0000 2a00 0000 0000 5e00 0102
    0008 a20e 10f0 0806 0001 0800 0604 0002
    0008 a20e 10f0 0a00 0001 0000 5e00 0102
    0a00 0001


  • LAYER 8 Netgate

    You will need to attach the capture file.

    I cannot properly test SG-3100 HA. I only have one unit.



  • The above is the capture file. Save as hexadecimal as dump.pcap


  • LAYER 8 Netgate

    Yeah, no thanks. attach the pcap please.




Log in to reply