Failover: WAN Master / LAN Backup on Secondary



  • I've configured CARP using the pfSense Book instructions as a guide. If I pull the WAN cable from the Primary unit to test failover (as suggested in the pfSense book as a test), the Secondary unit WAN VIP will become Master, but the Secondary LAN VIP will stay Backup. My expectation was that the Secondary LAN VIP would also become Master so that traffic would flow through the Secondary unit. I can get the Secondary LAN VIP to become Master if I pull the LAN cable as well from the Primary unit or cause a full shutdown or disable of CARP on Primary unit.

    Is the behavior that I am observing expected? I'd ideally like to see a single port failure on the Primary cause the Secondary to gain Master state on both the WAN and LAN VIPs so that traffic flow will not be interrupted.

    The basic configuration on an SG-2440 running 2.3.2-RELEASE-p1 is as follows:

    Public IP/28 (CARP VIP, VHID 73)
                                                                    |
                                                            WAN switch
                |–-------------------------------------|---------------------------------------|
    192.168.40.1/28                                                                              192.168.40.2/28
            WAN                                                                                                WAN

    Primary    OPT1(192.168.50.1/24) --------- (192.168.50.2/24)OPT1    Secondary

    LAN                                                                                                  LAN
    172.27.40.2/24                                                                                  172.27.40.3/24
                |---------------------------------------|---------------------------------------|
                                                            LAN switch           
                                                                    |
                                          172.27.40.1/24 (CARP VIP, VHID 1)

    Any guidance on this is appreciated.


  • Netgate

    What is in the system log when you do this test?

    All CARP VIPs should get their advskew raised by 240 by default for every interface with a CARP VIP that goes into a down state. That should be enough to make all VIPs on the secondary become master in a default 1/0 1/100 advbase/advskew configuration.

    Nov 16 04:21:20 kernel carp: 237@xn2: MASTER -> INIT (hardware interface down)
    Nov 16 04:21:20 kernel carp: demoted by 240 to 240 (interface down)
    Nov 16 04:21:20 kernel xn2: link state changed to DOWN
    Nov 16 04:21:20 kernel carp: 228@xn1: MASTER -> BACKUP (more frequent advertisement received)
    Nov 16 04:21:20 kernel carp: 236@xn0: MASTER -> BACKUP (more frequent advertisement received)



  • Here's what I see in the logs. In this case, I pulled the LAN cable out of the Primary. On the WAN VIP, I can see that at least for a moment, the Secondary did become Master, but then the Primary promptly became the Master.

    Primary Log:

    Nov 16 22:53:53 check_reload_status Carp backup event
    Nov 16 22:53:53 kernel carp: demoted by 240 to 240 (interface down)
    Nov 16 22:53:53 kernel igb1: link state changed to DOWN
    Nov 16 22:53:53 kernel carp: VHID 73@igb0: MASTER -> BACKUP (more frequent advertisement received)
    Nov 16 22:53:53 check_reload_status Linkup starting igb1
    Nov 16 22:53:53 check_reload_status Carp backup event
    Nov 16 22:53:54 php-fpm 97843 /rc.carpbackup: HA cluster member "(172.27.40.1@igb1): (LAN)" has resumed CARP state "BACKUP" for vhid 1
    Nov 16 22:53:54 php-fpm 97843 /rc.linkup: DEVD Ethernet detached event for lan
    Nov 16 22:53:54 check_reload_status Reloading filter
    Nov 16 22:53:54 kernel carp: demoted by -240 to 0 (vhid removed)
    Nov 16 22:53:54 kernel igb1: promiscuous mode disabled
    Nov 16 22:53:54 php-fpm 97843 /rc.carpbackup: HA cluster member "(x.x.x.x@igb0): (WAN)" has resumed CARP state "BACKUP" for vhid 73
    Nov 16 22:53:55 check_reload_status Carp master event
    Nov 16 22:53:55 kernel carp: VHID 73@igb0: BACKUP -> MASTER (preempting a slower master)
    Nov 16 22:53:55 xinetd 17605 Starting reconfiguration
    Nov 16 22:53:55 xinetd 17605 Swapping defaults
    Nov 16 22:53:55 xinetd 17605 readjusting service 6969-udp
    Nov 16 22:53:55 xinetd 17605 Reconfigured: new=0 old=1 dropped=0 (services)
    Nov 16 22:53:56 php-fpm 97843 /rc.carpmaster: HA cluster member "(x.x.x.x@igb0): (WAN)" has resumed CARP state "MASTER" for vhid 73

    Secondary Log:

    Nov 16 22:53:52 check_reload_status Carp master event
    Nov 16 22:53:52 kernel carp: VHID 73@igb0: BACKUP -> MASTER (preempting a slower master)
    Nov 16 22:53:53 php-fpm 33089 /rc.carpmaster: HA cluster member "(x.x.x.x@igb0): (WAN)" has resumed CARP state "MASTER" for vhid 73
    Nov 16 22:53:54 check_reload_status Carp backup event
    Nov 16 22:53:54 kernel carp: VHID 73@igb0: MASTER -> BACKUP (more frequent advertisement received)
    Nov 16 22:53:55 check_reload_status Carp master event
    Nov 16 22:53:55 kernel carp: VHID 1@igb1: BACKUP -> MASTER (master down)
    Nov 16 22:53:55 php-fpm 33089 /rc.carpbackup: HA cluster member "(x.x.x.x@igb0): (WAN)" has resumed CARP state "BACKUP" for vhid 73
    Nov 16 22:53:56 php-fpm 33089 /rc.carpmaster: HA cluster member "(172.27.40.1@igb1): (LAN)" has resumed CARP state "MASTER" for vhid 1


  • Netgate

    Nov 16 22:53:53    kernel      carp: demoted by 240 to 240 (interface down)
    Nov 16 22:53:54    kernel      carp: demoted by -240 to 0 (vhid removed)

    Hmm. demotes then apparently the vhid is removed from the interface so no more interface with a CARP VIP down so no more demotion. I think I can dig up a couple 2440s. Have to test it physical.

    If you feel like it take a backup config.xml from the primary (without rrd) and paste it into a PM so I can test your config more accurately.


  • Netgate

    Are you trying to cheat with private addresses on the interfaces and one public CARP VIP?

    That is not how it is done in the book, and is not a supported configuration, even though some appear to have gotten it to basically work.

    Please do it right with public IP addresses on the interfaces out of the same WAN /28 and see how it goes.



  • Yes, I am using private address space for the WAN interfaces with a public VIP. I'm not certain whether I can obtain additional public IPs, but will see what I can do. That's too bad that this isn't a supported scenario as I imagine obtaining multiple public IPs may present additional complications for some situations.


  • Netgate

    Found it.

    Set your LAN IPv6 to None. It's set to Track Interface.

    Seems like the logic there could be better but CARP/HA is not compatible with dynamic addressing on any CARP interface so setting to none there fixes what you're seeing.



  • I believe this may have been the solution. I seem to be able to pull the WAN or LAN cables from Primary now and get a full failover.