No email notification on CARP failover



  • Is there any additional configuration required to get CARP to send email notifications when it fails over? We just discovered today that our two systems have been flip-flopping between each other every few hours. I started looking into a way to monitor CARP status when I came across a mention of /etc/rc.carpbackup and rc.carpmaster files. I looked at them, and from what I can tell they should be sending an email notification whenever the CARP status changes.

    I double checked the notifications settings, and I get the test email when I submit the form, so I presume notifications are working. Am I missing a configuration setting somewhere?


  • Rebel Alliance Developer Netgate

    That should be enough, though truth be told I can't remember the last time I tested the CARP aspect of the notification code.

    If the test notification works, others should too.



  • If I manually run 'php -f /etc/rc.carpbackup' I got a notification email, but I am -not- receiving any notifications when they actually failover. Is there any reason why the rc.carp* files would not be run?

    Any idea where I should start looking? What script is actually supposed to call these files?


  • Rebel Alliance Developer Netgate

    Show the output of "ifconfig -a" as well as the contents of your /etc/rc.carpbackup and /etc/devd.conf

    If the script doesn't fire at all, it may be because the devd.conf file is looking for events on the wrong interface name.

    If the script fires but doesn't e-mail, it may be that it can't read the config for some reason in that script.



  • Here's the ifconfig:

    
    em0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
            options=209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic>ether 68:05:ca:03:8c:8f
            inet 172.16.1.149 netmask 0xffffff00 broadcast 172.16.1.255
            inet6 fe80::6a05:caff:fe03:8c8f%em0 prefixlen 64 scopeid 0x1
            nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
            status: active
    em1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
            options=209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic>ether 68:05:ca:03:8d:6b
            inet6 fe80::6a05:caff:fe03:8d6b%em1 prefixlen 64 scopeid 0x2
            inet 192.168.1.76 netmask 0xfffffff0 broadcast 192.168.1.79
            nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex>)
            status: active
    re0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
            options=389b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_ucast,wol_mcast,wol_magic>ether 00:21:5a:1e:1b:b5
            inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
            inet6 fe80::221:5aff:fe1e:1bb5%re0 prefixlen 64 scopeid 0x3
            nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex>)
            status: active
    pflog0: flags=100 <promisc>metric 0 mtu 33664
    pfsync0: flags=41 <up,running>metric 0 mtu 1460
            pfsync: syncdev: re0 syncpeer: 224.0.0.240 maxupd: 128 syncok: 1
    enc0: flags=41 <up,running>metric 0 mtu 1536
    lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
            options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000
            inet6 ::1 prefixlen 128
            inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
            nd6 options=3 <performnud,accept_rtadv>vip5: flags=49 <up,loopback,running>metric 0 mtu 1500
            inet 172.16.1.254 netmask 0xffffff00
            carp: MASTER vhid 5 advbase 1 advskew 0
    ovpns1: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1500
            options=80000 <linkstate>inet6 fe80::6a05:caff:fe03:8c8f%ovpns1 prefixlen 64 scopeid 0xe
            inet 172.16.248.1 --> 172.16.248.2 netmask 0xffffffff
            nd6 options=3 <performnud,accept_rtadv>Opened by PID 55909
    ovpns2: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1500
            options=80000 <linkstate>inet6 fe80::6a05:caff:fe03:8c8f%ovpns2 prefixlen 64 scopeid 0xf
            inet 172.16.249.1 --> 172.16.249.2 netmask 0xffffffff
            nd6 options=3 <performnud,accept_rtadv>Opened by PID 58221
    vip1: flags=49 <up,loopback,running>metric 0 mtu 1500
            inet 192.168.1.66 netmask 0xfffffff0
            carp: MASTER vhid 1 advbase 1 advskew 0
    vip2: flags=49 <up,loopback,running>metric 0 mtu 1500
            inet 192.168.1.67 netmask 0xfffffff0
            carp: MASTER vhid 2 advbase 1 advskew 0
    vip3: flags=49 <up,loopback,running>metric 0 mtu 1500
            inet 192.168.1.68 netmask 0xfffffff0
            carp: MASTER vhid 3 advbase 1 advskew 0
    vip4: flags=49 <up,loopback,running>metric 0 mtu 1500
            inet 192.168.1.69 netmask 0xfffffff0
            carp: MASTER vhid 4 advbase 1 advskew 0
    vip6: flags=49 <up,loopback,running>metric 0 mtu 1500
            inet 192.168.1.71 netmask 0xfffffff0
            carp: MASTER vhid 6 advbase 1 advskew 0</up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></performnud,accept_rtadv></linkstate></up,pointopoint,running,multicast></performnud,accept_rtadv></linkstate></up,pointopoint,running,multicast></up,loopback,running></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></up,running></up,running></promisc></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_ucast,wol_mcast,wol_magic></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic></up,broadcast,running,promisc,simplex,multicast> 
    

    Here's the devd.conf:

    
    # $Id$
    # $FreeBSD: src/etc/devd.conf,v 1.26.2.1 2005/09/03 22:49:22 sam Exp $
    
    options {
            directory "/etc/devd";
            directory "/usr/local/etc/devd";
            pid-file "/var/run/devd.pid";
            set scsi-controller-regex
                    "(aac|adv|adw|aha|ahb|ahc|ahd|aic|amd|amr|asr|bt|ciss|ct|dpt|\
                    esp|ida|iir|ips|isp|mlx|mly|mpt|ncr|ncv|nsp|stg|sym|trm|wds)\
                    [0-9]+";
    };
    
    # CARP notify hooks. This will call carpup/carpdown with the
    # interface (carp0, carp1) as the first parameter.
    notify 100 {
        match "system"          "IFNET";
        match "type"            "LINK_UP";
        match "subsystem"           "carp";
        action "/etc/rc.carpmaster $subsystem";
    };
    
    notify 100 {
        match "system"          "IFNET";
        match "type"            "LINK_DOWN";
        match "subsystem"           "carp";
        action "/etc/rc.carpbackup $subsystem";
    };
    
    # When a USB keyboard arrives, attach it as the console keyboard.
    attach 100 {
            device-name "ukbd0";
            action "kbdcontrol -k /dev/ukbd0 < /dev/console 2>/dev/null";
    };
    
    detach 100 {
            device-name "ukbd0";
            action "kbdcontrol -k /dev/kbd0 < /dev/console 2>/dev/null";
    };
    
    #
    # Signal upper levels that an event happened on ethernet class interface
    #
    notify 0 {
            match "system"          "IFNET";
            match "type"            "LINK_UP";
            media-type              "ethernet";
            action "/usr/local/sbin/pfSctl -c 'interface linkup start $subsystem'";
    };
    
    notify 0 {
            match "system"          "IFNET";
            match "type"            "LINK_DOWN";
            media-type              "ethernet";
            action "/usr/local/sbin/pfSctl -c 'interface linkup stop $subsystem'";
    };
    
    # Notify all users before beginning emergency shutdown when we get
    # a _CRT or _HOT thermal event and we're going to power down the system
    # very soon.
    notify 10 {
            match "system"          "ACPI";
            match "subsystem"       "Thermal";
            match "notify"          "0xcc";
            action "logger -p kern.emerg 'WARNING: system temperature too high, shutting down soon!'";
    };
    
    

    Here's the rc.backup:

    
    #!/usr/local/bin/php -f
    /* $Id$ */
    /*
        rc.carpdown
        part of pfSense (http://www.pfSense.com)
        Copyright (C) 2004 Scott Ullrich
        All rights reserved.
    
        Redistribution and use in source and binary forms, with or without
        modification, are permitted provided that the following conditions are met:
    
        1\. Redistributions of source code must retain the above copyright notice,
           this list of conditions and the following disclaimer.
    
        2\. Redistributions in binary form must reproduce the above copyright
           notice, this list of conditions and the following disclaimer in the
           documentation and/or other materials provided with the distribution.
    
        THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
        INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
        AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
        AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY,
        OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
        SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
        INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
        CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
        ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
        POSSIBILITY OF SUCH DAMAGE.
    */
    
    require_once("functions.inc");
    require_once("config.inc");
    require_once("notices.inc");
    
    $notificationmsg = "A carp cluster member has resumed the state 'BACKUP'";
    
    notify_via_smtp($notificationmsg);
    notify_via_growl($notificationmsg);
    
    ?>
    
    

    Thanks for taking a look at this.


  • Rebel Alliance Developer Netgate

    Yeah that's something we fixed a while back. Edit devd.conf, change the "subsystem" from carp to vip. Then you can killall -9 devd; devd (or reboot).



  • That took care of it. Thanks!


Locked