No email notification on CARP failover

nullifi · Aug 20, 2012, 7:08 PM

Is there any additional configuration required to get CARP to send email notifications when it fails over? We just discovered today that our two systems have been flip-flopping between each other every few hours. I started looking into a way to monitor CARP status when I came across a mention of /etc/rc.carpbackup and rc.carpmaster files. I looked at them, and from what I can tell they should be sending an email notification whenever the CARP status changes.

I double checked the notifications settings, and I get the test email when I submit the form, so I presume notifications are working. Am I missing a configuration setting somewhere?

jimp · Aug 20, 2012, 8:08 PM

That should be enough, though truth be told I can't remember the last time I tested the CARP aspect of the notification code.

If the test notification works, others should too.

nullifi · Aug 22, 2012, 7:50 PM

If I manually run 'php -f /etc/rc.carpbackup' I got a notification email, but I am -not- receiving any notifications when they actually failover. Is there any reason why the rc.carp* files would not be run?

Any idea where I should start looking? What script is actually supposed to call these files?

jimp · Aug 22, 2012, 7:58 PM

Show the output of "ifconfig -a" as well as the contents of your /etc/rc.carpbackup and /etc/devd.conf

If the script doesn't fire at all, it may be because the devd.conf file is looking for events on the wrong interface name.

If the script fires but doesn't e-mail, it may be that it can't read the config for some reason in that script.

nullifi · Aug 22, 2012, 8:53 PM

Here's the ifconfig:


em0: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
        options=209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic>ether 68:05:ca:03:8c:8f
        inet 172.16.1.149 netmask 0xffffff00 broadcast 172.16.1.255
        inet6 fe80::6a05:caff:fe03:8c8f%em0 prefixlen 64 scopeid 0x1
        nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
em1: flags=8943 <up,broadcast,running,promisc,simplex,multicast>metric 0 mtu 1500
        options=209b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic>ether 68:05:ca:03:8d:6b
        inet6 fe80::6a05:caff:fe03:8d6b%em1 prefixlen 64 scopeid 0x2
        inet 192.168.1.76 netmask 0xfffffff0 broadcast 192.168.1.79
        nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
re0: flags=8843 <up,broadcast,running,simplex,multicast>metric 0 mtu 1500
        options=389b <rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_ucast,wol_mcast,wol_magic>ether 00:21:5a:1e:1b:b5
        inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
        inet6 fe80::221:5aff:fe1e:1bb5%re0 prefixlen 64 scopeid 0x3
        nd6 options=3 <performnud,accept_rtadv>media: Ethernet autoselect (100baseTX <full-duplex>)
        status: active
pflog0: flags=100 <promisc>metric 0 mtu 33664
pfsync0: flags=41 <up,running>metric 0 mtu 1460
        pfsync: syncdev: re0 syncpeer: 224.0.0.240 maxupd: 128 syncok: 1
enc0: flags=41 <up,running>metric 0 mtu 1536
lo0: flags=8049 <up,loopback,running,multicast>metric 0 mtu 16384
        options=3 <rxcsum,txcsum>inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
        nd6 options=3 <performnud,accept_rtadv>vip5: flags=49 <up,loopback,running>metric 0 mtu 1500
        inet 172.16.1.254 netmask 0xffffff00
        carp: MASTER vhid 5 advbase 1 advskew 0
ovpns1: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1500
        options=80000 <linkstate>inet6 fe80::6a05:caff:fe03:8c8f%ovpns1 prefixlen 64 scopeid 0xe
        inet 172.16.248.1 --> 172.16.248.2 netmask 0xffffffff
        nd6 options=3 <performnud,accept_rtadv>Opened by PID 55909
ovpns2: flags=8051 <up,pointopoint,running,multicast>metric 0 mtu 1500
        options=80000 <linkstate>inet6 fe80::6a05:caff:fe03:8c8f%ovpns2 prefixlen 64 scopeid 0xf
        inet 172.16.249.1 --> 172.16.249.2 netmask 0xffffffff
        nd6 options=3 <performnud,accept_rtadv>Opened by PID 58221
vip1: flags=49 <up,loopback,running>metric 0 mtu 1500
        inet 192.168.1.66 netmask 0xfffffff0
        carp: MASTER vhid 1 advbase 1 advskew 0
vip2: flags=49 <up,loopback,running>metric 0 mtu 1500
        inet 192.168.1.67 netmask 0xfffffff0
        carp: MASTER vhid 2 advbase 1 advskew 0
vip3: flags=49 <up,loopback,running>metric 0 mtu 1500
        inet 192.168.1.68 netmask 0xfffffff0
        carp: MASTER vhid 3 advbase 1 advskew 0
vip4: flags=49 <up,loopback,running>metric 0 mtu 1500
        inet 192.168.1.69 netmask 0xfffffff0
        carp: MASTER vhid 4 advbase 1 advskew 0
vip6: flags=49 <up,loopback,running>metric 0 mtu 1500
        inet 192.168.1.71 netmask 0xfffffff0
        carp: MASTER vhid 6 advbase 1 advskew 0</up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></up,loopback,running></performnud,accept_rtadv></linkstate></up,pointopoint,running,multicast></performnud,accept_rtadv></linkstate></up,pointopoint,running,multicast></up,loopback,running></performnud,accept_rtadv></rxcsum,txcsum></up,loopback,running,multicast></up,running></up,running></promisc></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_ucast,wol_mcast,wol_magic></up,broadcast,running,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic></up,broadcast,running,promisc,simplex,multicast></full-duplex></performnud,accept_rtadv></rxcsum,txcsum,vlan_mtu,vlan_hwtagging,vlan_hwcsum,wol_magic></up,broadcast,running,promisc,simplex,multicast>

Here's the devd.conf:


# $Id$
# $FreeBSD: src/etc/devd.conf,v 1.26.2.1 2005/09/03 22:49:22 sam Exp $

options {
        directory "/etc/devd";
        directory "/usr/local/etc/devd";
        pid-file "/var/run/devd.pid";
        set scsi-controller-regex
                "(aac|adv|adw|aha|ahb|ahc|ahd|aic|amd|amr|asr|bt|ciss|ct|dpt|\
                esp|ida|iir|ips|isp|mlx|mly|mpt|ncr|ncv|nsp|stg|sym|trm|wds)\
                [0-9]+";
};

# CARP notify hooks. This will call carpup/carpdown with the
# interface (carp0, carp1) as the first parameter.
notify 100 {
    match "system"          "IFNET";
    match "type"            "LINK_UP";
    match "subsystem"           "carp";
    action "/etc/rc.carpmaster $subsystem";
};

notify 100 {
    match "system"          "IFNET";
    match "type"            "LINK_DOWN";
    match "subsystem"           "carp";
    action "/etc/rc.carpbackup $subsystem";
};

# When a USB keyboard arrives, attach it as the console keyboard.
attach 100 {
        device-name "ukbd0";
        action "kbdcontrol -k /dev/ukbd0 < /dev/console 2>/dev/null";
};

detach 100 {
        device-name "ukbd0";
        action "kbdcontrol -k /dev/kbd0 < /dev/console 2>/dev/null";
};

#
# Signal upper levels that an event happened on ethernet class interface
#
notify 0 {
        match "system"          "IFNET";
        match "type"            "LINK_UP";
        media-type              "ethernet";
        action "/usr/local/sbin/pfSctl -c 'interface linkup start $subsystem'";
};

notify 0 {
        match "system"          "IFNET";
        match "type"            "LINK_DOWN";
        media-type              "ethernet";
        action "/usr/local/sbin/pfSctl -c 'interface linkup stop $subsystem'";
};

# Notify all users before beginning emergency shutdown when we get
# a _CRT or _HOT thermal event and we're going to power down the system
# very soon.
notify 10 {
        match "system"          "ACPI";
        match "subsystem"       "Thermal";
        match "notify"          "0xcc";
        action "logger -p kern.emerg 'WARNING: system temperature too high, shutting down soon!'";
};

Here's the rc.backup:


#!/usr/local/bin/php -f
/* $Id$ */
/*
    rc.carpdown
    part of pfSense (http://www.pfSense.com)
    Copyright (C) 2004 Scott Ullrich
    All rights reserved.

    Redistribution and use in source and binary forms, with or without
    modification, are permitted provided that the following conditions are met:

    1\. Redistributions of source code must retain the above copyright notice,
       this list of conditions and the following disclaimer.

    2\. Redistributions in binary form must reproduce the above copyright
       notice, this list of conditions and the following disclaimer in the
       documentation and/or other materials provided with the distribution.

    THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
    INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
    AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
    AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY,
    OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
    SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
    INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
    CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
    ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
    POSSIBILITY OF SUCH DAMAGE.
*/

require_once("functions.inc");
require_once("config.inc");
require_once("notices.inc");

$notificationmsg = "A carp cluster member has resumed the state 'BACKUP'";

notify_via_smtp($notificationmsg);
notify_via_growl($notificationmsg);

?>

Thanks for taking a look at this.

jimp · Aug 22, 2012, 9:03 PM

Yeah that's something we fixed a while back. Edit devd.conf, change the "subsystem" from carp to vip. Then you can killall -9 devd; devd (or reboot).

nullifi · Aug 23, 2012, 3:02 PM

That took care of it. Thanks!