Should work.

CARP is mainly broadcastingtraffic. Have a look at http://www.countersiege.com/doc/pfsync-carp/ to see how it works.

Correc t, you need first a virtual IP to add 1:1 mappings (at least if we are not talking abou the real wan interface IP). On top of that you need firewallrules to let the desired traffic pass of course.

Thanks a lot
Heiko

I'm not a fulltime programmer so I don't think like a geek.  I just made it work for my case.  If it's usefull, please feel free to use it.  I will be very happy if I know that it's intergrated into pfsense.  I can sleep with a happy face  :)

The three main files that I modified are: filter.inc, dyndns.class, and pfsense-utils.inc  (see attached files).  As always, diff is really useful to checkout the differences.  It's not much coding, here are all of my codes & modifications:

//
/ File: /etc/inc/filter.inc                                                                    /
/ At the end of the function filter_configure_sync add                            /
/ the new wan_monitor() function in                                                    /
//

/* reload filter sync /
function filter_configure_sync() {
  …
        / sync carp entries to other firewalls */
        update_filter_reload_status("Syncing CARP data");
        carp_sync_client();

/* WAN monitoring */
        wan_monitor();

update_filter_reload_status("Done");

return 0;
}

function wan_monitor() {
        global $config;
        $failure = 0;
        update_filter_reload_status("Debug: WAN Monitoring");

foreach ($config['filter']['rule'] as $rule) {
                if ($rule['gateway'] <> "") {
                        $rulegw = $rule['gateway'];
                }
        }

foreach($config['load_balancer']['lbpool'] as $lb) {
                $poolname = $lb['name'];
                $servers = $lb['servers'];

if ($poolname == $rulegw) {
                        if($lb['behaviour'] == "failover") {
                                $routeto = exec("cat /tmp/rules.debug | grep route-to | cut -d '(' -f2 | cut -d ')' -f1 | /usr/bin/sed -e 's/^ //g'");
                                list($int, $gateway) = split(" ", $routeto);

$default_route = exec("netstat -rn | grep default | awk '{print $2, $6}'");
                                list($default_gw, $default_int) = split(" ", $default_route);

if ($default_gw != $gateway && $gateway <> "" ) {

$wan1_gw = $config['interfaces']['wan']['gateway'];
                                        list($int_name, $monIP) = split("|", $servers[0]);

/* Double check by ping to the host monitor IP 3 times */
                                        for ($i = 1; $i <= 3; $i++) {
                                                $pingstatus = exec("/sbin/ping -c 1 -t 2 -q -Q $monIP | grep 'packet loss' | cut -d ',' -f3 | /usr/bin/sed -e 's/^ //g' | cut -d '%' -f1");

if ($pingstatus == 100) {
                                                        $failure++;
                                                }
                                                sleep (5);
                                        }

/* If total failure is 3 times, then switch the default route /
                                        if ($failure == 3) {
                                                $switchroute = 1;
                                        }
                                        / Switch back to WAN1 if the host monitor IP is pingable /
                                        / and the default gateway is on WAN2                    */
                                        else if ($failure == 0 && $default_gw != $wan1_gw) {
                                                $switchroute = 1;
                                        }
                                        else {
                                                $switchroute = 0;
                                        }

update_filter_reload_status("Debug: Switch Route = $switchroute");
                                        update_filter_reload_status("Debug: Total Failure time = $failure");
                                }
                                else {
                                        update_filter_reload_status("Debug: Do nothing. Same route");
                                }

if ($switchroute == 1) {
                                        update_filter_reload_status("Debug: Changing the default gateway to $gateway");
                                        exec("/sbin/route delete default");
                                        exec("/sbin/route add default $gateway");

update_filter_reload_status("Debug: Sending email notification");
                                        $hostname = exec("hostname");
                                        $subject = "$hostname has switched the default gateway to $gateway";
                                        $msg = "This is an automate email notification that the default gateway has switched over to ";
                                        $msg .= "$gateway\n";
                                        $to = "email@company.com";
                                        exec("/usr/local/bin/php /root/my-scripts/phpmailer/smtp.php "$subject" "$msg" $to");
                                }

$is_carp_enable = get_carp_status();

if ($is_carp_enable == 1) {
                                        foreach($config['virtualip']['vip'] as $carp) {
                                                if ($carp['mode'] != "carp") continue;
                                                $ipaddress = $carp['subnet'];
                                                $carp_int = find_carp_interface($ipaddress);
                                                $carp_status = get_carp_interface_status($carp_int);
                                                update_filter_reload_status("Debug: $carp_status");
                                        }
                                }

if ($carp_status == "MASTER") {
                                        update_filter_reload_status("Debug: Updating with DynDNS");
                                        exec("/etc/rc.dyndns.update");
                                }
                                update_filter_reload_status("Debug: End");
                        }
                }
        }
}

//
/ File: /etc/inc/dyndns.class                                                              /
/ Replace $wan_ip to use $this->_checkip() function to                          /
/ get the real WAN IP to use with Dyndns                                            /
//

if(!$wan_ip)
                                $wan_ip = $this->_checkip();

function _checkip() {

//log_error("DynDns: Running _checkip() for real WAN IP");

exec("/usr/bin/netstat -rn | grep carp | awk '{print $1, $6}'", $getcarp);

list($gwip, $gwint) = split(" ", /usr/bin/netstat -rn | grep default | awk '{print $2, $6}');
                        $gwip = str_replace("\n", "", $gwip);
                        $gwint = str_replace("\n", "", $gwint);

if ($getcarp[0] <> "") {
                                foreach ($getcarp as $carpinfo) {

list($carpip, $carpname) = split(" ", $carpinfo);

$carp_int = $this->_getcarp_int($carpip);

if ($gwint == "$carp_int") {
                                                $match = exec("echo $carpinfo | grep $carpname");
                                                list($gwvip, $gwcarp) = split(" ", $match);
                                                $ip = $gwvip;
                                        }
                                }
                        }
                        else {
                                $ch = curl_init();
                                curl_setopt($ch, CURLOPT_URL, 'http://checkip.dyndns.com');
                                curl_setopt($ch, CURLOPT_HEADER, 0);
                                curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);

$data = curl_exec($ch);
                                curl_close($ch);

list($part1, $part2) = split(': ', $data, 2);
                                list($ip, $junk) = split('<', $part2);
                        }
                        return $ip;
                }
                /* End of function */

function _getcarp_int($carpip) {

global $config;

foreach($config['virtualip']['vip'] as $vip) {

if ($vip['subnet'] == "$carpip") {
                                        $int_name = $vip['interface'];
                                        $int = convert_friendly_interface_to_real_interface_name($int_name);
                                }

}
                        return $int;
                }

/*
                * Private Function (added 12 July 05) [beta]
                *  - Detect whether or not IP needs to be updated.
                *      | Written Specifically for pfSense (pfsense.com) may
                *      | work with other systems. pfSense base is FreeBSD.
                */
                function _detectChange() {
                        global $config;
                        log_error("DynDns: _detectChange() starting.");

$currentTime = time();

$wan_ip = $this->_checkip();
                        $this->_dnsIP = $wan_ip;
                        $this->_dnsHost = $config['dyndns']['host'];

$previousIP = exec("/sbin/ping -c1 $this->_dnsHost | grep PING | cut -d '(' -f2 | cut -d ')' -f1");

log_error("DynDns: Previous DNS IP: {$previousIP}");
                        log_error("DynDns: Current WAN IP: {$wan_ip}");

if (file_exists($this->_cacheFile)) {
                                if(file_exists($this->_cacheFile))
                                        $contents = file_get_contents($this->_cacheFile);
                                else
                                        $contents = "";
                                list($cacheIP,$cacheTime) = split(':', $contents);

$this->_debug($cacheIP.'/'.$cacheTime);
                                $initial = false;
                                log_error("DynDns: Cached IP: {$cacheIP}");
                        } else {
                                conf_mount_rw();
                                $file = fopen($this->_cacheFile, 'w');
                                fwrite($file, '0.0.0.0:'.$currentTime);
                                fclose($file);
                                conf_mount_ro();
                                $cacheIP = '0.0.0.0';
                                $cacheTime = $currentTime;
                                $initial = true;
                                log_error("DynDns: No Cached IP found.");
                        }

/*  use 2419200 for dyndns, dhs, easydns, noip, hn
                        *  zoneedit, dyns, ods
                        */
                        $time = '2160000';

$needs_updating = FALSE;

/* lets deterimine if the item needs updating /
                        if ($previousIP != $wan_ip) {
                                $needs_updating = TRUE;
                                log_error("DynDns: previousIP != wan_ip.  Updating.");
                        }
                        if ($cacheIP != $wan_ip) {
                                $needs_updating = TRUE;
                                log_error("DynDns: cacheIP != wan_ip.  Updating.");
                        }
                        $update_reason = "Cached IP: {$cacheIP} WAN IP: {$wan_ip} ";
                        if (($currentTime - $cacheTime) > $time ) {
                                $needs_updating = TRUE;
                                log_error("DynDns: More than 25 days.  Updating.");
                        }
                        $update_reason .= "{$currentTime} - {$cacheTime} > {$time} ";
                        if ($initial == TRUE) {
                                $needs_updating = TRUE;
                                $update_reason .= "Inital update. ";
                                log_error("DynDns: Initial run.  Updating.");
                        }
                        /  finally if we need updating then store the
                        *  new cache value and return true
                        */
                        if($needs_updating == TRUE) {
                                return TRUE;
                        } else {
                                return FALSE;
                        }

log_error("DynDns debug information: {$update_reason}");

}

filter.inc.txt
dyndns.class.txt
pfsense-utils.inc.txt

Excellent. Works perfectly. Thanks hoba.

pfsync and carp shouldn't interfere with each other. Swapping Master/Backup status can only be related to CARP, not to pfSync.

http://doc.pfsense.org/index.php/Setting_up_CARP_with_pfSense

Works like a charm.  Thanks a bunch!

I have found out the root causes.  It's not the problem with the switch.  It was b/c of the firewall rules.  When the FW outbound LAN rule got change to use the Failover pool, the default route is no long effective.  When master & slave send out the broadcast message of VRRP to 244.0.0.18, it used the Failover for it routing table and Failover pool is only routed to either WAN1 or WAN2 which doesn't know the route of the internal LAN subnet.  That's why stage MASTER/MASTER were on both machines.  Once I create the new rule for LAN subnet to allow traffic to 244.0.0.18 using the default gateway, then it fixed the problem.

It's working great now.  Disable on master –> switch over to salve.  Enable back --> fallback to master.

Exactly

thanks for yours help

This was a bug.  Just fixed it.

Thank you

To answer the first of my two questions, I just came up with this:

http://kerneltrap.org/node/1021

Which basically means you can't bind scripts to it, just monitor link status from cron and act upon it (MASTER -> dial out)

Which brings back the second question.

What's the proper way of initiating PPPoE dialout from shell?

@jmhoms:

when you talk about adding firewall rules that utilize the pool as gateway, these rules must be added in the LAN interface, isnt it ?

That is right, you have to make the rule on the interface the traffic comes in from.

@jmhoms:

and the gateway IPs in the load balacer configuration must be the WAN and WAN2 CARP addresses, isnt it ?

No, you use their gateways (it's a gateway pool). If you use the latest snapshot you'll have these as pulldown options so there is no footshooting with this setting anymore.
Don't forget to set your firewall>nat, outbound to advanced outbound nat to utilize your CARP VIPs.

@jmhoms:

If so, i'm trying to make this setup working in a wmare test environment, no luck for now, i'll keep trying.

I have heard that CARP is not happy inside vmware. Haven't tried it myself though.

@jmhoms:

I supose it will work, so , do you think that if i have 2 offices with this configuration, will be possible to do IPSEC between the WANs CARP addresses ?

Yes, just have a look at the failover tab at vpn>ipsec. I have a setup running in this configuration.

whats the way around that ?

would it just simply be

if destination is in this range and from WAN forward out DMZ interface
and back out again
If destination is from DMZ servers range to the net forward out WAN interface ?

That way you wouldnt need NAT or Bridging ?

http://faq.pfsense.com/index.php?action=artikel&cat=1&id=167&artlang=en&highlight=arp

I'm trying the same that jpinder70, but with 2 adsl connections (and later will try to setup a redundant balaced ipsec meshed network).

It seems obvious that each pfsense system must have a wan ip of each of the adsl/t1 connections in order to have a carp address for each connection. I only have 1 public static ip per adsl, and will belong to carp interface, because the traffic must go out with this ip, cos is the only routed to my connection by my isp. That way, as the wan adresses must be in the same subnet as the carp address, i will take 2+2 ip that not really belong to me, and i assume that my natted networks never will get to the real ips (anyway these probably doent have any public service that must be directly accessed by my users).

Actually i only have 3 nic in each pfsense. So i'm trying some setups to see if they work without need of 4rt nic, hope to hear your feedback.

I connected both adsl routers, and both wan of pfsenses to the same ethernet segment.
My pfsense1 sync to pfsense2. I tried also to activate that pfsense2 sync to pfsense1. It seems to work, but there is some delay when apply changes, maybe there is some kind of cyclic action :? i don't know if it's ok that setup.
Actually my  wan of pfsense1 have the adsl1 public ip, and wan of pfsense2 have the adsl2 public ip. I setup a carp address for adsl1 subnet in pfsense1, and a carp address for adsl2 suvnet in pfsense2. I was expecting for an error in sync, because pfsense1 doesnt know about adsl2 subnet, and pfsense2 neither of adsl1 net. Pfsense system have sync and now i have the carp adresses in both pfsenses. Maybe is not necessary that both pfsesne to be in both wan subnets ?¿¿ i think that yes it's mandatory, because don't seem to work (no error in frontend anyway).

Assuming that both subnets are mandatory, i would like to know if it's possible to setup a wan interface with the two wan ips (1 per each adsl conn).  Maybe with proxy arp virtual ip ?? i don't see any aliasing option to assign multiple ip to an interface in the frontend (like in rc.conf _alias method in freebsd). I read somewhere that is not recommended, anyone have any hint with this ? maybe this will be an issue in the way the traffic wil go out ??¿ maybe the balacer will not work properly ?

i keep monitoring this thread to see if the jpinder70 setup works.

Thanks.

Demonstration of what it does:
http://pfsense.com/mirror.php?section=tutorials/carp/carp_failoversim.htm

How to set it up:
http://pfsense.com/mirror.php?section=tutorials/carp/carp-cluster-new.htm
http://doc.pfsense.org/index.php/Setting_up_CARP_with_pfSense

Technical Info:
http://www.openbsd.org/faq/pf/carp.html
http://www.countersiege.com/doc/pfsync-carp/