CARP issues due to Layer 2 switching

maverickws

Hi all,

I am having an issue with CARP using Hetzner's vSwitch to connect servers across different DC's.
As CARP wasn't working out of the box, talked to them and they replied this:

Real Multicast isn't possible inside the vSwitch. All packets will be handled like Broadcast Traffic.
You can try to use use gratuitous ARP on both ends end to ensure that the MACs are correct updated. It takes a short time the switches to learn the MACs entries.

I made a follow up question and they simply insisted:

For this we suggest to you that to use GARP on both ends end to ensure that the MACs are correct updated.

So I looked at this topic: https://forum.netgate.com/topic/60600/gratuitous-arp-from-virtual-ips

And I put the first script on both pfsenses:

#!/usr/bin/env php -f
<?php
/*
 * Required Packages:
 * - arping
 *
 * v0.0.2 - Romeo Benzoni
 * - adopted to run on 2.3 (get_carp_interface_status changes signature)
 * - send arp response using the virtual MAC
 * v0.0.1 :: rc.viparp - created by Shahid Sheikh
 * - generates unsolicited arp response to a gateway.
*/
/* $Id$ */

/* parse the configuration and include all functions used below */
require_once("/etc/inc/config.inc");
require_once("/etc/inc/gwlb.inc");
require_once("/etc/inc/interfaces.inc");
require_once("/etc/inc/pkg-utils.inc");

$bad_arp_gw = trim($argv[1], " \n");
$package_name = "arping";
if (is_package_installed($package_name)) {
        $bad_arp_gw_int = lookup_gateway_interface_by_name($bad_arp_gw);
        $bad_arp_gw_ip = lookup_gateway_ip_by_name($bad_arp_gw);
        if (!empty($bad_arp_gw_int)) {
                foreach ($config['virtualip']['vip'] as $vip) {
                        if ($vip['mode'] == "carp" && $vip['interface'] == $bad_arp_gw_int) {
                                $vip_int = "{$vip['interface']}_vip{$vip['vhid']}";
                                $status = get_carp_interface_status("_vip{$vip['uniqid']}");
                                if ($status == "MASTER") {
                                        $mac = sprintf("00:00:5E:00:01:%02X", $vip['vhid']);
                                        log_error("Sending ARP for CARP VIP {$vip['subnet']}'s MAC $mac to {$bad_arp_gw_ip}");
                                        mwexec("/usr/local/sbin/arping -s $mac -S {$vip['subnet']} -c 3 {$bad_arp_gw_ip}");
                                }
                        }
                }
        }
} else {
        log_error($package_name . " package is not installed.");
}
?>

Put it to run every three minutes on both routers. But putting the primary router down didn't solve it. So I added changed to the second script also there:

#!/usr/bin/env php -f
<?php
require_once("/etc/inc/config.inc");
require_once("/etc/inc/gwlb.inc");
require_once("/etc/inc/interfaces.inc");
require_once("/etc/inc/pkg-utils.inc");

$gatewayName = trim($argv[1], " \n");
$packageName = "arping";

if (is_package_installed($packageName)) {
	$gatewayInterface 	= lookup_gateway_interface_by_name($gatewayName);
    $gatewayIp 			= lookup_gateway_ip_by_name($gatewayName);

	if (!empty($gatewayName)) {
		$found = false;
		foreach ($config['virtualip']['vip'] as $vip) {
			if ($vip['interface'] == $gatewayInterface) {
				log_error('Sending ARP for Virtual IP ' . $vip['subnet'] . ' to ' . $gatewayIp);
				mwexec('/usr/local/sbin/arping -S ' . $vip['subnet'] . ' -c 3 ' . $gatewayIp);

				$found = true;
			}
		}

		if (!$found) log_error('Found no Virtual IP tied to this gateway: '  . $gatewayName . '\n');
	} else  {
		log_error('You forgot to supply the Gateway name.');
	}
} else {
		log_error($packageName . " package is not installed.");
}
?>

And also ran it every three minutes. Also solved nothing, putting the primary down the secondary automatically changes the role to secondary but traffic doesn't flow.
I see on the secondary node all the mac addresses for the VIP's. The ARP table hadn't them there before (but neither on the first anyway they're there too after running the script), but traffic doesn't flow.

How can I implement the proposed solution? thank you.

Also, isn't there a way to make a hook to only trigger this when the role changes?

maverickws

Hi guys, I was just looking at: https://docs.netgate.com/pfsense/en/latest/releases/2-2-new-features-and-changes.html

2.2 New Features and Changes

...

Packages

...
Added the capability for package hooks in /etc/rc.carpmaster and /etc/rc.carpbackup
...

How does this work? Can this be used to call a hook when the state changes? Sorry guys I'm not a developer, suck at it. If anyone could direct me at a solution I'd be really appreciated.

apollo13

@maverickws said in CARP issues due to Layer 2 switching:

How does this work? Can this be used to call a hook when the state changes? Sorry guys I'm not a developer, suck at it. If anyone could direct me at a solution I'd be really appreciated.

This allows other packages (those that can be installed via System -> PackageManager) to register "hooks" that get called when the carp states change. I don't know how easily you could register your custom hooks there.

apollo13

I found a few more details about how this works: For instance haproxy registers for the carp hook via: https://github.com/pfsense/FreeBSD-ports/blob/308fec64d2b57fc9b3fb8c9cf0f7b350cb534e20/net/pfSense-pkg-haproxy/files/usr/local/pkg/haproxy.xml#L52 and pfsense then loops over the pkg information, sees that haproxy wants to be called and then dispatches to https://github.com/pfsense/FreeBSD-ports/blob/308fec64d2b57fc9b3fb8c9cf0f7b350cb534e20/net/pfSense-pkg-haproxy/files/usr/local/pkg/haproxy/haproxy.inc#L2258 -- in this hook haproxy can do whatever it needs when the carp state changes.

maverickws

Hi @apollo13 thank you for your answer!!
I'm not really a developer so code isn't really my thing, but I try to manage.

So, from what we gather so far, it seems like it would be possible to make a package that would respond to CARP event and execute the script performing GARP's so that the switches learn the new correct MAC address where to send traffic to, correct?

I was also looking at here: https://docs.netgate.com/pfsense/en/latest/development/developing-packages.html

Wondering where would I place those options? The <plugins> would be on the config_file.xml I assume? And where would I put the function to run upon state change?
Also, this is automatic, I mean, if I was to install such package/plugin so to call it, it would be automatically triggered upon state change?

Thanks!

apollo13

@maverickws said in CARP issues due to Layer 2 switching:

So, from what we gather so far, it seems like it would be possible to make a package that would respond to CARP event and execute the script performing GARP's so that the switches learn the new correct MAC address where to send traffic to, correct?

Yes.

Wondering where would I place those options? The <plugins> would be on the config_file.xml I assume?

Yes, see haproxy for an example: https://github.com/pfsense/FreeBSD-ports/blob/308fec64d2b57fc9b3fb8c9cf0f7b350cb534e20/net/pfSense-pkg-haproxy/files/usr/local/pkg/haproxy.xml

And where would I put the function to run upon state change?

It has to be in the file referenced by include_file: https://github.com/pfsense/FreeBSD-ports/blob/308fec64d2b57fc9b3fb8c9cf0f7b350cb534e20/net/pfSense-pkg-haproxy/files/usr/local/pkg/haproxy.xml#L31 And you have to name it <your_package_name>_plugin_carp taking one parameter $pluginparams

Also, this is automatic, I mean, if I was to install such package/plugin so to call it, it would be automatically triggered upon state change?

Yes, you can write a GUI for your package which would let you to disable those hooks etc… but I think generally they will be called always.

maverickws

Again I stress... not really a programmer, this kind of is a bit over my head. I will try to look into it with some spare time, but in the meanwhile I wonder if no one would also benefit from this and if there isn't someone who maybe could pick this up and wrap it quickly.