Enable CARP maintenance when state changes for one interface

mahomed

TL;DR What's the safest way to enable CARP maintenance mode when one interface changes from master to backup/init? By safe, I mean without editing pfSense core files that would get overwritten by, or block upgrades.

Long Version

System:		Netgate XG-7100 1U  (2 units)
Version:	2.4.4-RELEASE-p3 (amd64)
		built on Thu May 16 06:01:19 EDT 2019
		FreeBSD 11.2-RELEASE-p10

Unfortunately due a design "feature" in the XG-7100, it's not possible to failover all the interfaces when one of the switch ports fails despite setting net.inet.carp.preempt: 1. Unfortunately I discovered this gem of info a bit too late (we already bought two units and can't send them back)

The problem this causes is that we have both WAN and LAN ports on the switch, if either interface fails, only that interface switches to the other router and causes a split-brain scenario. E.g.

OK State
--------
			Router 1		Router 2
WAN (SW-Port-01)	Master			Backup
LAN (SW-Port-02)	Master			Backup


WAN Port failed
---------------
			Router 1		Router 2
WAN (SW-Port-01)	Backup			Master
LAN (SW-Port-02)	Master			Backup

Rather than buy the add-on card (because then the switch becomes useless to us), we would like to simply put the primary into maintenance mode when one of the interfaces fails or goes into backup/init mode.

I have searched the forum and worked out that I can edit /etc/devd.conf [1, 2] and run /usr/local/sbin/pfSsh.php playback enablecarpmaint [3] to enable maintenance mode when a specific port fails. This works great after I edit /etc/devd.conf and restart devd. The issue is that when I reboot, my changes no longer work. After running ps auxf | grep devd I found that pfSense has started devd with the file /etc/pfSense-devd.conf. So I have tried to put my notify conditions in my own file in /etc/devd but the actions are not being called. I figure that this is because the CARP and IFNET conditions that I am trying to match are already being met in /etc/pfSense-devd.conf [4]

So I added my own action line in /etc/pfSense-devd.conf and this works BUT this is obviously a file that is "part of pfSense" (it says so in the header :-) ) so my concern is about whether my changes in this file will get lost by a system upgrade or even block an upgrade.

Is there a better way of achieving all of this (that I haven't found) and that will survive an upgrade? Any help would be greatly appreciated. Thanks in advance.

Sources:
[1] - https://forum.netgate.com/topic/49078/how-to-make-dhcp-wans-trigger-carp-failover
[2] - https://forum.netgate.com/topic/41756/solved-still-problems-with-pfsense-carp-trigger
[3] - https://forum.netgate.com/topic/141326/carp-master-slave-manage-master-recover/2
[4] - https://forum.netgate.com/post/318954

JeGr

@mahomed said in Enable CARP maintenance when state changes for one interface:

So I added my own action line in /etc/pfSense-devd.conf and this works BUT this is obviously a file that is "part of pfSense" (it says so in the header :-) ) so my concern is about whether my changes in this file will get lost by a system upgrade or even block an upgrade.

If that is the main concern, have a look at the Filer Plugin. It can manage new files or existing ones and is HA compatbile. So you can add a file, paste the content it should have and the filer package will make sure, that content is correct in that file. As the package is HA compatible, that will also be synced to the standby node, too. And as a package, that saves its configuration in config.xml it will get re-installed after a pfSense upgrade and the new file content will get "restored" after the package is reinstalled and the config restored. That should take care of the "modified file content" problem I think.

Greets

mahomed

Thanks @JeGr . I've now installed Filer and I can definitely see the use in it for restoring/syncing my script files. I can see that I can also probably use it for /etc/pfSense-devd.conf. But that brings the next problem of what happens when the Netgate team updates this file? The "latest" and correct version would get overwritten by my file in Filer. Out of curiosity I've checked the file on GitHub and it was indeed updated 2 months ago and those changes are in the file on my routers. So that means it will definitely change with an upcoming upgrade.

Is there no other/better way to force the maintenance mode or execute the devd actions without modifying a system file?