Enable CARP maintenance when state changes for one interface
TL;DR What's the safest way to enable CARP maintenance mode when one interface changes from master to backup/init? By safe, I mean without editing pfSense core files that would get overwritten by, or block upgrades.
System: Netgate XG-7100 1U (2 units) Version: 2.4.4-RELEASE-p3 (amd64) built on Thu May 16 06:01:19 EDT 2019 FreeBSD 11.2-RELEASE-p10
Unfortunately due a design "feature" in the XG-7100, it's not possible to failover all the interfaces when one of the switch ports fails despite setting
net.inet.carp.preempt: 1. Unfortunately I discovered this gem of info a bit too late (we already bought two units and can't send them back)
The problem this causes is that we have both WAN and LAN ports on the switch, if either interface fails, only that interface switches to the other router and causes a split-brain scenario. E.g.
OK State -------- Router 1 Router 2 WAN (SW-Port-01) Master Backup LAN (SW-Port-02) Master Backup WAN Port failed --------------- Router 1 Router 2 WAN (SW-Port-01) Backup Master LAN (SW-Port-02) Master Backup
Rather than buy the add-on card (because then the switch becomes useless to us), we would like to simply put the primary into maintenance mode when one of the interfaces fails or goes into backup/init mode.
I have searched the forum and worked out that I can edit
/etc/devd.conf[1, 2] and run
/usr/local/sbin/pfSsh.php playback enablecarpmaint to enable maintenance mode when a specific port fails. This works great after I edit
devd. The issue is that when I reboot, my changes no longer work. After running
ps auxf | grep devdI found that pfSense has started
devdwith the file
/etc/pfSense-devd.conf. So I have tried to put my notify conditions in my own file in
/etc/devdbut the actions are not being called. I figure that this is because the
IFNETconditions that I am trying to match are already being met in
So I added my own action line in
/etc/pfSense-devd.confand this works BUT this is obviously a file that is "part of pfSense" (it says so in the header :-) ) so my concern is about whether my changes in this file will get lost by a system upgrade or even block an upgrade.
Is there a better way of achieving all of this (that I haven't found) and that will survive an upgrade? Any help would be greatly appreciated. Thanks in advance.
 - https://forum.netgate.com/topic/49078/how-to-make-dhcp-wans-trigger-carp-failover
 - https://forum.netgate.com/topic/41756/solved-still-problems-with-pfsense-carp-trigger
 - https://forum.netgate.com/topic/141326/carp-master-slave-manage-master-recover/2
 - https://forum.netgate.com/post/318954
So I added my own action line in /etc/pfSense-devd.conf and this works BUT this is obviously a file that is "part of pfSense" (it says so in the header :-) ) so my concern is about whether my changes in this file will get lost by a system upgrade or even block an upgrade.
If that is the main concern, have a look at the Filer Plugin. It can manage new files or existing ones and is HA compatbile. So you can add a file, paste the content it should have and the filer package will make sure, that content is correct in that file. As the package is HA compatible, that will also be synced to the standby node, too. And as a package, that saves its configuration in config.xml it will get re-installed after a pfSense upgrade and the new file content will get "restored" after the package is reinstalled and the config restored. That should take care of the "modified file content" problem I think.
Thanks @JeGr . I've now installed Filer and I can definitely see the use in it for restoring/syncing my script files. I can see that I can also probably use it for
/etc/pfSense-devd.conf. But that brings the next problem of what happens when the Netgate team updates this file? The "latest" and correct version would get overwritten by my file in Filer. Out of curiosity I've checked the file on GitHub and it was indeed updated 2 months ago and those changes are in the file on my routers. So that means it will definitely change with an upcoming upgrade.
Is there no other/better way to force the maintenance mode or execute the devd actions without modifying a system file?