This seems to be the same issue as https://redmine.pfsense.org/issues/13569 -- I'd love to debug this further but I am not sure what else to look into.

contacted support, reinstalled from scratch, same errors again. provided diagnosis data, support reproduced issue, now we have this issue in redmine: bug

it happened to me several times, don't change the port, delete the backend and redo it

@jimp These two pfSenses are in the middle of network, the issue didn't affect interfaces faced to syslog server, syslog source set as local pfSense interface, not as CARP VIP. We see in syslog other messages like FW rules actions during the issue period, but not CARP-related ones.

@damianhl If it has ZFS there is a Disks widget that can expand to show details: [image: 1685732278018-e80dceed-465d-4da2-9b03-30e91c0a4dcd-image.png] Not sure about hardware RAID, have never used it. Unless FreeBSD/pfSense includes a driver the pfSense OS will probably only be able to see what the BIOS shows it.

DHCPD LOG.txt

@decibel83 A problem at Layer 2 is the most common cause.

FYI- You can disable NAT and route without also disabling the firewall. Firewall > NAT, Outbound tab, set it to Disable Outbound NAT and save/apply.

@james92 Yes a dumb switch is fine.

@davidredekop Interesting. I have never had to change anything in proxmox for CARP. As an aside, while fc00::/7 is the ULA network space, fc00::/8 is currently undefined. fd00::/8 is proper ULA addressing. Recommend implementing RFC 4193 and randomly selecting a /48 for ULA usage.

i was able to track down a bit of a solution we had disabled hardware offloads , this is now turned back on which make xmlrpc sync much quicker and lower load and cpu. also we have two wans, on each wan we had two openvpn servers listening for different purposes, 7-8 years ago we were told that its best to listen on localhost with each vpn server, then nat port forward each external port so that each wan can listen on the same server, it appears if we do this now, each time an xmlproc sync occurs it causes pfctl and the reload scripts to thrash and loop 3 or more times. we this this occuring over and over with localhost php-fpm[6973]: /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed its IP. Reloading endpoints that may use VPN the solution now is listening on a single carp ip, this means were not able to openvpn in the backup wan, but atleast vpn works on master and backup servers, just not the backup wan all xmlrpc sync is re-enabled and no CARP timeouts so far.......

@nocternal Yup, I'll be doing just that, super nice we can do "micro" patches like this. Thanks again!!

Just to add for anyone else coming across this issue. Adding a vlan and therefore triggering a configuration reload and mini failover, caused exactly the same issue. Which was not fixable with restoring a configuration backup or even a restart of both firewalls. Applying this patch: Fix automatic firewall rules for HA DHCP server failover (Requires reboot or filter reload to activate, Redmine #13965) Fixed the issue with the DHCP server. The issue showed in Status / DHCP Leases a permanent status of My State - 'Recover', as well as previously mentioned 'communication-interrrupted'

@derelict At the moment, I only have one firewall. I will add the other one later when I have more WAN addresses.

@tony-soprano Any switch should support CARP. The protocol simply sends out mulitcasts to talk to the other node. So both has only to be within the same L2 network.

@viragomann I have read the Netgate documentation regarding Single Address CARP. This brief paragraph states there are significant challenges with updating, etc. I tried to look around for more information on this type of setup. What info I did encounter seemed to be people who were spoofing the WAN MAC Address on the secondary unit and using scripts to determine the interface state(ifup/ifdown) in an attempt to avoid collision. I would be hesitant to put any real trust in hacks of this nature in a corporate production environment. Edit: PPPoE is not being used.

@jakub_ Yes. The advertisements are sourced from the interface IP address and CARP MAC. Not sure why you are seen advertisements from both the primary (advskew 0) and secondary (advskew 100) there.