On CARP switchover to secondary, *some* replicated states disappear
-
First, this is pfSense CE 2.8.1, two VMs on KVM and conventional Linux bridge devices. vNICs are all virtio. Best I know, none of the latter (platform) has any bearing on this. Plenty of RAM on both VMs, avg. usage no more than about 60% and avg. around 1k entries in the state tables.
TL;DR: state and XMLRPC replication works great... then some, but not all - states disappear when the secondary takes over. When the primary takes over again, if not rebooted, the missing states are re-replicated and any hung connections resume. No apparent suspicious entries in the system log.
What works: in normal CARP pair operation, states are replicated - by inspection of the state table - about same number of by watching pfinfo. Spot checks also match, the VM configs are identical, including the interface names and enumeration order. When I do either a manual carp maintenance mode, or e.g. reboot the primary, the VIPs all migrate virtually instantly and as they should. If I'm 1/sec pinging a host routed through the firewall, it doesn't miss a ping on the takeover/giveback - good. There are no obvious errors in the system log during this process.
What doesn't work: even though the states are apparently replicated, just as the secondary takes over, it quietly drops 30-50% of its states by count. Not all, just some. If I'm logged in via TCP to that same host I'm pinging via firewall routing (requiring a state), that connection hangs until the primary takes over again. Manually inspecting the state tables for that host's IP on the backup/temp primary, all states for the host are among those that disappeared - so no surprise the connection hung. When I re-enable CARP and primary takes over again, state for that connection is re-replicated and data flows again. BTW this behavior occurs whether the VMs are on the same host or different hosts (latter the usual case for HW HA), so it's not a switch problem per se - seemingly already ruled that out. The loss of states appears to be occurring inside the secondary, because replication itself via the layers below, is otherwise working.
I've dug deep on the net for any clues, including running through the full CARP troubleshooting guide, including using unicast for state replication instead of the default directed multicast. (Both behave the same.) Still, something is causing loss of not all, but a portion of the states that were replicated and appeared in the secondary's state table. I don't see any clear pattern except that new(er) states seem to be the ones most likely lost.
I do use some policy routing on a couple internal interfaces, and tried disabling that with no behavior change. The topology here is pretty simple for the HA pair: a single WAN-facing interface towards my cable modem with outbound NAT enabled plus a handful of inbound port forwards, and several internal interfaces e.g. LAN, DMZ. One interface is dedicated to SYNC, has a wide-open firewall rule, and nothing about the interface seems to be a problem. All interfaces except SYNC host a CARP VIP. I am not using trunking inside pfSense but have brX devices on the host, one per VLAN defined via netplan. Once upon a time this worked seamlessly. I'm not aware of any config changes that would cause this.