@bmeeks:
@adam65535:
Changing Suricata config to live reload the rules stopped carp from failing over. It does seem like Suricata was causing the issue. I thought I didn't enable live reloading because of issues a few years ago but that was quite a few versions ago so maybe that isn't an issue anymore. There is a note that if live reloading causes problems that you should disable live reloading. Hopefully things keep going smoothly.
Thanks for the help.
Thanks for the follow-up. Using Live Reload should be OK. It is relatively mature now in Suricata.
I still have no good explanation for why Suricata restarting woud cycle the network connection. As I said earlier, the only thing it is doing with Legacy Mode blocking is starting up libpcap to get packet copies of traffic traversing the interface. Maybe that causes something to hiccup in FreeBSD someplace and CARP sees the hiccup because maybe it disrupts traffic very briefly. Strange issue.
Bill
Success! Looks like enabling "Live Swap" fixed the issue for me too. Only got past 1 "expected CARP failover event" thus far, but appears to be good.
Thanks for the suggestion. All I did to fix it on my side is filled in the checkbox "Enable "Live Swap" reload of rules after downloading an update" on my pfsense routers and so far so good. Typically the routers appeared to fail back and forth a lot as the general system logs showed >5000 logs of CARP failover. Gladly CARP works very well, so actual impact was approx 2-5 lost pings, slight freeze on RDP sessions, but SSH sessions would continue to work as expected.
Because I only just set this rule I have only gotten past one potential failure (update every 12 hours starting at 00:30. 00:30 did have failure, but at 11:40 I enabled the "Live Swap reload" in Suricata, and 12:30 typical CARP failover did NOT happen). In other words, I typically have two failures, one at 00:30 midnight, and a second at 12:30 noon. After changing this setting I have not had any failures.
Crossing my fingers this was the solution :)
PS. if helpful, my versions are:
pfSense: 2.4.2-RELEASE
Suricata: 4.0.3_1