Lost connectivity after exiting CARP maintenance

aborsic

Hi All,

I am inexperienced user of pfSense. By reading the online instructions I was able to setup a HA cluster configuration with two Netgate 7100 routers for my small business.

Recently I have updated them from pfSense 23.05.1 to 23.09.1 following the instructions for upgrading a HA cluster at https://docs.netgate.com/pfsense/en/latest/solutions/reference/highavailability/upgrading.html

I followed the instructions, and precisely I have:

upgraded the secondary
looked at the logs on the secondary and all was looking OK
on the primary set CARP status to maintenance
verified that the secondary kicked in and all was OK
upgraded the primary
looked at the logs on the primary and all was looking OK
on the primary exited the CARP maintenance mode

as soon as I hit the button "exit permanent maintenance" under CARP status, all connectivity on the cluster was lost and I had to ask someone local to power cycle the two routers, which fixed the problem.

Does anyone know what might have been the cause of this, or have any suggestion for investigating the problem? I would like to be able to reliably update the cluster in the future ...

Thanks for any kind suggestion,

Best Regards,

Andrea

SteveITS

@aborsic That's not normal behavior. Had you previously tried maintenance mode/failover in regular usage, outside of upgrading? You should be able to enter/leave that at will. Do the logs show anything useful at the time of exiting maintenance mode?

aborsic

@SteveITS

Hi Steve,

Thanks for your comments. This was the first time I was using the CARP maintenance mode. I did not find anything obvious (to me) in the logs at the time of switching out from maintenance mode. I will try entering / leaving the maintenance mode again - I am currently physically far from the site, and I will try that as soon as I can be at the site in case power on/off is required again to regain connectivity.

Thanks,

Andrea

SteveITS

@aborsic It is hard to say without more details but it could be something like:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/high-availability.html#both-nodes-appear-as-master
https://docs.netgate.com/pfsense/en/latest/troubleshooting/high-availability.html#both-nodes-in-maintenance-mode

If it's working correctly, you can normally just upgrade/reboot/maintenance mode the routers without anyone noticing.

aborsic

@SteveITS

Thank you, I will re-examine the logs and see if for any reason it appears I was in one of the two cases. I will test as well again entering and exiting the maintenance mode.

Andrea