Update: after turning the whole infrastructure from left to right we found the solution. It's the limiter bug that is already known. After removing the limiter from the firewall rule (it was a just one catch all rule for the whole NAT traffic), it works as before. Which also means: the same setting worked perfectly fine before the upgrade. I am some much hoping for a soon fix of the limiters in an official update or release!

@m4rek11 After applying the patches, I did not notice that the routers changed the roles of Master-> Backup, Backup-> Master. All the problems went with those when I made any changes to the rules, dns or DHCP. I found my configuration error early. For unknown reason, for 2 different networks I sent the same vhid for Virtual IP. But the problems were still there. After applying the patches, the problem was gone.

@digger30 Perfect! Glad I could be of assistance.

The maintenance mode switch is in the config and persists across reboots.

@chrullrich I replaced the pfSense 2.6 "local router/firewall"s in my test setup with OPNsense 22.1 (this is FreeBSD 13.0 instead of pfSense 2.6's 12.3) to get a second opinion. The behavior is the same: As soon as the CARP failover happens, everything sent towards the "Internet" goes out the default route with the NATed source address appropriate for the policy route. When I tried it the first time today I thought I saw ping (and only ping) work correctly, but now I cannot reproduce it. I probably just saw what I wanted to see.

@derelict Yeah, same conclusion i had. @viragomann Yup.

@viragomann Issue resolved with hostname override and haproxy listnening on LAN interface Thx

@skid9000 Perhaps some screenshots of the setup? Can you get it working without the VLANs and add those in after? I've not had occasion to set HA up with VLANs but have done so with aliases for other subnets on LAN.

Just for your info. We've now seen the issue on multiple installations (even different hardware and pfsense versions) and could solve it on every single system by moving the sync-vlan to a dedicated physical interface.

@viragomann i think it's that https://forum.netgate.com/topic/150505/xmlrpc-restore_config_section-error because my rule to NAT with CARP ip make the backup node not able to reach the gateway so as it explain on that like you sent Filter reload sees the down gateway and resets states, terminating the connection currently used for XMLRPC. it make sense Thanks you very much, i think you resolve my issue :)

@kayavila OK this is great info, thank you! I read your entire write up you linked to as well but I'm still trying to wrap my brain around it. Think I've got it figured out but wanted to pose an example. This particular one will be between different VLAN/subnets rather than with WAN as I personally don't ever allow those connections via the WAN. So in theory if you had VLAN1 and VLAN2 setup, and there was an any-any rule below a block "This Firewall" rule on VLAN1, and some device on VLAN1 tried to contact the LAN interface of VLAN2, due to state syncing this would be let through? Since the first node would see the connection to the VLAN2 IP and see that it's not in it's block list but matches the any-any rule, and then the state would sync to the secondary which wouldn't assess it's rules? If that is the case, I would imagine not having a rule on the primary node that allows access to any would solve the issue, but since some people do use an any rule for internet access it could pose a problem (though best practice is of course to use an alias for RFC1918 and explicitly allow the inverse of that).

@viragomann Thanks ! Went with the port forward + outbound option, NAT is working finally.

@starsandbars What questions do you have after reading this? https://docs.netgate.com/pfsense/en/latest/highavailability/index.html