Ok, little follow-up.
It looks like it's not possible (at all) to actually specify which address is advertised in a RA. I assume the host directly uses the source address of the RA
To speed up failover, one can hack the pfSense code which generates the actual radvd config and set MinRtrAdvInterval 3, MaxRtrAdvInterval 5 and, for the ::/0 route add AdvRouteLifetime 5. This reduces failover time to around 10 seconds - but I'm not sure if it is generally a good idea to mess with these values.
The fastest option (in terms of failover) seems to be actually not to use SLAAC in the first place, but to manually configure IPv6 on each host - to be able to specify the desired CARP IP as default gateway…
Does this make sense?
PS: Oh, just found this read: https://www.isc.org/blogs/routing-configuration-over-dhcpv6-2/