It can depend on the switch/router on the other end of the cable. For instance with Comcast routers often when replacing a router in an office (inside the Comcast router) I've found it's fastest to power off or reboot the Comcast router so it learns the IP has a new MAC. If you have the second router on, and are just plugging in cables, I would wonder if restarting the second router (or just leaving it off and powering it on) would help.
But overall CARP set up properly works basically instantly so that would be preferred. https://docs.netgate.com/pfsense/en/latest/book/highavailability/index.html