Pfsync IPsec Failover Issues

[xRadeon]

We are testing an IPsec site to site solution using pfSense. We have deployed two pfSense VMs in VMware and have setup pfsync and XMLRPC Sync between the two nodes (We enabled all the settings we need to on the port groups). We also configured two CARP VIPs, one for inside and one for outside. We've tested the failover while pinging through the firewall and CARP performs perfectily. We configured a IPsec VPN using the outside CARP address and we can bring up the tunnel to our remote S2S node.

The issue is with IPsec and how long it takes to failover when rebooting/master fails/etc. When we failover the firewall to the secondary node it takes up to 3 minutes to re-establish the IPsec tunnel and about 8 minutes to failback once the master is up (during both times the remote client is unreachable for the entire duration). My question is, with pfsync enabled, shouldn't the CARP failover happen seamlessly? The IPsec states should be synced between the two nodes so it should just take a second for CARP to flip the VIP over the secondary node so IPsec shouldn't even go down at all. Is there any special configuration settings we need to make to ensure an IPsec seamless failover can happen? If so where is it located at? If this is expected behavior is there any way to reduce the time it takes to failover?

See attached images for HA, CARP, and IPsec (P1, P2 and adv) settings.

Thank you for your time.