Looks like NAT and reauthentication is giving this issue in a certain case. The clients will start to get double virtual ip's if the NAT device expires/reboots/crashes. If I disable reauthentication on both sides it solves the issue.
I still can't explain why this works but for me it looks like it could be a bug in strongswan. It's 100 percent reproduceable with the follow setup
RW(client) -> Pfsense(nat) -> Pfsense(endpoint)
Rebooting the NAT will give double virtual ip's to the RW where one of the ip given doesn't work