Multi-wan with vpn to same remote endpoint?

[grumling]

Hi,

So this is my setup that im having problems with.

Site A:
Static WAN X.X.X.X

Site B:
WAN1: Y.Y.Y.Y
WAN2: Z.Z.Z.Z

Site A and B is connected via WAN(Site A) and WAN1(Site B)

The problem is now when I have set up the WAN2 as a failover. When WAN1 goes down I want WAN2 to connect to WAN via VPN. The multi-wan part is up and running thanks to Routing groups.

But I cant add a second Phase1 on the pfsense, when I do, routing breaks on the "first" phase1. Seems logical in a way, they both points to the same remote networks. But uses different interfaces on  pfsense.
But how should I set this up so when WAN1 is up, then it uses that for vpn, but when WAN1 fails, the vpn should use WAN2.

Sorry for spelling and grammar. Been up for 22h and im about to get some sleep.

Hope someone have some idea and the time to give some tip on how I can get this to work.

Thinking about using something like dyndns and use the dns as identifier and point SiteA to connect to the DNSname instead of WAN1 ip. But im not sure if that is viable.