Well, i'm one more with the same problem.
First of all, PFsense 2.4.2, both sides with Group Gateway Failover, DDNS on Remote Gateway.
So, i'm reading a lot of articles and, … i'll test a single change at IPSEC configuration. VPN > IPSEC > Advanced Configuration > Configure Unique IDs as NO.
Why ? https://blog.bravi.org/?p=1209
I don't know if i misunderstood, but, i'll try this shot …