Irregular times phase 2 not passing traffic

[sabern-terry]

We recently began experiencing problems with a phase 2 connection not receiving packages from the other end.
Both ends have this problem.
The local id and remote id are correct for both pfsense instances.

It looks to pretty random when it occurs, it can take 3-4 hours, one hour or event 15 minutes.

I tried playing with the lifetime and rekey settings as well as local/remote ip's and outbound nat rules (both ends are in a carp situation, with for the time being the slaves being turned off)

Logs don't give errors (IKE SA, IKE Child SA, Configuration backend and StrongSwan Lib in diag mode)

Is there anyone that could be help me out? Maybe had or currently has the same problems?