Problem solved.
In the remote branch there was another device (Sophos XG105) connected to the internet with a buggy 4G connection…
This device was setup with the same parameter (IPsec initiator) than the pfSense box and was, sometimes (no idea when/why), connecting the main XG appliance.
The message in the log (main XG appliance) is: "System received a P2 connexion request whose Localsubnet-Remotesubnet configuration conflicts with that of an already established connexion "XXXX-1". System is terminate connection "XXXX-1" to honor the incoming request."
That message leaded me to thing there was an issue on the pfSense box, trying to start several tunnels.
It was (obviously) not the case, it was another device...
Once that other device is shutdown, problem is solved.