IPsec VPN with NAT/BINAT goes up and fails after 60 seconds

[Midnight_Shadow]

I have a problem having the vpn stay up. It is getting up and the status goes to green, but after about a minute it's gone and the green goes to error.

The remote setup is managed by a 3rd party who gave me the details for building the vpn. Normally they deliver a netscreen for a fair amount plus a yearly fee which is just too much.
I wanted to do it myself with pfSense since I had it running anyway.

The goal is to have the local subnet connunicate with the remote subnet.

My setup:
Router external public ip: A
Internal subnet 192.168.100.0/24 GW 192.168.100.254
NAT/BINAT: 172.21.1.0/24
Rule in IPsec tab to allow everyting over IPsec

3rd party External public ip: B
3rd party External subnet 10.220.31.0/24

Disabled "Prefer older IPsec Sas"
Disabled MMS clamping
Added a rule for 10.220.31.0/24 as source to Lan net for TCP/UPD on the external and LAN tabs
Rule addes on the IPsec tab for protocol IPv4 to allow everything
No NAT 1:1 rule
NAT outbound setting: Automatic outbound NAT rule generation (IPsec passthrough included)

IPsec pahse 2 settings:
Local network: LAN subnet
NAT/BINAT type: network -> 172.21.1.0/24
Remote network type: Network -> 10.220.31.0/24

Starting the vpn seems to work as the icon turns green. After about a minute it fails. I don't have the idea that the tunnel is working at all.
Over the IPsec interface I only see icmp packet coming from B -> A. A is not responding to B or B is not accepting ICMP over the IPsec.

I see this 6 times in the packet capture log of the IPsec interface before the tunnel fails:
18:13:49.043078 (authentic,confidential) : SPI 0x01141aed: IP B > A: ICMP echo request, id 1024, seq 49112, lenght 24

Dec 20 09:22:49 racoon: INFO: unsupported PF_KEY message REGISTER
Dec 20 09:22:49 racoon: INFO: unsupported PF_KEY message REGISTER
Dec 20 09:22:49 racoon: ERROR: such policy already exists. anyway replace it: 192.168.100.0/24[0] 10.220.31.0/24[0] proto=any dir=out
Dec 20 09:22:49 racoon: ERROR: such policy already exists. anyway replace it: 10.220.31.0/24[0] 172.21.1.1/24[0] proto=any dir=in
Dec 20 09:23:12 racoon: INFO: unsupported PF_KEY message REGISTER
Dec 20 09:23:18 racoon: NOTIFY: no in-bound policy found: 10.220.31.0/24[0] 192.168.100.0/24[0] proto=any dir=in
Dec 20 09:23:18 racoon: [nimvpn1]: INFO: IPsec-SA request for External B queued due to no phase1 found.
Dec 20 09:23:18 racoon: [nimvpn1]: INFO: initiate new phase 1 negotiation: External A[500]<=>External B[500]
Dec 20 09:23:18 racoon: INFO: begin Identity Protection mode.
Dec 20 09:23:18 racoon: INFO: received Vendor ID: DPD
Dec 20 09:23:18 racoon: [nimvpn1]: INFO: ISAKMP-SA established External A[500]-External B[500] spi:babfe5d93bd745b3:c3376bbbdc20d228
Dec 20 09:23:19 racoon: [nimvpn1]: INFO: initiate new phase 2 negotiation: External A[500]<=>External B[500]
Dec 20 09:23:19 racoon: WARNING: attribute has been modified.
Dec 20 09:23:19 racoon: [nimvpn1: INFO: IPsec-SA established: ESP External A[500]->External B[500] spi=193227799(0xb846c17)
Dec 20 09:23:19 racoon: [nimvpn1]: INFO: IPsec-SA established: ESP External A[500]->External B[500] spi=3567566563(0xd4a4bee3)
Dec 20 09:23:54 racoon: INFO: purging ISAKMP-SA spi=f123f6587c2c4e84:1726fd9b22e849a0.
Dec 20 09:23:54 racoon: INFO: purged ISAKMP-SA spi=f123f6587c2c4e84:1726fd9b22e849a0.
Dec 20 09:23:54 racoon: [nimvpn1]: INFO: ISAKMP-SA deleted External A[500]-External B[500] spi:f123f6587c2c4e84:1726fd9b22e849a0
Dec 20 09:25:06 racoon: INFO: purged IPsec-SA proto_id=ESP spi=3567566563.
Dec 20 09:57:41 racoon: INFO: unsupported PF_KEY message REGISTER
Dec 20 09:58:52 racoon: INFO: unsupported PF_KEY message REGISTER

Any idea why the tunnel would fail? I can think if the gateway A not responding to B on the ICMP requests over the IPsec interface and B dropping the connection because it thinks A is dead.
Maybe I need to add rules to allow traffic etc.

Any help is greatly appreciated!

[Midnight_Shadow]

It seems to be related to:
https://redmine.pfsense.org/issues/3321