Problems with SIP over IPsec Tunnel

[johnlile]

My setup currently looks the following:
Site A has a FreePBX / Asterisk telephony server and multiple IP phones
Site B has just one IP phone

Both sites are connected to each other through an IPsec tunnel, which works without problems. All hosts on both site A's and site B's LAN subnets are reachable by each other. The Firewall Rules allow any host using any protocol.
The IP phone at site B can register itself on the PBX, though when you start a call, you can't hear the other person and the other person can't hear you. After about 6-7 seconds the call automatically hangs up.

This is the log output from Asterisk regarding said call:

[2020-11-02 19:01:37] WARNING[764] chan_sip.c: Retransmission timeout reached on transmission 612e364eea3d6118 for seqno 1007277218 (Critical Response) -- See https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions
Packet timed out after 6399ms with no response
[2020-11-02 19:01:37] WARNING[764] chan_sip.c: Hanging up call 612e364eea3d6118 - no reply to our critical packet (see https://wiki.asterisk.org/wiki/display/AST/SIP+Retransmissions).

I have already tried increasing the UDP Timeout under System -> Advanced -> Firewall & NAT to 120 seconds for all three entries. I set the Firewall Optimizations to Conservative. I also tried setting the State Type of the IPsec interfaces to none, but I set it back to keep, as none didn't work either.
Site A can call out to not just other local phones without issues. As soon as I connect the phone from site B directly to site A's network I can also make calls without problems, which leads me to believe that it has to be a problem with the VPN.
I am not bound to using IPsec, I also tried OpenVPN already, but unfortunatley, that didn't work either.

Thanks for your help.