The problem appears to be solvable with BINAT in the IPsec phase 2.
https://doc.pfsense.org/index.php/NAT_with_IPsec_Phase_2_Networks
Under local network I have put my LAN addresses.
In the NAT/BINAT configuration I have put my external IP, which the other end sees.
In remote network I have put the public /30 addresses.
Something was not entirely right in the other end, will have to wait for them to get back.