Now that you mentioned, it could be NAT problem indeed.
Here is the setup anyway:
HOST-A (behind nat)
private IP: 10.x.x.x. (translated into public IP)
LAN: 192.168.5.x
HOST-B (no nat)
public IP WAN: x.x.x.x
LAN: 192.168.10.x
On HOST-A I have disabled outbound NAT, as it's managed on the vmware side and on the host-b outbound NAT is set to auto.
EDIT:
Thanks jlevesque. It seems to be NAT issue indeed. I've tried to add third host who is not behind NAT and ipsec connectivity is working out-of-the-box with default settings. I've even tried to change between different encryption methods and change p1 and p2 a bit, but it was working.
I will investigate this more further.