There isn't an IPsec traceroute exactly, but you can use hping to get a similar effect.
For example, to see how far ESP (tunneled traffic) makes it, run this command:
$ hping -0 1.2.3.4 -H 50 -d 10 -t 1
Replace 1.2.3.4 with the far side, and increase the -t value until you do not get a response. That's where it gets dropped. I see -T in the hping settings now and it might be helpful as well.
If the tunnel won't connect at all, however, that would be a simple UDP test for port 500 and 4500. For that you can probably use a traditional traceroute command with -P UDP -p 500 and again for -p 4500.