There is a racoon configuration directive called 'log info|debug|debug2'. I added it manually to /var/etc/racoon.conf, killed racoon and started it manually using /usr/local/sbin/racoon -f /var/etc/racoon.conf. Too bad there isn't a GUI option for it somewhere (or is there?)
Anyway, more specifically, by setting it to debug2, I get this:
Jun 29 17:30:58 racoon: DEBUG: 32953411 3a24b070 00000000 00000000 01100400 00000000 0000010c 04000038 00000001 00000001 0000002c 01010001 00000024 01010000 800b0001 800c7080 80010007 800e0100 80030001 80020002 80040002 0a000084 8239ea94 e4bf1ad1 3c9a02d3 6103ba0b 50b669b5 8ca55b22 79f90a6f 62d4f840 85632dcb cfa7e7c5 ea5601da 724aa79e 5a8b6997 15739a07 79330d88 948ffa4c 20a19ce6 442538f0 d0182aaa caf80d76 9c47049f 11cd3c72 471e475a c6d675bc ca4a1f7d b1271636 52c30de3 2ac6ea4c bc945bd3 e9683a82 fc5b0d0a 236f2ef8 05000014 99e5be30 5910045b b768c0a6 89ef8c57 0d00000c 011101f4 40936165 00000014 afcad713 68a1f1c9 6b8696fc 77570100
Jun 29 17:30:58 racoon: DEBUG: resend phase1 packet 329534113a24b070:0000000000000000
Jun 29 17:30:58 racoon: DEBUG: ===
Jun 29 17:30:58 racoon: DEBUG: 40 bytes message received from 62.x.y.z[500] to 64.a.b.c[500]
Jun 29 17:30:58 racoon: DEBUG: 00000000 00000000 00000000 00000000 0b100500 f37b30ee 00000028 0000000c 00000000 0100001d
Jun 29 17:30:58 racoon: ERROR: malformed cookie received.
Any idea what might be causing the malformed cookie?
– james