Ipsec starts flapping when I enable ipv6 on WAN

[hhkb]

As soon as I enable ipv6 on my WAN, my ipsec connection starts to flap, disconnecting every 15 minutes or so. Disabling ipv6 again fixes the problem. I've tried digging into the logs and configuration but can't figure out why this would happen. ipsec logs just show it disconnecting which matches what I'm observing (my various services fail to send data during the downtime of the tunnel). I'm using ipv4 addresses and protocol in my ipsec configuration. Anyone have any idea?