Ok, I believe I've found the root cause and it was a misconfiguration.
I have "Manual Outbound NAT" configured. This is to use a 4 address pool instead of the firewall's WAN address for NAT.
So even though there was an wildcard "Auto Created Rule" for TCP port 500 (ISAKMP) using the WAN address, there wasn't a rule for the ESP protocol. I added a wildcard rule for ESP that used the WAN address last night, and haven't had any issues since.
In retrospect, this makes sense, since each time I lost connection, the source address was one of the addresses in the pool. What doesn't make sense to me is that it ever used the WAN IP address. Like so many things, it would have been a lot easier to diagnose than a connection that always failed than one that sometimes worked.
I think an argument could be made that if pfsense is going to add an Auto Created Outbound NAT rule for ISAKMP, it should probably create an Auto Created rule for ESP at the same time.