Good news. More forum searching with a few different terms and I came across this post. https://forum.pfsense.org/index.php?topic=76735.15
This lead me to upgrading the older pfsense install I had at the remote location. After upgrading the tunnel came up and I tested more backups with my original way of using rsync.
So far so good, I have transferred many gigabytes over this tunnel without any random crashes.