I did get this working now in version 2.2.4 after doing a lot of packet captures and troubleshooting. I have another different virtual IP address setup for IPsec and L2TP (both enabled) on the pfSense box itself. When I disabled those it started to work.
This leads me to believe that there might be a bug is IPsec & L2TP services on pfSense. When enabled they will not forward udp port 500 traffic on other virtual IP's. Once disabled they do pass the traffic.