Thanks for the replies. I got it working but I am not sure how. I was sniffing with wireshark and in the midst of my troubleshooting it started working. Did not change anything in the firewall config.
The router was working correctly as well. It was sending out ICMP packets that were getting a response and the ones that were not getting a response exactly the same way.
One key thing that I forgot to mention in my original post is that this is a cable connection.
Since the time the problem mysteriously fixed itself I have tore down the whole setup a couple of times and rebuilt it. And I have been able to reproduce the problem but not with consistency.
One thing I found that fixes the problem every time is rebooting the cable modem after I am done creating all the VIPs. Any VIP that did not exist when the cable modem was powered up has a random chance of working. But if I reboot the cable modem after creating the IP then it would work every time. I did not sniff the wire to see if the modem was somehow ping sweeping my CIDR block to see which IP is live and which is not. It wouldn't make sense for it to do that.
In any case, I'll post back if I find anything concrete. Right now my troubleshooting is not conclusive but at least I know how to get it working.
What a great product BTW. Very impressed.