For what it's worth, we figured out a way to do it with the Cisco ASA 5505. I was able to issue to the VPN users the IP addresses in the same subnet as the 10.4.0.0/20 network. Then I had to add some very strange looking acl's allowing 10.4.0.0/20 to talk to 10.4.0.0/20. That seems very strange to me, but it works.
I'd still really like to know how to make this work with Pfsense, so if anyone has any ideas, or has questions about my setup please chime in.
Thanks.