@tomwaller:
Am I doing something fundamentally wrong here. I thought the packet would come back the way it entered but I guess not.
No, return traffic follows the system routing table, which means it sends it out the directly-connected network in that circumstance.
If you dual home a system like that, don't connect cross-subnet to it. Strictly connect to its IP on the same subnet as the source machine, or the IP of the interface where the default gateway resides if it's sourced from a network where the destination system isn't directly connected. That'll ensure no issues along those lines.