Hi Derelict!
I am in the exact same situation as NekoSema and tried to solve it the same way, before stumbling upon this thread.
I already did what you said, except for:
"X.X.28.3 needs to know to route traffic for X.X.96.0/24 back to pfSense. (Guessing on the subnet since it was unspecified.)"
I don't know how to accomplish that. I thought it might be a static route, but I don't know how to define it.
I know this thread is old, but it is the exact description of the situation that I am facing.