Going to leave a reply here since I figured it out.
Turns out layer 3 routing doesn't work with a switch.
Which in retrospect obviously it doesn't, switches are level 2.
I had previously followed a guide that had me set up an interface group consisting of a few nics I have on the pfsense box.
And the guide had me set up an allow all rule on the interface group.
And any rules I placed on the interface group doesn't actually redirect packet to gateway according to source.
It just doesn't work.
But as it turns out, I didin't actually need the allow all rule on the interface group anyways.
I deleted the rule on the interface group, and then everything worked as it should on lan0.
Again, this should have been obvious in retrospect because the lan0 firewall rule never had any states, but the interface group rule had all the states.
Hope this helps someone else.