@phil.davis:
System: Advanced: Miscellaneous
Skip rules when gateway is down - By default, when a rule has a specific gateway set, and this gateway is down, rule is created and traffic is sent to default gateway.This option overrides that behavior and the rule is not created when gateway is down
Check that box - pfSense is too nice, and when the target gateway is down it changes the rule to just pass the traffic to the dwefault routing table (= out the default WAN in most cases). This box disables that "niceness".
That fixed it. It seemed like it was some type of failover because it wasn't immediate. I looked and looked but was in the wrong area. Thanks for the help!