Check Status/Queues to make sure the traffic isn't actually in the queue and the state counter just isn't counting up.
Check Diagnostics/States and see if the state is matching your source. For instance in order to lower priority on a download the state may be to the webserver:443 and the return traffic matches the state but not the rule. (so, doesn't seem to count up the counter...)
Is low_priority_src a LAN IP behind NAT? If so it can't match, see https://docs.netgate.com/pfsense/en/latest/trafficshaper/advanced.html#shaper-rule-matching-tips for info on tagging it.
Brilliant, thanks so much for the hint/info on tagging!