Check Status/Queues to make sure the traffic isn't actually in the queue and the state counter just isn't counting up.
Check Diagnostics/States and see if the state is matching your source. For instance in order to lower priority on a download the state may be to the webserver:443 and the return traffic matches the state but not the rule. (so, doesn't seem to count up the counter...)
If you are not using limiters, then note this from the guide;
The ALTQ framework is handled through pf and is closely tied to network card drivers. ALTQ can handle several types of schedulers and queue layouts. The traffic shaper wizard configures ALTQ and gives firewall administrators the ability to quickly configure QoS for common scenarios, and it allows custom rules for more complex tasks. ALTQ is inefficient, however, so the maximum potential throughput of a firewall is lowered significantly when it is active.
pfSense software also supports a separate shaper concept called Limiters. Limiters enforce hard bandwidth limits for a group or on a per-IP address or network basis. Inside of those bandwidth limits, limiters can also manage traffic priorities.