It's working as intended, but perhaps not the way you expect:
https://www.freebsd.org/cgi/man.cgi?query=pf.conf&apropos=0&sektion=0&manpath=FreeBSD+11.2-RELEASE&arch=default&format=html#STATEFUL_TRACKING_OPTIONS
For stateful TCP connections, limits on established connections (connec-
tions which have completed the TCP 3-way handshake) can also be enforced
per source IP.
max-src-conn <number>
Limits the maximum number of simultaneous TCP connections which
have completed the 3-way handshake that a single host can make.
max-src-conn-rate <number> / <seconds>
Limit the rate of new connections over a time interval. The con-
nection rate is an approximation calculated as a moving average.
Because the 3-way handshake ensures that the source address is not being
spoofed, more aggressive action can be taken based on these limits. With
the overload <table> state option, source IP addresses which hit either
of the limits on established connections will be added to the named ta-
ble. This table can be used in the ruleset to block further activity
from the offending host, redirect it to a tarpit process, or restrict its
bandwidth.
The optional flush keyword kills all states created by the matching rule
which originate from the host which exceeds these limits. The global
modifier to the flush command kills all states originating from the
offending host, regardless of which rule created the state.
For example, the following rules will protect the webserver against hosts
making more than 100 connections in 10 seconds. Any host which connects
faster than this rate will have its address added to the <bad_hosts> ta-
ble and have all states originating from it flushed. Any new packets
arriving from this host will be dropped unconditionally by the block
rule.
block quick from <bad_hosts>
pass in on $ext_if proto tcp to $webserver port www keep state \
(max-src-conn-rate 100/10, overload <bad_hosts> flush global)
You specified a rate. Not that it is not a literal count of X states in Y time, but a calculated average. If they connect at a rate faster than the one you chose, on average, they will be blocked, even if they did not explicitly exceed the stated limit over the entire time period.
