Captive portal settings : example of a possible setup :

Settings for a FreeRadius user (this one can use 3096 Mb ) :

Every minutes or so, you'll see this in in Status > System Logs > System > General

@bingo600

The reel checks are not done during start or stop.
Re read this part :
For a user set up in FreeRadius :

and this (Captive portal settings ):

and

You saw the snip of the logs above ?
These are repeated every minute, or 5 minutes , whatever delay you choose - see images above, to update the counters with used 'bytes'.
Btw : this fills up de system log really quickly - normally, I deactivate the line in the script that does does the logging, and I only keep the log line that says when the user is thrown of the portal because he used all of his amount of Download and Upload Traffic.
The magic is happening here : /usr/local/etc/raddb/scripts/datacounter_auth.sh

I'm not using FeeRadius on pfSense, and have not read how the manual says it should perform on pfSense.
This is just a "quess".

It could be that quota is only checked on "user login", as that is where Radius is involved.

My guess would be that the radius accounting records are saved on logout , and then at next login the data is available.

I have no idea if Radius is queried during an ongoing session, but i haven't seen that on systems i have used.

After exceeding the quota , have you tried to logout/login ?
To see if the the Radius accounting is being queried during logon , and if quota limit then would work ?

/Bingo