I have a similar problem. I have 1 500/100 Mbit/s Fiber thru vlan connection and a floating rule for limiters with interface wan and wan gateway, match, out.. I'm using this wan gateway in every out connection rule except one where I use a vpn gateway. As soon as I connect the VPN, my WAN only does 50 ish mbit/s upload. If I disable the floating rule, it returns to normal. If I disable the VPN, with floating ruleon, it returns to normal.
2.4.4-RELEASE-p3 (amd64)
built on Wed May 15 18:53:44 EDT 2019
FreeBSD 11.2-RELEASE-p10
Intel(R) Atom(TM) CPU C2558 @ 2.40GHz
4 CPUs: 1 package(s) x 4 core(s)
EDIT: Forgot to say that I solved my problem adding the queues in the rules and not using the floating rule.