OK, I think I found why it doesn't work.
My VOIP adapter is NATed. I guess that NAT rules are applied before the floating rules. So floating rules never "see" the LAN IP address of my adapter…
In this case, to send the VOIP traffic in the right queue, I have to choice to use the IP address of the SIP server as destination.