Don't cap the other bandwidth, set the Voip queue to have 320Kbps of realtime bandwidth.
i.e. 320Kbps is reserved for the Voip queue.
Since your phones should have static IPs, set their IPs into an alias and use a firewall rule to mask the source as the alias and pipe all the traffic into the Voip queue.
This is rudimentary, of course, since the 320Kbps is forever reserved for the voip purpose but tweaking for a more fair share is very heavily dependent on the specific traffic type you see on the network.
Example:
Assuming each voip packet is 1.6kb and you need 30ms max. for clear calls. This is for one way, you need another similar queue for the other direction.
qVoipUp & qVoipdown
realtime (m1 d m2): 6.4KbKb 30 160Kb
bandwidth (m1 d m2): 160Kb 100 160Kb
Assuming the phones are using 192.168.1.100 to 192.168.1.104 as their IPs:
Alias IPs 192.168.1.100 to 192.168.1.104 as 'voipips'
Set the firewall rules to:
voip outbound
Protocol: Any
Source port: ANY
Source IP: voipips
Dest. port: ANY
Dest. IP: ANY
Queue: qVoipUp
voip inbound
Protocol: Any
Source port: ANY
Source IP: ANY
Dest. port: ANY
Dest. IP: voipips
Queue: qVoipDown