VOIP over OpenVPN post call delay for reuse

[4o4rh]

I am not sure if this is a pfsense settings, openvpn or voip settings issue, but the problem i have is;

• i have 3 voip providers for incoming and 1 outgoing provider.
• voip is routed over openvpn connections to public provider
• calls to all provider test numbers have perfect quality as does normal calls.

Problem I have is, when the call hangs up, either by myself or from the other end,
i have to wait 4-5s (maybe longer) before i can make the next call. If i try to call too soon, i get a connection show but no voice.
So it seems like the previous connection is still held for a short while.

Is there a way to drop the connection quicker so i can make the next call right away?