That's part of the nature of conservative mode -- states will pile up more. If some client behavior changes and makes the clients open more states, then they'll hang out longer.

What you could do is keep the router itself on normal mode and setup custom state timeout rules to match the VoIP traffic which sets different state timeouts just for them, and perhaps only for VoIP/RTP traffic for example. Narrow down the longer state lifetimes as much as possible.