Yup it's that. If an interface goes down then IP addresses on the firewall change. I agree it's a bit of a big hammer and ideally could be far more nuanced. But that's what you're seeing.
Interestingly there is a quirk you can probably use to stop this happening. If you enable IPv6 on the wifi interface and set it to track a WAN for v6, even if there is no IPv6, it will by ignored.
Steve