Are you using the DNS Resolver, perhaps? Maybe DNS over TLS? or DNSBL?
We found out there are some memory leaks in the version of unbound shipped with 2.4.4. They were recently fixed upstream in Unbound, and we'll have them in 2.4.4-p1 soon.
That's the only known memory leaks at the moment that I can think of.