sg-1100 2.4.5 unbound python module + DHCP lease DNS registration memory leak
-
I turned off DHCP leases being registered at the same time, preliminary results look good here. I wonder if the reloads caused by DHCP renewals is the actual cause of the issue.... Turning it back on to see.
-
So this appears to be a memory leak that happens when you use a python module and have "register DHCP leases in DNS resolver" enabled.
-
@bruor said in sg-1100 2.4.5 unbound python module + DHCP lease DNS registration memory leak:
"register DHCP leases in DNS resolver"
has this nice habit of restarting the Resolver on every new lease on a LAN.
The cache get's ditched while doing so.
When you use a package like pfBlockerNG with a lot of feeds, the restart will take (a lot of) time, during which no DNS isn't working : another not so nice side effect.
You have to have special reasons to have "register DHCP leases in DNS resolver" activated (checked).Anyway.
I'll activate 'check' mine and see what happens with the memory. -
@Gertjan here I have about 40 devices on the network and it it takes about a day for unbound to consume 400ish MB of RAM.
My best guess is that the restart of unbound is just a reload and some stuff stays resident in mom, and gets launched again. If I do a full restart of the unbound service it'll free up the ram it's hogging and go back to normal.
In order to diagnose this I actually disabled pfblocker-ng and the issue still happened.
-
@bruor said in sg-1100 2.4.5 unbound python module + DHCP lease DNS registration memory leak:
My best guess is that the restart of unbound is just a reload and some stuff stays resident in mom, and gets launched again.
The thing is, it's worse... Dono ... depends which issue we're talking about.
When unbound "reloads", the process is stopped, all memory is freed. This in an "OS" thing, not an unbound thing.
Then, it starts again ...This repeatability process re starting can not leak memory. If it would, millions of FreeBSD installs would fault after several minutes ...
But .... unbound instantiates Python. and I presume (^^) that, when unbound stops, Python also 'stops' == freeing its used memory and resources. -
I agree, but I believe the DHCP leases cause an unbound reload and not a full restart. I have a feeling that reload doesn't clean up the python process but launches another. Full restart however, does. I think I could figure out how to just script 20x reloads for unbound back to back to see if this is the case.
-
@bruor said in sg-1100 2.4.5 unbound python module + DHCP lease DNS registration memory leak:
I agree, but I believe the DHCP leases cause an unbound reload and not a full restart
A boatload of forum threads already handle that questions over the last two, three years.
A couple of us actually looked at the source code of unbound. It's a 'free' project anyway, so : why would you choose to believe if you can see the thing in front of you : Unbound, when it receives a SIGHUP signal, stops. It will exit. It does not implement a something that looks like a "reload the config an go".I activated (checked) " DHCP Registration Register DHCP leases in the DNS Resolver" - I have about 40 LAN devices ** and a another LAN network for visitors/clients/BYOD devices.
My pfSense Memory usage is public info : https://www.test-domaine.fr/munin/brit-hotel-fumel.net/pfsense.brit-hotel-fumel.net/memory.html
Active memory is actually .. lowering. W'll see what happens after some days. -
Thanks for trying to reproduce on your end.
When looking at https://www.freebsd.org/cgi/man.cgi?unbound.conf it indicates that a HUP is handled as a config reload and is different than a full restart. Mentioned in the interface and username sections
-
Can you have a look at your unbound process and see if it is consuming more ram than expected? Based on your graphs it's definitely not impacting your setup as radically as it did mine at home.
-
@bruor said in sg-1100 2.4.5 unbound python module + DHCP lease DNS registration memory leak:
Thanks for trying to reproduce on your end.
When looking at https://www.freebsd.org/cgi/man.cgi?unbound.conf it indicates that a HUP is handled as a config reload and is different than a full restart. Mentioned in the interface and username sections
Doc is nice.
But pure BS - or at least the interpretation is wrong.
The nice thing of Open Software is : the real manual is the code. You can read it yourself.
Again : HUP = dump process and start over. It's just a couple of lines of Code.https://github.com/NLnetLabs/unbound/blob/68eab24db7b3ecbd9c13feeaa4e3f4e4aea9c08a/daemon/unbound.c
You find the main() - the rundeamon() and here https://github.com/NLnetLabs/unbound/blob/334498d9b94f080cd29fa5e1d3d426b9d1edfe6b/daemon/worker.c you see how, in a worker thread, the signals are handled.New seems : a HUP (SIGHUP) keeps the interfaces 'as is , and reloads the rest. That not ok for pfSEnse, where interfaces can come and go : cables are unplugged, OpenVPN instances are started or stopped, etc.
But, hey, who cares : see for yourself the ancient way : dump the resolver.log real time in an console/SSH window.
In another console/SSH, stop start / restart / reload / whatever your unbound process.
The log times in seconds will give you all the details.For me : stop + restart or reload or restart = identical.
To have it sweat a little bit, I bloated my pfBlockerNG-devel more then usual. More then a one GB of memory was being used.@bruor said in sg-1100 2.4.5 unbound python module + DHCP lease DNS registration memory leak:
Can you have a look at your unbound process and see if it is consuming more ram than expected? Based on your graphs it's definitely not impacting your setup as radically as it did mine at home.
I had pfBlockerNG-devel shut down for a day or two. It needed a day of.
-
Thanks, perhaps this is limited to the sg-1100 in some way. Opened a redmine to see if they can reproduce at their end
-
pfblocker has made a note about python modules being incompatible with DHCP lease registration updates in unbound needed a full restart vs sighup.
https://forum.netgate.com/topic/158592/pfblockerng-devel-v3-0-0-no-longer-bound-by-unbound/2
This led me to another search which seems to indicate that this is a long-standing issue.
https://redmine.pfsense.org/issues/5413 -
This has been addressed in the latest snapshots.
We are testing the changes and will include them in 23.01 which is due soon.
The issue is multifaceted.
I've submitted upstream patches to both Unbound and the MaxMind DB Python module.
The MaxMindDB Python module had several issues. The major issue though was a reference counting bug causing the Python garbage collector to prematurely free a heap-allocated structure. This led to a use-after-free causing Unbound to segfault.
Unbound reloads the built-in Python interpreter every time Unbound is reloaded either by a SIGHUP signal or using the unbound-control interface. Python was not designed to be reloaded like that in the same process.
I've fixed the refcounting bug in Maxmind, and patched Unbound so Python is only initialized and unwound once. I've also upgraded Python from 3.9 to 3.11.
The memory usage should be significantly improved.
The next improvement would be to rewrite the integration with ISC DHPCD to use a better interface with Unbound. That likely will have to wait until 23.05
