DNSBL - Difference Unbound Mode / Unbound Python Mode
-
Hi!
I have a general question about DNSBL.
To be precise, about the difference between the Unbound Mode and the Unbound Python Mode.
According to the description, Python mode has advantages and I don't know why it is not used by default.Are there any reasons or further information?
-
@deleted said in DNSBL - Difference Unbound Mode / Unbound Python Mode:
Unbound Mode
What was used before.
Basically, pfBlocker placed something like this in here :
where a file was listed that would be read during the startup.
This wile was one big list with host names and their desired IP addresses, typically all "0.0.0.0" or "10.10.10.1".
It was discovered quickly that people (also knows as the admins of pfSense), thanks to the big list with available DNSBL, and some of them are huge, they started to add them all (yeah, don't ask me why ..).
Like : xx million host names, all in one big file, and lets see what happens when unbound (the resolver) tries to read them in. Failures were multiple. Out of Memory (RAM) was one known reason, and if unbound managed to read in everything, it has been seen that it needed 15 minutes or more to do so. Not a real issue, but during this time DNS is not available.
As nobody knows what DNS really is, I translate this situation to a more known problem"Internet is broken again".
and the pfSense admin had to sleep in the dog house again.
Something needed to be done.
And then came the good news : nlnetlabs,nl the author of unbound, was also aware of the situation. ( or they also hated the dog house )
So, programmers @nlnetlabs did what everybody does these days : they add "scripting" or "API's" so internal unbound functionalities can be used by scripts written by you, me and everybody else.
Basically, nlnetlabs.nl did what Netgate does with pfSense, or Microsoft with Windows : you can add (in order) python scripts, pfSense packages or Programs to extend the functionally of their products.So, the python mode is nothing more as an Unbound 'extension' (written in Python, hence the name) so the author of pfBlockerng had more control over what happens. More stats became available, more functionalities became available, and .... roll the drums .... it is way faster.
Imho : always use python mode.
-
Hi @Gertjan,
Thank you for the detailed explanation.
Then I active the python mode now to test.
So activate the mode in Unbound and in pfblocker, right? -
Resolver default settings are :
as these are the settings Netgate has chosen for you.
They have their reasons to do so
So, nothing to do over there.
The default pfBLockerng(devel) DNSBL mode is (but I'm not sure) :
Edit : the resolver "Python Module Order" Validator mode is also a python mod.
It implements DNNSEC (DNSSEC is DNS with the guarantee that your head will hurts if you want to know what it does). Just set it and forget it. -
-
Not that I recommending against Unbound python mode (it seems to be almost a requirement with Kea), but there is a slight downside to using pfBlockerNG's Unbound python mode. Even with pfBlockerNG's DNSBL logging disabled, it increases disk IO. As an example, on my system, which has very little logging activity, enabling Unbound python mode in pfBlockerNS takes my disk IO from below 13bps to a bit over 40bps.
-
True.
Most of use rarely look at this :
The DNSBL actions are logged :
so the pfBlockerng candy bar system can do its work - generate this :
Stats, over time, need lots of details.
Remember the very beginning, these are my thoughts :
There was this firewall router called pfSense.
And we had a separate VM, or physical box with a Pi-hole.
All that pi-hole needed, was also present on pfSense.
So, some one decided to integrate "what pi-hiole" does into pfSense, as less (hardware) == better.pfBlockerng needs to write to disk (logs, or some database storage, whatever) so the admin can see what happens and when.
After all, DNSBL feeds are loaded, so, for now on, some host names, an action is takes that blocks the outcome of a URL (DNS) request. Most common is a web browser on some LAN client device that doesn't show an publicity anymore, up until "pfSense is broken as I can't visit facebook anymore and I also loaded pfBlockerng with 'some' DNSBL feeds - what's up ?" (no joke, these questions really exist, found one yesterday).That said, I agree, you've mentioned an advantage of the unbound (non python) mode.
On the other hand, also yesterday, I discovered that a 128 Gbytes SSD (not emmc) for my '4100' costs what ? 30 $ ?
Btw : running a pi-hole on any device with an emmc device, guess what will happen ? ^^
Maybe there should be an option where all logging and stat building can be stopped.
Or : Plan R : create a RAM disk, and have that synced to disk only during reboot - with the risk of data (stats) loss when a power fail arrives.
-
@Gertjan said in DNSBL - Difference Unbound Mode / Unbound Python Mode:
The DNSBL actions are logged
...
pfBlockerng needs to write to disk (logs, or some database storage, whatever) so the admin can see what happens and when.The increased IO does not come from logging. I have DNSBL logging completely disabled. I.E. Null Block (no logging).
The increased IO is intrinsic to pfBlockerNG's Unbound python mode, and comes from the write of a group counter to a sqlite3 database, /var/unbound/pfb_py_resolver.sqlite. Even if logging/reporting is completely disabled, there is no way to disable the counter update without turning off pfBlocketNG's python mode.
-
I worked-out the above the long way but nice to see the proper explanation.
Not that I am bothered about logging as mine goes to a good SSD, rather than the eMMC, and that will never wear out from logging.
️ -
Since there is still a lot going on here, a quick question;
How do I delete all the entries under Reporting?
Everything individually in the logs? -
@dennypage said in DNSBL - Difference Unbound Mode / Unbound Python Mode:
Unbound python mode, and comes from the write of a group counter to a sqlite3 database, /var/unbound/pfb_py_resolver.sqlite.
Forgot all about that one.
There are one (or two ?) small database files that gets updated constantly to reflect the current counters :
@deleted said in DNSBL - Difference Unbound Mode / Unbound Python Mode:
How do I delete all the entries under Reporting?
That's, afaik, not possible. Not without severe patching.
To minimize disk usage : in this order :
Use unbound mode, not python mode.
Do not use DNSBL,
Use RAM disk
Or, the best method : don't install any pfSense packages that use and need disk recording. After all, pfBlockerng doesn't come with pfSense pre installed, it's a option you activate by adding it yourself. -
The starting advice is just to move everything to a media that can take the additional writes with ease, such as an SSD made for the job, or export stuff to a logging/monitoring service somewhere on your network.
The joy of pfSense is all the other things it can do to support your network needs. Don't feel the need to disable stuff that is useful to you if you can just reconfigure things.
️ -
My only concern was that I generated a lot of entries during testing and I wanted to clean them up.
However, the idea with the SSD is good to reduce the load on the system.I'll find out the best way to do this.