DNSBL SQLite3 database [ lastevent ] corrupt messages in system logs
-
@jrey Thank's a lot for your advice! I'll dig into this. I have watchdog on because unbound (and somertimes kea dhcp) stopped working (I suspect after pfblocker updatding at night).
I'll try again without. -
@Klaus2314 said in DNSBL SQLite3 database [ lastevent ] corrupt messages in system logs:
stopped working
I've edited my post above with some observations.
You will find some very possible reasons why.@Klaus2314 said in DNSBL SQLite3 database [ lastevent ] corrupt messages in system logs:
and somertimes kea dhcp
Be gentile to yourself : switch back to ISC for the moment.
-
@Gertjan ok, thanks again. I can't delete the files. permission denied.
Doesn't sudo work on BSD? It says unknown command.Again, sorry for my novice ignorance :-D
-
@Gertjan said in DNSBL SQLite3 database [ lastevent ] corrupt messages in system logs:
It will take some serious time to do so.
you'd think that will default settings a zone file that big the system would be throwing memory errors. That's a lot of zones. of course there is a limit as to how many it loads and ignores the rest. Either way though the list is too big.
@Gertjan said in DNSBL SQLite3 database [ lastevent ] corrupt messages in system logs:
switch back to ISC for the moment.
Also true.
-
does your ssh prompt say you are root or admin?
if you ssh in with a "normal" user account you won't have permission.
ssh bob@... logs in won't work (bob doesn't have permission)
ssh admin@... or ssh root@... logs in will work. -
@jrey said in DNSBL SQLite3 database [ lastevent ] corrupt messages in system logs:
default settings a zone
The default pfb_py_zone.txt file size is 'zero' as pfBlockerng has no DNSBL pre set or pre loaded.
Then the admin comes along, and selects a whole bunch of these :
and then .... major issues happens. The system can't cope anymore.
Btw : unbound doesn't only 'allocate' memory to load the file, and stores it into that memort check.
Noop, it reads the fiie, interpretes every line, and starts to fill structures 'per line', making each line part of a big array with DNSBL to be matched on every host name request ...
This is done using arm or intel instrcution, true, but even so : it's very time consumming.
How long can be seen here :
so with 119414 entries, it took 15:57-15:22 = already 35 seconds.
Oh, I get it : @Klaus2314 thought that unbound was 'broken' ....
Klaus2314 : check for yourself, ask for apfBlockerng reload here and see what the log tells you :
and hit the blue run button.
-
@Gertjan OK, I have now managed to delete those files, de-installed watchdog and followed your other advice. I will see how it goes now and report back.
Thanks again for your patience! -
@Gertjan said in DNSBL SQLite3 database [ lastevent ] corrupt messages in system logs:
so with 119414 entries, it took 15:57-15:22 = already 35 seconds.
What version are you running?, and this is on a 4100 I think you said?
Below is from a crap little test box (my production box a 2100) and has much the same with regards to timing, but has a larger "list" than shown here
Original Matches Removed Final ---------------------------------------- 166351 61104 67425 98926 ----------------------------------------- TLD finalize... completed [ 09/15/24 04:15:10 ] Saving DNSBL statistics... completed Reloading Unbound Resolver (DNSBL python) Stopping Unbound Resolver. Unbound stopped in 2 sec. Additional mounts (DNSBL python): No changes required. Starting Unbound Resolver... completed [ 09/15/24 04:15:12 ] DNSBL update [ 98926 | PASSED ]... completedCertainly no 35 seconds..
Cache you say ?I've never really seen a benefit to "Resolver cache restored" that takes a lot of time.
restoring cache, number of entries in zone are generally not related.Cache works just fine... and any 2 hour sample here will run cache from about 35% to 60% cache hit depending who's doing what. The past 2 hours looks kind of like this.
-
@Klaus2314 said in DNSBL SQLite3 database [ lastevent ] corrupt messages in system logs:
Certainly no 35 seconds..
My pfb_py_zone.txt has 119 414 entries.
Yours has 98 926 entries, somewhat less.before, you showed this :
That's waaaay bigger. Like 68 Mbytes big.
Your pfSEnse has probably bit more 'hose power' as mine.
@Klaus2314 said in DNSBL SQLite3 database [ lastevent ] corrupt messages in system logs:
I've never really seen a benefit to "Resolver cache restored" that takes a lot of time.
The unbound cache - there are command line commands to dump it - isn't that big.
Its just useful to load it back in again. It's a cache after all.My cache : same info, shown differently :
gray = served out of cache.
Other colors : resolving was needed. -
You're confused, that file size you are quoting back is from the OP not me.
I was asking about your 35 second time from your system compared to my unbound and that section of the log where you show and calculate the 35 seconds.
a) yes on my little test box the "list" is a little smaller (process time 2 seconds) I said that.
b) I also said on my 2100 the list is a little larger than yours but the the timing of the unbound restart in that section of the log is similar to my little test box ~ 2-4 secondsThe only difference I can see is that you are restoring the unbound cache, that is where the time difference appears to be from. And I went on to say that I've never seen a benefit to doing that and my cache hit rate is just fine.
All I've said is I've never seen a benefit to restoring resolver cache...and therefore taking the time hit doing so. The cache works just fine here.
-
@jrey said in DNSBL SQLite3 database [ lastevent ] corrupt messages in system logs:
You're confused,
Noop. The @Klaus2314 didn't made into my post - I've edited.
-
Hi all,
since I deleted the files and de-installed watchdog no more errors occurred in the last 24 hours (which included a few cron jobs by pfblocker) so things seem to be fine again.
Oh, and yes, I have been running the python mode before (and still am) on DNSBL.
thanks again for helping.
