DNSBL Enable TLD RAM/freezing issues

[sjtorrie]

Is anyone else using DNSBL with with TLD enabled using lists from http://www.squidblacklist.org/ and having RAM/freezing on CRON update issues?

Squidblacklist pulls in a hearty 1947496 entries and everything is fine without TLD enabled - however, we really need TLD enabled for it to be an effective webfilter. Once enabled (with 8GB RAM) it struggles to do an initial CRON and then freezes constantly.

If I ramp it up to 10GB then it's stable, but with very high RAM use, but when manual/scheduled CRON runs it freezes up again, all users are locked out from the internet and I have to reboot the device.

I REALLY want to use this for filtering and I don't really want to use a smaller list - We've considered using squidguard as a filtered webproxy instead but it doesn't seem to be as advanced, and we lose all the IP list functons (we're using FireHOL to lockdown on this level)

(pfSense 2.4.1 & pfbockerng 2.1.2_1)

[BBcan177]

Unbound creates a pointer in memory for each "redirect" zone and this is why it uses more memory.

Keep in mind that the pkg will do a validation of the database after each cron event and it will require memory also for that purpose…. So initially it loads ok, but when cron runs, Unbound is already using quite a bit of memory and you need that much more for the validation process...

So you will need to bump the memory in the box to be able to use 2M domains... Not much I can do about that... I have worked with the Unbound devs but so far there is no change on how Unbound loads these domains into memory.

[f34rinc]

Another idea would be to use 3rd party DNS filtering for the TLD blocking.

[sjtorrie]

@f34rinc:

Another idea would be to use 3rd party DNS filtering for the TLD blocking.

I'm all ears for suggestions on what 3rd parties might be out there(?!)… ideally I want to keep everything on a single pfSense VM for each internet breakout > in the mean time I'll push the RAM a little higher - unfortunately the hosts I have out and about in our branch sites are somewhat limited...

[StyleNZ]

@BBcan177 .

Is this thread in particular in relation to this post?

https://www.reddit.com/r/PFSENSE/comments/bmmf7a/high_ram_usage/?utm_medium=android_app&utm_source=share

'm running into lock ups particulatly on updates/reloads.

One thing I don't currently have enabled is SWAP. If I managed to create a 4GB or so swap, could I thriretically resolve this issue and potentially add more lists?

Regards

[StyleNZ]

@sjtorrie

I managed to add a SWAP to my install and this has seemed to of fixed my issues. I know this is a dated post but this may resolve your/others issues of locking up and the potential of using more DNSBLs.

Regards