Is it Possible to Run the DNSBL UT1_Adult on a Netgate 1100?

mpfrench · Apr 4, 2023, 4:31 PM

Ref: Netgate 1100 running 23.01-RELEASE (arm64).
pfBlocker 3.2.0_3

I cannot get the UT1 DNSBL to run the Adult blocking suite. The reload procedure always is killed without issuing an error message. Here is the log:
20230404_DNSBL_Reload_Log.txt

I'm wondering whether or not the 1100 has enough capability to run this blocking list or whether or not I'm just missing some setting.

Can someone point me in the right direction?

nimrod · Apr 11, 2023, 1:56 AM

@mpfrench said in Is it Possible to Run the DNSBL UT1_Adult on a Netgate 1100?:

Ref: Netgate 1100 running 23.01-RELEASE (arm64).
pfBlocker 3.2.0_3

It works for me. Just tried opening the famous site that ends in "hub" and it was blocked.

I cannot get the UT1 DNSBL to run the Adult blocking suite. The reload procedure always is killed without issuing an error message. Here is the log:
20230404_DNSBL_Reload_Log.txt

I checked the log you attached and i dont see anything "killed" in there. There are no errors in your log.

I'm wondering whether or not the 1100 has enough capability to run this blocking list

1100 has plenty of power to handle that.

or whether or not I'm just missing some setting.

Can someone point me in the right direction?

You probably have a bad configuration somewhere. Make sure your clients are using pfSense as a DNS resolver. If you are using custom DNS like 1.1.1.1 or 8.8.8.8, make sure you have DNS Query Forwarding option enabled in your DNS Resolver settings.

Gertjan · Apr 11, 2023, 7:50 AM

@nimrod said in Is it Possible to Run the DNSBL UT1_Adult on a Netgate 1100?:

I checked the log you attached and i dont see anything "killed" in there. There are no errors in your log.

Line 11.
The download process was killed (stopped).
The 'Killed' message probably comes from PHP.

Other downloads, like :

[ UT1_dangerous_material ] Reload [ 04/4/23 10:44:54 ] . completed ..

have a 'completed' which indicate : download went ok.

As said earlier ( ? ) even for me, using a 4100, downloading this list makes things unstable.

Example :

[23.01-RELEASE][admin@pfSensemysite.tld]/usr: nslookup
> google.com
;; communications error to 127.0.0.1#53: timed out
Server:         127.0.0.1
Address:        127.0.0.1#53

Non-authoritative answer:
Name:   google.com
Address: 142.250.178.142
Name:   google.com
Address: 2a00:1450:4007:80c::200e

=> ;; communications error to 127.0.0.1#53: timed out : WTF ?!

I'm forwarding to 1.1.1.1 so "127.0.0.1" will get skipped, and 1.1.1.1 is happy to answer me.
But : where is unbound on 127.0.0.1 ?? (so no pfBlockerNG functionality neither)

sgw · Apr 14, 2023, 5:41 AM

I am also interested if that's possible. My tests on a SG-3100 failed. Maybe these settings for "table entries" etc would have to be different, I might have not set them back then or set them too low. I could need it on a SG-3100 if that's possible. Would be great to have a positive report before trying that "in production".

Gertjan · Apr 14, 2023, 7:31 AM

@sgw

I didn't insist on having that big edit : UT1_Adult DNSBL (x00 thousands of entries) by manipulating 'buffers' or 'memory' or whatever.

PHP, or for that matter, the way faster Python, shouldn't be used to handle files (lookups into files loaded into structures into memory) of those size for rather time critical DNS lookups. My IMHO of course.

I tried it @home with my 2.6.0 patched : it loaded.
But that pfSense is running a on PC with 16G memory and it's own i5, a overkill solution that uses 15 times more electric power as my 4100 at work.

sgw · Apr 14, 2023, 8:20 AM

@gertjan thanks, understood. This confirms my view and experience.

sgw · Apr 14, 2023, 8:36 AM

On a 7100 with 8GB RAM it should be possible, I assume?
I wonder if I have to adjust the default settings there also:

Firewall Maximum States: 811000
Firewall Maximum Table Entries: 403001

Gertjan · Apr 14, 2023, 10:59 AM

@sgw said in Is it Possible to Run the DNSBL UT1_Adult on a Netgate 1100?:

On a 7100 with 8GB RAM it should be possible, I assume?

As the newer "6100" : 8G also.
And yes, I presume.

SteveITS · Apr 14, 2023, 3:42 PM

@sgw said in Is it Possible to Run the DNSBL UT1_Adult on a Netgate 1100?:

I wonder if I have to adjust the default settings there also:
Firewall Maximum States: 811000
Firewall Maximum Table Entries: 403001

a few notes...

OP had another thread...the PHP process memory limit is by default set at 512 MB on amd64 (Intel compatible) and 128 MB on everything else.

Long ago I picked up somewhere here, IIRC from the package maintainer, to set the table entries at least at 2 million if using pfBlocker. Of course it depends on the number of entries needed so adjust up if necessary.

There is a long standing bug where the text for the "default" table entry size is whatever number you have typed into the field.

provels · Apr 14, 2023, 7:01 PM

FWIW, now that I have real hardware, with 16GB and an i7, I thought I'd try loading the UT-1 adult list. Odd that it doesn't even block Pornhub!

Gertjan · Apr 14, 2023, 10:59 PM

@provels said in Is it Possible to Run the DNSBL UT1_Adult on a Netgate 1100?:

FWIW, now that I have real hardware, with 16GB and an i7, I thought I'd try loading the UT-1 adult list. Odd that it doesn't even block Pornhub!

This time, it seems to work for me :
Selected :

🔒 Log in to view

A save - Reload - and took some time, and I'm sure it wasn't my new Gbit fiber connection.
My 4100 went in overdrive mode again.
Anyway, the process finished, unbound was restarted. Happy end.

Here it is :

🔒 Log in to view

Wt* 4 and a half million entries ....

Show time :

[23.01-RELEASE][admin@pfSense.pain-just-pain.hurt]/root: nslookup
> pornhub.com
;; communications error to 127.0.0.1#53: timed out
Server:         127.0.0.1
Address:        127.0.0.1#53

Non-authoritative answer:
Name:   pornhub.com
Address: 10.10.10.1
;; Got SERVFAIL reply from 127.0.0.1, trying next server

Ok, got a 10.10.10.1.

The last line isn't promising at all.

🔒 Log in to view

Ok, no AAAA for porn....

Last time when I was testing, ifconfig told me that interface lo0, 127.0.0.1 was gone, which is just plain catastrophic.

I'm removing the adult list, this is just to scary.

@provels can you please come over here and explain my wife why I'm posting about this porn site. She doesn't 'trust' my explanation ... (she doesn't read English neither).

edit : You could still visit that site with your browser .... maybe because your browser (and OS, local DNS) didn't need to ask 'pfSense' to resolve the domain. As it already had the answer ....
So that might explain why it still showed up.
Or your device doesn't use pfSense for DNS ?

I tested with nslookup on pfSense, after flushing the local unbound dns cache, and tailed /var/log/var/log/pfblockerng/dns_reply.log to check if pfblockerng was asked to resolve te site, and what it answered.

provels · Apr 14, 2023, 11:53 PM

@gertjan Tell her it's just business. Monkey business...

No, I use pfB and pfS for DNS. I have had the list in effect for about a week, and machines have been rebooted. I'm blocking DoH with several lists, too. Not whitelisted. I think it's the UT1 list... It's only me here so just playing around with lists.
🔒 Log in to view

Gertjan · Apr 15, 2023, 2:47 PM

@provels
🔒 Log in to view

Noop.
Look for

🔒 Log in to view

provels · Apr 15, 2023, 2:48 PM

@gertjan Oh, sure, there's a bunch of those, but I think www. is what folks think of first on this side or the planet!

mpfrench · Apr 17, 2023, 3:33 PM

To summarize this thread, the answer to the question posed in the title is that the 1100 will not run the DNSBL UT1_Adult.

I cannot get my 1100 to load DNSBL UT1_Adult.

@sgw cannot get a 3100 to load UT1_Adult.

@provels can get UT1_Adult to load on an i7 with 16 GB RAM.

@Gertjan can get UT1_Adult to load on a 4100 (4 GB RAM) as well as an i5 with 16 GB RAM.

@Gertjan found that UT1_Adult did not block pornhub.com and the list has 4.5M entries.

My conclusion is that implementing UT1_Adult is not worth the effort on any hardware. Furthermore, if one wants to block adult content, one should forward DNS quires to a "family" DNS service such as one run by OpenDNS[208.67.220.123, 208.67.222.123] or Cloudflare[1.1.1.3, 1.0.0.3], both of which are quite capable of blocking adult content.

provels · Apr 17, 2023, 5:19 PM

@mpfrench Certainly applies to the 1100. The 1100 does what it does well, providing basic protection. But 1GB RAM will never go very far. And I've never seen ANY application or OS get smaller with time. But if the resources are there, there's nothing wrong with running it. And other blocklists for adult sites are available. Also, you can create your own custom lists as needed should something be slipping through. And pfB uses deduplication to help minimize the clutter. But those features take resources without a doubt. Fact is, there are probably several dozen either malicious or adult sites created in the time it's taken me to type this. If you want to set/forget, 3rd party DNS is the way to go. but many in this community are gearheads that ~~enjoy~~ ~~tolerate~~ ~~endure~~ accept the maintenance of their environment. Anyway, sounds like you arrived at the right conclusion. Best luck.

Gertjan · Apr 18, 2023, 6:07 AM

@mpfrench said in Is it Possible to Run the DNSBL UT1_Adult on a Netgate 1100?:

@Gertjan found that UT1_Adult did not block pornhub.com and the list has 4.5M entries.

Correct. The first time I tried, using dnsbl at the end of (huge) list didi resolve to the real IP, not 0.0.0.0 ( I don't forward to 10.10.10.1 = internal pfB web server as https can't be redirected so the browser error will get shown ).
I saw that the complete pfBlockerNG reload / force all didn't terminated with a " UPDATE PROCESS ENDED", it just stopped somewhere in the middle, leaving the process in a undefined state.

Several days later, I redid the test, it took close to 10 minutes to finish, but it did finish this time. Now, it blocked dnsbl from the UT1 list.

Again : using a 4100 with 4 Mbytes of ram.

I share your conclusion.