2100 cannot load the UT1 adult table.
-
[ UT1_adult ] Downloading update .
Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 122618696 bytes) in /usr/local/pkg/pfblockerng/pfblockerng.inc on line 3962
PHP ERROR: Type: 1, File: /usr/local/pkg/pfblockerng/pfblockerng.inc, Line: 3962, Message: Allowed memory size of 134217728 byWhat do do please? I just want to block adult sites from the customers kids! Ideally sing the tool that is meant to be able to do it!
-
Maybe silly, but the answer explained the issue.
'PHP' is the thing that produces web pages, so your browser can show you something.
But it can also do things with files, do math, or even hosting game.
Its a language , and php itself is an interpreter of this language.
As such, it uses some system memory of its own needs.
On your system, it took 134217728 bytes (134 217 728 = about 134 Mbytes) to work with.
And no, it can take 'every byte up until the last', as this will kill your system (FreeBSD, the OS).But : it needed an extra 122 618 696 bytes or 122 Mbytes from this PHP 134 Mbytes memory pool to do what is had to do with this UT1_adult (marked as file it just downloaded.
Files (DNSBL feeds) downloaded by pfBlockerng need to be parsed, converted, sorted and assembled in one big file, so it can by used by the resolver (unbound) as one big lookup table, so see which host name should not pass, thus 'blocked'.The thing is, and you get it by now : with big files you need a lots (loads) of memory.
Worse : PHP memory is determined during PHP start-up, and related to the available, real total memory (RAM).
So, yes, with a 16 Gbytes RAM router PHP gets probably 15 % or so, that's 2 Gbytes.
And now, no surprise, it passes.Netgate doesn't even propose routers with that kind of memory (have to check).
Read also : Is it Possible to Run the DNSBL UT1_Adult on a Netgate 1100?
So, why is this list listed then ?
Well, in the beginning, as always, the list wasn't big, for some reason, it grew.What you could do : change something somewhere, so PHP gets more memory allocated when it starts. That's what I did to force my little 4 GBytes 4100-MAX to accept this file.
But that comes with ... risks, and that's ok for me, as I won't complain here when things go south. I'll know "I did it".
I finally removed the list, as it didn't had big scores : it didn't block close to nothing, and I was filtering a public wifi hot spot - a hotel.Btw :
"Large" in this context was a message for the pfSense admin and has no physical meaning
-
@Gertjan Thanks for that. So how would you filter porn out then? Customer has requested it
-
@orangehand
Blocking porn is a battle but if you are dealing with a resource constraint then its best to maybe use a Rasberry Pi [or another low-power device you have ] install PiHole or Adguard home to be your DNS server and start loading blocklists there.
Similar to what i did where i have Adguard home running and its used for a specific VLAN and that has all the blocklists i want without harming my firewall which is weak on cpu and memory. -
@michmoor Rather depressing that I bought a Ryzen mini box with Win 11, 32Gb and 1Tb SSD for pretty much the same money as they charge for this puny piece of junk. Also on the basis that every 2100 we have deployed has had to have an SSD installed in it to cope at all. I know pfSense has to be funded, but what they deliver hardware-wise for the money is a bit of a joke.
-
@orangehand I get it...
But yeah the memory requirement for pretty much any porn blocklist is pretty heavy.
This is why when anyone wants a recommendation on hardware the lowest i will suggest is the 4100. In my mind i cant justify the prices for the lower tier but thats just me.
If you really want to use UT1 or any porn blocklist - set up a custom build or start looking at the 4100.edit: To Netgates defense, its difficult sizing up hardware when you dont know the purpose/role it will play in someone's network. SG-2100 is perfectly fine if you have some remote access VPNs and some vlans. Its impossible to know what blocklist an admin will use and to build around that is just an impossible task. Some vendors do handle things a bit better when they give out a spec sheet and they tell you whats the expected throughput if you enable certain services. IMO, a better job needs to be done in the marketing pages for Netgates product line to help people make better buying decisions but thats just me. Perhaps a note stating that "URL Filtering using large blocklists are not recommended" or to that effect.
Thats why it's always best to over-spec. Do you need 4GB today? Maybe not but you have to anticipate the future so maybe you have to spring for the 8GB model.
Also, pfsense can be loaded with white box hardware which in your case i highly recommend as you can most likely get more bang for the buck. -
Console or SSH into your pfSense.
First, goto here /usr/local/www/
Create this file :<?php phpinfo(); ?>Now, visit the GUI of your pfSense, use the URL and 'visit' this file :
http://192.168.1.1/phpinfo.php(you just wrote your first web page on a web server
)Look for "memory_limit" :
Now, edit the file /etc/rc.php_ini_setup
Add the line
memory_limit = 256Mas indicated :
On the console main menu, use option 16 - Restart PHP.
Check http://192.168.1.1/phpinfo.php again : memory_limit is now 256 ?
If so : you know what to do to get back to the original settings.
Keep an eye on your system.The initial (for me) 128M was chosen by Netgate for a reason.
From now on : look for swap usage : if that happens, go back as swapping is a performance killer.
This worked 'better' for me, and I was using a 4100 which has 4 Gbytes.
PHP is a programming language which was meant to create dynamic (html) web pages. Handling big, no, huge (text) files was never a strong point of any high level interpreted language.
I hope that bbcan77 will adopt the language, C if possible, so the heavy file sort parse assembly can be done using the full processor power. It will still be 'hard' to do multi hundreds of mega file size handling in a system with 'no' memory (2 Gbytes is nothing these days - the kernel and other basic tools also need a place in RAM). -
@orangehand It's all about RAM. The 2100 seems to have 4GB RAM. But you need at least 8GB RAM to use the UT1_Adult list. Is it possible to upgrade the RAM?
If not, you need to tweak the PHP limit first (see @Gertjan post).
After that, here are some ideas to tweak the pfblockerng settings.
- Use python mode. This mode needs a lot less resources (I think 1/2)
- Increase the Firewall Maximum Table Entries under System/Advanced/Firewall & NAT
- Increase the pfblockerng memory limit which is a bit conservative.
You can find this limit in the file "/usr/local/pkg/pfblockerng/pfblockerng.inc" around line #7250.
The second part is for python mode:
// Determine max Domain count available for DNSBL TLD analysis (Avoid Unbound memory exhaustion) $pfs_memory = (round(get_single_sysctl('hw.physmem') / (1024*1024)) ?: 1000); if (!$pfb['dnsbl_py_blacklist']) { $pfb['pfs_mem'] = array( '0' => '100000', '1500' => '150000', '2000' => '200000', '2500' => '250000', '3000' => '400000', '4000' => '600000', '5000' => '1000000', '6000' => '1500000', '7000' => '2000000', '8000' => '2500000', '12000' => '3000000', '16000' => '4000000', '32000' => '8000000'); } else { $pfb['pfs_mem'] = array( '0' => '200000', '1500' => '300000', '2000' => '400000', '2500' => '500000', '3000' => '800000', '4000' => '1200000', '5000' => '2000000', '6000' => '3000000', '7000' => '4000000', '8000' => '5000000', '12000' => '6000000', '16000' => '8000000', '32000' => '16000000');To use the UT1_adult list with 8GB RAM I had to increase the value for 7GB and for 8GB, as in some installations 8GB RAM is recognized as 7963MB RAM.
I changed both values to 6000000:
'7000' => '6000000', '8000' => '6000000',With the UT1_adult and some other lists (ADs, Hacking, Phishing, EasyList, etc) and the "Wildcard Blocking (TLD)" feature on, the final count of domains is around 5 million.
Assembling DNSBL database...... completed [ 10/13/23 00:02:00 ] TLD: TLD analysis.................................................... completed [ 10/13/23 00:02:18 ] TLD finalize... ---------------------------------------- Original Matches Removed Final ---------------------------------------- 5117194 4515592 150573 4966621 ----------------------------------------- TLD finalize... completed [ 10/13/23 00:03:00 ] Saving DNSBL statistics... completed [ 10/13/23 00:03:06 ]While updating the list, memory usage is up to 60%. After that it's max 25% of my 8GB.
As one last thought, you can also use some smaller lists as a custom list. For example the StevenBlack porn list from here:
https://github.com/StevenBlack/hostsThis list should also include most adult sites (~ 54.000 unique domains).
-
@mOrbo said in 2100 cannot load the UT1 adult table.:
Is it possible to upgrade the RAM?
Euh ... no way.
There is the SMD hurdle : soldering on a sub millimeter scale is hard.@mOrbo said in 2100 cannot load the UT1 adult table.:
Use python mode. This mode needs a lot less resources (I think 1/2)
True.
But that economy happens when unbound is "told" to use the one and only main master DNSBL file, assembled from all your selected feeds.
The old method was : have unbound load this huge (?) text file upon startup ... don't be surprised that it took tens of seconds, or more. An dduring this time : no DNS.
The new method : unbound is told to use a script file : a python written script file that contains functions that unbound calls for most DNS lookup needs. This function can now interact upon the DNS request. The python scrip uses the file mentioned already above.
This method uses less memory (not sure) and loads much faster.Keep in mind that pfBlockerng doesn't do much when you use it.
It will "tail" the log files produced by unbound and/or the unbound script, to create nice usage stats. And that's it.Another part is the CRON based : it will scan your selected DNSBL and IP feeds, and 'check' if they needed to be updated. If so, they will be downloaded, and a new master IP and DNSBL file is created. This is where the big memory usage is happening.
Most feeds aren't updated every 5 minutes, every hour or even every day.
That's why I use :
that is ones a day.
The servers that host the feeds will thank me.@mOrbo said in 2100 cannot load the UT1 adult table.:
Increase the Firewall Maximum Table Entries under System/Advanced/Firewall & NAT
Yep, if you use big IP feeds, this will be needed.
And think about it : every packet that creates a new state in the firewall need to be parsed against all the IPs in the master IP alias table.
If possible, don't use this alias on the WAN interface. The very first mini doss you receive on your WAN will cripple (or worse) your firewall. Use the default "drop all" rule. Don't try to filter the net, you'll lose.@mOrbo said in 2100 cannot load the UT1 adult table.:
the final count of domains is around 5 million.
Yeah ! and that makes the question easy to answer : with about 15 or so characters for every domain name, that makes .... how big the final file ?
The RAM question is over now.
Lets check the HHD or SSD disk space@mOrbo said in 2100 cannot load the UT1 adult table.:
This list should also include most adult sites (~ 54.000 unique domains).
That's the one I'm using :
-
FWIW, when I used the UT1 adult category, even pornhub was not blocked.
When I added the Steven Black list it was. It's the default list for pi-hole and it works very well.
https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts -
@orangehand If you want to outsource it you can set DNS Resolver to forward to CloudFlare or others that block adult sites (1.1.1.3).
https://blog.cloudflare.com/introducing-1-1-1-1-for-families/
