TLD Domain count exceeded.
-
Hello,
I received the following message (pfsense 23.05.1-RELEASE):
Assembling DNSBL database...... completed [ 08/1/23 21:09:09 ] TLD: TLD analysis........................................xxxxxxxxxxxxxxxxxxx completed [ 08/1/23 21:09:33 ] ** TLD Domain count exceeded. [ 4000000 ] All subsequent Domains listed as-is ** TLD finalize... ---------------------------------------- Original Matches Removed Final ---------------------------------------- 5800359 2956267 836484 4963875 ----------------------------------------- TLD finalize... completed [ 08/1/23 21:10:13 ]It's not clear to me if I have to change anything. I read this topic: https://forum.netgate.com/topic/169369/how-to-increase-tld-domain-count-exceeded-4000000
I checked out this php file and normally with 8 GB of memory the limit of 4000000 should not be applied, but rather a higher limit.
if (!$pfb['dnsbl_py_blacklist']) { $pfb['pfs_mem'] = array( '0' => '100000', '1500' => '150000', '2000' => '200000', '2500' => '250000', '3000' => '400000', '4000' => '600000', '5000' => '1000000', '6000' => '1500000', '7000' => '2000000', '8000' => '2500000', '12000' => '3000000', '16000' => '4000000', '32000' => '8000000'); } else { $pfb['pfs_mem'] = array( '0' => '200000', '1500' => '300000', '2000' => '400000', '2500' => '500000', '3000' => '800000', '4000' => '1200000', '5000' => '2000000', '6000' => '3000000', '7000' => '4000000', '8000' => '5000000', '12000' => '6000000', '16000' => '8000000', '32000' => '16000000'); }Is this a bug?
-
-
-
@Unoptanio please check out https://www.reddit.com/r/pfBlockerNG/comments/15jpbmq/tld_domain_count_exceeded/
-
Extract from /usr/local/pkg/pfblockerng/pfblockerng.inc
// Determine max Domain count available for DNSBL TLD analysis (Avoid Unbound memory exhaustion) $pfs_memory = (round(get_single_sysctl('hw.physmem') / (1024*1024)) ?: 1000); if (!$pfb['dnsbl_py_blacklist']) { $pfb['pfs_mem'] = array( '0' => '100000', '1500' => '150000', '2000' => '200000', '2500' => '250000', '3000' => '400000', '4000' => '600000', '5000' => '1000000', '6000' => '1500000', '7000' => '2000000', '8000' => '2500000', '12000' => '3000000', '16000' => '4000000', '32000' => '8000000'); } else { $pfb['pfs_mem'] = array( '0' => '200000', '1500' => '300000', '2000' => '400000', '2500' => '500000', '3000' => '800000', '4000' => '1200000', '5000' => '2000000', '6000' => '3000000', '7000' => '4000000', '8000' => '5000000', '12000' => '6000000', '16000' => '8000000', '32000' => '16000000'); } foreach ($pfb['pfs_mem'] as $pfb_mem => $domain_max) { if ($pfs_memory >= $pfb_mem) { $pfb['domain_max_cnt'] = $domain_max; } }change "'7000' => '2000000'" and "'7000' => '4000000'" to "'7000' => '6000000'" in both sets.
change "'8000' => '2500000'" and "'8000' => '5000000'" to "'8000' => '6000000'" in both sets.
Update Reload | DNSBL after making these changes.
-
@Unoptanio said in TLD Domain count exceeded.:
Extract from /usr/local/pkg/pfblockerng/pfblockerng.inc
// Determine max Domain count available for DNSBL TLD analysis (Avoid Unbound memory exhaustion) $pfs_memory = (round(get_single_sysctl('hw.physmem') / (1024*1024)) ?: 1000); if (!$pfb['dnsbl_py_blacklist']) { $pfb['pfs_mem'] = array( '0' => '100000', '1500' => '150000', '2000' => '200000', '2500' => '250000', '3000' => '400000', '4000' => '600000', '5000' => '1000000', '6000' => '1500000', '7000' => '2000000', '8000' => '2500000', '12000' => '3000000', '16000' => '4000000', '32000' => '8000000'); } else { $pfb['pfs_mem'] = array( '0' => '200000', '1500' => '300000', '2000' => '400000', '2500' => '500000', '3000' => '800000', '4000' => '1200000', '5000' => '2000000', '6000' => '3000000', '7000' => '4000000', '8000' => '5000000', '12000' => '6000000', '16000' => '8000000', '32000' => '16000000'); } foreach ($pfb['pfs_mem'] as $pfb_mem => $domain_max) { if ($pfs_memory >= $pfb_mem) { $pfb['domain_max_cnt'] = $domain_max; } }change "'7000' => '2000000'" and "'7000' => '4000000'" to "'7000' => '6000000'" in both sets.
change "'8000' => '2500000'" and "'8000' => '5000000'" to "'8000' => '6000000'" in both sets.
Update Reload | DNSBL after making these changes.
@BBcan177 I run a Netgate 6100 Max and unfortunately every pfBlockerNG update requires me to re-edit these values.
Is there any chance these values could get increased more permanently? Or perhaps a UI option that allows tweaking the value according to a user's specific RAM utilisation rather than these rough estimates? I run pfBlockerNG, Snort, ZabbixAgent6 and Wireguard packages without breaking a sweat on this 6100 Max with 8GB. TLD count is:Original: 6786434
Matches: 5001323
Removed: 1184774
Final: 5601660That said, if I'm way off here and doing something wrong please do set me straight!
Thanks -
@Squuiid do you use Python mode or Unbound mode? I will see. Thanks.
-
@BBcan177 Thanks for the quick reply! Python mode.
-
-
Resurrecting this thread for two reasons:
1.) Because this is where I landed when newly confronted with the topic using
pfBlockerNG-devel 3.2.10onpfSense CE 2.8.1-RELEASE; and
2.) to confirm that the 'issue' and 'fix' here continue to be viable despite the TLD analysis function being considerably modified since the last post in September 2024.Current function @ L7255 of
/usr/local/pkg/pfblockerng/pfblockerng.inc:// Determine max Domain count available for DNSBL TLD analysis (Avoid Unbound memory exhaustion) $pfs_memory = (round(get_single_sysctl('hw.physmem') / (1024*1024)) ?: 1000); $pfb['pfs_mem'] = [ '0' => '100000', '1500' => '150000', '2000' => '200000', '2500' => '250000', '3000' => '400000', '4000' => '600000', '5000' => '1000000', '6000' => '1500000', '7000' => '2000000', '8000' => '2500000', '12000' => '3000000', '16000' => '4000000', '32000' => '8000000' ]; if ($pfb['dnsbl_py_blacklist']) { array_walk($pfb['pfs_mem'], function (&$value) { $value = $value * 3; }); } foreach ($pfb['pfs_mem'] as $pfb_mem => $domain_max) { if ($pfs_memory >= $pfb_mem) { $pfb['domain_max_cnt'] = $domain_max; } }On a system with 32 GB of RAM attempting to 'analyze' over 24M but less than 27M domains, the line "
'32000' => '8000000'" was changed to "'32000' => '9000000'" (i.e., permitting a maximum number of 27M domains to be 'analyzed') in order for the function to complete successfully.Analyzing (and then subsequently loading) precisely this number of domains...
Original Matches Removed Final ---------------------------------------- 24270656 21017552 6463516 17807140 -----------------------------------------...results in Unbound's stable operational consumption of ~6 GB of RAM and any/all pfBlockerNG 'Reload' options consuming as much as ~6 GB of RAM, concurrently. Therefore one should only attempt this DNSBL hack if they're confident that their system has at least 13 GB of memory 'headroom' (taking into account normal system operation and any other resource-consuming, installed packages).
