TLD Domain count exceeded.

float

Hello,

I received the following message (pfsense 23.05.1-RELEASE):

Assembling DNSBL database...... completed [ 08/1/23 21:09:09 ]
TLD:
TLD analysis........................................xxxxxxxxxxxxxxxxxxx completed [ 08/1/23 21:09:33 ]

  ** TLD Domain count exceeded. [ 4000000 ] All subsequent Domains listed as-is **

TLD finalize...
 ----------------------------------------
 Original    Matches    Removed    Final     
 ----------------------------------------
 5800359     2956267    836484     4963875   
 -----------------------------------------
TLD finalize... completed [ 08/1/23 21:10:13 ]

It's not clear to me if I have to change anything. I read this topic: https://forum.netgate.com/topic/169369/how-to-increase-tld-domain-count-exceeded-4000000

I checked out this php file and normally with 8 GB of memory the limit of 4000000 should not be applied, but rather a higher limit.

if (!$pfb['dnsbl_py_blacklist']) {
		$pfb['pfs_mem'] = array(   '0' => '100000', '1500' =>  '150000', '2000' =>  '200000', '2500' =>  '250000', '3000' =>  '400000',
					'4000' => '600000', '5000' => '1000000', '6000' => '1500000', '7000' => '2000000', '8000' => '2500000',
					'12000' => '3000000', '16000' => '4000000', '32000' => '8000000');
	} else {
		$pfb['pfs_mem'] = array(   '0' => '200000', '1500' =>  '300000', '2000' =>  '400000', '2500' =>  '500000', '3000' =>  '800000',
					'4000' => '1200000', '5000' => '2000000', '6000' => '3000000', '7000' => '4000000', '8000' => '5000000',
					'12000' => '6000000', '16000' => '8000000', '32000' => '16000000');
	}

Is this a bug?

Unoptanio

@float

Hi, I have the same problem.
Do you have any news? how to solve?

Unoptanio

@float

????

float

@Unoptanio please check out https://www.reddit.com/r/pfBlockerNG/comments/15jpbmq/tld_domain_count_exceeded/

Unoptanio

@float

Extract from /usr/local/pkg/pfblockerng/pfblockerng.inc

// Determine max Domain count available for DNSBL TLD analysis (Avoid Unbound memory exhaustion)
	$pfs_memory = (round(get_single_sysctl('hw.physmem') / (1024*1024)) ?: 1000);

	if (!$pfb['dnsbl_py_blacklist']) {
		$pfb['pfs_mem'] = array(   '0' => '100000', '1500' =>  '150000', '2000' =>  '200000', '2500' =>  '250000', '3000' =>  '400000',
					'4000' => '600000', '5000' => '1000000', '6000' => '1500000', '7000' => '2000000', '8000' => '2500000',
					'12000' => '3000000', '16000' => '4000000', '32000' => '8000000');
	} else {
		$pfb['pfs_mem'] = array(   '0' => '200000', '1500' =>  '300000', '2000' =>  '400000', '2500' =>  '500000', '3000' =>  '800000',
					'4000' => '1200000', '5000' => '2000000', '6000' => '3000000', '7000' => '4000000', '8000' => '5000000',
					'12000' => '6000000', '16000' => '8000000', '32000' => '16000000');
	}

	foreach ($pfb['pfs_mem'] as $pfb_mem => $domain_max) {
		if ($pfs_memory >= $pfb_mem) {
			$pfb['domain_max_cnt'] = $domain_max;
		}
	}

change "'7000' => '2000000'" and "'7000' => '4000000'" to "'7000' => '6000000'" in both sets.

change "'8000' => '2500000'" and "'8000' => '5000000'" to "'8000' => '6000000'" in both sets.

Update Reload | DNSBL after making these changes.

Squuiid

@Unoptanio said in TLD Domain count exceeded.:

@float

Extract from /usr/local/pkg/pfblockerng/pfblockerng.inc

// Determine max Domain count available for DNSBL TLD analysis (Avoid Unbound memory exhaustion)
	$pfs_memory = (round(get_single_sysctl('hw.physmem') / (1024*1024)) ?: 1000);

	if (!$pfb['dnsbl_py_blacklist']) {
		$pfb['pfs_mem'] = array(   '0' => '100000', '1500' =>  '150000', '2000' =>  '200000', '2500' =>  '250000', '3000' =>  '400000',
					'4000' => '600000', '5000' => '1000000', '6000' => '1500000', '7000' => '2000000', '8000' => '2500000',
					'12000' => '3000000', '16000' => '4000000', '32000' => '8000000');
	} else {
		$pfb['pfs_mem'] = array(   '0' => '200000', '1500' =>  '300000', '2000' =>  '400000', '2500' =>  '500000', '3000' =>  '800000',
					'4000' => '1200000', '5000' => '2000000', '6000' => '3000000', '7000' => '4000000', '8000' => '5000000',
					'12000' => '6000000', '16000' => '8000000', '32000' => '16000000');
	}

	foreach ($pfb['pfs_mem'] as $pfb_mem => $domain_max) {
		if ($pfs_memory >= $pfb_mem) {
			$pfb['domain_max_cnt'] = $domain_max;
		}
	}

change "'7000' => '2000000'" and "'7000' => '4000000'" to "'7000' => '6000000'" in both sets.

change "'8000' => '2500000'" and "'8000' => '5000000'" to "'8000' => '6000000'" in both sets.

Update Reload | DNSBL after making these changes.

@BBcan177 I run a Netgate 6100 Max and unfortunately every pfBlockerNG update requires me to re-edit these values.
Is there any chance these values could get increased more permanently? Or perhaps a UI option that allows tweaking the value according to a user's specific RAM utilisation rather than these rough estimates? I run pfBlockerNG, Snort, ZabbixAgent6 and Wireguard packages without breaking a sweat on this 6100 Max with 8GB. TLD count is:

Original: 6786434
Matches: 5001323
Removed: 1184774
Final: 5601660

That said, if I'm way off here and doing something wrong please do set me straight!
Thanks

BBcan177

@Squuiid do you use Python mode or Unbound mode? I will see. Thanks.

Squuiid

@BBcan177 Thanks for the quick reply! Python mode.

Unoptanio

@Squuiid @BBcan177
Just me:
Unbound python mode

but:
I increased my system's RAM to 32GB

my values:

I increased my system's RAM to 32GB