TLD Domain count exceeded

Unoptanio · Nov 5, 2023, 6:06 PM

Pfsense 2.7.0
pfBlockerNG-devel 3.2.0_6
RAM 8Gbyte

Hi, how do I increase the value?

Assembling DNSBL database...... completed [ 11/5/23 18:59:34 ]
TLD:
TLD analysis........................................xxxxxxxxx completed [ 11/5/23 19:00:00 ]

** TLD Domain count exceeded. [ 4000000 ] All subsequent Domains listed as-is **

🔒 Log in to view

Gertjan · Nov 8, 2023, 1:43 PM

@Unoptanio said in TLD Domain count exceeded:

Hi, how do I increase the value?

pfBlockerng is nothing more (neither less) as a lot of PHP scripts.
And PHP hasn't "all system RAM available", but far less.

Or you, (indirectly, I get it) asked pfBLcokerNG == the PHP scripts, to do something that asked far more then it can handle : merging, sorting, removing the doubles of 4,8 million host names will ..... well, it didn't explode but it just stopped doing some house keeping, and building the main DNSBL list "as is".

To remove " TLD Domain count exceeded", there is one easy way out : use less DNSBL (feeds).

Keep in mind that, for every DNS request (coming from your LANs) unbound (the resolver) has to parse this entire DNSBL list so it can look up for a match.

Unoptanio · Nov 7, 2023, 4:28 PM

@Gertjan

ok.
At the moment I have not detected any slowness in DNS resolution
Could it be a solution to increase the RAM of my system? Or does PHP see at most a certain fixed maximum amount of RAM?

If I increase the system RAM to 32GB, will PHP be able to use more RAM than now?

SteveITS · Nov 7, 2023, 6:13 PM

@Unoptanio Per the blue (i) icon in pfBlocker here:
🔒 Log in to view

"Once the TLD Domain limit below is exceeded, the balance of the Domains will be listed as-is. IE: Blocking only the listed Domain (Not Sub-Domains)
TLD Domain Limit Restrictions:

< 1.0GB RAM - Max 100k Domains
< 1.5GB RAM - Max 150k Domains
< 2.0GB RAM - Max 200k Domains
< 2.5GB RAM - Max 250k Domains
< 3.0GB RAM - Max 400k Domains
< 4.0GB RAM - Max 600k Domains
< 5.0GB RAM - Max 1.0M Domains
< 6.0GB RAM - Max 1.5M Domains
< 7.0GB RAM - Max 2.5M Domains
> 7.0GB RAM - > 2.5M Domains"

...so at a limit of 4 million I guess you have a lot of RAM. :) Without looking into the code I guess you can try it? I would have read that list as it stops at 2.5 million.

Unoptanio · Nov 8, 2023, 8:48 AM

@SteveITS

🔒 Log in to view

Unoptanio · Nov 8, 2023, 8:49 AM

This post is deleted!

Unoptanio · Nov 8, 2023, 8:52 AM

@SteveITS

🔒 Log in to view

SteveITS · Nov 8, 2023, 1:56 PM

@Unoptanio re: "table-entries hard limit", what is your setting of "System > Advanced > Firewall & NAT > Firewall Maximum Table Entries"? Looks like you'd need it to be at least 4.9 million to fit 4806104 entries.

(note pfSense has a longstanding bug where the sentence "On this system the default size is: ___" always shows whatever number you've entered, if you've entered a custom number)

Unoptanio · Nov 8, 2023, 2:08 PM

@SteveITS

🔒 Log in to view

SteveITS · Nov 8, 2023, 2:12 PM

@Unoptanio Assuming your router has enough RAM, change the 400,000 to a higher number.

Actually per https://docs.netgate.com/pfsense/en/latest/config/advanced-firewall-nat.html#firewall-maximum-table-entries it says twice the number you need so I guess 9 million for you.

Unoptanio · Nov 8, 2023, 2:14 PM

@SteveITS

now i have only 400.000

i try to change in 9.000.000 ??

Unoptanio · Nov 8, 2023, 2:43 PM

@Unoptanio

I did a test now with a value of Firewall Maximum Table Entries 6,000,000 but it didn't solve the problem.

The number of X is always the same

🔒 Log in to view

SteveITS · Nov 8, 2023, 2:59 PM

@Unoptanio Not sure what to tell you. According to the pfBlocker directions I posted above, over 7 GB RAM should be limited to 2.5 million domains. You may need to find the code in pfBlocker that is setting the limit to 4 million and/or add RAM to your router.

Unoptanio · Nov 8, 2023, 3:06 PM

@SteveITS
@BBcan177

Proverò ad acquistare altra ram

🔒 Log in to view

I will try to buy more RAM
my current ram consumption is around 50%

Unoptanio · Nov 10, 2023, 9:58 PM

@SteveITS

Resolved

Extract from /usr/local/pkg/pfblockerng/pfblockerng.inc

// Determine max Domain count available for DNSBL TLD analysis (Avoid Unbound memory exhaustion)
	$pfs_memory = (round(get_single_sysctl('hw.physmem') / (1024*1024)) ?: 1000);

	if (!$pfb['dnsbl_py_blacklist']) {
		$pfb['pfs_mem'] = array(   '0' => '100000', '1500' =>  '150000', '2000' =>  '200000', '2500' =>  '250000', '3000' =>  '400000',
					'4000' => '600000', '5000' => '1000000', '6000' => '1500000', '7000' => '2000000', '8000' => '2500000',
					'12000' => '3000000', '16000' => '4000000', '32000' => '8000000');
	} else {
		$pfb['pfs_mem'] = array(   '0' => '200000', '1500' =>  '300000', '2000' =>  '400000', '2500' =>  '500000', '3000' =>  '800000',
					'4000' => '1200000', '5000' => '2000000', '6000' => '3000000', '7000' => '4000000', '8000' => '5000000',
					'12000' => '6000000', '16000' => '8000000', '32000' => '16000000');
	}

	foreach ($pfb['pfs_mem'] as $pfb_mem => $domain_max) {
		if ($pfs_memory >= $pfb_mem) {
			$pfb['domain_max_cnt'] = $domain_max;
		}
	}

change "'7000' => '2000000'" and "'7000' => '4000000'" to "'7000' => '6000000'" in both sets.

change "'8000' => '2500000'" and "'8000' => '5000000'" to "'8000' => '6000000'" in both sets.

Update Reload | DNSBL after making these changes.

🔒 Log in to view