Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    TLD Domain count exceeded

    Scheduled Pinned Locked Moved pfBlockerNG
    15 Posts 3 Posters 1.9k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • GertjanG
      Gertjan @Unoptanio
      last edited by Gertjan

      @Unoptanio said in TLD Domain count exceeded:

      Hi, how do I increase the value?

      pfBlockerng is nothing more (neither less) as a lot of PHP scripts.
      And PHP hasn't "all system RAM available", but far less.

      Or you, (indirectly, I get it) asked pfBLcokerNG == the PHP scripts, to do something that asked far more then it can handle : merging, sorting, removing the doubles of 4,8 million host names will ..... well, it didn't explode but it just stopped doing some house keeping, and building the main DNSBL list "as is".

      To remove " TLD Domain count exceeded", there is one easy way out : use less DNSBL (feeds).

      Keep in mind that, for every DNS request (coming from your LANs) unbound (the resolver) has to parse this entire DNSBL list so it can look up for a match.

      No "help me" PM's please. Use the forum, the community will thank you.
      Edit : and where are the logs ??

      UnoptanioU 1 Reply Last reply Reply Quote 1
      • UnoptanioU
        Unoptanio @Gertjan
        last edited by Unoptanio

        @Gertjan

        ok.
        At the moment I have not detected any slowness in DNS resolution
        Could it be a solution to increase the RAM of my system? Or does PHP see at most a certain fixed maximum amount of RAM?

        If I increase the system RAM to 32GB, will PHP be able to use more RAM than now?

        S 1 Reply Last reply Reply Quote 0
        • S
          SteveITS Galactic Empire @Unoptanio
          last edited by

          @Unoptanio Per the blue (i) icon in pfBlocker here:
          44e34053-07a7-49a3-9a47-56a5f4079814-image.png

          "Once the TLD Domain limit below is exceeded, the balance of the Domains will be listed as-is. IE: Blocking only the listed Domain (Not Sub-Domains)
          TLD Domain Limit Restrictions:

          < 1.0GB RAM - Max 100k Domains
          < 1.5GB RAM - Max 150k Domains
          < 2.0GB RAM - Max 200k Domains
          < 2.5GB RAM - Max 250k Domains
          < 3.0GB RAM - Max 400k Domains
          < 4.0GB RAM - Max 600k Domains
          < 5.0GB RAM - Max 1.0M Domains
          < 6.0GB RAM - Max 1.5M Domains
          < 7.0GB RAM - Max 2.5M Domains
          > 7.0GB RAM - > 2.5M Domains"
          

          ...so at a limit of 4 million I guess you have a lot of RAM. :) Without looking into the code I guess you can try it? I would have read that list as it stops at 2.5 million.

          Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
          When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
          Upvote πŸ‘ helpful posts!

          UnoptanioU 2 Replies Last reply Reply Quote 0
          • UnoptanioU
            Unoptanio @SteveITS
            last edited by

            @SteveITS

            c7cf026e-ddfc-4ba4-a27b-029b2e6f0aee-image.png

            UnoptanioU S 2 Replies Last reply Reply Quote 0
            • UnoptanioU
              Unoptanio @Unoptanio
              last edited by

              This post is deleted!
              1 Reply Last reply Reply Quote 0
              • UnoptanioU
                Unoptanio @SteveITS
                last edited by

                @SteveITS

                5678eddb-2574-45bc-a5e6-d89a1c6500ab-image.png

                6838c7e7-0f0b-4e54-ad51-e75bfe42e61d-image.png

                1 Reply Last reply Reply Quote 0
                • S
                  SteveITS Galactic Empire @Unoptanio
                  last edited by

                  @Unoptanio re: "table-entries hard limit", what is your setting of "System > Advanced > Firewall & NAT > Firewall Maximum Table Entries"? Looks like you'd need it to be at least 4.9 million to fit 4806104 entries.

                  (note pfSense has a longstanding bug where the sentence "On this system the default size is: ___" always shows whatever number you've entered, if you've entered a custom number)

                  Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                  When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                  Upvote πŸ‘ helpful posts!

                  UnoptanioU 1 Reply Last reply Reply Quote 0
                  • UnoptanioU
                    Unoptanio @SteveITS
                    last edited by

                    @SteveITS

                    44b53cc1-7061-45b1-bf34-5abfc983f388-image.png

                    S 1 Reply Last reply Reply Quote 0
                    • S
                      SteveITS Galactic Empire @Unoptanio
                      last edited by

                      @Unoptanio Assuming your router has enough RAM, change the 400,000 to a higher number.

                      Actually per https://docs.netgate.com/pfsense/en/latest/config/advanced-firewall-nat.html#firewall-maximum-table-entries it says twice the number you need so I guess 9 million for you.

                      Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                      When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                      Upvote πŸ‘ helpful posts!

                      UnoptanioU 1 Reply Last reply Reply Quote 0
                      • UnoptanioU
                        Unoptanio @SteveITS
                        last edited by

                        @SteveITS

                        now i have only 400.000

                        i try to change in 9.000.000 ??

                        UnoptanioU 1 Reply Last reply Reply Quote 0
                        • UnoptanioU
                          Unoptanio @Unoptanio
                          last edited by Unoptanio

                          @Unoptanio

                          I did a test now with a value of Firewall Maximum Table Entries 6,000,000 but it didn't solve the problem.

                          The number of X is always the same

                          a5935726-dc63-4821-b44b-962f6dc05d2a-image.png

                          S 1 Reply Last reply Reply Quote 0
                          • S
                            SteveITS Galactic Empire @Unoptanio
                            last edited by

                            @Unoptanio Not sure what to tell you. According to the pfBlocker directions I posted above, over 7 GB RAM should be limited to 2.5 million domains. You may need to find the code in pfBlocker that is setting the limit to 4 million and/or add RAM to your router.

                            Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                            When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                            Upvote πŸ‘ helpful posts!

                            UnoptanioU 1 Reply Last reply Reply Quote 1
                            • S SteveITS referenced this topic on
                            • UnoptanioU
                              Unoptanio @SteveITS
                              last edited by Unoptanio

                              @SteveITS
                              @BBcan177

                              ProverΓ² ad acquistare altra ram

                              f54acef1-7e7b-428c-b89d-95e459421d5d-image.png

                              I will try to buy more RAM
                              my current ram consumption is around 50%

                              UnoptanioU 1 Reply Last reply Reply Quote 0
                              • UnoptanioU
                                Unoptanio @Unoptanio
                                last edited by Unoptanio

                                @SteveITS

                                Resolved

                                Extract from /usr/local/pkg/pfblockerng/pfblockerng.inc

                                // Determine max Domain count available for DNSBL TLD analysis (Avoid Unbound memory exhaustion)
                                	$pfs_memory = (round(get_single_sysctl('hw.physmem') / (1024*1024)) ?: 1000);
                                
                                	if (!$pfb['dnsbl_py_blacklist']) {
                                		$pfb['pfs_mem'] = array(   '0' => '100000', '1500' =>  '150000', '2000' =>  '200000', '2500' =>  '250000', '3000' =>  '400000',
                                					'4000' => '600000', '5000' => '1000000', '6000' => '1500000', '7000' => '2000000', '8000' => '2500000',
                                					'12000' => '3000000', '16000' => '4000000', '32000' => '8000000');
                                	} else {
                                		$pfb['pfs_mem'] = array(   '0' => '200000', '1500' =>  '300000', '2000' =>  '400000', '2500' =>  '500000', '3000' =>  '800000',
                                					'4000' => '1200000', '5000' => '2000000', '6000' => '3000000', '7000' => '4000000', '8000' => '5000000',
                                					'12000' => '6000000', '16000' => '8000000', '32000' => '16000000');
                                	}
                                
                                	foreach ($pfb['pfs_mem'] as $pfb_mem => $domain_max) {
                                		if ($pfs_memory >= $pfb_mem) {
                                			$pfb['domain_max_cnt'] = $domain_max;
                                		}
                                	}
                                

                                change "'7000' => '2000000'" and "'7000' => '4000000'" to "'7000' => '6000000'" in both sets.

                                change "'8000' => '2500000'" and "'8000' => '5000000'" to "'8000' => '6000000'" in both sets.

                                Update Reload | DNSBL after making these changes.

                                2e83ff06-6f9d-4627-a64d-71193a0c3608-image.png

                                2fb0b039-02d3-4859-9bb2-042eb7bde376-image.png

                                1 Reply Last reply Reply Quote 0
                                • First post
                                  Last post
                                Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.