Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    TLD Domain count exceeded

    Scheduled Pinned Locked Moved pfBlockerNG
    15 Posts 3 Posters 1.6k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • UnoptanioU
      Unoptanio
      last edited by Unoptanio

      Pfsense 2.7.0
      pfBlockerNG-devel 3.2.0_6
      RAM 8Gbyte

      Hi, how do I increase the value?

      Assembling DNSBL database...... completed [ 11/5/23 18:59:34 ]
      TLD:
      TLD analysis........................................xxxxxxxxx completed [ 11/5/23 19:00:00 ]

      ** TLD Domain count exceeded. [ 4000000 ] All subsequent Domains listed as-is **

      77110fb0-8dd6-4329-a2d4-12889c905b2a-image.png

      a042bb5d-29ed-47f2-b48e-5f258975e42f-image.png

      GertjanG 1 Reply Last reply Reply Quote 0
      • GertjanG
        Gertjan @Unoptanio
        last edited by Gertjan

        @Unoptanio said in TLD Domain count exceeded:

        Hi, how do I increase the value?

        pfBlockerng is nothing more (neither less) as a lot of PHP scripts.
        And PHP hasn't "all system RAM available", but far less.

        Or you, (indirectly, I get it) asked pfBLcokerNG == the PHP scripts, to do something that asked far more then it can handle : merging, sorting, removing the doubles of 4,8 million host names will ..... well, it didn't explode but it just stopped doing some house keeping, and building the main DNSBL list "as is".

        To remove " TLD Domain count exceeded", there is one easy way out : use less DNSBL (feeds).

        Keep in mind that, for every DNS request (coming from your LANs) unbound (the resolver) has to parse this entire DNSBL list so it can look up for a match.

        No "help me" PM's please. Use the forum, the community will thank you.
        Edit : and where are the logs ??

        UnoptanioU 1 Reply Last reply Reply Quote 1
        • UnoptanioU
          Unoptanio @Gertjan
          last edited by Unoptanio

          @Gertjan

          ok.
          At the moment I have not detected any slowness in DNS resolution
          Could it be a solution to increase the RAM of my system? Or does PHP see at most a certain fixed maximum amount of RAM?

          If I increase the system RAM to 32GB, will PHP be able to use more RAM than now?

          S 1 Reply Last reply Reply Quote 0
          • S
            SteveITS Galactic Empire @Unoptanio
            last edited by

            @Unoptanio Per the blue (i) icon in pfBlocker here:
            44e34053-07a7-49a3-9a47-56a5f4079814-image.png

            "Once the TLD Domain limit below is exceeded, the balance of the Domains will be listed as-is. IE: Blocking only the listed Domain (Not Sub-Domains)
            TLD Domain Limit Restrictions:

            < 1.0GB RAM - Max 100k Domains
            < 1.5GB RAM - Max 150k Domains
            < 2.0GB RAM - Max 200k Domains
            < 2.5GB RAM - Max 250k Domains
            < 3.0GB RAM - Max 400k Domains
            < 4.0GB RAM - Max 600k Domains
            < 5.0GB RAM - Max 1.0M Domains
            < 6.0GB RAM - Max 1.5M Domains
            < 7.0GB RAM - Max 2.5M Domains
            > 7.0GB RAM - > 2.5M Domains"
            

            ...so at a limit of 4 million I guess you have a lot of RAM. :) Without looking into the code I guess you can try it? I would have read that list as it stops at 2.5 million.

            Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
            When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
            Upvote ๐Ÿ‘ helpful posts!

            UnoptanioU 2 Replies Last reply Reply Quote 0
            • UnoptanioU
              Unoptanio @SteveITS
              last edited by

              @SteveITS

              c7cf026e-ddfc-4ba4-a27b-029b2e6f0aee-image.png

              UnoptanioU S 2 Replies Last reply Reply Quote 0
              • UnoptanioU
                Unoptanio @Unoptanio
                last edited by

                This post is deleted!
                1 Reply Last reply Reply Quote 0
                • UnoptanioU
                  Unoptanio @SteveITS
                  last edited by

                  @SteveITS

                  5678eddb-2574-45bc-a5e6-d89a1c6500ab-image.png

                  6838c7e7-0f0b-4e54-ad51-e75bfe42e61d-image.png

                  1 Reply Last reply Reply Quote 0
                  • S
                    SteveITS Galactic Empire @Unoptanio
                    last edited by

                    @Unoptanio re: "table-entries hard limit", what is your setting of "System > Advanced > Firewall & NAT > Firewall Maximum Table Entries"? Looks like you'd need it to be at least 4.9 million to fit 4806104 entries.

                    (note pfSense has a longstanding bug where the sentence "On this system the default size is: ___" always shows whatever number you've entered, if you've entered a custom number)

                    Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                    When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                    Upvote ๐Ÿ‘ helpful posts!

                    UnoptanioU 1 Reply Last reply Reply Quote 0
                    • UnoptanioU
                      Unoptanio @SteveITS
                      last edited by

                      @SteveITS

                      44b53cc1-7061-45b1-bf34-5abfc983f388-image.png

                      S 1 Reply Last reply Reply Quote 0
                      • S
                        SteveITS Galactic Empire @Unoptanio
                        last edited by

                        @Unoptanio Assuming your router has enough RAM, change the 400,000 to a higher number.

                        Actually per https://docs.netgate.com/pfsense/en/latest/config/advanced-firewall-nat.html#firewall-maximum-table-entries it says twice the number you need so I guess 9 million for you.

                        Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                        When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                        Upvote ๐Ÿ‘ helpful posts!

                        UnoptanioU 1 Reply Last reply Reply Quote 0
                        • UnoptanioU
                          Unoptanio @SteveITS
                          last edited by

                          @SteveITS

                          now i have only 400.000

                          i try to change in 9.000.000 ??

                          UnoptanioU 1 Reply Last reply Reply Quote 0
                          • UnoptanioU
                            Unoptanio @Unoptanio
                            last edited by Unoptanio

                            @Unoptanio

                            I did a test now with a value of Firewall Maximum Table Entries 6,000,000 but it didn't solve the problem.

                            The number of X is always the same

                            a5935726-dc63-4821-b44b-962f6dc05d2a-image.png

                            S 1 Reply Last reply Reply Quote 0
                            • S
                              SteveITS Galactic Empire @Unoptanio
                              last edited by

                              @Unoptanio Not sure what to tell you. According to the pfBlocker directions I posted above, over 7 GB RAM should be limited to 2.5 million domains. You may need to find the code in pfBlocker that is setting the limit to 4 million and/or add RAM to your router.

                              Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                              When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                              Upvote ๐Ÿ‘ helpful posts!

                              UnoptanioU 1 Reply Last reply Reply Quote 1
                              • S SteveITS referenced this topic on
                              • UnoptanioU
                                Unoptanio @SteveITS
                                last edited by Unoptanio

                                @SteveITS
                                @BBcan177

                                Proverรฒ ad acquistare altra ram

                                f54acef1-7e7b-428c-b89d-95e459421d5d-image.png

                                I will try to buy more RAM
                                my current ram consumption is around 50%

                                UnoptanioU 1 Reply Last reply Reply Quote 0
                                • UnoptanioU
                                  Unoptanio @Unoptanio
                                  last edited by Unoptanio

                                  @SteveITS

                                  Resolved

                                  Extract from /usr/local/pkg/pfblockerng/pfblockerng.inc

                                  // Determine max Domain count available for DNSBL TLD analysis (Avoid Unbound memory exhaustion)
                                  	$pfs_memory = (round(get_single_sysctl('hw.physmem') / (1024*1024)) ?: 1000);
                                  
                                  	if (!$pfb['dnsbl_py_blacklist']) {
                                  		$pfb['pfs_mem'] = array(   '0' => '100000', '1500' =>  '150000', '2000' =>  '200000', '2500' =>  '250000', '3000' =>  '400000',
                                  					'4000' => '600000', '5000' => '1000000', '6000' => '1500000', '7000' => '2000000', '8000' => '2500000',
                                  					'12000' => '3000000', '16000' => '4000000', '32000' => '8000000');
                                  	} else {
                                  		$pfb['pfs_mem'] = array(   '0' => '200000', '1500' =>  '300000', '2000' =>  '400000', '2500' =>  '500000', '3000' =>  '800000',
                                  					'4000' => '1200000', '5000' => '2000000', '6000' => '3000000', '7000' => '4000000', '8000' => '5000000',
                                  					'12000' => '6000000', '16000' => '8000000', '32000' => '16000000');
                                  	}
                                  
                                  	foreach ($pfb['pfs_mem'] as $pfb_mem => $domain_max) {
                                  		if ($pfs_memory >= $pfb_mem) {
                                  			$pfb['domain_max_cnt'] = $domain_max;
                                  		}
                                  	}
                                  

                                  change "'7000' => '2000000'" and "'7000' => '4000000'" to "'7000' => '6000000'" in both sets.

                                  change "'8000' => '2500000'" and "'8000' => '5000000'" to "'8000' => '6000000'" in both sets.

                                  Update Reload | DNSBL after making these changes.

                                  2e83ff06-6f9d-4627-a64d-71193a0c3608-image.png

                                  2fb0b039-02d3-4859-9bb2-042eb7bde376-image.png

                                  1 Reply Last reply Reply Quote 0
                                  • First post
                                    Last post
                                  Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.