• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

TLD Domain count exceeded

Scheduled Pinned Locked Moved pfBlockerNG
15 Posts 3 Posters 1.6k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • U
    Unoptanio
    last edited by Unoptanio Nov 6, 2023, 10:28 AM Nov 5, 2023, 6:06 PM

    Pfsense 2.7.0
    pfBlockerNG-devel 3.2.0_6
    RAM 8Gbyte

    Hi, how do I increase the value?

    Assembling DNSBL database...... completed [ 11/5/23 18:59:34 ]
    TLD:
    TLD analysis........................................xxxxxxxxx completed [ 11/5/23 19:00:00 ]

    ** TLD Domain count exceeded. [ 4000000 ] All subsequent Domains listed as-is **

    77110fb0-8dd6-4329-a2d4-12889c905b2a-image.png

    a042bb5d-29ed-47f2-b48e-5f258975e42f-image.png

    G 1 Reply Last reply Nov 7, 2023, 3:43 PM Reply Quote 0
    • G
      Gertjan @Unoptanio
      last edited by Gertjan Nov 8, 2023, 1:43 PM Nov 7, 2023, 3:43 PM

      @Unoptanio said in TLD Domain count exceeded:

      Hi, how do I increase the value?

      pfBlockerng is nothing more (neither less) as a lot of PHP scripts.
      And PHP hasn't "all system RAM available", but far less.

      Or you, (indirectly, I get it) asked pfBLcokerNG == the PHP scripts, to do something that asked far more then it can handle : merging, sorting, removing the doubles of 4,8 million host names will ..... well, it didn't explode but it just stopped doing some house keeping, and building the main DNSBL list "as is".

      To remove " TLD Domain count exceeded", there is one easy way out : use less DNSBL (feeds).

      Keep in mind that, for every DNS request (coming from your LANs) unbound (the resolver) has to parse this entire DNSBL list so it can look up for a match.

      No "help me" PM's please. Use the forum, the community will thank you.
      Edit : and where are the logs ??

      U 1 Reply Last reply Nov 7, 2023, 4:24 PM Reply Quote 1
      • U
        Unoptanio @Gertjan
        last edited by Unoptanio Nov 7, 2023, 4:28 PM Nov 7, 2023, 4:24 PM

        @Gertjan

        ok.
        At the moment I have not detected any slowness in DNS resolution
        Could it be a solution to increase the RAM of my system? Or does PHP see at most a certain fixed maximum amount of RAM?

        If I increase the system RAM to 32GB, will PHP be able to use more RAM than now?

        S 1 Reply Last reply Nov 7, 2023, 6:13 PM Reply Quote 0
        • S
          SteveITS Galactic Empire @Unoptanio
          last edited by Nov 7, 2023, 6:13 PM

          @Unoptanio Per the blue (i) icon in pfBlocker here:
          44e34053-07a7-49a3-9a47-56a5f4079814-image.png

          "Once the TLD Domain limit below is exceeded, the balance of the Domains will be listed as-is. IE: Blocking only the listed Domain (Not Sub-Domains)
          TLD Domain Limit Restrictions:

          < 1.0GB RAM - Max 100k Domains
          < 1.5GB RAM - Max 150k Domains
          < 2.0GB RAM - Max 200k Domains
          < 2.5GB RAM - Max 250k Domains
          < 3.0GB RAM - Max 400k Domains
          < 4.0GB RAM - Max 600k Domains
          < 5.0GB RAM - Max 1.0M Domains
          < 6.0GB RAM - Max 1.5M Domains
          < 7.0GB RAM - Max 2.5M Domains
          > 7.0GB RAM - > 2.5M Domains"
          

          ...so at a limit of 4 million I guess you have a lot of RAM. :) Without looking into the code I guess you can try it? I would have read that list as it stops at 2.5 million.

          Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
          When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
          Upvote πŸ‘ helpful posts!

          U 2 Replies Last reply Nov 8, 2023, 8:48 AM Reply Quote 0
          • U
            Unoptanio @SteveITS
            last edited by Nov 8, 2023, 8:48 AM

            @SteveITS

            c7cf026e-ddfc-4ba4-a27b-029b2e6f0aee-image.png

            U S 2 Replies Last reply Nov 8, 2023, 8:49 AM Reply Quote 0
            • U
              Unoptanio @Unoptanio
              last edited by Nov 8, 2023, 8:49 AM

              This post is deleted!
              1 Reply Last reply Reply Quote 0
              • U
                Unoptanio @SteveITS
                last edited by Nov 8, 2023, 8:52 AM

                @SteveITS

                5678eddb-2574-45bc-a5e6-d89a1c6500ab-image.png

                6838c7e7-0f0b-4e54-ad51-e75bfe42e61d-image.png

                1 Reply Last reply Reply Quote 0
                • S
                  SteveITS Galactic Empire @Unoptanio
                  last edited by Nov 8, 2023, 1:56 PM

                  @Unoptanio re: "table-entries hard limit", what is your setting of "System > Advanced > Firewall & NAT > Firewall Maximum Table Entries"? Looks like you'd need it to be at least 4.9 million to fit 4806104 entries.

                  (note pfSense has a longstanding bug where the sentence "On this system the default size is: ___" always shows whatever number you've entered, if you've entered a custom number)

                  Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                  When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                  Upvote πŸ‘ helpful posts!

                  U 1 Reply Last reply Nov 8, 2023, 2:08 PM Reply Quote 0
                  • U
                    Unoptanio @SteveITS
                    last edited by Nov 8, 2023, 2:08 PM

                    @SteveITS

                    44b53cc1-7061-45b1-bf34-5abfc983f388-image.png

                    S 1 Reply Last reply Nov 8, 2023, 2:12 PM Reply Quote 0
                    • S
                      SteveITS Galactic Empire @Unoptanio
                      last edited by Nov 8, 2023, 2:12 PM

                      @Unoptanio Assuming your router has enough RAM, change the 400,000 to a higher number.

                      Actually per https://docs.netgate.com/pfsense/en/latest/config/advanced-firewall-nat.html#firewall-maximum-table-entries it says twice the number you need so I guess 9 million for you.

                      Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                      When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                      Upvote πŸ‘ helpful posts!

                      U 1 Reply Last reply Nov 8, 2023, 2:14 PM Reply Quote 0
                      • U
                        Unoptanio @SteveITS
                        last edited by Nov 8, 2023, 2:14 PM

                        @SteveITS

                        now i have only 400.000

                        i try to change in 9.000.000 ??

                        U 1 Reply Last reply Nov 8, 2023, 2:37 PM Reply Quote 0
                        • U
                          Unoptanio @Unoptanio
                          last edited by Unoptanio Nov 8, 2023, 2:43 PM Nov 8, 2023, 2:37 PM

                          @Unoptanio

                          I did a test now with a value of Firewall Maximum Table Entries 6,000,000 but it didn't solve the problem.

                          The number of X is always the same

                          a5935726-dc63-4821-b44b-962f6dc05d2a-image.png

                          S 1 Reply Last reply Nov 8, 2023, 2:59 PM Reply Quote 0
                          • S
                            SteveITS Galactic Empire @Unoptanio
                            last edited by Nov 8, 2023, 2:59 PM

                            @Unoptanio Not sure what to tell you. According to the pfBlocker directions I posted above, over 7 GB RAM should be limited to 2.5 million domains. You may need to find the code in pfBlocker that is setting the limit to 4 million and/or add RAM to your router.

                            Pre-2.7.2/23.09: Only install packages for your version, or risk breaking it. Select your branch in System/Update/Update Settings.
                            When upgrading, allow 10-15 minutes to restart, or more depending on packages and device speed.
                            Upvote πŸ‘ helpful posts!

                            U 1 Reply Last reply Nov 8, 2023, 3:05 PM Reply Quote 1
                            • S SteveITS referenced this topic on Nov 8, 2023, 2:59 PM
                            • U
                              Unoptanio @SteveITS
                              last edited by Unoptanio Nov 8, 2023, 3:06 PM Nov 8, 2023, 3:05 PM

                              @SteveITS
                              @BBcan177

                              ProverΓ² ad acquistare altra ram

                              f54acef1-7e7b-428c-b89d-95e459421d5d-image.png

                              I will try to buy more RAM
                              my current ram consumption is around 50%

                              U 1 Reply Last reply Nov 10, 2023, 9:58 PM Reply Quote 0
                              • U
                                Unoptanio @Unoptanio
                                last edited by Unoptanio Nov 10, 2023, 10:00 PM Nov 10, 2023, 9:58 PM

                                @SteveITS

                                Resolved

                                Extract from /usr/local/pkg/pfblockerng/pfblockerng.inc

                                // Determine max Domain count available for DNSBL TLD analysis (Avoid Unbound memory exhaustion)
                                	$pfs_memory = (round(get_single_sysctl('hw.physmem') / (1024*1024)) ?: 1000);
                                
                                	if (!$pfb['dnsbl_py_blacklist']) {
                                		$pfb['pfs_mem'] = array(   '0' => '100000', '1500' =>  '150000', '2000' =>  '200000', '2500' =>  '250000', '3000' =>  '400000',
                                					'4000' => '600000', '5000' => '1000000', '6000' => '1500000', '7000' => '2000000', '8000' => '2500000',
                                					'12000' => '3000000', '16000' => '4000000', '32000' => '8000000');
                                	} else {
                                		$pfb['pfs_mem'] = array(   '0' => '200000', '1500' =>  '300000', '2000' =>  '400000', '2500' =>  '500000', '3000' =>  '800000',
                                					'4000' => '1200000', '5000' => '2000000', '6000' => '3000000', '7000' => '4000000', '8000' => '5000000',
                                					'12000' => '6000000', '16000' => '8000000', '32000' => '16000000');
                                	}
                                
                                	foreach ($pfb['pfs_mem'] as $pfb_mem => $domain_max) {
                                		if ($pfs_memory >= $pfb_mem) {
                                			$pfb['domain_max_cnt'] = $domain_max;
                                		}
                                	}
                                

                                change "'7000' => '2000000'" and "'7000' => '4000000'" to "'7000' => '6000000'" in both sets.

                                change "'8000' => '2500000'" and "'8000' => '5000000'" to "'8000' => '6000000'" in both sets.

                                Update Reload | DNSBL after making these changes.

                                2e83ff06-6f9d-4627-a64d-71193a0c3608-image.png

                                2fb0b039-02d3-4859-9bb2-042eb7bde376-image.png

                                1 Reply Last reply Reply Quote 0
                                1 out of 15
                                • First post
                                  1/15
                                  Last post
                                Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
                                  This community forum collects and processes your personal information.
                                  consent.not_received