Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    URL Table not working correctly

    Scheduled Pinned Locked Moved General pfSense Questions
    48 Posts 6 Posters 10.3k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • BBcan177B
      BBcan177 Moderator
      last edited by

      Why don't you try to use pfBlockerNG to load these text files… It will also have the option of de-duplicating the Lists (if there are any dups...)

      A small note... from the 2nd screen shot... You don't need to :
      cat filename | wc -l

      You can just use:
      wc -l filename

      The following is actually faster if your counting ms    :)
      grep -c ^ filename

      "Experience is something you don't get until just after you need it."

      Website: http://pfBlockerNG.com
      Twitter: @BBcan177  #pfBlockerNG
      Reddit: https://www.reddit.com/r/pfBlockerNG/new/

      1 Reply Last reply Reply Quote 0
      • J
        johns
        last edited by

        @doktornotor:

        Afraid that unless you make your blocklist available here, this won't get anywhere.

        I'm not sure the whole list is needed, but I've searched for the IP's showing up in the table in my file, but to no avail.

        
        [root@]# grep "1\.0\.209\.0" blocklist.txt
        [root@]# grep "1\.0\.155\.0" blocklist.txt
        [root@]# grep "1\.0\.167\.0" blocklist.txt
        [root@]# head blocklist.txt
        120.203.159.14/24
        118.244.254.17/24
        117.26.227.207/24
        27.153.210.22/24
        183.232.55.193/24
        211.119.86.147/24
        175.44.29.77/24
        125.77.142.168/24
        122.96.59.106/24
        190.216.229.68/24
        
        

        Here's to show grep is working… (selected an IP from the head command above):

        [root@]# grep "27\.153\.210\.22" blocklist.txt
        27.153.210.22/24
        
        1 Reply Last reply Reply Quote 0
        • J
          johns
          last edited by

          @BBcan177:

          Why don't you try to use pfBlockerNG to load these text files… It will also have the option of de-duplicating the Lists (if there are any dups...)

          A small note... from the 2nd screen shot... You don't need to :
          cat filename | wc -l

          You can just use:
          wc -l filename

          The following is actually faster if your counting ms    :)
          grep -c ^ filename

          I'm currently using pfBlockerNG (by selecting specific locations), but 1) wasn't aware I could load custom files, 2) need some automation, the URL Table's appear to offer the scheduled importing I need.

          And thanks for the wc tip, I didn't know that!

          1 Reply Last reply Reply Quote 0
          • BBcan177B
            BBcan177 Moderator
            last edited by

            Shouldn't the last octet be a "0" when using a /24 ?

            I don't think those IPs will load into a packet fence table..

            "Experience is something you don't get until just after you need it."

            Website: http://pfBlockerNG.com
            Twitter: @BBcan177  #pfBlockerNG
            Reddit: https://www.reddit.com/r/pfBlockerNG/new/

            1 Reply Last reply Reply Quote 0
            • BBcan177B
              BBcan177 Moderator
              last edited by

              @johns:

              I'm currently using pfBlockerNG (by selecting specific locations), but 1) wasn't aware I could load custom files, 2) need some automation, the URL Table's appear to offer the scheduled importing I need.

              I am the Dev of pfBNG … So I can confirm that it can use localfiles..  :)

              In the IPv4/6 Tab, enter the localfile path/filename in the URL field....

              "Experience is something you don't get until just after you need it."

              Website: http://pfBlockerNG.com
              Twitter: @BBcan177  #pfBlockerNG
              Reddit: https://www.reddit.com/r/pfBlockerNG/new/

              1 Reply Last reply Reply Quote 0
              • J
                johns
                last edited by

                @BBcan177:

                Shouldn't the last octet by "0" when using a /24 ?

                I don't think those IPs will load into a packet fence table..

                I've used them successfully in small alias tables and URL Tables, and from testing, appear to work correctly.

                1 Reply Last reply Reply Quote 0
                • J
                  johns
                  last edited by

                  @BBcan177:

                  @johns:

                  I'm currently using pfBlockerNG (by selecting specific locations), but 1) wasn't aware I could load custom files, 2) need some automation, the URL Table's appear to offer the scheduled importing I need.

                  I am the Dev of pfBNG … So I can confirm that it can use localfiles..  :)

                  In the IPv4/6 Tab, enter the localfile path/filename in the URL field....

                  Cool!  And I see I can set the update frequency!  Will this handle the 250k+ records?  If so, is there a limit, if not was is the limit?

                  1 Reply Last reply Reply Quote 0
                  • BBcan177B
                    BBcan177 Moderator
                    last edited by

                    The first post that Dok posted was about some issues with the pf Tables… I don't personally have a single table over 200,000IPs, but I do have over 200,000 IPs in total table size.

                    I tried to add that IP and it doesn't get added to the pf table… The last octet in a /24 needs to be "0" for it to load into the table... I assume that this is the issue you are having..

                    I would suggest taking this list and splitting it down into two files.. I assume you are collecting these IPs from a mail server/honey pot etc... Just start with a new 3rd file to keep the size down.

                    grep -c '/24' filename will show how many lines are /24.

                    As a test : (Change the pfB_PRI1 to any existing pf Table)

                    pfctl -t pfB_PRI1 -T add 20.203.159.14/24
                    0/1 addresses added.

                    But if I add the IP with a "0" in the last octet

                    pfctl -t pfB_PRI1 -T show | grep "20.203.159."
                      20.203.159.0
                      20.203.159.0/24

                    "Experience is something you don't get until just after you need it."

                    Website: http://pfBlockerNG.com
                    Twitter: @BBcan177  #pfBlockerNG
                    Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                    1 Reply Last reply Reply Quote 0
                    • D
                      doktornotor Banned
                      last edited by

                      Pretty much as noted above, those blocklists are just wrong. Use /32 (or just nothing) for individual IPs. Those subnets you have are not valid.

                      1 Reply Last reply Reply Quote 0
                      • J
                        johns
                        last edited by

                        @BBcan177:

                        I tried to add that IP and it doesn't get added to the pf table… The last octet in a /24 needs to be "0" for it to load into the table... I assume that this is the issue you are having..

                        I'm starting to think the same thing….however....I have a much smaller list, it's setup the exact same way...the difference is the size...this one has about 300 ip's, that's it...and it auto set the last octect to 0....

                        [root@]# head manualblocklist.txt
                        178.120.172.209/24
                        186.82.25.216/24
                        77.44.161.22/24
                        181.118.75.200/24
                        188.209.49.117/24
                        119.94.47.83/24
                        81.92.120.13/24
                        118.98.115.16/24
                        180.191.104.244/24
                        81.213.208.9/24
                        
                        

                        ![8-28-2015 1-05-55 pm.png](/public/imported_attachments/1/8-28-2015 1-05-55 pm.png)
                        ![8-28-2015 1-05-55 pm.png_thumb](/public/imported_attachments/1/8-28-2015 1-05-55 pm.png_thumb)

                        1 Reply Last reply Reply Quote 0
                        • J
                          johns
                          last edited by

                          I would suggest taking this list and splitting it down into two files.. I assume you are collecting these IPs from a mail server/honey pot etc… Just start with a new 3rd file to keep the size down.

                          I was thinking the same thing, thus my reason for asking what limits there were.  Thanks.

                          1 Reply Last reply Reply Quote 0
                          • D
                            doktornotor Banned
                            last edited by

                            How does it set the last octet?!? The list does not match in the least what you posted in the screenshot.

                            1/ Stop feeding invalid crap to aliases.
                            2/ Load this to pfBlockerNG and use some reputation features there to make the whole thing smaller. Like this:

                            1 Reply Last reply Reply Quote 0
                            • J
                              johns
                              last edited by

                              @doktornotor:

                              Pretty much as noted above, those blocklists are just wrong. Use /32 (or just nothing) for individual IPs. Those subnets you have are not valid.

                              Ah, this may be the issue….using my smaller url table I just checked, and the /24 are imported but not the single IP's listed as /32.  So perhaps removing these from my lists will solve the issue?  I'll try it and repost.

                              As a side note, I tried using pfbng to import and use the list and it semi working, the file was imported and did find dups, but it's not blocking anything (nothing shown in pfbng status widget) and when I try to view the table (via diag -> tables, nothing shows)...so I assume it may be the same issue.

                              1 Reply Last reply Reply Quote 0
                              • J
                                johns
                                last edited by

                                @doktornotor:

                                How does it set the last octet?!? The list does not match in the least what you posted in the screenshot.

                                1/ Stop feeding invalid crap to aliases.
                                2/ Load this to pfBlockerNG and use some reputation features there to make the whole thing smaller.

                                That's exactly the issue…when I look at the files, it shows legit info, but that's not what's being loaded into the aliases, thus the reason I posted here.  I'm going to remove the /32's and see if that doesn't help.

                                1 Reply Last reply Reply Quote 0
                                • D
                                  doktornotor Banned
                                  last edited by

                                  I think you are just very confused? You have no /32 anywhere. You have /24 there.

                                  1 Reply Last reply Reply Quote 0
                                  • J
                                    johns
                                    last edited by

                                    @doktornotor:

                                    I think you are just very confused? You have no /32 anywhere. You have /24 there.

                                    Sorry, perhaps I'm not being clear, but I do have /32's listed, just not a lot of them:

                                    
                                    [root@]# grep '\/32' manualblocklist.txt
                                    70.49.52.23/32
                                    69.46.128.29/32
                                    66.51.128.42/32
                                    109.169.22.220/32
                                    162.250.189.125/32
                                    198.23.140.98/32
                                    104.206.96.58/32
                                    

                                    And to clarify…I've tested and the /24 are imported and auto changed to .0/24, but the /32's are not imported, so I'm wondering if they're causing issues.

                                    1 Reply Last reply Reply Quote 0
                                    • J
                                      johns
                                      last edited by

                                      I take it back, they are being imported….screenshot one is my text file and screen shot two is from diag -> tables.  So I'm back to square one.  Any other ideas?

                                      ![8-28-2015 1-28-35 pm.png](/public/imported_attachments/1/8-28-2015 1-28-35 pm.png)
                                      ![8-28-2015 1-27-57 pm.png](/public/imported_attachments/1/8-28-2015 1-27-57 pm.png)

                                      1 Reply Last reply Reply Quote 0
                                      • J
                                        johns
                                        last edited by

                                        I tried using pfbng to no avail…file imports correctly, but lists shows as blank and status shows no blocks:

                                         UPDATE PROCESS START [ 08/28/15 13:31:55 ]
                                        
                                        [ pfB_Europe_v4 ]	 exists, Reloading File 
                                        [ pfB_Top_v4 ]		 exists, Reloading File 
                                        [ cbl ]			 Downloading New File ----------------------------------------------------------
                                        Original   Masterfile Outfile     [ Post Duplication count ]   
                                        ----------------------------------------------------------
                                        225549     213102     213102      [ Passed ]                   
                                        ----------------------------------------------------------
                                        
                                        ===[  Aliastables / Rules  ]================================
                                        
                                        Firewall Rule Changes Found, Applying Filter Reload
                                        
                                        ===[ FINAL Processing ]=====================================
                                        
                                           [ Original count   ]  [ 242994 ]
                                        
                                           [ Processed Count  ]  [ 230547 ]
                                        
                                        ===[ Deny List IP Counts ]===========================
                                        
                                          230547 total
                                          213102 /var/db/pfblockerng/deny/cbl.txt
                                        
                                        1 Reply Last reply Reply Quote 0
                                        • BBcan177B
                                          BBcan177 Moderator
                                          last edited by

                                          According to that screenshot, it collected 213102 IPs… The others are probably duplicates...

                                          I think your going to have strange results until you convert the file which has /24 to its proper format. ("0" in the last octet"... Don't worry about the /32 it will work with or without the /32.

                                          Here is a small script that will convert the /24's. (Change the /tmp/filename)

                                          
                                          #!/bin/sh
                                          data=$(grep '/24' /tmp/filename)
                                          
                                          for ip in $data; do
                                                  i="$(echo $ip | cut -d '.' -f 1-3)"
                                                  echo $i".0/24"
                                          done
                                          exit
                                          
                                          

                                          To collect all other IPs in the file and exclude the /24's
                                          grep -v '/24' filename

                                          If you want to sort that file…
                                          sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n filename

                                          "Experience is something you don't get until just after you need it."

                                          Website: http://pfBlockerNG.com
                                          Twitter: @BBcan177  #pfBlockerNG
                                          Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                                          1 Reply Last reply Reply Quote 0
                                          • J
                                            johns
                                            last edited by

                                            @BBcan177:

                                            According to that screenshot, it collected 213102 IPs… The others are probably duplicates...

                                            I think your going to have strange results until you convert the file which has /24 to its proper format. ("0" in the last octet"... Don't worry about the /32 it will work with or without the /32.

                                            Here is a small script that will convert the /24's. (Change the /tmp/filename)

                                            
                                            #!/bin/sh
                                            data=$(grep '/24' /tmp/filename)
                                            
                                            for ip in $data; do
                                                    i="$(echo $ip | cut -d '.' -f 1-3)"
                                                    echo $i".0/24"
                                            done
                                            exit
                                            
                                            

                                            To collect all other IPs in the file and exclude the /24's
                                            grep -v '/24' filename

                                            If you want to sort that file…
                                            sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n filename

                                            Thanks, I'll try it.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.