Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    PfBlocker list import problem ….

    Scheduled Pinned Locked Moved pfBlockerNG
    13 Posts 3 Posters 2.6k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A
      atrocity
      last edited by

      forgot to say that on the CSF list server, i SEE the Apache access ass successfull, and the lists are reachable in our whole network.
      It can import it correctly 1 time, can't imagine that he can't do it later …

      1 Reply Last reply Reply Quote 0
      • BBcan177B
        BBcan177 Moderator
        last edited by

        Is deduplication enabled? I would assume its due to that…

        "Experience is something you don't get until just after you need it."

        Website: http://pfBlockerNG.com
        Twitter: @BBcan177  #pfBlockerNG
        Reddit: https://www.reddit.com/r/pfBlockerNG/new/

        1 Reply Last reply Reply Quote 0
        • A
          atrocity
          last edited by

          Hello BBcan and thanks for your answer.
          I checked de-duplication and here are the result of the 2 :

          26 /var/db/aliastables/pfB_CSF.txt
                1 /var/db/aliastables/pfB_CSF2.txt

          33 /var/db/aliastables/pfB_CSF.txt

          So also with de-duplication, the numbers of imported IPs are not the same.
          Same thing for the other lists :

          37525 /var/db/aliastables/pfB_Europe_v4.txt
            30344 /var/db/aliastables/pfB_Blocklistde_SMTP25.txt
              4940 /var/db/aliastables/pfB_Europe_v6.txt
              3027 /var/db/aliastables/pfB_Blocklistde_SSH.txt
              1342 /var/db/aliastables/pfB_Blocklistde_bots.txt
              816 /var/db/aliastables/pfB_Spamhaus.txt
              776 /var/db/aliastables/pfB_Rutgers_Attackers.txt
              500 /var/db/aliastables/pfB_Blocklistde_ApacheDDOS.txt
              409 /var/db/aliastables/pfB_Blocklistde_FTP21.txt
              271 /var/db/aliastables/pfB_Blocklistde_bruteforce.txt

          46065 /var/db/aliastables/pfB_Europe_v4.txt
            29329 /var/db/aliastables/pfB_Blocklistde_SMTP25.txt
              6569 /var/db/aliastables/pfB_Europe_v6.txt
              3369 /var/db/aliastables/pfB_Blocklistde_SSH.txt
              1139 /var/db/aliastables/pfB_Blocklistde_bots.txt
              816 /var/db/aliastables/pfB_Spamhaus.txt
              252 /var/db/aliastables/pfB_Blocklistde_bruteforce.txt
              226 /var/db/aliastables/pfB_Rutgers_Attackers.txt

          BBcan, i'm one of the guy that give you a access to our firewalls in Europe, some months ago during your beta ;-)
          Thanks for any trick/idea to debug this.
          Regards

          1 Reply Last reply Reply Quote 0
          • BBcan177B
            BBcan177 Moderator
            last edited by

            Run a reload then execute the following command:

            rm /var/db/pfblockerng/test1
cat /var/db/pfblockerng/deny/*.txt >> /var/db/pfblockerng/test1

            Then after the next cron run:

            rm /var/db/pfblockerng/test2
cat /var/db/pfblockerng/deny/*.txt >> /var/db/pfblockerng/test2

            Then compare the differences:

            awk 'FNR==NR{a[$0];next}!($0 in a)' /var/db/pfblockerng/test2 /var/db/pfblockerng/test1 | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n
awk 'FNR==NR{a[$0];next}!($0 in a)' /var/db/pfblockerng/test1 /var/db/pfblockerng/test2 | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n

            You could also send the comparison to a file for review:

            awk 'FNR==NR{a[$0];next}!($0 in a)' /var/db/pfblockerng/test2 /var/db/pfblockerng/test1 | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n > /var/db/pfblockerng/test1dif
awk 'FNR==NR{a[$0];next}!($0 in a)' /var/db/pfblockerng/test1 /var/db/pfblockerng/test2 | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n > /var/db/pfblockerng/test2dif

            Then use a file editor like vi, ee, or nano to view the files
            [ /var/db/pfblockerng/test1dif    and    /var/db/pfblockerng/test2dif  ]

            You can then grep one of the missing IPs and see if the IP is now blocked by a larger CIDR entry:

            grep "1\.2\.3\.4"  /var/db/pfblockerng/deny/*
grep "1\.2\.3\."  /var/db/pfblockerng/deny/*
grep "1\.2\."  /var/db/pfblockerng/deny/*
grep "1\."  /var/db/pfblockerng/deny/*

            You can use this link to see what a CIDR range includes:
            https://www.ultratools.com/tools/netMask

            You can send the results via email to make it easier to reply. (See general tab for email)

            Please do your own testing first please :)

            "Experience is something you don't get until just after you need it."

            Website: http://pfBlockerNG.com
            Twitter: @BBcan177  #pfBlockerNG
            Reddit: https://www.reddit.com/r/pfBlockerNG/new/

            1 Reply Last reply Reply Quote 0
            • A
              atrocity
              last edited by

              Hello,

              well, here are the results on of of the 2 firewalls :

              [2.3.2-RELEASE][root@MyFirewallName]/var/db/pfblockerng: awk 'FNR==NR{a[$0];next}!($0 in a)' /var/db/pfblockerng/test2 /var/db/pfblockerng/test1 | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n
              [2.3.2-RELEASE][root@MyFirewallName]/var/db/pfblockerng: awk 'FNR==NR{a[$0];next}!($0 in a)' /var/db/pfblockerng/test1 /var/db/pfblockerng/test2 | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n
              [2.3.2-RELEASE][root@MyFirewallName]/var/db/pfblockerng: awk 'FNR==NR{a[$0];next}!($0 in a)' /var/db/pfblockerng/test2 /var/db/pfblockerng/test1 | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n > /var/db/pfblockerng/test1dif
              [2.3.2-RELEASE][root@MyFirewallName]/var/db/pfblockerng: awk 'FNR==NR{a[$0];next}!($0 in a)' /var/db/pfblockerng/test1 /var/db/pfblockerng/test2 | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n > /var/db/pfblockerng/test2dif

              nothing found :(

              Should i try to delete all the pfblocker/* and redo this tests ?
              Thanks
              Best regards

              1 Reply Last reply Reply Quote 0
              • L
                lispeedyg
                last edited by

                Hi,
                Has this ever been resolved??
                I've been having the same issues and the OP.  Currently on pfSense V 2.4.1 and pfBlockerNG v 2.1.2_1.
                And, specifically, the only way I can actually have the list updated is to uninstall pfBlocker completely, i.e., Don't keep settings.  And then re-start the process completely.  All other facets work as intended.  But, this has been a significant drawback.

                Is there any further help anyone can provide?

                1 Reply Last reply Reply Quote 0
                • BBcan177B
                  BBcan177 Moderator
                  last edited by

                  Post some more details about what you're seeing … Keep in mind that Feeds won't update if the source of the Feed is unchanged (ie: previously downloaded feed is the same as the source feed), and the pfblockerng.log will show "Update not required".

                  "Experience is something you don't get until just after you need it."

                  Website: http://pfBlockerNG.com
                  Twitter: @BBcan177  #pfBlockerNG
                  Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                  1 Reply Last reply Reply Quote 0
                  • L
                    lispeedyg
                    last edited by

                    Hi,
                    Thanks for the quick reply..
                    My feed is actually a text file I host on the pfSense box at /usr/local/www/DenyIPs.txt
                    I add to the file manually as I'm informed of possible new threats.
                    I've pointed to the file source in two seperate ways, i.e., using the direct path above and also as http://[server-ip]/DenyIPs.txt

                    In both scenarios I can see the new IP entries in the file/browser.  I then run both an Update and Reload for pfBlockerNG.  And in all cases (other than completely deleting the application) the result is "no changes".

                    1 Reply Last reply Reply Quote 0
                    • BBcan177B
                      BBcan177 Moderator
                      last edited by

                      You need to use this format:

                      /usr/local/www/DenyIPs.txt

                      And keep in mind that Force Update|Reload just downloads the feed if its new (IE: wasn't previously downloaded).

                      You need to define the Frequency setting to say "once per hour" and then it will download the referenced  Source Feed at that time via the Cron task.

                      "Experience is something you don't get until just after you need it."

                      Website: http://pfBlockerNG.com
                      Twitter: @BBcan177  #pfBlockerNG
                      Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                      1 Reply Last reply Reply Quote 0
                      • L
                        lispeedyg
                        last edited by

                        Thanks for that.
                        I've reverted to the original file pointer, i.e., /usr/local/www/DenyIPs.txt.
                        I've also revised the frequency to 1hour from weekly.
                        After the above, I manually ran Update/Cron/Reload.  And, the new IP's are still not visible and the number of IP's are still the same as before the process.  Do I need to wait the hour for it to run the CRON automatically?  If not I think the problem still exists.

                        1 Reply Last reply Reply Quote 0
                        • BBcan177B
                          BBcan177 Moderator
                          last edited by

                          You can goto the Log Browser tab and delete this feed in the "Deny" folder.
                          or from the shell:

                          rm /var/db/pfblockerng/deny/DenyIPs.*

                          Then Force Update to get the changes immediately… But after that... it will update the file as per the frequency setting as designed...

                          "Experience is something you don't get until just after you need it."

                          Website: http://pfBlockerNG.com
                          Twitter: @BBcan177  #pfBlockerNG
                          Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                          1 Reply Last reply Reply Quote 0
                          • L
                            lispeedyg
                            last edited by

                            Thank you.
                            I'll give that a try and post back if there are other issues.
                            Thanks again.

                            1 Reply Last reply Reply Quote 0
                            • First post
                              Last post
                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.