PfBlockerNG


  • Moderator

    @fsansfil:

    Very nice job on the package but…in the alert tab we need small country flags for the CC colum. Its not a real NG package w/o little flags :)

    :) I thought about it.. I was going to borrow the flags from the ntopng package but with so many countries, I think it's impractical for most users to reference all those Flags. While I agree it would look nice, I think Geography is not the intended topic. I may add it and let the user decide which they prefer. However, I am focusing my efforts on the next version which will have Unbound DNSBL integration.  :)

    Thanks for the overall positive feedback!



  • ROFL!

    @fsansfil:

    Very nice job on the package but…in the alert tab we need small country flags for the CC colum. Its not a real NG package w/o little flags :)


  • Banned

    I think you just need one flag for CN. That's where most of the unwanted junk traffic comes from. :P

    (I know about project getting pissed off bug reports about "You ignoramus don't know this state does not exist any more?! We have split from those nasty theyspeakotherdialect criminals whopping two years ago and you dare to not have noticed yet?! What an insult to our huge 500K headcount nation! And/or - this flag is not used any more, it was used by $badguy regime, you are insulting our brand new "democracy". Flags can become a slippery slope very quickly…)



  • Many thanks to all those that made this happen  ;D

    A recent upgrade to vDISASTER-1.0, has forced me to start over from scratch. Working bit by bit on preparing a system for the blueprint guides doesn't seem like a lot of work when you do it bit by bit. Starting over made me realize what a lot of people have been complaining about: complexity.

    Just about when the suricata part of the blueprints was finished (I deliberately delayed the IP lists, since I knew the package was coming), the package was made available. Great, perfect opportunity to test it I thought.

    Installed the package, went to configure it…my jaw dropped! BBcan177 told me it was good, but unfortunately due to a LOT of work coming my way, wasn't able to help out test it, but I never thought it would be THIS good!

    It basically takes configuring a system according to the blueprints from a process that takes 2 days (you do double check everything, right?) to a few hours. Biggest help is the auto-creation of floating rules using the lists. Bonus thing is, the way the package handles the lists, you don't even have to create the dozen (or so) floating rules suggested. Just one rule containing the alias, which in turn contains the lists. pfBlockerNG then handles the "ok, so what list is this IP on?" part in its alerts tab. Perfect!

    So far it makes a perfect addition to the blueprint guides (and the new guide, which will be a LOT simpler, yes I'm typing it up bit by bit, preparing it for other authors to contribute too). Only downside I find to it, is that sync doesn't want to sync, but that's another story, for another day. First I need to add the dozens of lists back, then figure out why it's not working. I understand that a much needed break is in order for the devs, so for the time being, I'm not going to bother with it  ;D Problem solved: There is an input validation that messes up my special passwords (I'm special, didn't you know?). Using alphanumeric ones solves it. Thanks to BBcan177 for keeping me company while troubleshooting this.



  • Any problem wizh iblocklist?

    After i add list, can t download any list from this site. I have gz and cidr format. Any help?



  • Have you clicked the "Force Cron" button on the Update tab? Is there anything in the Logs / Log Files / Error.log?



  • What a nice surprise to see an update for pfblocker. I didnt expect i to to come for a couple of weeks. Totally awsome.
    Everything works great. I see some really clever things in the new NG package. Lots of shiny new options.

    One minor thing. The header field was a little strange. Im used to a header meaning something to ignore  as actual input. So my immediate intuition was to input something like a # because my lists did not load right and the force update said:
    ** TERMINATED - Header contains Blank/International/Special or Spaces

    But I found in the forum that its a header like a description. So maybe a little description for header would be helpfull.

    Congratulations BBcan177 for your achievement



  • I am having a problem getting the lists from i-blocklist.com to work on this. What format should I select on the i-blocklist website and what format on the new pfblockerng? Anyone gotten this to work?

    Nevermind. Everything I need to know is there. I just looked over it. Sorry about that all.


  • Moderator

    In v1.01 I have found a bug that doesn't allow cron to Update the Lists as per the Frequency setting in the Alias. While adding the "Anonymous Proxy and Satellite Providers" to the code, I added a new function that is in my Dev version of pfBNG but I unfortunately missed to add a variable. You can see in the Widget, the last Updated Timestamp of the Aliases.

    I have posted a Pull Request to fix this issue. It also includes some other improvments.

    So until its merged, you can follow any ONE of these steps to fix this issue:

    Apply this patch using system patches      OR      Manually edit this File and add the part in blue.

    /usr/local/www/pfblockerng/pfblockerng.php  on  Line 150  and add the missing variable.

    function pfb_update_check($header_url, $list_url, $url_format) {
                  to
          function pfb_update_check($header_url, $list_url, $url_format, $pfbfolder) {

    
    --- /usr/local/www/pfblockerng/pfblockerng.php
    +++ /usr/local/www/pfblockerng/pfblockerng.php
    @@ -147,7 +147,7 @@
     if ($uname['machine'] == "amd64")
            ini_set('memory_limit', '256M');
    
    -function pfb_update_check($header_url, $list_url, $url_format) {
    +function pfb_update_check($header_url, $list_url, $url_format, $pfbfolder) {
            global $pfb;
            $pfb['cron_update'] = FALSE;
    
    

    After applying this patch or manually editing the file, running "Force Cron" will update the files but only if the Alias Frequency Setting is within the current Hour or wait for the next Cron event to update the files as required.

    Sorry about the trouble!


  • Moderator

    @JohnPFsense:

    One minor thing. The header field was a little strange. Im used to a header meaning something to ignore  as actual input. So my immediate intuition was to input something like a # because my lists did not load right and the force update said:
    ** TERMINATED - Header contains Blank/International/Special or Spaces

    But I found in the forum that its a header like a description. So maybe a little description for header would be helpfull.

    I was going to name it "filename" but I think it can also be misleading (users might think to enter the URL filename there)

    There is a description of what a Header should be just above the URL settings in each alias. Suggestions are always welcome!  :)



  • Seems like the Force Cron button deletes the cron job:

    
    0  	*  	*  	*  	*  	root  	/usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php cron >> /var/log/pfblockerng/pfblockerng.log 2>&1  
    
    

    Happens with and without that change to line 150 of pfblockerng.php. Can anyone else confirm this or have I managed to totally mess up my installation? :)


  • Moderator

    @fragged:

    Seems like the Force Cron button deletes the cron job:

    When you select any of the "Force" Icons, the first step is to remove any Cron tasks that are defined. This prevents a collision with a Cron task that could be called while the Manual task is still in process.

    Once the "Force" (Update, Cron or Reload) are completed, it will add the Cron task back to the Cron Manager.



  • @BBcan177:

    @fragged:

    Seems like the Force Cron button deletes the cron job:

    When you select any of the "Force" Icons, the first step is to remove any Cron tasks that are defined. This prevents a collision with a Cron task that could be called while the Manual task is still in process.

    Once the "Force" (Update, Cron or Reload) are completed, it will add the Cron task back to the Cron Manager.

    Ok, so the problem then is that the page never stops loading even than the log part shows the job finished.


  • Moderator

    @fragged:

    Ok, so the problem then is that the page never stops loading even than the log part shows the job finished.

    ok. That is a small bug… The Window is still spinning as its waiting for a string

    "UPDATE PROCESS ENDED"

    but in Force Cron, it never sees that String... I will add a fix for that. But the task did complete when it outputs "CRON  PROCESS  ENDED". You can move away from the Tab or click "End View"

    EDIT: Yup.. It won't add the Cron Task back as it didn't complete the function.

    Click "Save" in the "General Tab" and it will be added.



  • @BBcan177:

    @fragged:

    Ok, so the problem then is that the page never stops loading even than the log part shows the job finished.

    ok. That is a small bug… The Window is still spinning as its waiting for a string

    "UPDATE PROCESS ENDED"

    but in Force Cron, it never sees that String... I will add a fix for that. But the task did complete when it outputs "CRON  PROCESS  ENDED". You can move away from the Tab or click "End View"

    It seems to work (adds the cron job back) if it actually does something. It didn't recreate the cronjob if the result was "No Updates required.".

    Now I noticed another possible issue. The update code doesn't match the remote timestamp correctly for a particular list all of the sudden. (List = http://www.spamhaus.org/drop/edrop.txt)

    
    [ SpamHDrop1 ]
      Remote timestamp: Tue,  3 Feb 2015 10:41:35 GMT
      Local  timestamp: Tue, 03 Feb 2015 10:41:35 GMT
      Updates Found
    
    

    I seem to be able to break everything I touch today :(


  • Moderator

    @fragged:

    
    [ SpamHDrop1 ]
      Remote timestamp: Tue,  3 Feb 2015 10:41:35 GMT
      Local  timestamp: Tue, 03 Feb 2015 10:41:35 GMT
      Updates Found
    
    

    Spamhaus is using a single digit day while they should be using a two digit day. I have it on my list to follow up with them.



  • The filename should be "/usr/local/www/pfblockerng/pfblockerng.php"

    @BBcan177:

    /usr/local/www/pfblockerng.php  on  Line 150  and add the missing variable.

    function pfb_update_check($header_url, $list_url, $url_format) {
                  to
          function pfb_update_check($header_url, $list_url, $url_format, $pfbfolder) {



  • Feature Request:

    Allow 1,5,15,30 min interval for Update Frequency. I realize historically this has been forced minimum 1hr to force people to be good net-citizens, but having a faster update frequency would be preferred for internal lists. I admit 1min may be excessive, but the rest are reasonable for internal use lists.

    I like to use different software/scripts to generate block lists on my machines behind the pfSense box, and have pfBlocker poll those lists. 1 hour is too long, as most attackers will give up in that time frame. Think of the SSH script kiddy or email spammer that are pounding on the machine for a short period of time. Since I may have multiple IP's forwarded to similar services (behind pfSense) it is convenient to have the list on pfSense so it's protecting all hosts, instead of the one getting pounded before the person moves on to the second or third.

    This is actually all I use pfBlocker for, along with a few friends.


  • Moderator

    There is a new Pull Request #808 to fix the issues reported by Fragged and jflsakfja.

    pfBNG - v1.03

    1. When Using "Force Cron" and "no updates" were found, the final process would not complete. This resulted in the active Cron task not being restored.
    2. XMLRPC Sync - Added MaxMind "Anonymous Proxy and Satellite providers"

  • Moderator

    @Sn3ak:

    Feature Request:

    Allow 1,5,15,30 min interval for Update Frequency.

    Yes this is functionality that I want to add. I don't expect to get to it soon thou. Lots of things to do, so little time… However, I would expect to use 15,30,45 mins time frames...



  • Hi!
    Thanks for the new PfblockerNG.
    Should be exellent from what I understand. However I cannot use it since the countrylists IPv4 are empty. What am I missing? :-)

    I tried to delete the package and made a reinstall but the result is the same.

    Version 1.02

    Thanks
    Jonna


  • Moderator

    @jonna99:

    However I cannot use it since the countrylists IPv4 are empty. What am I missing? :-)

    Can you provide some more details? Screenshots etc…



  • Ok thank you.
    Using clean install of pfSense-memstick-2.2-RELEASE and Pfblocker 1.02. Have been using the old Pfblocker without problems before.
    I find nothing wrong in logs although I´m not sure where  to find everything.
    The countrylists top-20 spammers (jpeg5) are filled but not the other ones under tab Europe eg.

    Jonna











  • Moderator

    @jonna99:

    The countrylists top-20 spammers (jpeg5) are filled but not the other ones under tab Europe eg.

    Thanks jonna99,

    It seems that the Maxmind Files did not download? Thats why you see in the Continent pages

    "Warning: Invalid Argument Supplied"

    I can send you the command to update the MaxMind Database, but I would like to see if I can understand the circumstances that led to this issue for you? There is a log file called "geoip.log" in the Log Browser, can you PM me with those details.



  • (Edit: 2.2-RELEASE (amd64) pfBlockerNG 1.02)

    I found a minor bug on the Alerts tab where the displayed rule is sometimes incorrect.

    I've seen this a few times since yesterday (2 diff machines) and it seems to be displaying the rule from the previous entry.

    If I get several entries in from the same list in-a-row and the 1st one is wrong, the subsequent ones are wrong too.


  • Moderator

    Hi LinuxTracker,

    There seems to be a big spread of time between those Alerts. Could it be that you added the Asia Continent after Spamhaus was already downloaded?

    Do you have enable "De-duplication" checked in the General tab?

    I would suggest to clear the Firewall Log and execute a "Force Reload", this will refresh all of the databases and then see how it goes from there.



  • Make sure you dont have a rule like this on WAN

    WEB-CLIENT csv file download request"; flow:to_server,established; content:"GET"; nocase; http_method; content:".csv"; nocase; http_uri;

    Had one active and it prevented the download of maxmind country list in my first install…

    F.



  • Sidebar: Doing a spotcheck, I found a popular SSL cert reseller is on a widely used blocklist.

    As of this writing, the website for StartSSL is on SpamHaus's Don't Route Or Peer blocklist http://www.spamhaus.org/drop/.

    www.startssl.com = 192.116.242.20 https://www.whatsmydns.net/#A/www.startssl.com

    192.112.112.0/20 blocked by http://www.spamhaus.org/drop/drop.txt

    Edit: I should clarify that this isn't an issue with pfBlockerNG.
    It's just a reminder that reputable lists get conflicting data and there's no replacement for monitoring logs.

    and yes, I'm aware of Spamhause's heavy handed history.



  • @BBcan177:

    Hi LinuxTracker,

    There seems to be a big spread of time between those Alerts. Could it be that you added the Asia Continent after Spamhaus was already downloaded?

    Do you have enable "De-duplication" checked in the General tab?

    I would suggest to clear the Firewall Log and execute a "Force Reload", this will refresh all of the databases and then see how it goes from there.

    I had some better examples where they all seemed to happen in a short time but I didn't screenshot them.

    I'll follow up on your recommendations and post back if I see it happening again.

    Thanks!


  • Moderator

    I recommend to everyone to enable "De-Duplication".

    Scenario -

    1. Country/Continents are Downloaded first. (No De-dup is needed as MaxMind reports Ranges per Country, so no overlap.)

    2. As each subsequent Alias/List is downloaded, it will not add any IPs that are already being blocked by an existing Continent List.

    When changes are made like "De-Dup or Reputation, or the removal of a Continent/Country or any Lists", I recommend that a "Force Reload" is executed. This will refresh the Database to the current settings.



  • NA country list is blocking my HE ipv6 tunnel. Tried suppression and changing order. Also tried de-selecting North America from the NA list. I have kept all countries to deny inbound.

    Only way it works is by disabling NA country list entirely. Suggestions??



  • Hi BBcan177

    Here is the logfile you asked for.

    Thanks
    Jonna

    geoip.log.txt



  • So I've downloaded pfblockerNG but cant find a tab to block the United States or North America which is only in response to the US announcing yesterday they will keep all non-US traffic for 5 years. I value my privacy so I'd rather not have anything to do with US sites after this announcement which will probably make surfing interesting to say the least.

    So considering the number of CIDR's for the the US, would it be best to add them to Aliases and block via a firewall rule or use pfblockerng.

    Do have to say it looks comprehensive which is good and certainly an improvement over earlier attempts to block IP's en mass.


  • Banned

    @firewalluser:

    So I've downloaded pfblockerNG but cant find a tab to block the United States or North America

    Eeeeh?!



  • As a side note.., and a question..

    Upgrading from version 1.01 to version 1.02 removed all my newly added pfBlockerNG rules from version 1.01.
    Had to create them all again after the package update. Seeing that update 1.03 is available.. Will this also remove all my current pfBlockerNG rules?
    Lots of manual labor..

    Anyway the new package is great!


  • Banned

    @SkyNET:

    Upgrading from version 1.01 to version 1.02 removed all my newly added pfBlockerNG rules from version 1.01.
    Had to create them all again after the package update. Seeing that update 1.03 is available.. Will this also remove all my current pfBlockerNG rules?
    Lots of manual labor..



  • Just to be clear: This post is not a personal attack at anybody.

    Most of the problems as far as I can see where with user configuration errors. This isn't a fruity company's product. You DO actually need to configure it in order to run.

    Choosing every single list on the planet without first understanding the basics is wrong.
    Blocking by countries is wrong. Most web hosting "companies" (trust me, I know my competition better than they know themselves) are renting space in a "cheap-from-a-provider's POV" country (eg. US). Yes I do agree that the NSA is being run by criminals, and yes I do agree that most (all?) of them should be locked in cells with the door welded shut. Weigh in what you lose with what you get. Not arguing the "if you have nothing to hide, you have nothing to be afraid of" meme. I'm saying that by cutting off entire countries, you risk cutting off significant portion of the "good" part of the Internet as well. For a single user, who cares. For a provider, you will, trust me.
    Spend 5 minutes (time it) going through the package's options before complaining that something doesn't work. There are cases where it is a genuine bug (eg PS tab not getting synced) and other times it's a specific issue to you (eg sync not working because of special characters in the password).

    Take care, and wait for the configuration guides before letting the beast out of the cage  ;)



  • @doktornotor:

    @firewalluser:

    So I've downloaded pfblockerNG but cant find a tab to block the United States or North America

    Eeeeh?!

    @firewalluser:

    So I've downloaded pfblockerNG but cant find a tab to block the United States or North America which is only in response to the US announcing yesterday they will keep all non-US traffic for 5 years. I value my privacy so I'd rather not have anything to do with US sites after this announcement which will probably make surfing interesting to say the least.

    So considering the number of CIDR's for the the US, would it be best to add them to Aliases and block via a firewall rule or use pfblockerng.

    Do have to say it looks comprehensive which is good and certainly an improvement over earlier attempts to block IP's en mass.



  • @Asterix:

    NA country list is blocking my HE ipv6 tunnel. Tried suppression and changing order. Also tried de-selecting North America from the NA list. I have kept all countries to deny inbound.

    Only way it works is by disabling NA country list entirely. Suggestions??

    Anyone?



  • @firewalluser:

    So I've downloaded pfblockerNG but cant find a tab to block the United States or North America which is only in response to the US announcing yesterday they will keep all non-US traffic for 5 years. I value my privacy so I'd rather not have anything to do with US sites after this announcement which will probably make surfing interesting to say the least.

    So considering the number of CIDR's for the the US, would it be best to add them to Aliases and block via a firewall rule or use pfblockerng.

    Do have to say it looks comprehensive which is good and certainly an improvement over earlier attempts to block IP's en mass.

    hmmm, so does this mean you wont be accessing the forum anymore?


Log in to reply