Taming the beasts… aka suricata blueprint


  • Moderator

    @justsomeone:

    $ pkg_add -r grepcidr
    tar: Failed to set default locale
    

    First, I would recommend that you make a Full Backup  Diagnostics:Backup/Restore  (Backup Area "all") and Download configuration… :)

    I haven't come across this error on any of my installs or with the ones that I have helped to get working.

    What version of pfSense are you using? Is it a Full Install or a Nano version?

    Are you seeing any other errors? Could you post the full output of that command?

    Can you download the file manually? Maybe a Firewall Rule or Snort is blocking it?

    Try to [  [b]ping ftp.freebsd.org   ] and see if you get a reply?

    [  [b]fetch ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8.3-release/Latest/grepcidr.tbz  ]
                                                          (This path is for the amd64 Release)

    [  [b]pkg_add grepcidr.tbz  ]



  • My cron job is throwing errors.
    Since manually running "/usr/bin/nice -n20 /home/badips/pfiprep >> /home/badips/download.log 2>&1"  gives me an error "Ambiguous output redirect." .
    Leaving out the log output redirect it works fine.  (aka >> …  or > ...)  So I could just remove the log output, but I'd rather keep it. :)

    For the rest, perfect.
    I edited the main script to leave out some lists, and also moved to using the main IR_ lists.
    Also great work on the widget.



  • @foetus:

    My cron job is throwing errors.
    Since manually running "/usr/bin/nice -n20 /home/badips/pfiprep >> /home/badips/download.log 2>&1"  gives me an error "Ambiguous output redirect." .
    Leaving out the log output redirect it works fine.  (aka >> …  or > ...)  So I could just remove the log output, but I'd rather keep it. :)

    for the cron job "/usr/bin/nice -n20 /home/badips/pfiprep >> /home/badips/download.log 2>&1"  works for me, but if i run it via directly from the shell, I get the same error you're seeing

    running from the shell, i type "/usr/bin/nice -n20 /home/badips/pfiprep >> /home/badips/download.log" instead



  • Hmm, I'll just let the cron job do its job and see if it throws an error then.
    Thx for that.


  • Moderator

    Hi Foetus,

    Welcome Aboard!

    Be careful not to run the command from the shell when Cron is scheduled to run or there may be some unexpected behavior. I added some code to avoid having them collide but just be aware to only run the  [  [b]./iprep  ] command when CRON is not scheduled to run or is still in process.

    You can just run  [ [b]./pfiprep ] from the shell and scroll up to see the output. You can also look at the "Daily.log" which shows a summary of the Downloads. Look out for any "FAIL" downloads.

    The High Level function of the script:

    Download Individual List
      Extract IPs
      Save copy to /orig Folder
      Check for Ranges that have 255 IPs and mark a single /24 Range
      Process /24 (Which looks for repeat Offenders in a /24 Range) (max variable) Individual Blocklist Only.
      Duplication Check

    Once all of the Downloads are completed that were scheduled to run:

    The Following is performed Globally on ALL Lists, except for the ones that were marked as "p24=no" on the Collect Line.

    p-Deduplication - Looks for Repeat Offenders that are over the pmax variable regardless of Country Code.

    d-DeDuplication - Looks for Repeat Offenders that are over the dmax variable but uses the Country Code Whitelist function.

    If the Sanity Checks passes, it will create the TIER (Group) lists and perform the "pfctl" commands to update the pfSense Alias Tables.

    If you decide to remove a list, you need to add "remove" after the collect line. When the script runs at its next scheduled run, it will remove the list from the database properly. Don't try to do this manually.

    If you follow the High level steps, when you use the p24 process in d-deduplicaton, it will look for a repeat range of malicious IPs and find all of the Blocklists that have this IP listed.

    The FIRST blocklists get a single x.x.x.0/24 Block and all of the other Lists that have the range are deleted.

    So if a List is removed, and it happens to be a list that had the p24 process and was the first list processed as above, then you have no Blocklists for that range. This will correct itself on when the Lists are re-downloaded but that could be 1-4hrs depending on when the Lists are scheduled to run.

    To get back into Sync, you can run this function:

    [  [b]./pfiprep killdb  ]

    Which will wipe the Database (Settings are not touched) and it will resync the database.

    Out of Curiosity, which Lists did you disable?

    Another Function is to use the "IR_Match" Alias in the Floating Rules as a "Match" Rule. This will show you activity for the IP Ranges that passed the Country Code Whitelist process. Because its a "Match" rule, it will not block, but just log the activity.

    Since I have been running the script, I have not found too many False Positives, but I always recommend not to disable a list but to create a "SAFE Alias" Rule that is defined above the "Block/Reject" Rules. And just add the IPs that you want to allow.

    The Patch for diag_dns.php will also work when looking at the Snort/Suricata Alert Logs.

    If you are running Snort/Suricata, when you click on the "!" ICON to Resolve an IP, you will find that most of the IPs are already listed in the BlockLists. You will also see over time that it will pickup an Alert for an IP but the Blocklists do not have the specific IP but there are several IPs within the same Range that are being Blocked.

    Also in diag_dns.php, there are several IP Reputation Links that can help you determine the Reputation of any Blocked IP before you remove a list, or Add an IP to the SAFE Alias list.

    Let me know if you need any clarification or any other help.



  • The 2>&1 at the end means don't bother emailing me everytime you run this job. It doesn't make any sense running that from the console, since the system wasn't intending to email you anyway.



  • @justsomeone:

    Awesome post, thanks!

    I'm trying to install pfiprep, and am getting an error when installing one of the dependencies.

    $ pkg_add -r grepcidr
    tar: Failed to set default locale
    

    Can someone please tell me what to do to fix this?

    Dont run the command from Diagnostics -> Command.
    Run it from SSH console (shell) or direct console (shell) and it should work.


  • Moderator

    I have added a "PATCH" to make the necessary changes to the pfBlocker Widget so that you don't need to modify that file from the shell.

    If you don't have the pfSense Package "System Patches", it is available in the pfSense System:Packages list under "System Patches"

    Click the "+" Icon to add a new Patch
      Enter a Description (pfBlocker Widget Patch)
      In the Patch Contents Dialog Box - Copy/Paste from my Gist the contents of this
      link below:

    [  [b]https://gist.github.com/BBcan17/67e8c456cb399fbe02ee#file-pfblocker-widget-php_patch  ]

    Keep the other default settings as is.

    Click "Test" and confirm that it can be applied Successfully. Then click "Apply"

    In my Gist, is also a "Patch" to Include the pf IP Reputation Blocklists in the "Firewall Logs "!" Lookup functions.

    [  [b]https://gist.github.com/BBcan17/67e8c456cb399fbe02ee#file-diag_dns-php_patch  ]

    UPDATE:

    When Applying a Patch, the File that you want to modify has to be an original pfSense Version for the Patch to be applied successfully.



  • @BBcan177:

    Out of Curiosity, which Lists did you disable?

    Anything related to Spamhaus. Reason is simple, location where my current test setup is already has this filtered on a higher up level. No point doing the same twice :)
    I was already using a white-list for my classic pfblocker lists, so that will just be extended (which wont be allot since most anti-spam lists are disabled by default). Its not the reason for disabling them :).

    On production networks I'm pretty much gonna use the defaults since most people here and a couple of fellow testers in my area are confirming a low false positives rate.


  • Moderator

    @foetus:

    @BBcan177:

    Out of Curiosity, which Lists did you disable?

    Anything related to Spamhaus. Reason is simple, location where my current test setup is already has this filtered on a higher up level. No point doing the same twice :)
    I was already using a white-list for my classic pfblocker lists, so that will just be extended (which wont be allot since most anti-spam lists are disabled by default). Its not the reason for disabling them :).

    On production networks I'm pretty much gonna use the defaults since most people here and a couple of fellow testers in my area are confirming a low false positives rate.

    The Spamhaus drop and edrop are also included in the ET Lists, but there are still times where one list hasn't sync'd with the others and there are Gaps. I always like to go to the source of the Lists and use those. IBlock has a lot of references to other lists but they just re-package them.

    Its not a lot of extra IPs and I would consider adding them just in case there is any issues from Higher Ups syncing at different times. The Daily Log does help to see how often Lists are updating by looking at the "Count"

    Emerging Threats fwip rules.

    Raw IPs for the firewall block lists. These come from:

    C&C servers identified by Shadowserver (www.shadowserver.org)

    Spam nes identified by Spamhaus (www.spamhaus.org)

    Top Attackers listed by DShield (www.dshield.org)

    More information available at www.emergingthreats.net

    Please submit any feedback or ideas to emerging@emergingthreats.net or the emerging-sigs mailing li

    Spamhaus also has a Botnet Command and Control List, that is Free but it is not readily available. You need to request access to that for your Downloading IP address and use RSYNC. The code to do that is included in the script. I have been working them, to see if they will just release it like the drop and edrop so it makes it easier.

    Its also nice to see that Grepcidr was partially funded by Spamhaus also.

    http://www.spamhaus.org/news/article/714/new-ipv6-cidr-searching-tools-released-grepcidrs

    I have also been working on Integrating the Script to use the New Emerging Threats IQRISK IP Rep lists. Its currently being beta tested by CINO. Unfortunately they want approx $1400 a year per license for it…  :o :o  But If you are a business, I would recommend that over BS locally install Virus Detection Software.  I know that Bill has also been trying to get them to reduce the price for "Home Use"!

    I am also trying to get access to "ShadowServers" Lists, but they are taking forever to approve.

    http://www.shadowserver.org/ccfull.php
    http://www.shadowserver.org/ccdns.php

    If you guys find any bugs in the code or have some "Alternative" Methods to find Offending IPs, I am always open to see if I can make the Script better/more efficient.


  • Moderator

    @BBcan177:

    I have added a "PATCH" to make the necessary changes to the pfBlocker Widget so that you don't need to modify that file from the shell.

    If you don't have the pfSense Package "System Patches", it is available in the pfSense System:Packages list under "System Patches"

    Click the "+" Icon to add a new Patch
      Enter a Description (pfBlocker Widget Patch)
      In the Patch Contents Dialog Box - Copy/Paste from my Gist the contents of this
      link below:

    [  [b]https://gist.github.com/BBcan17/67e8c456cb399fbe02ee#file-pfblocker-widget-php_patch  ]

    Keep the other default settings as is.

    Click "Test" and confirm that it can be applied Successfully. Then click "Apply"

    In my Gist, is also a "Patch" to Include the pf IP Reputation Blocklists in the "Firewall Logs "!" Lookup functions.

    [  [b]https://gist.github.com/BBcan17/67e8c456cb399fbe02ee#file-diag_dns-php_patch  ]

    Cino has found a small bug in one of the patches.

    For the pfBlocker Widget, you will need to click on "Revert" back to previous version.

    Edit the patch and copy/paste the new code from my Gist.

    Than "Save", "Test", and "Apply"

    If you don't Have pfBlocker Installed, you could download the pfIPreputation.widget.php and save that in /usr/local/www/widgets/widgets

    And adding it to the Status:Dashboard page with the "+" icon.

    NOTE:  For [ [b]  diag_dns.php   ]  Don't forget to change the path to the /pf folder
        from the Patch - /YOUR/BLOCKLIST/FOLDER/*   
        Example  /home/USER/pf/*

    UPDATE:

    If you are having issues applying the patches, it could be due to copy/paste issues.

    You can also use the "URL/Commit ID" in the Patch Edit Menu, and use these Links for each of the patches.

    https://gist.githubusercontent.com/BBcan17/67e8c456cb399fbe02ee/raw/f3ca0e1d3dd4a07a21796d033dad06a4ce1cc218/diag_dns.php_PATCH
    (Once you "fetch" the patch, you will need to manually edit the path to the [ /pf folder ]
    example  [  /home/USER/pf/*  ], then click "Test" and then "Apply". Do not paste any code into the "patch contents" the url will do that.

    https://gist.githubusercontent.com/BBcan17/67e8c456cb399fbe02ee/raw/3c3d508cec136788cea6abd98d49d367f9b75b7a/pfBlocker.widget.php_PATCH

    UPDATE2:

    When Applying a Patch, the File that you want to modify has to be an original pfSense Version for the Patch to be applied successfully.



  • and a day later both bing and google are blocked  ;D



  • How are they blocked? Lists or suricata?


  • Moderator

    If you use the MTA list it's blocking those sites.

    That is a malware analysis website and if you take a look at their website you will see how they report their work. Unfortunately Google dns 8.8.8.8 was involved in malware (believe it or not lol)

    Add those IPs to a pass list above the Block Rules.

    If you added the Diag_dns.php patch, you will see which blacklist is blocking what.



  • yea, having issues applying the patch. does not seem to be reading out my lists. out of time today, whitelists ftw.


  • Moderator

    @foetus:

    yea, having issues applying the patch. does not seem to be reading out my lists. out of time today, whitelists ftw.

    You need to edit the path in the patch to point to your pf folder.

    In pfiprep is a pfdir= path

    Make sure the patch has the correct path. If you make changes, you need to revert, make the changes and then re-apply.

    If you are still having difficulties with it, send me a PM.



  • I cried to quickly. Clean reboot and re-applied. Working as intended :)
    And exactly : MTA.txt

    Once again nice job with that script. This will really help out.


  • Moderator

    @foetus:

    I cried to quickly. Clean reboot and re-applied. Working as intended :)
    And exactly : MTA.txt

    Once again nice job with that script. This will really help out.

    I cried sometimes while writing the script!!!  ;D

    Thanks for the feedback. Really appreciate it. If your using Snort/Suricata, the updated diag_dns.php will also help when you click on the "!" Icon to resolv the Alerted IPs.

    I would say that over 90% of the alerts in Snort/Suricata are already being blocked by the Blocklists.



  • hey guys, thanks to everyone for the work here in developing the code, scripts and instructions in this thread.
    Can I check some newb stuff. When I create my aliases, I like to verify they appear right by mouseovering in the Firewall:Rules page, a drop down appears which shows the loaded data sets.
    I noticed all my Alias were 2998 lines long which seemed odd as file sizes and data when scripts were running suggested they were larger than this. I checked one script by loading it in via a pfBlocker list and it shows as much larger (142k entries)….Can I check that the Firewall->Alias was of creating aliases doesn't truncate the data set and its just the display thats limited.
    Sorry if this is a dumb question - some of this is hard to get your head round first timers.


  • Moderator

    Hi irj972,

    If you run this command:

    [  [b]tail -200 download.log  ]

    It will show the last 200 lines of the download.log

    You will see a section that looks something like this:

    Alias Table IP Counts (w/o 1.1.1.1)
    –---------------------------
      281343 total
      145545 /usr/local/www/aliastables/IR_SEC3
      51863 /usr/local/www/aliastables/IR_IB
      30389 /usr/local/www/aliastables/IR_PRI1
      27565 /usr/local/www/aliastables/IR_PRI2
      23143 /usr/local/www/aliastables/IR_SEC1
        2351 /usr/local/www/aliastables/IR_TOR
        391 /usr/local/www/aliastables/IR_SEC2
          57 /usr/local/www/aliastables/android
          39 /usr/local/www/aliastables/ponmocup

    Forget about the bottom two. But the counts in your list should match what you see in the pfSense Rules Count and/or the widget.

    The alias should be in this format:

    [ https://127.0.0.1:[port]/aliastables/IR_PRI1 ]

    You can also check to see that the alias tables in pfSense are Large enough:

    pfSense Table Stats
    –-----------------
    table-entries hard limit 12000000
    Table Usage Count        316805

    You can edit the tables size in Advanced:Firewall/NAT:Firewall Max Table Entries

    One thing that is odd, is that you have "IR_SEC3" listed in the pfIP_Reputation Window below? Can you explain what that window represents?



  • I don't know why, its not likely right but my download.log is zero bytes…..
    I just re-ran the script and it shows the following...

    
    Alias Table IP Counts (w/o 1.1.1.1)
    -----------------------------
      256918 total
      142929 /usr/local/www/aliastables/IR_SEC3
       51854 /usr/local/www/aliastables/IR_IB
       28441 /usr/local/www/aliastables/IR_PRI2
       24370 /usr/local/www/aliastables/IR_SEC1
        4994 /usr/local/www/aliastables/IR_TOR
        3811 /usr/local/www/aliastables/IR_PRI1
         519 /usr/local/www/aliastables/IR_SEC2
    
    Alias Table (Match) IP Counts
    -----------------------------
       21218 /usr/local/www/aliastables/IR_Match
    
    pfSense Table Stats
    -------------------
    table-entries hard limit 10000000
    Table Usage Count        897695
    
    

    I can confirm all lists over 2998 entries (i.e everything other than IR_SEC2) appear to be "capped".

    The IR_SEC3 thing is just a comment where i created the alias.


  • Moderator

    In the script, pfiprep,

    There is a line 210, pfupdate=yes

    Can you confirm if that is set to "yes"

    After the Alias Table list, you will see something that looks like this that shows pfctl Updating the Alias Tables:

    **Updating  [ IR_PRI1 ] [  ET_IPrep ET_Comp ET_Block Spamhaus_drop Spamhaus_edrop Spamhaus_CC CIArmy AbuseZeus AbuseSpyeye AbusePalevo dShield_Top dShield_Block SnortBL ISC_top10 Snort64 ]
    94 addresses added.

    Updating  [ IR_PRI2 ] [  ALIENVAULT Atlas_Attacks Atlas_Botnets Atlas_Fastflux Atlas_Phishing Atlas_Scans Atlas_SSH SRI_Attackers SRI_CC HoneyPot ]
    48 addresses added.
    61 addresses deleted.

    No Updates [ IR_PRI3 ]

    No Updates [ IR_SEC1 ]

    No Updates [ IR_SEC2 ]

    No Updates [ IR_SEC3 ]

    No Updates [ IR_IB ]

    No Updates [ IR_TOR ]

    No Updates [ IR_MAIL ]

    No Updates [ IR_CC ]**



  • this looks right…..



  • it looks right…..

    update was set to yes....

    Updating   [ IR_PRI1 ] [  ET_Comp ET_Block Spamhaus_drop Spamhaus_edrop CIArmy AbuseZeus AbuseSpyeye AbusePalevo dShield_Top dShield_Block SnortBL ISC_top10 ]
    813 addresses added.
    
    Updating   [ IR_PRI2 ] [  ALIENVAULT Atlas_Attacks Atlas_Botnets Atlas_Fastflux Atlas_Phishing Atlas_Scans Atlas_SSH SRI_Attackers SRI_CC HoneyPot ]
    25533 addresses added.
    90 addresses deleted.
    
    No Updates [ IR_PRI3 ]
    
    No Updates [ IR_SEC1 ]
    
    No Updates [ IR_SEC2 ]
    
    No Updates [ IR_SEC3 ]
    
    No Updates [ IR_IB ]
    
    No Updates [ IR_TOR ]
    
    No Updates [ IR_MAIL ]
    
    No Updates [ IR_CC ]
    

    looks like the mouseover thing is wrong, to me at least.


  • Moderator

    Make sure you created the Alias URL Tables correctly. Cino posted his setup here

    https://forum.pfsense.org/index.php?topic=78062.msg427132#msg427132



  • yeah, there's where I copied the setup from, its been a long day so its completely possible Ive screwed something up but it seems basic enough (isn't this where all errors are made, the easy stuff!?)

    each rule looks like this….

    Edit: Hang on: it says use small IP lists under 3000 there….

    and the alias page looks like this.....(not sure why mine shows a section of the data under each rule mind.....version difference with 2.1.4?)


  • Moderator

    We all make mistakes and I think the issue is that you created a "URL" Alias instead of a "URL Table" Alias.

    When its a URL, it has a max amount of IPs that it can hold.

    You should also keep the Description the same as the Alias Name, so its easier to see in the Logs and Rules GUI.

    Easy Fix…  ;)



  • you are right…..knew it would be a newb mistake  :-[
    thanks for your help this evening....and again, thanks for all the work you've put into this.


  • Moderator

    My pleasure! Keep those Bastards out of your Network….  8)



  • Sorry….me again :)

    I've just ploughed through configuring Suricata and have a few issues in the log I was curious about understanding, are these anything I should worry about or just disable these rules?
    FYI: I'm running this with jflsakfja's suggested list and the ETFree list (want to try this out before coughing up some serious coin (compared to VRT home licence) for the ETPro list.

    
     <snip>2/7/2014 -- 18:16:11 - <error>-- [ERRCODE: SC_ERR_WITHIN_INVALID(106)] - within argument "-4" is less than the content length "4" which is invalid, since this will never match.  Invalidating signature
    2/7/2014 -- 18:16:11 - <error>-- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET any <> $HOME_NET any (msg:"ET POLICY hidden zip extension .pif"; flow:established; content:"|50 4b 03 04|"; byte_jump:2,22,relative,little, post_offset +2; content:".pif"; within:-4; reference:url,doc.emergingthreats.net/2001407; classtype:suspicious-filename-detect; sid:2001407; rev:11;)" from file /usr/pbi/suricata-amd64/etc/suricata/suricata_43304_igb0/rules/suricata.rules at line 4808
    2/7/2014 -- 18:16:11 - <error>-- [ERRCODE: SC_ERR_WITHIN_INVALID(106)] - within argument "-4" is less than the content length "4" which is invalid, since this will never match.  Invalidating signature
    2/7/2014 -- 18:16:11 - <error>-- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET any <> $HOME_NET any (msg:"ET POLICY hidden zip extension .scr"; flow:established; content:"|50 4b 03 04|"; byte_jump:2,22,relative,little, post_offset +2; content:".scr"; within:-4; reference:url,doc.emergingthreats.net/2001408; classtype:suspicious-filename-detect; sid:2001408; rev:12;)" from file /usr/pbi/suricata-amd64/etc/suricata/suricata_43304_igb0/rules/suricata.rules at line 4809
    2/7/2014 -- 18:16:13 - <error>-- [ERRCODE: SC_ERR_PCRE_COMPILE(5)] - pcre compile of ""/(obj.data|\object.data).+file\x3A\x2F\x2F127\x2E[0-9]/si"" failed at offset 11: missing opening brace after \o
    2/7/2014 -- 18:16:13 - <error>-- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET $HTTP_PORTS -> $HOME_NET any (msg:"ET WEB_CLIENT Possible Microsoft Internet Explorer Dynamic Object Tag/URLMON Sniffing Cross Domain Information Disclosure Attempt"; flow:established,to_client; content:"obj"; nocase; content:"data"; nocase; within:10; content:"file|3A|//127."; nocase; within:20; pcre:"/(obj.data|\object.data).+file\x3A\x2F\x2F127\x2E[0-9]/si"; reference:url,tools.cisco.com/security/center/viewAlert.x?alertId=19873; reference:url,tools.cisco.com/security/center/viewAlert.x?alertId=20610; reference:url,www.microsoft.com/technet/security/bulletin/ms10-035.mspx; reference:url,www.coresecurity.com/content/internet-explorer-dynamic-object-tag; reference:cve,2010-0255; reference:url,doc.emergingthreats.net/2011695; classtype:attempted-user; sid:2011695; rev:4;)" from file /usr/pbi/suricata-amd64/etc/suricata/suricata_43304_igb0/rules/suricata.rules at line 8069
    2/7/2014 -- 18:16:13 - <error>-- [ERRCODE: SC_ERR_NEGATED_VALUE_IN_PORT_RANGE(56)] - Can't have a negated value in a range.
    2/7/2014 -- 18:16:13 - <error>-- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET [!21:23,!2100,!3535] -> $HOME_NET 1024:65535 (msg:"ET WEB_CLIENT Possible GnuTLS Client ServerHello SessionID Overflow CVE-2014-3466"; flow:established,to_client; content:"|16 03|"; depth:2; byte_test:1,<,4,2; content:"|02|"; distance:3; within:1; content:"|03|"; distance:3; within:1; byte_test:1,<,4,0,relative; byte_test:4,>,1370396981,1,relative; byte_test:4,<,1465091381,1,relative; byte_test:1,>,32,33,relative; reference:url,radare.today/technical-analysis-of-the-gnutls-hello-vulnerability/; reference:cve,2014-3466; classtype:attempted-user; sid:2018537; rev:1;)" from file /usr/pbi/suricata-amd64/etc/suricata/suricata_43304_igb0/rules/suricata.rules at line 8225
    2/7/2014 -- 18:16:18 - <info>-- 2 rule files processed. 14110 rules successfully loaded, 4 rules failed
    2/7/2014 -- 18:16:53 - <info>-- 14119 signatures processed. 16 are IP-only rules, 4158 are inspecting packet payload, 11827 inspect application layer, 77 are decoder event only</info></info></error></error></error></error></error></error></error></error></snip> 
    

    Are there any known issues about using the '+' to clone the interface to another interface, I was trying to clone my WAN to cover my VPN_WAN but had some issues starting? Will investigate further…..

    EDIT: Scratch this.....the interface duplication appears to be working fine, I just needed to let everything settle down.


  • Moderator

    I Have updated the pf IP Reputation Manager Script to version 2.3.3

    You can review the revisions in my GIST.

    https://gist.github.com/BBcan17/67e8c456cb399fbe02ee

    For pfiprep make the changes to your existing file or just overwrite and add your changes as required.

    For pfiprepman, just backup the previous 2.3.2 version and replace with the latest 2.3.3 version.

    CHANGELOG

    ***** Added Support to use the Emerging Threats IQRISK IP Reputation Lists 
          (Requires Subscription)
      ***** Some more of the Lists now support HTTPS downloads, and the collect lines
          have been updated.
      ***** Added a [ [b]./pfiprep killdb dskip  ] function which will reset the database with
          the existing Downloaded Files
      ***** Moved Blocklist.de Blocklist from the Mail Server Section to the Regular 
          Section, as this list has more than Mail Server Blocklists. Refer to INFO URL in the
          script.
      ***** Added a few other Blocklists
      ***** Script can now process IBlock Subscription Lists
      ***** Script can now process SquidBlock lists that are IP based.
      ***** Added "plog=yes" option to Log Errors to the pfSense System Log

    I recommend running

    [ [b] ./pfiprep killdb  ]  with version changes  or  [  [b]./pfiprep killdb dskip  ]

    If you find any Bugs please let me know and I will promptly fix them.

    Feedback is always Welcome!


  • Moderator

    I have Updated the pfIP_Reputation.widget.php file:

    1.  It now displays the Last Update (Date/Time) per Alias Table.

    2.  Displays a Total Blocklist Count

    3. Displays "All Downloads Successful" or it will List any "FAILED" Downloads.

    The updated file can be found in my Gist @
    https://gist.github.com/BBcan17/67e8c456cb399fbe02ee#file-pfip_reputation-widget-php

    The file will need to be saved in the

    /usr/local/www/widgets/widgets    folder    [ [b]pfIP_Reputation.widget.php ]

    Lines 36 and 37 need to be edited to the "Masterfile" and the "Daily.log" file locations.

    From the Status:Dashboard, click on the "+" Icon to add the widget.

    See attached for a screenshot of what the Updated Widget Looks like:

    Previous pfIP_Rep widget at Top
        New - Widget showing (All Downloads Successful)
        New - Widget Showing a Failed Download

    With these changes, you can effectively manage the pfIP_Reputation Manager without needing to use the Shell to see its status. If You have any other suggestions to improve the widget, please let me know.

    ![pfIP_Reputation Widget.png_thumb](/public/imported_attachments/1/pfIP_Reputation Widget.png_thumb)
    ![pfIP_Reputation Widget.png](/public/imported_attachments/1/pfIP_Reputation Widget.png)



  • nice  :)



  • Working. Sadly the no packets counted bug is back for me.
    Existing install without importing back-up xml files. Just added the new aliases and rules as usual. Normally importing back-up firewall rules breaks it..

    Checking log file works, update status also. Just no packets :p (yes yes, quick option, logging option, the whole shambles - rules are working fine, DNS lookup also, just no packets counted on dashboard widget)



  • The rules that are using the Aliases (IR_PRI1, IR_SEC1, etc),  does the Description start with the same name of the Alias? This is how the widget can match the rule to grab the hit count.

    Look at my post https://forum.pfsense.org/index.php?topic=78062.msg427132#msg427132 and take a look at how I labeled the Description field.



  • just adding that it worked for me too after following Cino's method.


  • Moderator

    @Cino:

    The rules that are using the Aliases (IR_PRI1, IR_SEC1, etc),  does the Description start with the same name of the Alias? This is how the widget can match the rule to grab the hit count.

    Look at my post https://forum.pfsense.org/index.php?topic=78062.msg427132#msg427132 and take a look at how I labeled the Description field.

    Thanks Cino, we must be on the same wavelength. I just sent Foetus a PM with the exact same thing!  ;D

    I have to take a look at that regex for the Description matching part. But for now, keep the Rule Description the same as the Alias Name.



  • And idd. Naming the rules anything else breaks it.  :-X
    Oh well, now that we know that.. :)



  • @foetus:

    And idd. Naming the rules anything else breaks it.  :-X
    Oh well, now that we know that.. :)

    The first word in it has to match the alias… after that, you can add whatever you want.



  • @jflsakfja:

    Next up Floating tab:
    Set up a rule but make these changes:

    | Action | Block |
    | Quick | TICKED!!! |
    | Interface | Hold CTRL and click on all interfaces EXCEPT LAN(admin) and SYNC |
    | Direction | any |
    | Source | any |
    | Destination | any |

    I've read your post 3 times and I'm having a difficult time understanding the floating rule.  The default nature of the firewall is to block incoming traffic unless you add a pass rule. As I understand it, floating rules are evaluated first.  So wouldn't this rule always block incoming packets on the interface regardless of the interface rules?


Log in to reply