Taming the beasts… aka suricata blueprint

BBcan177

@foetus:

yea, having issues applying the patch. does not seem to be reading out my lists. out of time today, whitelists ftw.

You need to edit the path in the patch to point to your pf folder.

In pfiprep is a pfdir= path

Make sure the patch has the correct path. If you make changes, you need to revert, make the changes and then re-apply.

If you are still having difficulties with it, send me a PM.

foetus

I cried to quickly. Clean reboot and re-applied. Working as intended :)
And exactly : MTA.txt

Once again nice job with that script. This will really help out.

BBcan177

@foetus:

I cried to quickly. Clean reboot and re-applied. Working as intended :)
And exactly : MTA.txt

Once again nice job with that script. This will really help out.

I cried sometimes while writing the script!!! ;D

Thanks for the feedback. Really appreciate it. If your using Snort/Suricata, the updated diag_dns.php will also help when you click on the "!" Icon to resolv the Alerted IPs.

I would say that over 90% of the alerts in Snort/Suricata are already being blocked by the Blocklists.

q54e3w

hey guys, thanks to everyone for the work here in developing the code, scripts and instructions in this thread.
Can I check some newb stuff. When I create my aliases, I like to verify they appear right by mouseovering in the Firewall:Rules page, a drop down appears which shows the loaded data sets.
I noticed all my Alias were 2998 lines long which seemed odd as file sizes and data when scripts were running suggested they were larger than this. I checked one script by loading it in via a pfBlocker list and it shows as much larger (142k entries)….Can I check that the Firewall->Alias was of creating aliases doesn't truncate the data set and its just the display thats limited.
Sorry if this is a dumb question - some of this is hard to get your head round first timers.

BBcan177

Hi irj972,

If you run this command:

[ [b]tail -200 download.log ]

It will show the last 200 lines of the download.log

You will see a section that looks something like this:

Alias Table IP Counts (w/o 1.1.1.1)
–---------------------------
281343 total
145545 /usr/local/www/aliastables/IR_SEC3
51863 /usr/local/www/aliastables/IR_IB
30389 /usr/local/www/aliastables/IR_PRI1
27565 /usr/local/www/aliastables/IR_PRI2
23143 /usr/local/www/aliastables/IR_SEC1
2351 /usr/local/www/aliastables/IR_TOR
391 /usr/local/www/aliastables/IR_SEC2
57 /usr/local/www/aliastables/android
39 /usr/local/www/aliastables/ponmocup

Forget about the bottom two. But the counts in your list should match what you see in the pfSense Rules Count and/or the widget.

The alias should be in this format:

[ https://127.0.0.1:[port]/aliastables/IR_PRI1 ]

You can also check to see that the alias tables in pfSense are Large enough:

pfSense Table Stats
–-----------------
table-entries hard limit 12000000
Table Usage Count 316805

You can edit the tables size in Advanced:Firewall/NAT:Firewall Max Table Entries

One thing that is odd, is that you have "IR_SEC3" listed in the pfIP_Reputation Window below? Can you explain what that window represents?

q54e3w

I don't know why, its not likely right but my download.log is zero bytes…..
I just re-ran the script and it shows the following...


Alias Table IP Counts (w/o 1.1.1.1)
-----------------------------
  256918 total
  142929 /usr/local/www/aliastables/IR_SEC3
   51854 /usr/local/www/aliastables/IR_IB
   28441 /usr/local/www/aliastables/IR_PRI2
   24370 /usr/local/www/aliastables/IR_SEC1
    4994 /usr/local/www/aliastables/IR_TOR
    3811 /usr/local/www/aliastables/IR_PRI1
     519 /usr/local/www/aliastables/IR_SEC2

Alias Table (Match) IP Counts
-----------------------------
   21218 /usr/local/www/aliastables/IR_Match

pfSense Table Stats
-------------------
table-entries hard limit 10000000
Table Usage Count        897695

I can confirm all lists over 2998 entries (i.e everything other than IR_SEC2) appear to be "capped".

The IR_SEC3 thing is just a comment where i created the alias.

BBcan177

In the script, pfiprep,

There is a line 210, pfupdate=yes

Can you confirm if that is set to "yes"

After the Alias Table list, you will see something that looks like this that shows pfctl Updating the Alias Tables:

**Updating [ IR_PRI1 ] [ ET_IPrep ET_Comp ET_Block Spamhaus_drop Spamhaus_edrop Spamhaus_CC CIArmy AbuseZeus AbuseSpyeye AbusePalevo dShield_Top dShield_Block SnortBL ISC_top10 Snort64 ]
94 addresses added.

Updating [ IR_PRI2 ] [ ALIENVAULT Atlas_Attacks Atlas_Botnets Atlas_Fastflux Atlas_Phishing Atlas_Scans Atlas_SSH SRI_Attackers SRI_CC HoneyPot ]
48 addresses added.
61 addresses deleted.

No Updates [ IR_PRI3 ]

No Updates [ IR_SEC1 ]

No Updates [ IR_SEC2 ]

No Updates [ IR_SEC3 ]

No Updates [ IR_IB ]

No Updates [ IR_TOR ]

No Updates [ IR_MAIL ]

No Updates [ IR_CC ]**

q54e3w

this looks right…..

q54e3w

it looks right…..

update was set to yes....

Updating   [ IR_PRI1 ] [  ET_Comp ET_Block Spamhaus_drop Spamhaus_edrop CIArmy AbuseZeus AbuseSpyeye AbusePalevo dShield_Top dShield_Block SnortBL ISC_top10 ]
813 addresses added.

Updating   [ IR_PRI2 ] [  ALIENVAULT Atlas_Attacks Atlas_Botnets Atlas_Fastflux Atlas_Phishing Atlas_Scans Atlas_SSH SRI_Attackers SRI_CC HoneyPot ]
25533 addresses added.
90 addresses deleted.

No Updates [ IR_PRI3 ]

No Updates [ IR_SEC1 ]

No Updates [ IR_SEC2 ]

No Updates [ IR_SEC3 ]

No Updates [ IR_IB ]

No Updates [ IR_TOR ]

No Updates [ IR_MAIL ]

No Updates [ IR_CC ]

looks like the mouseover thing is wrong, to me at least.

BBcan177

Make sure you created the Alias URL Tables correctly. Cino posted his setup here

https://forum.pfsense.org/index.php?topic=78062.msg427132#msg427132

q54e3w

yeah, there's where I copied the setup from, its been a long day so its completely possible Ive screwed something up but it seems basic enough (isn't this where all errors are made, the easy stuff!?)

each rule looks like this….

Edit: Hang on: it says use small IP lists under 3000 there….

and the alias page looks like this.....(not sure why mine shows a section of the data under each rule mind.....version difference with 2.1.4?)

BBcan177

We all make mistakes and I think the issue is that you created a "URL" Alias instead of a "URL Table" Alias.

When its a URL, it has a max amount of IPs that it can hold.

You should also keep the Description the same as the Alias Name, so its easier to see in the Logs and Rules GUI.

Easy Fix… ;)

q54e3w

you are right…..knew it would be a newb mistake :-[
thanks for your help this evening....and again, thanks for all the work you've put into this.

BBcan177

My pleasure! Keep those Bastards out of your Network…. 8)

q54e3w

Sorry….me again :)

I've just ploughed through configuring Suricata and have a few issues in the log I was curious about understanding, are these anything I should worry about or just disable these rules?
FYI: I'm running this with jflsakfja's suggested list and the ETFree list (want to try this out before coughing up some serious coin (compared to VRT home licence) for the ETPro list.


 <snip>2/7/2014 -- 18:16:11 - <error>-- [ERRCODE: SC_ERR_WITHIN_INVALID(106)] - within argument "-4" is less than the content length "4" which is invalid, since this will never match.  Invalidating signature
2/7/2014 -- 18:16:11 - <error>-- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET any <> $HOME_NET any (msg:"ET POLICY hidden zip extension .pif"; flow:established; content:"|50 4b 03 04|"; byte_jump:2,22,relative,little, post_offset +2; content:".pif"; within:-4; reference:url,doc.emergingthreats.net/2001407; classtype:suspicious-filename-detect; sid:2001407; rev:11;)" from file /usr/pbi/suricata-amd64/etc/suricata/suricata_43304_igb0/rules/suricata.rules at line 4808
2/7/2014 -- 18:16:11 - <error>-- [ERRCODE: SC_ERR_WITHIN_INVALID(106)] - within argument "-4" is less than the content length "4" which is invalid, since this will never match.  Invalidating signature
2/7/2014 -- 18:16:11 - <error>-- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET any <> $HOME_NET any (msg:"ET POLICY hidden zip extension .scr"; flow:established; content:"|50 4b 03 04|"; byte_jump:2,22,relative,little, post_offset +2; content:".scr"; within:-4; reference:url,doc.emergingthreats.net/2001408; classtype:suspicious-filename-detect; sid:2001408; rev:12;)" from file /usr/pbi/suricata-amd64/etc/suricata/suricata_43304_igb0/rules/suricata.rules at line 4809
2/7/2014 -- 18:16:13 - <error>-- [ERRCODE: SC_ERR_PCRE_COMPILE(5)] - pcre compile of ""/(obj.data|\object.data).+file\x3A\x2F\x2F127\x2E[0-9]/si"" failed at offset 11: missing opening brace after \o
2/7/2014 -- 18:16:13 - <error>-- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET $HTTP_PORTS -> $HOME_NET any (msg:"ET WEB_CLIENT Possible Microsoft Internet Explorer Dynamic Object Tag/URLMON Sniffing Cross Domain Information Disclosure Attempt"; flow:established,to_client; content:"obj"; nocase; content:"data"; nocase; within:10; content:"file|3A|//127."; nocase; within:20; pcre:"/(obj.data|\object.data).+file\x3A\x2F\x2F127\x2E[0-9]/si"; reference:url,tools.cisco.com/security/center/viewAlert.x?alertId=19873; reference:url,tools.cisco.com/security/center/viewAlert.x?alertId=20610; reference:url,www.microsoft.com/technet/security/bulletin/ms10-035.mspx; reference:url,www.coresecurity.com/content/internet-explorer-dynamic-object-tag; reference:cve,2010-0255; reference:url,doc.emergingthreats.net/2011695; classtype:attempted-user; sid:2011695; rev:4;)" from file /usr/pbi/suricata-amd64/etc/suricata/suricata_43304_igb0/rules/suricata.rules at line 8069
2/7/2014 -- 18:16:13 - <error>-- [ERRCODE: SC_ERR_NEGATED_VALUE_IN_PORT_RANGE(56)] - Can't have a negated value in a range.
2/7/2014 -- 18:16:13 - <error>-- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET [!21:23,!2100,!3535] -> $HOME_NET 1024:65535 (msg:"ET WEB_CLIENT Possible GnuTLS Client ServerHello SessionID Overflow CVE-2014-3466"; flow:established,to_client; content:"|16 03|"; depth:2; byte_test:1,<,4,2; content:"|02|"; distance:3; within:1; content:"|03|"; distance:3; within:1; byte_test:1,<,4,0,relative; byte_test:4,>,1370396981,1,relative; byte_test:4,<,1465091381,1,relative; byte_test:1,>,32,33,relative; reference:url,radare.today/technical-analysis-of-the-gnutls-hello-vulnerability/; reference:cve,2014-3466; classtype:attempted-user; sid:2018537; rev:1;)" from file /usr/pbi/suricata-amd64/etc/suricata/suricata_43304_igb0/rules/suricata.rules at line 8225
2/7/2014 -- 18:16:18 - <info>-- 2 rule files processed. 14110 rules successfully loaded, 4 rules failed
2/7/2014 -- 18:16:53 - <info>-- 14119 signatures processed. 16 are IP-only rules, 4158 are inspecting packet payload, 11827 inspect application layer, 77 are decoder event only</info></info></error></error></error></error></error></error></error></error></snip>

Are there any known issues about using the '+' to clone the interface to another interface, I was trying to clone my WAN to cover my VPN_WAN but had some issues starting? Will investigate further…..

EDIT: Scratch this.....the interface duplication appears to be working fine, I just needed to let everything settle down.

BBcan177

I Have updated the pf IP Reputation Manager Script to version 2.3.3

You can review the revisions in my GIST.

https://gist.github.com/BBcan17/67e8c456cb399fbe02ee

For pfiprep make the changes to your existing file or just overwrite and add your changes as required.

For pfiprepman, just backup the previous 2.3.2 version and replace with the latest 2.3.3 version.

CHANGELOG

***** Added Support to use the Emerging Threats IQRISK IP Reputation Lists
(Requires Subscription)
***** Some more of the Lists now support HTTPS downloads, and the collect lines
have been updated.
***** Added a [ [b]./pfiprep killdb dskip ] function which will reset the database with
the existing Downloaded Files
***** Moved Blocklist.de Blocklist from the Mail Server Section to the Regular
Section, as this list has more than Mail Server Blocklists. Refer to INFO URL in the
script.
***** Added a few other Blocklists
***** Script can now process IBlock Subscription Lists
***** Script can now process SquidBlock lists that are IP based.
***** Added "plog=yes" option to Log Errors to the pfSense System Log

I recommend running

[ [b] ./pfiprep killdb ] with version changes or [ [b]./pfiprep killdb dskip ]

If you find any Bugs please let me know and I will promptly fix them.

Feedback is always Welcome!

BBcan177

I have Updated the pfIP_Reputation.widget.php file:

1. It now displays the Last Update (Date/Time) per Alias Table.

2. Displays a Total Blocklist Count

3. Displays "All Downloads Successful" or it will List any "FAILED" Downloads.

The updated file can be found in my Gist @
https://gist.github.com/BBcan17/67e8c456cb399fbe02ee#file-pfip_reputation-widget-php

The file will need to be saved in the

/usr/local/www/widgets/widgets folder [ [b]pfIP_Reputation.widget.php ]

Lines 36 and 37 need to be edited to the "Masterfile" and the "Daily.log" file locations.

From the Status:Dashboard, click on the "+" Icon to add the widget.

See attached for a screenshot of what the Updated Widget Looks like:

Previous pfIP_Rep widget at Top
New - Widget showing (All Downloads Successful)
New - Widget Showing a Failed Download

With these changes, you can effectively manage the pfIP_Reputation Manager without needing to use the Shell to see its status. If You have any other suggestions to improve the widget, please let me know.

![pfIP_Reputation Widget.png_thumb](/public/imported_attachments/1/pfIP_Reputation Widget.png_thumb)
![pfIP_Reputation Widget.png](/public/imported_attachments/1/pfIP_Reputation Widget.png)

justsomeone

nice :)

foetus

Working. Sadly the no packets counted bug is back for me.
Existing install without importing back-up xml files. Just added the new aliases and rules as usual. Normally importing back-up firewall rules breaks it..

Checking log file works, update status also. Just no packets :p (yes yes, quick option, logging option, the whole shambles - rules are working fine, DNS lookup also, just no packets counted on dashboard widget)

Cino

The rules that are using the Aliases (IR_PRI1, IR_SEC1, etc), does the Description start with the same name of the Alias? This is how the widget can match the rule to grab the hit count.

Look at my post https://forum.pfsense.org/index.php?topic=78062.msg427132#msg427132 and take a look at how I labeled the Description field.