Taming the beasts… aka suricata blueprint

q54e3w

Sorry….me again :)

I've just ploughed through configuring Suricata and have a few issues in the log I was curious about understanding, are these anything I should worry about or just disable these rules?
FYI: I'm running this with jflsakfja's suggested list and the ETFree list (want to try this out before coughing up some serious coin (compared to VRT home licence) for the ETPro list.


 <snip>2/7/2014 -- 18:16:11 - <error>-- [ERRCODE: SC_ERR_WITHIN_INVALID(106)] - within argument "-4" is less than the content length "4" which is invalid, since this will never match.  Invalidating signature
2/7/2014 -- 18:16:11 - <error>-- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET any <> $HOME_NET any (msg:"ET POLICY hidden zip extension .pif"; flow:established; content:"|50 4b 03 04|"; byte_jump:2,22,relative,little, post_offset +2; content:".pif"; within:-4; reference:url,doc.emergingthreats.net/2001407; classtype:suspicious-filename-detect; sid:2001407; rev:11;)" from file /usr/pbi/suricata-amd64/etc/suricata/suricata_43304_igb0/rules/suricata.rules at line 4808
2/7/2014 -- 18:16:11 - <error>-- [ERRCODE: SC_ERR_WITHIN_INVALID(106)] - within argument "-4" is less than the content length "4" which is invalid, since this will never match.  Invalidating signature
2/7/2014 -- 18:16:11 - <error>-- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET any <> $HOME_NET any (msg:"ET POLICY hidden zip extension .scr"; flow:established; content:"|50 4b 03 04|"; byte_jump:2,22,relative,little, post_offset +2; content:".scr"; within:-4; reference:url,doc.emergingthreats.net/2001408; classtype:suspicious-filename-detect; sid:2001408; rev:12;)" from file /usr/pbi/suricata-amd64/etc/suricata/suricata_43304_igb0/rules/suricata.rules at line 4809
2/7/2014 -- 18:16:13 - <error>-- [ERRCODE: SC_ERR_PCRE_COMPILE(5)] - pcre compile of ""/(obj.data|\object.data).+file\x3A\x2F\x2F127\x2E[0-9]/si"" failed at offset 11: missing opening brace after \o
2/7/2014 -- 18:16:13 - <error>-- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET $HTTP_PORTS -> $HOME_NET any (msg:"ET WEB_CLIENT Possible Microsoft Internet Explorer Dynamic Object Tag/URLMON Sniffing Cross Domain Information Disclosure Attempt"; flow:established,to_client; content:"obj"; nocase; content:"data"; nocase; within:10; content:"file|3A|//127."; nocase; within:20; pcre:"/(obj.data|\object.data).+file\x3A\x2F\x2F127\x2E[0-9]/si"; reference:url,tools.cisco.com/security/center/viewAlert.x?alertId=19873; reference:url,tools.cisco.com/security/center/viewAlert.x?alertId=20610; reference:url,www.microsoft.com/technet/security/bulletin/ms10-035.mspx; reference:url,www.coresecurity.com/content/internet-explorer-dynamic-object-tag; reference:cve,2010-0255; reference:url,doc.emergingthreats.net/2011695; classtype:attempted-user; sid:2011695; rev:4;)" from file /usr/pbi/suricata-amd64/etc/suricata/suricata_43304_igb0/rules/suricata.rules at line 8069
2/7/2014 -- 18:16:13 - <error>-- [ERRCODE: SC_ERR_NEGATED_VALUE_IN_PORT_RANGE(56)] - Can't have a negated value in a range.
2/7/2014 -- 18:16:13 - <error>-- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET [!21:23,!2100,!3535] -> $HOME_NET 1024:65535 (msg:"ET WEB_CLIENT Possible GnuTLS Client ServerHello SessionID Overflow CVE-2014-3466"; flow:established,to_client; content:"|16 03|"; depth:2; byte_test:1,<,4,2; content:"|02|"; distance:3; within:1; content:"|03|"; distance:3; within:1; byte_test:1,<,4,0,relative; byte_test:4,>,1370396981,1,relative; byte_test:4,<,1465091381,1,relative; byte_test:1,>,32,33,relative; reference:url,radare.today/technical-analysis-of-the-gnutls-hello-vulnerability/; reference:cve,2014-3466; classtype:attempted-user; sid:2018537; rev:1;)" from file /usr/pbi/suricata-amd64/etc/suricata/suricata_43304_igb0/rules/suricata.rules at line 8225
2/7/2014 -- 18:16:18 - <info>-- 2 rule files processed. 14110 rules successfully loaded, 4 rules failed
2/7/2014 -- 18:16:53 - <info>-- 14119 signatures processed. 16 are IP-only rules, 4158 are inspecting packet payload, 11827 inspect application layer, 77 are decoder event only</info></info></error></error></error></error></error></error></error></error></snip>

Are there any known issues about using the '+' to clone the interface to another interface, I was trying to clone my WAN to cover my VPN_WAN but had some issues starting? Will investigate further…..

EDIT: Scratch this.....the interface duplication appears to be working fine, I just needed to let everything settle down.

BBcan177

I Have updated the pf IP Reputation Manager Script to version 2.3.3

You can review the revisions in my GIST.

https://gist.github.com/BBcan17/67e8c456cb399fbe02ee

For pfiprep make the changes to your existing file or just overwrite and add your changes as required.

For pfiprepman, just backup the previous 2.3.2 version and replace with the latest 2.3.3 version.

CHANGELOG

***** Added Support to use the Emerging Threats IQRISK IP Reputation Lists
(Requires Subscription)
***** Some more of the Lists now support HTTPS downloads, and the collect lines
have been updated.
***** Added a [ [b]./pfiprep killdb dskip ] function which will reset the database with
the existing Downloaded Files
***** Moved Blocklist.de Blocklist from the Mail Server Section to the Regular
Section, as this list has more than Mail Server Blocklists. Refer to INFO URL in the
script.
***** Added a few other Blocklists
***** Script can now process IBlock Subscription Lists
***** Script can now process SquidBlock lists that are IP based.
***** Added "plog=yes" option to Log Errors to the pfSense System Log

I recommend running

[ [b] ./pfiprep killdb ] with version changes or [ [b]./pfiprep killdb dskip ]

If you find any Bugs please let me know and I will promptly fix them.

Feedback is always Welcome!

BBcan177

I have Updated the pfIP_Reputation.widget.php file:

1. It now displays the Last Update (Date/Time) per Alias Table.

2. Displays a Total Blocklist Count

3. Displays "All Downloads Successful" or it will List any "FAILED" Downloads.

The updated file can be found in my Gist @
https://gist.github.com/BBcan17/67e8c456cb399fbe02ee#file-pfip_reputation-widget-php

The file will need to be saved in the

/usr/local/www/widgets/widgets folder [ [b]pfIP_Reputation.widget.php ]

Lines 36 and 37 need to be edited to the "Masterfile" and the "Daily.log" file locations.

From the Status:Dashboard, click on the "+" Icon to add the widget.

See attached for a screenshot of what the Updated Widget Looks like:

Previous pfIP_Rep widget at Top
New - Widget showing (All Downloads Successful)
New - Widget Showing a Failed Download

With these changes, you can effectively manage the pfIP_Reputation Manager without needing to use the Shell to see its status. If You have any other suggestions to improve the widget, please let me know.

![pfIP_Reputation Widget.png_thumb](/public/imported_attachments/1/pfIP_Reputation Widget.png_thumb)
![pfIP_Reputation Widget.png](/public/imported_attachments/1/pfIP_Reputation Widget.png)

justsomeone

nice :)

foetus

Working. Sadly the no packets counted bug is back for me.
Existing install without importing back-up xml files. Just added the new aliases and rules as usual. Normally importing back-up firewall rules breaks it..

Checking log file works, update status also. Just no packets :p (yes yes, quick option, logging option, the whole shambles - rules are working fine, DNS lookup also, just no packets counted on dashboard widget)

Cino

The rules that are using the Aliases (IR_PRI1, IR_SEC1, etc), does the Description start with the same name of the Alias? This is how the widget can match the rule to grab the hit count.

Look at my post https://forum.pfsense.org/index.php?topic=78062.msg427132#msg427132 and take a look at how I labeled the Description field.

q54e3w

just adding that it worked for me too after following Cino's method.

BBcan177

@Cino:

The rules that are using the Aliases (IR_PRI1, IR_SEC1, etc), does the Description start with the same name of the Alias? This is how the widget can match the rule to grab the hit count.

Look at my post https://forum.pfsense.org/index.php?topic=78062.msg427132#msg427132 and take a look at how I labeled the Description field.

Thanks Cino, we must be on the same wavelength. I just sent Foetus a PM with the exact same thing! ;D

I have to take a look at that regex for the Description matching part. But for now, keep the Rule Description the same as the Alias Name.

foetus

And idd. Naming the rules anything else breaks it. :-X
Oh well, now that we know that.. :)

Cino

@foetus:

And idd. Naming the rules anything else breaks it. :-X
Oh well, now that we know that.. :)

The first word in it has to match the alias… after that, you can add whatever you want.

tikcolg

@jflsakfja:

Next up Floating tab:
Set up a rule but make these changes:

| Action | Block |
| Quick | TICKED!!! |
| Interface | Hold CTRL and click on all interfaces EXCEPT LAN(admin) and SYNC |
| Direction | any |
| Source | any |
| Destination | any |

I've read your post 3 times and I'm having a difficult time understanding the floating rule. The default nature of the firewall is to block incoming traffic unless you add a pass rule. As I understand it, floating rules are evaluated first. So wouldn't this rule always block incoming packets on the interface regardless of the interface rules?

Mr. Jingles

Aside from extremely enjoying your funny writing style ( ;D ) I also think you can't get too much karma for all that you are doing with regards to helping people set up Suricata (and before that: Snort).

Thank you, secret man :P

A Former User

@tikcolg:

I've read your post 3 times and I'm having a difficult time understanding the floating rule. The default nature of the firewall is to block incoming traffic unless you add a pass rule. As I understand it, floating rules are evaluated first. So wouldn't this rule always block incoming packets on the interface regardless of the interface rules?

The giant red warning under that rule should explain it. It's a rule that will ONLY apply to traffic destined for pfsense's ports. By default pfsense could open up the webgui to an undesired interface, which will not be covered by the default rule. Depending on how far away you sit from the fan, it leads to varying amounts of brown stuff raining down when "it" hits the fan. ;)

@Hollander: Finished today analyzing logs for 3,997,696 IP addresses. Those (almost) 4mil IPs were what tripped up our security systems in the first 6 months of this year. Needless to say they gained a magical place in my "Permanently Banned" Hall of Shame.

If pfsense/suricata/other logs can help identify 4mil malicious IPs, then sure as hell they deserve all the support we can give them.

@all: List has had a couple of updates don't forget to check it regularly. I'm trying to add descriptions when I edit the list, so it's obvious what I added/removed/changed, without needing to go through the entire list.

tikcolg

@jflsakfja:

@tikcolg:

I've read your post 3 times and I'm having a difficult time understanding the floating rule. The default nature of the firewall is to block incoming traffic unless you add a pass rule. As I understand it, floating rules are evaluated first. So wouldn't this rule always block incoming packets on the interface regardless of the interface rules?

The giant red warning under that rule should explain it. It's a rule that will ONLY apply to traffic destined for pfsense's ports. By default pfsense could open up the webgui to an undesired interface, which will not be covered by the default rule. Depending on how far away you sit from the fan, it leads to varying amounts of brown stuff raining down when "it" hits the fan. ;)

@Hollander: Finished today analyzing logs for 3,997,696 IP addresses. Those (almost) 4mil IPs were what tripped up our security systems in the first 6 months of this year. Needless to say they gained a magical place in my "Permanently Banned" Hall of Shame.

If pfsense/suricata/other logs can help identify 4mil malicious IPs, then sure as hell they deserve all the support we can give them.

@all: List has had a couple of updates don't forget to check it regularly. I'm trying to add descriptions when I edit the list, so it's obvious what I added/removed/changed, without needing to go through the entire list.

jflsakfja,

Thank you for the clarification! Makes sense now.

Mr. Jingles

An intermezzo question: did anybody try to print this thread? I wanted to start working on this, and print it to study it thoroughly first. The printing leads to iny tiny small text on the paper, not readable. I tried this from three computers, 3 browsers, all the same.

Is this a forum software thing? Would an admin perhaps mind to verify?

(text in red so admin notices it)

Thank you ;D

A Former User

Never tried printing anything from around here, but I've been getting weird errors when posting replies. It's time to abandon the clusterf*** that is the current forum software*. I believe it will also be the solution to the black hole creation problem as well, and who knows, maybe one day we too can edit our old posts. One can only hope.

Notes:

This is my personal opinion and I'm allowed to say it based on provisions in my country's constitution, as well as international human rights treaties.

Disclaimer:
If you are in any way related to the current clusterf*** forum software, then you should not be offended by a single person's opinion of it. If it is the majority's opinion of it though, that means that the current forum software is indeed a clusterf***, and in that case you should seriously consider abandoning the project and letting it die the slow and horrible death it deserves.

Mr. Jingles

@jflsakfja:

Never tried printing anything from around here, but I've been getting weird errors when posting replies. It's time to abandon the clusterf*** that is the current forum software*. I believe it will also be the solution to the black hole creation problem as well, and who knows, maybe one day we too can edit our old posts. One can only hope.

Notes:

This is my personal opinion and I'm allowed to say it based on provisions in my country's constitution, as well as international human rights treaties.

Disclaimer:
If you are in any way related to the current clusterf*** forum software, then you should not be offended by a single person's opinion of it. If it is the majority's opinion of it though, that means that the current forum software is indeed a clusterf***, and in that case you should seriously consider abandoning the project and letting it die the slow and horrible death it deserves.

;D ;D ;D

(Love your funny writing style :P ).

Your royalness, I am currently clusterf*cking around on my box with your tuto. For one, the floating rule blocks everything out, so I had to disable that. I've noticed in a follow up post that you wrote the floating rule was meant to prevent access to the pfSense GUI, but I think that is not what your initial instruction does (but again, keep in mind I will be the eternal noob).

A more serious question for me is, while I am now currently looking at the script: which are the lines I need to comment out to select lists? I am looking but honestly have no clue :-[ Could you give an example of a line that contains a list? Is it the ones all the way at the bottom?

Thank you ;D

A Former User

Sorry about that floating rule, I later added the giant red warning. The floating rule should only apply to the pfsense's ports, and that shouldn't block any other traffic.

As far as the lists go, it's near the end of the script. There are instructions in the script to enable/disable the lists. Enabling a list is usually removing the # in front of the line containing the list.

Mr. Jingles

:'( :-[ ???

( >:( )

How on earth can this be possible? I even logged out and logged in again. Where on earth does it get this directory from?

It is not difficult to remain the eternal noob when this happens ( ;D >:( )

[b]EDIT: it appears it hadn't saved the first: userfolder=/home/badips

Probably because I had the file open in both WinSCP and via the Diagnostics/edit file. I saved it in the latter, but apparently since it was also open in WinSCP it didn't tell me it couldn't write but simply said nothing.

pfsense_pfiprep.jpg_thumb

Mr. Jingles

@jflsakfja:

Sorry about that floating rule, I later added the giant red warning. The floating rule should only apply to the pfsense's ports, and that shouldn't block any other traffic.

As far as the lists go, it's near the end of the script. There are instructions in the script to enable/disable the lists. Enabling a list is usually removing the # in front of the line containing the list.

Thanks Jflsakfja ;D

Next up Floating tab:
Set up a rule but make these changes:

| Action | Block |
| Quick | TICKED!!! |
| Interface | Hold CTRL and click on all interfaces EXCEPT LAN(admin) and SYNC |
| Direction | any |
| Source | any |
| Destination | any |

DON'T CHANGE DESTINATION PORT RANGE!!!

So when I add new floating rule, the above reads to block any source, any direction, to any destination (I left the ports to 'other', which is the default when I created a new floating rule, so I didn't change it per the red text), effectively blocking all LAN out (I think, at least when I disabled the rule I had internet access again).

Remember: eternal noobs will be eternal noobs ;D