Taming the beasts… aka suricata blueprint

The main problem I see is that you can't edit the old posts. I would love to add more information to the topic as things move forward/things change, but I simply can't do so in an organized way. Expecting everyone to go through 100 pages of posts is not exactly user intuitive ;)

I also enjoy seeing people take over the "technical support"/troubleshooting for other users. It means at least a few understood what the topic is trying to do, how to do it, and are willing to contribute to it ;D. The downside to that, is that updates to the topic get pushed further back in the thread, which takes us back to the "trying to keep things organized" subject. Having the main topic be outside the thread would (IMHO) help with keeping discussions/insulting-"innocent"-passers-by organized. For example, we could have a "ridicule the industry leaders week" (*see note) without affecting the actual guides.

I'm looking at you Google, time to take your own advise** and migrate away from SHA-1 certificates on your selected sites, eg www.google.com.cy (at time of writing:PKCS #1 SHA-1 With RSA Encryption). Especially when you yourself are a CA and can just sign a new certificate with no cost, other than the 2 minutes it takes to generate/re-issue/sign a new certificate and change/revoke the old certificate. Or you can take my advice, if you think that your depreciation policy is adequate. I've been screaming since March (this year) that everybody had plenty of time to migrate away from the old (and weak) certificates, time to force everyone to move on. As usual, no industry leader should listen to the "who-the-f***-are-you" random bloke on the Internet.

**http://googleonlinesecurity.blogspot.com/2014/09/gradually-sunsetting-sha-1.html which brings us to ***

***9 years is indeed a lot of time. That's why for the past few months I've been literally (not just using the word without any context), literally SCREAMING at the CAs to force everyone off the old certificates already FFS (my exact words). which takes us to ****

**** Comodo just recently (after the Heartbleed security feature*****, and after 2 all round re-issues) has finally decided to force SHA-2 certificates.

***** In open source, bugs are not bugs. They are features. A bug introduced in the most respected (previous to the Heartbleed bug, now (as the Cypriot saying goes) not even dogs will piss on it) cryptographic software would imply that someone was directly paid to put it there. Especially when the ZOMGWTFNEW1337FEATURE is not needed. We wouldn't go around accusing anyone of getting paid to subvert security software now, would we? Of course not, don't be silly. The bloke accidentally introduced a "spil your guts out" feature. See? Only open source can offer you 2in1 features****(something, lost count)

****(something, lost count) to RMS and gang: I'm a die-hard open source supporter.

Cino

Here's a laugh for you…

So I'm browsing around http://suricata-ids.org and click on their News - Planet link http://planet.suricata-ids.org/


TCP  Potentially Bad Traffic  96.43.130.12 80  x.x.x.x 38263  1:2100498  GPL ATTACK_RESPONSE id check returned root

gotta love it

@Cino:

Here's a laugh for you…

So I'm browsing around http://suricata-ids.org and click on their News - Planet link http://planet.suricata-ids.org/
TCP  Potentially Bad Traffic  96.43.130.12 80  x.x.x.x 38263  1:2100498  GPL ATTACK_RESPONSE id check returned root
gotta love it

Analyzing the packets just before the website was blocked brings us to this: (code tags not added because they interfere with my red emphasis)
–---------------------------------------------------------------------------
"Lets kick it up notch.....

We want to search through -

all the generated alerts that have
a printable payload data
that have the following string: uid=0(root)

Easy, here is the query:

payload_printable:"uid=0(root)"

You should enter it like this in Kibana:
[–---------------------------------------------------------------------------
The rule says this:

msg:"GPL ATTACK_RESPONSE id check returned root"; content:"uid=0|28|root|29|"; classtype:bad-unknown; sid:2100498; rev:7;

The rule is not an FP rule, but a DANGEROUS FP rule. Rules based on plaintext should be banned by legislation. They are the kind of rules that wake you up in the middle of the night when it starts hitting the fan.

I'll add the rule to the list, as well as make the small correction you pointed out on github.

While we are at it, does anyone have anything to say on the proposed way forward as mentioned in a previous post of mine?](http://2.bp.blogspot.com/-u81HF0TVrr4/U_C1BmpQdwI/AAAAAAAAAhU/dJdR8iAQk-k/s1600)

Cino

I didn't have time to look at the packet capture but I will over the weekend. You explaining it is a good example on how rules are triggered by Suricata.

I think your proposed way going forward will work. Maybe start a new thread linking to github while posting links to this thread and the snort one for background info. Ideally it would be nice if we could just edit post like every other forum out there…

BBcan177

@Cino:

Here's a laugh for you…

So I'm browsing around http://suricata-ids.org and click on their News - Planet link http://planet.suricata-ids.org/
TCP  Potentially Bad Traffic  96.43.130.12 80  x.x.x.x 38263  1:2100498  GPL ATTACK_RESPONSE id check returned root
gotta love it

I have similar Alerts when Emerging Threats or Snort posts details of a rule in any of their Newsletters.

Instead of suppressing SIDs, i would probably suppress these Sites to avoid FPs.

bmeeks

What I would like to do is migrate both the Suricata and Snort packages to a new paradigm where you have alerts and blocks, but not a block from every alert like you have today. In a true IPS, you manually edit rules to change the "alert" action to "drop" only for those rules that detect really bad threats. For other things a simple alert to notify you is enough. However, in both Suricata and Snort today on pfSense, the output plugin that handles blocking simply keys off every alert. So any alert results in a block (unless the offending IP is in a Pass List).

I have been running some ideas around in my head on a method for presenting this new paradigm in the GUI. I am thinking about a series of behavior options the admin can choose from. Maybe something along these lines:

1. Legacy behavior where every alert results in a block unless the offender is in a Pass List. This is how things work today.

2. Only blocking when the PRIORITY field in the associated rule is less than or equal to a set value. For example, block on Priority 1 alerts but only alert for Priority 2 and 3 alerts. Still not sure how useful this option would really be, nor how reliable the Priority field values are.

3. A more conventional IPS setup where all rules are only alerts (with no blocks) unless the admin explicitly changes the rule action from "alert" to "drop". This could be handled fairly easily using the new SID MGMT tab feature. It would still require the admin to do a lot of work, though.

4. A refinement of #3 above where choosing one of the Snort VRT IPS policy settings results in blocks from all alerts originating from any of the IPS Policy rules, while other rules (non-IPS policy ones) simply alert. Of course the admin would still have the ability to manually edit any of the other rules to block (drop) by changing the action keyword from "alert" to "drop".

This would set both packages up to be ready to implement true IPS mode when the pfSense kernel is ready to support it. As both packages exist today where any alert results in a block, things could be a pain to administer if those blocks were true inline traffic drops.

I would like the community's input on the above. What changes would you like to see in the blocking behavior?

Thanks,
Bill

@bmeeks:

1. Legacy behavior where every alert results in a block unless the offender is in a Pass List. This is how things work today.

That would be a nice option.
@bmeeks:

2. Only blocking when the PRIORITY field in the associated rule is less than or equal to a set value. For example, block on Priority 1 alerts but only alert for Priority 2 and 3 alerts. Still not sure how useful this option would really be, nor how reliable the Priority field values are.

Not at all useful, since I've seen priorities that really don't match the rules.
@bmeeks:

3. A more conventional IPS setup where all rules are only alerts (with no blocks) unless the admin explicitly changes the rule action from "alert" to "drop". This could be handled fairly easily using the new SID MGMT tab feature. It would still require the admin to do a lot of work, though.

If you turn that around, I'd love to have the functionality. Instead of all rules being alert and you had to change them to drop, I would prefer all rules to be drop and you can change them to alert, if need be. Makes a lot more sense when the inline parts are finally available.

That's what I would like to see, alert/blocking wise. I didn't even gave it much thought, until I saw your post.

What I would like though (WANTED FEATURE ALERT!) is a way to turn off the category + priority from the syslog logs. It's driving me (and our loggers) insane. Every IDS/IPS/firewall log should be:
Source IP - Source Port - Destination IP - Destination Port - Protocol - Rule ID - Reason for blocking/dropping/rule explanation.

Warning:RANT!
Anything more than that is, pardon my Greek, BS. Having the line wrap at just the exact point where the IP (and splitting the IP across the screen) is annoying to say the least ;D. I do remember you mentioning that the logging facility is taken straight from the binary (which BTW, everyone reading this* go here:https://redmine.openinfosecfoundation.org/issues/809, then stalk** the developers until they fix it. Bonus points if you print out a large banner and display it in public). That would require the upstream devs to fix it, correct? While they are at it, they can fix the bug linked above. When the time comes (and I can sign a note stating it WILL come), and it hits the fan, everybody will run around screaming "IT WAS THE LOGGING BUG REPORTED OVER A YEAR AGO". Just like the recent POODLE attack. Nobody pays any attention to the guy screaming "stop using an (almost) 20 year old cryptographic protocol. It's proven broken time and time again!". NOPE. Everybody pays attention to Google's engineers when they publish a paper asking everyone to stop using a broken protocol. Doesn't matter, I pointed and laughed because I (unlike the "leaders") eat my own dogfood ;D

all 5 of you

** literally stalk them: show up at random times at their houses, stand next to them in queues staring, peek through their windows while they are having sex. Stalk them. I find it personally offending when a serious bug is reported over a year ago and is yet to be determined when it will be fixed (looks over at the traffic shaping bug…oh wait ;))

bmeeks

@jflsakfja:

@bmeeks:

1. Legacy behavior where every alert results in a block unless the offender is in a Pass List. This is how things work today.

That would be a nice option.
@bmeeks:

2. Only blocking when the PRIORITY field in the associated rule is less than or equal to a set value. For example, block on Priority 1 alerts but only alert for Priority 2 and 3 alerts. Still not sure how useful this option would really be, nor how reliable the Priority field values are.

Not at all useful, since I've seen priorities that really don't match the rules.
@bmeeks:

3. A more conventional IPS setup where all rules are only alerts (with no blocks) unless the admin explicitly changes the rule action from "alert" to "drop". This could be handled fairly easily using the new SID MGMT tab feature. It would still require the admin to do a lot of work, though.

If you turn that around, I'd love to have the functionality. Instead of all rules being alert and you had to change them to drop, I would prefer all rules to be drop and you can change them to alert, if need be. Makes a lot more sense when the inline parts are finally available.

That's what I would like to see, alert/blocking wise. I didn't even gave it much thought, until I saw your post.

What I would like though (WANTED FEATURE ALERT!) is a way to turn off the category + priority from the syslog logs. It's driving me (and our loggers) insane. Every IDS/IPS/firewall log should be:
Source IP - Source Port - Destination IP - Destination Port - Protocol - Rule ID - Reason for blocking/dropping/rule explanation.

Thank you for the comments. Option #1 is sort of a no-brainer since that is how things work today and no changes in any code would be required.

Your assessment of Option #2 reinforces what I suspected – the assignment of the Priority value in a rule is not necessarily perfect.

Your comment on Option #3 regarding turning the logic around (making all alerts DROP by default and allowing changing some to only ALERT) would require rewriting some of the underlying binary code in both Suricata and Snort. All of the internal routines are geared to act on the action keyword (either ALERT or DROP) in the rule. So without essentially rewriting the core mechanisms of both binaries, implementing as you suggest would not be possible. And for several very valid reasons, the pfSense team wants packages to stay in sync with upstream to the maximum amount possible. That means no or very limited patching for use on pfSense.

Turning off fields in the syslog output would also require upstream changes in the binary. Today the fields are hard-coded in that Suricata source code module.

Bill

dstroot

I am attempting the red pill. However it is not going down all that well. ;) I am attempting the very first portion of the tutorial, blocking everything and setting up explicit "whitelist rules" I am using version 2.1.5.

First setup your port aliases - check.
Next setup your anti-lockout rule - check. (and it works… phew)
Next setup a floating rule that blocks basically everything in/out on WAN - check.
Next setup rule(s) that allow LAN services access to services (DNS, DHCP, NTP) - check. I get a DHCP address and I can resolve DNS.
Finally, setup your explicit outbound ports rule to allow traffic out from the LAN - No Joy.

Here's what I suspect is happening - can someone help me figure this out?

Floating rules are parsed before rules on other interfaces. Thus, if a packet matches a floating rule and the Quick option is active (as we were told to do) on that rule, pfsense will not attempt to filter that packet against any rule on any other interface. Thus in step 5 the explicit outbound rules are not being hit because the floating "block everything in/out" has "quick" checked.

Does this make any sense? Has anyone else followed the first post in the thread and got working? I'm stumped.

Cheers,
Dan

rules.png_thumb

I'm thinking that you too got confused by the floating rule. The floating "block everything" rule isn't meant to be a "block everything" rule. It's meant to be a rule to block everything to pfsense's ports!, hence the giant red warning. I'm working on version 2.0 of this guide, which will clear up some confusion (and possibly create confusion elsewhere :p).

dstroot

I'm still confused. If you "DON'T CHANGE DESTINATION PORT RANGE!!!" on a new rule then you end up with "any" on the rule. It looks like the attached. And yes, it blocks everything. ;)

So instead I should use the alias for the pfSense ports? i.e. instead of "DON'T CHANGE DESTINATION PORT RANGE!!!" on a new rule it should say "Destination port range 'pfsense_ports' as above"?

a 2.0 guide would be much appreciated - this thread is a long hard slog. ;D

edit: I did find what I was looking for somewhere in the middle of this thread you give a very clear explanation:

Re-reading it does make sense on why it blocked out traffic. I meant to say create a new floating rule, based on the previous allow rule, but this time around change the pass to a block, keeping the destination ports the same.

A)1 normal pass rule for the ports active on the interface you want to administer pfsense from.

B)1 floating rule block rule for ALL interfaces EXCEPT the one you want to administer pfsense from.

floating.png_thumb

fsansfil

@bmeeks:

What I would like to do is migrate both the Suricata and Snort packages to a new paradigm where you have alerts and blocks, but not a block from every alert like you have today. In a true IPS, you manually edit rules to change the "alert" action to "drop" only for those rules that detect really bad threats. For other things a simple alert to notify you is enough. However, in both Suricata and Snort today on pfSense, the output plugin that handles blocking simply keys off every alert. So any alert results in a block (unless the offending IP is in a Pass List).

I have been running some ideas around in my head on a method for presenting this new paradigm in the GUI. I am thinking about a series of behavior options the admin can choose from. Maybe something along these lines:

1. Legacy behavior where every alert results in a block unless the offender is in a Pass List. This is how things work today.

2. Only blocking when the PRIORITY field in the associated rule is less than or equal to a set value. For example, block on Priority 1 alerts but only alert for Priority 2 and 3 alerts. Still not sure how useful this option would really be, nor how reliable the Priority field values are.

3. A more conventional IPS setup where all rules are only alerts (with no blocks) unless the admin explicitly changes the rule action from "alert" to "drop". This could be handled fairly easily using the new SID MGMT tab feature. It would still require the admin to do a lot of work, though.

4. A refinement of #3 above where choosing one of the Snort VRT IPS policy settings results in blocks from all alerts originating from any of the IPS Policy rules, while other rules (non-IPS policy ones) simply alert. Of course the admin would still have the ability to manually edit any of the other rules to block (drop) by changing the action keyword from "alert" to "drop".

This would set both packages up to be ready to implement true IPS mode when the pfSense kernel is ready to support it. As both packages exist today where any alert results in a block, things could be a pain to administer if those blocks were true inline traffic drops.

I would like the community's input on the above. What changes would you like to see in the blocking behavior?

Thanks,
Bill

Well I'm alot like you on those feelings.

1,2,3 - Maybe the answer is in the "log" operator. Alert, Drop = block, Pass = Pass and … Log = Log; No drop, No block, just log.. There are many rules these days that I would want to be "log" only in pfsense. Also...all those complains about false positives, you can switch them manualy to "log" only. What I hate the most is when a rule is disabled because of too many false positives....well hell, let me decide whats FP...

"Pass" is sweet too, I have a set of rules that are true firewall rules with barely any payload inspection. "Pass" comes handy in this ruleset...

I personnaly replace all Alert with Drop and monitor the traffic, then, I change single rules to log only, one by one.

Keep up the good work Bill!

Cheers.

F.

bmeeks

@fsansfil:

@bmeeks:

What I would like to do is migrate both the Suricata and Snort packages to a new paradigm where you have alerts and blocks, but not a block from every alert like you have today. In a true IPS, you manually edit rules to change the "alert" action to "drop" only for those rules that detect really bad threats. For other things a simple alert to notify you is enough. However, in both Suricata and Snort today on pfSense, the output plugin that handles blocking simply keys off every alert. So any alert results in a block (unless the offending IP is in a Pass List).

I have been running some ideas around in my head on a method for presenting this new paradigm in the GUI. I am thinking about a series of behavior options the admin can choose from. Maybe something along these lines:

1. Legacy behavior where every alert results in a block unless the offender is in a Pass List. This is how things work today.

2. Only blocking when the PRIORITY field in the associated rule is less than or equal to a set value. For example, block on Priority 1 alerts but only alert for Priority 2 and 3 alerts. Still not sure how useful this option would really be, nor how reliable the Priority field values are.

3. A more conventional IPS setup where all rules are only alerts (with no blocks) unless the admin explicitly changes the rule action from "alert" to "drop". This could be handled fairly easily using the new SID MGMT tab feature. It would still require the admin to do a lot of work, though.

4. A refinement of #3 above where choosing one of the Snort VRT IPS policy settings results in blocks from all alerts originating from any of the IPS Policy rules, while other rules (non-IPS policy ones) simply alert. Of course the admin would still have the ability to manually edit any of the other rules to block (drop) by changing the action keyword from "alert" to "drop".

This would set both packages up to be ready to implement true IPS mode when the pfSense kernel is ready to support it. As both packages exist today where any alert results in a block, things could be a pain to administer if those blocks were true inline traffic drops.

I would like the community's input on the above. What changes would you like to see in the blocking behavior?

Thanks,
Bill

Well I'm alot like you on those feelings.

1,2,3 - Maybe the answer is in the "log" operator. Alert, Drop = block, Pass = Pass and … Log = Log; No drop, No block, just log.. There are many rules these days that I would want to be "log" only in pfsense. Also...all those complains about false positives, you can switch them manualy to "log" only. What I hate the most is when a rule is disabled because of too many false positives....well hell, let me decide whats FP...

"Pass" is sweet too, I have a set of rules that are true firewall rules with barely any payload inspection. "Pass" comes handy in this ruleset...

I personnaly replace all Alert with Drop and monitor the traffic, then, I change single rules to log only, one by one.

Keep up the good work Bill!

Cheers.

F.

Utilizing LOG and PASS in addition to ALERT and DROP are excellent suggestions. I now have more options to be thinking about …

Bill

Soloam

Hi, I made the transition from snort to suricata, I followed your tutorial from the beginning, and I think that I have all working. But one thing is wired, on snord I had several alerts a day, some were identified as false positives, others I wasn't able to determine, so a keep them blocked until I identified the real source. On Suricata, I have it running now at 5 days, and not even one alert. Should I go back to snort in my production system until Suricata is in a more mature state?

fsansfil

Suricata is more than mature…

The problem is, some of Snort VRT rules arent compatible, out of the box, with suricata.

Majority of ET rules are. So the alerts you were seeing with Snort and not, right now, with suricata are most probably coming from Snort VRT ruleset.

Take those rules you were seeing, paste them in the custom box of your suricata interfaces and youll get a warning on why they arent compatible.

Bingo...just modify the rules now according to the warning and you all set.

You can even use the SID Mgmt tab to mod many rules at the same time.

Cheers.

F.

Soloam

How can I test if suricata is working? I tried for example in the "files.rules" to activate all (for example jpg) and try to access jpg files on the web, and no alert where triggered, shouldn't suricata deny those files?

fsansfil

It depends if those rules are a GET or a POST JPG rules…if they are a GET, then it should trigger after 2-3 webpages max, if they are a POST, then dont worry.

Best advice, use ET ruleset, go visit the same webpages or generate the same traffic you are use to see when you were using Snort, make sure same rules are active and compare...

Every year I see some articles about plp doing some lab test comparing Snort and Suricata, they throw some malware and trojan in, see who catch what...and most of the time Suricata wins...

F.

fsansfil

Precision:

Snort is an IDS engine and a propriatary ruleset, also compatible with ET ruleset.

Suricara is an IDS engine and no propriatary ruleset, also compatible with ET ruleset, but not optimized….

So its important, when comparing, to understand if you are comparing an engine to detect a vulnerability or an engine to detect a vulnerability made on their syntax of a rule...

Suricata come naked, but it can perform alot better than Snort in some cases if you have a appropriate ruleset...

Hope I hvent confused the topic ;)

F.

@fsansfil Snort is an IDS. As soon as people started working on the IPS part, the IPS part blew apart. Development on snort-inline has ceased (the IPS) and the devs have moved on to suricata. This is what gave birth to suricata (well, that and the fact that snort wasn't scalable enough for NSA). I don't understand the "not optimized" part. Suricata has been seen analyzing 10Gbps connections (multiple, if anyone cares), something that snort can (and forever will) only dream about.

Snort VRT on the other hand are the rules that are designed for snort. Some of them work with suricata. Personally, I avoid them.

@soloam: This guide has been tested time and time again. If you set it up according to the guide, then suricata will alert correctly. Just to let everyone know, the transition was also tested. A cluster using snort was migrated to suricata by uninstalling snort, installing suricata and configuring it identically to this topic.

Did you restart after removing snort? After installing suricata?Is the suricata interface even running? Any logs showing some more info?What's the HOME net that suricata sees?

Soloam

Hi jflsakfja, I don't doubt that the system works, I was just trying to see if I did something wrong, or if there is something wrong I uninstalled snort but I can't remember if I restarted after uninstalling and before installing suricata. So probably not, can it cause problems? I restarted after installing suricata.

In the home network I see the same entry's that I did in snort, my lan addresses and my wan address.

One thing. I didn't used the script to make the alias to the block ip's, I had everything set with pfblocker (from your previews tutorial) and I didn't mess with it. I only unistalled snort and installed suricata.

One question, any idea way Suricata doesn't have portscan alerts? I looked on the web and it's not supported. Any reason to way is that?

Thank You
Best Regards
Soloam