Taming the beasts… aka suricata blueprint

justsomeone

What we really need is a simpler way to manage the Rules. Enabling/Disabling one at a time is too cumbersome and leads to human error and is too taxing on Time (Which we all wish we had more of ..  :) )

I agree. If there was a function or feature to show relationships between or how different rules affect others, that would be very useful and could possibly prevent human error. Especially for those less experienced or knowledgeable.

@BBcan177:

I think having both Snort and Suricata around will promote a healthier environment. From what I have seen, I don't think that Cisco will close Snort to "closed Source", they may add new features that are only for Paid Subscribers or for their own Cisco Equipment.

While I think that ET and OISF are capable of the same behavior.

What we really need is a simpler way to manage the Rules. Enabling/Disabling one at a time is too cumbersome and leads to human error and is too taxing on Time (Which we all wish we had more of .. :) )

I hope the Devs listen and give us the right tools for the job so we can manage the IDS/IPS of our own Choosing in the most efficient way.

If the rule maintainers even said for once "yes, I've read that crazy guy's list", even for a single time, and corrected the rules I've been screaming about for years, then we wouldn't have all those rules to disable. An example? See the previous post about the win 6.1 rule. It's an invalid rule that must be deleted, end of story. If the rule maintainer doesn't delete it, I will figure out a way to work around that limitation by disabling the rule. I will not stop using the software just because a single person out of the thousands of persons that have to do with suricata doesn't realize he made a mistake.
Standing up to your actions and saying "I'm sorry, I made a mistake" is magnitudes of order more respectable than saying "I will not delete the rule because kids,cyber,terrorism,syria,russia,china,ukraine".

Want an easier process to setting up suricata? Convince upstream to actually delete false positive rules. Not disable them, delete them. If they don't want to delete the win 6.1 rule, give microsoft a call, then put them on conference with the rule maintainer next to you and ask: "Is windows 7 actually windows 6.1?". Even after they hear the answer from microsoft they insist on not deleting the rule, a public name and shame is the industry accepted practice of dealing with it. Bonus points if you get it on video. Until then, I'm personally happy with maintaining my own false positives list and manually disabling the rules. Yes the disabling process as mentioned earlier should be changed, but it sort of works for now. Even if nobody else maintains the rules, I have my own rules for suricata to do what I want it to do.

A huge thanks to bmeeks (and all others that helped in the past) for getting snort/suricata packages were they are. I'm not playing down any of that work, if they didn't make the packages, we wouldn't be here arguing today on what rules and how we should disable them. Think about it. When IDS/IPS systems first started getting imagined (not created, imagined) they came with a price of a few million $. Now it's just a click and follow the guide away.

This is where the economics comes in (Hollander will be thrilled :P): If there is a single user of your software, don't expect much in the sense of support from it (donations/subscriptions). If everybody out there is using your software, and you are not a greedy SoB and actually charge a logical price for your software (eg subscriptions), a whole lot of people will chime in to support you. Yes this is the ideal scenario, and no it's not what actually happens in reality. It should happen though, because it's a shame to see projects die because of lack of user support. No sympathy to openssl. As Linus would have said:"The sooner the developers die in a horrific car accident, the better". Not my words, and put the phone down, it's not a "terrorist threat".

There is a (not so) fine line between fixing bugs and ignoring feature requests, and implementing feature requests but ignoring bugs. OpenSSL went over that line years ago.

If you see a man running around waving his arms up high, screaming "CHARGE FOR SOFTWARE????ARE YOU F***ING INSANE???SOFTWARE SHOULD ALWAYS BE FREE", it's RMS, please ignore him. His words have no relation to reality (no pun intended).

Theoretically speaking now, don't want to offend anyone, but I'll go ahead and do so anyway: If a certain IDS/IPS offers me all I want, then I'll contribute to its creation (documentation,donations,beer for the developers, hookers and blow (let's see how many got that joke) and so on). If said IDS/IPS's rule maintainer doesn't listen to me when I say a certain rule is buggy and must be corrected, don't expect said maintainer to get any support from me. Just saying, don't want to offend anyone.

And yes, I've put my money where my mouth is, remember the donations for snort's sync to slave system. It was a feature I wanted badly, the package maintainer impemented it without any noticable side effects to the general usage, and I've donated to those that made it happen. Everybody should do the same and stop sucking on RMS's meme that software should always be free and developers must die of hunger.

bmeeks

My vision for both Suricata and Snort includes a true inline operation option. For that to happen we need some changes to the pfSense kernel code. I think those will come, but it will take a little while. Obviously changes down at that level should not just be rushed out. So please continue to have patience.

As for some improvements that can actually happen without pfSense kernel changes, here are some highlights from my internal road map of new features I have planned for each package.

Suricata:
Migration to the new 2.0.2 binary code. This offers several new detection/inspection features, especially around DNS requests. There are also a multitude of other changes and improvements compared to the existing 1.4.6 binary code base in the current package.

Adding the same XMLRPC sync feature to Suricata that currently exists in Snort so you can synchronize Suricata setups across multiple firewalls.

Fixing the list of bugs you guys have submitted over the last few months. There are several of them.

–---------------------------------------------------------------------------------

Snort:
Adding support for some of the new file capture features that have been added to the Snort binary. These mimic in large part what Suricata does in this area.

Both Suricata and Snort:
Backporting significant functionality changes between Snort and Suricata. This means reverting to the old force rule enable/disable icon behavior that some folks so desperately want (I'm looking at you jflsakfja… ;))

Adding a filter option to the ALERTS tab so you can see only what you want. This will be modeled after the firewall log filter functionality.

Providing rule management functionality equivalent to PulledPork with regex matches for enabling or disabling rules. In other words, the ability to read and interpret enablesid.conf, modifysid.conf and disablesid.conf files. You would be able to edit these offline and upload to the firewall, or edit in place using the same interface as I implemented for the IP REP lists management tab in Snort.

Bill

BBcan177

Hi Bill,

This is Fantastic News. Lots to look forward to. We are all deeply in your debt! ;)

We've never had a poll for the Best Package Maintainer! But I think we should start…

If you need any help to Test, I am sure several of us will throw our hats into the ring.

Supermule

Just wondering…

Would it be feasible/easy/doable to implement squid like behavior in Suricata/Snort to make it independant of the inline capabilities of Pfsense?

Instead of Snort inspecting copies of packets, then going through as a transparent proxy, then the inline could be there?

Doable?

@bmeeks:

My vision for both Suricata and Snort includes a true inline operation option. For that to happen we need some changes to the pfSense kernel code. I think those will come, but it will take a little while. Obviously changes down at that level should not just be rushed out. So please continue to have patience.

As for some improvements that can actually happen without pfSense kernel changes, here are some highlights from my internal road map of new features I have planned for each package.

Suricata:
Migration to the new 2.0.2 binary code. This offers several new detection/inspection features, especially around DNS requests. There are also a multitude of other changes and improvements compared to the existing 1.4.6 binary code base in the current package.

Adding the same XMLRPC sync feature to Suricata that currently exists in Snort so you can synchronize Suricata setups across multiple firewalls.

Fixing the list of bugs you guys have submitted over the last few months. There are several of them.

–---------------------------------------------------------------------------------

Snort:
Adding support for some of the new file capture features that have been added to the Snort binary. These mimic in large part what Suricata does in this area.

Both Suricata and Snort:
Backporting significant functionality changes between Snort and Suricata. This means reverting to the old force rule enable/disable icon behavior that some folks so desperately want (I'm looking at you jflsakfja… ;))

Adding a filter option to the ALERTS tab so you can see only what you want. This will be modeled after the firewall log filter functionality.

Providing rule management functionality equivalent to PulledPork with regex matches for enabling or disabling rules. In other words, the ability to read and interpret enablesid.conf, modifysid.conf and disablesid.conf files. You would be able to edit these offline and upload to the firewall, or edit in place using the same interface as I implemented for the IP REP lists management tab in Snort.

Bill

If The Company was hiring, you would be one of the first people it would hire, trust me on that one ;D

Many thanks for helping the community with those packages. I'm sure as the features everybody wants are added we can all chime in to make it worth a bit for the time you've spent on them.

WRT the blue text, maybe we should agree on a bare minimum of disabled rules (the absolute lowest number of general use rules) that should be disabled on all installations and somehow integrate that into newer installs? So people don't have to get choked into configuring snort/suricata right away after install, but actually get on the internet and get some help if they have a problem, without the IDS/IPS banning everything (coughsimple http requestcough).

bmeeks

@jflsakfja:

WRT the blue text, maybe we should agree on a bare minimum of disabled rules (the absolute lowest number of general use rules) that should be disabled on all installations and somehow integrate that into newer installs? So people don't have to get choked into configuring snort/suricata right away after install, but actually get on the internet and get some help if they have a problem, without the IDS/IPS banning everything (coughsimple http requestcough).

Not a bad idea. When I get to that point, I will be in touch. That could be offered as an option with the default being "yes" (meaning, auto-disable some bare minimum of community-agreed upon quasi-useless rules).

Bill

Mr. Jingles

@bmeeks:

Not a bad idea.

Can I apply for having a good idea too?

( ;D )

The checkboxes that we have in the firewall rules to select multiple rules: also in Snort/Suricata? So you can quickly select multiple rules in a category and then mass enable/disable them?

Select the rules -> click 'disable' -> click 'apply', and move on to the next category.

Or a CLI-operation where you can edit a *.conf quicker than one by one as currently is the case?

Let me guess, probably not good ideas from the eternal noob :-[

( ;D )

Thanks for all you do, Bill :P

@Hollander:

The checkboxes that we have in the firewall rules to select multiple rules: also in Snort/Suricata? So you can quickly select multiple rules in a category and then mass enable/disable them?

bangs head on desk Why didn't I think of that? ;D Selecting a dozen rules then hitting disable selected rules would be very nice indeed.

bmeeks

@Hollander:

@bmeeks:

Not a bad idea.

Can I apply for having a good idea too?

( ;D )

The checkboxes that we have in the firewall rules to select multiple rules: also in Snort/Suricata? So you can quickly select multiple rules in a category and then mass enable/disable them?

Select the rules -> click 'disable' -> click 'apply', and move on to the next category.

Or a CLI-operation where you can edit a *.conf quicker than one by one as currently is the case?

Let me guess, probably not good ideas from the eternal noob :-[

( ;D )

Thanks for all you do, Bill :P
[/quote]

Excellent idea. Let me see if I can find enough real estate to show the checkboxes. For non-widescreen configurations, I'm running out of space on the RULES tab when displaying the rules. When everyone migrates to pfSense 2.2, it has widescreen as a built-in theme. That will help out greatly.

I do also plan to offer what is a kind of CLI interface for this. Do a Google search for enablesid.conf, modifysid.conf and disablesid.conf. You should get some hits showing how these work with packages such as PulledPork. The idea for the Suricata and Snort packages is to be able to edit these files offline and then upload them to the firewall.

Bill

wcrowder

Ran into a small problem with bbcan177's script. Was working wonderfully then quit updating after the last Alpha update. Running pfSense 2.2 alpha.

Here's the error:

SSL options: 81004bff
Peer verification enabled
Using CA cert file: /etc/ssl/cert.pem
Certificate verification failed for /O=www.projecthoneypot.org/OU=Domain Control Validated/CN=www.projecthoneypot.org
675141692:error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed:/usr/pfSensesrc/src/secure/lib/libssl/../../../crypto/openssl/ssl/s3_clnt.c:1169:
fetch: https://www.projecthoneypot.org/list_of_ips.php?t=d&rss=1: Authentication error

Anyone else run into this?

NEVER MIND, I'm stupid… LOL..

Mr. Jingles

Well thank you Bill and JFL for boosting my self esteem ;D ;D ;D

But now… 8)

( ;D )

I am struggling with a dnsmasq problem (blocking ad servers via dnsmasq, can't get the script to run in cron - although manually it works), and on the site that provides the lists I also noted a format for Snort:

http://pgl.yoyo.org/adservers/

(Select the second drop down - list ad servers IP adresses): there is also a Snort format of the list there.

Would it make sense to use Snort to block ad servers using that list?

I know there will be reservations; 'noob, shut up'.

Ok, I will :P

bmeeks

@Hollander:

Well thank you Bill and JFL for boosting my self esteem ;D ;D ;D

But now… 8)

( ;D )

I am struggling with a dnsmasq problem (blocking ad servers via dnsmasq, can't get the script to run in cron - although manually it works), and on the site that provides the lists I also noted a format for Snort:

http://pgl.yoyo.org/adservers/

(Select the second drop down - list ad servers IP adresses): there is also a Snort format of the list there.

Would it make sense to use Snort to block ad servers using that list?

I know there will be reservations; 'noob, shut up'.

Ok, I will :P

When you can't get a script to run via cron, the #1 cause is forgetting to provide the complete and entire path to all files used in the script. When a cron task executes, it does not inherit the user environment you have when working at a shell prompt (CLI). Some examine the script and make sure full paths are provided for all referenced files.

Bill

BBcan177

@wcrowder:

Ran into a small problem with bbcan177's script. Was working wonderfully then quit updating after the last Alpha update. Running pfSense 2.2 alpha.

Here's the error:
SSL options: 81004bff
Peer verification enabled
Using CA cert file: /etc/ssl/cert.pem
Certificate verification failed for /O=www.projecthoneypot.org/OU=Domain Control Validated/CN=www.projecthoneypot.org
675141692:error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed:/usr/pfSensesrc/src/secure/lib/libssl/../../../crypto/openssl/ssl/s3_clnt.c:1169:
fetch: https://www.projecthoneypot.org/list_of_ips.php?t=d&rss=1: Authentication error
Anyone else run into this?

NEVER MIND, I'm stupid… LOL..

SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
The server certificate is invalid, either because it is signed by an invalid CA (internal CA, self signed,…), doesn't match the server's name or because it is expired.

Hi wcrowder, is this still an issue for you?

BBcan177

@bmeeks:

I do also plan to offer what is a kind of CLI interface for this. Do a Google search for enablesid.conf, modifysid.conf and disablesid.conf. You should get some hits showing how these work with packages such as PulledPork. The idea for the Suricata and Snort packages is to be able to edit these files offline and then upload them to the firewall.

This will make the Rules Configuration Process a Breeze. All we would have to do, is copy the text based conf file to any other Snort/Suricata Interface or to another Box. Rules can be disabled or enabled by SID, Category or by PCRE (ie : Autodesk or Adobe - Which will disable/enable any rules that has any of those Keywords in them. This way even as new rules are added, the PCRE will disable them at each rule-update automatically. :)

wcrowder

Sorta, I just removed the S from each HTTP and it updated, not the best practice. LOL. I've updated the cert.pem with the latest version of Mozilla's certs, no go, assuming it's something with the 2.2 Alpha. Also takes 7 minutes to get through "configuring Firewall" upon boot. Have "./pfiprep killdb" twice then pfiprep with bypass=YES after update and it works. Still now SSL. Just messing around. Really fine work on your part.

BBcan177

@wcrowder:

Sorta, I just removed the S from each HTTP and it updated, not the best practice. LOL. I've updated the cert.pem with the latest version of Mozilla's certs, assuming it's something with the 2.2 Alpha. Also takes 7 minutes to get through "configuring Firewall" upon boot. Have "./pfiprep killdb" twice. Just messing around. Really fine work on your part.

Hi wcrowder,

You need to check the pfSense Cert, as that is the machine that is fetching those files.

BBcan177

https://www.mail-archive.com/freebsd-security@freebsd.org/msg05085.html
Looks like FreeBSD 9 doesn't have a default pem file in /etc/ssl/cert.pem ?

fetch
–ca-cert=file
[SSL] Path to certificate bundle containing trusted CA cer-
tificates. If not specified, /etc/ssl/cert.pem is used. The
file may contain multiple CA certificates. The port
security/ca_root_nss is a common source of a current CA bun-
dle.

Do you have a file in that directory?
[ [b] ls -lah /etc/ssl/ ]

I'm not the best at certs. But I think you should just be able to generate a Self-signed one from the pfSense Cert Manager? I don't want to suggest you disable the cert check in the fetch cmd line.

http://www.freebsd.org/cgi/man.cgi?query=fetch(1)
My script fetchs with this line:

$pathfetch -v -o $workdir$infile -T 20 $addr$infile

Try this:

cd /tmp
fetch -v -o honeypot.txt "https://www.projecthoneypot.org/list_of_ips.php?t=d&rss=1"

Or you can switch to using WGET: (Script pfiprep has an option to switch between fetch and wget)

Script Command:
$pathwget –no-check-certificate -T 20 -O $workdir$infile $addr$infile

Try:
wget –no-check-certificate -O honeypot.txt "https://www.projecthoneypot.org/list_of_ips.php?t=d&rss=1"

I think Foetus had a similar issue with 2.2. Not sure how he made out.

Supermule

Isnt 2.2 FreeBSD 10 based??

Mr. Jingles

@bmeeks:

@Hollander:

Well thank you Bill and JFL for boosting my self esteem ;D ;D ;D

But now… 8)

( ;D )

I am struggling with a dnsmasq problem (blocking ad servers via dnsmasq, can't get the script to run in cron - although manually it works), and on the site that provides the lists I also noted a format for Snort:

http://pgl.yoyo.org/adservers/

**(Select the second drop down - list ad servers IP adresses): there is also a Snort format of the list there.

Would it make sense to use Snort to block ad servers using that list?**

I know there will be reservations; 'noob, shut up'.

Ok, I will :P

When you can't get a script to run via cron, the #1 cause is forgetting to provide the complete and entire path to all files used in the script. When a cron task executes, it does not inherit the user environment you have when working at a shell prompt (CLI). Some examine the script and make sure full paths are provided for all referenced files.

Bill

Thanks Bill ;D

I didn't mean to bother you with the problem I have to get a cron job running (I did have the full path, btw): I was only meandering away from the point in question: the text in bold in the above :P