Taming the beasts… aka suricata blueprint

My vision for both Suricata and Snort includes a true inline operation option. For that to happen we need some changes to the pfSense kernel code. I think those will come, but it will take a little while. Obviously changes down at that level should not just be rushed out. So please continue to have patience.

As for some improvements that can actually happen without pfSense kernel changes, here are some highlights from my internal road map of new features I have planned for each package.

Suricata:
Migration to the new 2.0.2 binary code. This offers several new detection/inspection features, especially around DNS requests. There are also a multitude of other changes and improvements compared to the existing 1.4.6 binary code base in the current package.

Adding the same XMLRPC sync feature to Suricata that currently exists in Snort so you can synchronize Suricata setups across multiple firewalls.

Fixing the list of bugs you guys have submitted over the last few months. There are several of them.

–---------------------------------------------------------------------------------

Snort:
Adding support for some of the new file capture features that have been added to the Snort binary. These mimic in large part what Suricata does in this area.

Both Suricata and Snort:
Backporting significant functionality changes between Snort and Suricata. This means reverting to the old force rule enable/disable icon behavior that some folks so desperately want (I'm looking at you jflsakfja… ;))

Adding a filter option to the ALERTS tab so you can see only what you want. This will be modeled after the firewall log filter functionality.

Providing rule management functionality equivalent to PulledPork with regex matches for enabling or disabling rules. In other words, the ability to read and interpret enablesid.conf, modifysid.conf and disablesid.conf files. You would be able to edit these offline and upload to the firewall, or edit in place using the same interface as I implemented for the IP REP lists management tab in Snort.

Bill

If The Company was hiring, you would be one of the first people it would hire, trust me on that one ;D

Many thanks for helping the community with those packages. I'm sure as the features everybody wants are added we can all chime in to make it worth a bit for the time you've spent on them.

WRT the blue text, maybe we should agree on a bare minimum of disabled rules (the absolute lowest number of general use rules) that should be disabled on all installations and somehow integrate that into newer installs? So people don't have to get choked into configuring snort/suricata right away after install, but actually get on the internet and get some help if they have a problem, without the IDS/IPS banning everything (coughsimple http requestcough).

bmeeks

@jflsakfja:

WRT the blue text, maybe we should agree on a bare minimum of disabled rules (the absolute lowest number of general use rules) that should be disabled on all installations and somehow integrate that into newer installs? So people don't have to get choked into configuring snort/suricata right away after install, but actually get on the internet and get some help if they have a problem, without the IDS/IPS banning everything (coughsimple http requestcough).

Not a bad idea. When I get to that point, I will be in touch. That could be offered as an option with the default being "yes" (meaning, auto-disable some bare minimum of community-agreed upon quasi-useless rules).

Bill

Mr. Jingles

@bmeeks:

Not a bad idea.

Can I apply for having a good idea too?

( ;D )

The checkboxes that we have in the firewall rules to select multiple rules: also in Snort/Suricata? So you can quickly select multiple rules in a category and then mass enable/disable them?

Select the rules -> click 'disable' -> click 'apply', and move on to the next category.

Or a CLI-operation where you can edit a *.conf quicker than one by one as currently is the case?

Let me guess, probably not good ideas from the eternal noob :-[

( ;D )

Thanks for all you do, Bill :P

@Hollander:

The checkboxes that we have in the firewall rules to select multiple rules: also in Snort/Suricata? So you can quickly select multiple rules in a category and then mass enable/disable them?

bangs head on desk Why didn't I think of that? ;D Selecting a dozen rules then hitting disable selected rules would be very nice indeed.

bmeeks

@Hollander:

@bmeeks:

Not a bad idea.

Can I apply for having a good idea too?

( ;D )

The checkboxes that we have in the firewall rules to select multiple rules: also in Snort/Suricata? So you can quickly select multiple rules in a category and then mass enable/disable them?

Select the rules -> click 'disable' -> click 'apply', and move on to the next category.

Or a CLI-operation where you can edit a *.conf quicker than one by one as currently is the case?

Let me guess, probably not good ideas from the eternal noob :-[

( ;D )

Thanks for all you do, Bill :P
[/quote]

Excellent idea. Let me see if I can find enough real estate to show the checkboxes. For non-widescreen configurations, I'm running out of space on the RULES tab when displaying the rules. When everyone migrates to pfSense 2.2, it has widescreen as a built-in theme. That will help out greatly.

I do also plan to offer what is a kind of CLI interface for this. Do a Google search for enablesid.conf, modifysid.conf and disablesid.conf. You should get some hits showing how these work with packages such as PulledPork. The idea for the Suricata and Snort packages is to be able to edit these files offline and then upload them to the firewall.

Bill

wcrowder

Ran into a small problem with bbcan177's script. Was working wonderfully then quit updating after the last Alpha update. Running pfSense 2.2 alpha.

Here's the error:

SSL options: 81004bff
Peer verification enabled
Using CA cert file: /etc/ssl/cert.pem
Certificate verification failed for /O=www.projecthoneypot.org/OU=Domain Control Validated/CN=www.projecthoneypot.org
675141692:error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed:/usr/pfSensesrc/src/secure/lib/libssl/../../../crypto/openssl/ssl/s3_clnt.c:1169:
fetch: https://www.projecthoneypot.org/list_of_ips.php?t=d&rss=1: Authentication error

Anyone else run into this?

NEVER MIND, I'm stupid… LOL..

Mr. Jingles

Well thank you Bill and JFL for boosting my self esteem ;D ;D ;D

But now… 8)

( ;D )

I am struggling with a dnsmasq problem (blocking ad servers via dnsmasq, can't get the script to run in cron - although manually it works), and on the site that provides the lists I also noted a format for Snort:

http://pgl.yoyo.org/adservers/

(Select the second drop down - list ad servers IP adresses): there is also a Snort format of the list there.

Would it make sense to use Snort to block ad servers using that list?

I know there will be reservations; 'noob, shut up'.

Ok, I will :P

bmeeks

@Hollander:

Well thank you Bill and JFL for boosting my self esteem ;D ;D ;D

But now… 8)

( ;D )

I am struggling with a dnsmasq problem (blocking ad servers via dnsmasq, can't get the script to run in cron - although manually it works), and on the site that provides the lists I also noted a format for Snort:

http://pgl.yoyo.org/adservers/

(Select the second drop down - list ad servers IP adresses): there is also a Snort format of the list there.

Would it make sense to use Snort to block ad servers using that list?

I know there will be reservations; 'noob, shut up'.

Ok, I will :P

When you can't get a script to run via cron, the #1 cause is forgetting to provide the complete and entire path to all files used in the script. When a cron task executes, it does not inherit the user environment you have when working at a shell prompt (CLI). Some examine the script and make sure full paths are provided for all referenced files.

Bill

BBcan177

@wcrowder:

Ran into a small problem with bbcan177's script. Was working wonderfully then quit updating after the last Alpha update. Running pfSense 2.2 alpha.

Here's the error:
SSL options: 81004bff
Peer verification enabled
Using CA cert file: /etc/ssl/cert.pem
Certificate verification failed for /O=www.projecthoneypot.org/OU=Domain Control Validated/CN=www.projecthoneypot.org
675141692:error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed:/usr/pfSensesrc/src/secure/lib/libssl/../../../crypto/openssl/ssl/s3_clnt.c:1169:
fetch: https://www.projecthoneypot.org/list_of_ips.php?t=d&rss=1: Authentication error
Anyone else run into this?

NEVER MIND, I'm stupid… LOL..

SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
The server certificate is invalid, either because it is signed by an invalid CA (internal CA, self signed,…), doesn't match the server's name or because it is expired.

Hi wcrowder, is this still an issue for you?

BBcan177

@bmeeks:

I do also plan to offer what is a kind of CLI interface for this. Do a Google search for enablesid.conf, modifysid.conf and disablesid.conf. You should get some hits showing how these work with packages such as PulledPork. The idea for the Suricata and Snort packages is to be able to edit these files offline and then upload them to the firewall.

This will make the Rules Configuration Process a Breeze. All we would have to do, is copy the text based conf file to any other Snort/Suricata Interface or to another Box. Rules can be disabled or enabled by SID, Category or by PCRE (ie : Autodesk or Adobe - Which will disable/enable any rules that has any of those Keywords in them. This way even as new rules are added, the PCRE will disable them at each rule-update automatically. :)

wcrowder

Sorta, I just removed the S from each HTTP and it updated, not the best practice. LOL. I've updated the cert.pem with the latest version of Mozilla's certs, no go, assuming it's something with the 2.2 Alpha. Also takes 7 minutes to get through "configuring Firewall" upon boot. Have "./pfiprep killdb" twice then pfiprep with bypass=YES after update and it works. Still now SSL. Just messing around. Really fine work on your part.

BBcan177

@wcrowder:

Sorta, I just removed the S from each HTTP and it updated, not the best practice. LOL. I've updated the cert.pem with the latest version of Mozilla's certs, assuming it's something with the 2.2 Alpha. Also takes 7 minutes to get through "configuring Firewall" upon boot. Have "./pfiprep killdb" twice. Just messing around. Really fine work on your part.

Hi wcrowder,

You need to check the pfSense Cert, as that is the machine that is fetching those files.

BBcan177

https://www.mail-archive.com/freebsd-security@freebsd.org/msg05085.html
Looks like FreeBSD 9 doesn't have a default pem file in /etc/ssl/cert.pem ?

fetch
–ca-cert=file
[SSL] Path to certificate bundle containing trusted CA cer-
tificates. If not specified, /etc/ssl/cert.pem is used. The
file may contain multiple CA certificates. The port
security/ca_root_nss is a common source of a current CA bun-
dle.

Do you have a file in that directory?
[ [b] ls -lah /etc/ssl/ ]

I'm not the best at certs. But I think you should just be able to generate a Self-signed one from the pfSense Cert Manager? I don't want to suggest you disable the cert check in the fetch cmd line.

http://www.freebsd.org/cgi/man.cgi?query=fetch(1)
My script fetchs with this line:

$pathfetch -v -o $workdir$infile -T 20 $addr$infile

Try this:

cd /tmp
fetch -v -o honeypot.txt "https://www.projecthoneypot.org/list_of_ips.php?t=d&rss=1"

Or you can switch to using WGET: (Script pfiprep has an option to switch between fetch and wget)

Script Command:
$pathwget –no-check-certificate -T 20 -O $workdir$infile $addr$infile

Try:
wget –no-check-certificate -O honeypot.txt "https://www.projecthoneypot.org/list_of_ips.php?t=d&rss=1"

I think Foetus had a similar issue with 2.2. Not sure how he made out.

Supermule

Isnt 2.2 FreeBSD 10 based??

Mr. Jingles

@bmeeks:

@Hollander:

Well thank you Bill and JFL for boosting my self esteem ;D ;D ;D

But now… 8)

( ;D )

I am struggling with a dnsmasq problem (blocking ad servers via dnsmasq, can't get the script to run in cron - although manually it works), and on the site that provides the lists I also noted a format for Snort:

http://pgl.yoyo.org/adservers/

**(Select the second drop down - list ad servers IP adresses): there is also a Snort format of the list there.

Would it make sense to use Snort to block ad servers using that list?**

I know there will be reservations; 'noob, shut up'.

Ok, I will :P

When you can't get a script to run via cron, the #1 cause is forgetting to provide the complete and entire path to all files used in the script. When a cron task executes, it does not inherit the user environment you have when working at a shell prompt (CLI). Some examine the script and make sure full paths are provided for all referenced files.

Bill

Thanks Bill ;D

I didn't mean to bother you with the problem I have to get a cron job running (I did have the full path, btw): I was only meandering away from the point in question: the text in bold in the above :P

BBcan177

@Supermule:

Isnt 2.2 FreeBSD 10 based??

Yes it is, but I couldn't find much info on this issue with my google FU…

I did find this link:
http://smyck.net/2014/01/22/freebsd-authentication-error/

I don't have a 2.2 box, so I can't test it myself. If anyone else has 2.2, can you see if these two commands work? Don't need to be using my Script to test if fetch and https work on 2.2 Alpha.

cd /tmp

fetch -v -o honeypot.txt "https://www.projecthoneypot.org/list_of_ips.php?t=d&rss=1"
wget –no-check-certificate -O honeypot.txt "https://www.projecthoneypot.org/list_of_ips.php?t=d&rss=1"

wcrowder

@BBcan177

My problem was a PEBKAC, I had a self signed cert installed in pfSense. I removed it and it works just fine. Thanks for your hard work. I can report that I'm not having any issues with the script or the widget in 2.2, all seems to be working fine except the patches for dns look up report that they can not be installed cleanly. Not important.

Thanks,
Bill

BBcan177

@wcrowder:

all seems to be working fine except the patches for dns look up report that they can not be installed cleanly. Not important.

Good to hear!

Is this the 2.2 diag_dns.php file that matches what you have on your box?
https://github.com/pfsense/pfsense/blob/master/usr/local/www/diag_dns.php

If its the same, I will post an updated version of the Patch.

wcrowder

I believe it is. On the current snapshot.

Patch can NOT be applied cleanly (detail)
Patch can NOT be reverted cleanly (detail)

Thanks,
Bill

wcrowder

Also neither widget displays list count or correctly determines failed downloads. It is correctly displaying other correct information. This is on Today's snapshot of 2.2. I didn't look into it because I can read logs… :)

PfIP Reputation2
minimize
close
(All Downloads Successful)

List Count
Alias CIDRs Packets Updated Status
IR_IB 60359 0 Jul 29 22:42
IR_PRI1 2418 0 Jul 29 20:42
IR_PRI2 24652 40 Jul 30 15:52