Taming the beasts… aka suricata blueprint

Cino

@jflsakfja:

They were supposed to remove those categories since they are no longer maintained. I'll add them in the do not use.

Do you think they will? after reading the comments in your enable rule list; hell will have to freeze over…

Supposed is the key word :D

ToxIcon

Thank you for taking the time to write up and
giving us this great suricata blueprint,

maybe you can help me with this setup

I would like to put pfsense in front of an
existing firewall, but just as an
Suricata IDS and also use the IP Reputation Manager script
is this possible, if it is how would i go about doing
so. The system that i am using has 3 ports, 1 port will be for managing
pfs as a Suricata IDS and the other 2 ports 1 will be connected to the wan of the other firewall
and the other port would be connect to the isp modem.

I would like the modem to not see the pfsense Suricata IDS box to act like its not even their and still pass the external address to the
other firewall behind the pfsense Suricata IDS while the pfsense Suricata IDS is still catching the nasty stuff and blocking them.

bmeeks

@Cino:

great work jflsakfja!

You may want to add decoder-events - 2200013 IPv6 truncated packet to the list for noise. My alert file is mostly this alert. What is strange, none of these show up in the GUI, only in the Alert file itself… Because there is no IP, i'm thinking the GUI doesn't display it. But that's for another thread after I confirm my findings.


06/25/2014-06:27:46.251740  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 01 EA 40 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
06/25/2014-06:27:46.250371  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 01 EA 40 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
06/25/2014-06:27:46.318718  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 01 EA 40 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
06/25/2014-06:27:46.251489  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 01 EA 40 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
06/25/2014-06:27:46.319106  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 01 EA 40 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
06/25/2014-06:27:46.319352  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 01 EA 40 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
06/25/2014-06:27:46.543947  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 0A B8 B0 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
06/25/2014-06:27:46.319496  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 01 EA 40 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
06/25/2014-06:27:46.543837  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 0A B8 B0 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
06/25/2014-06:27:46.986163  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 05 E7 64 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
06/25/2014-06:27:46.986289  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 05 E7 64 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]

The GUI is not currently showing these alerts because it is not properly detecting them. The Suricata alert log output is modified to produce a CSV (comma separated values) format. There is a built-in PHP function that can parse a CSV file and split the result into fields. For now the pfSense PHP code is counting fields and only showing on the ALERTS tab those alerts that parse into 13 (I think it is 13) fields. The decoder events currently don't output 13 distinct fields, and hence are dropped by the ALERTS tab PHP code.

I can fix that in an upcoming update.

Bill

bmeeks

@jflsakfja:

whitelists were renamed to passlists not long ago. That is a definite bug in the suricata package. please post a link pointing to that post, on the most recent version(ed?) topic on suricata or re-post the same thing there. I'm not the suricata package maintainer, bmeeks is ;D

bmeeks got a tunnel working a few days back, so IPv6 support for both packages should improve.

I do finally have a working IPv6 tunnel, and I did see the bug report in a separate post and responded. I will fix it.

Bill

@Arist:

Thank you for taking the time to write up and
giving us this great suricata blueprint,

maybe you can help me with this setup

I would like to put pfsense in front of an
existing firewall, but just as an
Suricata IDS and also use the IP Reputation Manager script
is this possible, if it is how would i go about doing
so. The system that i am using has 3 ports, 1 port will be for managing
pfs as a Suricata IDS and the other 2 ports 1 will be connected to the wan of the other firewall
and the other port would be connect to the isp modem.

I would like the modem to not see the pfsense Suricata IDS box to act like its not even their and still pass the external address to the
other firewall behind the pfsense Suricata IDS while the pfsense Suricata IDS is still catching the nasty stuff and blocking them.

What you are looking for is running pfsense (+suricata) as a transparent bridge in front of the "normal" firewall. Suricata might have trouble deciding its home net values if running as a bridge (snort did) but that's easily corrected by manually entering the home net.

A transparent bridge is not visible on the network. If you send a packet to one interface, and the rules allow it, it will pop up through the other interface. You can actually do some pretty clever stuff with it, provided you are using a single host (no CARP). Think of it as merging the two interfaces into a single interface, with filtering applied.

A couple of years back I was able to access a public server hosted behind a transparent bridge, from a host on a NATed interface on that transparent bridge (let's say the admin interface), using the server's public IP (universally understood as NOT possible to do). Then again I'm the only person on the planet that managed to get IPv6 working through 30 year old switches :)

That said, my personal recommendation is NOT to run pfsense like that. If you are trying to protect a small network, put pfsense directly as the core router (which allows you to move onto CARP if you so wish), which also saves time (money) + space (1 pc instead of 2 daisy chained firewalls) + power. Then set up firewalls on network hosts, along with other security measures (brute force protection for example) working together to protect your hosts.

EDIT: Clarification: I'm not saying don't ever use transparent bridges. If all you need is a single firewall host, it's actually better to run it as a transparent bridge since the host is not visible from the network. Depending on law mandated paranoia (called certifications in the industry), that might actually be exactly what you should use.
There are also downsides, an example is that the firewall hosts themselves don't have internet access (cannot check for updates), as set up in a plain vanilla transparent bridge (permission is hereby granted to correct me if I'm wrong).

justsomeone

Awesome post, thanks!

I'm trying to install pfiprep, and am getting an error when installing one of the dependencies.

$ pkg_add -r grepcidr
tar: Failed to set default locale

Can someone please tell me what to do to fix this?

BBcan177

@justsomeone:

$ pkg_add -r grepcidr
tar: Failed to set default locale

First, I would recommend that you make a Full Backup Diagnostics:Backup/Restore (Backup Area "all") and Download configuration… :)

I haven't come across this error on any of my installs or with the ones that I have helped to get working.

What version of pfSense are you using? Is it a Full Install or a Nano version?

Are you seeing any other errors? Could you post the full output of that command?

Can you download the file manually? Maybe a Firewall Rule or Snort is blocking it?

Try to [ [b]ping ftp.freebsd.org ] and see if you get a reply?

[ [b]fetch ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8.3-release/Latest/grepcidr.tbz ]
(This path is for the amd64 Release)

[ [b]pkg_add grepcidr.tbz ]

foetus

My cron job is throwing errors.
Since manually running "/usr/bin/nice -n20 /home/badips/pfiprep >> /home/badips/download.log 2>&1" gives me an error "Ambiguous output redirect." .
Leaving out the log output redirect it works fine. (aka >> … or > ...) So I could just remove the log output, but I'd rather keep it. :)

For the rest, perfect.
I edited the main script to leave out some lists, and also moved to using the main IR_ lists.
Also great work on the widget.

Cino

@foetus:

My cron job is throwing errors.
Since manually running "/usr/bin/nice -n20 /home/badips/pfiprep >> /home/badips/download.log 2>&1" gives me an error "Ambiguous output redirect." .
Leaving out the log output redirect it works fine. (aka >> … or > ...) So I could just remove the log output, but I'd rather keep it. :)

for the cron job "/usr/bin/nice -n20 /home/badips/pfiprep >> /home/badips/download.log 2>&1" works for me, but if i run it via directly from the shell, I get the same error you're seeing

running from the shell, i type "/usr/bin/nice -n20 /home/badips/pfiprep >> /home/badips/download.log" instead

foetus

Hmm, I'll just let the cron job do its job and see if it throws an error then.
Thx for that.

BBcan177

Hi Foetus,

Welcome Aboard!

Be careful not to run the command from the shell when Cron is scheduled to run or there may be some unexpected behavior. I added some code to avoid having them collide but just be aware to only run the [ [b]./iprep ] command when CRON is not scheduled to run or is still in process.

You can just run [ [b]./pfiprep ] from the shell and scroll up to see the output. You can also look at the "Daily.log" which shows a summary of the Downloads. Look out for any "FAIL" downloads.

The High Level function of the script:

Download Individual List
Extract IPs
Save copy to /orig Folder
Check for Ranges that have 255 IPs and mark a single /24 Range
Process /24 (Which looks for repeat Offenders in a /24 Range) (max variable) Individual Blocklist Only.
Duplication Check

Once all of the Downloads are completed that were scheduled to run:

The Following is performed Globally on ALL Lists, except for the ones that were marked as "p24=no" on the Collect Line.

p-Deduplication - Looks for Repeat Offenders that are over the pmax variable regardless of Country Code.

d-DeDuplication - Looks for Repeat Offenders that are over the dmax variable but uses the Country Code Whitelist function.

If the Sanity Checks passes, it will create the TIER (Group) lists and perform the "pfctl" commands to update the pfSense Alias Tables.

If you decide to remove a list, you need to add "remove" after the collect line. When the script runs at its next scheduled run, it will remove the list from the database properly. Don't try to do this manually.

If you follow the High level steps, when you use the p24 process in d-deduplicaton, it will look for a repeat range of malicious IPs and find all of the Blocklists that have this IP listed.

The FIRST blocklists get a single x.x.x.0/24 Block and all of the other Lists that have the range are deleted.

So if a List is removed, and it happens to be a list that had the p24 process and was the first list processed as above, then you have no Blocklists for that range. This will correct itself on when the Lists are re-downloaded but that could be 1-4hrs depending on when the Lists are scheduled to run.

To get back into Sync, you can run this function:

[ [b]./pfiprep killdb ]

Which will wipe the Database (Settings are not touched) and it will resync the database.

Out of Curiosity, which Lists did you disable?

Another Function is to use the "IR_Match" Alias in the Floating Rules as a "Match" Rule. This will show you activity for the IP Ranges that passed the Country Code Whitelist process. Because its a "Match" rule, it will not block, but just log the activity.

Since I have been running the script, I have not found too many False Positives, but I always recommend not to disable a list but to create a "SAFE Alias" Rule that is defined above the "Block/Reject" Rules. And just add the IPs that you want to allow.

The Patch for diag_dns.php will also work when looking at the Snort/Suricata Alert Logs.

If you are running Snort/Suricata, when you click on the "!" ICON to Resolve an IP, you will find that most of the IPs are already listed in the BlockLists. You will also see over time that it will pickup an Alert for an IP but the Blocklists do not have the specific IP but there are several IPs within the same Range that are being Blocked.

Also in diag_dns.php, there are several IP Reputation Links that can help you determine the Reputation of any Blocked IP before you remove a list, or Add an IP to the SAFE Alias list.

Let me know if you need any clarification or any other help.

The 2>&1 at the end means don't bother emailing me everytime you run this job. It doesn't make any sense running that from the console, since the system wasn't intending to email you anyway.

foetus

@justsomeone:

Awesome post, thanks!

I'm trying to install pfiprep, and am getting an error when installing one of the dependencies.
$ pkg_add -r grepcidr
tar: Failed to set default locale
Can someone please tell me what to do to fix this?

Dont run the command from Diagnostics -> Command.
Run it from SSH console (shell) or direct console (shell) and it should work.

BBcan177

I have added a "PATCH" to make the necessary changes to the pfBlocker Widget so that you don't need to modify that file from the shell.

If you don't have the pfSense Package "System Patches", it is available in the pfSense System:Packages list under "System Patches"

Click the "+" Icon to add a new Patch
Enter a Description (pfBlocker Widget Patch)
In the Patch Contents Dialog Box - Copy/Paste from my Gist the contents of this
link below:

[ [b]https://gist.github.com/BBcan17/67e8c456cb399fbe02ee#file-pfblocker-widget-php_patch ]

Keep the other default settings as is.

Click "Test" and confirm that it can be applied Successfully. Then click "Apply"

In my Gist, is also a "Patch" to Include the pf IP Reputation Blocklists in the "Firewall Logs "!" Lookup functions.

[ [b]https://gist.github.com/BBcan17/67e8c456cb399fbe02ee#file-diag_dns-php_patch ]

UPDATE:

When Applying a Patch, the File that you want to modify has to be an original pfSense Version for the Patch to be applied successfully.

foetus

@BBcan177:

Out of Curiosity, which Lists did you disable?

Anything related to Spamhaus. Reason is simple, location where my current test setup is already has this filtered on a higher up level. No point doing the same twice :)
I was already using a white-list for my classic pfblocker lists, so that will just be extended (which wont be allot since most anti-spam lists are disabled by default). Its not the reason for disabling them :).

On production networks I'm pretty much gonna use the defaults since most people here and a couple of fellow testers in my area are confirming a low false positives rate.

BBcan177

@foetus:

@BBcan177:

Out of Curiosity, which Lists did you disable?

Anything related to Spamhaus. Reason is simple, location where my current test setup is already has this filtered on a higher up level. No point doing the same twice :)
I was already using a white-list for my classic pfblocker lists, so that will just be extended (which wont be allot since most anti-spam lists are disabled by default). Its not the reason for disabling them :).

On production networks I'm pretty much gonna use the defaults since most people here and a couple of fellow testers in my area are confirming a low false positives rate.

The Spamhaus drop and edrop are also included in the ET Lists, but there are still times where one list hasn't sync'd with the others and there are Gaps. I always like to go to the source of the Lists and use those. IBlock has a lot of references to other lists but they just re-package them.

Its not a lot of extra IPs and I would consider adding them just in case there is any issues from Higher Ups syncing at different times. The Daily Log does help to see how often Lists are updating by looking at the "Count"

Emerging Threats fwip rules.

Raw IPs for the firewall block lists. These come from:

C&C servers identified by Shadowserver (www.shadowserver.org)

Spam nes identified by Spamhaus (www.spamhaus.org)

Top Attackers listed by DShield (www.dshield.org)

More information available at www.emergingthreats.net

Please submit any feedback or ideas to emerging@emergingthreats.net or the emerging-sigs mailing li

Spamhaus also has a Botnet Command and Control List, that is Free but it is not readily available. You need to request access to that for your Downloading IP address and use RSYNC. The code to do that is included in the script. I have been working them, to see if they will just release it like the drop and edrop so it makes it easier.

Its also nice to see that Grepcidr was partially funded by Spamhaus also.

http://www.spamhaus.org/news/article/714/new-ipv6-cidr-searching-tools-released-grepcidrs

I have also been working on Integrating the Script to use the New Emerging Threats IQRISK IP Rep lists. Its currently being beta tested by CINO. Unfortunately they want approx $1400 a year per license for it… :o :o But If you are a business, I would recommend that over BS locally install Virus Detection Software. I know that Bill has also been trying to get them to reduce the price for "Home Use"!

I am also trying to get access to "ShadowServers" Lists, but they are taking forever to approve.

http://www.shadowserver.org/ccfull.php
http://www.shadowserver.org/ccdns.php

If you guys find any bugs in the code or have some "Alternative" Methods to find Offending IPs, I am always open to see if I can make the Script better/more efficient.

BBcan177

@BBcan177:

I have added a "PATCH" to make the necessary changes to the pfBlocker Widget so that you don't need to modify that file from the shell.

If you don't have the pfSense Package "System Patches", it is available in the pfSense System:Packages list under "System Patches"

Click the "+" Icon to add a new Patch
Enter a Description (pfBlocker Widget Patch)
In the Patch Contents Dialog Box - Copy/Paste from my Gist the contents of this
link below:

[ [b]https://gist.github.com/BBcan17/67e8c456cb399fbe02ee#file-pfblocker-widget-php_patch ]

Keep the other default settings as is.

Click "Test" and confirm that it can be applied Successfully. Then click "Apply"

In my Gist, is also a "Patch" to Include the pf IP Reputation Blocklists in the "Firewall Logs "!" Lookup functions.

[ [b]https://gist.github.com/BBcan17/67e8c456cb399fbe02ee#file-diag_dns-php_patch ]

Cino has found a small bug in one of the patches.

For the pfBlocker Widget, you will need to click on "Revert" back to previous version.

Edit the patch and copy/paste the new code from my Gist.

Than "Save", "Test", and "Apply"

If you don't Have pfBlocker Installed, you could download the pfIPreputation.widget.php and save that in /usr/local/www/widgets/widgets

And adding it to the Status:Dashboard page with the "+" icon.

NOTE: For [ [b] diag_dns.php ] Don't forget to change the path to the /pf folder
from the Patch - /YOUR/BLOCKLIST/FOLDER/*
Example /home/USER/pf/*

UPDATE:

If you are having issues applying the patches, it could be due to copy/paste issues.

You can also use the "URL/Commit ID" in the Patch Edit Menu, and use these Links for each of the patches.

https://gist.githubusercontent.com/BBcan17/67e8c456cb399fbe02ee/raw/f3ca0e1d3dd4a07a21796d033dad06a4ce1cc218/diag_dns.php_PATCH
(Once you "fetch" the patch, you will need to manually edit the path to the [ /pf folder ]
example [ /home/USER/pf/* ], then click "Test" and then "Apply". Do not paste any code into the "patch contents" the url will do that.

https://gist.githubusercontent.com/BBcan17/67e8c456cb399fbe02ee/raw/3c3d508cec136788cea6abd98d49d367f9b75b7a/pfBlocker.widget.php_PATCH

UPDATE2:

When Applying a Patch, the File that you want to modify has to be an original pfSense Version for the Patch to be applied successfully.

foetus

and a day later both bing and google are blocked ;D

How are they blocked? Lists or suricata?