Taming the beasts… aka suricata blueprint

A Former User

Recommended lists.
This MAY or MAY NOT get changed in the future. Please go through the thread in the future to see if an updated version is available. I'll try to keep the updates to posts, for as much as I can, after that, I'm afraid to cause the forum's black hole.
The left column is my alias name, and the right column is the file (/usr/local/www/badips/$file), See post above how to add this to the aliases

abuse_palevo AbusePalevo.txt
abuse_spyeye AbuseSpyeye.txt
abuse_zeus AbuseZeus.txt
alienvault ALIENVAULT.txt
atlas_attacks Atlas_Attacks.txt
atlas_botnets Atlas_Botnets.txt
atlas_fastflux Atlas_Fastflux.txt
atlas_phishing Atlas_Phishing.txt
atlas_scans Atlas_Scans.txt
atlas_ssh Atlas_SSH.txt
ciarmy CIArmy.txt
danger_rules DangerRulez.txt
drg_http DRG_http.txt
drg_ssh DRG_SSH.txt
drg_vnc DRG_VNC.txt
dshield dShield.txt
et_comp ET_Comp.txt
feodo_bad Feodo_Bad.txt
feodo_block Feodo_Block.txt
geopsy Geopsy.txt
iblock_badpeer IBlock_Badpeer.txt
iblock_bt_fs IBlock_BT_FS.txt
iblock_bt_hijack IBlock_BT_Hijack.txt
iblock_bt_spy IBlock_BT_Spy.txt
iblock_bt_web IBlock_BT_Web.txt
iblock_onion IBlock_Onion.txt
infiltrated Infiltrated.txt
malc0de malc0de.txt
malware_group MalwareGroup.txt
mdl MDL.txt
nothink_bl NOThink_BL.txt
nothink_malware NOThink_Malware.txt
nothink_ssh NOThink_SSH.txt
openbl OpenBL.txt
p24_spamcop p24Spamcop24.txt
shunlist Shunlist.txt
spamhaus_drop Spamhaus_drop.txt
spamhaus_edrop Spamhaus_edrop.txt
sri_attackers SRI_Attackers.txt
sri_cc SRI_CC.txt
tor TOR.txt
virbl VirBL.txt
vmx VMX.txt
watchguard WatchGuard.txt

A bit of searching around in the script should be enough to match my list to the lists enabled in the script. I could provide a script snippet showing the enabled lists, but meh :P
This particular list of lists (again, no pun intended) is in use on a production network, providing connectivity to servers and clients. I can't remember the last time I had to fiddle around with the lists due to a false positive. I do remember the last time was removing alienvault from the lists. Been running like this for a few months now (on an old customize version of the script, and that's why I had to edit the above post multiple times, migrated to a newer version).

It's a total of 44 lists. Double that (each list needs 2 floating rules) and you only have to set 88 floating rules.
A reason why I don't use the script's tier functionality (combining multiple lists to a single alias) is because since this is a production network I might have to identify false positives and make changes fast to it. Haven't had to change the floating rules for a while though :D.

Now, class, finish your homework by the next time. And make sure you brush up on the previous parts when that time comes . Hint: writing custom suricata rules for use on a network gateway. Yes diving straight to the deep waters. You've already got so far in your training, yet still fail to understand the reason for all this work…tsk..tsk...young grasshoppers...
You are already ahead of the curve, just the information provided in these two parts could land you a 5K/month job as a network administrator for a Fortune 500 company. Trust me, you know stuff the competition doesn't even imagine exists. Next up, we'll take your training to the next level and offer you the chance to become one of the best network administrators/security experts in the world by awakening the beast...

A Former User

Suricata

Everyone should have completed the two previous parts of the guide before moving on to this part. This sets everything up so that when the beast is truelly awaken, it will work in your favor, instead of against you. Some instructions in this part (and explanations) apply equally to suricata and snort. I'll mention only suricata, since this IS the suricata topic after all.

I know most of you are anxious and want to dive right in. First things first. We need a "wax on, wax off" session first.

Who will be our attackers
Bet you didn't see that coming. Our attackers are persons/corporations/state-funded-actors that are not previously identified. Since we are using IP lists to block traffic from known sources, if the attacker's packets pass through pfsense (reaching the wan side is ok, since known are dropped) then the attacker has not attacked a critical sensor system (one that adds to the lists). Keeping with snort's topic's NSA theme "the SIGINT simply isn't there".

Since we have already identified the "usual suspects", we are dealing with, let's say 20% of the total bad traffic (assuming 80% is known).

How will you know that packets are arriving from an unknown bad IP?

Those packets will reach a legitimate port and pass across pfsense to a server/process listening behind pfsense. In this case they will be analyzed in transit (or intercepted if running in IPS mode) by our fire breathing dragon (suricata).
Those packets will NOT reach a legitimate (open) port and will be dropped by pfsense's default block rule. We could monitor the default rule and set up bans based on that. We'll do it in a far more interesting way, using suricata.

The way suricata works on pfsense can be thought as being as close to the network as possible. Packets (technically copies of packets so far, since we are not running in IPS mode) are passed through suricata at the same time pfsense gets the actuall packet. So let's say a bad source packet arrives at our firewall. The packet is copied, then the original is forwarded to pfsense at the same time the copy is forwarded for analysis. The original will be dropped by pfsense, but the copy can fire up an alert.

Why is this useful? We can leverage our suricata systems to keep their own IP lists, in the form of banned hosts. Bad packets (bad refers to a closed port destination) will be dropped by pfsense with no indication to the attacker, and at the same time our attacker will be identified and put on our own list. If the attacker decides to perform another port scan, for example, a week later, suricata will still pick up on that and fire up a new alert, refreshing the initial ban day. So instead of being 28-7=21 days from now, with the refreshed alert is will be 28 days from now (pfsense is already dropping the original packets, since that host is now on an internal alias, and the packets will not be logged). You can see that as long as the attacker attacks us, he will be kept on a perpetual ban. The harder they try, the worst it is for them.

NOTE: The rules explained in this part are rules that should be run on each and every single gateway system. They are rules especially designed for such use, since our bastion (first systems that can be reached from the network) must be our bastions, our "impenetrable" outer perimeter. Quotes because as proven by history, even the mightiest of the castles have fallen.

QUIZ: Who remembers why low (privileged ports) are useful?

Using the low ports we can detect port scans looking like legitimate traffic reaching "official" server ports. Let's examine a random example:
The attacker sends traffic for port 514 (syslog), containing a legitimate syslog entry.
We have 2 choices to deal with this traffic. Set up a rule especially for it, or set up a general unused ports rule. Remember, suricata will add a new alert each time a rule is triggered, which means if an attacker scans 80 low ports, there will be 80 alerts logged for it, making the blocked tab page a mess. If the alert matches the previous one though, only the timestamp is updated, which keeps everything running smoothly. Further analysis can be performed through the logs.

In other words, packets from a source NOT our home network (or belonging to an external network) and a source port of any, reaching a port NOT in use on our home network, will alert us.
Congratulations young grasshopper, you have just created the world's top snort/suricata rule. It is internally referred to in The Company as "The Golden Standard for writing IDS/IPS rules" or, depending on the day "The two rules to rule them all". Why is it the golden standard? Those of you that failed to see that it is impossible to cause a false positive by this rule so far are dangerously close to failing the class. Unless you ignored a port in use (which shouldn't happen all that often) a packet that triggers this rule is a packet that will not route across pfsense. Simply put: a legitimate alert.
Be extremely careful with the knowledge. It can be used for good, or it can cause great harm. These two (a TCP and a UDP), the "The two rules to rule them all", rules accounts for 10K monthly banned hosts (hosts set up to be unbanned after 28 days, but trigger the rule before that).

Let's examine why the rule is so successful. We have already mentioned its ability not to cause false positives. One of its other abilities is that it acts as close to the actual protocol as possible. There is no need to pass a packet through deep packet inspection to see what is actually in it, since it is a packet that would otherwise be dropped. Saves us the processing and provides a further ability: detection of traffic that would otherwise be missed by suricata.

"Wax on, wax off" session: HTTP traffic comes in wanting to initiate a connection to pfsense's HTTP port. We are already running HTTP servers behind pfsense, but they use a different IP for that. A legitimate connection attempt to a non legitimate destination.

So, a packet has arrived that is sourced NOT from our network, having a source port of any, and is destined for a NOT HTTP server, to an HTTP port. Get it? ;)

The next question will separate those that want to use this topic as a way of getting a 10K/month job (5K was when you finished a previous part, you are now in the position to basically walk in to any company you want and tell them to hire you for an arbitrary salary) and those that are truelly interested in network security.

Should a low (<1024) port communicate directly with another low port?
The answer is no. Low ports should always be used as the DESTINATION of the connection, NOT the source.
So, a packet that claims to come from a port which starts at 0 and ends at 1023 shouldn't be reaching a port that starts at 0 and ends at 1023. If most potential employers failed to see why you are one of the best there is in the network security industry, tell them the above rule. They will not be able to find the pen to sign your contract fast enough.

You now have the necessary knowledge of setting up custom rules for use in pfsense's suricata use case (a network gateway). This allows you to shift processing from Layers 5-7 of the OSI model (actually opening up the packet and looking inside, which is useless if using encryption), to Layer 4 (TCP is layer 4, you are forgeting the first level, physical (cables)). This offers lower load, since you are spending less time processing the traffic, while at the same time not reducing your security. Simply put, you have just awaken the beast…

Those of you that reached this far and "saw the light" will go on in their lives becoming the top of your classes in the best universities of the world, and upon graduating from said universities, will pursue a career in network security, quite possibly for an alphabet-soup-agency. What ever your future goals are, I have a simple request from you: "Always try to the best at what you do. There is the first place and there is the last place. Second place is not an option."

You have now set up the battle so that the sun is behind you, your shield is on your left, the hill is on your right, and the enemy is straight ahead. Sun Tsu was right, you are now only dealing with determined attackers. Suricata is screaming "bring it on".

That completes the "static" part of this topic. Since old posts cannot be edited, I wanted to put this as close to the top as possible. The IP lists list and the enabled suricata rules will be posted from time to time to this thread, so please check the entire thread for the most up to date version.

FAQs
"Why didn't you just gave away your custom rules?" Because quite frankly I've spent 17 years of my life studying and don't want to see my efforts being used as a way for an idiot to get a high paying job. Those that truelly want to learn about suricata (and snort) rules have already picked up the hints and are on their way to writing their rules already.
"Why did you give away all this?" You having a more secure network is inversely proportional to my time spent sifting through logs, trying to identify infected/malicious/compromised remote hosts :D. That and the fact that setting up IDS/IPS systems my way has been proven to be the best way to set them up.
"Who are you?" As mentioned in the first post, "Security through obscurity".
"Who do you work for?" As mentioned in the first post, "Security through obscurity".

Next up, and after much anticipation, suricata enabled rules. :D

BBcan177

I have been working on a script that downloads over 50 different Blocklists and performs a duplication check to reduce the size of the data. It can download .CSV, .TXT, ,GZ, .ZIP files and also scrape from certain websites that post only a web copy of their Blocklists.

ie : ET, Spamhaus, IBlock, dShield, Atlas, Alienvault etc.. I have been researching Blocklists for several Months and have found the current list to be beneficial in Blocking Malicious IPs.

It utilizes a tool called "Grepcidr" to make the de-duplication work.

It also looks at the number of IP addresses found in a /24 range and can condense the list and enter a /24 block instead. This is done in three ways

Using a "max" variable, if it finds over the Max variable it will perform a local Maxmind Geoip Database lookup and will process a /24 block for configured Foreign Countries on an individual Blocklist Basis.
Using a "dmax" variable if it finds over the dmax variable it will perform a Maxmind Geoip Database lookup and will process a /24 block for configured Foreign Countries at the end of the download process on all of the Blocklists together.
Using a "pmax" variable, if it finds over the dmax variable it will process a /24 Block excluding Country Code whitelist at the end of the download process on all of the Blocklists together.

So I set max to 5, dmax to 5 and pmax to 50 in my setup.

Depending on how aggressive / conservative an admin wants to configure the processes or disable them completely and just use the de-duplication processes.

I have been testing it for several weeks without the need for pfBlocker to reload the Alias tables. Options also exist to place the Repeat Offender Ranges into a "Match" List that can be used with "Floating Rules" to report when these suspected ranges are in your network.

I also found a way to use the Maxmind database to make a Country Code Specific Blocklist, excluding whitelisted countries.

Here is a snapshot of the IP count for each list (mail server lists not included)

Orginal list of approx 700,000 IPs (worst of the worst!)

===[ [b]Blocklist IP Counts ]===========================

240818 total
127685 /home/user/pf/BadIPs.txt
47095 /home/user/pf/IBlock_Badpeer.txt
23423 /home/user/pf/ALIENVAULT.txt
6406 /home/user/pf/Geopsy.txt
6282 /home/user/pf/SRI_Attackers.txt
5307 /home/user/pf/VMX.txt
4818 /home/user/pf/IBlock_Onion.txt
3692 /home/user/pf/Infiltrated.txt
2904 /home/user/pf/IBlock_BT_Spy.txt
2813 /home/user/pf/HackedReport.txt
1570 /home/user/pf/ET_Block.txt
1423 /home/user/pf/IBlock_BT_Web.txt
1139 /home/user/pf/DRG_http.txt
1057 /home/user/pf/MDL.txt
1037 /home/user/pf/ET_Comp.txt
559 /home/user/pf/SnortBL.txt
446 /home/user/pf/Atlas_SSH.txt
378 /home/user/pf/Greensnow.txt
371 /home/user/pf/CIArmy.txt
358 /home/user/pf/Spamhaus_CC.txt
357 /home/user/pf/IBlock_BT_FS.txt
275 /home/user/pf/MTA.txt
239 /home/user/pf/NOThink_Malware.txt
188 /home/user/pf/Maxmind_Proxy.txt
152 /home/user/pf/DRG_VNC.txt
85 /home/user/pf/dShield_Top.txt
83 /home/user/pf/Blut_TOR.txt
76 /home/user/pf/DRG_SSH.txt
68 /home/user/pf/BotScout.txt
66 /home/user/pf/Juniper_Spam.txt
56 /home/user/pf/Snort64.txt
49 /home/user/pf/HoneyPot.txt
40 /home/user/pf/malc0de.txt
39 /home/user/pf/IBlock_BT_Hijack.txt
39 /home/user/pf/DangerRulez.txt
37 /home/user/pf/NOThink_BL.txt
36 /home/user/pf/MalwareGroup.txt
28 /home/user/pf/Feodo_Block.txt
23 /home/user/pf/Spamhaus_edrop.txt
21 /home/user/pf/OpenBL.txt
17 /home/user/pf/WatchGuard.txt
15 /home/user/pf/dShield_Block.txt
14 /home/user/pf/SRI_CC.txt
10 /home/user/pf/Atlas_Fastflux.txt
8 /home/user/pf/Shunlist.txt
8 /home/user/pf/Atlas_Phishing.txt
7 /home/user/pf/Atlas_Botnets.txt
6 /home/user/pf/NOThink_SSH.txt
4 /home/user/pf/Atlas_Attacks.txt
3 /home/user/pf/Atlas_Scans.txt
1 /home/user/pf/Spamhaus_drop.txt <–-- These below are actually empty with
1 /home/user/pf/ISC_top10.txt a "1.1.1.1" as a placeholder.
1 /home/user/pf/Feodo_Bad.txt
1 /home/user/pf/AbuseZeus.txt
1 /home/user/pf/AbuseSpyeye.txt
1 /home/user/pf/AbusePalevo.txt

(Here is the status of the pfSense Alias Tabes (Groups/Tiers) update process)

===> Sanity Checks PASSED, Updating Group Lists <===

Updating [ IR_PRI1 ] [ ET_Comp ET_Block Spamhaus_drop Spamhaus_edrop Spamhaus_CC CIArmy AbuseZeus AbuseSpyeye AbusePalevo dShield_Top dShield_Block SnortBL ISC_top10 Snort64 ]
no changes.

Updating [ IR_PRI2 ] [ ALIENVAULT Atlas_Attacks Atlas_Botnets Atlas_Fastflux Atlas_Phishing Atlas_Scans Atlas_SSH SRI_Attackers SRI_CC HoneyPot ]
no changes.

Updating [ IR_SEC1 ] [ MDL NOThink_BL NOThink_SSH NOThink_Malware DangerRulez Shunlist Infiltrated DRG_SSH DRG_VNC DRG_http Feodo_Block Feodo_Bad WatchGuard VMX Geopsy MTA Maxmind_Proxy BotScout HackedReport Juniper_Spam Greensnow ]
no changes.

Updating [ IR_SEC2 ] [ MalwareGroup OpenBL malc0de ]
no changes.

Updating [ IR_SEC3 ] [ BadIPs ]
no changes.

Updating [ IR_IB ] [ IBlock_BT_Hijack IBlock_BT_FS IBlock_BT_Web IBlock_BT_Spy IBlock_Badpeer ]
no changes.

Updating [ IR_TOR ] [ IBlock_Onion Blut_TOR ]
no changes.

pfSense Table Stats
–-----------------
table-entries hard limit 800000
Table Usage Count 512459

Thanks to jflsakfja, for helping to test it out. ;) ;)

Here is a link to GitHub(Gist) where you can download the files. This provides more functionality than what is available in the vanilla pfBlocker.

[ [b]https://gist.github.com/BBcan17/67e8c456cb399fbe02ee ]

There are installation instructions in the script. If you need more help send me a PM or a post.
Please be careful when working in the shell.

** [ From time-time please review any Revisions in the Gist ]**

foetus

"also scrape from certain websites that post only a web copy of their Blocklists. "

That last part could become quite handy indeed.
Thanks for sharing your script.

BBcan177

UPDATE: I have updated my script pf IP Reputation Manager to v2.3.1

https://gist.github.com/BBcan17/67e8c456cb399fbe02ee

If anyone needs any help to install please let me know.

Your Feedback is always welcome.

Cino

@jflsakfja great write up! I have to agree with you; block everything and open up what is needed… To bad I don't do this on my box at home, well my lan is open but my other interfaces are not. So i'm half way there.

@BBcan177 Thank you so much for sharing. When jflsakfja mention the script the other day, I started to search everywhere (including github) but couldn't find it :-( I've been a big fan of pfBlocker but there are too many dups because of list overlap and it doesn't allow you to import other formats.. Maybe someday your script and pfBlocker can get married and have a honeymoon...

I'm using the script basically out of the box without making any major changes to it (Other then disabling the TOR EXIT lists, enabling 2 list from the MAILSERVER section and BadIPs.

Since pfIPRep creates and updates the following pf Tables:
IR_PRI1
IR_PRI2
IR_PRI3
IR_SEC1
IR_SEC2
IR_SEC3
IR_IB
IR_TOR
IR_MAIL
IR_CC
I use them for my URL Alias name. This way, pfSense doesn't create a new pf Table

Url Aliases:

I've created my floating rules like this:

Example of my Outbound Block rule:

Notice in the the description, I started it with the alias name.. This could be used for label matching within your USER: ruleset if needed.

Example of my inbound Reject rule:

urlalias.JPG_thumb

floatingrules.JPG_thumb

blockoutbound.JPG_thumb

rejectinbound.JPG_thumb

BBcan177

Thanks Cino, I'm glad that you are using the Script ;)

I suggest that you add "enable logging" on the Floating Rules so you can see what is getting Blocked.

I have also added your recommendations for the Widget. (pfIP_Reputation.widget.php)

The widget can be found in my Github GIST.

Copy and save that file in [ [b]/usr/local/www/widgets/widgets ]

From pfSense Status:Dashboard

Click the "+" Icon and add the new "pf_IP_Reputation" Widget"

You may also notice that there is also a "mastermatch" alias that can also be placed into the Floating Rules as a "MATCH" Rule. These are IP Ranges that are in the Safe Country List, but are repeat offending IP Ranges. This can help to indicate malicious behaviour.

A Former User

@Cino:

Example of my Inbound Block rule:
…
Example of my Outbound Reject rule:

FTFY :)
~~Other than that, everything looks good.~~
EDIT!!!! MISSED THE QUICK CHECKBOX. TICK THAT!

A small update on the suricata rules. I'm delaying their release because I wanted to enable all the rules and work my way towards an FP free install, just like snort. The trouble is that I've had to do all the work I did over a few years, in a few weeks. The good news is that most disabled rules are the same on snort and suricata. The medium news is that I still haven't hit all the FPs :p.

I haven't made up my mind if I'll post on the topic recommendations to disable rules no longer needed, since they are handled by the custom rules mentioned previously. Maybe I'll add a small note to them that they can be disabled if using custom rules.

BBcan177

I have posted version 2.3.2 of pf IP Reputation Manager to my Gist.

https://gist.github.com/BBcan17/67e8c456cb399fbe02ee#file-pfiprepman_2-3-2

No recent changes to the pfiprep script. You can replace the pfiprepman script completely with this one.

I have also added some code to the pfSense php script /usr/local/www/diag_dns.php

ScreenShot http://imgur.com/L9GeYa2

This will allow you to click on a Blocked IP in the Firewall Log, and it will show you which Blocklists have the Blocked IP Listed. At the bottom are several Threat Sources that can be referenced as well. When 2.1.4 comes out, I will post a new Patch as there are some other changes being made in this file also.

I have created a "patch" that can be applied to pfSense v 2.1.3 (Haven't tested it on other versions, but should work?)

If you don't have the pfSense Package "System Patches", it is available in the pfSense System:Packages list under "System Patches"

Click the "+" Icon to add a new Patch
Enter a Description (diag_dns.php Patch)
In the Patch Contents Dialog Box - Copy/Paste from my Gist the contents of this
link below:

https://gist.github.com/BBcan17/67e8c456cb399fbe02ee#file-diag_dns-php_patch

After you paste the patch, look for this line and change it to your pf folder location.

```
$query_list = grep {$hosttrim} /YOUR/BLOCKLIST/FOLDER/* | sed 's/^.*[a-zA-Z]\///';


  These changes need to be made also:

  Base Directory - **/usr/local/www/**

  Click **"Save"**

  Click **"Test"**

  Look at the top of the patch that it can be successfully added, then
  Click **"Apply"**

BBcan177

Here is a good example about why Reputation in a /24 network is important to monitor:

A recent DNS amplification attack caught by Snort.

The IP Range is listed by 5 different Blocklists, but none of them have that exact IP Range.

The Country code for 204.124.183.211 is in the US. I currently have my script set to not apply a /24 block to a repeated offender IP Range based in the US.

( See ScreenShot ) http://imgur.com/xkRXSFi

A Former User

Suricata Disabled Rules

ALWAYS DISABLE RULES! SUPPRESSING HAS OTHER USES, HIDING KNOWN FALSE POSITIVES IS NOT ONE OF THEM!

What this list is
This list is the definitive guide on what rules should be edited/deleted upstream. Ideally the ET guys will come along and the next iteration of this list will be blank. Practically I've spent months trying to get certain rules deleted, with no success. An example:
2010228 ET POLICY Suspicious Microsoft Windows NT 6.1 User-Agent Detected <<< so WTF is windows 7 then??? DELETE UPSTREAM

Windows 7 was (since RTM), is (currently publicly available versions) and always will be, 6.1. Yes it doesn't make sense. It's actually the bitter truth. Windows 7 IS Windows 6.1. Other than Microsoft's NSA ties, there is nothing suspicious about it. An OS used by the majority of PC users should not generate a "suspicious" alert when performing the initial updates after a clean install.

How to use this list
First of all head over to the interface's categories page. Go through the list and enable all categories NOT having "DO NOT USE!" in front of them. Leave the rest disabled. There are some suricata internal rules, that will not be shown on the categories page.

After that, move over to the rules page. Select the first entry in the dropdown, and enable all rules. Then manually search for the IDs listed, and click the icon next to them. Open a second browser tab that shows the dashboard. Look at the CPU load. Back on the rules page, hit apply. Back to the dashboard, and wait for the CPU load to return to idle. When that's done, THEN move to the next category in the dropdown.
That's one way to do it. The other way is to enable all in a category,hit apply, and wait for the CPU to idle again, before moving to the next category. THERE ARE 2 EXCEPTIONS TO THIS (the following 2 rules MUST be disabled after hitting enable all, BEFORE HITTING APPLY!

decoder-events > 2200003 SURICATA IPv4 truncated packet
emerging-policy > 2010815 ET POLICY Incoming Connection Attempt From Amazon EC2 Cloud

Rule 1 will generate a billion alerts per second (trust me on this) and rule 2 is (here's the first revelation of it on the entire internet) responsible for 500MB+ RAM usage. Disabling this rule makes sure that the suricata interface does not ran out of memory while trying to load its rules.

With our selected rules enabled, it's time to head over to the alerts tab. Refresh it after a couple of minutes and watch for rule IDs listed here. Click the icon next to them to force them to a manually disabled state (and never having to deal with them again). While disabling a rule, make sure to keep an eye out on your second tab (dashboard). Only remove a host from the blocked list (icon next to IP) AFTER the cpu returns to idle, otherwise you risk generating an alert again since the rule has not yet been unloaded.
Note to bmeeks: Pretty please bring back the old way of handling manually disabled rules. Manually disabling a rule from either the alerts tab or the rules page, should turn the rule into a manually disabled rule (pale yellow). Currently the rules page turns it into the rule's default state. This is NOT recommended when using this list. Having both setting to manually disabled, allows the list to be used as it was meant to be used. Enable all, then find the 10 that need to be disabled, disable them, and apply. Rinse, repeat.

Explanations:DO NOT use the list without reading this!
DO NOT USE! = categories that should be disabled
xxxxxx (number) = rule's ID
text before <<< = rule's explanation
text after <<< = why the rule was disabled

before xxxxxx = WILL ONLY BE FOUNT ON SOME CATEGORIES.Rules that generated false positives on a suricata production network. I go with the enable all, then manually disable based on the alerts route. These are rules that generated alerts, but the rest of the rules in that category (NO # in front) are known FPs, but have NOT yet generated an FP on the suricata install. Hope that makes sense.

and no # rules explained. BLUE rules are rules that generated FPs on suricata

decoder-events
2200029 SURICATA ICMPv6 unknown type <<< breaks IPv6 tunnels
2200079 SURICATA ICMPv6 invalid checksum <<< breaks sixxs's IPv6 tunnel

emerging-chat
2001595 ET CHAT Skype VOIP Checking Version (Startup)
#2002157 ET CHAT Skype User-Agent detected <<< obvious
Hope that clears it up. Sorry for that, but I had to start from the snort topic's list and work my way forward, so I had to copy over the list and manually disable the alerts as they are generated. You are welcome!

The Definitive Enabled Rules List
Getting a 500 internal error while trying to upload the list. ~~Will try again (either edit this post, or a new post)~~ The list cannot be uploaded either to this reply, or a new reply, due to a 500 error. Will see where I can paste it so that I can also edit it in the future, which should also help with keeping the thread clean.
List is here:
https://github.com/jflsakfja/suricata-rules/blob/master/list.txt

A Former User

Writing custom rules for deep packet inspection
Suricata's intended use is intercepting the packets as they are passing through the network, reassemble them (if needed), open them up and peek inside, looking for specific patterns, and deciding whether to alert or not (or drop, if running in IPS).

We'll examine a specific example of a known spammer trying to send spam to our email server.
The logs show this:
2014-06-24T16:35:52+03:00 emailserver1 postfix/smtpd[4324]: NOQUEUE: reject: RCPT from unknown[192.168.1.1]: 554 5.7.1 Service unavailable; Client host [192.168.1.1] blocked using zen.spamhaus.org; http://www.spamhaus.org/query/bl?ip=192.168.1.1; from= example@example.comto= example2@example2.comproto=ESMTP helo=<[192.168.1.1]>

The red part shows that the email was rejected because we have set up our email server so that it first checks if the sender is a known spammer, then decides whether to accept the email or not.
The blue part is the server's reply sent to the client (the spammer).

Why not use the IP directly in an alias on pfsense? This is not the scope of this post. This post tries to show how to write custom rules based on what you are trying to detect.

Back on topic. We know so far that we are looking for a packet that is originating from our email server (port 25 since that is unencrypted), and could end up anywhere on the internet, containing a specific reply.

So far the rule is:
alert tcp $SMTP_SERVERS 25 -> $EXTERNAL_NET any
This will not get you far, since you detected ALL traffic originating from port 25, with no specific contents.

Examining our reply sent back to the client we see that the easiest way to detect this specific traffic is to use the part in red shown below. This is the common part for all replies, and states the reason why the email was denied:
554 5.7.1 Service unavailable; Client host [192.168.1.1] blocked using zen.spamhaus.org; http://www.spamhaus.org/query/bl?ip=192.168.1.1

Ok, now we know what we are looking for, but it's time for us to understand our alert.
Our chosen message will be:
Known SPAMMER
The classification for this alert will be bad-unknown (use http://manual.snort.org/node31.html#SECTION00446000000000000000 and choose the most appropriate classtype for your rule)
Our rule version (revision) will be 1
Our sid (signature ID) must be a high value so that it will not interfere with other rules. Each rule MUST have a unique ID! I recommend using a value in the 9mil range. In this rule we use 9900003 because we have other custom rules before it.

Don't forget that we are only interested in established connections (actively communicating) and the traffic direction is from the server to the client.

Putting it all together gives us this:
alert tcp $SMTP_SERVERS 25 -> $EXTERNAL_NET any (msg:"Known SPAMMER"; flow:established,from_server; content:"blocked using"; classtype:bad-unknown; sid:9900003; rev:1;)

An example of the alert generated:
2014-06-24T16:09:49+03:00 pfsense1 suricata[21435]: [1:9900003:1] Known SPAMMER [Classification: Potentially Bad Traffic] [Priority: 2] {TCP} 192.168.200.1:25 -> 192.168.1.1:63651

Since our IP is not banned, suricata will try to ban both IPs (source+destination) (ALWAYS set it up like that) but ignore the source.

We have caught our first offender using suricata's deep packet inspection! :D

The spammer will return after 28 days (since his IP will be unbanned) and try to send another spam email. He will be blocked again for a further 28 days. 1 email per month is nothing compared to 1000 per day.

Taking apart the rule and finding flaws (if any)
The rule should only fire up on messages originating from our email servers. Good!
The rule cannot generate a false positive since the email WILL be rejected regardless by our email server. Good!
The rule shows a generic alert message. Good AND Bad!
The rule's alert message is good in the sense that it does not give too much information about the origin of the block (no list's name appears).This is the bad part. This also allows us to keep the alerts tab clean, since a known spammer is a known spammer regardless of where the intel is coming from. If an IP is listed on list ABC on one month, and list DEF on the next month, two alerts will be logged in the alerts tab. For now it's nothing. Detecting a portscan using my previously recommended rule would end up with a thousand alerts logged on the alerts tab for that IP (NOT if using a generic "Incoming connection to unused port" message). This is the good part!

Congratulations. You have now finished your training and are part of the industry's elite in network security. You are one of the best in this field! Use the knowledge wisely./example2@example2.com /example@example.com

Cino

great work jflsakfja!

You may want to add decoder-events - 2200013 IPv6 truncated packet to the list for noise. My alert file is mostly this alert. What is strange, none of these show up in the GUI, only in the Alert file itself… Because there is no IP, i'm thinking the GUI doesn't display it. But that's for another thread after I confirm my findings.


06/25/2014-06:27:46.251740  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 01 EA 40 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
06/25/2014-06:27:46.250371  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 01 EA 40 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
06/25/2014-06:27:46.318718  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 01 EA 40 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
06/25/2014-06:27:46.251489  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 01 EA 40 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
06/25/2014-06:27:46.319106  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 01 EA 40 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
06/25/2014-06:27:46.319352  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 01 EA 40 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
06/25/2014-06:27:46.543947  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 0A B8 B0 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
06/25/2014-06:27:46.319496  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 01 EA 40 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
06/25/2014-06:27:46.543837  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 0A B8 B0 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
06/25/2014-06:27:46.986163  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 05 E7 64 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]
06/25/2014-06:27:46.986289  [**] [1:2200013:1] SURICATA IPv6 truncated packet [**] [Classification: (null)] [Priority: 3] [**] [Raw pkt: 00 25 90 02 31 65 00 01 5C 33 54 41 86 DD 60 05 E7 64 05 B4 06 34 26 10 01 60 00 11 00 11 00 00 ]

A Former User

Getting the suricata logs on a remote syslog, and I'm running IPv6 tunnels because my upstream provider has vowed not to support IPv6 until AFTER they are completely cut off from the Internet (it will be the last ISP on earth that moves to IPv6). It might be due to the fact that their "3rd level" support technicians have no clue what an IP (4 or 6) address looks like. This is the same company that has to pay a daily fine for "excessive profits".

I've not seen that alert so far, but I'll keep an eye out for it, thanks for pointing it out.

So far the list seems to have stabilized a LOT, and I'm thinking about clearing up the list (# and no # rules). What gets on my nerves is that although the list originates from the snort topic (which is almost a year old!) most of the rules are still there.

Cino

I'm wondering since i'm dual stack IPv4/IPv6(DHCP for both on the WAN), that is why i'm seeing this alert.. My ISP(TWC) doesn't officially support IPv6 but its enabled and working pretty well. Better then IPv4 traffic at times…

A Former User

I also dual stack, but have not seen that alert yet. Same here, I get better speeds over the tunnel, than the native IPv4.

On a side note, I'm updating the list. Mostly cleaning it up, moving the DO NOT USE! categories to the top so it's easier to find them, removing the # and no # rules…a line added here, a line taken away there...deleting some rules, updating the counts. I said I'm rarely mistaken (mathematically proven by the world's top universities, and the possibility of me being mistaken comes out to 0.000000000001%) but ffs guys. 5 rules on top of a count that says 4 rules are disabled should be noticed by someone by now :D.

EDIT: Finished cleaning up. The list is now starting fresh for suricata (old rules coming over from snort that did not FP within a reasonable timeframe were removed), cleaned up and ready to go. Enjoy!

A Former User

whitelists were renamed to passlists not long ago. That is a definite bug in the suricata package. please post a link pointing to that post, on the most recent version(ed?) topic on suricata or re-post the same thing there. I'm not the suricata package maintainer, bmeeks is ;D

bmeeks got a tunnel working a few days back, so IPv6 support for both packages should improve.

Cino

@jflsakfja:

I also dual stack, but have not seen that alert yet. Same here, I get better speeds over the tunnel, than the native IPv4.

On a side note, I'm updating the list. Mostly cleaning it up, moving the DO NOT USE! categories to the top so it's easier to find them, removing the # and no # rules…a line added here, a line taken away there...deleting some rules, updating the counts. I said I'm rarely mistaken (mathematically proven by the world's top universities, and the possibility of me being mistaken comes out to 0.000000000001%) but ffs guys. 5 rules on top of a count that says 4 rules are disabled should be noticed by someone by now :D.

EDIT: Finished cleaning up. The list is now starting fresh for suricata (old rules coming over from snort that did not FP within a reasonable timeframe were removed), cleaned up and ready to go. Enjoy!

You're dual stack and using a Tunnel? When I mean dual stack; I mean receiving IPv4 IPv6 addresses on the same WAN interface.. The alerts are on that interface… Once I get my stuff fine tune, i'm going to copy it over to LAN and see if there is a difference. I should had started with my LAN interface first but oh well...

I noticed the counts were wrong, figured your eyes were getting crossed or something from the long post :o Thanks again for a great thread!

@avink use this topic https://forum.pfsense.org/index.php?topic=73906.msg428032#msg428032 to report the issue

A Former User

Statistics on suricata's system usage

While idling, the system is using about 2% CPU and 23% RAM (4GB,32bit so not all available).

Under normal load (not extreme "ZOMG! multiple 10Gbps duplex laser links into space!") the CPU occasionally spikes to <10% for a brief moment, and the RAM stays about the same.

While reloading rules (when hitting save on the rules page or disabling a rule from the alerts tab) the CPU spikes to 50% with it occasionally going higher (seen up to 92% for a few seconds). CPU is a dual core, so 50% means 100% of one core. RAM usage drops at the start of the reload, and continues to slowly climb to ~23% while the rules are reloaded into RAM.

System has been set up according to this topic, with about 50 custom rules. It's the network gateway to a small datacenter, which also provides Internet connectivity for hosts outside the datacenter.

Matcher is AC, suricata interface is WAN.

@Cino : Having a tunnel means somehow getting the IPv6 addresses onto pfsense. Unless you are doing some pretty complicated stuff (routing packets to Mars, then Venus, then the end of the galaxy :D) all other interfaces that need to use those addresses (except the tunnel's "terminating" WAN interface) ARE dual stacked ;). Clients on the LAN side can connect either using an IPv4 address or completely stop using IPv4 and use an IPv6 address. That is the definition of dual stacked :D

dotOne

@Cino, Thanks. I removed my message from this thread.