Taming the beasts… aka suricata blueprint

Respect for the little atoms that could ;D. The newer 4 core models (technically a celeron, or is it the other way around?) are interesting, thinking about getting a couple for testing. A fully loaded psfsense system based on those should be close to 30W (cpu+cards+hdd).

Cino

I want to say my D510 box is running around 20-25watts. Have to find my build notes to confirm. off topic but I have a few interfaces, traffic shaping, snort, pfblocker, suricta, squid, ntop, vnstat… she runs good... Need to change her it 64bit so I can use all of the 4gb of memory but I think i'll wait for 2.2 to be release then do a fresh install and rebuild the config for fun :o

@Cino:

I want to say my D510 box is running around 20-25watts. Have to find my build notes to confirm. off topic but I have a few interfaces, traffic shaping, snort, pfblocker, suricta, squid, ntop, vnstat… she runs good... Need to change her it 64bit so I can use all of the 4gb of memory but I think i'll wait for 2.2 to be release then do a fresh install and rebuild the config for fun :o

Ah, the Debian bug. Nothing happens to it, to the point where you want to upgrade to testing just for the hope of something breaking? :p

Atoms are perfect for personal use, IMHO.

bmeeks

@jflsakfja:

@dmitripr:

Thanks, jflsakfja. That rule syntax worked.

I see that the latest Suricata release is 2.0.3. Any ideas when pfsense package will be upgraded to that version? Is there a way to manually update?

I have noticed that as of late I'm being pinged from multiple hosts for several minutes at a time. Feels like an attack from hijacked hosts. I put a rule to block those, and I was able to block almost 500 hosts in 24 hours via suricata.

Thanks again!

I think 2.x is coming with the next release of the suricata package. How long that takes, dunno though.

The pings are regular internet noise. Don't worry about it. As long as you are not responding back, then you are still flying under the radar.

I am hopefully just a week or so away from posting the Pull Request for 2.0.x Suricata. I ran into a small snag compiling the new package for 2.2 of pfSense, but I think I have a solution for that now. I have been developing/testing with 2.0.2, but if it's not too big of a change I will bump it to 2.0.3 before I post the Pull Request.

Bill

Mr. Jingles

@bmeeks:

I am hopefully just a week or so away from posting the Pull Request for 2.0.x Suricata. I ran into a small snag compiling the new package for 2.2 of pfSense, but I think I have a solution for that now. I have been developing/testing with 2.0.2, but if it's not too big of a change I will bump it to 2.0.3 before I post the Pull Request.

Bill

Hi Bill ;D

Would that also include the suggestion from one of the biggest noobs on this board to have an easy way to multi-enable/disable the rules per category (the same check boxes you see in the left side of the firewall rules screens)?

That would be quite lovely, so to speak :P

Mr. Jingles

A most stupid question, for which I am by now famous: wat is the OpenVPN-interface; WAN or LAN?

Virtual Private Network would suggest LAN, but on the other hand: it is connected to the WAN ???

dmitripr

@jflsakfja:

It would be interesting to see more details about your setup. Did you disable the rules I recommended in this topic? Even the amazon one (yes that single rule does matter)? How much RAM was used? Nice to see that a dual core atom @ 1.86Ghz can (nearly) max out 100Mbps. I'm sure with a bit of tuning it could get there, unless you have already removed suricata and installed snort.

Don't worry about the VRT rules.

My setup is pretty simple, bough off the newegg:
** OEM Production 2550L2D-MxPC Intel NM10 Black Mini / Booksize Barebone System - OEM (http://www.newegg.com/Product/Product.aspx?Item=N82E16856205007)
** 4GB of RAM
** 32GB SSD
** Latest version of pfsense
** 1 LAN+ 1 WAN + IPsec + OpenVPN
** Bind, snort/suricata (not at the same time), pfblocker

It has dual Broadcom nics, which is not too bad. If I disable the snort/suricata IPS, then @108 mbps down the CPU load is only 33% or so. So, theoretically this thing should be able to push 250 mbps easily. Not too shabby.

On the suricata setup, I followed your instructions for the rules. So, I did turn off the ones you mentioned in the posts here. However, I did have dshield and DROP categories enabled – apparently pfblocker doesn't have all the latest IPs for those, and some get though to IPS and are blocked there. For comparison sake, I have the same rules enabled for snort (just keeping it apples-to-apples).

I'm a believer in suricata, based on what I read, but probably not quite prime-time ready (at least in my setup, based on my limited testing). I have not uninstalled it, just disabled it at this point. I'll try again once 2.x.x comes out, hopefully soon.

Double K

@BBcan177:

…
The High Level function of the script:

Download Individual List
Extract IPs
Save copy to /orig Folder
Check for Ranges that have 255 IPs and mark a single /24 Range
Process /24 (Which looks for repeat Offenders in a /24 Range) (max variable) Individual Blocklist Only.
Duplication Check

Once all of the Downloads are completed that were scheduled to run:

The Following is performed Globally on ALL Lists, except for the ones that were marked as "p24=no" on the Collect Line.

p-Deduplication - Looks for Repeat Offenders that are over the pmax variable regardless of Country Code.

d-DeDuplication - Looks for Repeat Offenders that are over the dmax variable but uses the Country Code Whitelist function.

If the Sanity Checks passes, it will create the TIER (Group) lists and perform the "pfctl" commands to update the pfSense Alias Tables.

If you decide to remove a list, you need to add "remove" after the collect line. When the script runs at its next scheduled run, it will remove the list from the database properly. Don't try to do this manually.

If you follow the High level steps, when you use the p24 process in d-deduplicaton, it will look for a repeat range of malicious IPs and find all of the Blocklists that have this IP listed.

The FIRST blocklists get a single x.x.x.0/24 Block and all of the other Lists that have the range are deleted.

So if a List is removed, and it happens to be a list that had the p24 process and was the first list processed as above, then you have no Blocklists for that range. This will correct itself on when the Lists are re-downloaded but that could be 1-4hrs depending on when the Lists are scheduled to run.

To get back into Sync, you can run this function:

[ [b]./pfiprep killdb ]

Which will wipe the Database (Settings are not touched) and it will resync the database.

Out of Curiosity, which Lists did you disable?

Another Function is to use the "IR_Match" Alias in the Floating Rules as a "Match" Rule. This will show you activity for the IP Ranges that passed the Country Code Whitelist process. Because its a "Match" rule, it will not block, but just log the activity.

Since I have been running the script, I have not found too many False Positives, but I always recommend not to disable a list but to create a "SAFE Alias" Rule that is defined above the "Block/Reject" Rules. And just add the IPs that you want to allow.

The Patch for diag_dns.php will also work when looking at the Snort/Suricata Alert Logs.

If you are running Snort/Suricata, when you click on the "!" ICON to Resolve an IP, you will find that most of the IPs are already listed in the BlockLists. You will also see over time that it will pickup an Alert for an IP but the Blocklists do not have the specific IP but there are several IPs within the same Range that are being Blocked.

Also in diag_dns.php, there are several IP Reputation Links that can help you determine the Reputation of any Blocked IP before you remove a list, or Add an IP to the SAFE Alias list.

Let me know if you need any clarification or any other help.

Hi BBcan177, thanks for all your help setting up the scripts. I managed to get everything setup, including the widgets & the DNS patch.

Could you please shed some more light on the max, dmax, and pmax variables? Not completely clear on the differences between them and how they operate.
For example, upon first run (using max=5, dmax=5, pmax=50), I had 7 unique addresses in the 104.28.7.x range in the IR_Match alias file. This also created a 104.28.7.0/24 entry as well. I tried accessing a site that was not one of the unique addresses (ie. not blocked by the lists), but it was blocked by the /24 range. When I changed max=10 and dmax=10, not only does the /24 range not appear (good) but the 7 unique blocked addresses don't appear either (bad).
Thus, just need a better understanding of how max, dmax, and pmax work, and what happens when you change the values.

BBcan177

@Double:

Hi BBcan177, thanks for all your help setting up the scripts. I managed to get everything setup, including the widgets & the DNS patch.

Good job!

Could you please shed some more light on the max, dmax, and pmax variables?

Using a "max" variable, if it finds over the Max variable it will perform a Maxmind Geoip Database lookup and will process a /24 block for configured Foreign Countries on an individual Blocklist Basis.
Using a "dmax" variable if it finds over the dmax variable it will perform a Maxmind Geoip Database lookup and will process a /24 block for configured Foreign Countries at the end of the download process on all of the Blocklists together.
Using a "pmax" variable, if it finds over the dmax variable it will process a /24 Block excluding Country Code whitelist at the end of the download process on all of the Blocklists together.

I had 7 unique addresses in the 104.28.7.x range in the IR_Match alias file. This also created a 104.28.7.0/24 entry as well. I tried accessing a site that was not one of the unique addresses (ie. not blocked by the lists), but it was blocked by the /24 range.

You are referencing "Match" aliases here.

Here is a snipet from the pfiprep script-


# country Code p24 Process (pass/match)
ccwhite=match
# Define what to do with IP Ranges found that are in the
# Country Code p24 Process (block/match)
ccblack=block
# For pfSense, the "Match" IPs can be "Monitored" with
# "Floating Rules" which can log packets from these IP Ranges,
# but still allow the Blocking of the Individual IPs found in the
# same /24 Range.

So the script will Block a whole /24 range depending if you select ccblack=block.
ccwhite=match will put the IP ranges that are in the Safe Country list into a match alias.

So the match file has all of the IPs that are being blocked with a "!" at the start of the IP to tell pfSense not to match the "!" excluded IPs, and match on anything else in the /24 range.

I would suggest leaving the "match" alone until you get everything else working. Change the ccwhite=match to ccwhite=pass

So at a high level, the max,dmax and pmax variables look at how many IPs are repeat offenders in all of the blocklists. And then depending on these settings block/match as required.

BBcan177

@dmitripr:

However, I did have dshield and DROP categories enabled – apparently pfblocker doesn't have all the latest IPs for those, and some get though to IPS and are blocked there.

I don't believe that this is correct.

In pfBlocker or in my pfiprep script, you can use the following dShield URL:

https://feeds.dshield.org/block.txt

and for Spamhaus:

http://www.spamhaus.org/drop/drop.txt
http://www.spamhaus.org/drop/edrop.txt

You don't need to have them enabled in Snort/Suricata if its in pfBlocker/pfIPrep

You are probably seeing Snort/Suricata Alerting on these alerts because Snort/Suricata is not a true-inline IPS. These packages are inspecting a "copy" of each packet, so even thou the pf filter blocked the IP, Snort/Suricata will still alert because it is seeing a 'copy' of the packets.

I posted here :

https://forum.pfsense.org/index.php?topic=78062.msg432804#msg432804

with what Rules can be disabled as these can be implemented in pfBlocker/pfIPrep.

dmitripr

@BBcan177:

@dmitripr:

However, I did have dshield and DROP categories enabled – apparently pfblocker doesn't have all the latest IPs for those, and some get though to IPS and are blocked there.

I don't believe that this is correct.

In pfBlocker or in my pfiprep script, you can use the following dShield URL:

https://feeds.dshield.org/block.txt

and for Spamhaus:

http://www.spamhaus.org/drop/drop.txt
http://www.spamhaus.org/drop/edrop.txt

You don't need to have them enabled in Snort/Suricata if its in pfBlocker/pfIPrep

You are probably seeing Snort/Suricata Alerting on these alerts because Snort/Suricata is not a true-inline IPS. These packages are inspecting a "copy" of each packet, so even thou the pf filter blocked the IP, Snort/Suricata will still alert because it is seeing a 'copy' of the packets.

I posted here :

https://forum.pfsense.org/index.php?topic=78062.msg432804#msg432804

with what Rules can be disabled as these can be implemented in pfBlocker/pfIPrep.

Thanks BB. I didn't have those lists added. I will do so and disable the categories on snort/suricata.

BBcan177

@dmitripr:

Thanks BB. I didn't have those lists added. I will do so and disable the categories on snort/suricata.

Anytime.

Take a look at my script. It has a lot of Threat Sources. Not all of them can be used in pfBlocker as it can't recognize the different formats like my script can.

wetspark

Thanks to those that have spent their time to make document so much and distribute solid advice.
Its been a long time since I administered or needed to administer a firewall and was about to setup SNORT and such to build a better kid trap for my kids to have access to the internet. The efforts here convinced me to use Suricata and I'm happy I did. While a lot of the initial setup is redundant to me, it does cover the general theme for my multii-WAN-LAN FW.

Thanks for all the efforts!!

@Hollander:

A most stupid question, for which I am by now famous: wat is the OpenVPN-interface; WAN or LAN?

Virtual Private Network would suggest LAN, but on the other hand: it is connected to the WAN ???

Think of it like a dedicated ethernet line connecting two adjacent houses, without the dedicated ethernet line. That's why it's called a "private" network, because the tunnel is really a direct connection between two points, and (supposedly) it actively tries to be secure at it, for example by tearing the tunnel down and establishing a new one when attackers interfere with it. Like digging up the cable and moving it a couple feet over, when detecting that someone is tampering with it.

There are a couple of ways to set up openvpn:

Terminating the tunnel on an internal (separate) interface, then use rules to direct traffic where it's supposed to go.
or
An even safer way is terminating the tunnel on a separate host connected to a separate LAN-type interface. A raspberrypi/cubox-i/other-cheap-ARM-thingy is perfect for this, unless trying to route loads and loads of bandwidth through it. If it's just for remotely administering a pfsense (I'll save my don't do it speech ;D) then go for it. Don't forget the interface rules.

@dmitripr:

On the suricata setup, I followed your instructions for the rules. So, I did turn off the ones you mentioned in the posts here. However, I did have dshield and DROP categories enabled – apparently pfblocker doesn't have all the latest IPs for those, and some get though to IPS and are blocked there. For comparison sake, I have the same rules enabled for snort (just keeping it apples-to-apples).

I'm a believer in suricata, based on what I read, but probably not quite prime-time ready (at least in my setup, based on my limited testing). I have not uninstalled it, just disabled it at this point. I'll try again once 2.x.x comes out, hopefully soon.

Dshield/drop is a lot of work for an IDS. Try enabling suricata without those categories and test again, if it's not too much trouble.

You've demonstrated the reason this topic was created. Using the IDS part of a gateway is using 3 times the power the firewall part is using, which is exactly the reason I started this and the snort topics.

@all:
WRT the low power, a particular asrock mobo caught my eye: http://www.asrock.com/mb/Intel/Q1900M/. It has space for 6 intels (2x1 port + 1x4 ports) + extra onboard nic. 2 uplinks, 2 downlinks, 2 to play with, 1 spare. It's all you want in a small cute package ;D. Anyone out there care to try it/has tried it and want to share experiences? I'm sure for a top-of-rack firewall it should be perfect.

bmeeks

@Hollander:

@bmeeks:

I am hopefully just a week or so away from posting the Pull Request for 2.0.x Suricata. I ran into a small snag compiling the new package for 2.2 of pfSense, but I think I have a solution for that now. I have been developing/testing with 2.0.2, but if it's not too big of a change I will bump it to 2.0.3 before I post the Pull Request.

Bill

Hi Bill ;D

Would that also include the suggestion from one of the biggest noobs on this board to have an easy way to multi-enable/disable the rules per category (the same check boxes you see in the left side of the firewall rules screens)?

That would be quite lovely, so to speak :P

Yes, but that functionality is already there. On the RULES tab in both Snort and Suricata are icons for "Enable All Rules in this Category" and "Disable All Rules in this Category". There are "X" and "+" icons on the right side of the upper middle of the page. Hover your mouse over them to see tooltips pop up with descriptions of what they do. The "category" refers to the rule set currently selected in the drop-down combo-box on that tab.

Bill

Mr. Jingles

@bmeeks:

@Hollander:

@bmeeks:

I am hopefully just a week or so away from posting the Pull Request for 2.0.x Suricata. I ran into a small snag compiling the new package for 2.2 of pfSense, but I think I have a solution for that now. I have been developing/testing with 2.0.2, but if it's not too big of a change I will bump it to 2.0.3 before I post the Pull Request.

Bill

Hi Bill ;D

Would that also include the suggestion from one of the biggest noobs on this board to have an easy way to multi-enable/disable the rules per category (the same check boxes you see in the left side of the firewall rules screens)?

That would be quite lovely, so to speak :P

Yes, but that functionality is already there. On the RULES tab in both Snort and Suricata are icons for "Enable All Rules in this Category" and "Disable All Rules in this Category". There are "X" and "+" icons on the right side of the upper middle of the page. Hover your mouse over them to see tooltips pop up with descriptions of what they do. The "category" refers to the rule set currently selected in the drop-down combo-box on that tab.

Bill

Hi Bill ;D

There might be a misunderstanding, probably due to English not being my native language

(As you may know by now education has been completely abandoned in the netherlands - we barely even know where the US, the UK or Canada is on the map. That is, however, not a problem so informs us our government in the weekly propaganda: knowledge can be found on google ('google' is a new language in dutch schools: english was dropped for it), it is 'what you do with it' that counts, not pure, factual, simple knowledge. Such as english words. That all is obsolete in the 'knowledge society' that the netherlands is. I heard it seamingly is the same in the rest of the west).

So I'll use the simple techniques our ancestors used ( ;D ): a picture. The old people once said that a picture can say more than a thousand words.

Because if I understand you correctly the current system only allows to enable/disable all rules in a category. What JFL writes is to selectively disable/enable for example 70 rules in 1 category. And that means the current Way Of Working is quite inefficient (click one rule disable/enable, wait for pfSense to respond which can take up to 1 minute, move on to the next rule, and then 70 minutes have passed for only 1 category. And there are many categories. It can easily take up to a full day to customize 1 interface).

pfsense_snort_checkboxes.jpg_thumb

@Hollander, it's not just in the Netherlands, it's the world trend now. They feel the need to dumb down the students so their authority is not threatened in the future when they grow up to be the tax-paying zombies they are meant to be ;D

Agree with the checkboxes approach. It could save a lot of time to enable all, then disable the 20 rules that need to be disabled per interface, and click the disable selected rules button.

Imagine, the package is that good that we are picking on the details ;D

BBcan177

@bmeeks:

Providing rule management functionality equivalent to PulledPork with regex matches for enabling or disabling rules. In other words, the ability to read and interpret enablesid.conf, modifysid.conf and disablesid.conf files. You would be able to edit these offline and upload to the firewall, or edit in place using the same interface as I implemented for the IP REP lists management tab in Snort.

I think the best approach is to wait for the changes that Bill has posted above.

We should only be clicking these ICONS for some exceptions. The majority of this work being done with these configuration files (disablesid, enablesid and modifysid) will greatly improve our use of time and ultimately have less potential for errors.

We just need to be patient to get it coded.

bmeeks

@BBcan177:

@bmeeks:

Providing rule management functionality equivalent to PulledPork with regex matches for enabling or disabling rules. In other words, the ability to read and interpret enablesid.conf, modifysid.conf and disablesid.conf files. You would be able to edit these offline and upload to the firewall, or edit in place using the same interface as I implemented for the IP REP lists management tab in Snort.

I think the best approach is to wait for the changes that Bill has posted above.

We should only be clicking these ICONS for some exceptions. The majority of this work being done with these configuration files (disablesid, enablesid and modifysid) will greatly improve our use of time and ultimately have less potential for errors.

We just need to be patient to get it coded.

I favor the enablesid, disablesid and modifysid approach, but it will take some time to get this coded. It and filtering for the ALERTS tab are my next tasks. I have the new 2.0.3 Suricata package pretty much ready to go. If I can get it coded in time, I might include the filtering and enablesid stuff in it, but I can't promise. It's more important in my view to get "current" with the Suricata binary first.

Bill

bmeeks

@Hollander:

Hi Bill ;D

There might be a misunderstanding, probably due to English not being my native language

(As you may know by now education has been completely abandoned in the netherlands - we barely even know where the US, the UK or Canada is on the map. That is, however, not a problem so informs us our government in the weekly propaganda: knowledge can be found on google ('google' is a new language in dutch schools: english was dropped for it), it is 'what you do with it' that counts, not pure, factual, simple knowledge. Such as english words. That all is obsolete in the 'knowledge society' that the netherlands is. I heard it seamingly is the same in the rest of the west).

So I'll use the simple techniques our ancestors used ( ;D ): a picture. The old people once said that a picture can say more than a thousand words.

Because if I understand you correctly the current system only allows to enable/disable all rules in a category. What JFL writes is to selectively disable/enable for example 70 rules in 1 category. And that means the current Way Of Working is quite inefficient (click one rule disable/enable, wait for pfSense to respond which can take up to 1 minute, move on to the next rule, and then 70 minutes have passed for only 1 category. And there are many categories. It can easily take up to a full day to customize 1 interface).

Sorry, I did misunderstand. I remember us having this conversation a while back. I had forgotten until I saw your picture to remind me. See my reply to BBcan177's post where I favor using the upcoming enablesid, disablesid and modifysid functionality. That is ultimately less work and will make it easy to "share" rule setups with other users. Just upload the file and apply it to an interface.

In the interim, one sort of workaround would be to "enable all" the rules in the category using the current icon, then selectively click the ones you want to disable (or vice-versa). In the upcoming Snort and Suricata updates (the Snort one is waiting for approval right now), the response should be a bit faster when clicking as it does not save everything and regenerate rules until you click APPLY.

Bill