Taming the beasts… aka suricata blueprint

BBcan177

Trying to support, for example, three different OS environments is a bit much for a one-man package maintainer. For instance, the installation paths differ between pfSense 2.0 and 2.1 and 2.2; plus the variety of system calls differs. For example, support for IPv6 varies between the versions. So this makes the code in a package sometimes quite complex as it tries to deal with all the subtle differences in pfSense versions. The more complex a package's code becomes, the higher the potential for bugs.

I agree, keeping up with the updates is important, especially if we have the support of the Maintainer to fix issues that arise…. If it wasn't for Bill maintaining these packages, I am sure we would be more hesitant to upgrade. ;) :o

Double K

@BBcan177:

Could you please shed some more light on the max, dmax, and pmax variables?

Using a "max" variable, if it finds over the Max variable it will perform a Maxmind Geoip Database lookup and will process a /24 block for configured Foreign Countries on an individual Blocklist Basis.

Using a "dmax" variable if it finds over the dmax variable it will perform a Maxmind Geoip Database lookup and will process a /24 block for configured Foreign Countries at the end of the download process on all of the Blocklists together.

Using a "pmax" variable, if it finds over the dmax variable it will process a /24 Block excluding Country Code whitelist at the end of the download process on all of the Blocklists together.

I had 7 unique addresses in the 104.28.7.x range in the IR_Match alias file. This also created a 104.28.7.0/24 entry as well. I tried accessing a site that was not one of the unique addresses (ie. not blocked by the lists), but it was blocked by the /24 range.

You are referencing "Match" aliases here.

Here is a snipet from the pfiprep script-
# country Code p24 Process (pass/match)
ccwhite=match
# Define what to do with IP Ranges found that are in the
# Country Code p24 Process (block/match)
ccblack=block
# For pfSense, the "Match" IPs can be "Monitored" with
# "Floating Rules" which can log packets from these IP Ranges,
# but still allow the Blocking of the Individual IPs found in the
# same /24 Range.
So the script will Block a whole /24 range depending if you select ccblack=block.
ccwhite=match will put the IP ranges that are in the Safe Country list into a match alias.

So the match file has all of the IPs that are being blocked with a "!" at the start of the IP to tell pfSense not to match the "!" excluded IPs, and match on anything else in the /24 range.

I would suggest leaving the "match" alone until you get everything else working. Change the ccwhite=match to ccwhite=pass

So at a high level, the max,dmax and pmax variables look at how many IPs are repeat offenders in all of the blocklists. And then depending on these settings block/match as required.

Thanks BBCan177, I've read & re-read this about 15 times, drawn it out on paper, changed the floating rule for IR_Match to "Match" instead of Block, and still can't figure it out.
Unfortunately, all I seem to be doing now is adding individual IP addresses to the Floating rules to allow them to pass through these Blocks - because legitimate IP addresses such as bing.com's IP address are being blocked by these Blocking Lists.

BBcan177

@Double:

Thanks BBCan177, I've read & re-read this about 15 times, drawn it out on paper, changed the floating rule for IR_Match to "Match" instead of Block, and still can't figure it out.
Unfortunately, all I seem to be doing now is adding individual IP addresses to the Floating rules to allow them to pass through these Blocks - because legitimate IP addresses such as bing.com's IP address are being blocked by these Blocking Lists.

Take a look at posts in this thread weeks ago where the Blocklist Source 'MTA' is blocking some legit sites.

Take a look at the pfiprep script (near the bottom) and read the 'INFO' comments above the Threat Source URLS.

You can disable a Source by commenting it out with "#" at the start of the line.

Unfortunately MTA posts all IPs that were involved in malicious activity. So it you want to keep this list, you need to create a "SAFE Alias" Rule above the "Block/Reject" rules to allow any false positives thru.

If you remove lists, I suggest running

./pfiprep killdb or ./pfiprep killdb skip

dskip option will re-use your existing downloads.

You don't need to play with 'Match', set that to 'pass' for now.

A Former User

@BBcan177:

I agree, keeping up with the updates is important, especially if we have the support of the Maintainer to fix issues that arise…. If it wasn't for Bill maintaining these packages, I am sure we would be more hesitant to upgrade. ;) :o

+1

From my point of view I want the software I'm using as bug free as possible. If that means keeping up with upstream, then so be it. I'll adapt to the software, instead of the software adapting to me.

I can't remember the exact details (mentioned before I'm not the "keep a google tab open" type of person), but a couple years back support for 286 (or was it up to 386?) was removed from the linux kernel, taking with it a crapload of unneeded code. If memory serves right, the same goes for 32bit popular ZFS distros out there (freenas?). That's the spirit of open source IMHO. The developers try their best to make a decent software for all to use, but within a few reasonable limits. But support for a 286? Why are you still using a 286?

Bought my laptop (a 17" toshiba which I love) in 2009 for €450. It's regular price was €800, but had some coupons (yes I'm that kind of guy) that dropped it down to €450. I don't actually have the need anymore for a 17" laptop. It was used for 3D designing in uni.

During its lifetime, this laptop has seen windows, 3d designing, linux, running linux on top of windows, running windows on top of linux, flashing firmwares to various ARM devices, it only shuts down when I need to be out of town and take it with me, otherwise it's a 24/7-on laptop. It's a dual core 2GHz Turion for crying out loud and it has served my well.

Was looking at a few prices the other day for laptops. A 15.6" freedos (you know it will get formatted right out of the box and debian installed on it) costs about €270-€340, depending who you ask (brand). It's a dual core pentium, with integrated intel graphics. An i3 15.6" costs somewhere around €440 (and that's just the first result that I found, windows 8 (this time formatted before even getting out of the packaging, since technically microsoft must refund the software license if you don't intent on using it, just ask the shop to format it for you before buying it.)). What I will do when my laptop is no longer supported by debian is run debian as long as I can (within the stable supported lifecycle) then air-gap it when support is officially dropped and turn it into an offline laptop for as long as it still works. For the online laptop (the daily use one) I'll just get a newer lower priced laptop, that can do everything this little (well, if you consider 17" little) guy did and more.

I don't expect the developers supporting my hardware forever, and I do understand that they can't actually support it forever. The same goes for a pfsense box. If that box can no longer run a newer version of pfsense, maybe spend the money to get a supported card (if that's the case) or just buy a newer box. As shown in this thread, you don't have to spend a fortune to get the best firewall/IDS. 5 years down the line, if the box still runs the latest version then great. If not, replace it. The electricity + time spent troubleshooting the old box + the speed increase of the newer box, will make up for the newer box.

And for the record, I'm still running pfsense (latest)+suricata on a 3.4 p4 (the "ZOMGITHASHYPERTHREADINGITSLIKEDUALCORES" northwood one). Don't hate it, embrace it. It's old, it's a power hog, it's loud, but it still works. The upstream connection to it is only 10mbps (a dedicated ethernet line that costs 5 arms, 8 legs, and 3 kidneys). If tomorrow support for it was dropped, then it would get upgraded straight to a little celeron that could, without thinking twice about it. I've used atoms, celerons, core 2s, i3s, e3s, and i7s. As long as you don't expect extraordinary things from them, each does wonders in its particular bandwidth category. Do the developers (and the rest of us) a favor and upgrade to a box capable of running the latest version of the software that's protecting your neck.

Don't be that corporate guy that never upgrades installed software in the fear of something breaking and getting fired. If one of my employees came up to me and said "oops, I just broke the email server by upgrading it to a newer version because of a vulnerability", he wouldn't get fired, but instead congratulated on taking the steps to ensure the system was as secure as possible. Who will get fired though is the backups guy, if backups fail to work ;D. If the backups are ok, the first guy gets a box to play with, to make sure the newer version is supported, and works upstream to get the newer version supporting our old system out as fast as they can. That's nothing compared to a compromise caused by an unpatched vulnerability.

Disclaimer
By you I'm NOT talking about anyone in particular, it's the general you.

SCSI_2K

jflsakfja, your write-ups are always great to read. Most topic on this thread is out of my head but I'm learning. Thank to you and BBCan117 for the great work!

I've been reading and re-reading this thread for a week now and almost done setting it up. The only issue that I'm getting is that most of my list from pfiprep only contains 1.1.1.1 IP address. Under the PfIP Reputation widget, I get a lot of fail errors with only IR_PRI1 list getting downloaded.

Running pfiprep from shell give me this kind of errors: ./pfiprep: /usr/local/bin/grepcidr: not found

Example:

looking up rules.emergingthreats.net
connecting to rules.emergingthreats.net:443
SSL connection established using DHE-RSA-AES256-SHA
Certificate subject: /serialNumber=cmxRbct3onRzhDapr8DEMQb9Au33Dqrp/OU=GT44212215/OU=See www.rapidssl.com/resources/cps 14/OU=Domain Control Validated - RapidSSL(R)/CN=*.emergingthreats.net
Certificate issuer: /C=US/O=GeoTrust, Inc./CN=RapidSSL CA
requesting https://rules.emergingthreats.net/open/suricata/rules/tor.rules
local size / mtime: 303563 / 1407962999
remote size / mtime: 303563 / 1407962999
/home/badips/download/tor.rules 100% of 296 kB 725 kBps

Download file count [ 699 ] Outfile count [ 3418 ]

Process /24 Stats
–-------------------------
Found [ 11 ] IP range(s) over the threshold of [ 5 ] on the CC Blacklist
Found [ 3 ] IP range(s) over the threshold of [ 5 ] on the CC Whitelist

Found [ 60 ] CC Blacklisted IP Address(es) are being set to [ block ]
Found [ Skipped ] CC Whitelisted IP Address(es) are being set to [ match ]

Removed the following IP Ranges
81.150.197.|178.32.181.|62.220.148.|46.105.162.|95.130.15.|81.89.0.|185.61.148.|94.198.98.|
–-------------------------
Post /24 Count [ 3366 ]
./pfiprep: /usr/local/bin/grepcidr: not found

Post Duplication count
–-------------------------
Masterfile Count [ 0 ]
Outfile Count [ 0 ] [ Passed ]
–--------------------------------------------------------------

Thanks for this thread, I switch from snort to suricata.

BBcan177

@SCSI_2K:

Running pfiprep from shell give me this kind of errors: ./pfiprep: /usr/local/bin/grepcidr: not found

You need to install this application called Grepcidr first.

Since FreeBSD has moved the pkg files to archive, you need to make a few changes:

[amd64]

setenv PACKAGESITE http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/ports/amd64/packages-8.3-release/net-mgmt/

or

[i386]

setenv PACKAGESITE http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/ports/i386/packages-8.3-release/net-mgmt/

then:

pkg_add -r grepcidr-1.3_1.tbz

Once this is installed. Try to see if it loads properly:

/usr/local/bin/grepcidr -V

grepcidr 1.3
Copyright (C) 2004 - 2013 Jem E. Berkes jem@berkes.caThen:

./pfiprep killdb/jem@berkes.ca

SCSI_2K

Thanks for the very fast response! It is working now. ;D

I must have mis-configured Snort before since it always uses 85% of my 3GB RAM. If I turn it off and use suricata, my mem usage drops to just 14% with all the rules enabled from this thread. WOW!

Is pfblocker still needed if I'm already using suricata and pfiprep?

BBcan177

What memory setting did you have for Snort. AC-BNFA-NQ seems to be the best for memory usage and performance.

You can disable pfBlocker. You don't need to uninstall it. Just don't have them both enabled at the same time.

Mr. Jingles

@jflsakfja:

@Hollander: smack Snap out of it! They got you too?!?! smack. All hope is lost :(

DISCLAIMER
This is NOT a direct attack on anyone around here, nor elsewhere. I'm merely expressing my opinion. If you do not agree with me, discuss in a calm, well-mannered way. Swearing is acceptable, as long as it's directed at the ET guys*, cisco* (for charging a night with your wife, your first born's right kidney, and your left kidney for a support contract), or the developers of either unity*,gnome 3*, or internet explorer* (all versions ok). Windows 8* accepted/encouraged.

;D ;D ;D ;D ;D

SCSI_2K

@BBcan177:

What memory setting did you have for Snort. AC-BNFA-NQ seems to be the best for memory usage and performance.

You can disable pfBlocker. You don't need to uninstall it. Just don't have them both enabled at the same time.

I had Snort set to AC-STD. Thanks again for answering my question on pfBlocker.

wetspark

Well, a week later and my Pfsense/Suricata/block-list protector has become boring. :)

In my years of IT, boring is good. It does its job, does it well, and stays out of your way.

I'm really interested in the update to 2.x Suricata and will be all over that. Mostly so I can easily plug it into logstash/kibana to see pretty graphs and such.

A Former User

Does the guide provide a stable starting point?

What sort of problems did you encounter while setting up suricata?

What could be improved in the guide?

wetspark

@jflsakfja:

Does the guide provide a stable starting point?

What sort of problems did you encounter while setting up suricata?

What could be improved in the guide?

I'm not a good candidate for a startup review as I know too much and that makes me dangerous. :)
I leveraged the information provided here and the great tools that have been developed for downloading/updating block lists.

Since I started early in the thread's life I ran into a couple of questions that were later answered/clarified in the thread. So those are no problem for anyone that reads the thread I think. I have multiple WAN circuits and multiple LAN circuits and I basically applied a lot of this to all of them but the "kid LAN" to "general purpose ISP WAN" is configured very closely with these recommendations.

I started writing the golden rules and let me say, that's an awesome thing right there. I truly love elegant designs and strive myself to create such designs. Those briefings are really elegant once implemented and that's the part that I still need to get around to doing.

In conclusion I'd say you guys have given some super costly consulting away. I appreciate the efforts!

A Former User

Good to hear the thread was so much help.

Thanks to all the guys that made it happen :)

BBcan177

@wetspark:

In conclusion I'd say you guys have given some super costly consulting away. I appreciate the efforts!

I think the general concensus is when people help you in this forum, that you pass that information down to those that need it… That's why this forum is so good!

:) :)

A Former User

Since wetspark's post, I've been meaning to ask all how things are going if you followed this guide.

@ALL Is everything working like it should? Does it get the job done? How stable was the resulting system?

If anyone has anything to add, please do so. The more discussion that goes into pfsense/suricata/lists the better for us all ;D

EDIT: In other words, was the guide successful in taming the beasts? :D

A Former User

And this is what comes out of the guide:

suricata_record.jpg_thumb

Mr. Jingles

@jflsakfja:

Since wetspark's post, I've been meaning to ask all how things are going if you followed this guide.

@ALL Is everything working like it should? Does it get the job done? How stable was the resulting system?

If anyone has anything to add, please do so. The more discussion that goes into pfsense/suricata/lists the better for us all ;D

EDIT: In other words, was the guide successful in taming the beasts? :D

I was away for a while, so sorry for not responding sooner :-*

It is very stable. Your guide about the firewall rules shows a lot of things being blocked albeit the internet experience doesn't suffer from it (e.g. websites can still be accessed, WIFE doesn't complain although I see numerous blocks from her LAN-IP out to the internets).

BB's lists also works very nice, although there are quite some false positives in it –- although BB can't be blamed for that, of course not.

I can not yet comment on the Suricata matters, as I currently have Snort:
1. I wanted to move over to Suricata but your rules instructions are massive, and the Suricate GUI doesn't allow me to fix the rule instructions relatively fast.
2. The great Bill anounced some modifications to the GUI, I don't know where that currently is at, because:
2A. I had to uninstall Suricata and some other packages since one of them was making my box crash - and I don't know which one it was nor did I have the time to find out;
2B. I was away, busy, playing 'Zorro' for some people out there ;D

So:
1. Firewall rules seem very useful;
2. BB's script is an artwork that performs very well too;
3. I am dying to dive into Suricata but I need to catch up to where it is at, avoiding the repeated crashed of some weeks ago.

In the end:
Great JFL;
Great BB;
Great Bill;
Great mysterious man who refuses me to buy him a coffee;
Great pfSense team;
Great many other helpful members;
Not so great few members ( ;D ;D ;D ;D ;D ).

wcrowder

@jflsakfja

Can that beast ever really be tamed? But your posts have went along way taming them for my meager home network. With kids and all my toys, having this king of control over that beast is assume. Thank you for your time putting this all to together.

foetus

I just found an issue and would like some input (Really need a vacation soon..).

So, in regards to the block-lists, I use floating rules instead of interface specific rules. (pfblocker)
All good and well, be it I use a couple of white-lists also (generated by a whois cron job, or just plain manual).

For example : I have a Google white-list to allow both gmail, google dns, the bunch. General floating rules for both directions.
BUT, if I apply the "Apply the action immediately on match." my NAT rules are completely ignored. So forget forwarding a match in the whitelist floating rule to an exchange server internally.

Causing the very nice result of not a single Gmail message getting delivered anymore.

Using the white-list in the specific interfaces will have no use, since the block lists floating rules will always take priority.
So, any other solution except for changing all floating rules to interface specifics? Be it manually or pfblocker.

*edit
to be specific. I just want to whitelist the whole list I create. Not start with specific port ranges.