Taming the beasts… aka suricata blueprint

Mr. Jingles

@Hollander:

@bmeeks:

I am hopefully just a week or so away from posting the Pull Request for 2.0.x Suricata. I ran into a small snag compiling the new package for 2.2 of pfSense, but I think I have a solution for that now. I have been developing/testing with 2.0.2, but if it's not too big of a change I will bump it to 2.0.3 before I post the Pull Request.

Bill

Hi Bill ;D

Would that also include the suggestion from one of the biggest noobs on this board to have an easy way to multi-enable/disable the rules per category (the same check boxes you see in the left side of the firewall rules screens)?

That would be quite lovely, so to speak :P

Yes, but that functionality is already there. On the RULES tab in both Snort and Suricata are icons for "Enable All Rules in this Category" and "Disable All Rules in this Category". There are "X" and "+" icons on the right side of the upper middle of the page. Hover your mouse over them to see tooltips pop up with descriptions of what they do. The "category" refers to the rule set currently selected in the drop-down combo-box on that tab.

Bill

Hi Bill ;D

There might be a misunderstanding, probably due to English not being my native language

(As you may know by now education has been completely abandoned in the netherlands - we barely even know where the US, the UK or Canada is on the map. That is, however, not a problem so informs us our government in the weekly propaganda: knowledge can be found on google ('google' is a new language in dutch schools: english was dropped for it), it is 'what you do with it' that counts, not pure, factual, simple knowledge. Such as english words. That all is obsolete in the 'knowledge society' that the netherlands is. I heard it seamingly is the same in the rest of the west).

So I'll use the simple techniques our ancestors used ( ;D ): a picture. The old people once said that a picture can say more than a thousand words.

Because if I understand you correctly the current system only allows to enable/disable all rules in a category. What JFL writes is to selectively disable/enable for example 70 rules in 1 category. And that means the current Way Of Working is quite inefficient (click one rule disable/enable, wait for pfSense to respond which can take up to 1 minute, move on to the next rule, and then 70 minutes have passed for only 1 category. And there are many categories. It can easily take up to a full day to customize 1 interface).

pfsense_snort_checkboxes.jpg_thumb

@Hollander, it's not just in the Netherlands, it's the world trend now. They feel the need to dumb down the students so their authority is not threatened in the future when they grow up to be the tax-paying zombies they are meant to be ;D

Agree with the checkboxes approach. It could save a lot of time to enable all, then disable the 20 rules that need to be disabled per interface, and click the disable selected rules button.

Imagine, the package is that good that we are picking on the details ;D

BBcan177

@bmeeks:

Providing rule management functionality equivalent to PulledPork with regex matches for enabling or disabling rules. In other words, the ability to read and interpret enablesid.conf, modifysid.conf and disablesid.conf files. You would be able to edit these offline and upload to the firewall, or edit in place using the same interface as I implemented for the IP REP lists management tab in Snort.

I think the best approach is to wait for the changes that Bill has posted above.

We should only be clicking these ICONS for some exceptions. The majority of this work being done with these configuration files (disablesid, enablesid and modifysid) will greatly improve our use of time and ultimately have less potential for errors.

We just need to be patient to get it coded.

bmeeks

@BBcan177:

@bmeeks:

Providing rule management functionality equivalent to PulledPork with regex matches for enabling or disabling rules. In other words, the ability to read and interpret enablesid.conf, modifysid.conf and disablesid.conf files. You would be able to edit these offline and upload to the firewall, or edit in place using the same interface as I implemented for the IP REP lists management tab in Snort.

I think the best approach is to wait for the changes that Bill has posted above.

We should only be clicking these ICONS for some exceptions. The majority of this work being done with these configuration files (disablesid, enablesid and modifysid) will greatly improve our use of time and ultimately have less potential for errors.

We just need to be patient to get it coded.

I favor the enablesid, disablesid and modifysid approach, but it will take some time to get this coded. It and filtering for the ALERTS tab are my next tasks. I have the new 2.0.3 Suricata package pretty much ready to go. If I can get it coded in time, I might include the filtering and enablesid stuff in it, but I can't promise. It's more important in my view to get "current" with the Suricata binary first.

Bill

bmeeks

@Hollander:

Hi Bill ;D

There might be a misunderstanding, probably due to English not being my native language

(As you may know by now education has been completely abandoned in the netherlands - we barely even know where the US, the UK or Canada is on the map. That is, however, not a problem so informs us our government in the weekly propaganda: knowledge can be found on google ('google' is a new language in dutch schools: english was dropped for it), it is 'what you do with it' that counts, not pure, factual, simple knowledge. Such as english words. That all is obsolete in the 'knowledge society' that the netherlands is. I heard it seamingly is the same in the rest of the west).

So I'll use the simple techniques our ancestors used ( ;D ): a picture. The old people once said that a picture can say more than a thousand words.

Because if I understand you correctly the current system only allows to enable/disable all rules in a category. What JFL writes is to selectively disable/enable for example 70 rules in 1 category. And that means the current Way Of Working is quite inefficient (click one rule disable/enable, wait for pfSense to respond which can take up to 1 minute, move on to the next rule, and then 70 minutes have passed for only 1 category. And there are many categories. It can easily take up to a full day to customize 1 interface).

Sorry, I did misunderstand. I remember us having this conversation a while back. I had forgotten until I saw your picture to remind me. See my reply to BBcan177's post where I favor using the upcoming enablesid, disablesid and modifysid functionality. That is ultimately less work and will make it easy to "share" rule setups with other users. Just upload the file and apply it to an interface.

In the interim, one sort of workaround would be to "enable all" the rules in the category using the current icon, then selectively click the ones you want to disable (or vice-versa). In the upcoming Snort and Suricata updates (the Snort one is waiting for approval right now), the response should be a bit faster when clicking as it does not save everything and regenerate rules until you click APPLY.

Bill

dmitripr

@jflsakfja:

Dshield/drop is a lot of work for an IDS. Try enabling suricata without those categories and test again, if it's not too much trouble.

You've demonstrated the reason this topic was created. Using the IDS part of a gateway is using 3 times the power the firewall part is using, which is exactly the reason I started this and the snort topics.

Alright, I ran some tests after disabling those categories – though during the peak hours (5-6pm), so, won't get the best throughput, but close. With snort my 2 consecutive results were: 99.17 then 101.95 mbps. Snort maxed out at about 65% CPU utilization. Disabled snort, enabled Suricata and retested: 95.39 then 101.32 mbps. Suricata maxed out at 95% (probably hit close to 100%) CPU utilization. So, it did get marginally better it seems. But I still feel that the performance is not on par with snort.

Maybe it's just my setup, or suricata + pfsense combination. I'm looking forward to 2.0.3 suricata release shortly (hopefully).

The way I see it: for most people with lower speed tiers (50/10 and below), this probably won't be noticeable at all. It just seems that once you get over 100 mbps, you start running into performance limitations of the CPU (in my case Atom) with suricata -- and snort too, for that matter, but it's fairing a little better so far.

Once 2.0.3 is out, I'll retest and let you know. Looking forward to it.

All this is making me wonder about all those 1 gbps lines poping up from Google and AT&T now, people raving about them. But will regular consumers be able to actually take advantage of the speed with an over-the-counter routers? Certainly not with OTC routers that run any sort of IPS/IDS (e.g. Small business Cisco). You really would need a Xeon type of system to push those sort of speeds, not to mention top of the line NICs. Just curious. (maybe a bit off topic here)

Cheers.

@dmitripr:

Alright, I ran some tests after disabling those categories – though during the peak hours (5-6pm), so, won't get the best throughput, but close. With snort my 2 consecutive results were: 99.17 then 101.95 mbps. Snort maxed out at about 65% CPU utilization. Disabled snort, enabled Suricata and retested: 95.39 then 101.32 mbps. Suricata maxed out at 95% (probably hit close to 100%) CPU utilization. So, it did get marginally better it seems. But I still feel that the performance is not on par with snort.

At least it's a step in the right direction. Thank you for taking the time to test and provide those results. There is nothing wrong with (nearly) maxing out the CPU, it's there for a reason afterall. As long as you don't literally max it out, then things start getting weird. If it stays at 95% and never sees 100%, then it should be good to go.
@dmitripr:

Maybe it's just my setup, or suricata + pfsense combination. I'm looking forward to 2.0.3 suricata release shortly (hopefully).

The way I see it: for most people with lower speed tiers (50/10 and below), this probably won't be noticeable at all. It just seems that once you get over 100 mbps, you start running into performance limitations of the CPU (in my case Atom) with suricata – and snort too, for that matter, but it's fairing a little better so far.

Once 2.0.3 is out, I'll retest and let you know. Looking forward to it.

Oh it will get better, trust me. I've never seen a (serious) program get worse with newer versions. Serious excludes any version of windows, gnome, and firefox with the idiotic chrome-wannabe interface. How the hell is moving the close search button from the left (right next to the search text box) all the way to the right (on the other side of the screen) a "user experience improvement". When will people realize that programmers must be constantly educated with the help of a baseball bat in order for them to be productive?
@dmitripr:

All this is making me wonder about all those 1 gbps lines poping up from Google and AT&T now, people raving about them. But will regular consumers be able to actually take advantage of the speed with an over-the-counter routers? Certainly not with OTC routers that run any sort of IPS/IDS (e.g. Small business Cisco). You really would need a Xeon type of system to push those sort of speeds, not to mention top of the line NICs. Just curious. (maybe a bit off topic here)
Cheers.

An atom will not be able to pull that off. You really need something in the i3 league (not e3 (xeon)) to get there. This is assuming full duplex on the connection (1gbps down/1gbps up fully maxed out). If it's just for downloading (1gbps down maxed out/let's say a quartergbps up) as long as it's a 1155 socket or newer it will work OK. Not saying get the cheapest 1150 you can get and expect it to get you there, but there are some powerful options, WRT their price. I'd (personally) take a $400 (total) option getting 900mbps than a $2000 option getting 1gbps. More so if it's for a home. Maybe this too will change with the newer pfsense (multithreaded pf) + newer quad core atoms. Oh excuse me, "celerons" ;) (see asrock above). If it still can't get you there, at least suricata offers the chance of using the GPU to help along.
Don't confuse specialized CPUs with general CPUs. It's the reason switches have <1Ghz CPUs and can still route at multi-dozen-gbps speeds. Not arguing that a linksys/asus/cisco/whatever router can get there, much less so when using any IDS.

There is no need for top of the line, intel nics are already the best there is ;D. For their price, I wouldn't complain.

Beeing off topic is what keeps this topic at the top. It's an exception to the "keep on topic" forum etiquette most people are accustomed, but it gives us a chance to talk about things we wouldn't have if it was strictly on topic. For example this discussion about the (expected) performance. "Ah, a thread about how to set up suricata. That's strange, this guy says he has tested suricata on X and can get Ymbps out of it. I'm only using Zmbps, so it must be able to pull it off. Let's see if using that other guy's suggestions can help out". A couple days later it's "ZOMGWTFWRTBBLLOL! I never knew suricata could do that". How many would find such an advise useful when buying parts for their next box, without having to start separate topics of "will this work for suricata/snort@Xmbps?".

wetspark

@jflsakfja:

Beeing off topic is what keeps this topic at the top. It's an exception to the "keep on topic" forum etiquette most people are accustomed, but it gives us a chance to talk about things we wouldn't have if it was strictly on topic. For example this discussion about the (expected) performance. "Ah, a thread about how to set up suricata. That's strange, this guy says he has tested suricata on X and can get Ymbps out of it. I'm only using Zmbps, so it must be able to pull it off. Let's see if using that other guy's suggestions can help out". A couple days later it's "ZOMGWTFWRTBBLLOL! I never knew suricata could do that". How many would find such an advise useful when buying parts for their next box, without having to start separate topics of "will this work for suricata/snort@Xmbps?".

Perfect example. I was tinkering again this morning while waiting on some other fools at work and thought, "Hmm, wouldn't it be cool if I could use some of these expensive ASICs we use at work for Suricata processing". Then I read this which lead me to search and discover that there's CUDA support being developed for Suricata.

Networks I work on have lots of IDS/IPS and tons of other acronyms running. Its amazing that we have commercial systems that will only do 1Gbps and we are struggling to find/operate 10Gbps effectively while this discussion board is talking about stuff that anyone can download and run on old hardware that can do most/all of 1Gbps.

I primarily work on ADC/ADN and SDN stuff these days. I have several pieces of commercial hardware in the cabinet behind me that I could be using for my network but I'm happy with this P4 running Pfsense and Suricata.

@wetspark:

Perfect example. I was tinkering again this morning while waiting on some other fools at work and thought, "Hmm, wouldn't it be cool if I could use some of these expensive ASICs we use at work for Suricata processing". Then I read this which lead me to search and discover that there's CUDA support being developed for Suricata.

Networks I work on have lots of IDS/IPS and tons of other acronyms running. Its amazing that we have commercial systems that will only do 1Gbps and we are struggling to find/operate 10Gbps effectively while this discussion board is talking about stuff that anyone can download and run on old hardware that can do most/all of 1Gbps.

I primarily work on ADC/ADN and SDN stuff these days. I have several pieces of commercial hardware in the cabinet behind me that I could be using for my network but I'm happy with this P4 running Pfsense and Suricata.

My point exactly. Suricata/snort on pfsense is more than enough for the majority of users out there. There are exceptions (like multiple 10Gbps connections) but for the majority out there, including me, cost of getting the connection far outweighs the need for a faster firewall/IDS. The software side is there (has been for quite a few years actually), it's only a matter of getting the hardware that satisfies the needs. Hardware getting faster/more efficient, while at the same time costing less and less, leads to hardware that can do what 5 years ago could only be dreamed about.

Let's take the example of a pure datacenter pfsense with suricata. It doesn't need to analyze traffic for all the exploits out there about browsers, since none of the computers behind it run that software. Out of all the publicly available rules out there, only a handful actually make sense in that particular environment. You can do a whole lot more with 50 custom rules.

Another example is the typical home connection. It doesn't need to analyze traffic for all the servers out there (web/email/etc). Trimming down the rules to only the bare essentials, and using the recommended rules for a network gateway, coupling it with a (relatively) cheap atom + intel nics gets you a firewall that can be on par with most (if not all) the commercial alternatives out there. The bonus is that since the software is open source, you can still use it 10 years down the line without worrying about an unpatched vulnerability existing. Another bonus is that the software will get better and better.

dgcom

@jflsakfja:

The bonus is that since the software is open source, you can still use it 10 years down the line without worrying about an unpatched vulnerability existing. Another bonus is that the software will get better and better.

I can't agree with this… Reading this forum I see a lot of people with old versions of pfSense having their packages broken because support is removed.
And if some new vulnerability is discovered (like we had earlier with OpenSSL) - you are screwed, because you can't really fix it without pfSense-specific build tools...

So you either upgrade or get outdated.

bmeeks

@dgcom:

@jflsakfja:

The bonus is that since the software is open source, you can still use it 10 years down the line without worrying about an unpatched vulnerability existing. Another bonus is that the software will get better and better.

I can't agree with this… Reading this forum I see a lot of people with old versions of pfSense having their packages broken because support is removed.
And if some new vulnerability is discovered (like we had earlier with OpenSSL) - you are screwed, because you can't really fix it without pfSense-specific build tools...

So you either upgrade or get outdated.

This is true, but to a large extent it is also true for more expensive commercial packages. They frequently move on and drop support for older stuff or don't backport some fixes. Depending on the particular vendor, some are better and some are worse than pfSense in this regard.

Snort on pfSense is a recent example of a package that had to move on and drop support for 2.0.x. That's mainly because the underlying FreeBSD 8.1 OS is EOL, so security updates would not be happening. Also, the old package system using *.tgz files was unwieldy in the way it handled shared libraries. You frequently hit the equivalent of Window's "DLL hell". So moving on to supporting only 2.1 and higher with PBI packages was an inevitable change. Granted FreeBSD 8.3 which pfSense 2.1.x uses also just went EOL, but at least there is ongoing frenzied development to move to FreeBSD 10 with pfSense 2.2.

So while I feel the pain of users with older installations losing support of a package, that has to be tempered with the concept of staying more "current" and keeping the software development task more manageable. Trying to support, for example, three different OS environments is a bit much for a one-man package maintainer. For instance, the installation paths differ between pfSense 2.0 and 2.1 and 2.2; plus the variety of system calls differs. For example, support for IPv6 varies between the versions. So this makes the code in a package sometimes quite complex as it tries to deal with all the subtle differences in pfSense versions. The more complex a package's code becomes, the higher the potential for bugs.

Bill

Mr. Jingles

And if some new vulnerability is discovered (like we had earlier with OpenSSL) - you are screwed, because you can't really fix it without pfSense-specific build tools…

This is a serious problem, indeed.

@bmeeks:

Trying to support, for example, three different OS environments is a bit much for a one-man package maintainer.

Even if it were more than 1 maintainer it would be difficult, would be my guess.

Life evolves, and so does software. It is not always nice to upgrade, because new problems will come along because of that.

But then again: we grow older, and new problems come from that too.

Life is life.

To me the value of open source software is that we are not dependent on crappy companies that sell us product X on day 1, and on day 1 + 1 simply drop support with an arrogant 'sue me, stupid customer'.

For that reason I don't consider open source 'free' as in: money-free: I've donated to many open source projects. I do realize devs want to have a beer too. The value of open source for me lies in: much more serious.

But, in the end: old software has to be replaced by new software. Nature's law, the new replace the old.

Mr. Jingles

@jflsakfja:

@Hollander, it's not just in the Netherlands, it's the world trend now. They feel the need to dumb down the students so their authority is not threatened in the future when they grow up to be the tax-paying zombies they are meant to be ;D

Hmmm 8)

I think you are a little bit too negative.

Have some faith in mankind, in what TPTB cook up for us. After all, the most talented, the most skilled, only the very few that manage to pass all the top tests, are hand picked to become a member of government, or parliament.

People like you and I don't get there, we don't have what it takes. This is a fact of life, to be observed in every country across the globe.

So you might be a little bit too negative.

The central planners know what is right, they are the creme de la creme. We should not criticize. We don't have the brains to do that. That's why we are here and they are there.

I don't want to ridicule you, but I mean, what is next?

You saying that all these highly intellectual real life social insight television shows about pawn shops, cooks that have a tough time in their kitchen (lots of them, given the number of shows about it), 'dr. fil', 'opla', 'who is the best singer/dancer/whiner', 'who can put a cucumber in his ear the fastest', 'remodel your house', 'rent a house from the bank in a place under the sum and pay it off 3 times in intrest and charges' and thelikes are fake?

Given your negative way of looking at the world, I would not be surprised if you would invent a new line for it. 'Scripted reality', or something like that.

Have faith, young man, have faith.

Right, I am off now, watching on our ever reliable state television how bad that certain country in eastern-europe is.

( ;D ;D ;D ;D ;D )

@dgcom:

I can't agree with this… Reading this forum I see a lot of people with old versions of pfSense having their packages broken because support is removed.
And if some new vulnerability is discovered (like we had earlier with OpenSSL) - you are screwed, because you can't really fix it without pfSense-specific build tools...

So you either upgrade or get outdated.

Don't confuse the need to stay up to date with updating the existing software. They are not the same.

Staying on top of vulnerabilities is one thing, supporting every single hardware/software configuration out there is another.

I did mention on a Debian mailing list that one of the biggest mistakes Debian made was forking firefox to iceweasel. Hold your horses, even if I killed someone the judge would still hear me out, so hear me out.

Debian is extremely conservative with their release cycle. A new version is usually a couple years away. These days a "couple years away" is a disaster. An utter and plain disaster. Not in the stuff breaks sense, hell I would trust a life sustaining machine running Debian more than I would trust the multi-million machine in the hospital, but in the needed new features sense. No, not the Gnome 3 f***-me-upside-down-and-inside-out-if-it's-a-usable-gui. The syslog logging for nginx for example. An extremely useful feature, that isn't in the version currently in wheezy (7.x). In my use case I have to import the logs into rsyslog, process them, then spit them out to remote hosts >and< on top of that split them per nginx vhost >and< split them at the same time to normal logs and errors. rsyslog isn't meant to be a text file tailing software. It's meant to take a single line, decide where it should go and get it there. I can make it work though, and patiently await for the time that the newer version gets to stable. I completely know that you can patch nginx and recompile it to get to where I want to be, but that's out of the question for a production machine. I'm not a developer, never will be, I leave that job to others. More on this following.

Debian stable (as already mentioned) is as solid as concrete. Recently they decided to go ahead with the squeeze-lts (long term support), which ties all this to firefox/iceweasel) citing the fact that I-forget-right-now-which-company-exactly recently migrated 5K servers to ubuntu LTS. Most people involved with the decision failed to see the actual reason for the migration: more frequent updates, even if it's for an LTS version.

What does running an LTS mean exactly? It means continuing to support old software, that has fixed the flaws you are trying to backport changes for, in newer versions of that software (bingo for firefox!). They also added a few newer features along the way, as an added bonus. Or extra shafting, depends (like australis). The whole reason iceweasel was born was that mozilla wouldn't let them use their trademarks for supporting EOL versions of firefox.

Canonical can support an LTS version, SPI (Software in the Public Interest, the guys behind Debian) cannot. Why? Hollander hit it right on the head: developer support.

Developers need to get payed to work on software. RMS's extremely distorted reality does NOT apply in the real world. Either contribute to the developers, or forget their support for the software. Nobody will spend a week of their own time working on a version you (not you you, general you) are using, just because you like it and fear installing a newer version, and the reason is they prefer actually working on the feature that 100 others desperately want implemented. Or that recent 0-day vulnerability that needs urgent fixing.

Support doesn't only come in the $$$ sense. It also comes in the documentation sense. How many hours did this topic save a developer? Or a user? A day? A week? A year? Instead of developers spending their time trying to weed out the false positives and improve them, they can instead take a look at the list and immediately see the rules they need to focus on. Users can also do the same, which saves them time asking if that rule is a false positive or not, spending a week arguing on the forums if it truly is a false positive, which by the way is probably a week of a developer spending his time troubleshooting with them/arguing/agreeing, then disabling the rule. Documentation is the most hated thing to do, an I can understand that (this topic is a form of documentation, you can't imagine the time I've spent behind the scenes working on it). It's why I rage about the ET guys ignoring the list ;D. The list is there, and it's accurate, now start removing the rules I suggested.

Got a little carried away, back to our mailing list discussion: The argument was that if I want a patch backported to a software I refuse to update, for personal reasons, it's up to me to make sure the developer has the incentive to do it. If the developer needs to spend a week, at $100/day to backport and troubleshoot the update I need (which by the way comes standard in the newer version) and I can spend $500 to get newer hardware that doesn't have a problem with the newer version, then I could do myself a favor and save the $200. Get newer hardware and don't expect the developer to work exclusively for me.

In the particular case of snort, going by what bmeeks said, I would personally choose to drop support for the older versions of pfsense. Why? Because they will never be completely up to date. By making a conscious choice of running an older version of a critical part of your infrastructure, you are making the conscious choice of getting your network broken into. Do you have the knowledge to backport all the necessary patches into the version you use? Great, the previous statement doesn't apply. Are you a regular user that waits on others to fix it for you? Then try to help them out, by not expecting them to work for free.

Braking something in a newer version is inevitable (although I still don't get how the Debian guys don't break stuff ;)). Our job as users is to let the developers know that they broke something, and work with them to the best of our abilities to get it fixed. If we can provide log samples, we do that. If we can provide kernel oopses, we do that. If we can provide money for the developer to get a pizza for him and his children (hey, 'murica does it and it works, so think of the children), then we do that.

That is what open source means. It doesn't mean "we need to make those businesses fail because ALL software should be free". A starving developer is a soon-to-be-dead developer, simple documented fact.

@Hollander: smack Snap out of it! They got you too?!?! smack. All hope is lost :(

DISCLAIMER
This is NOT a direct attack on anyone around here, nor elsewhere. I'm merely expressing my opinion. If you do not agree with me, discuss in a calm, well-mannered way. Swearing is acceptable, as long as it's directed at the ET guys*, cisco* (for charging a night with your wife, your first born's right kidney, and your left kidney for a support contract), or the developers of either unity*,gnome 3*, or internet explorer* (all versions ok). Windows 8* accepted/encouraged.

*all relation to actual names, claimed trademarks, registered trademarks, physical and/or legal entities, is purely coincidental. Nope, no relation at all to the real world. Nothing to see here, move along.

BBcan177

@bmeeks:

Trying to support, for example, three different OS environments is a bit much for a one-man package maintainer. For instance, the installation paths differ between pfSense 2.0 and 2.1 and 2.2; plus the variety of system calls differs. For example, support for IPv6 varies between the versions. So this makes the code in a package sometimes quite complex as it tries to deal with all the subtle differences in pfSense versions. The more complex a package's code becomes, the higher the potential for bugs.

I agree, keeping up with the updates is important, especially if we have the support of the Maintainer to fix issues that arise…. If it wasn't for Bill maintaining these packages, I am sure we would be more hesitant to upgrade. ;) :o

Double K

@BBcan177:

Could you please shed some more light on the max, dmax, and pmax variables?

Using a "max" variable, if it finds over the Max variable it will perform a Maxmind Geoip Database lookup and will process a /24 block for configured Foreign Countries on an individual Blocklist Basis.

Using a "dmax" variable if it finds over the dmax variable it will perform a Maxmind Geoip Database lookup and will process a /24 block for configured Foreign Countries at the end of the download process on all of the Blocklists together.

Using a "pmax" variable, if it finds over the dmax variable it will process a /24 Block excluding Country Code whitelist at the end of the download process on all of the Blocklists together.

I had 7 unique addresses in the 104.28.7.x range in the IR_Match alias file. This also created a 104.28.7.0/24 entry as well. I tried accessing a site that was not one of the unique addresses (ie. not blocked by the lists), but it was blocked by the /24 range.

You are referencing "Match" aliases here.

Here is a snipet from the pfiprep script-
# country Code p24 Process (pass/match)
ccwhite=match
# Define what to do with IP Ranges found that are in the
# Country Code p24 Process (block/match)
ccblack=block
# For pfSense, the "Match" IPs can be "Monitored" with
# "Floating Rules" which can log packets from these IP Ranges,
# but still allow the Blocking of the Individual IPs found in the
# same /24 Range.
So the script will Block a whole /24 range depending if you select ccblack=block.
ccwhite=match will put the IP ranges that are in the Safe Country list into a match alias.

So the match file has all of the IPs that are being blocked with a "!" at the start of the IP to tell pfSense not to match the "!" excluded IPs, and match on anything else in the /24 range.

I would suggest leaving the "match" alone until you get everything else working. Change the ccwhite=match to ccwhite=pass

So at a high level, the max,dmax and pmax variables look at how many IPs are repeat offenders in all of the blocklists. And then depending on these settings block/match as required.

Thanks BBCan177, I've read & re-read this about 15 times, drawn it out on paper, changed the floating rule for IR_Match to "Match" instead of Block, and still can't figure it out.
Unfortunately, all I seem to be doing now is adding individual IP addresses to the Floating rules to allow them to pass through these Blocks - because legitimate IP addresses such as bing.com's IP address are being blocked by these Blocking Lists.

BBcan177

@Double:

Thanks BBCan177, I've read & re-read this about 15 times, drawn it out on paper, changed the floating rule for IR_Match to "Match" instead of Block, and still can't figure it out.
Unfortunately, all I seem to be doing now is adding individual IP addresses to the Floating rules to allow them to pass through these Blocks - because legitimate IP addresses such as bing.com's IP address are being blocked by these Blocking Lists.

Take a look at posts in this thread weeks ago where the Blocklist Source 'MTA' is blocking some legit sites.

Take a look at the pfiprep script (near the bottom) and read the 'INFO' comments above the Threat Source URLS.

You can disable a Source by commenting it out with "#" at the start of the line.

Unfortunately MTA posts all IPs that were involved in malicious activity. So it you want to keep this list, you need to create a "SAFE Alias" Rule above the "Block/Reject" rules to allow any false positives thru.

If you remove lists, I suggest running

./pfiprep killdb or ./pfiprep killdb skip

dskip option will re-use your existing downloads.

You don't need to play with 'Match', set that to 'pass' for now.

@BBcan177:

I agree, keeping up with the updates is important, especially if we have the support of the Maintainer to fix issues that arise…. If it wasn't for Bill maintaining these packages, I am sure we would be more hesitant to upgrade. ;) :o

+1

From my point of view I want the software I'm using as bug free as possible. If that means keeping up with upstream, then so be it. I'll adapt to the software, instead of the software adapting to me.

I can't remember the exact details (mentioned before I'm not the "keep a google tab open" type of person), but a couple years back support for 286 (or was it up to 386?) was removed from the linux kernel, taking with it a crapload of unneeded code. If memory serves right, the same goes for 32bit popular ZFS distros out there (freenas?). That's the spirit of open source IMHO. The developers try their best to make a decent software for all to use, but within a few reasonable limits. But support for a 286? Why are you still using a 286?

Bought my laptop (a 17" toshiba which I love) in 2009 for €450. It's regular price was €800, but had some coupons (yes I'm that kind of guy) that dropped it down to €450. I don't actually have the need anymore for a 17" laptop. It was used for 3D designing in uni.

During its lifetime, this laptop has seen windows, 3d designing, linux, running linux on top of windows, running windows on top of linux, flashing firmwares to various ARM devices, it only shuts down when I need to be out of town and take it with me, otherwise it's a 24/7-on laptop. It's a dual core 2GHz Turion for crying out loud and it has served my well.

Was looking at a few prices the other day for laptops. A 15.6" freedos (you know it will get formatted right out of the box and debian installed on it) costs about €270-€340, depending who you ask (brand). It's a dual core pentium, with integrated intel graphics. An i3 15.6" costs somewhere around €440 (and that's just the first result that I found, windows 8 (this time formatted before even getting out of the packaging, since technically microsoft must refund the software license if you don't intent on using it, just ask the shop to format it for you before buying it.)). What I will do when my laptop is no longer supported by debian is run debian as long as I can (within the stable supported lifecycle) then air-gap it when support is officially dropped and turn it into an offline laptop for as long as it still works. For the online laptop (the daily use one) I'll just get a newer lower priced laptop, that can do everything this little (well, if you consider 17" little) guy did and more.

I don't expect the developers supporting my hardware forever, and I do understand that they can't actually support it forever. The same goes for a pfsense box. If that box can no longer run a newer version of pfsense, maybe spend the money to get a supported card (if that's the case) or just buy a newer box. As shown in this thread, you don't have to spend a fortune to get the best firewall/IDS. 5 years down the line, if the box still runs the latest version then great. If not, replace it. The electricity + time spent troubleshooting the old box + the speed increase of the newer box, will make up for the newer box.

And for the record, I'm still running pfsense (latest)+suricata on a 3.4 p4 (the "ZOMGITHASHYPERTHREADINGITSLIKEDUALCORES" northwood one). Don't hate it, embrace it. It's old, it's a power hog, it's loud, but it still works. The upstream connection to it is only 10mbps (a dedicated ethernet line that costs 5 arms, 8 legs, and 3 kidneys). If tomorrow support for it was dropped, then it would get upgraded straight to a little celeron that could, without thinking twice about it. I've used atoms, celerons, core 2s, i3s, e3s, and i7s. As long as you don't expect extraordinary things from them, each does wonders in its particular bandwidth category. Do the developers (and the rest of us) a favor and upgrade to a box capable of running the latest version of the software that's protecting your neck.

Don't be that corporate guy that never upgrades installed software in the fear of something breaking and getting fired. If one of my employees came up to me and said "oops, I just broke the email server by upgrading it to a newer version because of a vulnerability", he wouldn't get fired, but instead congratulated on taking the steps to ensure the system was as secure as possible. Who will get fired though is the backups guy, if backups fail to work ;D. If the backups are ok, the first guy gets a box to play with, to make sure the newer version is supported, and works upstream to get the newer version supporting our old system out as fast as they can. That's nothing compared to a compromise caused by an unpatched vulnerability.

Disclaimer
By you I'm NOT talking about anyone in particular, it's the general you.

SCSI_2K

jflsakfja, your write-ups are always great to read. Most topic on this thread is out of my head but I'm learning. Thank to you and BBCan117 for the great work!

I've been reading and re-reading this thread for a week now and almost done setting it up. The only issue that I'm getting is that most of my list from pfiprep only contains 1.1.1.1 IP address. Under the PfIP Reputation widget, I get a lot of fail errors with only IR_PRI1 list getting downloaded.

Running pfiprep from shell give me this kind of errors: ./pfiprep: /usr/local/bin/grepcidr: not found

Example:

looking up rules.emergingthreats.net
connecting to rules.emergingthreats.net:443
SSL connection established using DHE-RSA-AES256-SHA
Certificate subject: /serialNumber=cmxRbct3onRzhDapr8DEMQb9Au33Dqrp/OU=GT44212215/OU=See www.rapidssl.com/resources/cps 14/OU=Domain Control Validated - RapidSSL(R)/CN=*.emergingthreats.net
Certificate issuer: /C=US/O=GeoTrust, Inc./CN=RapidSSL CA
requesting https://rules.emergingthreats.net/open/suricata/rules/tor.rules
local size / mtime: 303563 / 1407962999
remote size / mtime: 303563 / 1407962999
/home/badips/download/tor.rules 100% of 296 kB 725 kBps

Download file count [ 699 ] Outfile count [ 3418 ]

Process /24 Stats
–-------------------------
Found [ 11 ] IP range(s) over the threshold of [ 5 ] on the CC Blacklist
Found [ 3 ] IP range(s) over the threshold of [ 5 ] on the CC Whitelist

Found [ 60 ] CC Blacklisted IP Address(es) are being set to [ block ]
Found [ Skipped ] CC Whitelisted IP Address(es) are being set to [ match ]

Removed the following IP Ranges
81.150.197.|178.32.181.|62.220.148.|46.105.162.|95.130.15.|81.89.0.|185.61.148.|94.198.98.|
–-------------------------
Post /24 Count [ 3366 ]
./pfiprep: /usr/local/bin/grepcidr: not found

Post Duplication count
–-------------------------
Masterfile Count [ 0 ]
Outfile Count [ 0 ] [ Passed ]
–--------------------------------------------------------------

Thanks for this thread, I switch from snort to suricata.

BBcan177

@SCSI_2K:

Running pfiprep from shell give me this kind of errors: ./pfiprep: /usr/local/bin/grepcidr: not found

You need to install this application called Grepcidr first.

Since FreeBSD has moved the pkg files to archive, you need to make a few changes:

[amd64]

setenv PACKAGESITE http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/ports/amd64/packages-8.3-release/net-mgmt/

or

[i386]

setenv PACKAGESITE http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/ports/i386/packages-8.3-release/net-mgmt/

then:

pkg_add -r grepcidr-1.3_1.tbz

Once this is installed. Try to see if it loads properly:

/usr/local/bin/grepcidr -V

./pfiprep killdb/jem@berkes.ca