RFC (make up a number not in use) - Blueprint for setting up snort + pfblocker
-
A couple of months back I contacted the snort guys to remove some rules that were carved in stone back in pre-historic times, since the stone was chipped and some pieces of it went missing. A few thousand years later, the rules are still there, and that's why I decided to contact them. They completely ignored me, thinking (and if any of them passes by here, please confirm) that no matter how old a rule is, it's still worth it to spend the man hours to maintain it. That's one X for snort.
You can see based on Bill's posts and mine around here that snort hasn't updated the imap commands yet. A critical part of an IDS for an "enterprise" use is therefore missing. That's two Xs for snort.
snort-inline. No need to comment on this, other than R.I.P. snort-inline. Three Xs so far. At this very same point, snort was voted out of the game. We decided to give it another chance though.
suricata comes along with its (far) superior stream and detection engine. Fanboysm aside, suricata was directly funded by the U.S. government for use as a toy in NSA's plans. As always, NEVER trust a single word, let alone a dot, of what I say. You can verify everything I say online. If it's good for them, it sure as hell it's good for me. That funding gave birth to a system that can max out a 10Gbps duplex connection, and have spare cycles to render a 3D video, at the same time. As it stands >right now<, comparing snort and suricata, suricata offers the ability to take advantage of our systems at their max potential, as they are right now. Since most people followed another member's advise (and basically ignored me), they used i3+ systems. Those systems went underutilized so far, since I have mentioned a (few) thousand times that snort doesn't actually act on the original packet, but a copy of the packet. To put it simply, the less than a handful of forum members that did decide to take my advice, saw snort running as fast on a p4 as an overclocked i7. Then again, who am I to know, right? Those systems (i3+) are completely ready for suricata's inline part, and will max out more bandwidth than you can afford. This is a HUGE advantage to suricata over snort.
Suricata is like building a solid foundation for the future. Think of it like snort on steroids. Better log management, better capture facilities. When the inline part finally arrives in pfsense, that will transform pfsense into a multi million $ firewall system, capable of taking even the best of the best on (cough USISC, later assimilated into NETCOM).
Speed wise, they both are pretty much the same as it stands right now.
Putting both systems side by side, most users will not notice the difference, right now. A year later when the inline part arrives, those users will run around reconfiguring their systems from the ground up. The rest of us will just tick a checkbox and forget about it.
For the upcoming suricata topic, I've decided to completely ignore the snort rules. Those rules that made it into ET (gpl for example) will be there, since they are technically part of ET, but the rest will be ignored. I believe it's best if we cut all ties with snort and let it die the horrendous death it deserves.
WRT the money needed for a subscription, I would prefer it if anyone finds any of the two topics interesting, instead of contacting me for donations, they donate to ET by purchasing a subscription. Rule maintainers have to make a living you know.
-
@jflsakfja:
A couple of months back I contacted the snort guys to remove some rules that were carved in stone back in pre-historic times, since the stone was chipped and some pieces of it went missing. A few thousand years later, the rules are still there, and that's why I decided to contact them. They completely ignored me, thinking (and if any of them passes by here, please confirm) that no matter how old a rule is, it's still worth it to spend the man hours to maintain it. That's one X for snort.
You can see based on Bill's posts and mine around here that snort hasn't updated the imap commands yet. A critical part of an IDS for an "enterprise" use is therefore missing. That's two Xs for snort.
snort-inline. No need to comment on this, other than R.I.P. snort-inline. Three Xs so far. At this very same point, snort was voted out of the game. We decided to give it another chance though.
suricata comes along with its (far) superior stream and detection engine. Fanboysm aside, suricata was directly funded by the U.S. government for use as a toy in NSA's plans. As always, NEVER trust a single word, let alone a dot, of what I say. You can verify everything I say online. If it's good for them, it sure as hell it's good for me. That funding gave birth to a system that can max out a 10Gbps duplex connection, and have spare cycles to render a 3D video, at the same time. As it stands >right now<, comparing snort and suricata, suricata offers the ability to take advantage of our systems at their max potential, as they are right now. Since most people followed another member's advise (and basically ignored me), they used i3+ systems. Those systems went underutilized so far, since I have mentioned a (few) thousand times that snort doesn't actually act on the original packet, but a copy of the packet. To put it simply, the less than a handful of forum members that did decide to take my advice, saw snort running as fast on a p4 as an overclocked i7. Then again, who am I to know, right? Those systems (i3+) are completely ready for suricata's inline part, and will max out more bandwidth than you can afford. This is a HUGE advantage to suricata over snort.
Suricata is like building a solid foundation for the future. Think of it like snort on steroids. Better log management, better capture facilities. When the inline part finally arrives in pfsense, that will transform pfsense into a multi million $ firewall system, capable of taking even the best of the best on (cough USISC, later assimilated into NETCOM).
Speed wise, they both are pretty much the same as it stands right now.
Putting both systems side by side, most users will not notice the difference, right now. A year later when the inline part arrives, those users will run around reconfiguring their systems from the ground up. The rest of us will just tick a checkbox and forget about it.
For the upcoming suricata topic, I've decided to completely ignore the snort rules. Those rules that made it into ET (gpl for example) will be there, since they are technically part of ET, but the rest will be ignored. I believe it's best if we cut all ties with snort and let it die the horrendous death it deserves.
WRT the money needed for a subscription, I would prefer it if anyone finds any of the two topics interesting, instead of contacting me for donations, they donate to ET by purchasing a subscription. Rule maintainers have to make a living you know.
Very well said. I would like to see the Emerging Threats guys offer a slightly more affordable subscription package for home users. $300-$500 USD bruises your wallet quite a bit more than $29.99 USD. For a commercial entity, it's a complete no-brainer. For home enthusiasts, it might be harder to get approval from the better half for $500… ;)...after all, that could buy another set of curtains or maybe that dress she has been looking at... ;D.
Bill
-
Very well said. I would like to see the Emerging Threats guys offer a slightly more affordable subscription package for home users. $300-$500 USD bruises your wallet quite a bit more than $29.99 USD. For a commercial entity, it's a complete no-brainer. For home enthusiasts, it might be harder to get approval from the better half for $500… ;)...after all, that could buy another set of curtains or maybe that dress she has been looking at... ;D.
Bill
Even for a commercial entity, $300 here, $300 there, quickly adds up to bills waiting to be paid next year and continues until the business goes bust.
What I would be more that willing to pay? $50/year/pc. Everybody is happy. A home user is unlikely to install the rules on more than 1 pc, so the cost is kept low. An enterprise is not likely to install the rules on more than a dozen PCs, again cost is proportionate to the stuff you get out of them. And the best part is rule maintainers gets payed as well, from everybody.
Since this topic has evolved into a "security" "experts" (not meaning to offend anyone here, it's directed at the individuals charging a few millions a year to provide network "security", yet still connect a scada system on the internet), why not move it up a notch and ridicule the MBAs as well, with a short lesson on how to build a solid successful business.
Lets say you are selling a blank box. You start your company saying "hey, if I sell even 1 of them for a million dollars, I'd be a millionaire!". So you set the initial price for a blank box for 1 million $. You set up a factory to produce them, hire workers, build your new store, and wait. Days, months, years go by, and you are sitting at your new store, waiting for someone to come in and sell them that box to be a millionaire. 20 years later, after racking up a 60 million $ total bill (electricity,workers, etc.) a box collector comes in to your store. He is totally amazed at the quality and design of your box, and is willing to buy one right now. Your reply is "what good is 1 million when I have to pay 60?" and reluctantly sell a box. 5 years later, the business goes bankrupt.
Now let's examine the example of someone trained by me personally. He knows the secret to a successful business is repeated business. He sets up a factory, and hires a single worker, automating most of the process of manufacturing a box. He sets up the new store, with samples of the new boxes. His secret is his price. He charges $1 for each new box, while everybody else charges $2. His total expenses to manufacture that box amount up to 50% of its price (cardboard,electricity,worker's salary split over the amount of boxes he can make in a shift, etc etc). He earns a respectable 50% on each box sold. A store owner passes by the new store and sees the prices and falls in love with them. He storms into the shop, already signing a check for a 100,000 pieces order. He is also willing to sign a contract for monthly delivery of boxes to his store. At his next golf game, he can't stop talking to his buddies about the extremely low priced boxes he bought. One of those buddies is a shoe making business owner. He walks into the new store the very next day, with a 200,000 pieces contract waiting to be signed. Everyone that passes by the store is stunned by the price. Their reaction is "ZOMG!!!111oneelevenhundredandelevenWTF!?!? THESE BOXES ARE HALF THE PRICE OF ALL OTHER BOXES". A year later the business has grown so much, it now employees 20 guys in the factory, and has opened up a dozen new stores. The business owner is also a millionaire, and has set up the foundation for the future.
Those 2 examples come directly from stuff I've seen in my life, and although they seem off topic for most people, are actually spot on topic.
What is the operational expense for a PC downloading the ET rules? If I say $500, I'd be willingly lying out of my teeth. I'm sorry guys, but I cannot do that. The operational expense for a single PC downloading the rules is not more than $10. What does it take to keep those rules updated? I'm sure an out of job developer would be more than willing to spend a few hours a week maintaining the rules, for $500 a month.
Making a profit FROM A SINGLE PC means charging $520/subscription (for $10 profit per subscription)
Spreading the profit onto a thousand PCs, means charging $20.50 (for $10 profit per subscription).
See where I'm getting at? You could even charge $50 and people would think "it's a f***ing steal!", yet you would be making 39.50/subscription. Just scale up from there and build a successful business on repeated business.
-
@jflsakfja:
why not move it up a notch and ridicule the MBAs as well, with a short lesson on how to build a solid successful business.
Finally something I can say something about with complete backing of my education (I hold quite some degrees in this very area) ;D ;D ;D
Not all 'MBA's' are the same: unfortunately around 95% of them are. And are exactly what you describe. This in itself has no longer anything to do with education, and as such I'd like to quote one of my favourite quotes:
Hyperinflation is not restricted to monetary events
With which I am referring to the current education scam, that is hitting the youngsters hard. As such, a simple google will lead you to many articles (even from trustworthy sites) about the insane college tuition bubbles that have been blewn around the world.
That aside: what you are illustrating is somebody who did not ever complete macro-economics 101, let alone even an introduction to Marketing, more specifically SOP, Sales & Operations Planning. Because in there we teach the simple concept of supply and demand. Most people know these two words, but they don't know that in SOP we use this to determine the optimal sales price. All others assumed standard, that means that sales price in which our profits are maximized.
The last lines of you are what is called 'marginal profitability'. Which specifically goes for digital goods, where the marginal profitability is around 100%. Given the concept of 'sunk cost', it means you, for short term sales pricing, should accept any price above the variable cost. Of course, if you are a victim of 1 of these 95%-'MBA's' you will pick the highest price you can think of. If you are one of the 5%, you know about SOP ;D
I by the way have a subscription to Snort, and the 30 USD/year is most fair. Obviously, somebody over there understands SOP. I also contacted the people over at ET some time ago, and was amazed by their arrogance, even when I explained SOP and marginal profits to them, in an attempt to have them lower the price to the levels of Snort for home users. And this only goes to prove the quote I put above. Hyperinflation is also observed in signatures of rather moron-alike people like the severely mentally handicapped person I had the chat with. He was a very important [fill in all kinds of expensive sounding words here] 'senior' 'manager' who told me: 'we don't care about your explanations, obviously, if you don't want to pay our price, you are not the customer we are looking for'.
I think it is all these processed foods people eat that have evaporated their brains :'(
( ;D )
-
@jflsakfja:
A couple of months –-SNIP ---
I forgot: thanks for the clarification :D
(As to paying ET as you suggest: I have no problem with that, but I explained right above this reply from me what mentally deceased person I had the pleasure of communicating with trying to get a home-user affordable price).
-
A little help with the snort configuration you suggested on the first page - just how much memory does it need? I threw 4GB at the setup thinking that would be plenty but whenever I try to enable one of the larger groups like ET-trojan.rules or ET-malware.rules, snort turns itself off for that interface.
Jun 9 16:29:21 check_reload_status: Syncing firewall Jun 9 16:29:21 php: /snort/snort_rulesets.php: [Snort] Updating rules configuration for: LAN ... Jun 9 16:29:23 php: /snort/snort_rulesets.php: [Snort] Enabling any flowbit-required rules for: LAN... Jun 9 16:29:23 php: /snort/snort_rulesets.php: [Snort] Building new sig-msg.map file for LAN... Jun 9 16:29:27 php: /snort/snort_interfaces.php: Toggle (snort starting) for LAN(LAN)... Jun 9 16:29:27 php: /snort/snort_interfaces.php: [Snort] Updating rules configuration for: WAN ... Jun 9 16:29:27 php: /snort/snort_interfaces.php: [Snort] Enabling any flowbit-required rules for: WAN... Jun 9 16:29:27 php: /snort/snort_interfaces.php: [Snort] Building new sig-msg.map file for WAN... Jun 9 16:29:27 php: /snort/snort_interfaces.php: [Snort] Updating rules configuration for: LAN ... Jun 9 16:29:29 php: /snort/snort_interfaces.php: [Snort] Enabling any flowbit-required rules for: LAN... Jun 9 16:29:29 php: /snort/snort_interfaces.php: [Snort] Building new sig-msg.map file for LAN... Jun 9 16:29:30 check_reload_status: Syncing firewall Jun 9 16:29:30 php: /snort/snort_interfaces.php: [Snort] Snort START for LAN(em1)...And that's the last line - no actual error, just snort no longer running on the interface. Is it memory? If not, what else is going on?
-
A little help with the snort configuration you suggested on the first page - just how much memory does it need? I threw 4GB at the setup thinking that would be plenty but whenever I try to enable one of the larger groups like ET-trojan.rules or ET-malware.rules, snort turns itself off for that interface.
Jun 9 16:29:21 check_reload_status: Syncing firewall Jun 9 16:29:21 php: /snort/snort_rulesets.php: [Snort] Updating rules configuration for: LAN ... Jun 9 16:29:23 php: /snort/snort_rulesets.php: [Snort] Enabling any flowbit-required rules for: LAN... Jun 9 16:29:23 php: /snort/snort_rulesets.php: [Snort] Building new sig-msg.map file for LAN... Jun 9 16:29:27 php: /snort/snort_interfaces.php: Toggle (snort starting) for LAN(LAN)... Jun 9 16:29:27 php: /snort/snort_interfaces.php: [Snort] Updating rules configuration for: WAN ... Jun 9 16:29:27 php: /snort/snort_interfaces.php: [Snort] Enabling any flowbit-required rules for: WAN... Jun 9 16:29:27 php: /snort/snort_interfaces.php: [Snort] Building new sig-msg.map file for WAN... Jun 9 16:29:27 php: /snort/snort_interfaces.php: [Snort] Updating rules configuration for: LAN ... Jun 9 16:29:29 php: /snort/snort_interfaces.php: [Snort] Enabling any flowbit-required rules for: LAN... Jun 9 16:29:29 php: /snort/snort_interfaces.php: [Snort] Building new sig-msg.map file for LAN... Jun 9 16:29:30 check_reload_status: Syncing firewall Jun 9 16:29:30 php: /snort/snort_interfaces.php: [Snort] Snort START for LAN(em1)...And that's the last line - no actual error, just snort no longer running on the interface. Is it memory? If not, what else is going on?
I found some posts via a Google search on the Snort mailing lists (or maybe it was Google Groups…) about the 2.9.6.0 version gobbling up memory fairly aggressively in higher traffic situations. This could also happen with lots of rules enabled. The ET-Trojan and ET-Malware rules groups are a bit large. There are settings for memory caps for the various preprocessors on the PREPROCESSORS tab. Adjusting those may help. It's also not outside the realm of possibility the Snort binary for this version (2.9.6.0) has a hidden bug. Signal 11 means a SEGFAULT occurred (or in less geeky terms, some piece of code tried to write to or access memory outside of its assigned address space). At least one other user had some "snort exited on signal 11" messages in their log during rule updates.
Bill
-
With regards to RAM usage, I've never seen snort go above 40% on a 2GB system.set up identical to this topic.
-
OK some more info here:
Using the Connectivity ruleset, 17% of 3051MB usage. And that's without most of the EmergingThreats rules.
Using the Security ruleset, 82%.I was amused to see this in my log one of the times it failed…
Jun 9 19:27:50 php: /snort/snort_interfaces.php: The command '/usr/local/bin/snort -R 60190 -D -q -l /var/log/snort/snort_em160190 --pid-path /var/run --nolock-pidfile -G 60190 -c /usr/pbi/snort-i386/etc/snort/snort_60190_em1/snort.conf -i em1' returned exit code '255', the output was 'Spawning daemon child... My daemon child 35849 lives... Daemon parent exiting (-1)'Is there some way to export the config file so you can have a look to see what I might have done wrong?
-
Let me take a wild guess that you are using AC. Use AC-BNFA.
EDIT: Actually AC-BNFA-NQ
-
Actually, AC-STD. However I seem to have solved it with the following process:
- Set the search method to AC-BNFA - this brought memory usage down to 10%
- Enable the desired rules - memory usage 13%
- Change search method back to AC-STD - memory usage 14%.
Voila!
-
And it will choke on the next rule update. Please use AC-BNFA-NQ. AC-BNFA-NQ replaced AC-BNFA, which in turn replaced AC*. Trust me, I was running AC-BNFA-NQ on multiple production networks, and I haven't created a black hole while doing so.
-
@jflsakfja:
And it will choke on the next rule update. Please use AC-BNFA-NQ. AC-BNFA-NQ replaced AC-BNFA, which in turn replaced AC*. Trust me, I was running AC-BNFA-NQ on multiple production networks, and I haven't created a black hole while doing so.
jflsakfja is correct. AC-BNFA-NQ is the best these days. Some of the other pattern matchers can gobble memory like crazy. And Suricata can be worse than Snort in this regard.
Bill
-
@jflsakfja:
And it will choke on the next rule update. Please use AC-BNFA-NQ. AC-BNFA-NQ replaced AC-BNFA, which in turn replaced AC*. Trust me, I was running AC-BNFA-NQ on multiple production networks, and I haven't created a black hole while doing so.
jflsakfja is correct. AC-BNFA-NQ is the best these days. Some of the other pattern matchers can gobble memory like crazy. And Suricata can be worse than Snort in this regard.
Bill
Not if you have a lot of RAM lol…
EDIT:
I have been using "AC" on a 32GB box and "AC-SPLIT" on all others.
I will try to see if changing the "AC-SPLIT" to "AC-BNFA-NQ" fixes the crashing at rule updates sometimes.
-
AC-BNFA-NQ.
AC-NQ is about 30% more ram efficient than AC-SPLIT, with an increased CPU usage.
AC (plain) is like killing a fly with a deathstar. AC-NQ replaced it, as in AC (plain) is now obsolete, you get no added benefits from AC over AC-NQ.
The best balance between RAM usage (more interfaces/more rules) and CPU is AC-BNFA-NQ. It's a single dropdown change, and an interface restart. Just try it, it will not bite.
On a side note, 32GB RAM is suricata's 10Gbps territory.
-
Not that I don't believe you, but Its hard to find some good docs on this.. Do you have any links?
-
If memory serves right, ac-bnfa-nq is the default if no method is specified. In order to be the default, it's probably the best, unless pushing snort to its limits, which unless you are in multiple-1Gbps territory, is nowhere near close to them.
A quick search gives this: http://sourceforge.net/p/snort/mailman/message/27113364/
A quick re-read of the entire thread will show that there are persons that think this topic is the absolute best way to set up snort on the entire internet. It might even give the hint to a certain group of people that want to build a statue of me, thinking I'm the messiah or something. A quick search around the forums will show others even going out of their way and lying that an i3 is faster than a p4 for a regular home connection while running snort.
If it meant completely tearing down your existing system and rebuilding it from scratch, then I understand the skepticism. As I said, it's a simple dropdown selection. Just try it, if you see that under your use case it chokes then try another method.
-
@jflsakfja:
A quick re-read of the entire thread will show that there are persons that think this topic is the absolute best way to set up snort on the entire internet. It might even give the hint to a certain group of people that want to build a statue of me, thinking I'm the messiah or something. A quick search around the forums will show others even going out of their way and lying that an i3 is faster than a p4 for a regular home connection while running snort.
I guess I am one of those people who sing your praise here… on my home system using your methodology and cookbook to get there, I cut my memory usage by more than half and CPU usage rarely comes off idle (AMD athlon 64 x2 with 4GB). I'm retired now but I spent enough years in the IT community (I was around before IT was IT) both as a worker bee and GOD that I can spot those who get it... vs. those who really understand it, live it, breathe it. I also learned not to argue with success until the next bright boy comes up with something better - and I've been on both sides of that coin. So yeah, I'd chip in a few coins towards your statue....BUT, considering you are still holding out on some of the rules that would be pretty helpful to us home users… we're talking "Winged Victory" here, not "David" (hint, one is missing their head) <wink>Rick</wink>
-
The upcoming suricata topic will include a long section dedicated to creating suricata rules specific to a network gateway. There is no need to release my custom rules, since it will be explained how to create those rules on your own.
If most of the stuff I've written so far is considered short by me, then you'll realize what long means when you see the topic ;)
I'm actually putting the finishing touches (writing entire paragraphs here and there) on the first part, the firewalling part right now.
-
@jflsakfja:
that want to build a statue of me, thinking I'm the messiah or something.
;D ;D ;D
