Snort 3
-
Since I maintain the Snort 2.9.x package on pfSense, I guess it's logical that I will be the one to create the Snort3 package...
I tried to compile Snort3 as a test package for pfSense back a few months ago. The compilation failed. I have not tried since then, but I do have plans to see about introducing a Snort3 package in the future. Mostly waiting for it to go to RELEASE from the current BETA mode.
I will also have to completely rewrite the custom blocking plugin used on pfSense. The new Snort3 architecture is quite different in terms of the internal plugin plumbing as compared to Snort 2.9.x. Because of that, it is likely the first version of Snort3 might offer IDS mode only with no blocking available. Depends on how hard it is to rewrite the blocking plugin and integrate it with Snort3.
So to shorten that response down, I would not expect to see Snort3 on pfSense in the near term (where near term means in the next 3 to 6 months). Maybe later this year, though.
-
Thanks for the quick reply. It sounds like it's not just as simple as just dropping it in.
If there is any way for me to help, just let me know.
You probably don't have a way to do this, or allowed, but I would be willing to donate $100 to a fund to help expedite this process, so you could make it a priority**. Also, by what you said, this may not even be worth the effort until it leaves BETA.
**(I say this with the assumption that others like me have the same motivation, where a legitimate pool could be created and funds released upon completion (in a reasonable amount of time).
Just trying to find a positive way to make everyone happy :) If this is even possible, maybe someone else knows a way to set this up. (or the reply may be NO, don't even try)
A year sounds SOOO far away. Sigh...
-
@talaverde said in Snort 3:
Thanks for the quick reply. It sounds like it's not just as simple as just dropping it in.
If there is any way for me to help, just let me know.
You probably don't have a way to do this, or allowed, but I would be willing to donate $100 to a fund to help expedite this process, so you could make it a priority**. Also, by what you said, this may not even be worth the effort until it leaves BETA.
**(I say this with the assumption that others like me have the same motivation, where a legitimate pool could be created and funds released upon completion (in a reasonable amount of time).
Just trying to find a positive way to make everyone happy :) If this is even possible, maybe someone else knows a way to set this up. (or the reply may be NO, don't even try)
A year sounds SOOO far away. Sigh...
pfSense has a mechanism called "Bounties". Here is a link to that sub-forum: https://forum.netgate.com/category/30/bounties.
I will try compiling Snort3 again in the near future (this month) and see what happens. I am also working on improving Netmap integration in DAQ, the data acquisition library used by Snort. I have been having an email conversation with the Snort principals (Cisco folks, now) about this work. Still a long way from being ready because Netmap documentation is sparse and programming examples are even rarer. The Netmap work should carry over to both Snort 2.9.x and Snort3.
I dropped the Snort3 work primarily because of the compile failure and because Snort 2.9.12 came out along with Suricata 4.1.x. Both of those packages took the higher priority.
-
Great Answer. I'll check out the bounties.
If it wasn't for the the lack multi-threading, I wouldn't really care. I have a gigabit fiber connection and Snort takes a big toll, even on my 4+ GHz i7. I opted against $uricata because their paid options aren't affordable for the little guy. I'm willing to pay for a subscription, but not $500 or even thousands per year for home use.
As you know, the multi-threading is just one benefit. I can't wait to put 3.0 into action. Thanks for the dedication and hard work.
-
@talaverde said in Snort 3:
Great Answer. I'll check out the bounties.
If it wasn't for the the lack multi-threading, I wouldn't really care. I have a gigabit fiber connection and Snort takes a big toll, even on my 4+ GHz i7. I opted against $uricata because their paid options aren't affordable for the little guy. I'm willing to pay for a subscription, but not $500 or even thousands per year for home use.
As you know, the multi-threading is just one benefit. I can't wait to put 3.0 into action. Thanks for the dedication and hard work.
Be aware that multithreading is not necessarily all that it's marketed as. Suricata has multithreading and can still have bottlenecks in some situations. There are several whitepapers scattered around the web that document testing scenarios with multithreading on Suricata. Also, the Snort folks for years pooh-poohed the advantages of multithreading; especially after Suricata came out and had multithreading while Snort 2.9.x did not ... . Funny how multithreading wasn't that big of a deal until Snort3 got it.
Not saying multithreading can't be a good thing, but the performance gains are all wrapped up in the details of its implementation and exactly how packets pass from module to module (decoding, protocol detection, signature matching and alerting).
-
Beware that multi-threading is not necessarily all that it's marketed as...
Being that I currently have Snort installed, instead of Suricata, is a way of agreeing with that comment (to a point). I'm not expecting the holy grail, but i am 'excited' to see what improvements there are.
I took a 50% hit with Snort (though who really needs 80MB/s+ for daily driving). If Snort 3.0 doesn't give back some of my bandwidth, I'll probably drop my fiber down to 30MB/s, even it's on $20/mo difference.
Not to have a topic change, but I originally had Snort monitoring both my WAN and LAN. I lost too much bandwidth. So, I cut it to just my LAN. I'm hoping I can either go back to monitoring both WAN and LAN, or sticking with LAN, but getting my speeds back... some? a little? hopefully it's measurable.
We'll see. Thanks.
-
BTW, just in case there may have been some confusion, my comment about multithreading not being a big deal until Snort3 had it was a small jab/joke aimed at the Snort developers and not you personally . Just wanted to clarify that to be sure no offense was taken.
Personally, for home use, I see no point in running the IDS/IPS on the WAN. The default WAN rule set will drop all unsolicited traffic anyway, so putting the IDS/IPS there is just going to get a lot of stuff logged that the firewall is dropping anyway. Snort sits in front of the firewall on the WAN side, so it sees and logs everything before the firewall rules processing occurs.
I suggest home users, and even many business users, put Snort (or Suricata) on the LAN. That does two things. First, it removes a bunch of "noise" from the logs that the firewall dropped. No sense in my view of logging and alerting on a bunch of stuff the firewall rules are going to drop anyway. Second, it means all the local host addresses in alerts will be the actual LAN address of the host and not the WAN's public IP (like happens when you run Snort on the WAN). This is because on the WAN, Snort sees traffic before the NAT rules are "undone". Of course if you are not using NAT, then this second point is moot.
A 50% performance hit is unusual unless your NIC and/or CPU is a little low on horsepower. Of course the number of enabled rules plays a significant role as well. The packet size can also be a big factor in performance. A bunch of small packets like ICMP replies consume a lot of CPU as opposed to fewer packets but with bigger payload sizes. So this means your MB/sec will increase with larger packets while CPU utilization may even decrease a bit.
-
re: multi-threading, completely understood your jab. I just meant I don't expect a miracle. I think it's funny how they've been careful to use the proper wording for that new 'feature'.
re: WAN. Just like pfBlockerNG, the WAN blocks by default anyway, so why filter? I was torn on that one with Snort. I thought maybe it's good to have another check in case an internal (bad) app requested data (i.e. virus). Your vote helped and your logic is sound. LAN only -- got it.
I'm using the 'Security' IPS Policy, all the ET Open Rules, and all the OPENAPPI rules. So, a lot of rules! (I'm still tweaking)
(AT&T Gigabit Fiber, Intel NICs w teaming, 10GB RAM, i7-4790, SSD)
I've had to disable many APPI rules for apps I'm running (or false alerts), and still working on that.
The 50% hit was when I had both WAN and LAN enabled. When I have time, I'll post a comparison of speeds w/ Snort on and off. May do separate thread. I wish I had a better way to identify false alerts while also test effectiveness (see Snort work in action). I have lots to learn.
(Side question - would using RAM disks help, as I have plenty of RAM, or is Snort all in RAM anyway? Could it hurt?)
-
@talaverde said in Snort 3:
(Side question - would using RAM disks help, as I have plenty of RAM, or is Snort all in RAM anyway? Could it hurt?)
Snort is totally a RAM resident beast other than of course the log files. So a RAM disk is of zero benefit really, and can actually lead to issues if it's not large enough as Snort needs 256 MB or so of space in /tmp to download, extract and install rule updates.
-
A 50% performance hit is unusual unless your NIC and/or CPU is a little low on horsepower. Of course the number of enabled rules plays a significant role as well. The packet size can also be a big factor in performance. A bunch of small packets like ICMP replies consume a lot of CPU as opposed to fewer packets but with bigger payload sizes. So this means your MB/sec will increase with larger packets while CPU utilization may even decrease a bit.
In a very rough test, I'm getting 700 Mb/s down / 600 Mb/s up on my 1Gb/s fiber w Snort OFF. With Snort on, it drops to 496 Mb/s down / 525 Mb/s up.
I turned off all the OPENAPPI rules as I'm not really sure how they are helping me with just one or two people in my home. With those off, my speeds went back up to 680 Mb/s down / 600 Mb/s up.
(Note - all these speeds are through a VPN. Big Brother AT&T has BLOCKED all speed test sites. The only way I can run an legitimate speed tests are through a VPN. The main goal here is to measure the affect of Snort anyway)
Would you agree that the OPENAPPI rules aren't worth the speed drop they produce?
-
@talaverde said in Snort 3:
A 50% performance hit is unusual unless your NIC and/or CPU is a little low on horsepower. Of course the number of enabled rules plays a significant role as well. The packet size can also be a big factor in performance. A bunch of small packets like ICMP replies consume a lot of CPU as opposed to fewer packets but with bigger payload sizes. So this means your MB/sec will increase with larger packets while CPU utilization may even decrease a bit.
In a very rough test, I'm getting 700 Mb/s down / 600 Mb/s up on my 1Gb/s fiber w Snort OFF. With Snort on, it drops to 496 Mb/s down / 525 Mb/s up.
I turned off all the OPENAPPI rules as I'm not really sure how they are helping me with just one or two people in my home. With those off, my speeds went back up to 680 Mb/s down / 600 Mb/s up.
(Note - all these speeds are through a VPN. Big Brother AT&T has BLOCKED all speed test sites. The only way I can run an legitimate speed tests are through a VPN. The main goal here is to measure the affect of Snort anyway)
Would you agree that the OPENAPPI rules aren't worth the speed drop they produce?
Yes, I would not use the OpenAppID rules in a home environment. Those are primarily aimed at the corporate world to help IT Security admins enforce employee policies (such as no Twitter, Facebook or Instagram on company time, etc.). That is of limited benefit at home unless you are just curious and want to experiment (or you want to torture any teenage kids in the house ).
-
Yes, I would not use the OpenAppID rules in a home environment. Those are primarily aimed at the corporate world to help IT Security admins enforce employee policies (such as no Twitter, Facebook or Instagram on company time, etc.). That is of limited benefit at home unless you are just curious and want to experiment (or you want to torture any teenage kids in the house ).
Thanks for the confirmation.
I've managed to manipulate my AT&T modem to allow speed tests (DSLReports / Ookla). With no VPN, I'm getting roughly 850 Mb/s down / 825 Mb/s up. That's through pfSense (HyperV VM), 24-port L2 Switch, Snort ON, and pfBlockerNG ON. (Oh, yea, I turned off OpenAppID in Snort.) Losing 150 Mb/s to all that isn't bad IMO.
As a VPN is a MUST these days, those test above aren't reality. Doing the same test as above but with a VPN has been a nightmare. I can't get any consistent results. Not only are VPN services notoriously inconsistent, Snort is throwing all sorts of alerts with DSLReprorts speed test. Even w/o the alerts, I can't get any constant speed test results worth reporting.
Some of the Snort alerts, as an example:
Sensitive Data was Transmitted Across the Network / (spp_sdf) SDF Combination Alert
Sensitive Data was Transmitted Across the Network / SENSITIVE-DATA Email Addresses
... multiple times
Other alerts too, I think.Both DSLReports and Ookla are giving inconstant results.
In short, best I can tell, I'm getting roughly 600-700 Mb/s down / 300-500 Mb/s up with VPN. Sometimes upload drops to <100Mb/s for some reason (my guess, Snort is blocking it)
I've tried both PIA and NortdVPN (integrated into pfSense)
Once I can provide more meaningful data, I'll report back. In short, it seems like Snort isn't playing well with either VPN or the speed tests, or the combination of the two.
-
In attempt to close this out:
After a reboot, my speeds were 800 Mbps / 655 Mbps through VPN, Snort and pfBlockerNG. I'm quite happy with these results. (Only concern is that this required a reboot. What happens after time?? Only time will tell). Moral of this story is that Snort doesn't have a 'debilitating' affect on speeds with reasonable number of rules. Nor does pfB. (VPN is the most inconsistent)
The biggest problem is combining inconsistent VPN speeds, along with Snort analyzing various speed test scripts. Too many variables to give good results. Best bet is to measure each of these individually.
(For the average users) The best set of Snort rules seems to be:
- LAN (Only)
- 'Security' IPS Policy (Subscribe to paid Oinkmaster code)
- ET Open rules (Non-paid - too expensive for personal use)
- No OpenAppi Rules - Unless you have a family to torture ;)
I haven't seen any significant performance issues with this setup. My main conclusion from all this is that pfSense needs a separate section to discuss VPN speeds. Most articles on the web are paid for and don't take pfSense into consideration. (To those listening - PLEASE set up separate board for VPN discussions. thanks!)
-
@talaverde
Why do you think a VPN is a necessity for web browsing? For most home users, the only real need for a VPN is to allow secure remote access back into a home network from outside (think Remote Desktop or SSH, for example). I am not an advocate of VPN use for privacy reasons. But to each his own, I guess.As for your Snort alerts on Sensitive Data, you do not need those rules enabled in my opinion. They are, much like the OpenAppID rules, aimed primarily at corporations and others who have either HIPPA or financial data on their networks that needs protection from ex-filtration. Those rules will false positive on a lot of normal home network traffic (such as simply signing up on a web site with your email address). Who wants an alert and block about that?
Dont' take this in a negative way, 'cause that's not how I mean it, but you need to do some research on IT security and tuning an IDS/IPS. You have way too many rules enabled for a home network. You really should not be running the IPS-Security policy either at home. IPS-Connectivity is more than adequate for home users and will generate next to no false positives. For the truly paranoid home user, maybe bump up to IPS-Balanced. IPS-Security is more for Ft. Knox or the pentagon ... . You will get false positive with IPS-Security enabled.
-
(Nothing taken negative. I welcome contrarian views. - so now that's out of the way..)
Multiple topics here:
I'm actually an IT consultant, installing, configuring and training users on financial software for firms larger than I'd like to mention here. Most connections are through... wait for it... VPN connections. With that said, I still want to harden my local environment the best I can. Nightmare scenario would be a virus I get through casual browsing which somehow gets passed on to their corporate environment. (1st thing you'd say is separate the hardware I use to connect to these clients. Reality isn't always so cut and dry)
Regardless of my clients and my career, I know too much about these vulnerabilities to not worry about my own data. HOWEVER, I agree with you to a point. I'm trying to find a balance. I may not need all these rules, but some may be helpful. I sure wish I had some good documentation to figure out which is which. I do this for a living with other software. In those cases, I have resources available. (Manuals, Knowledge Bases)
Now, the VPN topic could go own for ages. We have two types of VPNs. A 'tunnel' to connect a remote machine to a hardened corporate network. Then, you have personal VPNs so your ISP, Google, Microsoft, AT&T, Apple, and everyone else don't have a never-ending history of your every move for all time, which can be used against you in any legal matter for the rest of your life. Am I exaggerating? Maybe. Do I want to take the chance? No. I tell you what - take a vote. See how many others prefer to use a VPN like me vs the option of 'why bother'.
Avoiding that endless conversation, one thing you said that caught my interest:
"Snort alerts on sensitive data". How did you identify those? How can * I * identify those?
I feel like there is some Snort rule/alert catalog that I've missed somewhere.
Thanks for the feedback.
-
Extension on my last post:
Do you feel 'Emerging Threat (ET) Rules are over-kill for non-corporate systems? Or, are you just saying I should use a 'less strict' IPS Policy rule set?
"you need to do some research on IT security and tuning an IDS/IPS." - OK, so where do you think is the best place to start? The Snort manual seems to be nothing more than a code book. Are there better documents available for people that don't want to make a profession out of interpreting these rules? (or is that the idea? ;) ;).
I simply want to be able to:
- Get an alert
- Identify what triggered it (app, protocol, port, what ever)
- Have some idea if it's a threat
- Have some idea how much of a risk it is to disable the rule
This all doesn't have to be black and white. Just something better than programming code. I get enough of that during the day.
-
@talaverde
It's true the documentation for all the Snort and Emerging Threats rules is very sparse. Rarely can you find a complete description of what a rule is looking for and how it is looking for it. If you become a rule syntax guru, you can discern generally what payload content a particular rule is looking for within a packet, but that may not be much help in determining "why" it is looking for that content. It gets even murkier with Snort's shared object rules (those rules files with "so" in the extension). Those are actually precompiled (source code written in C) binary modules that Snort loads to assist with content inspection. The vast majority of those rules are treated as proprietary info by the authors, so the content matching information is never provided. Truly good IDS/IPS admins are rare and usually well paid. It takes knowledge of the rule syntax and then of course very specific knowledge of the various threats to decipher alerts and make the call of "real alert" or "false positive". Or stated another way, you just about need to be one of those black-hat guys who actually writes the trojans to also be a good IDS/IPS admin who can parse the alerts and pick the real ones from the false positives. You need to understand how the threats work and how they spread in order to write detection rules for them. That same knowledge is needed when poring through alerts to decide if they represent a real threat or not. SIEM tools can help here by correlating lots of data and organizing it by timeline to help important things stand out. A popular big-dollar tool for this is ArcSight, but there are some open-source alternatives.I fully understand using a VPN for remote corporate work. The company I worked for before retirement used a Citrix remote desktop system via a VPN with two-factor authentication. The only thing that worked across the connection was keystrokes and the display. All clipboard activity (copy and paste) was blocked, as was any file access on my local machine. My PC became essentially a dumb TTY terminal when connected to the corporate network.
So I support VPNs for point-to-point networking transport across the Internet and for secure remote access. I'm just not a fan of using one for general browsing. One reason I say that is I see a fair number of issues folks post about here on the forums when using (or trying to use) VPNs. VPNs for secure access is one thing, but using VPNs for personal privacy is treated by many Internet content providers as quasi-nefarious. Netflix is a case in point: they ban all the VPN provider IP ranges because folks were using VPNs to circumvent Netflix's viewing restrictions. I don't mean everyone using a VPN for privacy is a potential nefarious character, but there is some amount of "guilt-by-association" in the eyes of many content providers the same as there is with TOR. VPNs also inevitably add CPU loading to the firewall and increase overall latency as your connection bounces around to get from you to the VPN host and then from the VPN host on to the actual final destination. But as I said in my post, to each his own.
With regards to the Sensitive Data Rules in Snort, you can find a little bit about them in the Snort manual at Snort.org. Basically they are just looking for patterns of numbers or letters (in the case of email addresses). So think of them as a fancy use of regular expression searches for data patterns typical of things like credit card numbers, phone numbers, Social Security numbers and email addresses. That's mostly what the rules are looking for. The main purpose of using them would be in a situation where you are concerned about the possible exfiltration of such data from your network. However, today everything pretty much goes over HTTPS/SSL in one form or the other, so these rules are becoming less effective unless you have MITM proxy. I mean only a truly dumb cyber criminal would steal your credit card info database and then send it out of your corporate network as cleartext using HTTP or FTP ... .
Finally, here is how I use the Emerging Threats rules. First off, there are two versions: the very expensive paid ET-Pro version which has quite comprehensive coverage of current threats; and the free ET-Open version which has much less coverage of current threats. I use the ET-Open rules, but only six categories total. I use emerging-botcc, emerging-malware, emerging-mobile_malware and emerging-trojan rules on my LAN along with Snort's IPS Policy-Balanced (I ran the IPS Policy-Connectivity for several years, though). On my WAN, I use two of the ET-Open "bad IP actors" rules, but only so I see regular alerts from Snort. I like to have those to assist with my package coding tasks. I don't use those WAN rules for any security reason. They are solely to generate alerts I can use when working on the PHP source code for the Snort package. Those two WAN rules are emerging-ciarmy and emerging-dshield. I get several alerts per minute from those rules, but they are alerting on stuff my firewall drops anyway. I get maybe one or two alerts per month on my LAN, and those are usually false positives. But that is expected. Just me and my wife here browsing.
-
Great stuff. All very interesting. (Though I'm surprised we haven't had any other users join the thread. This feels like we're having an email conversation ;)
I was really hoping there was some resource out there (e-book, website) that would help me (slowly) begin to interpret these alerts and decide which were false or not. My training is currently focused on T-SQL, PowerShell, C# and dotNet, I don't think I have the bandwidth to learn IP translation (in regards to Snort). Maybe next year.
It seems that I'm completely blind when it comes to dividing false alerts to true threats, and there is no easy way to change that. So, I think I might put more thought into getting Barnyard2 up and running. Maybe if I start logging enough data over an extended period of time, I could begin to analyze that data. If understanding the language isn't an option, maybe focusing on the patterns will tell me more. It may tell me nothing in respects to this conversation, but at least I'll learn a new skill (using Barnyard2).
Thanks for all the great information.
(My anticipation for Snort3 coming to pfSense is even GREATER with this new information. I can't wait !! :)
-
@talaverde
The best place to start learning how to interpret IDS alerts is to do a little studying on the art of the "darkside". By that I mean learning about how things like buffer overflows work and how they are utilized to escalate privilege on computer systems. If you have a programming background, that really helps in understanding the underlying mechanisms of such compromise techniques.Next you want to have a decent handle on how TCP/IP networking really works down at the packet level. There are lots of tutorials about this on the web. The SANS Institute web site has a Reading Room that is an excellent source of this information as well as information on how popular exploits work.
Armed with this information, it becomes a little easier to sort through the alerts. But to really dive in and truly dissect a Snort or Suricata alert, you will need the captured raw packets and a tool to analyze them. This lets you look into the packet payloads to see if the byte patterns present match up with known exploits. But I would say 6 or 7 times out of 10 you will see just false positives. A lot of false positives come from tacky programming by web folks (not strictly following the RFCs with regards to session setup and teardown, for example). Many rules, especially the Snort HTTP_INSPECT rules, are examining web traffic against the RFC standards. Any little deviation from the standards can generate an alert, but usually the deviation is not harmful and thus the alert is considered a false positive.
The fact we have poor documentation of the rules is our own fault as computer folk. You say you program in a few languages, so you will likely identify with this rhetorical question. What's more fun: (1) documenting code you've already written; or (2) writing new code? For me, it's certainly writing new code! I think the same applies to the rule authors. They know what their rule is or was looking for when they wrote it, but they don't have time to document it for someone else to reference in the future because they are off right now to write another rule for that next big exploit... .
-