Blog suricata-vs-snort - Snort 3.0?
-
@michmoor said in Blog suricata-vs-snort - Snort 3.0?:
That said, the strength is always in the community of course. Although the warm and fuzzy part of me would always want a paid dev (cough cough bmeeks) on staff to always be around.
I'm happy not being a paid dev, already put in my 36 years at the corporate grindstone . Just doing the package maintainer duties now to keep my mind engaged. Been officially fully retired since 2014.
So at some point down the road it will be time for a new "whipper-snapper" to take over the IDS/IPS package duties. Perhaps that could start by someone developing the Snort3 package ???
-
@michmoor FYI Netgate has a package list here and maintains several.
@bmeeks Being a programmer I definitely appreciate your time involved. Probably, most non-coders do not.
Just for discussion purposes is Snort 3 a situation where it needs its own package?
Hmm, can packages be renamed? Having "pfblockerNG-devel" as the current recommended package probably means they can't? Using a package name like "snort3" could be confusing when it is using Snort version 6 down the road. "snort-gen2"? "snort-ng"?
Not even sure how to get people to switch. One last package update that adds a banner saying to use the new package, and it doesn't migrate?
I emailed our Partner contact. Maybe they can have someone update the blog.
-
@bmeeks I have like...20 more years of the grind. Writing this with a heavy sigh.
That said, Would love to use the Snort3 package in an official way. One day it will come. Im sure of it. -
@steveits said in Blog suricata-vs-snort - Snort 3.0?:
@bmeeks Being a programmer I definitely appreciate your time involved. Probably, most non-coders do not.
Kind of strange you say that. Im a IT admin and sometimes i do code just not in PHP etc. But i appreciate every code that is creating and maintaining these.
@bmeeks snort it works yes. but when there is like snort3 which is faster beter etc.
Sadly i dont have enough time to learn alle these things like PHP and how to create a package etc that would bee snort3.
Creating something is 1 thing but also maintaining and fixing bugs and things is a other thing.
-
@steveits said in Blog suricata-vs-snort - Snort 3.0?:
@michmoor FYI Netgate has a package list here and maintains several.
Just for discussion purposes is Snort 3 a situation where it needs its own package?
My personal opinion is "yes" because it is substantially different technology under the hood. All the configuration is done via Lua files, and it is multithreaded where Snort 2.9.x is not. The mismatch between the current Snort binary version and the GUI package version numbering is my fault from a long time ago. With Suricata I have been striving to keep the major GUI version equal to the underlying binary. Hence 6.0.8_x currently in pfSense development snapshots and 6.0.4_x in release. These major version numbers in the GUI package match the underlying version of the Suricata binary.
With
pkg
, it's the version number that matters in addition to the name. The higher versions are considered "newer", so once you release a 3.x something package, you can't go back to 2.x something becausepkg
will always think that is an older version. Remember Snort was my first package maintainer duties, and I did not think far enough in advance when I was making lots of updates and creating new versions. I have tried to not repeat that mistake with Suricata.Hmm, can packages be renamed? Having "pfblockerNG-devel" as the current recommended package probably means they can't? Using a package name like "snort3" could be confusing when it is using Snort version 6 down the road. "snort-gen2"? "snort-ng"?
I will not judge the use of "devel" in the name of the most current release of pfBlockerNG. It is something I personally would not encourage, though. I think it is very confusing to users conditioned by the way most other software works. You normally don't expect a "devel" version of something to be the one that is really the most current and suggested for production use. I think it would be very bad to create a Snort-NG or Snort-devel package and have it alongside the release package in the same repo. To the best of my knowledge,
pkg
cannot rename an existing package. You can create totally new ones, but you can't rename existing ones. You can, however, remove packages or mark them as deprecated. That is the proper way to do things in my view.I don't think a Snort3 package would be confusing. It would actually immediately identify the underlying binary. At the time there was never a consideration of Snort3, and I was a rookie anyway, but in hindsight it might have been useful to name the current Snort package on pfSense "Snort2-xxx" instead of simply "Snort-xxx". Then whenever Snort4 comes along, you do the same.
Not even sure how to get people to switch. One last package update that adds a banner saying to use the new package, and it doesn't migrate?
You mostly get them to switch by removing the old package. That would not make
pkg
remove it from their firewall, but there would never be another update available for that version. That's the way pfSense handles packages today when a new release of pfSense comes out. Thepkg
repo for the old pfSense version is frozen and never gets any updates. Similarly, if a completely new version of a package emerged (say Snort3 is out and you wanted to deprecate Snort2) you could just freeze updates for the older package for a time, then remove it from the repo so users could not even install it anymore. Of course you would want to communicate this information. And probably somebody would complain about it, but such is life in the world of software -- time marches on whether it is convenient for you personally or not . -
@steveits said in Blog suricata-vs-snort - Snort 3.0?:
FYI Netgate has a package list here and maintains several.
Thanks for this. What would it take for packages to be moved to the 'Maintained by Netgate' status?
I noticed some are just supported by install or custom config which isnt the same as maintained. -
@musicwizard said in Blog suricata-vs-snort - Snort 3.0?:
Kind of strange you say that. Im a IT admin
I probably phrased what I said poorly. It wasn't meant to be an insult, just that programming often takes a lot of time.
@michmoor said in Blog suricata-vs-snort - Snort 3.0?:
What would it take for packages to be moved to the 'Maintained by Netgate' status
No idea. They do help on pfBlocker occasionally, I think I've seen, not sure about others. One might think a popular package without a maintainer would be to Netgate's advantage to keep running.
-
@steveits said in Blog suricata-vs-snort - Snort 3.0?:
@michmoor said in Blog suricata-vs-snort - Snort 3.0?:
What would it take for packages to be moved to the 'Maintained by Netgate' status
No idea. They do help on pfBlocker occasionally, I think I've seen, not sure about others. One might think a popular package without a maintainer would be to Netgate's advantage to keep running.
Netgate is probably not eager to step in and take over a package mainly because they already have plenty of work on their plates maintaining TNSR, CE and now Plus core pfSense software.
They have stepped up and made quick fixes to popular packages when some pretty significant show-stopper issue surfaces -- as in the package won't install or run at all, or has some other major defect. I think this has mostly happened when the normal package maintainer is not immediately available.
And there are some packages that were actually created by Netgate staff. Service Watchdog is one that I recall, and I think Acme is another. Those packages were created to fill specific niches. OpenVPN Client Export is another such package.
-
@steveits said in Blog suricata-vs-snort - Snort 3.0?:
One might think a popular package without a maintainer would be to Netgate's advantage to keep running.
Agreed. Better to have it written on 'paper' for it to be official.
-
@bmeeks said in Blog suricata-vs-snort - Snort 3.0?:
Netgate is probably not eager to step in and take over a package mainly because they already have plenty of work on their plates maintaining TNSR, CE and now Plus core pfSense software.
This is one of those areas where I understand a bit. I agree with what you are saying completely. With limited time and resources its all about where to throw the bodies to fix or improve core features.
Users want more apps on top of what's available and i get it. At the end of the day, the devs and maintainers are doing the best they can. -
@michmoor said in Blog suricata-vs-snort - Snort 3.0?:
@bmeeks said in Blog suricata-vs-snort - Snort 3.0?:
Netgate is probably not eager to step in and take over a package mainly because they already have plenty of work on their plates maintaining TNSR, CE and now Plus core pfSense software.
This is one of those areas where I understand a bit. I agree with what you are saying completely. With limited time and resources its all about where to throw the bodies to fix or improve core features.
Users want more apps on top of what's available and i get it. At the end of the day, the devs and maintainers are doing the best they can.And certain features can actually work better not as add-on packages but as core constituents of the system. That's how Suricata is handled on OPNsense. It is always there as a core component, and the user has the option of enabling it or not. But the code is a part of the base system install. It's similar to how the DNS Resolver and DNS Forwarder are handled today in pfSense.
So I would expect that if the features of a particular package were very popular and almost all users installed that package, and it became associated with "pfSense" itself, the functionality of the package would be absorbed into the base system.
-
@bmeeks said in Blog suricata-vs-snort - Snort 3.0?:
So I would expect that if the features of a particular package were very popular and almost all users installed that package, and it became associated with "pfSense" itself, the functionality of the package would be absorbed into the base system.
How difficult is it getting a package intergrated? Id imagine Suricata and pfblocker are the most important and popular package to date.
-
@michmoor said in Blog suricata-vs-snort - Snort 3.0?:
@bmeeks said in Blog suricata-vs-snort - Snort 3.0?:
So I would expect that if the features of a particular package were very popular and almost all users installed that package, and it became associated with "pfSense" itself, the functionality of the package would be absorbed into the base system.
How difficult is it getting a package intergrated? Id imagine Suricata and pfblocker are the most important and popular package to date.
It's not a terribly difficult task to actually integrate, but testing everything to be sure hidden bugs did not break something huge would be time consuming.
While Suricata and Snort both seem popular now, to be honest I really don't understand why that is so on pfSense. The vast majority of network traffic today is encrypted. That means without MITM the IDS/IPS is almost blind. It can see source and destination IP addresses and ports, and it can see a little bit of what's going on via the SNI header when that is present, but that's about all it can see for encrypted traffic. There is currently no built-in pathway for the IDS/IPS to see unencrypted data even when MITM is in place. Creating such a pathway would require some major reworking of the internal network plumbing of FreeBSD. If I were Netgate, I would not really consider the effort involved in bringing an IDS/IPS into the core system worth it over the long term. To make it really functional, you would also need to rework the network stack AND offer a ready-to-go MITM setup.
The features of pfBlockerNG-devel would be a more natural fit.
I personally believe that most protections are going to have to move more to the endpoints and be used less (and eventually almost not all) on the network perimeter.
-
@bmeeks said in Blog suricata-vs-snort - Snort 3.0?:
There is currently no built-in pathway for the IDS/IPS to see unencrypted data even when MITM is in place. Creating such a pathway would require some major reworking of the internal network plumbing of FreeBSD.
I think we discussed this on a previous thread as well.
The integration with FreeBSD is the issue. If you take the landscape of how other firewalls are doing IPS(Threat Prevention), they break into TLS and then inspect traffic. Those other firewalls have a very customizable OS so thats likely why such an integration is possible.
The function is separated right now within PF or OPN and so the integration perhaps with Squid and Suricata would need to happen within a combined binary perhaps? Please correct me if I'm wrong with my analysis. -
@michmoor said in Blog suricata-vs-snort - Snort 3.0?:
@bmeeks said in Blog suricata-vs-snort - Snort 3.0?:
There is currently no built-in pathway for the IDS/IPS to see unencrypted data even when MITM is in place. Creating such a pathway would require some major reworking of the internal network plumbing of FreeBSD.
I think we discussed this on a previous thread as well.
The integration with FreeBSD is the issue. If you take the landscape of how other firewalls are doing IPS(Threat Prevention), they break into TLS and then inspect traffic. Those other firewalls have a very customizable OS so thats likely why such an integration is possible.
The function is separated right now within PF or OPN and so the integration perhaps with Squid and Suricata would need to happen within a combined binary perhaps? Please correct me if I'm wrong with my analysis.No, it's a bit more complicated than just a simple change to a couple of binaries. It has to do with the physical path packets travel from NIC to kernel and back. This gets into the User Space versus Kernel Space problem, too. The current IDS/IPS packages run in User Space. For Inline IPS Mode, they use the special netmap kernel device. What netmap does is set up two common memory buffers that both the hardware NIC driver, Suricata (or Snort) running in User Space, and the kernel network stack share. The NIC copies incoming data to the common memory area designated for the NIC and then tells the netmap device the data is waiting there. The netmap device tells Suricata that data is present in the shared memory area (in the netmap rings) from the NIC driver. Suricata reads and examines the data and compares it against the rule signatures. For traffic that does not trigger a DROP rule, Suricata copies the packet data from the NIC common data area to the separate common data area it shares with the kernel stack (called the Host Stack in netmap). For traffic that triggers a DROP rule, the packet data is simply not copied to the kernel shared memory area. Therefore we call the packet "dropped" because it was not transferred from the NIC driver memory buffer over to the kernel network stack memory buffer. Netmap, by creating and exposing those two shared memory areas, allows a User Space application to process network packets as fast as possible (almost line rate if the netmap pipe is between two physical interfaces as opposed to one physical interface and the Host Stack).
The problem here is that in FreeBSD all this netmap stuff lives immediately after the NIC driver (when talking about inbound traffic). At this point all of the payload data is still encrypted as it is passed to the kernel. It will stay encrypted until the kernel stack delivers it to the MITM application (typically a proxy server app running also in User Space). The reverse is also true. Data outbound from the system is already encrypted by the time it reaches the last step of the process where netmap is used to copy it from the kernel space buffer over to the NIC driver buffer (via Suricata again where Suricata inspects and does the actual copy of data).
To make this work you will need to rewrite how some of the kernel network stack works so that your IDS/IPS sees inbound data AFTER the MITM has decrypted it, and it sees outbound data BEFORE the MITM has encrypted it. That's a tall order unless you are creating a proprietary MITM and IDS/IPS configuration by radically altering the operating system's network stack and how data packets travel around.
There are other ways perhaps to do this via sockets where apps swap data with each other, but that would be terribly slow. You could also perhaps do some creative routing with several physical interfaces to send traffic to and from a proxy and Suricata, but it would use several physical interfaces and be difficult to set up.
Not saying this is impossible - just that it's a tall order. The few big boy firewall vendors that offer this have a highly customized OS on their box. Creation and maintenance of that is one of the reasons their cost is so astronomical, too.
-
@bmeeks This is extremely insightful here. Thanks for the added detail that im glad i know about now.
So the way i see it, the IPS is only 'useless' on pf from the standpoint of lack of integration with any MITM process. In a perfect world with unlimited resources if there can be some integration then the IPS component would be of more value.
As it stands, its pretty useless (no disrespect to you of course), and this is more in line with what you have been saying for quite some time about its usefulness. -
@michmoor said in Blog suricata-vs-snort - Snort 3.0?:
@bmeeks This is extremely insightful here. Thanks for the added detail that im glad i know about now.
So the way i see it, the IPS is only 'useless' on pf from the standpoint of lack of integration with any MITM process. In a perfect world with unlimited resources if there can be some integration then the IPS component would be of more value.
As it stands, its pretty useless (no disrespect to you of course), and this is more in line with what you have been saying for quite some time about its usefulness.Correct. But it's not just
pf
. The same is true with many other operating system configurations.My point in my other postings about the "usefulness" of IPS is that on the perimeter it is becoming less and less effective due to the encryption. Thus having it on the firewall inspecting traffic flowing between hosts is not really doing a whole lot unless the payload data is cleartext.