IDS/IPS and antivirus on 2.1 release
-
Just because some functionality is well integrated, tested and part of the distribution doesn't mean you have to use it and enable it.
There's plenty of functionality in pfSense that I don't use, and a bunch of packages that sort of bite each other in unpredictable way, because there are no "standard" packages and the various interactions are not part of the testing.There may be other valid reasons not to include package <x>as part of the distribution, but the fact that n% of users won't use it or don't need it, isn't one of them.</x>
Actually, it is a perfectly valid reason. We recently removed OLSRD and RIP (routed) from base for that very reason. They were better as packages, they were no longer used by enough people to justify keeping them in the base system.
Quite true, but the integration needs to be better. e.g. naming conventions: the base system names the menus usually BY FUNCTION, while packages tend to name the menues BY PROJECT NAME, which frankly is annoying.
That's no reason to put them in base. Just fix/change the menu names. But that's a Bikeshed discussion for another thread. Naming by function breaks the moment you have two packages that have the same function but different actual names, and you can't tell them apart.
And again, if a package doesn't work like you want, submit patches or convince the maintainer to fix/make changes. That does not have any relationship to it being in base.
Similarly, the packages as we have them now, do not signal in any way what is mutually exclusive, e.g. Dansguardian does not exclude HAVP and/or SquidGuard from being installed, which is something that could be solved with meta packages that are sorted by function, such that one can install only one IDS/IPS system, only one content filter, etc.
It could use some "conflicts" type checking but the examples you gave aren't always really conflicts. I'm sure someone has a valid reason for combining those in some weird way, and if we stopped them from being installed, something else would break.
It's probably the majority of users that have one or more packages installed, but unfortunately, due to a lack of conventions and/or a system of enforcing these conventions, the fit and finish of the entire installation quickly takes a nose dive once a few packages are installed, which makes it feel inferior to other products, just because things get inconsistent, not because things really are inferior in substance.
If you have some examples of this with non-beta/non-development packages, feel free to cite actual examples and not generalities that aren't really true. Lots of us run many packages with no problems. Certain specific packages may have issues, but again, that's a problem with the individual package's code that isn't going to magically get fixed by bringing it into the base system.
If you want better packages, then help the maintainers make them better, submit patches to make them better, or fund the maintainers to help them get better.
-
Actually, it is a perfectly valid reason. We recently removed OLSRD and RIP (routed) from base for that very reason. They were better as packages, they were no longer used by enough people to justify keeping them in the base system.
The biggest downside to me using a package as opposed to something built in by default is what happens when you upgrade a system. There is extra downtime while the package gets re-installed. With CARP enabled for HA and you upgrade your primary server it will take over after it reboots but before packages are done upgrading/reinstalled. This is one of the reasons I prefer not to use haproxy. Snort, openvpn client export, etc and non-critical packages like that are acceptable to come later but important routing and loadbalancing packages are critical functions.
A way to help with that scenario though would be if there was an easy way to manually disable CARP on the primary that would survive reboots. Better yet would be for the firewall to know when it is done with upgrades and just delay CARP failover back to primary automatically. I would actually prefer the manual method though to make sure that all the packages get upgraded properly before putting load on it. The automatic method though would just be a nice thing for people who don't want to worry about it but where hosted webserver,etc uptime is still very critical to them.
-
Actually, it is a perfectly valid reason. We recently removed OLSRD and RIP (routed) from base for that very reason. They were better as packages, they were no longer used by enough people to justify keeping them in the base system.
The biggest downside to me using a package as opposed to something built in by default is what happens when you upgrade a system. There is extra downtime while the package gets re-installed. With CARP enabled for HA and you upgrade your primary server it will take over after it reboots but before packages are done upgrading/reinstalled. This is one of the reasons I prefer not to use haproxy. Snort, openvpn client export, etc and non-critical packages like that are acceptable to come later but important routing and loadbalancing packages are critical functions.
A way to help with that scenario though would be if there was an easy way to manually disable CARP on the primary that would survive reboots. Better yet would be for the firewall to know when it is done with upgrades and just delay CARP failover back to primary automatically. I would actually prefer the manual method though to make sure that all the packages get upgraded properly before putting load on it. The automatic method though would just be a nice thing for people who don't want to worry about it but where hosted webserver,etc uptime is still very critical to them.
Which is still, technically, not a package system issue, but a general CARP issue.
Firmware upgrades, for most users, happen once or twice a year, or less often. A few minutes of downtime during a scheduled maintenance window for a firmware upgrade is not a huge issue for most people.
Yes, if you're tracking snapshots it's more often and more disruptive, but the fact of the matter is that is not how most people work.
Still, that's an issue to take up with the CARP procedures, it's not specific to things being a package really.
-
I agree with your reasoning. Upgrades don't happen often on production sites. When they happen for me I fail over to a backup site so it isn't such a big deal. I was somehow overlooking that seemingly obvious point.
I also failed to mention that main reason I prefer the built-in load balancer is because it doesn't drop connections during firewall fail-over where haproxy does. That again shouldn't happen often.
-
I also failed to mention that main reason I prefer the built-in load balancer is because it doesn't drop connections during firewall fail-over where haproxy does. That again shouldn't happen often.
The built-in balancer works because relayd works like NAT rules, so it fails over with the normal state table entries.
Until HAproxy grows a way to sync its internal connection states in a cluster-like fashion, there isn't anything we can do for that. There's a separate thread for that somewhere, I seem to recall posting about it in the last couple weeks. Usually web sessions are fleeting enough that it doesn't matter a ton that the states are gone.
-
Well, there's still the killer argument: unneccessary complexity.
More complex systems are more likely to break. I guess we don't need to discuss this.
Is IDS neccessary? For most of us, no.
Of course, everyone has his favouriet package "which should be in the base system". Yup, if it's in the base system it gives the user the illusion of real solid support. However, when thinking again about illusions and additonal complexity, I'd rather prefer the modular approach, where I can simply remove addons which begin to act up.
-
"Well, there's still the killer argument: unneccessary complexity." ;D
-
How about a system like the WP Plugin Repository?
Bring in some sort of order to the chaos of broken, semi-functional packages that gives Pfsense a bad name?
-
Well, there's still the killer argument: unneccessary complexity.
More complex systems are more likely to break. I guess we don't need to discuss this.
Is IDS neccessary? For most of us, no.
It all depends how you use pfSense. e.g. if I tried to program the firewall, it would be a fairly insane effort for a little home network, because there's no starting point, and the number and kinds of software people run these days isn't anything like the old days where you'd open up the mail, DNS and SSH ports, and shut everything else.
We have things like dropbox, drobo, back-to-my-mac, iCloud, email, DNS, VoIP, Skype, P2P, Tor, RDP, VNC, Apple Remote Desktop, remote server admin tools, push services, photo streams, proprietary protocols of ever more license demons, web apps, etc.
I'd end up with a mess of rules, and probably still a bunch of things that don't work quite well.Or I can use SNORT's blocking features, which simply selectively block hosts for a few hours if they do things that look suspicious, and features like the ssh lock-out feature that block a host after a certain number of failed login attempts, and pfBlocker with some list subscription of known bad hosts in combination with something like Apple's application firewall, which isn't ports/protocol based, but decides which apps can accept incoming network traffic.
As things get more and more internet driven, old school firewall programming gets more and more difficult, and smart, behavior driven thread detection/blocking more important.
As such I think IDS/IPS along with crowd-sourced blocking lists are the future of firewalls, and while rule based firewalls work well for systems that are essentially shut out of the internet except for a very few carefully poked holes, that model doesn't work too well for the way most people use the net these days and in the future.
-
Fine, if you want SNORT, install it as a package.
As such I think IDS/IPS along with crowd-sourced blocking lists are the future of firewalls
Nope, definitely not.
In case you haven't noticed yet: blocking lists are regularily manipulated to put companies out of business. You are probably not a prime target, but others definitely are.
However, if you're not a target, who bother? And if you are, why open up a new attack vector to get DoSed?
-
I think having pfsense start up with nothing, exactly as it does now, is the best way to go. It ensures maximum compatibility across a broad range of platforms. I think if you want a package, add a package is the correct mentality. SNORT especially, is something that some just MUST have and others should avoid like the plague.
-
Let's take an objective look at this:
Pros to having things as a package:
- Able to update more frequently independent of a firmware release
- Allows the package development to proceed with more freedom
- Faster response to upstream changes for things like snort versions, rule formats, AV databases, CVEs, etc.
- Keeps the main system simpler, smaller, etc. (less complexity)
- Less bloat in the main system for things many users do not need (snort users are in the minority)
- Most packages works fine as-is for most users
- Smaller download sizes, less bandwidth used for hosting pfSense downloads and mirrors
- Lots of other good things discussed here and elsewhere about the package system.
Cons for having things as a package
- Making those 4 clicks to install the package can be tiresome and tough work the one time you have to do it. I'm surprised people haven't broken their fingers.
- Some people want it pre-installed
- Somehow magically better code and support because it's in the base system.
Things that have little or no bearing on whether or not it's a package:
- The menu name (snort vs IDS/IPS vs some other confusing name)
- How the package behaves during a firmware upgrade that happens for most people 1-2 times per year, if that.
- How a package interacts with other features such as CARP
So, in other words:
- Some valid complaints are not about the package itself, but about how packages in general are handled. This is a base system issue, and you can't "fix" that by moving a package into the base system. You're treating the symptom not curing the disease.
- Some valid complaints are about the code of the packages themselves. This will not be solved by moving it to the base system. Someone still has to fix it. Bringing it into base would only happen if a package was already in good shape, or if someone funding getting it into good shape and other factors deemed it worthy. The same fixes would happen if it were kept a package and someone fixed the package. Hence, broken code is broken no matter where it is, and good code is good no matter where it is.
In conclusion:
- Help the package maintainers fix packages by contributing code, funds, etc.
- Help identify and fix issues with package handling in the base system (patches are always welcome, doubly so if they actually fix things in an elegant and useful way)
- Keep using packages and make them even more awesome than they already are.
If anything arguments could be made for taking even more things out of the base system, though what we have now strikes a good default balance.
If you want a real debate about whether a package should be a package, or in the base system, take a closer look at a small, simple package such as sudo or blinkled. It's a much better debate than this one for large/complex add-ons. ;D
-
I think modularizing things into packages is per se a good thing, but having started with pfSense as a total noob not all that long time ago, I remember the confusions the packages create.
And that boils largely down to competing/conflicting packages that are not recognizable as such, and the vastly differing quality and level of support for various packages.
If there were some sort of curation process, e.g. official packages (things that are maintained by the pfSense team, and are just taken out of the base system for a more modular setup), qualified packages (very commonly used things that for license or other reasons are not part of the base system or official packages, but are tested and have a seal of approval that they work and are consistent with the UI and naming conventions), and user contributed packages (for which no assertion is made in regards to anything), that would be a step forward, particularly if one could restrict the list of packages shown in the "available packages" section to a certain quality level, e.g. only official packages, or qualified and official packages, or everything.
As long as there's a better way of handling package conflicts, and as long as package isn't synonym with less support, less quality, less consistency, etc. I'm all for modularizing things. However right now, I consistently get the feel that anything that's in a package is "second class citizen", and that's pretty much the main reason that I'd love to see everything that I consider important part of the base system.
-
You do see that Alpha, Beta, RC, Stable tag in the packages right? Are those tags confusing?
Also, in every disro of Unix like OS, packages are handled like that. I wouldn't want the "Microsoft" one stop shop o-code method anyway.
Sounds like you just don't like the way opensource projects work in general.
There are some pretty sweet commercial appliances you can buy for an arm and a leg.
Of course, no one can see the code most of the time. No idea if its protecting you or rooting you.
Actually, I have a pretty good idea… Huawei makes some pretty nice stuff. Install it liberally. -
You do see that Alpha, Beta, RC, Stable tag in the packages right? Are those tags confusing?
No, but they turn out to be meaningless. These tags make a statement about the stability of the package, not about the quality of the integration, whether or not they are a modularized part of the official project, or some more or less supported/approved add-on
Also, in every disro of Unix like OS, packages are handled like that. I wouldn't want the "Microsoft" one stop shop o-code method anyway.
This wouldn't be the Microsoft model, more along the lines the Apple AppStore model, except more liberal: for a package to be either official (something that is done by the core project team, but put into a package to make things more modular) or qualified (third party contribution, but tested), it would have to fulfill certain standards e.g. the way their interface works, if it can use aliases properly, what naming conventions are used, etc.
If it doesn't do these things, it would be marked as unqualified user contributed package. -
If you want a real debate about whether a package should be a package, or in the base system, take a closer look at a small, simple package such as sudo or blinkled. It's a much better debate than this one for large/complex add-ons. ;D
- vote for iftop in base ;)
Let the debate begin!
You do see that Alpha, Beta, RC, Stable tag in the packages right? Are those tags confusing?
Actually they are confusing. Take a look at haproxy, haproxy-full, haproxy-devel. They are all marked as release. Something about devel being release status seems a big odd :). I brought that up in another discussion and it was mentioned the tags are not maintained very well. I am not really trying to complain about it though. After a post on the forums I got it figured out. I am just trying to state that the tags are not very accurate it seems for some packages.
-
The RC/Beta/etc tags are largely meaningless. I'd love to fix that, but that means either personally testing each one or getting reliable feedback about the actual quality of a package.
Some, like sudo, are simple and probably could be release quality, but they are also very new and not many have given feedback to say it works or has issues. So to me it may be release, but in general it's "beta" since it's really not quite well-tested and reviewed.
- vote for iftop in base ;)
Let the debate begin!
My heart says +1 because I love iftop but my brain says no because it doesn't really fit in base. :-)
-
My experience is that Stable means it will probably work, beta means alpha and alpha means don't make any freakin plans. You are going to be pulling your hair out for the next several hours if not days.
-
At least they don't mean the same as Google's release and beta which means it will generally work but they might pull the plug at any time :).
-
LESS IS MORE - *Nix philosophy No. 1
I am reiterating that it is always the best to separte base OS from the extra goodies (packages). It keeps simple, both development and system adminstration.
The demand from some of the users who might come from the GUI culture that a monolithic (including packages) pfSense shall deviate the focus from developers. Everyone's need is possible with the current package system (though I am finding pbi a bit exhaustive) on top of base OS. Else I suggest to play a bit on command line and add desired packages in jails to separate the OS from the base OS.
Alternatively, they can compile whatever they need from the pfSense source directly as per their own requirments than asking for the developers. I preferred this way maybe because I am comfortable with the command line since the last century. The exhaustive menus and options on a single page of GUI exhaust me, yet I am for GUIs for those in my team who does not want to follow the sharp learning curve of command line tools, mostly coming from windows and mac cultures (now I can add ubuntu users, too in this group).
DO NOT UPGRADE UNLESS IT BREAKS - *nix Philosophy No. 2
As far as the broken packages with frequent updates of pfSense base are concerned (which is preferred by most of the non-*nix users these days), I would like to remind *nix philosophy that do not upgrade if it does not break. However for snapshot releases you may need to do so, but before check the changelog. FYI, I am still on July 18 snapshot because it is working for my own needs. ;-)
SEPARATION OF PACKAGES - A Suggestion
pfSense 2.1 is alright as it is now, yet I suggest the developers of pfSense to separate the active packages from not-maintained packages to avoid confusion that it is not the problem with pfSense developers' part when the package breaks.Thanks!
