HEADS UP: Squid3 package is broken with a default config. Will break FS perms
-
Just to comfirm this is indeed the case, and using mtree did restore file rights to a working system again.
I would stay clear of the current version untill we get some dev feedback. And hopefully a fix.
-
I pushed a quick set of safety belts to stop it from causing harm, but it doesn't fix the root cause. Package version 0.3.5.1 will be up shortly.
Whoever is maintaining the package these days will want to track down what is causing it to try to chown an empty/blank value.
-
New version is up now, unstickied topic since the real danger isn't there now.
-
I applaud the dedication of the support team. Thanks guys.
I'll be testing this ASAP. -
Considering how the Squid package seems to be extremely popular, it might make sense to move its development and maintenance in-house. Marcello has done an amazing job of running it (better than I could do, for sure), but pfSense can't be risking its reputation on 3rd-party packages maintained by volunteers. It doesn't look professional, and you don't have the same commitment to testing & stability.
-
Once we reach 2.3 where squid3 will be the only choice it may end up that way. We have already had on-staff developers clean it up a bit from time to time.
We have stayed away from claiming squid3 because it has been quite unstable not because of the core features but because of what people attempt to use it for (icap, AV, SSL interception, reverse proxy, etc) – things we don't really want to "own" for support because they are, by and large, not recommended, stable, or functional in the way people expect.
If a squid3 package existed that only had the "features" of the current squid 2.x package -- mostly a stable forward proxy that can be used with squidGuard -- we'd happily support that.
The problem with bringing packages like that in-house is that once we do, we'll be expected to make every aspect of it work even for things that are broken upstream or are half-implemented, or things that don't work on FreeBSD, and so on.
We'll see what happens when 2.3 gets closer though.
-
Once we reach 2.3 where squid3 will be the only choice it may end up that way. We have already had on-staff developers clean it up a bit from time to time.
We have stayed away from claiming squid3 because it has been quite unstable not because of the core features but because of what people attempt to use it for (icap, AV, SSL interception, reverse proxy, etc) – things we don't really want to "own" for support because they are, by and large, not recommended, stable, or functional in the way people expect.
If a squid3 package existed that only had the "features" of the current squid 2.x package -- mostly a stable forward proxy that can be used with squidGuard -- we'd happily support that.
Hi Jimp,
I understand that situation, but none of that would have happend if the Core Team developed the package from the start. Since you refused to work on it, someone would do it.
Here we come again to the difference between what ESF think we need on pfSense and what people really do with it. There are security concerns and worload to be considered, but it should be clear now, a lot of packages and features are so widely used that you should keep the "under the roof".
The problem with bringing packages like that in-house is that once we do, we'll be expected to make every aspect of it work even for things that are broken upstream or are half-implemented, or things that don't work on FreeBSD, and so on.
We'll see what happens when 2.3 gets closer though.
Well, as I stated above, if you implemented the package from the start, no crazy feature would have been merged into the solution. Moreover, other solution do offer those features in the market…
Clear example? Single Sign On with Squid using NTLM. Multiple times people tried to bring a minimal implementation of Samba to use it only as a "bridge" to a DC, in onder to auth the users. You put that aside under the argument that using pfSense as a DC is not secure... well I AGREE, but we are not using it as a DC, just a member to do the SSO.
So much is changing in 2.3... no better time will come to reconsider some positions and move pfSense forward.
-
In an ideal world, we would develop it internally. The reality is that we don't have the staff, time, and resources to develop every package that people want internally. We also don't have much pressure from our hardware and support customers to support squid3 internally so we have not been in any hurry to take it on. So, that's where community-developed packages fit in.
When we "take on" a package we incur a lot of technical debt and support overhead once we claim it. Any bugs in the original code become our problem that we have to support and correct. We need to have the resources available to handle the load, and in the case of complicated packages, that is a significant burden.
At the moment we're all focused on 2.3 itself, and not so much on the packages. Certainly not looking at adding any features currently. Maybe after 2.3-RELEASE, who knows, but not right this moment. It's definitely not the right time to add more work to the large list we already have in front of us.
-
@jimp: Make a list of what you want to be gone from the current 3.4.x package. Nuking stuff from the code certainly goes faster than cleaning the huge thing up. (At round 4 now, definitely not finished).
When I have a list, I can do a PR for something like squid3-minimal (or -official) or whatever, you can take that with lots less of cruft and buggy code and do whatever you want with it. The current one can stay as squid3-community or whatever other name, otherwise you'll just piss off users by removing features from their package.
Also, cleaning up the PBI-specific garbage alone from there will help tremendously, that package would be for 2.3+ only. Certainly NO PBI. Never again.
The 2.7 thing is dead and needs to be burried.
-
2.3 is right around the corner, it's probably best to let 2.7.x die a slow death along with 2.2.x without putting too much effort into killing it, but it may not be that much effort to replace it.
A minimal squid3 does sound nice for those that don't need the extra bits. I don't have a list to compare at the moment but really one could just look at the 2.7.x GUI and take it from there.
For names, "squid" would be fine for the basic no-frills one and I'd be inclined to call the other one something that implied that is has more features, like squid-extras or squid-plus (ugh), squid-xtreme (double ugh), squid-giant, squid-architeuthis-dux, squid-kraken, etc.
The binaries for both would have to be the same since with pkgng we can't (yet) have multiple packages with different options, but the difference in GUI alone is likely enough to keep people from getting into much trouble with it.
-
otherwise you'll just piss off users by removing features from their package.
We wouldn't do that, as far as I know. When we've had something like that come up in the past the package was split, like was done with HAproxy until it stabilized and then came back into one package.
Also, cleaning up the PBI-specific garbage alone from there will help tremendously, that package would be for 2.3+ only. Certainly NO PBI. Never again.
The 2.3 package files are in our freebsd-ports repository and not in the pfsense-package repository but at the moment there is some syncing going on to keep things updated. I'm not sure at what point those will split off but eventually the freebsd-ports versions will split and then all the older code can be cut out with no worry about backward compatibility. You might want to start a thread in the 2.3 board about that.
-
In an ideal world, we would develop it internally. The reality is that we don't have the staff, time, and resources to develop every package that people want internally. We also don't have much pressure from our hardware and support customers to support squid3 internally so we have not been in any hurry to take it on. So, that's where community-developed packages fit in.
When we "take on" a package we incur a lot of technical debt and support overhead once we claim it. Any bugs in the original code become our problem that we have to support and correct. We need to have the resources available to handle the load, and in the case of complicated packages, that is a significant burden.
At the moment we're all focused on 2.3 itself, and not so much on the packages. Certainly not looking at adding any features currently. Maybe after 2.3-RELEASE, who knows, but not right this moment. It's definitely not the right time to add more work to the large list we already have in front of us.
Yep, as I said above, I understand your reasoning to not adopt packages too often.
Now, remember when we had this same argument over Samba like a year ago? Well, the same argument was made. Here we are with like 30% of the user base using Samba(just a guess based on the posts about it).
You talk about pressure from your "paid support" customers, well, the money comes from them, but do you actually have a poll statistics about which features people want? You know, maybe some people would love if you implement some features and maybe hire your support? I suggest you do a general poll on the community about this.
Now, Marcelloc did a Pull Request on Git for a feature that covers hit count on Firewall Rules. That feature is present on a F. ton of other solutions. A week passed and no one from ESF answered there again. As I follow it almost seems you want to keep some features for "Gold" only, that's the point where things starts to get ugly.
I'm sorry for hijacking this thread, I just think whenever someone from the community really tries to contribute, you shot it down.
-
Here we are with like 30% of the user base using Samba(just a guess based on the posts about it).
From the people we talk to and deal with on a daily basis, the number of people wanting or needing that is likely not even a single digit percentage. There are hundreds of thousands of pfSense installs out there, to claim "30% of the user base" is absurd without data to back that up. I'd say less than a couple dozen people are using it total, but that's also just a guess that has about as much a chance of being correct. A poll, done properly, may not be a bad idea, but you are off base in multiple ways suggesting what you're saying.
I did some searching in our support system (which also includes pre-sales questions) and I found only 3 people who asked about Samba on pfSense and two of those were for file sharing and not SSO. Only one was for AD auth. If I looked for SSO, I got 6 hits ever that were about Squid and AD SSO. Sure there may be a few more on the forum but it's far from a significant portion of the user base. It may be important to you, and important to the few people who seem to yell loudest for it, but it's not a key feature for most people. If it was that important, we'd be seeing a massive number of requests for it, and the demand just is not there.
Now, Marcelloc did a Pull Request on Git for a feature that covers hit count on Firewall Rules. That feature is present on a F. ton of other solutions. A week passed and no one from ESF answered there again. As I follow it almost seems you want to keep some features for "Gold" only, that's the point where things starts to get ugly.
Not sure what you're looking at, but we are working with marcelloc to get that into 2.3 but it has to be implemented differently. We are working on adding it into the pfSense PHP module, as his implementation had some issues. https://github.com/pfsense/pfsense/pull/1901 Has all the discussion. These things take time, and we have limited resources and a large workload. It's being worked on. It won't magically make it in overnight since it wasn't usable as it was submitted.
-
Here we are with like 30% of the user base using Samba(just a guess based on the posts about it).
From the people we talk to and deal with on a daily basis, the number of people wanting or needing that is likely not even a single digit percentage. There are hundreds of thousands of pfSense installs out there, to claim "30% of the user base" is absurd without data to back that up. I'd say less than a couple dozen people are using it total, but that's also just a guess that has about as much a chance of being correct. A poll, done properly, may not be a bad idea, but you are off base in multiple ways suggesting what you're saying.
Well, as you said, your guess is empty as mine in this aspect. I even stated it was my POV. Now… this is one of those sittuations when if you had the feature available, a LOT of people would use it.
Of all the pfSense servers running, a small portion participate on the forums or buy Support subscriptions. An even smaller portion care enough to actually participate, now cut that down even more to people that really care enough about the development process and wants to help.
IT IS HARD to get some good feedback from a dispersed community like that, but a small practical number in this case may have secondary implications.
Let's drop the weapons and bring this down to: You are changing "everything" on 2.3, IMO it's the best time to ask the community(from paid support to idiots like me that wants to help). I mean, not only about SSO, but other features in general.
I did some searching in our support system (which also includes pre-sales questions) and I found only 3 people who asked about Samba on pfSense and two of those were for file sharing and not SSO. Only one was for AD auth. If I looked for SSO, I got 6 hits ever that were about Squid and AD SSO. Sure there may be a few more on the forum but it's far from a significant portion of the user base. It may be important to you, and important to the few people who seem to yell loudest for it, but it's not a key feature for most people. If it was that important, we'd be seeing a massive number of requests for it, and the demand just is not there.
Well, the acronym SSO is not that widely known, specially in the market pfSense targets at. Ask Ingrid about Spiceworks community, most of them don't understand the concept. But take the time to explain it, whatever makes the "User" do less for more, they(we) want.
Create a poll, if possible with translations, make Renato Botelho post the poll on our Facebook page, I'll also ask people to take part in it. Post it on Spiceworks too.
Now, Marcelloc did a Pull Request on Git for a feature that covers hit count on Firewall Rules. That feature is present on a F. ton of other solutions. A week passed and no one from ESF answered there again. As I follow it almost seems you want to keep some features for "Gold" only, that's the point where things starts to get ugly.
Not sure what you're looking at, but we are working with marcelloc to get that into 2.3 but it has to be implemented differently. We are working on adding it into the pfSense PHP module, as his implementation had some issues. https://github.com/pfsense/pfsense/pull/1901 Has all the discussion. These things take time, and we have limited resources and a large workload. It's being worked on. It won't magically make it in overnight since it wasn't usable as it was submitted.
It was usable for 2.2.4, the changes are necessary for 2.3.x because the front end is all new. Now, this feature is present in almost all competitors on the same market sector pfSense targets, isn't that little change worth the effort?(It's a question, no irony intended).
-
Well, as you said, your guess is empty as mine in this aspect. I even stated it was my POV. Now… this is one of those sittuations when if you had the feature available, a LOT of people would use it.
No, they wouldn't. The majority of users do not use squid. It may be popular, and the most popular package, but it's not installed on the majority of systems, and even less of those have AD or could even use SSO. Again, you can't make claims without support to back them. I have, through the years here in the community and through support, interacted with a significant portion of the user base and I'm in a better position to know what is being used. It may be wildly popular with a portion of the user base, but many would just as soon put squid on a second box and not try to run all of that on the firewall. Or they wouldn't use a proxy at all. We are, don't forget, primarily a firewall and not a UTM platform.
Of all the pfSense servers running, a small portion participate on the forums or buy Support subscriptions. An even smaller portion care enough to actually participate, now cut that down even more to people that really care enough about the development process and wants to help.
Yes, but even so, the ones we talk to via support, pre-sales, and so on are a good cross-section because not all of those actually are intending to make a purchase, they just want to know what we can do. And very few of those are asking for it.
Let's drop the weapons and bring this down to: You are changing "everything" on 2.3, IMO it's the best time to ask the community(from paid support to idiots like me that wants to help). I mean, not only about SSO, but other features in general.
We're not changing "everything" though – just the GUI, base OS, package backend, update mechanism, and a few other things feature-wise. Lots more than that will change for 3.0. That still doesn't have any impact on what packages can do, other than it makes building them a bit easier.
Well, the acronym SSO is not that widely known, specially in the market pfSense targets at. Ask Ingrid about Spiceworks community, most of them don't understand the concept. But take the time to explain it, whatever makes the "User" do less for more, they(we) want.
I didn't just search literally for "SSO" but variations and things like active directory and so on. I just said "SSO" for simplicity.
Create a poll, if possible with translations, make Renato Botelho post the poll on our Facebook page, I'll also ask people to take part in it. Post it on Spiceworks too.
We have done some surveys before, and we'd need to be more careful about how things are worded and where it's put up. The Facebook group is not a good representation of the community in general.
These are things that wouldn't be done by me or any one of us in development or support though, those are things you'd want to be talking to someone in sales about. Drop a line to sales@pfsense.org and see what you can get there.
It was usable for 2.2.4, the changes are necessary for 2.3.x because the front end is all new. Now, this feature is present in almost all competitors on the same market sector pfSense targets, isn't that little change worth the effort?(It's a question, no irony intended).
It was not something we could import into 2.2.4 as-is either. The changes made to accommodate the hit counter broke other things in the process like rule descriptions and lookups. It's not something that could be imported without changes no matter where it went. It might have been OK for people to apply to their systems, it might have functioned, but it was NOT in a state we could import into the code base due to the way it was done. Cool, yes, useful, also yes, but it was not implemented in a way that was good for everyone. It will be fixed, but it will take some time.
Locking this thread since it's outlived its usefulness.
