Squid 5.8 ---> 5.9
-
So Squid will not be removed from pfSense?
-
@slu I am hopeful it stays
-
No "help me" PM's please. Use the forum, the community will thank you.
Edit : and where are the logs ?? -
Is there any information from Netgate?
Last info was:
https://www.netgate.com/blog/deprecation-of-squid-add-on-package-for-pfsense-softwarepfSense Gold subscription
-
You don't need to wait for what Netgate has to say.
It's open source. So use the open source advantages golden rule : "do - look it up - it yourself".
I don't mean that you have to build it yourself, but you can check the open source squid repository (github, where else) and support forums if there is any progress been done.The issue was (my words) : squid wasn't supported anymore by the maintainers, they couldn't follow the passe, lost interest - or it became just to complex. This can have big repercussions on the security of the package, so Netgate "warned' that they have thresholds that can make them decide to not include Squid as a package. So see the Netgate blog post as a public message to all parties : move your a** or wi'll stop including it. This means that most active squid users now contact the squid support , the maintainers, directly, to motivate them (in the modern world, this means : sending a check would do it - or harassing them on social media, very popular).
Or propose developer Micky, my son, to help them (if he gets clearence) as most devs are contaminated with the XZ syndrome right now.So, see the Netgate blog as a "Please help us helping them helping you" message.
All this is 100 % IMHO.
-
I think you're being more than a little disingenuous here.
@Gertjan said in Squid 5.8 ---> 5.9:
The issue was (my words) : squid wasn't supported anymore by the maintainers, they couldn't follow the passe, lost interest - or it became just to complex.
What maintainers ? Upstream squid maintainers ? Squid package maintainers at Netgate ?
At the time the deprecation announcement was made the Squid version in PFSense was version 5.8 and version 6.5 of Squid was available. Since then three more point releases of Squid have come out. Whose fault was it that PFSense was 7 point releases and one major release version behind and complaining about bugs not being fixed ?
How is squid "not supported" anymore by the developers ? Have you even looked at the releases page on the squid website ?
https://www.squid-cache.org/Versions/
The most recent version is 6.8 released on March 4th this year. There have been 10 point releases of Squid in the last year alone from 6.0 through to 6.8, in that time there has been one update to the Squid package in PFSense CE. (Not sure about Plus)
Squid 5 saw 18 point releases from 2020 to 2023, so development has actually accelerated if anything since then.
@Gertjan said in Squid 5.8 ---> 5.9:
This can have big repercussions on the security of the package, so Netgate "warned' that they have thresholds that can make them decide to not include Squid as a package. So see the Netgate blog post as a public message to all parties : move your a** or wi'll stop including it. This means that most active squid users now contact the squid support , the maintainers, directly, to motivate them (in the modern world, this means : sending a check would do it - or harassing them on social media, very popular).
This is incredibly arse about face and shows what is wrong with how open source projects are treated these days. The Squid developers don't owe anyone ANYTHING, least of all Netgate.
Netgate is taking an open source project - Squid, for free, building it into their firewall product, and charging people for it, at least for PFSense Plus, and making money selling hardware whose software feature list includes (or used to include) a web proxy/filter which included transparent proxying and filtering.
Many people will have bought Netgate hardware or started using PFSense plus partially on the basis that a content filtering web proxy (an essential part of any enterprise firewall) is in the feature set. I know I would not have started using PFSense if it did not have Squid/Squidguard in it as it is a feature I require, and the lack of this feature will force me to abandon PFSense when the time comes as I doubt they are going to change course on removing Squid.
Ok, so lets say Netgate have identified some specific security bugs in Squid that concern them, first of all, see if these bugs are fixed in more recent versions and actually update your package to the latest squid. If you're 7 versions and nearly a year behind then yeah, there's going to be bugs, self inflicted bugs because you're not keeping the package up to date.
But lets say that the specific bugs that concern them are not fixed in the latest Squid version either - guess what, it's open source, they can fix the bugs directly, compile a patched version of the package including the fixes for their customers and contribute the fixes upstream! Just like they do with the FreeBSD kernel... I'm sure the Squid guys would be very appreciative to get some help and get some fixes pushed upstream in the spirit of Open Source cooperation rather than just get demands and threats that bugs must be fixed.
Netgate's blog post is basically saying "We're annoyed that this package that we get for free and build into our commercial product has some security bugs but we don't think Squid is a high priority feature so we don't want to spend the effort in house to fix these bugs ourselves and contribute the fixes upstream so we're going to threaten to remove the package in the hopes that something will happen upstream or that our users will complain upstream"
That's just a terrible take on how open source works. The upstream Squid developers don't care if PFSense drops Squid support - why would they ? If PFSense users go and complain to the Squid developers as you're suggesting they're complaining to the wrong people.
If Netgate wants to sell a firewall product that includes a web proxy with filtering (I would argue a core piece of functionality of any enterprise edge firewall/router) and the only open source project that meets that need, albeit in an imperfect way is Squid and Squidguard, then it's beholden on Netgate to make the effort to help address any perceived security bugs that open source project might have, and in doing so, contribute back to the open source community that their PFSense product is built on. (Nearly everything in PFSense below the web GUI is open source projects including the FreeBSD OS base)
I can only see two possible reasons why Netgate is really taking the action they currently are with Squid:
-
They don't have the interest or resources to have a Netgate staffer work on the Squid source code to attempt to fix security bugs, (and upstream them) but instead of admitting this they want to pass all the blame upstream and just call the project that they've included in PFSense for free for over a decade a steaming pile of security bugs and "not their problem". Ripping it out is the easy, least effort path to follow.
-
They genuinely don't think a web proxy/filter is an important feature of a firewall, and have no qualms about removing a feature that many people bought their hardware for.
Granted, I am using PFSense CE (in an education environment) so I haven't bought their hardware, but I have seriously considered over the last year I've been using CE upgrading to PFSense Plus on the white box hardware that I'm using to get official TAC support, but now that Squid removal has been announced that definitely won't be happening, and in fact I will be forced to leave PFSense when Squid is actually removed. I would have been pretty pissed if I'd upgraded to Plus and a few months later one of the key feature I need was removed with no remorse and no workable alternatives. (Other than leaving PFSense)
Personally I think that if they really do believe Web proxy/content filter is not an important feature they're making a big mistake and it's an incredibly short sighted move. I can't think of any other comparable firewall/router from other vendors that does not have some kind of web content filtering in it. All the removal of Squid will do is lose them userbase and also paying customers.
-
-
Again after that announcement Squid has been updated to 5.9 in Plus on ARM so something was fixed in the package, if they were going to abandon it why take the time to release a new version... right?
-
@Gertjan 6.6 ??? So cool
-
@JonathanLee said in Squid 5.8 ---> 5.9:
@DBMandrake That's amazing so the free version of pfsense non plus version has 6.3?
Also do you use the cache?
if you do I highly recommend you enable
pipeline_prefetch 100 in custom area this lets Squid request more than one item at a time really speeds it upCan't say I've really noticed any performance issues with Squid - and that's with around 800 clients, but I will check out this option anyway, thanks.
The only real issue I have with squid is this issue....
-
@JonathanLee said in Squid 5.8 ---> 5.9:
Again after that announcement Squid has been updated to 5.9 in Plus on ARM so something was fixed in the package, if they were going to abandon it why take the time to release a new version... right?
Continuing to update the package (at least in Plus, the last time it was updated in CE was with 2.7.0 I think) despite saying they're deprecating it is a bit odd I agree, but I don't think that means they've changed their mind. I guess we will see.
Unless they explicitly come out and say they've changed their mind, anybody who relies on the feature will already be actively investigating alternatives - I know I am.
-
Thanks for your very detailed words.
You're some one who knows about the why part, as it is clear that you've been looking for the real reasons on the right places
My post wasn't about finger pointing to some one or something like that : everybody is free to decide what they do with their software.
I do get some times a bit upset as I read that a lot is taken for granted, so I started to enumerate possible reasons. And by no means I've listed all the reasons, or the correct reasons. Just that these projects are driven by people and they can have 'reasons'.As you've stated very clearly, and I guess a lot of us don't know : "pfSense" (the pfSense squid package maintainer) grabs the squid source, adapts it so it compile in a pfSense environment, then he write, a whole lot of GUI glue script code and whatever else that is needed so that it shows up and can be used as a package under pfSense. Then rince and repeat, as this is an never ending story.
I'm not sure if it is a Netgate employee, or some other, 'freelance' maintainer.The reason of my post above was not at all because I know why Netgate posted this : Deprecation of Squid Add-On Package For pfSense Software I've just read the blog post and interpreted my thoughts about it. I saw it as a message like a "be aware that we could pull the plug on this package because ..." as people (pfSense users) depend on it, so that they were warned that Netgate can pull the plug.
edit : your case 1) is somewhat what I thought. Not that I would use the word 'blame', as it might be the simple result of the equation : what does it deliver, and what are the costs.
Ripping it out is the easy, least effort path to follow.
and probably the only choice they have.I was surprised of the reaction of some pfSEnse users that were asking ; can Netgate not just make Squid better, fork it, or do something to solve the issue, as they don't know that squid is ... HUGE ....
I agree with everything you said, although I really think that :
Personally I think that if they really do believe Web proxy/content filter is not an important feature they're making a big mistake and it's an incredibly short sighted move
this can't be the case. Or, differently, I hope that's not the case.
Because of what you said :All the removal of Squid will do is lose them userbase and also paying customers.
Important info : I'm not a squid user myself.
And yes, It might be possible that my humor was a bit borderline. -
@DBMandrake said in Squid 5.8 ---> 5.9:
I think you're being more than a little disingenuous here.
@Gertjan said in Squid 5.8 ---> 5.9:
The issue was (my words) : squid wasn't supported anymore by the maintainers, they couldn't follow the passe, lost interest - or it became just to complex.
What maintainers ? Upstream squid maintainers ? Squid package maintainers at Netgate ?
At the time the deprecation announcement was made the Squid version in PFSense was version 5.8 and version 6.5 of Squid was available. Since then three more point releases of Squid have come out. Whose fault was it that PFSense was 7 point releases and one major release version behind and complaining about bugs not being fixed ?
How is squid "not supported" anymore by the developers ? Have you even looked at the releases page on the squid website ?
https://www.squid-cache.org/Versions/
The most recent version is 6.8 released on March 4th this year. There have been 10 point releases of Squid in the last year alone from 6.0 through to 6.8, in that time there has been one update to the Squid package in PFSense CE. (Not sure about Plus)
Squid 5 saw 18 point releases from 2020 to 2023, so development has actually accelerated if anything since then.
@Gertjan said in Squid 5.8 ---> 5.9:
This can have big repercussions on the security of the package, so Netgate "warned' that they have thresholds that can make them decide to not include Squid as a package. So see the Netgate blog post as a public message to all parties : move your a** or wi'll stop including it. This means that most active squid users now contact the squid support , the maintainers, directly, to motivate them (in the modern world, this means : sending a check would do it - or harassing them on social media, very popular).
This is incredibly arse about face and shows what is wrong with how open source projects are treated these days. The Squid developers don't owe anyone ANYTHING, least of all Netgate.
Netgate is taking an open source project - Squid, for free, building it into their firewall product, and charging people for it, at least for PFSense Plus, and making money selling hardware whose software feature list includes (or used to include) a web proxy/filter which included transparent proxying and filtering.
Many people will have bought Netgate hardware or started using PFSense plus partially on the basis that a content filtering web proxy (an essential part of any enterprise firewall) is in the feature set. I know I would not have started using PFSense if it did not have Squid/Squidguard in it as it is a feature I require, and the lack of this feature will force me to abandon PFSense when the time comes as I doubt they are going to change course on removing Squid.
Ok, so lets say Netgate have identified some specific security bugs in Squid that concern them, first of all, see if these bugs are fixed in more recent versions and actually update your package to the latest squid. If you're 7 versions and nearly a year behind then yeah, there's going to be bugs, self inflicted bugs because you're not keeping the package up to date.
But lets say that the specific bugs that concern them are not fixed in the latest Squid version either - guess what, it's open source, they can fix the bugs directly, compile a patched version of the package including the fixes for their customers and contribute the fixes upstream! Just like they do with the FreeBSD kernel... I'm sure the Squid guys would be very appreciative to get some help and get some fixes pushed upstream in the spirit of Open Source cooperation rather than just get demands and threats that bugs must be fixed.
Netgate's blog post is basically saying "We're annoyed that this package that we get for free and build into our commercial product has some security bugs but we don't think Squid is a high priority feature so we don't want to spend the effort in house to fix these bugs ourselves and contribute the fixes upstream so we're going to threaten to remove the package in the hopes that something will happen upstream or that our users will complain upstream"
That's just a terrible take on how open source works. The upstream Squid developers don't care if PFSense drops Squid support - why would they ? If PFSense users go and complain to the Squid developers as you're suggesting they're complaining to the wrong people.
If Netgate wants to sell a firewall product that includes a web proxy with filtering (I would argue a core piece of functionality of any enterprise edge firewall/router) and the only open source project that meets that need, albeit in an imperfect way is Squid and Squidguard, then it's beholden on Netgate to make the effort to help address any perceived security bugs that open source project might have, and in doing so, contribute back to the open source community that their PFSense product is built on. (Nearly everything in PFSense below the web GUI is open source projects including the FreeBSD OS base)
I can only see two possible reasons why Netgate is really taking the action they currently are with Squid:
-
They don't have the interest or resources to have a Netgate staffer work on the Squid source code to attempt to fix security bugs, (and upstream them) but instead of admitting this they want to pass all the blame upstream and just call the project that they've included in PFSense for free for over a decade a steaming pile of security bugs and "not their problem". Ripping it out is the easy, least effort path to follow.
-
They genuinely don't think a web proxy/filter is an important feature of a firewall, and have no qualms about removing a feature that many people bought their hardware for.
Granted, I am using PFSense CE (in an education environment) so I haven't bought their hardware, but I have seriously considered over the last year I've been using CE upgrading to PFSense Plus on the white box hardware that I'm using to get official TAC support, but now that Squid removal has been announced that definitely won't be happening, and in fact I will be forced to leave PFSense when Squid is actually removed. I would have been pretty pissed if I'd upgraded to Plus and a few months later one of the key feature I need was removed with no remorse and no workable alternatives. (Other than leaving PFSense)
Personally I think that if they really do believe Web proxy/content filter is not an important feature they're making a big mistake and it's an incredibly short sighted move. I can't think of any other comparable firewall/router from other vendors that does not have some kind of web content filtering in it. All the removal of Squid will do is lose them userbase and also paying customers.
Everything I’ve been saying in the forums since the announcement of the removal of Squid in pfsense. Glad I’m not alone
edit: To be fair to netgate, their documentation does point out that using proxies is no longer effective. https://docs.netgate.com/pfsense/en/latest/recipes/block-websites.html#using-a-proxy
That being the case there really is no excuse to be behind on numerous releases and possible package fixes and then at the same time state the package is insecure. I always found the Netgate response to be nonsensical at best and at worst extremely disingenuous because as I and other have pointed out - they are taking freeware , bundling it into a paid product and then complaining that the package is insecure without contributing anything upstream to the project. The same criticism they give to the OPNsense devs may i add (They pull from upstream but contribute nothing back). A classic example of don't throw rocks in a glass house.
edit2: One last thing. There is no current mainteainer of the Squid package for pfSense. I have reached out to the maintainer of record for SquidGuard and he informed me over email that he hasn't been involved in the package for quite some time. So the package may be getting updated but improvements to it will require someone to take charge of the plugin. I believe Netgate's position should be rather - because there is no maintainer we cannot continue to offer the package as we cant ensure its continued security. Thats much better than saying "Upstream doesn't fix anything" which turns out to be silly.
-
-
@DBMandrake I have never seen that error with my configuration, I will send it to you once it reboots
-
@JonathanLee said in Squid 5.8 ---> 5.9:
@DBMandrake I have never seen that error with my configuration, I will send it to you once it reboots
You won't see that error in the logs unless you add the debug options that I describe in the ticket. If you add the debug options in the (normally hidden) advanced settings, restart squid, then use the transparent proxy with some clients for a while and check the cache.log (at /var/squid/logs/cache.log) you'll see lines like those, particularly with any large CDN.
This is the single biggest limitation of Squid by far at the moment, and it's actually a bug, as the configuration option relating to this dns test does not behave as described in the squid documentation. Anyway, it's all in the ticket.
-
@michmoor said in Squid 5.8 ---> 5.9:
Everything I’ve been saying in the forums since the announcement of the removal of Squid in pfsense. Glad I’m not alone
I'm sad because I think PFSense is one of the best and most feature rich firewalls out there - although I've heard about it for years I only tried it the first time about a year and a half ago and started using it seriously a year ago.
I was forced at very short notice to abandon the firewall system we had previously been running, (vendor went bankrupt) and spent a lot of time setting up PFSense in a fairly complex environment that makes full use of a large percentage of the features (Things like OpenVPN, DHCP relay, Avahi, udpbroadcast forwarder, etc) and after a couple of teething problems it has been solid as a rock and has never failed to have a feature that I have needed.
Removal of Squid/Squidguard will force me to yet again look for another firewall and to be honest there is little out there that can compete with the feature rich functionality of PFSense, because I have come to rely on so many of the features in the configuration I'm now running.
edit: To be fair to netgate, their documentation does point out that using proxies is no longer effective. https://docs.netgate.com/pfsense/en/latest/recipes/block-websites.html#using-a-proxy
"No longer effective" depends what you're using it for... Is a Squid proxy still effective as a bandwidth saving cache ? No, and it hasn't been for about 8 years due to ubiquitous use of SSL/TLS. I don't have caching enabled.
Is it still useful as an explicit proxy server (proxy configured on the client) optionally with user authentication ? Yes.
Is it still useful as a content filter ? Yes, partially. Without MITM inspection and client side certificates (which is complex and problematic to set up) it's impossible for it to do any kind of deep inspection like keyword matching, but SNI inspection still works.
SNI inspection doesn't allow the full URL to be found but it allows the domain name for the request to be found whether you're using explicit proxy or transparent proxying.
This is way more useful that it sounds. Coupled with Squidguard this makes a powerful domain name based block list - and importantly for us, you can apply different domain blocklists for different IP ranges / VLAN's, something that is very difficult to do with DNS based blocking.
As a means of filtering out specific websites (based on categories and/or individual domain names) the Squid/Squidguard combination is more powerful and more thorough that any form of DNS blocking like PFBlocker.
SNI based domain name filtering is what we use Squid/Squidguard for - as an explicit proxy for devices where this can be configured, and also as a transparent proxy as a fall-back. Unfortunately the bug I linked to earlier does cause some issues in transparent mode, but this bug should be fixable.
That being the case there really is no excuse to be behind on numerous releases and possible package fixes and then at the same time state the package is insecure. I always found the Netgate response to be nonsensical at best and at worst extremely disingenuous because as I and other have pointed out - they are taking freeware , bundling it into a paid product and then complaining that the package is insecure without contributing anything upstream to the project. The same criticism they give to the OPNsense devs may i add (They pull from upstream but contribute nothing back). A classic example of don't throw rocks in a glass house.
I don't know for certain that Netgate don't or haven't contributed to Squid in the past (maybe they have) but if the issue they're not happy with is security bugs, the best thing to do is dig into the code and help fix those bugs and upstream the fixes.
Bug fixes are generally accepted into upstream projects much more readily than new features or major code refactors.
edit2: One last thing. There is no current mainteainer of the Squid package for pfSense. I have reached out to the maintainer of record for SquidGuard and he informed me over email that he hasn't been involved in the package for quite some time. So the package may be getting updated but improvements to it will require someone to take charge of the plugin. I believe Netgate's position should be rather - because there is no maintainer we cannot continue to offer the package as we cant ensure its continued security. Thats much better than saying "Upstream doesn't fix anything" which turns out to be silly.
Some of the packages in PFSense are indeed maintained by volunteers who don't work for Netgate, although I'm not sure if that was the case with Squid and/or Squidguard.
If volunteers for key packages are burnt out / busy / not interested anymore then that's an issue Netgate needs to find a solution to. The proxy features were listed on the website as features of the product right up until the announcement that it was going to be removed.
As I said, I'd have been pretty annoyed if I'd forked out for hardware (or even the TAC subscription) based on the feature set at the time I bought it, only to have the feature deprecated a few months later.
While I'm not hopeful, I do hope they reconsider, because if nothing else PFSense will no longer be the "swiss army knife" of firewalls that it is now if a key feature like this is removed. It's hard to recommend a firewall with a key feature like that missing.
The ideal solution from the point of view of end users would be if Netgate put the resources into maintaining packages such as Squid and actively working on security fixes for them where necessary. Here's hoping.
-
FYI------>
Hello fellow Netgate community
Squid 6.6 is available in bata version 24 I am working on some bugs on it but it's more secure and fixes all the concerns as it is the latest version of Squid. Squid is very effective at blocking URLS and I am working on finding a way to access the menu with Squid support also. I have some GitHub pulls open for the issues. More to come -
@slu that stateement was released before 6.6 was available
-
@JonathanLee said in Squid 5.8 ---> 5.9:
FYI------>
Hello fellow Netgate community
Squid 6.6 is available in bata version 24 I am working on some bugs on it but it's more secure and fixes all the concerns as it is the latest version of Squid. Squid is very effective at blocking URLS and I am working on finding a way to access the menu with Squid support also. I have some GitHub pulls open for the issues. More to comeAre you working on the official squid package in PFSense or on an alternative ?
If the former this is interesting news...
-
@DBMandrake Official Version only issue with it is
Per Squid Support Amos Jeffries
"You do have direct proxy (and thus manager) access via the 192.168.1.1:3128 so this URL should work:
http://192.168.1.1:3128/squid-internal-mgr/menu"Per Alex Rousskov Squid Support
"Currently, you may need to figure out what hostname Squid considers to self-identify as and use that hostname in cache manager requests. The following bug report may help, but there are several overlapping problems here, and that makes it difficult to triage without more information: https://bugs.squid-cache.org/show_bug.cgi?id=5283"https://bugs.squid-cache.org/show_bug.cgi?id=5283
It works great Blocks URLs some software convergence issues but nothing really major might need a new SSL/TLS cert made but that's about it
-
@JonathanLee said in Squid 5.8 ---> 5.9:
@DBMandrake Official Version only issue with it is
Per Squid Support Amos Jeffries
"You do have direct proxy (and thus manager) access via the 192.168.1.1:3128 so this URL should work:
http://192.168.1.1:3128/squid-internal-mgr/menu"Per Alex Rousskov Squid Support
"Currently, you may need to figure out what hostname Squid considers to self-identify as and use that hostname in cache manager requests. The following bug report may help, but there are several overlapping problems here, and that makes it difficult to triage without more information: https://bugs.squid-cache.org/show_bug.cgi?id=5283"https://bugs.squid-cache.org/show_bug.cgi?id=5283
It works great Blocks URLs some software convergence issues but nothing really major might need a new SSL/TLS cert made but that's about it
What about this bug ?
https://redmine.pfsense.org/issues/14390
This is a biggy - it has existed in Squid for over 10 years and causes major problems with CDN networks with rapidly rotated multiple IP address hostnames. (only with transparent proxying)
The bug has been there for years, what has changed is CDN's have started to very aggressively rotate DNS entries with TTL's as short as 30 seconds or less, this has made the symptoms trigger far more often than in the past, and this single issue is responsible for nearly all intermittent behaviour and connection failures (HTTP/409) in transparent proxy mode.
A fix for this would be massive.
