Squid 5.8 ---> 5.9
-
@DBMandrake I have never seen that error with my configuration, I will send it to you once it reboots
-
@JonathanLee said in Squid 5.8 ---> 5.9:
@DBMandrake I have never seen that error with my configuration, I will send it to you once it reboots
You won't see that error in the logs unless you add the debug options that I describe in the ticket. If you add the debug options in the (normally hidden) advanced settings, restart squid, then use the transparent proxy with some clients for a while and check the cache.log (at /var/squid/logs/cache.log) you'll see lines like those, particularly with any large CDN.
This is the single biggest limitation of Squid by far at the moment, and it's actually a bug, as the configuration option relating to this dns test does not behave as described in the squid documentation. Anyway, it's all in the ticket.
-
@michmoor said in Squid 5.8 ---> 5.9:
Everything I’ve been saying in the forums since the announcement of the removal of Squid in pfsense. Glad I’m not alone
I'm sad because I think PFSense is one of the best and most feature rich firewalls out there - although I've heard about it for years I only tried it the first time about a year and a half ago and started using it seriously a year ago.
I was forced at very short notice to abandon the firewall system we had previously been running, (vendor went bankrupt) and spent a lot of time setting up PFSense in a fairly complex environment that makes full use of a large percentage of the features (Things like OpenVPN, DHCP relay, Avahi, udpbroadcast forwarder, etc) and after a couple of teething problems it has been solid as a rock and has never failed to have a feature that I have needed.
Removal of Squid/Squidguard will force me to yet again look for another firewall and to be honest there is little out there that can compete with the feature rich functionality of PFSense, because I have come to rely on so many of the features in the configuration I'm now running.
edit: To be fair to netgate, their documentation does point out that using proxies is no longer effective. https://docs.netgate.com/pfsense/en/latest/recipes/block-websites.html#using-a-proxy
"No longer effective" depends what you're using it for... Is a Squid proxy still effective as a bandwidth saving cache ? No, and it hasn't been for about 8 years due to ubiquitous use of SSL/TLS. I don't have caching enabled.
Is it still useful as an explicit proxy server (proxy configured on the client) optionally with user authentication ? Yes.
Is it still useful as a content filter ? Yes, partially. Without MITM inspection and client side certificates (which is complex and problematic to set up) it's impossible for it to do any kind of deep inspection like keyword matching, but SNI inspection still works.
SNI inspection doesn't allow the full URL to be found but it allows the domain name for the request to be found whether you're using explicit proxy or transparent proxying.
This is way more useful that it sounds. Coupled with Squidguard this makes a powerful domain name based block list - and importantly for us, you can apply different domain blocklists for different IP ranges / VLAN's, something that is very difficult to do with DNS based blocking.
As a means of filtering out specific websites (based on categories and/or individual domain names) the Squid/Squidguard combination is more powerful and more thorough that any form of DNS blocking like PFBlocker.
SNI based domain name filtering is what we use Squid/Squidguard for - as an explicit proxy for devices where this can be configured, and also as a transparent proxy as a fall-back. Unfortunately the bug I linked to earlier does cause some issues in transparent mode, but this bug should be fixable.
That being the case there really is no excuse to be behind on numerous releases and possible package fixes and then at the same time state the package is insecure. I always found the Netgate response to be nonsensical at best and at worst extremely disingenuous because as I and other have pointed out - they are taking freeware , bundling it into a paid product and then complaining that the package is insecure without contributing anything upstream to the project. The same criticism they give to the OPNsense devs may i add (They pull from upstream but contribute nothing back). A classic example of don't throw rocks in a glass house.
I don't know for certain that Netgate don't or haven't contributed to Squid in the past (maybe they have) but if the issue they're not happy with is security bugs, the best thing to do is dig into the code and help fix those bugs and upstream the fixes.
Bug fixes are generally accepted into upstream projects much more readily than new features or major code refactors.
edit2: One last thing. There is no current mainteainer of the Squid package for pfSense. I have reached out to the maintainer of record for SquidGuard and he informed me over email that he hasn't been involved in the package for quite some time. So the package may be getting updated but improvements to it will require someone to take charge of the plugin. I believe Netgate's position should be rather - because there is no maintainer we cannot continue to offer the package as we cant ensure its continued security. Thats much better than saying "Upstream doesn't fix anything" which turns out to be silly.
Some of the packages in PFSense are indeed maintained by volunteers who don't work for Netgate, although I'm not sure if that was the case with Squid and/or Squidguard.
If volunteers for key packages are burnt out / busy / not interested anymore then that's an issue Netgate needs to find a solution to. The proxy features were listed on the website as features of the product right up until the announcement that it was going to be removed.
As I said, I'd have been pretty annoyed if I'd forked out for hardware (or even the TAC subscription) based on the feature set at the time I bought it, only to have the feature deprecated a few months later.
While I'm not hopeful, I do hope they reconsider, because if nothing else PFSense will no longer be the "swiss army knife" of firewalls that it is now if a key feature like this is removed. It's hard to recommend a firewall with a key feature like that missing.
The ideal solution from the point of view of end users would be if Netgate put the resources into maintaining packages such as Squid and actively working on security fixes for them where necessary. Here's hoping.
-
FYI------>
Hello fellow Netgate community
Squid 6.6 is available in bata version 24 I am working on some bugs on it but it's more secure and fixes all the concerns as it is the latest version of Squid. Squid is very effective at blocking URLS and I am working on finding a way to access the menu with Squid support also. I have some GitHub pulls open for the issues. More to come -
@slu that stateement was released before 6.6 was available
-
@JonathanLee said in Squid 5.8 ---> 5.9:
FYI------>
Hello fellow Netgate community
Squid 6.6 is available in bata version 24 I am working on some bugs on it but it's more secure and fixes all the concerns as it is the latest version of Squid. Squid is very effective at blocking URLS and I am working on finding a way to access the menu with Squid support also. I have some GitHub pulls open for the issues. More to comeAre you working on the official squid package in PFSense or on an alternative ?
If the former this is interesting news...
-
@DBMandrake Official Version only issue with it is
Per Squid Support Amos Jeffries
"You do have direct proxy (and thus manager) access via the 192.168.1.1:3128 so this URL should work:
http://192.168.1.1:3128/squid-internal-mgr/menu"Per Alex Rousskov Squid Support
"Currently, you may need to figure out what hostname Squid considers to self-identify as and use that hostname in cache manager requests. The following bug report may help, but there are several overlapping problems here, and that makes it difficult to triage without more information: https://bugs.squid-cache.org/show_bug.cgi?id=5283"https://bugs.squid-cache.org/show_bug.cgi?id=5283
It works great Blocks URLs some software convergence issues but nothing really major might need a new SSL/TLS cert made but that's about it
-
@JonathanLee said in Squid 5.8 ---> 5.9:
@DBMandrake Official Version only issue with it is
Per Squid Support Amos Jeffries
"You do have direct proxy (and thus manager) access via the 192.168.1.1:3128 so this URL should work:
http://192.168.1.1:3128/squid-internal-mgr/menu"Per Alex Rousskov Squid Support
"Currently, you may need to figure out what hostname Squid considers to self-identify as and use that hostname in cache manager requests. The following bug report may help, but there are several overlapping problems here, and that makes it difficult to triage without more information: https://bugs.squid-cache.org/show_bug.cgi?id=5283"https://bugs.squid-cache.org/show_bug.cgi?id=5283
It works great Blocks URLs some software convergence issues but nothing really major might need a new SSL/TLS cert made but that's about it
What about this bug ?
https://redmine.pfsense.org/issues/14390
This is a biggy - it has existed in Squid for over 10 years and causes major problems with CDN networks with rapidly rotated multiple IP address hostnames. (only with transparent proxying)
The bug has been there for years, what has changed is CDN's have started to very aggressively rotate DNS entries with TTL's as short as 30 seconds or less, this has made the symptoms trigger far more often than in the past, and this single issue is responsible for nearly all intermittent behaviour and connection failures (HTTP/409) in transparent proxy mode.
A fix for this would be massive.
-
@DBMandrake Was the Bug also listed in Squid Bugs? I thought that was closed out in Squid 6.6 they had an open report on the Squid side also. Again they now have the version 6.6 installed if you have the ability to do BE (Boot Environments) Check it out I opened some pulls for it just small issues with Squid -k parse. Again the menu issue with making squid know its identity. Main thing is it works, I can block URLS and cache traffic in 24. I might need a new certificate. All just seem like Squid to GUI software convergence for user convenience is all. Key issue is resolved the security concerns it is running Squid 6.6 and it has all the updates in that version. The Squid DEV version is 7.X right now the stable version is 6.X
I wish I knew more I love this package I am glad Netgate did not give up on it. Squid really shines when you configure it correctly.
-
@DBMandrake They have had to do something they use acceleration systems globally. Squid is used all over it's huge. It is technically green technology as items are not downloaded a million times over and over they are downloaded once and stored closer to the client. Again software hardware convergence is a bit complicated.
This really interests me (see below), Facebook is actively working on improving cacheing, they even have an open source project. It does save energy.
2016
https://research.facebook.com/blog/2016/4/the-evolution-of-advanced-caching-in-the-facebook-cdn/
2021
https://engineering.fb.com/2021/09/02/core-infra/cachelib/
https://cachelib.orgOverall cacheing fascinates me as a computer science student, the complications, the protocols, and the challenge. It is amazing when it works correctly. This tool is amazing when it's configured correctly.
-
@JonathanLee said in Squid 5.8 ---> 5.9:
@DBMandrake Was the Bug also listed in Squid Bugs? I thought that was closed out in Squid 6.6 they had an open report on the Squid side also.
I originally reported this bug 11 months ago and only to PFSense (discussed on the forum here in another thread then I opened the ticket) as at the time it was running a very out of date version of squid.
I haven't seen any upstream reports for this issue, do you have a link to that if you think it is fixed in 6.6 ?
There has been no update in the original redmine ticket so I assumed that the problem has not been fixed.
Again they now have the version 6.6 installed if you have the ability to do BE (Boot Environments) Check it out I opened some pulls for it just small issues with Squid -k parse. Again the menu issue with making squid know its identity. Main thing is it works, I can block URLS and cache traffic in 24. I might need a new certificate. All just seem like Squid to GUI software convergence for user convenience is all. Key issue is resolved the security concerns it is running Squid 6.6 and it has all the updates in that version. The Squid DEV version is 7.X right now the stable version is 6.X
Unfortunately I'm running 2.7.2 CE which currently only has Squid 6.3, and when I checked recently version 6.3 still has this issue, and boot environments are not supported in CE either. Based on previous release cycles of CE it could be a long time (6 months or more) before 6.6 found its way into CE.
Are you able to try the Python test script I attached to the PFSense ticket ? This can be run on any PC with Python installed.
For the test to be valid transparent proxying needs to be enabled on PFSense with the client going through the transparent proxy with no explicit proxy settings.
The way it works is it resolves a test hostname known to cause issues (I have at least 4 in the script that can be tried) and saves the IP address, it then crafts an HTTPS query to the same IP address every 30 seconds instead of freshly looking up the hostname. This is to simulate use of a "stale" DNS record.
When this IP address times out of the (rapidly rotated) DNS records, Squid will start to refuse the requests with HTTP/409.
If the same test is run with no transparent proxy there will be no errors, so this script is a reliable way to reproduce the issue.
I wish I knew more I love this package I am glad Netgate did not give up on it. Squid really shines when you configure it correctly.
So far I've seen no official word from Netgate that they've reversed their decision, so I hope you're right.
If they have changed their mind and are bringing the package right up to date and as a side effect this long standing Squid bug is fixed that would be absolutely fantastic.
-
@DBMandrake I test it when I get some time, I can’t do tests while family is home so I have what I call the everything bagel boot environment running sometime in the week I will check it the script.
-
@JonathanLee I've done a little testing myself today.
Using the following I was able to install a Squid 6.8 build on 2.7.2 CE on a spare test firewall:
https://forum.netgate.com/topic/186911/squid-6-8-available
Unfortunately this does not fix the issue as it still fails my python testing script (note that some of the domain names I've given as examples in the script are no longer valid so you'll need to find some that work) and also the errors are still logged in cache.log.
So as of Squid version 6.8 this problem still exists unfortunately. I have also not been able to find mention of any fixes for this issue in recent squid release notes.
-
@JonathanLee Here is the discussion about this issue:
https://bugs.squid-cache.org/show_bug.cgi?id=4940
A patch was first provided nearly 4 years ago but it has still not been accepted. (!)
The problem has been known about for over 10 years, as the discussion shows.
There is a patch attached to the ticket which is basically a one liner change that fixes the issue. This is something that netgate could choose to include in their version of the package if they wanted to. (And if they were satisfied it was safe)
-
@DBMandrake I also had a ticket open for this, it was closed as a duplicate I do know what you're talking about, iTunes does it also.
