Squid explicit https filtering with remote cache

datacore

Hi@all!

I'm trying to use my pfsense as an explicit proxy for http AND https Connections.
I've set up wpad, deployed the certs to the devices and everything works fine execpt one thing: parent proxy aka remote cache.

In the moment I check the box for "enable SSL filtering" Squid stops using the remote cache.
Every request is forwarded with "HIER_DIRECT" directive instead of "FIRSTUP_PARENT".

Does anyone know something about this or how to get remote cache working again?

Squid version is 0.4.23
pfsense version is 2.3.2

Konan

Nothing to offer but I have exactly the same issue, so it's not unique to your setup.

Any help appreciated!

KOM

You don't need SSL filtering checked and client certificates installed when running explicit. That's the beauty of explicit – it just works. And what remote cache are you referring to? Default squid wants a local cache on the pfSense box itself. Unless you have an upstream squid cache on some other box, remote cache is irrelevant.

Konan

@KOM:

You don't need SSL filtering checked and client certificates installed when running explicit. That's the beauty of explicit – it just works. And what remote cache are you referring to? Default squid wants a local cache on the pfSense box itself. Unless you have an upstream squid cache on some other box, remote cache is irrelevant.

Sorry - should've mentioned that I'm running mine in transparent.

I am referring to a remote upstream cache which I have specified in the remote cache settings. If I use only transparent http only, the remote cache is used. If I use transparent http AND https, neither use will the remote cache.

KOM

Oh ok, you confused me when you said explicit proxy. I don't run transparent and have no idea about your problem. Check the System log and /var/squid/logs/cache.log for clues.

yqjrnnxq

Same problem. Any solutions yet? Mine works fine without transparent mode and forwards everything (http & https) to the upstream parent remote cache, but insists on a direct connection when in transparent mode.

doktornotor

No, no solution and zero interest in finding any; certainly have better things to do than setting up daisy-chained proxies and testing similar nonsense, just in case one proxy wasn't enough of a PITA.

Post to http://lists.squid-cache.org/listinfo/squid-users perhaps.

SaschaITM

Sometimes I wonder why doktornotor is bothering to answer questions here at all, when nearly everything asked about is "nonsense" to him. It isn't nonsense, because (for instance) in order to work around a bug in pfSense (or FreeBSD) you need a parent proxy to make load balancing/failover work correctly.

Anway, on topic: I'm not using HTTPS/SSL interception, but transparent mode + a parent proxy. This is working perfectly fine.

I didn't use the GUI settings though, but added the parent proxy on the "General" tab in the "Custom Options (Before Auth)" box like so:

cache_peer <parent cache="" ip=""> parent 3128 0 no-query default
never_direct allow all
nonhierarchical_direct off
acl DIRECT src <parent cache="" ip="">
always_direct allow DIRECT</parent></parent>

doktornotor

I'm getting an impression that having the Squid GUI available is actually a bad thing. Why? Since people want to use pfSense as proxy appliance with completely whacky setups such as:

Oh I want a completely transparent non-firewall with completely transparent proxy on a supertransparent invisible bridge behind two other routers (WTF are you sticking it on a router/firewall then)
I want to do MITM and daisy chain 5 parent proxies in front of that (yeah, that makes for great user experience, maybe in the center of a desert with 56Kbps dialup line.)
I want to cache Windows Update (use WSUS, FFS!)
I want to cache AV updates (leaving the "AVs are evil" fact behind, any AV that doesn't do streaming push updates hundreds of times a day is dead, 20 years old technology)
I want to load balance with Squid on a 10 WANs setup with CARP failover (sorry, Squid doesn't do HA failover and sucks with LB as well when run on a router)

I just bother answering since someone asked whether there's a solution planned and I'm pretty much the only person who keeps fixing the Squid package on pfSense, over and over again.

No, there's no solution planned for this issue, since it's a complete non-issue for the purpose the package is intended on pfSense. The package works as is for ~95% of users on pfSense, for the normal use cases.

If you want any of the above listed examples, or other whacky "geek" things, the package GUI does not aim to provide solutions for that.


pkg remove pfSense-pkg-squid
pkg install squid

Now, you can use shell and mess with the configuration manually as you wish and nothing will limit you in those efforts.

SaschaITM

Sorry to say that doktornotor, but you sound like a little kid who doesn't want others to play with his toy.

I really appreciate what you've done with the package(s), but from reading many of your posts it looks to me you're "destroying with your a*s what you've built with your hands", to (probably badly) translate a german saying. On one hand you're developing the package, on the other hand you're saying that everything is shit and doesn't work (hyperbole, but everyone can read the staments for themselves).

Look, your use case may not be the use case of other people. I highly doubt statements like "The package works as is for ~95% of users on pfSense, for the normal use cases" are valid, because I bet no one knows for sure what those 95 % of pfSense users are using pfSense for, or (and that is an important point) what they would use pfSense for if the options were exposed via the GUI.

I'm a "small" IT support guy, and my customers are mostly small-ish companies. For those "SOHO-type" situations it makes total sense to have one pfSense instance that handles "everything internet" (mostly because most of the time this is all you get – either a cheap router with zero functionality, or an old PC you can install pfSense on and at least try to create something more professional and functional), and caching via Squid for example is an important part of "everything internet". No, I don't want to load balance 10 lines, but I need to load balance two fucking slow ADSL lines because otherwise my customer can't receive E-Mails while someone else in his network is downloading something. No, I don't want to not use WSUS, I can't use it because it's not worth to implement for a handful of machines, but at the same time these machines are saturating the limited bandwidth when they're updating. I could go on. And I'm pretty sure I'm not the only pfSense user with a need for this type of solution, contrary to what you seem to think. After all, if I can go "big budget" I buy "big names", like Cisco or Sophos or whatever, for the ease of use and support alone. One of the "powers" of OSS has (in my opinion, at least) always been that it enables us to create solutions which are professional enough for productive use on a very limited budget, if need be.

doktornotor

What I can tell you is that

Any sort of refresh patterns is not going back to the package, since they are either not working, not useful, breaking Squid for vast majority of users, or are just not worth the maintenance effort, at all. You can add them as you wish, completely unsupported though. If you don't want to use WSUS, then perhaps get W10 and they'll grab updates via P2P from your other LAN machines (see, this is another reason why these efforts are absurd. Those refresh patterns widely published are KNOWN to be broken, not maintained by anyone, plus the technology itself is obsolete as well.)
As for LB with Squid, already linked above. (The email rant is completely unrelated here, do some traffic shaping instead.)
As for exposing obscure options in the GUI, nope. The resulting code bloat/complexity and confusion for the rest of users is just not worth it.
If you are at the point that the GUI gets in your way no matter what you try to do, well then just stop using it. It's pointless to have a GUI that will ignore any GUI configuration and let you paste complete Squid config there, there's no reason to use the GUI in the first place.

SaschaITM

You misunderstand, or maybe you're trying to evade my point. The GUI as it is now is good enough for me. I just wish you'd stop telling users stuff like "this is shit" or "that is a piece of garbage" just because you're not using the functionality they need or can't understand why they need it. And maybe be open to the possibility of implementing/supporting features you can't "understand" but which other users need, instead of shooting down ideas or discouraging users from utilizing features that you don't use or don't want to use.

Load balancing/failover isn't perfect, but it works in the sense that it's possible to utilize multiple WANs in a "usable" way from the user perspective (they can live with the occasional browser error message when a gateway goes down as long as waiting a few seconds and hitting "reload" will allow them to continue their work). Same with caching Windows updates - it may not be perfect, but it works "good enough" in the sense that it enables us to conserve at least some bandwidth. And to return to the topic of this thread, forwarding to a parent proxy also is working fine (at least for http) when using custom parameters. Maybe the "Parent Proxy" GUI functions just need a little work in order to make this work in a more convenient way.

doktornotor

No, I'm not evading your point at all. The stuff like WU/Avast/godknows what caching was already there. It was removed because it was BROKEN. If it works for you, add it manually and move on, It didn't work for vast majority of users, worse, it broke other things, noone has time to maintain similar things. Squid is NOT the way to distribute Windows updates. Even if you can use every tool as a hammer, it's just not a good idea.

Just to be crystal clear about this, look at

https://redmine.pfsense.org/issues/3847
http://wiki.squid-cache.org/ConfigExamples/DynamicContent/YouTube#Discussion

So yeah, it just doesn't work any more. Then there was Avast – they've switched to streaming updates ages ago. Nothing to cache there, dead code. Symantec - ditto. The only thing that might possibly be working is the Avira stuff, but that's just due to the fact that their AV is very much dead and has not moved anywhere for past 10 years or so, except for inventing more and more aggressive ways of nagging users with fullscreen advertising pop-ups. Why should a pfSense package care about someone using a dead AV?

When you have barely 1 person to occasionally maintain the code, you just do not add bloat well known to break every couple of months to the code. And if it's already there, you remove it.