Squid HTTPS/SSL filtering 2017 [Is this it?]

pfsensation

Hi everyone,

I'm new to the forums, and new to Pfsense. My goals are to setup HTTPS/SSL filtering at home. As most of you know, since Google pushed HTTPS, more and more websites started using it. On top of that hackers are getting more and more advanced, and are using HTTPS themselves to bypass detection from IDS's etc. So I find it extremely stupid to have filtering that doesn't cover HTTPS, as there's also all those web based proxies and even ClamAV won't detect viruses sent over HTTPS (tested with eicar).
I have read a lot of threads, however I am running into a lot of issues and just can't seem to crack it. I've spent a few days frantically trying to sort it out.

For starters, I have Squid and Squidguard setup, I've got WPAD setup and devices seem to be picking up the proxy settings just fine, and I have also created a CA certificate and have it installed on my devices. Here's the issue I am getting as of now, it seems squid is changing the domain from the domain typed in to "http" :

Some of the other issues I am getting is slow browsing intermittently. Could it be related to PowerD? I have tried changing around the settings but it doesn't seem to have helped much. Furthermore what kind of hardware is recommended for HTTPS decryption? I'm literally just running it on a old machine that I've thrown in an extra dual NIC.

Another thing I want to ask is how are you all getting past HSTS? Do you guys setup bypasses for HSTS sites such as Google? It's really frustrating. My college has a similar to what I want to do here, they run Smoothwall and everything seems to work perfectly. Anyone is able to bring their own device into college, and use the internet as long as they install the certificate. And there's barely any errors at all. Their blocking system seems to work even with random websites, for example if iadoi3.com was a porn site, it would detect it. Not based on looking at the URL but actually looking through the website and inspecting it. Which I think is incredibly clever and advanced.
How would I setup something like this to work efficiently? Is it even possible with pfsense? As of now, squid seems to be quite finnicky.
I've of course also got friends and guests coming over, I don't want to spend ages fiddling around with their device trying to set it up with my network. In terms of the certificate, I have got a link on my captive portal for people to download and install it.

Oh, and another thing I nearly forgot. Android devices are really getting a hard time picking up the proxy settings. Could I just force the traffic to go through Squid? How would I do that? Just with port forward?

Here's my setup:
2.3.2-RELEASE-p1 (i386)
Intel(R) Celeron(R) CPU 420 @ 1.60GHz
2GB of RAM
140GB Hard drive
NIC : HP NC360T
Squid version: 0.4.29_1
SquidGuard version: 1.14_4

Installed packages:

Hope someone could really help me out with this, I'm sure many people are going through the same trouble.
Thank you in advance! Happy new year!

doktornotor

@pfsensation:

it seems squid is changing the domain from the domain typed in to "http"

No, it's not. See https://redmine.pfsense.org/issues/6777#note-2

If Squidguard is blocking Google for you, that's unfortunate, you'll need to fix that issue instead. (No idea how, the package is broken and the code not maintainable.)

pfsensation

@doktornotor:

@pfsensation:

it seems squid is changing the domain from the domain typed in to "http"

No, it's not. See https://redmine.pfsense.org/issues/6777#note-2

If Squidguard is blocking Google for you, that's unfortunate, you'll need to fix that issue instead. (No idea how, the package is broken and the code not maintainable.)

This is just extremely strange. What would be best to use as a replacement for Squid? There really does seem no proper fix for this.

doktornotor

There is no need to replace Squid, this is not a Squid issue. Anyway, see https://www.freshports.org/www/e2guardian/ (the GUI/pfSense package got never finished - https://github.com/pfsense/pfsense-packages/pull/866, https://forum.pfsense.org/index.php?topic=87526.0). As for squidGuard replacements, there's https://www.freshports.org/www/ufdbguard/ (not available in pfSense and obviously no GUI for that.)

C0RR0SIVE

I wouldn't bother with MITM with a CA to be honest… If you are using WPAD your browser and SQUID both should be letting HTTPS traffic pass through properly. This happens even for HSTS based websites like Google.com. However this also negates the ability of ClamAV working, which, is useless anyways.

pfsensation

@C0RR0SIVE:

I wouldn't bother with MITM with a CA to be honest… If you are using WPAD your browser and SQUID both should be letting HTTPS traffic pass through properly. This happens even for HSTS based websites like Google.com. However this also negates the ability of ClamAV working, which, is useless anyways.

Is it even possible to work like this? From what I understand, I cannot do any HTTPS filtering without MITM as the traffic is encrypted.

@doktornotor:

There is no need to replace Squid, this is not a Squid issue. Anyway, see https://www.freshports.org/www/e2guardian/ (the GUI/pfSense package got never finished - https://github.com/pfsense/pfsense-packages/pull/866, https://forum.pfsense.org/index.php?topic=87526.0). As for squidGuard replacements, there's https://www.freshports.org/www/ufdbguard/ (not available in pfSense and obviously no GUI for that.)

Thanks for that, I will have a look at e2guardian, from the description, it does look like what I would wanna use but it's quite outdated. However, what I really wanted to know is how others are filtering HTTPS content using Pfsense in this day and age. I'm not an expert, but I know there's many hoops to jump through as HTTPS was designed so that inspection couldn't be done easily (HSTS for example).

I would really appreciate if someone who is already doing this could message, and give me pointers to the system they're using and how they're jumping through the hoops I mentioned above.

C0RR0SIVE

shrugs I don't use MITM and CA's on my network, but I use WPAD and I seem to get a "block page" any time I hit a blocked secure website, granted it doesn't show the actual block page since you can't intercept, so SQUID just breaks the tunnel.

Blocked Secure Site

Allowed Secure Site

I also have Transparent disabled… If you are wanting to get the block page that you would normally use however, not sure how it would be possible all because of HSTS.

pfsensation

@C0RR0SIVE:

shrugs I don't use MITM and CA's on my network, but I use WPAD and I seem to get a "block page" any time I hit a blocked secure website, granted it doesn't show the actual block page since you can't intercept, so SQUID just breaks the tunnel.

Blocked Secure Site

Allowed Secure Site

I also have Transparent disabled… If you are wanting to get the block page that you would normally use however, not sure how it would be possible all because of HSTS.

I'm having that same issue, but I think it maybe linked to chrome. The block page correctly displays on other browers, and devices. It's kinda sad really, I'm sure others have the setup I want but not sure if they're not willing to share. Or haven't seen this thread yet.

Squid is making me want to bash my head on the wall…

EDIT : The page has been fixed on Chrome now, after the latest update. I'm using Chrome beta.

sichent

This might be related - https://docs.diladele.com/faq/squid/cannot_connect_to_site_using_https.html
In short - browser ignores everything but 200 Connection established from initial CONNECT tunnel setup to remote HTTPS sites thorough proxy (hope this is not too technical)

pfsensation

@sichent:

This might be related - https://docs.diladele.com/faq/squid/cannot_connect_to_site_using_https.html
In short - browser ignores everything but 200 Connection established from initial CONNECT tunnel setup to remote HTTPS sites thorough proxy (hope this is not too technical)

Thank you for the reply @Sichent, so is Diladele the only filtering system available for pfsense that's actually actively updated and maintained? Doktornotor seems to be working extremely hard trying to fix Squid itself, but issues like this just remain. I haven't been able to use anything else decent enough that also does SSL inspection of sorts, I'm stuck relying on SquidGuard which is half broken and OpenDNS.

I'm not sure if others are not doing this, or just don't want to share their methods. But it seems bizarre to me, how can everyone be filtering without HTTPS… As most do just use Squid + Squid Guard.

doktornotor

Fixing Squid (pretty much done as of 0.4.33) sadly will not do anything for the SquidGuard package. It's not just buggy, but that code is completely unreadable and something I definitely won't touch. Would hugely benefit from rewriting from scratch.

spanguel

I've got HTTPS filtering running with squid+clamav+squidguard on pfsense.
If you want, i can send you my configs ive changed and try to get it running that way.
If that works for you too, we can make it public in this forum.

I still have to exclude some sites. for example some sites with certificate pinning activated…

pfsensation

@spanguel:

I've got HTTPS filtering running with squid+clamav+squidguard on pfsense.
If you want, i can send you my configs ive changed and try to get it running that way.
If that works for you too, we can make it public in this forum.

I still have to exclude some sites. for example some sites with certificate pinning activated…

Sounds great, I'd love to try it. How did you go about excluding sites? Excluding doesn't seem like a good method because more and more websites are jumping on the HSTS bandwagon, Github, Python, and of course Google, just to name a few. Furthermore, Squid guard also doesn't do any kind of content inspection, it just seems to work off a black list, I'm not sure, but I think I can't create different levels of user groups and block certain websites for certain people either.

ha100

Yes, hello. Can you please share the config to me. I am also having big https problems with squidguard. Some https sites working, others not.

aGeekhere

https://forum.pfsense.org/index.php?topic=112335.0

ha100

Thanks for that.

pfsensation

But what about actually decrypting and scanning?

aGeekhere

@pfsensation:

But what about actually decrypting and scanning?

Do you mean using the pfsense antivirus to scan https files? Or something else?

pfsensation

@aGeekHere:

@pfsensation:

But what about actually decrypting and scanning?

Do you mean using the pfsense antivirus to scan https files? Or something else?

I mean using pfsense to scan and look into SSL/HTTPS traffic to do certain things like identify new websites to block. Currently I am using OpenDNS + SquidGuard and Shalla and it pretty much sucks. If there's a new domain which isn't in those lists, it'll open. E2 Guardian seems like it's perfect for me but it's still got issues as I understand, and doesn't work as well as it should.

aGeekhere

The best way I found is to force google or bing into safe mode and use squidguard to catch the rest.