Squid HTTPS/SSL filtering 2017 [Is this it?]

pfsensation

I wouldn't bother with MITM with a CA to be honest… If you are using WPAD your browser and SQUID both should be letting HTTPS traffic pass through properly. This happens even for HSTS based websites like Google.com. However this also negates the ability of ClamAV working, which, is useless anyways.

Is it even possible to work like this? From what I understand, I cannot do any HTTPS filtering without MITM as the traffic is encrypted.

@doktornotor:

There is no need to replace Squid, this is not a Squid issue. Anyway, see https://www.freshports.org/www/e2guardian/ (the GUI/pfSense package got never finished - https://github.com/pfsense/pfsense-packages/pull/866, https://forum.pfsense.org/index.php?topic=87526.0). As for squidGuard replacements, there's https://www.freshports.org/www/ufdbguard/ (not available in pfSense and obviously no GUI for that.)

Thanks for that, I will have a look at e2guardian, from the description, it does look like what I would wanna use but it's quite outdated. However, what I really wanted to know is how others are filtering HTTPS content using Pfsense in this day and age. I'm not an expert, but I know there's many hoops to jump through as HTTPS was designed so that inspection couldn't be done easily (HSTS for example).

I would really appreciate if someone who is already doing this could message, and give me pointers to the system they're using and how they're jumping through the hoops I mentioned above.

C0RR0SIVE

shrugs I don't use MITM and CA's on my network, but I use WPAD and I seem to get a "block page" any time I hit a blocked secure website, granted it doesn't show the actual block page since you can't intercept, so SQUID just breaks the tunnel.

Blocked Secure Site

Allowed Secure Site

I also have Transparent disabled… If you are wanting to get the block page that you would normally use however, not sure how it would be possible all because of HSTS.

pfsensation

@C0RR0SIVE:

shrugs I don't use MITM and CA's on my network, but I use WPAD and I seem to get a "block page" any time I hit a blocked secure website, granted it doesn't show the actual block page since you can't intercept, so SQUID just breaks the tunnel.

Blocked Secure Site

Allowed Secure Site

I also have Transparent disabled… If you are wanting to get the block page that you would normally use however, not sure how it would be possible all because of HSTS.

I'm having that same issue, but I think it maybe linked to chrome. The block page correctly displays on other browers, and devices. It's kinda sad really, I'm sure others have the setup I want but not sure if they're not willing to share. Or haven't seen this thread yet.

Squid is making me want to bash my head on the wall…

EDIT : The page has been fixed on Chrome now, after the latest update. I'm using Chrome beta.

sichent

This might be related - https://docs.diladele.com/faq/squid/cannot_connect_to_site_using_https.html
In short - browser ignores everything but 200 Connection established from initial CONNECT tunnel setup to remote HTTPS sites thorough proxy (hope this is not too technical)

pfsensation

@sichent:

This might be related - https://docs.diladele.com/faq/squid/cannot_connect_to_site_using_https.html
In short - browser ignores everything but 200 Connection established from initial CONNECT tunnel setup to remote HTTPS sites thorough proxy (hope this is not too technical)

Thank you for the reply @Sichent, so is Diladele the only filtering system available for pfsense that's actually actively updated and maintained? Doktornotor seems to be working extremely hard trying to fix Squid itself, but issues like this just remain. I haven't been able to use anything else decent enough that also does SSL inspection of sorts, I'm stuck relying on SquidGuard which is half broken and OpenDNS.

I'm not sure if others are not doing this, or just don't want to share their methods. But it seems bizarre to me, how can everyone be filtering without HTTPS… As most do just use Squid + Squid Guard.

doktornotor

Fixing Squid (pretty much done as of 0.4.33) sadly will not do anything for the SquidGuard package. It's not just buggy, but that code is completely unreadable and something I definitely won't touch. Would hugely benefit from rewriting from scratch.

spanguel

I've got HTTPS filtering running with squid+clamav+squidguard on pfsense.
If you want, i can send you my configs ive changed and try to get it running that way.
If that works for you too, we can make it public in this forum.

I still have to exclude some sites. for example some sites with certificate pinning activated…

pfsensation

@spanguel:

I've got HTTPS filtering running with squid+clamav+squidguard on pfsense.
If you want, i can send you my configs ive changed and try to get it running that way.
If that works for you too, we can make it public in this forum.

I still have to exclude some sites. for example some sites with certificate pinning activated…

Sounds great, I'd love to try it. How did you go about excluding sites? Excluding doesn't seem like a good method because more and more websites are jumping on the HSTS bandwagon, Github, Python, and of course Google, just to name a few. Furthermore, Squid guard also doesn't do any kind of content inspection, it just seems to work off a black list, I'm not sure, but I think I can't create different levels of user groups and block certain websites for certain people either.

ha100

Yes, hello. Can you please share the config to me. I am also having big https problems with squidguard. Some https sites working, others not.

aGeekhere

https://forum.pfsense.org/index.php?topic=112335.0

ha100

Thanks for that.

pfsensation

But what about actually decrypting and scanning?

aGeekhere

@pfsensation:

But what about actually decrypting and scanning?

Do you mean using the pfsense antivirus to scan https files? Or something else?

pfsensation

@aGeekHere:

@pfsensation:

But what about actually decrypting and scanning?

Do you mean using the pfsense antivirus to scan https files? Or something else?

I mean using pfsense to scan and look into SSL/HTTPS traffic to do certain things like identify new websites to block. Currently I am using OpenDNS + SquidGuard and Shalla and it pretty much sucks. If there's a new domain which isn't in those lists, it'll open. E2 Guardian seems like it's perfect for me but it's still got issues as I understand, and doesn't work as well as it should.

aGeekhere

The best way I found is to force google or bing into safe mode and use squidguard to catch the rest.

pfsensation

@aGeekHere:

The best way I found is to force google or bing into safe mode and use squidguard to catch the rest.

That's pretty old and basic. I'm already using those but it's not good enough. What about uncategorized websites, that's where OpenDNS, shalla list and these methods really fail.

aGeekhere

@pfsensation:

@aGeekHere:

The best way I found is to force google or bing into safe mode and use squidguard to catch the rest.

That's pretty old and basic. I'm already using those but it's not good enough. What about uncategorized websites, that's where OpenDNS, shalla list and these methods really fail.

Well you cannot find these uncategorized websites using google/bing safe search so someone would have to find a new website (not using safe search) that is not added to shalla list yet and go to that website direct. Well like with all things if there is a will there is a way.

andyschmid

@doktornotor:

@pfsensation:

it seems squid is changing the domain from the domain typed in to "http"

No, it's not. See https://redmine.pfsense.org/issues/6777#note-2

If Squidguard is blocking Google for you, that's unfortunate, you'll need to fix that issue instead. (No idea how, the package is broken and the code not maintainable.)

@doktornotor:

Fixing Squid (pretty much done as of 0.4.33) sadly will not do anything for the SquidGuard package. It's not just buggy, but that code is completely unreadable and something I definitely won't touch. Would hugely benefit from rewriting from scratch.

So this looks like the SquidGuard package shouldn't be used at all which leads me to 2 questions:

With 2.3 a lot of packages got cleaned up that weren't maintained. So if SquidGuard isn't maintained and broken why didn't it get removed as well?
If SquidGuard is a no go and the other two solutions e2guarding and ufDBGuard are not available as packages how do we do content filtering on pfSense?

While I understand all the issues mentioned just keen to understand what's the solution moving forward.

Thanks!

pfsensation

@andyschmid:

@doktornotor:

@pfsensation:

it seems squid is changing the domain from the domain typed in to "http"

No, it's not. See https://redmine.pfsense.org/issues/6777#note-2

If Squidguard is blocking Google for you, that's unfortunate, you'll need to fix that issue instead. (No idea how, the package is broken and the code not maintainable.)

@doktornotor:

Fixing Squid (pretty much done as of 0.4.33) sadly will not do anything for the SquidGuard package. It's not just buggy, but that code is completely unreadable and something I definitely won't touch. Would hugely benefit from rewriting from scratch.

So this looks like the SquidGuard package shouldn't be used at all which leads me to 2 questions:

With 2.3 a lot of packages got cleaned up that weren't maintained. So if SquidGuard isn't maintained and broken why didn't it get removed as well?

If SquidGuard is a no go and the other two solutions e2guarding and ufDBGuard are not available as packages how do we do content filtering on pfSense?

While I understand all the issues mentioned just keen to understand what's the solution moving forward.

Thanks!

It's a big doddle!

E2 Guardian is actually quite fantastic, I've been testing it in a virtual machine. It's way better than SquidGuard for sure, however it still has this issue of not working with chrome as it still uses Squid as a upstream proxy. I've found this website explaining a bit more : http://stackoverflow.com/questions/43665243/chrome-invalid-self-signed-ssl-cert-subject-alternative-name-missing

However, anyone reading this. Remember that E2 Guardian is still unofficial and the GUI isn't perfect. But so far it seems to work half decently, and be able to phrase match - Something I always wanted. It's stupid to rely solely on URL filtering these days, you want your firewall to be able to scan a website and see if it should be blocked or allowed based on whats on the page. This also means that any new URL or proxy that may slip through the nets will also be blocked.

fptelecom

I used PFSENSE for almost two years. It is an AWESOME firewall and router, capable to support about 500 users concurrently whit just a CPU and a couple of good NICs. It's just sad to see one of their "killer apps", the https web filtering falling down defeated and without any hope. Is this the begining of the end for PFSENSE? I hope not, but unfortunaty I need to search another options. Push play baby!!! GOODBYYYYYEEEEE MOON MEN, GOODBYYYYYEEEE GOODBYYYYYEEEE…...... :'(