HTTPS Filtering, once and for all!

Draven666

Dear pfSense users,

I don't want to start a debate but the fact is, I probably will :D . Ok, here's my situation. I'm configuring a pfSense box for a public hotspot with a 8Mbps/1Mbps link so, traffic management and filtering is quite important if I don't want one user to steal all the available bandwidth. So, I've installed pfSense, Squid and Squidguard and I'm at the point where I have to choose between transparent proxy with man-in-the-middle or non-transparent proxy for my http/https filter. From my reading, both seems to have advantages and disadvantages so, I think it's time to close the debate once and for all. Which one would you choose and why?

For what I understand, transparent proxy with man-in-the-middle won't require me to manually configure all hotspot connected devices with proxy settings but will issue a warning on the user's browser so, it may represent a problem. I don't want to explain this to everyone coming at the front desk and asking if their devices will be compromised if they add an exception nor that I want to compromise the security of my business and my clients. So, it leads me to non-transparent proxy…

But, again, there's a problem. Non-transparent proxy needs configuration on the client's devices except if I use WPAD and PAC files to deliver the configurations options. But, it seems that Android devices (and maybe some others...I don't know), don't offer support for WPAD (https://code.google.com/p/android/issues/list?can=2&q=wpad&colspec=ID+Status+Priority+Owner+Summary+Stars+Reporter+Opened&cells=tiles) without the use of third party applications. In a public hotspot, that's a MAJOR issue. Again, it seems like people at the front desk will have to do a lot of troubleshooting, so, bad idea.

So, from my point of view, it seems that we don't have any CLEAR winner here. I want a simple and clean solution. I don't know how many clients I'll be able to serve until the end of the summer but I clearly don't want to configure all their devices, one by one or troubleshoot every end-user who only wants to surf on the net. And since I won't be on-site 24/7, I would like a SIMPLE solution. A solution easily manageable by computer illiterate peoples.

Waiting for your answer,
Draven

P.S.: Sorry for my english, I'm french.

pfBasic

For what it's worth, I'm sending this message from a Galaxy S7 running stock Android Marshmallow and I'm using Squid with Squid guard and clamAV via a WPAD .PAC file. I think it's older Android OS that doesn't support it?

I did have to change my wifi settings to auto configure the proxy settings and point it towards the file on my pfSense box. It took about 15 seconds but that might be too much to ask if you have a lot of clients coming in and out?

pfBasic

What about using pfSense traffic shaper for bandwidth management and using a DNS service to blacklist URLs and force your interface to use only that DNS? That way you don't have to mess with squid & SSL. I don't think there is a clean solution to filter SSL to a bunch of different clients who aren't computer literate or simply don't want to be bothered.

chris4916

Draven (BTW, I'm French to and can read your English w/o any problem ;)),

I don't think there is any clean and easy solution, especially if your goal is to filter HTTPS "content".
Few years ago, I wrote a short paper for another platform trying to explain what are the main differences between transparent and explicit proxy.
Given what you explain, I'm sure you understand this already.

The only point is potentially with anti-virus witch doesn't work for HTTPS without MITM but deciding to go for MITM design, transparently or explicitly is not that simple. Aside the "no, you certificate or web site is not corrupted" stuff (and BTW, yes you encrypted access IS corrupted! :o) , French law requires, especially for public access, that you inform your customers, in any case, that you can access their bank account if they access it through platform you provide to them.

I'm, kind of, joking. French law doesn't require this explicitly but you do need to inform about this risk and need to make it public.

Thus, in term of design, transparent proxy is very appealing but is not, from legal perspective, so simple, not to say that this is less secure and much more exposed to attacks.

This been said, configuring each and every device not able to benefit from WPAD is painful and there is no magic. DNS stuff is only a workaround. If you need these devices to access through proxy without WPAD help, then you have to inform users about "howto" and to me, simplest way is to set up captive portal with page explaining "how to" so that each impacted user can do it by itself.

Before going there, be sure you have deployed all the WPAD mechanisms ;) then deal with exceptions manually… and think twice about MITM.... and transparent proxy

lavd

here is the answer for transparent
just do
acl blocked_sites ssl::server_name .facebook.com
or
acl white_sitesreg ssl::server_name_regex (facebook)

https://forum.pfsense.org/index.php?topic=117339.0

chris4916

@lavd:

here is the answer for transparent
just do
acl blocked_sites ssl::server_name .facebook.com
https://forum.pfsense.org/index.php?topic=117339.0

This doesn't work, at least with proxy in transparent mode without SSL-bump.
Why? because with transparent proxy, only HTTP (port 80) is (transparently) redirected to proxy.
HTTPS is not and therefore goes directly to internet.

Then if decision is to deploy SSL-Bump, HTTPS is filtered like HTTP.

lavd

For this to work I have "Enable SSL filtering" with below of course

ssl_bump terminate blocked_sites
ssl_bump peek all
ssl_bump splice all

chris4916

@lavd:

For this to work I have "Enable SSL filtering" with below of course

What I meant to say is that SSL-Bump is useful if you want to analyse content, not URL
If goal is only to prevent access to facebook, then you are not obliged to enable SSL-Bump… but you can't filter in transparent mode.

In term of design, transparent proxy has more cons than pros.
e.g. have a look at this ;) Quite old but nevertheless illustrate how inserting transparent feature in the middle may have negative side effect.

You will easily find other information like this or this directly linked to transparent proxy implementation.

Think twice before going in this direction because it looks so easy.

There is no such thing as a free lunch ;)

Kababayan

user have to install that certificate if we want to cache and analyze the content of encrypted traffic in squid that what I understand. Having a setting like this I just disable ssl natting in NAT. If anyone has a better solution without having a certificate problem while caching encrypted content would like to hear it also.

chris4916

For sure if goal is to inspect HTTPS content (for content filtering or caching purpose), then there is no other option thant SSL-bump ::)

Issue with certificate is also due to self-signed certificates or signed with certificate authority that is not known (trusted) by browsers.

lavd

no doubt transparent proxy has issues, as everything else does; that's why we have this version numbers increasing

doubt I was in search of fee lunch with transparent proxy, the main concern is making it easy for users
setting up those certificates is creating to much mess and is definitely not an option unless you seat next to those people all the time

so the only real solution is ssl_bump, and seems to work so far - https://facebook.com would just not load
no fancy "blocked by …" but works

Kababayan

OP is not just filtering URL he is also caching and analyzing the content as well so he need to bumb those certificates and users has to install proxy certificate to use https proxy. my solution is disable https transparancy without making any wpad. some user might not have wpad service enabled, just add no redirect option for port 443 to make https not transparent. with squidguard he can still filter https traffic.

chris4916

Well… HTTP proxy can run in both explicit and transparent mode and this has nothing to do with SSL-Bump.
These are 2 different aspects and the only relationship is when you want/need to deal with HTTPS in transparent mode or if you want/need to analyse HTTPS content.

However, what needs to be understood is that from browser view point, proxy is either defined (whatever the way you define it) or not. If proxy is defined, then this is an explicit proxy. If not, this is transparent proxy. Which means that if proxy side, one can have both in parallel, client side (browser) you have only 3 choices with no overlap:

explicit proxy
no proxy -> with transparent proxy intercepting
no proxy... without proxy ;D

If SSL-bump is configured (whatever proxy mode, explicit or transparent), trusting certificate generated to intercept HTTPS flow is mandatory to prevent warning messages.
This often means to deploy, client side, CA public key to be trusted.

Keep in mind we are only discussing technical aspects here, not all the legal aspects with HTTPS flow being broken and intercepted.

While designing your proxy, you have to determine whether your goal is to filter access (e.g. prevent facebook access) which doesn't require any SSL-bump, even for HTTPS, but does require explicit proxy or if you need/want to intercept HTTPS, which means SSL-Bump thus certificate.