Guide to filtering web content (http and https) with pfsense 2.3

vielfede · Jun 19, 2017, 2:37 PM

If I use it in transparent mode, https does not work!
Better sometimes it works, sometimes it does not! (http works always!)

If anyone managed to get http+https in splice all + transparent mode work please let me know… ;)

aGeekhere · Jun 19, 2017, 11:57 PM

I just enabled it mitm, added a cert, set it to splice all and its working.

techbee · Jun 21, 2017, 2:51 AM

The guide is update but confusing. Can you please clean up the guide and have it step by step. Its somewhat hard to follow if you are not familiar with wpad and related firewall rules.

Thanks for the guide.

aGeekhere · Jun 21, 2017, 7:33 AM

hi techbee, yeah after discovering a few new things the guide is a little messy now.

The transparent proxy with MITM (splice all) vs wpad should really be divided into two different choices. When I made the guide I did not know that all you need to get transparent proxy for https working is set splice all and create a cert for the router. So in fact you can now choose either using a wpad or the transparent proxy MITM or both.

when i get time I will try an clean it up a bit.

vielfede · Jun 23, 2017, 7:56 AM

@aGeekHere:

The transparent proxy with MITM (splice all) vs wpad should really be divided into two different choices. When I made the guide I did not know that all you need to get transparent proxy for https working is set splice all and create a cert for the router. So in fact you can now choose either using a wpad or the transparent proxy MITM or both.

Indeed I do not think MITM and splice all can work toghether, as stated on squid documentation MITM is associated to the "bump" directive that is something complety different from splice directive.
With MITM (bump) squid is able to decrypt traffic (and analyse it) meanwhile with splice all you can do is just "web filtering".
Summarizing
Directive Advantages/features Disadvantage
Splice (all) No needs of certificate installation on Clients + webfiltering No traffic analysis i.e no AntiVirus
Bump (MITM) Traffic analysis i.e noYES AntiVirus + webfiltering Needs to install certs on clients

Please take a look here and let me know if I missed something. Thanks
http://wiki.squid-cache.org/Features/SslPeekAndSplice#Actions
http://marek.helion.pl/install/squid.html

AR15USR · Jun 22, 2017, 5:08 PM

@vielfede:

@aGeekHere:

The transparent proxy with MITM (splice all) vs wpad should really be divided into two different choices. When I made the guide I did not know that all you need to get transparent proxy for https working is set splice all and create a cert for the router. So in fact you can now choose either using a wpad or the transparent proxy MITM or both.

Indeed I do not think MITM and splice all can work toghether, as stated on squid documentation MITM is associated to the "bump" directive that is something complety different from splice directive.
With MITM (bump) squid is able to decrypt traffic (and analyse it) meanwhile with splice all you can do is just "web filtering".
Summarizing
Directive Advantages/features Disadvantage
Splice (all) No needs of certificate installation on Clients + webfiltering No traffic analysis i.e no AntiVirus
Bump (MITM) Traffic analysis i.e no AntiVirus + webfiltering Needs to install certs on clients

Please take a look here and let me know if I missed something. Thanks
http://wiki.squid-cache.org/Features/SslPeekAndSplice#Actions
http://marek.helion.pl/install/squid.html

Shouldn't the second one not have the word "no"..

vielfede · Jun 23, 2017, 7:58 AM

@AR15USR:

Shouldn't the second one not have the word "no"..

Sorry cut & paste mistake, I fixed it.

mahnonsaprei · Jun 29, 2017, 9:59 PM

Hi at all,

I'm Marcello from italy. Nice to meet you, i'm a newbie of pfsense.

I have read the tutorial, but i have one problem.
When i try to connect to an https website in blacklist, the browser show me a generic error see the attachment: (ERR_TUNNEL_CONNECTION_FAILED) while i want to see a message of pfsense that explain the block. In http it's ok

I'm looking for in google but i not found the answer.

Please help me.
Thank you veeeeery much!!!

aGeekhere · Jun 30, 2017, 12:10 AM

When i try to connect to an https website in blacklist, the browser show me a generic error see the attachment: (ERR_TUNNEL_CONNECTION_FAILED) while i want to see a message of pfsense that explain the block. In http it's ok

Known issue with squid, I do not think there is a fix for it.

mahnonsaprei · Jun 30, 2017, 2:08 PM

Thank you so much. There aren't alternatives to redirect https via proxy also blank page or other?

B.r.

techbee · Jul 3, 2017, 9:45 AM

hello ageekhere,

can you update your guide now.

aGeekhere · Jul 3, 2017, 1:44 PM

what part are you stuck on?

techbee · Jul 3, 2017, 2:03 PM

1. I dont know what exact firewall rules to add
2. I am confused if I need to install unofficial wpad package of marcelloc.
3. I am confused if there is a need to have separate webserver of wpad.
4. I am confused if I can enable both transparent and ssl mitm filtering on squid.
5. I cant nslookup wpad on clients command prompt to resolve to my pfsense box ip.

aGeekhere · Jul 4, 2017, 3:15 AM

The firewall rules that you need are


IPv4+6 TCP	*	*	*	80 (HTTP)	*	none	 	port 80 block
IPv4+6 TCP	*	*	*	443 (HTTPS)	*	none	 	port 443 block

This blocks access to port 80 and 443, it stops users from bypassing the proxy.

Also if you want to force pfsense to be the DNS server use.


In Firewall/NAT/Port forward 
add a new rule

Interface = LAN 
Protocol = TCP/UDP
Source ports = *
Dest address = *
Dest ports = 53
NAT IP = 127.0.0.1
NAT Ports = 53
Description = Redirect DNS
LAN TCP/UDP * * * 53 127.0.0.1 53 Redirect DNS
Save

Have a look through a few of these videos https://www.youtube.com/playlist?list=PLE726R7YUJTePGvo0Zga2juUBxxFTH4Bk
There are a few really good configs he does there.

2. Wait for the package to become stable and available in the package manager, in the mean time you can use the pfsense webserver, put the wpad files in /usr/local/www/ read Part 3. (all that needs to be done is add the wpad files (wpad.da, proxy.pac and wpad.dat) to /usr/local/www/).

3. Just use pfsense webserver /usr/local/www/ (I have not used an external webserver)

4. This is what I did, I set up a WPAD and transparent proxy with ssl mitm (splice all so you do not have to install cert on every device) and set your device to auto configure. I found that just using WPAD a lot of programs that cannot use the proxy were getting blocked so the transparent proxy catches that and something the transparent proxy ssl has issues (like windows updates) however the WPAD is used for that. Best of both worlds.

5 I have had issues as well, if you put http://192.168.1.1/wpad.dat in your web browser and it downloads than it is working.

This guide goes a few steps fever by forcing google and bing to be in safe mode (even youtube).

So what I would do is get WPAD working then enable the transparent proxy with ssl mitm (create a cert for the router).

Hope this helps

techbee · Jul 4, 2017, 4:19 AM

thanks for the reply aGeekHere.

What is the tcp port that you used on your webconfigurator ? If I block port 80, then, I will also be blocked opening the pfsense gui.

Also, regarding to my question number 4, do you mean that you did not enable the http transparent proxy on port 3128 but only the HTTPS/SSL MITM on port 3129 and setup wpad. did I understood you correctly ?

aGeekhere · Jul 4, 2017, 4:58 AM

What is the tcp port that you used on your webconfigurator ?

set pfsense Protocol to http (This is a MUST, it will not work if you do not do this)
System: Advanced: Admin Access Protocol http

If I block port 80, then, I will also be blocked opening the pfsense gui.

No it should not (have the rule above the anti lockout rule);

Also, regarding to my question number 4

Enable squid proxy 3128
Get WPAD working follow guide

Enable transparent mode to forward all requests for destination port 80 to the proxy server.

Enable SSL filtering
Splice all
create Certificate (for pfsense).

Summery
Set Advanced: Admin Access Protocol to http
Enable squid proxy
Follow guide on setting up WPAD
Enable transparent mode
Enable SSL filtering Splice all

techbee · Jul 4, 2017, 5:09 AM

Yep, done that all.

I can download the wpad files from the browser. I can access the http sites but still cannot access the https sites.

aGeekhere · Jul 4, 2017, 5:22 AM

what web browser?

Clear your cache?

What Operating system?

Operating system update work?

Did you set auto configure proxy?

Try Diagnostics States States clear?

What sites, google?

techbee · Jul 4, 2017, 5:41 AM

@aGeekHere:

what web browser?

Clear your cache?

What Operating system?

Operating system update work?

Did you set auto configure proxy?

Try Diagnostics States States clear?

What sites, google?

'

Hi aGeekHere,

Yes, cache is cleared,
os=win7 pro sp1,
I disabled the update coz I dont need it,
autoconfigure is set,
yes, google and yahoo https sites. facebook and youtube does not load and shows error.

I use splice-all. Don't I really need to install certs to client device ?

If https site is successfully blocked, what error page do I expect to see ? Can you show me the screenshot for the error page that will show !

aGeekhere · Jul 4, 2017, 5:52 AM

firefox ie or chrome?

I use splice-all. Don't I really need to install certs to client device ?

not when splice all

If https site is successfully blocked, what error page do I expect to see ?

The default cannot access page of the web browser.

If you enabled squid, transparent mode and SSL filtering Splice all with a cert on pfsense and you still cannot access https sites, then something is broken.
Try reinstall.
If you still have the same issue post screen shots of your setup.