Guide to filtering web content (http and https) with pfsense 2.3

mahnonsaprei · Jun 30, 2017, 2:08 PM

Thank you so much. There aren't alternatives to redirect https via proxy also blank page or other?

B.r.

techbee · Jul 3, 2017, 9:45 AM

hello ageekhere,

can you update your guide now.

aGeekhere · Jul 3, 2017, 1:44 PM

what part are you stuck on?

techbee · Jul 3, 2017, 2:03 PM

1. I dont know what exact firewall rules to add
2. I am confused if I need to install unofficial wpad package of marcelloc.
3. I am confused if there is a need to have separate webserver of wpad.
4. I am confused if I can enable both transparent and ssl mitm filtering on squid.
5. I cant nslookup wpad on clients command prompt to resolve to my pfsense box ip.

aGeekhere · Jul 4, 2017, 3:15 AM

The firewall rules that you need are


IPv4+6 TCP	*	*	*	80 (HTTP)	*	none	 	port 80 block
IPv4+6 TCP	*	*	*	443 (HTTPS)	*	none	 	port 443 block

This blocks access to port 80 and 443, it stops users from bypassing the proxy.

Also if you want to force pfsense to be the DNS server use.


In Firewall/NAT/Port forward 
add a new rule

Interface = LAN 
Protocol = TCP/UDP
Source ports = *
Dest address = *
Dest ports = 53
NAT IP = 127.0.0.1
NAT Ports = 53
Description = Redirect DNS
LAN TCP/UDP * * * 53 127.0.0.1 53 Redirect DNS
Save

Have a look through a few of these videos https://www.youtube.com/playlist?list=PLE726R7YUJTePGvo0Zga2juUBxxFTH4Bk
There are a few really good configs he does there.

2. Wait for the package to become stable and available in the package manager, in the mean time you can use the pfsense webserver, put the wpad files in /usr/local/www/ read Part 3. (all that needs to be done is add the wpad files (wpad.da, proxy.pac and wpad.dat) to /usr/local/www/).

3. Just use pfsense webserver /usr/local/www/ (I have not used an external webserver)

4. This is what I did, I set up a WPAD and transparent proxy with ssl mitm (splice all so you do not have to install cert on every device) and set your device to auto configure. I found that just using WPAD a lot of programs that cannot use the proxy were getting blocked so the transparent proxy catches that and something the transparent proxy ssl has issues (like windows updates) however the WPAD is used for that. Best of both worlds.

5 I have had issues as well, if you put http://192.168.1.1/wpad.dat in your web browser and it downloads than it is working.

This guide goes a few steps fever by forcing google and bing to be in safe mode (even youtube).

So what I would do is get WPAD working then enable the transparent proxy with ssl mitm (create a cert for the router).

Hope this helps

techbee · Jul 4, 2017, 4:19 AM

thanks for the reply aGeekHere.

What is the tcp port that you used on your webconfigurator ? If I block port 80, then, I will also be blocked opening the pfsense gui.

Also, regarding to my question number 4, do you mean that you did not enable the http transparent proxy on port 3128 but only the HTTPS/SSL MITM on port 3129 and setup wpad. did I understood you correctly ?

aGeekhere · Jul 4, 2017, 4:58 AM

What is the tcp port that you used on your webconfigurator ?

set pfsense Protocol to http (This is a MUST, it will not work if you do not do this)
System: Advanced: Admin Access Protocol http

If I block port 80, then, I will also be blocked opening the pfsense gui.

No it should not (have the rule above the anti lockout rule);

Also, regarding to my question number 4

Enable squid proxy 3128
Get WPAD working follow guide

Enable transparent mode to forward all requests for destination port 80 to the proxy server.

Enable SSL filtering
Splice all
create Certificate (for pfsense).

Summery
Set Advanced: Admin Access Protocol to http
Enable squid proxy
Follow guide on setting up WPAD
Enable transparent mode
Enable SSL filtering Splice all

techbee · Jul 4, 2017, 5:09 AM

Yep, done that all.

I can download the wpad files from the browser. I can access the http sites but still cannot access the https sites.

aGeekhere · Jul 4, 2017, 5:22 AM

what web browser?

Clear your cache?

What Operating system?

Operating system update work?

Did you set auto configure proxy?

Try Diagnostics States States clear?

What sites, google?

techbee · Jul 4, 2017, 5:41 AM

@aGeekHere:

what web browser?

Clear your cache?

What Operating system?

Operating system update work?

Did you set auto configure proxy?

Try Diagnostics States States clear?

What sites, google?

'

Hi aGeekHere,

Yes, cache is cleared,
os=win7 pro sp1,
I disabled the update coz I dont need it,
autoconfigure is set,
yes, google and yahoo https sites. facebook and youtube does not load and shows error.

I use splice-all. Don't I really need to install certs to client device ?

If https site is successfully blocked, what error page do I expect to see ? Can you show me the screenshot for the error page that will show !

aGeekhere · Jul 4, 2017, 5:52 AM

firefox ie or chrome?

I use splice-all. Don't I really need to install certs to client device ?

not when splice all

If https site is successfully blocked, what error page do I expect to see ?

The default cannot access page of the web browser.

If you enabled squid, transparent mode and SSL filtering Splice all with a cert on pfsense and you still cannot access https sites, then something is broken.
Try reinstall.
If you still have the same issue post screen shots of your setup.

techbee · Jul 4, 2017, 6:17 AM

@aGeekHere:

firefox ie or chrome?

I use splice-all. Don't I really need to install certs to client device ?

not when splice all

If https site is successfully blocked, what error page do I expect to see ?

The default cannot access page of the web browser.

If you enabled squid, transparent mode and SSL filtering Splice all with a cert on pfsense and you still cannot access https sites, then something is broken.
Try reinstall.
If you still have the same issue post screen shots of your setup.

I am using chrome. most of the users are chrome.
On the other hand, the firewall rules you shown was to block port 80 and 443 right and redirect dns resolution to pfsense dns server and no more else. That means, I don't need to redirect traffic from port 80 to 3128 and 443 to 3129 right ?

aGeekhere · Jul 4, 2017, 6:32 AM

That means, I don't need to redirect traffic from port 80 to 3128 and 443 to 3129 right

That is what the transparent proxy is doing, remove that rule, that could be what is causing the issue.

techbee · Jul 4, 2017, 7:07 AM

@aGeekHere:

That means, I don't need to redirect traffic from port 80 to 3128 and 443 to 3129 right

That is what the transparent proxy is doing, remove that rule, that could be what is causing the issue.

Alright

So what I have now on my firewall rules are:
1. antilock rule on port 80 which says (allow all on tcp port 80)
2. block LAN_NET on tcp port 80 and 443
3. redirect rule for dns resolution

would that be ok ?

On the other hand, what is the default cannot access page of the web browser, is it the same like of that of http transparent proxy error page ?

techbee · Jul 5, 2017, 4:37 AM

aGeekHere,

My wpad now works. I can download it from chrome browser.
I can ping and nslookup wpad now.
All squidguard group ACL and its target rules are applied to block facebook and youtube for testing purposes for now.
I HAVE NOT INSTALLED THE CA CERT ON CLIENT DEVICES.

I used http and https filtering with splice all option so that I dont need to install the cert to client devices.

I also cleared browser cache and history.

The result is I can browse http but cannot browse https. Https sites like yahoo and google mail works though.

I have screenshot of the error for browsing https site.

aGeekhere · Jul 5, 2017, 5:11 AM

Is squidguard blocking it by mistake? is it set to deny all?

Try allowing all in squidguard rules to test.

Post screen shots of squidguard.

Try clearing states in pfsense.

Reboot.

techbee · Jul 5, 2017, 6:13 AM

Squidguard is not blocking by mistake.

I tried clearing firewall states and Rebooted, still the same situation.

techbee · Jul 5, 2017, 6:27 AM

I also have this wireshark done where I can see that it made a CONNECT method to facebook, then it has established connection, send client hello but later have RST and there is no SERVER HELLO thus no application data received.

What could I miss here.

aGeekhere · Jul 5, 2017, 7:03 AM

lets try without the wpad and use just the transparent, set proxy direct does it work then?
If you still cannot access https sites then post you squid config.

techbee · Jul 5, 2017, 7:08 AM

@aGeekHere:

lets try without the wpad and use just the transparent, set proxy direct does it work then?
If you still cannot access https sites then post you squid config.

If I use only the http proxy without the ssl mitm and set browser to auto, then the http proxy filter works.

If you are saying to setup http transparent proxy and ssl mitm filtering and install cert to client device and set browser proxy, then I have not done this yet.