Guide to filtering web content (http and https) with pfsense 2.3

[DerHeldvomErdbeerfeld]

Hi there,

I just read this guide and also tried a little bit an the most I tried worked. Now I just have some
questions for better understanding.

@aGeekHere:

Guide to filtering web content (http and https) with pfsense 2.3 updated 09 June 2017

To stop users from bypassing your proxy setup a new firewall lan rule and block port 80 and 443
IPv4 TCP * * * 80 - 443 * none.
Save

If I set it up like this FROM 80 TO 443 (this is what it mean to me 80-443) didnt I block also all ports
between 80 and 443? Wouldnt it be better to set up two rules or an alias (Firewall/Aliases/Ports)
for the pot 80 and 443 or did I understood it wrong?

I also want to use a dual wan configuration, is there something to take care of?

–-
greets from Germany

DerHeldvomErdbeerfeld

[aGeekhere]

hi
1. I created two rules, one for port 80 and one for 443 (I will update this part of the guide)

2. I do not think squid works on dual wan

[Tecnergy]

Hi, thanks for this guide it has helped me a lot, bear with me pls no bully, I just installed pfsense

"the wpad will be setup to use port 80 and 443. The transperrent proxy is going to catch every thing that the wpad misses, enable transperrent proxy in squid once you have the wpad setup."

shouldn't be "the proxy will be setup…"?

Guide is missing configuration of mime type, I don't know which file to edit. I've read other guides and most of them say something about adding mime types for .da, .dat and .pac (if it's not there already)

Anyways, my problem is that I cannot download the files so no HTTPS filtering for me. I've made a folder inside www called wpad "www/wpad"
I cannot reach wpad.mydomain.local/wpad/wpad.da nor any other. Ping to wpad.mydomain.local responds to firewall IP

It's probably something dumb like a firewall rule but I can't see it.

-I have changed access to GUI to port 9999 and HTTP
-squid and squidguard services up
-with MiTM enabled I can't access HTTPS sites (because no downloading of proxy files I guess)
-Im running pfblocker too
-pfsense is my DHCP and DNS server
-have two WAN (I've read it squid proxy doesn't work with dual wan but it is blocking http OK like this)

Can you help me to realize whats wrong with the downloading of files for wpad?

thanks.

[Tecnergy]

Hi,

I managed to make it work by implementing another recurrent instance of nginx following another guide I found on the internet (I don't know if I'm allowed to post external links so I won't).

it seems to be working so far with 2 WAN's in load balance mode and it is filtering both http and https it seems with out MitM.

Thanks!

[J7529]

Hello,

Just following the Guide, I am getting a issue where when searching on https i get "This site can't be reached" and when browsing on http I get "The requested URL could not be retrieved" I am only trying to get to Google which is let through!

How I am currently setting up is behind a Router/Firewall and we already have a DNS server so I haven't enabled DNS Resolver, would this cause issues?

Thanks!

[MartinN]

@aGeekHere:

Go to Services/DNS/Resolver/General Settings
in custom option enter

server:
include: /var/unbound/forecegoogle.conf

You have mis-spelled the filename here, which prevents it from being saved by pfSense. Please fix it  8)

Thanks for the guide, excellent stuff!

[aGeekhere]

Fixed spelling

[nib01]

@nib01:

I got confused when you say add (blue). Did you mean add (blue) below yellow within the same wpad.da file? or..

"Part 3
Now we are going to set up a wpad read more here about wpad https://doc.pfsense.org/index.php/WPAD_Autoconfigure_for_Squid
ssh in to pfsense
8
cd /
create the wpad.da file
vi /usr/local/www/wpad.da
wq

Create two new symbolic link files
Code: [Select]
ln -s /usr/local/www/wpad.da /usr/local/www/wpad.dat
ln -s /usr/local/www/wpad.da /usr/local/www/proxy.pac

Then go Diagnostics /Edit File
click browse
user
local
www
click wpad.da
add

Code: [Select]
function FindProxyForURL(url, host)
{
    if (isPlainHostName(host) ||
        shExpMatch(host, "*.local") ||
        isInNet(dnsResolve(host), "192.168.1.0",  "255.255.255.0"))
        return "DIRECT";

return "PROXY 192.168.1.1:3128";
}

save"

[aGeekhere]

You have to create 3 files in /usr/local/www/
wpad.da
wpad.dat
proxy.pac

Each having this code

function FindProxyForURL(url, host) 
{ 
    if (isPlainHostName(host) ||
        shExpMatch(host, "*.local") ||
        isInNet(dnsResolve(host), "192.168.1.0",  "255.255.255.0"))
        return "DIRECT";

    return "PROXY 192.168.1.1:3128";
} 

Instead of maintaining changes for all 3 files you can create a main file e.g. wpad.da and create a symbolic link for the other 2 files, that way all changes from wpad.da are copied over to the other files.

So now you only need to make changes to wpad.da.

[kpoman]

Is there any way to force safesearch (google youtube etc…) if your DNS is on a Windows Server AD but your gateway pfSense is intercepting all http on a squid instance ?

[aGeekhere]

@kpoman:

Is there any way to force safesearch (google youtube etc…) if your DNS is on a Windows Server AD but your gateway pfSense is intercepting all http on a squid instance ?

Read the DNS resolver part in the guide and see if you can use that method.

[jopeme]

I see a problem with this option and it is that many of my clients have problems to watch YouTube videos, when they search from Google, since it shows that they are restricted videos and that you have to register.

Click add under Host overrides
Host = www
Domain = youtube.com
IP =  216.239.38.120
Description = youtube
Save
NOTE: Safe search for youtube is not as advanced as google safe search, which results in a lot of safe content be filtered out.

[vielfede]

Hello,
Maybe I missed something, I read the entire topic, but I did not find any post about access url on different port then standard 80/443….
For example if you try to access the following site:

http://fatturazione.comieco.org:82/anagrafica.php

The proxy respond with an error message.
Loooking to the squid log I noticed that when I tried to connect to http://fatturazione.comieco.org:82/ squid tried to connect still to port 80…
To get it work, I inserted an exception  on wpad configuration, something like

        if (shExpMatch(url, "http://fatturazione.comieco.org:82/*"))
        { return "DIRECT"; }

That allows the client to go out straight through the FW on port 82…. and hence now it works...

Indeed I'd like a solution to allow "different ports" work without WPAD exceptions. Otherwise you have to insert an exception for every site not using std port.
I did not notice anyone pointing out this kind of issue... Can I be the only one with this kind of issue?
Thanks in advance.

[aGeekhere]

I am not sure on this, if you use the transparent proxy then that only listens on port 80 and 443 for MITM.

[vielfede]

Finally I found out the solution… (http and https)
https://forum.pfsense.org/index.php?topic=96782.0

[KARMA]

I am trying to block facebook and youtube and some unwanted sites using Squid and SquidGuard filtering but those sites are't blocked. I can still access the blocked sites. Please can any one help me where i went wrong?

[pfrickroll]

This post is deleted!

[stuck]

@aGeekhere
First thank you for the detail guide. I'm running netgate 2.4.3_1 connected to TorguardVPN service and have had http transparent proxy (squid) running fine for some time. I've recently decided to add https filtering with squid/squidguard on netgate and WPAD on a separate http server. I can access the wpad files just fine using both the wpad.localdomain address or ip address. But the SSL filtering doesn't seem to work.
It seems on my system, transparent proxy only works on http sites. In order to get to https/ssl sites I have to explicitly setup my proxy address and port in windows.

Any help or hint is appreciated

[aGeekhere]

@stuck In squid SSL - Man In the Middle Filtering - HTTPS/SSL Interception tic Enable SSL filtering and SSL/MITM Mode set to splice all

[zonda]

Thank you for this guide aGeekHere!

Would you be able provide instructions on how to exclude websites from being proxied please? I have transparent proxy setup, no WPAD, SSL/MITM mode set to Splice All. Filtering web content is working great except that Microsoft Office 365 applications seem to be having connectivity issues. When user traffic is no longer being proxied, issue is resolved. Would like to exclude all Office 365 related domains/sites from the proxy.

Thank you again for any help or advice in advance.