Guide to filtering web content (http and https) with pfsense 2.3
-
Thanks, will update it soon.
Done
-
Is "click wpad.da" and other "wpad.da" a typo?
If it's correct, it might be worth commenting after it that you do mean "da" not "dat", because having two files called wpad.da and wpad.dat might not be noticed, looks like a typo, is confusing, etc :) -
Hi, there are 3 wpad files
wpad.dat
wpad.da
proxy.pac
https://en.wikipedia.org/wiki/Web_Proxy_Autodiscovery_ProtocolI made the wpad.da the main file you edit and made a symbolic link for wpad.dat and proxy.pac (so all you need to do is just edit the wpad.da file).
If this is still confusing me know and I will update the guide. -
thanks for the guide ;)
-
Thank you for the guide!
Some guides have a step to assign MIME file types eg: ".wpad" => "application/x-ns-proxy-autoconfig",
Is this not necessary if using the webconfig http file server?
-
Some guides have a step to assign MIME file types eg: ".wpad" => "application/x-ns-proxy-autoconfig",
Is this not necessary if using the webconfig http file server?
I do not believe it is needed.
-
Can you please expand on the "use our route as the DNS server." part? Your list of settings is not very complete compared to the options available in the WebUI. Maybe be a little more specific, I'm counting 13 or so available line items available.
edit: Also, regarding the new firewall rule, which level on the Firewall list hierarchy do we here to put this new rule? If we put it at the top will we not block ourselves out?
Also, how/where does the certs for https inspection get created/used?
Thanks
-
Can you please expand on the "use our route as the DNS server." part?
Sure, since we enabled the DNS resolver to do Host overrides to force safe search on a few search engines I also set up a rule that will force the local network to use the pfsense router as the DNS server. The pfsense router will cache DNS addresses and use them instead of calling a DNS server, you can also change the cache size.
Read more here https://doc.pfsense.org/index.php/Unbound_DNS_ResolverYour list of settings is not very complete compared to the options available in the WebUI. Maybe be a little more specific, I'm counting 13 or so available line items available.
Depending on your setup this could change, this is what I use
Tic Enable DNS resolver
Listen Port default which is 53
Network Interfaces all
Outgoing Network Interfaces all
System Domain Local Zone Type transperrent
Tic Enable DNSSEC Support
Untic Enable Forwarding Mode
Tic Register DHCP leases in the DNS Resolver
Tic Register DHCP static mappings in the DNS ResolverFor advance settings
Tic
Tic
Tic
Tic
Tic
Set cache size (I set 100MB)
10
10
4096
513
200
86400
0
15 min
20000
Disable
1
Untic
UnticAlso, regarding the new firewall rule, which level on the Firewall list hierarchy do we here to put this new rule? If we put it at the top will we not block ourselves out?
It should be just under the anti block out rule, so the second rule
Also, how/where does the certs for https inspection get created/used?
WPAD does not use certs, as there is no man in the middle attack
Hope this helps
-
Thanks aGeek
Actually I was referring to the "Firewall/NAT/Port forward" settings but I got it figured out anyhow.
WPAD does not use certs, as there is no man in the middle attack
Hope this helps
After using your guide, can I enable the MITM section in Squid, step up the CA and certs and it should then inspect https?
Also, under System/General/DNS Server Settings should we have/not have any DNS servers listed? I'm guessing we need some listed as this is where pfsense will get its DNS cache from?
-
After using your guide, can I enable the MITM section in Squid, step up the CA and certs and it should then inspect https?
No, the wpad is doing that, the reason why we are using a wpad is so we do not have to use certs. So no transperrent proxy or man in the middle. Do not enable these options!
Also, under System/General/DNS Server Settings should we have/not have any DNS servers listed? I'm guessing we need some listed as this is where pfsense will get its DNS cache from?
You should include your isp dns (there is an option for that) and add the fastest DNS servers in your area, fill the list up, the DNS cache will get build up as you search the net, it does not just download every DNS address from the DNS server, you would run out of ram if it did.
I hope that I have cleared a few things up, have you got your setup working?
-
After using your guide, can I enable the MITM section in Squid, step up the CA and certs and it should then inspect https?
No, the wpad is doing that, the reason why we are using a wpad is so we do not have to use certs. So no transperrent proxy or man in the middle. Do not enable these options!
Also, under System/General/DNS Server Settings should we have/not have any DNS servers listed? I'm guessing we need some listed as this is where pfsense will get its DNS cache from?
You should include your isp dns (there is an option for that) and add the fastest DNS servers in your area, fill the list up, the DNS cache will get build up as you search the net, it does not just download every DNS address from the DNS server, you would run out of ram if it did.
I hope that I have cleared a few things up, have you got your setup working?
no, not working. As soon as I enable the firewall rule there is no access to the internet. With the firewall rule disabled I have access but the eicar test files do not get blocked like they did when in transparent mode indicating the proxy is not being used.
-
Can't get HTTPS scanning to work. See https sites passing through squid but clamd won't scan the files. Works fine on HTTP and it catches the virus files. Using proxy setting in browser since wpad isn't giving the results I am expecting.
-
no, not working. As soon as I enable the firewall rule there is no access to the internet. With the firewall rule disabled I have access but the eicar test files do not get blocked like they did when in transparent mode indicating the proxy is not being used.
Is that the port 80 and 443 block rule? If so than that is correct, now set your device to auto configure proxy. In windows go to global internet settings and there is an option for that, and for each program you have set its proxy settings. For programs with no proxy settings create a pass rule.
Can't get HTTPS scanning to work. See https sites passing through squid but clamd won't scan the files. Works fine on HTTP and it catches the virus files. Using proxy setting in browser since wpad isn't giving the results I am expecting.
Hi, first try above post, second I have not tried clamd scanning because I have found it to have issues, best to ask this question in the proxy forum.
Using proxy setting in browser since wpad isn't giving the results I am expecting
You have to tell your browser to use system settings and in global internet settings set proxy to auto configure.
-
Can't get HTTPS scanning to work. See https sites passing through squid but clamd won't scan the files. Works fine on HTTP and it catches the virus files. Using proxy setting in browser since wpad isn't giving the results I am expecting.
Same issue here now.
I enabled the 80-443 block rule, unchecked the 'Transparent" option in Squid and I can only get access if I manually enter the wpad.dat location into my local computer(s) settings. Auto discovery does not work. I'm on all Apple computers/devices here btw. Same issue with ClamAV, it scans http but not https as poster above.
-
Auto config works for me. What I meant in the original post was that the WPAD direction info I was writing was not proper hence was using straight proxy settings for the time being. Will be experimenting with the WPAD file at a later time.
I seriously doubt HTTPS scanning (not filtering) with clamd is working. I have followed the configuration directions to the T and yet the only thing which does not work is https clamd scans for viruses. I believe its a Squid issue.
-
Post a link to the https fake virus test file that you are testing and I will see if it works for me.
-
http://www.eicar.org/85-0-Download.html
There is one group for http, and one group for https.
-
There are 4 https links
https://secure.eicar.org/eicar.com
https://secure.eicar.org/eicar.com.txt
https://secure.eicar.org/eicar_com.zip
https://secure.eicar.org/eicarcom2.zip -
Would you know how to get the below google safesearch info in pfSense BIND DNS?
server: include: /var/unbound/forecegoogle.conf
-
Ok done some research squidclamav only supports http not https because it is encrypted.
would you know how to get the below google safesearch info in pfSense BIND DNS?
Not sure you will need to ask that in the proxy forum.
-
Ok done some research squidclamav only supports http not https because it is encrypted.
I just did a test. Squidclamav will scan https traffic when using Squids MITM option..
-
Ok done some research squidclamav only supports http not https because it is encrypted.
I just did a test. Squidclamav will scan https traffic when using Squids MITM option..
And that would need certificates have to be installed on the clients..
-
Ok done some research squidclamav only supports http not https because it is encrypted.
I just did a test. Squidclamav will scan https traffic when using Squids MITM option..
And that would need certificates have to be installed on the clients..
Correct.
-
To help stop virus or spywhere we can enable squidguard block list blk_BL_spyware,
-
What is the point of all the safesearch nonsense and redirecting users to only use your dns.. You do understand when a proxy is being used, the proxy does the query not the client..
-
While setting up wpad we'r supposed ot enter the following into /usr/local/www/wpad.da:
function FindProxyForURL(url, host)
{
if (isPlainHostName(host) ||
shExpMatch(host, "*.local") ||
isInNet(dnsResolve(host), "192.168.1.0", "255.255.255.0"))
return "DIRECT";return "PROXY 192.168.1.1:3128";
}I've actually changed my pfsense server ip to 192.168.0.1 - do I need to edit both ip addresses listed above?
-
What is the point of all the safesearch nonsense and redirecting users to only use your dns.. You do understand when a proxy is being used, the proxy does the query not the client..
I think the point of the safe search is to stop, say your young kids, from Googling porn images. I'm not sure about the dns redirecting.
-
What is the point of all the safesearch nonsense and redirecting users to only use your dns.. You do understand when a proxy is being used, the proxy does the query not the client..
I think the point of the safe search is to stop, say your young kids, from Googling porn images. I'm not sure about the dns redirecting.
1. What is the point of forcing search engines from using safe search?
Answer: To aid in the filtering of adult content, this is most importantly for google images as squidguard does not block them, if you do not want to filter web content then this guide is not designed for you.2. What is the point of redirecting users to use THEIR pfsense router as the DNS server.
Answer: There are many advantages not all relating to web filtering however tries and stops the user from bypassing the dns redirect rule.While setting up wpad we'r supposed ot enter the following into /usr/local/www/wpad.da:
function FindProxyForURL(url, host)
{
if (isPlainHostName(host) ||
shExpMatch(host, "*.local") ||
isInNet(dnsResolve(host), "192.168.1.0", "255.255.255.0"))
return "DIRECT";return "PROXY 192.168.1.1:3128";
}I've actually changed my pfsense server ip to 192.168.0.1 - do I need to edit both ip addresses listed above?
Yes, set it to 192.168.0.0
-
I'm constantly getting these entries in the Squid Real Time log:
01.07.2016 06:06:34 192.168.1.5 TCP_DENIED/403 127.0.0.1:59488 - - 01.07.2016 06:06:34 192.168.1.5 TCP_DENIED/403 127.0.0.1:55735 - - 01.07.2016 06:06:34 192.168.1.5 TCP_DENIED/403 127.0.0.1:49806 - - 01.07.2016 06:06:34 192.168.1.5 TCP_DENIED/403 127.0.0.1:46365 - - 01.07.2016 06:06:34 192.168.1.5 TCP_DENIED/403 127.0.0.1:38156 - - 01.07.2016 06:06:34 192.168.1.5 TCP_DENIED/403 127.0.0.1:25012 - - 01.07.2016 06:06:34 192.168.1.5 TCP_DENIED/403 127.0.0.1:24866 - - 01.07.2016 06:06:33 192.168.1.5 TCP_DENIED/403 127.0.0.1:14826 - - 01.07.2016 06:06:33 192.168.1.5 TCP_DENIED/403 127.0.0.1:10196 - - 01.07.2016 06:06:33 192.168.1.5 TCP_DENIED/403 127.0.0.1:6263 - -
192.168.1.5 is my local machine. Is this normal?
Update: Resolved this in another thread..
-
To stop users from bypassing your proxy setup a new firewall lan rule and block port 80 and 443
Just to clarify - this is to prevent users from bypassing the proxy altogether, rather than just bypassing the autoproxy/wpad stuff?
-
The Squid Guard settings aren't working for me. I download the blacklist and then go to Common ACL (I've already completed the target categories step on a previous run through), but the "Target Rules" only contains "^whitelist all" and there are only [whitelist] and Default access [all] options there.
On another point, I don't want to have to set up auto proxy for every device connecting to my network - would it work if I use transparent proxy and then explicitly set the https_proxy on the couple of machines I really need locked down?
-
To stop users from bypassing your proxy setup a new firewall lan rule and block port 80 and 443
Just to clarify - this is to prevent users from bypassing the proxy altogether, rather than just bypassing the autoproxy/wpad stuff?
If port 80 and 443 are lefted open then the user can simply untic the auto configure proxy (on their PC, Mac, phone etc) and set the setting to go direct. This will not call for the wpad and the user will go direct and not use the proxy.
The Squid Guard settings aren't working for me. I download the blacklist and then go to Common ACL (I've already completed the target categories step on a previous run through), but the "Target Rules" only contains "^whitelist all" and there are only [whitelist] and Default access [all] options there.
On another point, I don't want to have to set up auto proxy for every device connecting to my network - would it work if I use transparent proxy and then explicitly set the https_proxy on the couple of machines I really need locked down?
1.With squidguard set default access to allow then set the categories you want to deny. Save and then go to squidguard General settings and hit apply. Please read the tip under this section it is very important.
Read this https://doc.pfsense.org/index.php/SquidGuard_package2. Transperrent proxy does not use a wpad, if you want to use transperrent proxy for http then you need SSL Man In the Middle Filtering for https which must have a certificate installed on evey device.
So you can either set auto configure proxy on all devices or install a certificate on all devices.
You cannot use a transperrent proxy and a wpad at the same time. If you did get a transperrent proxy with SSL mitm working then there is no need for manual proxy mode.
-
1.With squidguard set default access to allow then set the categories you want to deny. Save and then go to squidguard General settings and hit apply. Please read the tip under this section it is very important.
Read this https://doc.pfsense.org/index.php/SquidGuard_packageThat was the problem - there were no other categories than whitelist and default access. I managed to get the others by enabling "Blackilist" under the general settings of Squid Guard. Forgive me if I just missed it, but I couldn't see that step in the guide.
-
I added
In squidguard under General settings
Tic enable
Tic Enable log
Tic Enable log rotation
Tic enable blacklist
Under Blacklist URL add http://www.shallalist.de/Downloads/shallalist.tar.gz
Save
apply (you must always hit apply for any changes you made to squidguard). -
I found this post and I am researching to see how well it will work.
https://forum.pfsense.org/index.php?topic=106016.0
The idea is to use the wpad for https content and have a transperrent proxy for http traffic. This could remove the bypass rules for programs with no proxy settings and need to use port 80.
Update
OK it looks to be working fine, now all the traffic that was block on port 80 is now using the transperrent proxy, you will still need a pass rule for port 443 but not for port 80.So you can use the wpad for http and https filtering (or for just https) and enable the transperrent proxy to catch the leftovers.
-
Just to point out that this approach does not play well with many Roku channels - at least on my Roku 3 anyway.
Forcing youtube to be restricted causes all videos to fail on the Roku app.
The step " Do not allow IP-Addresses in URL" breaks Netflix.
There is also something that breaks the ITV Hub channel - although I haven't figured out what yet. My experiments suggest that it's something to do with Squid, but probably no squid guard.
The Roku remote is also somewhat less reliable in connecting after a reboot of the Roku, although I'm surprised at that, because I thought the Roku used it's own wifi to connect to the remote independently of the main network.
I'm not saying all this because I'm asking for fixes (although any suggestions that might provide fixes would be useful), but as information for anyone setting this up who has a Roku on their network.
I'm still, largely, using this approach, although, obviously, it's somewhat weaker than it would be if I didn't have to make compromises to get my Roku working.
-
Bypass the proxy for Roku and Netflix.
-
So i have got wpad all working fine using some other guides, everything goes via the autodisovery and all is good.
That is except android devices. They are not allowing auto wpad config, and i'm not setting them all up one by one.
If i set a firewall rule to block 80 - 443 on the LAN it works, and all the devices work except the android devices, which is what i expect to happen.
What i want to do is redirect all 80 - 443 requests to the squid port 3128.
But i have read somewhere that as my squid is the same ip as firewall and same subnet as lan it will cause some redirect loops.
Can someone advise me the best way to force all android traffic via squid without manually setting up each device..
-
That is except android devices. They are not allowing auto wpad config, and i'm not setting them all up one by one.
Manual setup is the only way I got it to work for android (5.1) and for older versions manual setup is the only option.
Try enabling transperrent proxy with the wpad, this will help with port 80 getting blocked.
But i have read somewhere that as my squid is the same ip as firewall and same subnet as lan it will cause some redirect loops.
I have read that too (never tried) might be your best option.
Or just get your users to setup the proxy?
-
But i have read somewhere that as my squid is the same ip as firewall and same subnet as lan it will cause some redirect loops.
You are not obliged to redirect to LAN IP. Thik about localhost ;) then I don't think there is any loop.
What makes this "dual proxy" option difficult is that, in order to set-up transparent proxy, you will have to allow requests ton internet on port 80, while, when configuring explicit proxy, you may (should) want to deny such access so that you ensure everything goes through your proxy.
And same for HTTPS except that transparent proxy, unless you configure ssl-bump, will not intercept HTTPS flow.