Guide to filtering web content (http and https) with pfsense 2.3

aGeekhere · Jun 7, 2016, 10:17 AM

Hi, there are 3 wpad files
wpad.dat
wpad.da
proxy.pac
https://en.wikipedia.org/wiki/Web_Proxy_Autodiscovery_Protocol

I made the wpad.da the main file you edit and made a symbolic link for wpad.dat and proxy.pac (so all you need to do is just edit the wpad.da file).
If this is still confusing me know and I will update the guide.

agixdota · Jun 14, 2016, 1:48 AM

thanks for the guide ;)

Dogfish · Jun 16, 2016, 10:02 PM

Thank you for the guide!

Some guides have a step to assign MIME file types eg: ".wpad" => "application/x-ns-proxy-autoconfig",

Is this not necessary if using the webconfig http file server?

aGeekhere · Jun 16, 2016, 10:41 PM

Some guides have a step to assign MIME file types eg: ".wpad" => "application/x-ns-proxy-autoconfig",

Is this not necessary if using the webconfig http file server?

I do not believe it is needed.

AR15USR · Jun 18, 2016, 9:32 PM

Can you please expand on the "use our route as the DNS server." part? Your list of settings is not very complete compared to the options available in the WebUI. Maybe be a little more specific, I'm counting 13 or so available line items available.

edit: Also, regarding the new firewall rule, which level on the Firewall list hierarchy do we here to put this new rule? If we put it at the top will we not block ourselves out?

Also, how/where does the certs for https inspection get created/used?

Thanks

aGeekhere · Jun 18, 2016, 11:57 PM

Can you please expand on the "use our route as the DNS server." part?

Sure, since we enabled the DNS resolver to do Host overrides to force safe search on a few search engines I also set up a rule that will force the local network to use the pfsense router as the DNS server. The pfsense router will cache DNS addresses and use them instead of calling a DNS server, you can also change the cache size.
Read more here https://doc.pfsense.org/index.php/Unbound_DNS_Resolver

Your list of settings is not very complete compared to the options available in the WebUI. Maybe be a little more specific, I'm counting 13 or so available line items available.

Depending on your setup this could change, this is what I use
Tic Enable DNS resolver
Listen Port default which is 53
Network Interfaces all
Outgoing Network Interfaces all
System Domain Local Zone Type transperrent
Tic Enable DNSSEC Support
Untic Enable Forwarding Mode
Tic Register DHCP leases in the DNS Resolver
Tic Register DHCP static mappings in the DNS Resolver

For advance settings
Tic
Tic
Tic
Tic
Tic
Set cache size (I set 100MB)
10
10
4096
513
200
86400
0
15 min
20000
Disable
1
Untic
Untic

Also, regarding the new firewall rule, which level on the Firewall list hierarchy do we here to put this new rule? If we put it at the top will we not block ourselves out?

It should be just under the anti block out rule, so the second rule

Also, how/where does the certs for https inspection get created/used?

WPAD does not use certs, as there is no man in the middle attack

Hope this helps

AR15USR · Jun 19, 2016, 3:19 AM

@aGeekHere:

Thanks aGeek

Actually I was referring to the "Firewall/NAT/Port forward" settings but I got it figured out anyhow.

WPAD does not use certs, as there is no man in the middle attack

Hope this helps

After using your guide, can I enable the MITM section in Squid, step up the CA and certs and it should then inspect https?

Also, under System/General/DNS Server Settings should we have/not have any DNS servers listed? I'm guessing we need some listed as this is where pfsense will get its DNS cache from?

aGeekhere · Jun 19, 2016, 4:27 AM

After using your guide, can I enable the MITM section in Squid, step up the CA and certs and it should then inspect https?

No, the wpad is doing that, the reason why we are using a wpad is so we do not have to use certs. So no transperrent proxy or man in the middle. Do not enable these options!

Also, under System/General/DNS Server Settings should we have/not have any DNS servers listed? I'm guessing we need some listed as this is where pfsense will get its DNS cache from?

You should include your isp dns (there is an option for that) and add the fastest DNS servers in your area, fill the list up, the DNS cache will get build up as you search the net, it does not just download every DNS address from the DNS server, you would run out of ram if it did.

I hope that I have cleared a few things up, have you got your setup working?

AR15USR · Jun 19, 2016, 1:48 PM

@aGeekHere:

After using your guide, can I enable the MITM section in Squid, step up the CA and certs and it should then inspect https?

No, the wpad is doing that, the reason why we are using a wpad is so we do not have to use certs. So no transperrent proxy or man in the middle. Do not enable these options!

Also, under System/General/DNS Server Settings should we have/not have any DNS servers listed? I'm guessing we need some listed as this is where pfsense will get its DNS cache from?

You should include your isp dns (there is an option for that) and add the fastest DNS servers in your area, fill the list up, the DNS cache will get build up as you search the net, it does not just download every DNS address from the DNS server, you would run out of ram if it did.

I hope that I have cleared a few things up, have you got your setup working?

no, not working. As soon as I enable the firewall rule there is no access to the internet. With the firewall rule disabled I have access but the eicar test files do not get blocked like they did when in transparent mode indicating the proxy is not being used.

asterix · Jun 19, 2016, 8:38 PM

Can't get HTTPS scanning to work. See https sites passing through squid but clamd won't scan the files. Works fine on HTTP and it catches the virus files. Using proxy setting in browser since wpad isn't giving the results I am expecting.

aGeekhere · Jun 19, 2016, 11:17 PM

no, not working. As soon as I enable the firewall rule there is no access to the internet. With the firewall rule disabled I have access but the eicar test files do not get blocked like they did when in transparent mode indicating the proxy is not being used.

Is that the port 80 and 443 block rule? If so than that is correct, now set your device to auto configure proxy. In windows go to global internet settings and there is an option for that, and for each program you have set its proxy settings. For programs with no proxy settings create a pass rule.

Can't get HTTPS scanning to work. See https sites passing through squid but clamd won't scan the files. Works fine on HTTP and it catches the virus files. Using proxy setting in browser since wpad isn't giving the results I am expecting.

Hi, first try above post, second I have not tried clamd scanning because I have found it to have issues, best to ask this question in the proxy forum.

Using proxy setting in browser since wpad isn't giving the results I am expecting

You have to tell your browser to use system settings and in global internet settings set proxy to auto configure.

AR15USR · Jun 20, 2016, 5:16 PM

@Asterix:

Can't get HTTPS scanning to work. See https sites passing through squid but clamd won't scan the files. Works fine on HTTP and it catches the virus files. Using proxy setting in browser since wpad isn't giving the results I am expecting.

Same issue here now.

I enabled the 80-443 block rule, unchecked the 'Transparent" option in Squid and I can only get access if I manually enter the wpad.dat location into my local computer(s) settings. Auto discovery does not work. I'm on all Apple computers/devices here btw. Same issue with ClamAV, it scans http but not https as poster above.

asterix · Jun 20, 2016, 2:40 PM

Auto config works for me. What I meant in the original post was that the WPAD direction info I was writing was not proper hence was using straight proxy settings for the time being. Will be experimenting with the WPAD file at a later time.

I seriously doubt HTTPS scanning (not filtering) with clamd is working. I have followed the configuration directions to the T and yet the only thing which does not work is https clamd scans for viruses. I believe its a Squid issue.

aGeekhere · Jun 20, 2016, 11:13 PM

Post a link to the https fake virus test file that you are testing and I will see if it works for me.

AR15USR · Jun 21, 2016, 12:53 AM

http://www.eicar.org/85-0-Download.html

There is one group for http, and one group for https.

asterix · Jun 21, 2016, 3:23 AM

There are 4 https links

https://secure.eicar.org/eicar.com
https://secure.eicar.org/eicar.com.txt
https://secure.eicar.org/eicar_com.zip
https://secure.eicar.org/eicarcom2.zip

asterix · Jun 21, 2016, 3:34 AM

Would you know how to get the below google safesearch info in pfSense BIND DNS?

server: include: /var/unbound/forecegoogle.conf

aGeekhere · Jun 21, 2016, 4:42 AM

Ok done some research squidclamav only supports http not https because it is encrypted.

would you know how to get the below google safesearch info in pfSense BIND DNS?

Not sure you will need to ask that in the proxy forum.

AR15USR · Jun 21, 2016, 1:03 PM

@aGeekHere:

Ok done some research squidclamav only supports http not https because it is encrypted.

I just did a test. Squidclamav will scan https traffic when using Squids MITM option..

asterix · Jun 21, 2016, 1:38 PM

@AR15USR:

@aGeekHere:

Ok done some research squidclamav only supports http not https because it is encrypted.

I just did a test. Squidclamav will scan https traffic when using Squids MITM option..

And that would need certificates have to be installed on the clients..