Filtering HTTPS / SSL Traffic on pfSense 2.1 using Squid Proxy

Heli0s · Dec 17, 2014, 5:57 PM

Since Diladele is a paid service, is there a way to use one of the other free packages that are already in the package list (DansGuardian, etc.)?

Also, instead of setting up SSL proxies and risking security issues, would it be easier using something like OpenDNS web filtering to do the filtering on the DNS level?

https://www.opendns.com/enterprise-security/solutions/web-filtering/

longhorn · Dec 21, 2014, 1:05 AM

So, after re-installing pfsense 2.1.5 (64-bit version), Squid3-dev, and Diladele 4.0…. thunk... no proxy serving or filtering worked. After A LOT of frustrated effort, I've finally got it working again. Here are the settings and steps that I used to get it working again. Hopefully, this will save some frustration for someone else (or perhaps remind myself if this happens to me again... knock on wood... LOL)

On another note... to clarify a point others have made but which I did not fully understand until I went through this process again... you cannot - from what I can tell - have a transparent SSL / HTTPS proxy with Squid and Diladele. You CAN setup a SSL proxy that is not transparent, but that requires accepting the pfSense CA on every device you want to do this for. In my situation, this wasn't helpful. However, I am happy with non-HTTPS as it catches most issues I was concerned about (home network protection and monitoring).

These instructions pertain to setting up the following

Squid3-dev proxy server
Diladele version 4.0 Web Safety
Transparent proxy for HTTP (non-SSL) ONLY (details on also filtering SSL below)

Steps:

Install pfsense 2.1.5
Setup basic configuration, firewall rules, etc.
Install package Squid3-dev (I used version 3.3.10 pkg 2.2.8)
Establish proxy server settings and test via Real Time reporting in squid
Reboot
Install Diladele using scripts
Reboot
Login to Diladele Web interface and verify operation
Add custom ACLS to pfsense Proxy Server
Restart Proxy Server
Test Diladele again and verify it is now reporting real time monitoring

Visit the Diladele pfSense tutorial, and you'll find the scripts I'm referring to. They make installing Diladele and its dependencies MUCH easier. You can find that tutorial here: http://docs.diladele.com/tutorials/filtering_https_traffic_squid_pfsense/integrate.html

Here are the proxy server settings that I used in pfSense (Services > Proxy Server):
- Proxy interface(s): Highlight and select (use CTRL + Left Click) all the interfaces you want to use the proxy for HTTP traffic
- Proxy port 3128
- Allow users on interface: check this box if you want to use a transparent proxy and have all users go thru the proxy by default
- Transparent HTTP proxy: check this box
- Transparent Proxy interface(s): check off all the interfaces; presumably you'll want them to match the interfaces above
- Bypass proxy for these source IP's: I left this blank
- Bypass proxy for these destination IPs: Put anything here you don't want cached; I included my pfSense box's IP, eveonline.com, and crashplan.com (I use crashplan for offsite backups)
- Scroll down to Logging Settings > Enabled logging: check this box. I found this to be key. If I didn't check this, Diladele did not work
- Scroll down to Custom Settings > Custom ACLS (Before_Auth): enter the string of text found in the Diladele website tutorial for pfSense or previously in this thread

That's it. Make sure you press Save and restart Squid.

Now launch Diladele and do some surfing. Watch Diladele's real-time monitoring/surfing. It should work now.

ghanel · Jan 9, 2015, 5:11 PM

@longhorn:

So, after re-installing pfsense 2.1.5 (64-bit version), Squid3-dev, and Diladele 4.0…. thunk... no proxy serving or filtering worked. After A LOT of frustrated effort, I've finally got it working again. Here are the settings and steps that I used to get it working again. Hopefully, this will save some frustration for someone else (or perhaps remind myself if this happens to me again... knock on wood... LOL)

On another note... to clarify a point others have made but which I did not fully understand until I went through this process again... you cannot - from what I can tell - have a transparent SSL / HTTPS proxy with Squid and Diladele. You CAN setup a SSL proxy that is not transparent, but that requires accepting the pfSense CA on every device you want to do this for. In my situation, this wasn't helpful. However, I am happy with non-HTTPS as it catches most issues I was concerned about (home network protection and monitoring).

These instructions pertain to setting up the following

Squid3-dev proxy server

Diladele version 4.0 Web Safety

Transparent proxy for HTTP (non-SSL) ONLY (details on also filtering SSL below)

Steps:

Install pfsense 2.1.5

Setup basic configuration, firewall rules, etc.

Install package Squid3-dev (I used version 3.3.10 pkg 2.2.8)

Establish proxy server settings and test via Real Time reporting in squid

Reboot

Install Diladele using scripts

Reboot

Login to Diladele Web interface and verify operation

Add custom ACLS to pfsense Proxy Server

Restart Proxy Server

Test Diladele again and verify it is now reporting real time monitoring

Visit the Diladele pfSense tutorial, and you'll find the scripts I'm referring to. They make installing Diladele and its dependencies MUCH easier. You can find that tutorial here: http://docs.diladele.com/tutorials/filtering_https_traffic_squid_pfsense/integrate.html

Here are the proxy server settings that I used in pfSense (Services > Proxy Server):
- Proxy interface(s): Highlight and select (use CTRL + Left Click) all the interfaces you want to use the proxy for HTTP traffic
- Proxy port 3128
- Allow users on interface: check this box if you want to use a transparent proxy and have all users go thru the proxy by default
- Transparent HTTP proxy: check this box
- Transparent Proxy interface(s): check off all the interfaces; presumably you'll want them to match the interfaces above
- Bypass proxy for these source IP's: I left this blank
- Bypass proxy for these destination IPs: Put anything here you don't want cached; I included my pfSense box's IP, eveonline.com, and crashplan.com (I use crashplan for offsite backups)
- Scroll down to Logging Settings > Enabled logging: check this box. I found this to be key. If I didn't check this, Diladele did not work
- Scroll down to Custom Settings > Custom ACLS (Before_Auth): enter the string of text found in the Diladele website tutorial for pfSense or previously in this thread

That's it. Make sure you press Save and restart Squid.

Now launch Diladele and do some surfing. Watch Diladele's real-time monitoring/surfing. It should work now.

Hopefully just missed a step in here but tried a couple of times. After going through the steps as described above I get an ICAP protocol error - essentially after adding the custom acls (before AUTH). Can anyone provide any hints. I did see on the diladele's website under the licensing section that the 2 month trial license had been removed - is that what's need to get it running?

Cheers
Gavin

ICAP ERROR
The following error was encountered while trying to retrieve the URL: http://www.google.co.uk/
ICAP protocol error.
The system returned: [No Error]
This means that some aspect of the ICAP communication failed.
Some possible problems are:
The ICAP server is not reachable.
An Illegal response was received from the ICAP server.

FROM DILADELE WEBSITE
The trial license which was active during the last year has finally expired. Please purchase the commercial license if you think product is worth and it if not - share your thoughts and we will try to make it better!

ghanel · Jan 13, 2015, 1:58 PM

Just on the above post the qlproxy service isn't starting because of an expired license key. Hoping Diladele can provide a month trial so we can test it out.

mzarrugh · Apr 26, 2015, 7:25 PM

Could you please update it so it works with pfSense 2.2.2 and squid 3.4.10_2 pkg 0.2.8.

NetVicious · May 22, 2015, 12:19 PM

On 2.2.2-RELEASE (i386) with these package versions:

squid3 0.2.8
squidGuard 1.9.14
squidGuard-devel 1.5_1beta pkg v.1.5.6

I get one error saying squid needs the module to bypassing SSL

# squid -k check
2015/05/22 14:15:33| FATAL: tproxy/intercept on https_port requires ssl-bump which is missing.
FATAL: Bungled /usr/local/etc/squid/squid.conf line 7: https_port 127.0.0.1:3129 intercept
Squid Cache (Version 3.4.10): Terminated abnormally.
CPU Usage: 0.024 seconds = 0.018 user + 0.006 sys
Maximum Resident Size: 36000 KB
Page faults with physical i/o: 0

It seems we need a new package from the pfSense developers

NetVicious · Jun 9, 2015, 8:51 AM

Hi! Any progress on the squid update to fix the HTTPS filtering problem we have on 2.2.2?

lockye · Aug 26, 2015, 4:58 PM

I have everything setup and it is filtering HTTPS sites correctly but I do have a couple of issues with using iPad’s/Android’s on the network.

I have installed the certificates but It seems that some of the apps do not like going through the man in the middle filtering, the app store, banking apps and Facebook to name a few.

Is there a way to setup some sites to bypass the proxy filtering completely?

KOM · Aug 26, 2015, 5:58 PM

Is there a way to setup some sites to bypass the proxy filtering completely?

The main Squid config page has this option:

Bypass proxy for these destination IPs

lockye · Aug 26, 2015, 8:14 PM

KOM

I have tried using the bypass proxy with apples iTunes store address but for some reason I can not get it to work for various Apps.

nhgdesign · Nov 10, 2015, 4:35 PM

I Have about 70 workstations on the network.

Installing a certificate in each and every browser would be a terrible idea for me.

:o

KOM · Nov 10, 2015, 4:44 PM

Installing a certificate in each and every browser would be a terrible idea for me.

And everyone else, too. That's why WPAD is, IMO, the preferred method.

nhgdesign · Nov 10, 2015, 5:59 PM

How about using E2Guardian? Although I can't find it in the packages available, it's probably being ported sometime soon. I've read somewhere that it supports HTTPS filtering.

KOM · Nov 10, 2015, 6:27 PM

I haven't paid it much attention as I don't need a heavy content filter, just a simple URL filter. Others have likened it to a substitute for DansGuardian but I have no knowledge of that.

sichent · Apr 26, 2016, 7:10 PM

Updated the guide for pfSense 2.3 and web safety 4.4 - http://docs.diladele.com/tutorials/filtering_https_traffic_squid_pfsense/index.html

alex_lebbrom · May 27, 2016, 6:25 PM

Question, Can I filter HTTPS traffic but not at all websites? For example, we could open websites for bank but we couldn't open social networks?

Thanks,

sichent · May 27, 2016, 7:49 PM

Hate to say but not in pfSense version :( In pfSense SSL filtering settings are managed by PfSense's Squid GUI. In Linux version there are two modes - bump all or filter targeted. And it is also possible to bump by categories - i.e. never bump banks