Filtering HTTPS / SSL Traffic on pfSense 2.1 using Squid Proxy

longhorn

Could you please give us more information how you solved the problem so that other users may benefit from it.

I don't believe that I answered your question very well a few weeks ago. To be more specific, this is what helped me to get it working completely (in particular the HTTPS filtering):

@firefish5000:

To setup transparent https caching: (posting here because it seems relevant & I was unable to find it documented anywhere)
1. Follow the guide's instructions for creating and installing an internal certificate authority
2. Goto 'Services ->Proxy Server' and Check 'HTTPS/SSL interception'
3. Set 'SSL Intercept interface(s):' to 'loopback' & 'SSL Proxy port:' to '3129'. Then Select the CA you created (For most it should already be selected).
4. (squid-dev 3.3.10 pkg 2.2.2 specific) Scroll to 'Custom Settings/Custom ACLS (Before_Auth)' section and Add 'always_direct allow all; ssl_bump server-first all' or your preferred ssl_bump setting there. (This was added automatically in previous packages. It's removal is likely a bug)(This is needed for manually proxied connections as well)
5. Goto 'Firewall -> NAT' and Under 'Port Forward' Click the Plus Button to add a new entry.
6. Set
Interface: LAN | Protocol: TCP
Source: any any (you may wish to set this to a specific ip or alias. At least until you confirm it's working properly)
Destination: NOT (Check this)
Choose 'LAN address' OR 'Single Host/alias' and add pfsense's lan ip (Else squid will lock you out of pfsense because it doesn't like pfsense's self signed certificate, you may also wish to add pfsense's ip in to squid's bypass list)
Destination Port: from HTTPS to HTTPS
Redirect target IP: 127.0.0.1 | Redirect target port: (other) 3129

Heli0s

Since Diladele is a paid service, is there a way to use one of the other free packages that are already in the package list (DansGuardian, etc.)?

Also, instead of setting up SSL proxies and risking security issues, would it be easier using something like OpenDNS web filtering to do the filtering on the DNS level?

https://www.opendns.com/enterprise-security/solutions/web-filtering/

longhorn

So, after re-installing pfsense 2.1.5 (64-bit version), Squid3-dev, and Diladele 4.0…. thunk... no proxy serving or filtering worked. After A LOT of frustrated effort, I've finally got it working again. Here are the settings and steps that I used to get it working again. Hopefully, this will save some frustration for someone else (or perhaps remind myself if this happens to me again... knock on wood... LOL)

On another note... to clarify a point others have made but which I did not fully understand until I went through this process again... you cannot - from what I can tell - have a transparent SSL / HTTPS proxy with Squid and Diladele. You CAN setup a SSL proxy that is not transparent, but that requires accepting the pfSense CA on every device you want to do this for. In my situation, this wasn't helpful. However, I am happy with non-HTTPS as it catches most issues I was concerned about (home network protection and monitoring).

These instructions pertain to setting up the following

Squid3-dev proxy server
Diladele version 4.0 Web Safety
Transparent proxy for HTTP (non-SSL) ONLY (details on also filtering SSL below)

Steps:

Install pfsense 2.1.5
Setup basic configuration, firewall rules, etc.
Install package Squid3-dev (I used version 3.3.10 pkg 2.2.8)
Establish proxy server settings and test via Real Time reporting in squid
Reboot
Install Diladele using scripts
Reboot
Login to Diladele Web interface and verify operation
Add custom ACLS to pfsense Proxy Server
Restart Proxy Server
Test Diladele again and verify it is now reporting real time monitoring

Visit the Diladele pfSense tutorial, and you'll find the scripts I'm referring to. They make installing Diladele and its dependencies MUCH easier. You can find that tutorial here: http://docs.diladele.com/tutorials/filtering_https_traffic_squid_pfsense/integrate.html

Here are the proxy server settings that I used in pfSense (Services > Proxy Server):
- Proxy interface(s): Highlight and select (use CTRL + Left Click) all the interfaces you want to use the proxy for HTTP traffic
- Proxy port 3128
- Allow users on interface: check this box if you want to use a transparent proxy and have all users go thru the proxy by default
- Transparent HTTP proxy: check this box
- Transparent Proxy interface(s): check off all the interfaces; presumably you'll want them to match the interfaces above
- Bypass proxy for these source IP's: I left this blank
- Bypass proxy for these destination IPs: Put anything here you don't want cached; I included my pfSense box's IP, eveonline.com, and crashplan.com (I use crashplan for offsite backups)
- Scroll down to Logging Settings > Enabled logging: check this box. I found this to be key. If I didn't check this, Diladele did not work
- Scroll down to Custom Settings > Custom ACLS (Before_Auth): enter the string of text found in the Diladele website tutorial for pfSense or previously in this thread

That's it. Make sure you press Save and restart Squid.

Now launch Diladele and do some surfing. Watch Diladele's real-time monitoring/surfing. It should work now.

ghanel

@longhorn:

So, after re-installing pfsense 2.1.5 (64-bit version), Squid3-dev, and Diladele 4.0…. thunk... no proxy serving or filtering worked. After A LOT of frustrated effort, I've finally got it working again. Here are the settings and steps that I used to get it working again. Hopefully, this will save some frustration for someone else (or perhaps remind myself if this happens to me again... knock on wood... LOL)

On another note... to clarify a point others have made but which I did not fully understand until I went through this process again... you cannot - from what I can tell - have a transparent SSL / HTTPS proxy with Squid and Diladele. You CAN setup a SSL proxy that is not transparent, but that requires accepting the pfSense CA on every device you want to do this for. In my situation, this wasn't helpful. However, I am happy with non-HTTPS as it catches most issues I was concerned about (home network protection and monitoring).

These instructions pertain to setting up the following

Squid3-dev proxy server

Diladele version 4.0 Web Safety

Transparent proxy for HTTP (non-SSL) ONLY (details on also filtering SSL below)

Steps:

Install pfsense 2.1.5

Setup basic configuration, firewall rules, etc.

Install package Squid3-dev (I used version 3.3.10 pkg 2.2.8)

Establish proxy server settings and test via Real Time reporting in squid

Reboot

Install Diladele using scripts

Reboot

Login to Diladele Web interface and verify operation

Add custom ACLS to pfsense Proxy Server

Restart Proxy Server

Test Diladele again and verify it is now reporting real time monitoring

Visit the Diladele pfSense tutorial, and you'll find the scripts I'm referring to. They make installing Diladele and its dependencies MUCH easier. You can find that tutorial here: http://docs.diladele.com/tutorials/filtering_https_traffic_squid_pfsense/integrate.html

Here are the proxy server settings that I used in pfSense (Services > Proxy Server):
- Proxy interface(s): Highlight and select (use CTRL + Left Click) all the interfaces you want to use the proxy for HTTP traffic
- Proxy port 3128
- Allow users on interface: check this box if you want to use a transparent proxy and have all users go thru the proxy by default
- Transparent HTTP proxy: check this box
- Transparent Proxy interface(s): check off all the interfaces; presumably you'll want them to match the interfaces above
- Bypass proxy for these source IP's: I left this blank
- Bypass proxy for these destination IPs: Put anything here you don't want cached; I included my pfSense box's IP, eveonline.com, and crashplan.com (I use crashplan for offsite backups)
- Scroll down to Logging Settings > Enabled logging: check this box. I found this to be key. If I didn't check this, Diladele did not work
- Scroll down to Custom Settings > Custom ACLS (Before_Auth): enter the string of text found in the Diladele website tutorial for pfSense or previously in this thread

That's it. Make sure you press Save and restart Squid.

Now launch Diladele and do some surfing. Watch Diladele's real-time monitoring/surfing. It should work now.

Hopefully just missed a step in here but tried a couple of times. After going through the steps as described above I get an ICAP protocol error - essentially after adding the custom acls (before AUTH). Can anyone provide any hints. I did see on the diladele's website under the licensing section that the 2 month trial license had been removed - is that what's need to get it running?

Cheers
Gavin

ICAP ERROR
The following error was encountered while trying to retrieve the URL: http://www.google.co.uk/
ICAP protocol error.
The system returned: [No Error]
This means that some aspect of the ICAP communication failed.
Some possible problems are:
The ICAP server is not reachable.
An Illegal response was received from the ICAP server.

FROM DILADELE WEBSITE
The trial license which was active during the last year has finally expired. Please purchase the commercial license if you think product is worth and it if not - share your thoughts and we will try to make it better!

ghanel

Just on the above post the qlproxy service isn't starting because of an expired license key. Hoping Diladele can provide a month trial so we can test it out.

mzarrugh

Could you please update it so it works with pfSense 2.2.2 and squid 3.4.10_2 pkg 0.2.8.

NetVicious

On 2.2.2-RELEASE (i386) with these package versions:

squid3 0.2.8
squidGuard 1.9.14
squidGuard-devel 1.5_1beta pkg v.1.5.6

I get one error saying squid needs the module to bypassing SSL

# squid -k check
2015/05/22 14:15:33| FATAL: tproxy/intercept on https_port requires ssl-bump which is missing.
FATAL: Bungled /usr/local/etc/squid/squid.conf line 7: https_port 127.0.0.1:3129 intercept
Squid Cache (Version 3.4.10): Terminated abnormally.
CPU Usage: 0.024 seconds = 0.018 user + 0.006 sys
Maximum Resident Size: 36000 KB
Page faults with physical i/o: 0

It seems we need a new package from the pfSense developers

NetVicious

Hi! Any progress on the squid update to fix the HTTPS filtering problem we have on 2.2.2?

lockye

I have everything setup and it is filtering HTTPS sites correctly but I do have a couple of issues with using iPad’s/Android’s on the network.

I have installed the certificates but It seems that some of the apps do not like going through the man in the middle filtering, the app store, banking apps and Facebook to name a few.

Is there a way to setup some sites to bypass the proxy filtering completely?

KOM

Is there a way to setup some sites to bypass the proxy filtering completely?

The main Squid config page has this option:

Bypass proxy for these destination IPs

lockye

KOM

I have tried using the bypass proxy with apples iTunes store address but for some reason I can not get it to work for various Apps.

nhgdesign

I Have about 70 workstations on the network.

Installing a certificate in each and every browser would be a terrible idea for me.

:o

KOM

Installing a certificate in each and every browser would be a terrible idea for me.

And everyone else, too. That's why WPAD is, IMO, the preferred method.

nhgdesign

How about using E2Guardian? Although I can't find it in the packages available, it's probably being ported sometime soon. I've read somewhere that it supports HTTPS filtering.

KOM

I haven't paid it much attention as I don't need a heavy content filter, just a simple URL filter. Others have likened it to a substitute for DansGuardian but I have no knowledge of that.

sichent

Updated the guide for pfSense 2.3 and web safety 4.4 - http://docs.diladele.com/tutorials/filtering_https_traffic_squid_pfsense/index.html

alex_lebbrom

Question, Can I filter HTTPS traffic but not at all websites? For example, we could open websites for bank but we couldn't open social networks?

Thanks,

sichent

Hate to say but not in pfSense version :( In pfSense SSL filtering settings are managed by PfSense's Squid GUI. In Linux version there are two modes - bump all or filter targeted. And it is also possible to bump by categories - i.e. never bump banks