Filtering HTTPS / SSL Traffic on pfSense 2.1 using Squid Proxy


  • Banned

    This article will tell you how to install and configure Squid proxy capable of filtering encrypted HTTPS connections using Diladele Web Safety ICAP content filtering server running on pfSense firewall 2.1. Being able to look into HTTPS contents greatly increases your ability to control what is allowed and accepted within your network while keeping inappropriate contents away.

    Why do We Need to Filter HTTPS

    HTTPS protocol was designed to provide secure means of communications between internet browser and remote web servers. In order to achieve this goal HTTPS protocol encrypts data passing through established connections so that it cannot be decrypted in reasonable amount of time thus preventing anyone from sniffing the contents interchanged over this connection. This protocol was primarily invented to enable safe and secure communication between the user and financial sites or government institutions over the insecure medium such as the Internet.

    Recently more and more web sites started to use HTTPS encrypted communications to increase online privacy of users. Google who as first enabled HTTPS for all its searches by default probably initiated this trend. Although there are no doubts that HTTPS encryption is a good thing for safety on the wire we must take into account that it also creates several problems for controlled networks typically found at home or offices. The main problem here is the essence of the HTTPS protocol itself – no one except the browser and the web server is able to see and thus filter transferred data. This may not always be desired. Contents that are usually blocked suddenly become immediately accessible by anyone. As an example imagine a school network where minors can see questionable content by just mistyping a search term in Google. Moreover the law often forces administrators in educational institutions to block access to such content (e.g. CIPA for educational environments) and encrypted access to web sites makes it nearly impossible to fulfill such an obligation.

    In order to overcome these limitations it is advised to setup HTTPS filtering of web contents with help of SSL bump feature of Squid proxy server and Diladele Web Safety web filter.

    How It Works

    In order to filter web requests user’s browser needs to be explicitly directed to use the proxy that is deployed in the same network. It is also possible to set the transparent proxy but we are not going to explain how this is done in this tutorial because steps involved are quite different from explicit proxy setup.

    When a user tries to navigate to a web site, browser sends the request to proxy server, asking it to get the requested page on his behalf. The proxy establishes a new connection to the remote site and returns the response to browser. If normal HTTP is used then proxy is able to see the original contents of the response and filter it. In case of HTTPS the flow of data is a little different. Browser asks the proxy to establish a virtual tunnel between itself and remote server and then sends encrypted data through the proxy. Domain name to which a virtual tunnel is being established is usually known, so proxy is able to block this virtual tunnel when it finds out that domain name belongs to a prohibited category. Unfortunately this is not a complete solution as there are a lot of sites on the Internet which are general in nature (like Google or YouTube) but allow you to easily navigate to something undesired.

    To improve the quality of web filtering and get access to contents in encrypted connections, browsers in the network may be setup to trust proxy to act on their behalf for establishing HTTPS connections, filtering them and passing the allowed data to clients while blocking everything that is not allowed. Although this assumption is too strict to be implemented in public networks, it is easily doable in controlled home, educational or corporate environments where administrators act as sole owners of network devices and may force any trusting rules.  After established trust browser is able to ask proxy to connect to a remote site in a safe manner with HTTPS, proxy is able to decrypt the traffic, filter it, encrypt it again and pass it to browser. As browser trusts the proxy it continues working with filtered HTTS without any errors or warnings.

    Assumptions

    I assume you have already installed pfSense 2.1 amd64 with two NIC (LAN and WAN). Lan IP address is 192.168.1.1. I also assume you have already done the initial login to the Web UI of pfSense and completed the initial setup wizard and successfully rebooted the pfSense box at least once.

    Step 1 – Install Squid built with SSL decryption support.

    Login to Web UI and select System -> Packages -> Available Packages. Find and install package squid3-dev.

    Wait until squid3-dev package is installed and after installation, reboot your pfSense box.

    Login to Web UI, select Services -> Proxy server. The big red dot on the right side will indicate Squid service was not able to start successfully and is currently stopped.

    To understand the reason why Squid count not start on reboot, login to the console of pfSense box and run the following command /usr/local/sbin/squid -v. The output clearly indicates something is wrong with the installation process as several run time dependences for Squid are missing.

    
    [2.1-RELEASE][root@pfsense.localdomain]/root(2): /usr/local/sbin/squid -v
    /libexec/ld-elf.so.1: Shared object "libheimntlm.so.10" not found, required by "squid"
    
    

    To fix this issue we will download missing libraries from web site of squid3-dev package maintainer in pfSense. After download finishes run the /usr/local/sbin/squid -v command again. Now it should finish successfully displaying information about current Squid version and its compile flags.

    
    fetch -o /usr/local/lib/libasn1.so.10 http://e-sac.siteseguro.ws/pfsense/8/amd64/All/ldd/libasn1.so.10
    fetch -o /usr/local/lib/libgssapi.so.10 http://e-sac.siteseguro.ws/pfsense/8/amd64/All/ldd/libgssapi.so.10
    fetch -o /usr/local/lib/libheimntlm.so.10 http://e-sac.siteseguro.ws/pfsense/8/amd64/All/ldd/libheimntlm.so.10
    fetch -o /usr/local/lib/libhx509.so.10 http://e-sac.siteseguro.ws/pfsense/8/amd64/All/ldd/libhx509.so.10
    fetch -o /usr/local/lib/libkrb5.so.10 http://e-sac.siteseguro.ws/pfsense/8/amd64/All/ldd/libkrb5.so.10
    fetch -o /usr/local/lib/libroken.so.10 http://e-sac.siteseguro.ws/pfsense/8/amd64/All/ldd/libroken.so.10
    
    

    Login to Web UI, select Status -> Services, find Squid service and push the Start Service button.

    After a while you will notice that Squid service still cannot be started. This time the reason for this is incorrect default configuration containing ACL for proxy authentication although no authentication scheme is configured by default. We could fix this by modifying configuration generation scripts but for the purposes of this howto we will just enable local authentication.

    Select Services -> Proxy server and click the Authentication tab. In the combo box select Local Authentication. Then scroll all the way down and click Save. Now select the Users tab and add a user “user1” with password “P@ssw0rd”.

    Click Save and try starting Squid service. This time it should start without errors.

    Now adjust your browser configuration to point to pfSense box and try browsing the web.

    The response from Squid indicates that we do not have access to it.

    To get the access we will add our subnet to the allowed ACL. Select Services -> Proxy server and click the ACL tab. Add 192.168.1.0/24 to the “Allowed subnets” field. Scroll all the way down and click Save. Reload Squid service in Status -> Services. Try browsing web again, enter credentials we created earlier and see that Squid is now working.

    Install Diladele Web Safety for Squid Proxy

    SSL Bumping feature alone is not enough to block questionable web content. We also need the filtering server that could be paired with Squid. We will use Diladele Web Safety (called qlproxy) for the filtering and blocking part. It is an ICAP daemon capable of integrating existing Squid proxy and providing rich content filtering functionality out of the box. It may be used to block illegal or potentially malicious file downloads, remove annoying advertisements, prevent access to various categories of the web sites and block resources with explicit content.

    We will use version 3.1 of qlproxy that was released this month. It was designed specifically with HTTPS filtering in mind and contains rich web administrator console to perform routine tasks right from the browser.

    By default, qlproxy comes with four polices preinstalled. Strict policy contains web filter settings put on maximum level and is supposed to protect minors and K12 students from inappropriate contents on the Internet. Relaxed policy blocks only excessive advertisements and was supposed to be used by network administrators, teachers and all those who do not need filtered access to web but would like to evade most ads. Third policy is tailored to white list only browsing and the last group contains less restrictive web filtering settings suitable for normal web browsing without explicitly adult contents shown.

    In order to install Diladele Web Safety for Squid Proxy, download package for FreeBSD 8 64-bit Diladele B.V. web site at http://www.quintolabs.com using browser and upload the TBZ binary to you pfSense box.

    Install Diladele Web Safety

    Diladele Web Safety uses qlproxy user and group to run. Normally it creates those upon installation but for some reason they are not saved during reboots so we will create required users manually. Go to System -> User Manager select Groups Tab and add a new group qlproxy.

    Click Save and then select the Users tab to add a new user qlproxy. Do not forget to make in a member of qlproxy group. Enter some arbitrary password.

    Again click Save.

    Administration console of Diladele Web Safety is built using Python Django framework and is usually managed by Apache web server. To install Python package run the following command.

    
    [2.1-RELEASE][root@pfSense.localdomain]/root(11): pkg_add -r python27 py27-sqlite3 py27-pip
    Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8.3-release/Latest/python27.tbz... Done.
    
    ====
    Note that some of the standard modules are provided as separate
    ports since they require extra dependencies:
    
    bsddb           databases/py-bsddb
    gdbm            databases/py-gdbm
    sqlite3         databases/py-sqlite3
    tkinter         x11-toolkits/py-tkinter
    
    Install them as needed.
    ====
    
    Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8.3-release/Latest/py27-sqlite3.tbz... Done.
    Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8.3-release/All/pkg-config-0.25_1.tbz... Done.
    Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8.3-release/All/sqlite3-3.7.10.tbz... Done.
    Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8.3-release/Latest/py27-pip.tbz... Done.
    Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8.3-release/All/py27-setuptools-0.6c11_3.tbz... Done.
    
    

    To install Django Python framework run the following command.

    
    [2.1-RELEASE][root@pfSense.localdomain]/root(13): /usr/local/bin/pip install Django==1.5
    Downloading/unpacking Django==1.5
      Downloading Django-1.5.tar.gz (8.0Mb): 8.0Mb downloaded
      Running setup.py egg_info for package Django
    
        warning: no previously-included files matching '__pycache__' found under directory '*'
        warning: no previously-included files matching '*.py[co]' found under directory '*'
    Installing collected packages: Django
      Running setup.py install for Django
        changing mode of build/scripts-2.7/django-admin.py from 644 to 755
    
        warning: no previously-included files matching '__pycache__' found under directory '*'
        warning: no previously-included files matching '*.py[co]' found under directory '*'
        changing mode of /usr/local/bin/django-admin.py to 755
    Successfully installed Django
    Cleaning up...
    
    

    To install Apache Web Server with mod_wsgi, run the following commands.

    
    [2.1-RELEASE][root@pfSense.localdomain]/root(15): pkg_add -r apache22 ap22-mod_wsgi
    Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8.3-release/Latest/apache22.tbz... Done.
    Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8.3-release/All/expat-2.0.1_2.tbz... Done.
    Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8.3-release/All/perl-5.12.4_4.tbz... Done.
    Removing stale symlinks from /usr/bin...
        Skipping /usr/bin/perl
        Skipping /usr/bin/perl5
        Removing /usr/local/bin/perl5.16.3 installed by an older perl port
    Done.
    Creating various symlinks in /usr/bin...
        Symlinking /usr/local/bin/perl5.12.4 to /usr/bin/perl
        Symlinking /usr/local/bin/perl5.12.4 to /usr/bin/perl5
    Done.
    cd: can't cd to /usr/include
    Cleaning up /etc/make.conf... Done.
    Spamming /etc/make.conf... Done.
    Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8.3-release/All/pcre-8.30_1.tbz... Done.
    Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8.3-release/All/gdbm-1.9.1.tbz... Done.
    Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8.3-release/All/db42-4.2.52_5.tbz... Done.
    Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8.3-release/All/apr-ipv6-devrandom-gdbm-db42-1.4.5.1.3.12_1.tbz... Done.
    pkg_add: warning: package 'apr-ipv6-devrandom-gdbm-db42-1.4.5.1.3.12_1' requires 'libiconv-1.13.1_2', but 'libiconv-1.14_1' is installed
    ===> Creating users and/or groups.
    Using existing group 'www'.
    Using existing user 'www'.
    pkg_add: warning: package 'apache-2.2.22_5' requires 'libiconv-1.13.1_2', but 'libiconv-1.14_1' is installed
    
    To run apache www server from startup, add apache22_enable="YES"
    in your /etc/rc.conf. Extra options can be found in startup script.
    
    Your hostname must be resolvable using at least 1 mechanism in
    /etc/nsswitch typically DNS or /etc/hosts or apache might
    have issues starting depending on the modules you are using.
    
    Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/amd64/packages-8.3-release/Latest/ap22-mod_wsgi.tbz... Done.
    [activating module `wsgi' in /usr/local/etc/apache22/httpd.conf]
    pkg_add: warning: package 'ap22-mod_wsgi-3.3_2' requires 'libiconv-1.13.1_2', but 'libiconv-1.14_1' is installed
    
    ============================================================================
    
    If you want to use daemon mode you will need to make sure Apache and APR
    include thread support.
    
    For more information, see http://www.modwsgi.org/wiki/InstallationOnFreeBSD.
    
    ============================================================================
    
    

    Now install the qlproxy-3.1.0-amd64.tbz package you have downloaded earlier. The console output should look something like this.

    
    [2.1-RELEASE][root@pfSense.localdomain]/root(20): pkg_add qlproxy-3.1.0-amd64.tbz
    Pre-install of Diladele Web Safety is starting...
    Searching for group qlproxy...
    Group qlproxy already exists.
    Searching for user qlproxy...
    User qlproxy already exists.
    Pre-install of Diladele Web Safety finished successfully.
    Post-install of Diladele Web Safety is starting...
    Cannot 'start' qlproxyd. Set qlproxyd_enable to YES in /etc/rc.conf or use 'onestart' instead of 'start'.
    Post-install of Diladele Web Safety finished successfully.
    
    ******************************************************************************
    Diladele Web Safety for Squid Proxy is installed successfully.
    
    Additional manual steps:
    
    1\. Enable automatic startup of Diladele Web Safety at system boot by adding
       qlproxyd_enable="YES" entry in /etc/rc.conf or
       /usr/local/etc/rc.conf
    
    2\. Integrate qlproxy as ICAP server with Squid Proxy
    
    3\. Configure Web UI of Diladele Web Safety as described in the Online Docs
    
    Thank you for choosing Diladele B.V.! :)
    
    ******************************************************************************
    
    

    As we did not use built in pfSense packages for Apache Web Server and qlproxy we will need to adjust them a little. Start by running the following commands to make qlproxy and apache autostart when pfSense box is rebooted.

    
    [2.1-RELEASE][root@pfSense.localdomain]/root(22): cp /usr/local/etc/rc.d/qlproxyd /usr/local/etc/rc.d/qlproxyd.sh
    [2.1-RELEASE][root@pfSense.localdomain]/root(23): cp /usr/local/etc/rc.d/apache22 /usr/local/etc/rc.d/apache22.sh
    
    

    We need to edit the qlproxyd.sh script,  find the ${qlproxyd_enable:="NO"} and change it to ${qlproxyd_enable:="YES"}. Then edit apache22.sh script, find [ -z "$apache22_enable" ]      && apache22_enable="NO" and change it to [ -z "$apache22_enable" ]  && apache22_enable="YES".

    Default installation of apache makes it listening on port 80. Port 80 is already taken by pfSense Web UI so we need to tell apache to listen on another port - like 8080. To do it open the /usr/local/etc/apache22/extra/qlproxy_virtual_host file and change <virtualhost *:8080="">to <virtualhost *:8080="">.

    Also change the Listen 80 directive to Listen 8080 in /usr/local/etc/apache22/httpd.conf and uncomment the Include etc/apache22/extra/httpd-vhosts.conf directive.

    Remove all VirtualHost directives in /usr/local/etc/apache22/extra/httpd-vhosts.conf so that you have only Include /usr/local/etc/apache22/extra/qlproxy_virtual_host in this file.

    If you try to access qlproxy web ui now you will get into trouble as it may not be fully configured (the time zone settings are incorrect). We need to change the default 'America/Chicago' to 'EST'. Open /usr/local/var/qlproxy/console/qlproxy/settings.py file, find the TIME_ZONE = 'America/Chicago' and change it to TIME_ZONE = 'EST'.

    Restart Apache by running /usr/local/etc/rc.d/apache22.sh restart and login to 192.169.1.1:8080 to finally see the Web UI of Diladele Web Safety.

    Integrate Squid Proxy and Diladele Web Safety

    Please restart the pfSense box now to be sure qlproxy and apache22 start normally after reboot. Then to integrate qlproxy and squid, go to Services / Proxy Server, scroll all the way down and in Custom Settings / Custom Options field type:

    
    icap_enable on
    icap_preview_enable on
    icap_preview_size 4096
    icap_persistent_connections on
    icap_send_client_ip on
    icap_send_client_username on
    icap_client_username_header X-Client-Username
    icap_service qlproxy1 reqmod_precache bypass=0 icap://127.0.0.1:1344/reqmod
    icap_service qlproxy2 respmod_precache bypass=0 icap://127.0.0.1:1344/respmod
    
    acl qlproxy_icap_edomains dstdomain "/usr/local/etc/qlproxy/squid/icap_exclusions_domains.conf"
    acl qlproxy_icap_etypes rep_mime_type "/usr/local/etc/qlproxy/squid/icap_exclusions_contenttypes.conf"
    
    adaptation_access qlproxy1 deny qlproxy_icap_edomains
    adaptation_access qlproxy2 deny qlproxy_icap_edomains
    adaptation_access qlproxy2 deny qlproxy_icap_etypes
    adaptation_access qlproxy1 allow all
    adaptation_access qlproxy2 allow all
    
    

    Click Save and restart Squid proxy. Try to browse to some adult site and see that HTTP filtering works correctly.

    Now we need to enable SSL Filtering to make Diladele Web Safety filter the HTTPS requests too. Create a certification authority in System / Cert Manager.

    Click Save.

    This certificate will be used to bump the HTTPS connections . Go to Services / Proxy Server, scroll to SSL Man-in-the-Middle filtering and fill the fields as indicated on the following screenshot. Note we are not filling the port settings as we are not doing transparent HTTPS filtering for now. Save and restart Squid service.

    If you navigate to google.com you may clearly see the HTTPS connection was bumped.

    To get rid of this warning, we need to install the root CA certificate from pfSense box as trusted in your browsers.  Reopen your browser, navigate to Google and make sure the certificate warning is away. If you click on the lock icon in the internet address box then it clearly indicates the google.com was signed by proxy’s certificate and not by original certificate by google.

    If you try to search Google with some adult only terms (e.g. NSFW) Diladele Web Safety blocks the access to explicit contents showing its denied page.

    Resume

    Now we have HTTPS web filtering up and running and our network environment become a little safer for those who need protection at most. Next steps would be direct all client browsers to use Squid proxy,  correctly setup authentication and authorization to get user specific reports in Diladele Web Safety and optionally setup transparent HTTPS filtering. It is also advisable to enable caching DNS server on pfSense proxy to further increase speed of connections.

    Links

    1. Diladele B.V. web site at http://www.quintolabs.com
    2. Online Documentation of Diladele Web Safety https://github.com/ra-at-diladele-com/qlproxy_external/wiki/Administrators-Guide
    3. Squid Proxy Wiki on SSL Bumping http://wiki.squid-cache.org/Features/SslBump</virtualhost></virtualhost>



  • Awesome tutorial, i have a question, can i transparent proxying HTTPS without ssl bump?, thanks, keep up the good work



  • great tutorial.

    personally however i have "moral" issue's with ssl filtering.
    especially because young children will get used to being monitored from an early age. When they get older they will no longer see any harm in their privacy being violated.

    but that's just my opinion offcourse.



  • I'm stuck on installing the Diladele web safety package.. I downloaded it & uploaded it to the pfsense where it is now placed under /tmp/ & now what I should do??? how to install it?? confused



  • Good howto.

    However, qlproxy 3.2 was just released and I'm getting sql errors if I try an install using these instructions.  Note:  The download links you provide lead to qlproxy3.2.  You have go to the git repository for the project to download 3.1.

    I know it was just released, but it would be nice to troubleshoot this and figure out these sql errors.  Once I get my head around 3.1, I may try troubleshooting myself

    Thanks.



  • Wanted to follow up.

    Rafael helped me solve this.  The problem was that the sqlite dbase was not writable to Apache.  I chmodded /usr/local/var/qlproxy and the three sqlite files in /usr/local/var/qlproxy/console to 775 and all was good.

    Greg



  • Hi, best tutorial I found about Diladele and pfSense. But in my instalation the HTTPS filter not working. When I put the https://google.nl for example, this pass direct and not bump my certificate.

    Whats is wrong? I put manualy the proxy configuration in the brownser.

    When i try http://playbloy.com, diladele block my web surfing, but when I put https:// no…

    Regards



  • SSL bump did not function at the first attempt, After enabling authentication under squid 3.3.10, adding users to squid and user manager in pfsense the ssl bump function started working.



  • Hi,

    i have tested it in transparent mode without authentification.

    squid does not filter https connection

    any help?

    Rgds

    ps : using authentification, it works



  • Hi,

    got it, for the latest version (2.1.2) add first lines in custom acl (before auth)

    always_direct allow all
    ssl_bump server-first all



  • I had it working for version 2.1.2, but I would rather have the transparent proxy.  I tried to use a NAT rule to forward to the proxy as the instructions to Dansguardian had done, but Diladele does not have the same structure.  I will have to build an alternate experimental router to see if a rule for a different port will work.  I do like the additional filtering and the secure search features too much to give up on this package.  But I was pressed for time and had to get something else working.

    Meanwhile, I have successfully installed Dansguardian with a transparent proxy.  It seems to work well and I have successfully tested it with some of the sites missed by OpenDNS and Squidguard (Diladele was also successful).  However I think that the blacklist that I loaded in completely shut down YouTube.  My wife was very understanding :).  I have also upgraded to 2.1.3 as of tonight.  I had to only reinstall the Dansguardian package to get it working again.

    If I can get time this weekend, I'll try to setup my second experimental router with Diladele.  I sure would like some help with this!



  • I have the transparent proxy working for pfSense 2.1.3, but I'm not 100% sure of why it is working.  I have some steps that I used to get this working.

    1.  Install pfSense 2.1.3
    2.  Install Diladele Software per Sichent's very helpful, detailed instructions
    3.  Grab the first DNS entry (after 127.0.0.1 if present) in the DNS servers portion of the Status->Dashboard page
    4.  Paste this DNS ip into the first DNS Server box in the DNS servers portion of the System->General Setup page
    5.  Save
    6.  Navigate to Services->Proxy server page
    7.  Check the Transparent HTTP proxy checkbox and apply to the LAN interface.  All other settings in this section are defaults.
    8.  Save
    9.  Restart Squid
    10.  Disable the proxy server implementations in your browser.
    11.  Test that the Diladele software is working by accessing a porn site and also through google.

    If this does not work, please also try rebooting the router.  I hope this can work for others.



  • Has there been any push to package and maintain this by anyone?



  • Great topic. Am trying to follow the steps but am having this error when installing ….Include file squid.inc could not be found for inclusion.  I am using pfsense 2.1.3-RELEASE (amd64) running on FreeBSD  8.3-RELEASE-p16...help please



  • @toddw911:

    I have the transparent proxy working for pfSense 2.1.3, but I'm not 100% sure of why it is working.

    If this does not work, please also try rebooting the router.  I hope this can work for others.

    I don't believe you need to setup a static DNS, but what you put down should get http proxying/filtering working via squid.
    This should not, however, transparently filter your https sites, so I would be surprised if it worked on google. (Though I am only caching, so if Diladele or it's dependencies enabled transparent ssl forwarding, I would not know)

    Notes for 'squid-dev 3.3.10 pkg 2.2.2' package (on pfSense 2.1.3-RELEASE (amd64) ):
    1. The aforementioned library files have been added to the squid package and do not need to be manually installed.
    2. You need to add 'always_direct allow all; ssl_bump server-first all' to the 'Custom Settings/Custom ACLS (Before_Auth)' section in 'Services ->Proxy Server'
    3. The 'Custom Settings / Custom Options' field is now 'Custom Settings/Custom ACLS (Before_Auth)' (and (After_Auth)). So keep this in mind when reading sichent's 'Integrate Squid Proxy and Diladele Web Safety' instructions.

    To setup transparent https caching: (posting here because it seems relevant & I was unable to find it documented anywhere)
    1. Follow the guide's instructions for creating  and installing an internal certificate authority
    2. Goto 'Services ->Proxy Server' and Check 'HTTPS/SSL interception'
    3. Set 'SSL Intercept interface(s):' to 'loopback'  &  'SSL Proxy port:' to '3129'. Then Select the CA you created (For most it should already be selected).
    4.  (squid-dev 3.3.10 pkg 2.2.2 specific) Scroll to 'Custom Settings/Custom ACLS (Before_Auth)' section and Add 'always_direct allow all; ssl_bump server-first all' or your preferred ssl_bump setting there. (This was added automatically in previous packages. It's removal is likely a bug)(This is needed for manually proxied connections as well)
    5. Goto 'Firewall -> NAT'  and Under 'Port Forward' Click the Plus Button to add a new entry.
    6. Set
    Interface: LAN  |  Protocol: TCP
    Source:  any any (you may wish to set this to a specific ip or alias. At least until you confirm it's working properly)
    Destination: NOT (Check this)
        Choose 'LAN address' OR 'Single Host/alias' and add pfsense's lan ip (Else squid will lock you out of pfsense because it doesn't like pfsense's self signed certificate, you may also wish to add pfsense's ip in to squid's bypass list)
    Destination Port: from HTTPS to HTTPS
    Redirect target IP: 127.0.0.1  |  Redirect target port: (other)  3129

    Note:
    0. I did not cover filtering or transparent http proxying (Just check 'Transparent HTTP proxy', select the lan Interface, and follow sichent's instructions for Diladele)
    1. Squid currently has issues verifying some sites, for which an error page will be displayed. (ex. https://moto360.motorola.com/)
    2. There should be little reason for you to need transparent https proxying (Add the proxy setting when installing the certificate).
    3. Although Saffari recognizes user added CAs on IOS devices, other applications may not (notable eg. Google Chrome). Also It appears that you cannot add CAs to android devices.
    4. As I am not using Diladele, I cannot confirm that it will work with this. (I believe it should)

    If I forgot/misconfigured a step or you have a better/more logical idea, please tell me (I am assuming that there is a better way to not redirect traffic to pfsence than using Port Forward NOT 192.168.1.1).
    PS. I am new to PFsense and FreeBSD (almost a week after first install), So if you think you know a better way to do anything, you probably do.



  • dear sir,

    thank you very much i have 2 question

    1- does this tutorial make squid cache https sites content such as facebook pages youtube in https mode

    2- can i make browser version including the certificate generated by pfsense



  • I'm trying to block Facebook at teh office, I had no idea it would be such a task.

    my confusion lies in the proxy settings on the browser.  This isn't feasible for me to manually setup each client on the network to point ot a proxy.

    We use Windows machines many use Firefox others use IE, what options do I have.

    Maybe open dns is the better route?



  • Don't forget that by doing this, you're man-in-the-middling your own connections and breaking server authentication. Some major security issues have happened in the past over doing this. It is highly recommended against if you care about security. but if you don't care about security, it's a great way to limit what HTTPS sites LAN devices can connect to.

    Once IPv6+IPSec starts becoming popular, I'm not sure you will be able to transparently proxy anymore, because port numbers will also be encrypted. Enjoy while you can.



  • Do you mind explaining this further? AM i compromising security using squid3 if so please explain or using open DNS?

    Thank you,



  • Worked great for me! Few notes.

    1. Diladele is a paid for subscription but you do get a trial by default.
    2. To import CA on Android I used an app from the play store called Certificate Installer https://play.google.com/store/apps/details?id=it.nicola_amatucci.android.certificate_installer
    3. Works nice since I also use this CA to self sign certs for websites and now they are all trusted.
    4. Android will now show a warning about the network bring monitored, you can remove it with a Xposed module if you like.
    5. It filtered explicit YouTube when using a browser on Android but it does not filter when using the YouTube App. I haven't checked the logs yet to see how the app is connecting or if there is a way to modify the rules/policies to filter when using the app.

    @nambi if you want to block Facebook as a whole just use dnsmasq and add faceboom.com and point it to some other IP.



  • @nambi:

    Do you mind explaining this further? AM i compromising security using squid3 if so please explain or using open DNS?

    Thank you,

    All HTTPS proxies compromise security in order to gain surveillance abilities. Sometimes this is a requirement, like schools needing to filter what content students may access. But in doing so, the client loses the ability to validate what server they are connecting to because the proxy is connecting for them.

    If you're in a business where you don't need the ability to validate remote HTTPS servers, then it's not an issue, but with more businesses moving services into the cloud, you need to be careful how to setup your networks.

    Window's semi-recently had a security issue because Windows Update was being forced through transparent HTTPS proxies and malware took advantage of this and could infect other local machines and relatively easily convince other machines to install malicious software via Windows Update. Microsoft said (paraphrased)"They did this to themselves by using HTTPS proxies for Windows Update, which is a bad idea".



  • Hi,

    if I use ssl interception, SquidGuard will be bypassed because of ssl_bump server-first all.
    Has anyone found a solution to that problem?

    SSL inspection alone is a little bit useless in my opinion.



  • Thanks. You helped me figure out why 2.1.5 was not capturing SSL properly w/Squid3-dev and Diladele. Now all the parts work properly!



  • Could you please give us more information how you solved the problem so that other users may benefit from it.



  • Sorry im new here in pfsense. I can't access diladele




  • @webstor:

    Could you please give us more information how you solved the problem so that other users may benefit from it.

    I just followed sichent's guide and the other related posts in this thread.

    One note, however… after using Diladele + Squid3-dev for around 2 months now, I had to disable SARG to get it to keep working consistently. For some reason, while also using the SARG package, I would get frequent disconnects of Squid. After a forced restart (of Squid), everything was fine for a couple of days, and then the proxy would spontaneously stop and I'd lose all filtering. Disabling SARG seems to have fixed the problem (running over 3 weeks now with no hiccups).



  • @webstor:

    Could you please give us more information how you solved the problem so that other users may benefit from it.

    I don't believe that I answered your question very well a few weeks ago. To be more specific, this is what helped me to get it working completely (in particular the HTTPS filtering):

    @firefish5000:

    To setup transparent https caching: (posting here because it seems relevant & I was unable to find it documented anywhere)
    1. Follow the guide's instructions for creating  and installing an internal certificate authority
    2. Goto 'Services ->Proxy Server' and Check 'HTTPS/SSL interception'
    3. Set 'SSL Intercept interface(s):' to 'loopback'  &  'SSL Proxy port:' to '3129'. Then Select the CA you created (For most it should already be selected).
    4.  (squid-dev 3.3.10 pkg 2.2.2 specific) Scroll to 'Custom Settings/Custom ACLS (Before_Auth)' section and Add 'always_direct allow all; ssl_bump server-first all' or your preferred ssl_bump setting there. (This was added automatically in previous packages. It's removal is likely a bug)(This is needed for manually proxied connections as well)
    5. Goto 'Firewall -> NAT'  and Under 'Port Forward' Click the Plus Button to add a new entry.
    6. Set
    Interface: LAN  |  Protocol: TCP
    Source:  any any (you may wish to set this to a specific ip or alias. At least until you confirm it's working properly)
    Destination: NOT (Check this)
        Choose 'LAN address' OR 'Single Host/alias' and add pfsense's lan ip (Else squid will lock you out of pfsense because it doesn't like pfsense's self signed certificate, you may also wish to add pfsense's ip in to squid's bypass list)
    Destination Port: from HTTPS to HTTPS
    Redirect target IP: 127.0.0.1  |  Redirect target port: (other)  3129



  • Since Diladele is a paid service, is there a way to use one of the other free packages that are already in the package list (DansGuardian, etc.)?

    Also, instead of setting up SSL proxies and risking security issues, would it be easier using something like OpenDNS web filtering to do the filtering on the DNS level?

    https://www.opendns.com/enterprise-security/solutions/web-filtering/



  • So, after re-installing pfsense 2.1.5 (64-bit version), Squid3-dev, and Diladele 4.0…. thunk... no proxy serving or filtering worked. After A LOT of frustrated effort, I've finally got it working again. Here are the settings and steps that I used to get it working again. Hopefully, this will save some frustration for someone else (or perhaps remind myself if this happens to me again... knock on wood... LOL)

    On another note... to clarify a point others have made but which I did not fully understand until I went through this process again... you cannot - from what I can tell - have a transparent SSL / HTTPS proxy with Squid and Diladele. You CAN setup a SSL proxy that is not transparent, but that requires accepting the pfSense CA on every device you want to do this for. In my situation, this wasn't helpful. However, I am happy with non-HTTPS as it catches most issues I was concerned about (home network protection and monitoring).

    These instructions pertain to setting up the following

    • Squid3-dev proxy server
    • Diladele version 4.0 Web Safety
    • Transparent proxy for HTTP (non-SSL) ONLY (details on also filtering SSL below)

    Steps:

    1. Install pfsense 2.1.5
    2. Setup basic configuration, firewall rules, etc.
    3. Install package Squid3-dev (I used version 3.3.10 pkg 2.2.8)
    4. Establish proxy server settings and test via Real Time reporting in squid
    5. Reboot
    6. Install Diladele using scripts
    7. Reboot
    8. Login to Diladele Web interface and verify operation
    9. Add custom ACLS to pfsense Proxy Server
    10. Restart Proxy Server
    11. Test Diladele again and verify it is now reporting real time monitoring

    Visit the Diladele pfSense tutorial, and you'll find the scripts I'm referring to. They make installing Diladele and its dependencies MUCH easier. You can find that tutorial here: http://docs.diladele.com/tutorials/filtering_https_traffic_squid_pfsense/integrate.html

    Here are the proxy server settings that I used in pfSense (Services > Proxy Server):
        - Proxy interface(s): Highlight and select (use CTRL + Left Click) all the interfaces you want to use the proxy for HTTP traffic
        - Proxy port 3128
        - Allow users on interface: check this box if you want to use a transparent proxy and have all users go thru the proxy by default
        - Transparent HTTP proxy: check this box
        - Transparent Proxy interface(s): check off all the interfaces; presumably you'll want them to match the interfaces above
        - Bypass proxy for these source IP's: I left this blank
        - Bypass proxy for these destination IPs: Put anything here you don't want cached; I included my pfSense box's IP, eveonline.com, and crashplan.com (I use crashplan for offsite backups)
        - Scroll down to Logging Settings > Enabled logging: check this box. I found this to be key. If I didn't check this, Diladele did not work
        - Scroll down to Custom Settings > Custom ACLS (Before_Auth): enter the string of text found in the Diladele website tutorial for pfSense or previously in this thread

    That's it. Make sure you press Save and restart Squid.

    Now launch Diladele and do some surfing. Watch Diladele's real-time monitoring/surfing. It should work now.



  • @longhorn:

    So, after re-installing pfsense 2.1.5 (64-bit version), Squid3-dev, and Diladele 4.0…. thunk... no proxy serving or filtering worked. After A LOT of frustrated effort, I've finally got it working again. Here are the settings and steps that I used to get it working again. Hopefully, this will save some frustration for someone else (or perhaps remind myself if this happens to me again... knock on wood... LOL)

    On another note... to clarify a point others have made but which I did not fully understand until I went through this process again... you cannot - from what I can tell - have a transparent SSL / HTTPS proxy with Squid and Diladele. You CAN setup a SSL proxy that is not transparent, but that requires accepting the pfSense CA on every device you want to do this for. In my situation, this wasn't helpful. However, I am happy with non-HTTPS as it catches most issues I was concerned about (home network protection and monitoring).

    These instructions pertain to setting up the following

    • Squid3-dev proxy server
    • Diladele version 4.0 Web Safety
    • Transparent proxy for HTTP (non-SSL) ONLY (details on also filtering SSL below)

    Steps:

    1. Install pfsense 2.1.5
    2. Setup basic configuration, firewall rules, etc.
    3. Install package Squid3-dev (I used version 3.3.10 pkg 2.2.8)
    4. Establish proxy server settings and test via Real Time reporting in squid
    5. Reboot
    6. Install Diladele using scripts
    7. Reboot
    8. Login to Diladele Web interface and verify operation
    9. Add custom ACLS to pfsense Proxy Server
    10. Restart Proxy Server
    11. Test Diladele again and verify it is now reporting real time monitoring

    Visit the Diladele pfSense tutorial, and you'll find the scripts I'm referring to. They make installing Diladele and its dependencies MUCH easier. You can find that tutorial here: http://docs.diladele.com/tutorials/filtering_https_traffic_squid_pfsense/integrate.html

    Here are the proxy server settings that I used in pfSense (Services > Proxy Server):
        - Proxy interface(s): Highlight and select (use CTRL + Left Click) all the interfaces you want to use the proxy for HTTP traffic
        - Proxy port 3128
        - Allow users on interface: check this box if you want to use a transparent proxy and have all users go thru the proxy by default
        - Transparent HTTP proxy: check this box
        - Transparent Proxy interface(s): check off all the interfaces; presumably you'll want them to match the interfaces above
        - Bypass proxy for these source IP's: I left this blank
        - Bypass proxy for these destination IPs: Put anything here you don't want cached; I included my pfSense box's IP, eveonline.com, and crashplan.com (I use crashplan for offsite backups)
        - Scroll down to Logging Settings > Enabled logging: check this box. I found this to be key. If I didn't check this, Diladele did not work
        - Scroll down to Custom Settings > Custom ACLS (Before_Auth): enter the string of text found in the Diladele website tutorial for pfSense or previously in this thread

    That's it. Make sure you press Save and restart Squid.

    Now launch Diladele and do some surfing. Watch Diladele's real-time monitoring/surfing. It should work now.

    Hopefully just missed a step in here but tried a couple of times.  After going through the steps as described above I get an ICAP protocol error - essentially after adding the custom acls (before AUTH).  Can anyone provide any hints.  I did see on the diladele's website under the licensing section that the 2 month trial license had been removed - is that what's need to get it running?

    Cheers
    Gavin

    ICAP ERROR
    The following error was encountered while trying to retrieve the URL: http://www.google.co.uk/
    ICAP protocol error.
    The system returned: [No Error]
    This means that some aspect of the ICAP communication failed.
    Some possible problems are:
    The ICAP server is not reachable.
    An Illegal response was received from the ICAP server.

    FROM DILADELE WEBSITE
    The trial license which was active during the last year has finally expired. Please purchase the commercial license if you think product is worth and it if not - share your thoughts and we will try to make it better!



  • Just on the above post the qlproxy service isn't starting because of an expired license key.  Hoping Diladele can provide a month trial so we can test it out.



  • Could you please update it so it works with pfSense 2.2.2 and squid 3.4.10_2 pkg 0.2.8.



  • On 2.2.2-RELEASE (i386) with these package versions:

    • squid3 0.2.8
    • squidGuard 1.9.14
    • squidGuard-devel 1.5_1beta pkg v.1.5.6

    I get one error saying squid needs the module to bypassing SSL

    # squid -k check
    2015/05/22 14:15:33| FATAL: tproxy/intercept on https_port requires ssl-bump which is missing.
    FATAL: Bungled /usr/local/etc/squid/squid.conf line 7: https_port 127.0.0.1:3129 intercept
    Squid Cache (Version 3.4.10): Terminated abnormally.
    CPU Usage: 0.024 seconds = 0.018 user + 0.006 sys
    Maximum Resident Size: 36000 KB
    Page faults with physical i/o: 0

    It seems we need a new package from the pfSense developers



  • Hi! Any progress on the squid update to fix the HTTPS filtering problem we have on 2.2.2?



  • I have everything setup and it is filtering HTTPS sites correctly but I do have a couple of issues with using iPad’s/Android’s on the network.

    I have installed the certificates but It seems that some of the apps do not like going through the man in the middle filtering, the app store, banking apps and Facebook to name a few.

    Is there a way to setup some sites to bypass the proxy filtering completely?



  • Is there a way to setup some sites to bypass the proxy filtering completely?

    The main Squid config page has this option:

    Bypass proxy for these destination IPs



  • KOM

    I have tried using the bypass proxy with apples iTunes store address but for some reason I can not get it to work for various Apps.



  • I Have about 70 workstations on the network.

    Installing a certificate in each and every browser would be a terrible idea for me.

    :o



  • Installing a certificate in each and every browser would be a terrible idea for me.

    And everyone else, too.  That's why WPAD is, IMO, the preferred method.



  • How about using E2Guardian? Although I can't find it in the packages available, it's probably being ported sometime soon. I've read somewhere that it supports HTTPS filtering.