Filter porn, virus's and ads with squid3, HAVP, Dansguardian and privoxy.



  • Create an internet filter using Pfsense, squid3, HAVP, Dansguardian and privoxy. This filter will screen incoming web traffic for virus's, banner ads and inappropriate content such as porn. The filter will also be set up transparently so there is no client configuration necessary or that kids/employees can bypass the filter.

    To set the filter up you will ideally need an old PC with 2 network ports. The machine used in this howto is an old P4 1.7Ghz with 1GB ram. One network port will plug directly into your network and one into your modem/router. This way there is no direct path from the network to the internet without going though the filter box.

    The Main reason for writing this howto is because I simply couldn’t find a howto out on the interwobble on this setup. I could find plenty of howto's to set up Pfsense with virus and porn blocking but non whatsoever on setting up Pfsense with ad blocking, let alone all 3 combined. I used to use IPcop with this setup for years instead of Pfsense but my network outgrew what could be done with IPcop.

    Please note that Privoxy is not an official Pfsense addon so you may not get any support for it on these forums. Also I originally wrote this howto for howto forge (http://www.howtoforge.com/node/7237). All of this howto except this paragraph is a direct copy of the howto forge howto, which is also a direct copy and past from my personal internal docs wiki.

    First head over to www.pfsense.org and get your self a Pfsense ISO and burn it to disk. Once you have done that install it onto what will become your filter box. There are plenty of howto's out there on how to burn ISO's and default install of Pfsense so this will not be covered in this howto. Also make sure you have installed any updates for Pfsense! The version of Pfsense used in this howto is 2.0.1.

    Network parameters for this howto

    WAN

    *IP - 192.168.1.10
    *netmask - 255.255.255.0
    *gateway - 192.168.1.1

    LAN

    *IP - 10.20.99.254
    *netmask - 255.255.0.0
    *This is the gateway so no gateway is needed however once the filter is set up clients on your lan will use this machine as there gateway.

    You may need to change these perimeters for your own network

    Once you have a default install of pfsense first make sure you can access the internet from a client inside the network through the pfsense box.

    Client configuration

    *IP - 10.20.99.5
    *Netmask - 255.255.0.0
    *Gateway - 10.20.99.254 (Pfsense lan IP)

    If you can access the internet things are looking good and you have a default install of Pfsense up and running.

    Now for the real work to begin.

    Head over to your Pfsense lan IP in your browser, 10.20.99.254 and login (defaults are user: admin Pass; pfsense)

    First we need to change the default port on which Pfsense serves the config pages. In the system menu click on advanced and change the TCP port, I used 8443. Save the setup and enter the new address (10.20.99.254:8443) in your web browser and relogin. We have done this because to make the filter transparent we need to use and redirect port 80.

    Now in the system menu select packages, and then click the available packages tab. Install Squid3, this is our proxy server to cache websites and help speed things up a little (Very handy when a throttled 2.5Mbps connection, even more annoying when 2 streets away you get 350Mbps fibre). Once Squid is installed the config page will appear in the services menu under proxy server.

    The only things we have to change in squid is the listening interface. Make sure only the loopback interface is selected. DO NOT tick the transparent proxy check box! And add the lines

    acl all src 0.0.0.0/0.0.0.0
    never_direct allow allsrc
    cache_peer 127.0.0.1 parent 3125 0 name=havp no-query no-digest no-netdb-exchange default

    to the custom options section and save the settings

    You can now see if squid is running by clicking on services in the status menu.

    Next we setup the virus scanning.

    We need to install the 'HAVP antivirus' package. We do this the same as we did for squid. System menu and select packages and click on the available packages tab and install 'HAVP antivirus'. Once installed the config page will be under Antivirus in the services menu.

    In the config page for the antivirus click on the http proxy tab and make sure that Enable is ticked, proxy mode is set to 'standard' (DO NOT SELECT "PARENT FOR SQUID", this can cause problems after a reboot.) and proxy interface is set to loopback. Set Parent proxy to 127.0.0.1:8118. Then save the settings. You can again check that HAVP is running by viewing the services page in the status menu.

    Now we will set up Dansguardian.

    This will block sites deemed as inappropriate such as porn. Again we do this by going to the system menu, select packages and click on the available packages tab. Then install Dansguardian. The Dansguardian config is in the service menu under you guessed it in Dansguardian.

    Again not a lot to set up in the config just make sure the 'Enable Dansguardian' checkbox is ticked and the listening interface is loopback. Because Dansguardian runs before squid we set dansguardian to pass its traffic to squid by setting the proxy IP to 127.0.0.1 and the proxy port to 3128 and save the config.

    You can check that Dansguardian is running by checking Status > Services.

    So far so good. Now for the banner ad blocking. Unfortunately there is no Pfsense package for doing this which makes setting up privoxy a little more complex but well worth it. However Privoxy is a great piece of software for doing this. To install Privoxy you need shell access to the Pfsence box either directly sat at the machine with a keyboard or via SSH.

    at the Pfsense command prompt enter

    pkg_add -r ftp://ftp-archive.freebsd.org/pub/FreeBSD-Archive/ports/i386/packages-8.1-release/Latest/privoxy.tbz

    This should install privoxy.

    While still in the command line run the following commands

    mkdir /var/run/privoxy
    chown privoxy:privoxy /var/run/privoxy
    touch /usr/local/etc/rc.d/privoxy.sh
    chmod +x /usr/local/etc/rc.d/privoxy.sh
    squid -z

    Now back in the Pfsense webconfig head to Edit file in the diagnostics menu and click browse. Select the /usr/local/etc/privoxy/config file. Due to some funny things inside pfsense it causes problems for privoxy reading the log file after a reboot, so I will disable loggings by commenting out the logdir and logfile directives by changing the lines

    logdir /var/log/privoxy

    to

    #logdir /var/log/privoxy

    and the line

    logfile logfile

    to

    #logfile logfile

    then save the file.

    Now we need to edit the /usr/local/etc/rc.d/privoxy.sh file. Click browse again and select /usr/local/etc/rc.d/privoxy.sh.

    The file will be empty. Past the following into the file and then save it

    #!/bin/sh

    rc_start() {
    /usr/local/etc/rc.d/privoxy forcestart

    }

    rc_stop() {
    /usr/local/etc/rc.d/privoxy stop
    }

    case $1 in
        start)
            rc_start
            ;;
        stop)
            rc_stop
            ;;
        restart)
            rc_stop
            rc_start
            ;;
    esac

    Now back on the command line on your Pfsense box issue the command

    sh /usr/local/etc/rc.d/privoxy.sh start

    This should start privoxy.

    You can check that privoxy is running by running

    /usr/local/etc/rc.d/privoxy forcestatus

    We are nearly done!

    We have set each service to only listen on the loopback service. This is to stop users on your network manually setting proxy settings in there browser and bypassing any of the filtering. However this makes it so no one can access any of the filters. But we we want to set it up transparently which just so happens to fix this problem. To do this go to Firewall > NAT in the web config. Click on the + icon on right to create a new nat rule.

    Set interface to LAN
    Set Destination to any
    Set Destination port range to HTTP
    Set redirect target IP to 127.0.0.1
    Set redirect target port to 8080
    Leave all other settings as they are

    and save the rule and click apply changes

    To make sure all is fine reboot your pfsense box leave it 5 mins to reboot and try accessing the internet, make sure porn, virus's and ads are still blocked.

    You should now have a working Pfsense box filtering out ads, Virus's and porn.

    You can check that virus's are blocked by trying to download the test virus from http://www.eicar.org/85-0-Download.html You should get an access is denied message when trying to download one of the test virus's.

    Check porn access by heading over to a porn site such as www.redtube.com. Again you should get an access is denied message.

    Now check for ad and banner removal by heading over to a site known to have ads on it such as www.slashdot.org. All being well there shouldn't be any ads at the top or sides of the page.

    If all is well you now have a very powerful internet filter built around Pfsense.

    If you find any errors or improvements please post them below.



  • Berrance, is this the correct Parent proxy for HAVP?  Parent proxy to 127.0.0.1:8118?  Shouldn't be 3125, 3128, or 8080?



  • I believe so. The setup is,

    Client > Nat to dansguardian (127.0.0.1:8080) > squid (127.0.0.1:3128) > Havp (127.0.0.1:3125) > Privoxy (127.0.0.1:8118) > internet

    With no parent proxy in Havp you will get the internet but not running through privoxy so no ad blocking.

    Please drop another line if your having problems.

    Berrance



  • I was able to get Dansguardian, Squid, and HAVP working before system resources were getting tight so I did not install Privoxy. Although my configuration was a little different.  I will post it later.

    EDIT:

    I did almost everything up above except Privoxy (for now?) and the following:

    1.  I place Squid on Transparent instead of turning it off.
    2.  I clicked LAN\Opt1 interfaces on Squid and HAVP.  Squid proxy point at HAVP (I tried loopback but did not work)
    3.  I did set HAVP as parent for squid. (I tried standard and transparent but did not work)
    4.  I clicked LAN\Opt1\Loopback for Dansguardian - Proxy server point at Squid (tried loopback but did not work)
    5.  I NAT'd Opt1 traffic to Opt1 interface IP address and LAN Subnet to the LAN interface IP address instead of the Loopback. (tried 127.0.0.1  but did not work).
    6.  Added ignore_expect_100 on to Squid Proxy customizations.
    7.  Tested connectivity and "questionable material" was blocked and eicar test was blocked as well.

    On a side note, do you know how many resources does Privoxy take up?  I am hovering around 80% with all of the services I have running.

    The firewall hardware consists of dual P-III 850 and 1 gig of RAM (everything is maxed).



  • Glad to hear you've got going. Just a few notes on your points from my experience (may help other people). I could only get it working by doing it in the order from my howto did a reinstall to check but thats not to say I have missed something out.

    1. For me placing squid on transparent made the web traffic bypass Dansguardian as Dansguardian was running before squid in the filter chain. Squid transparent mode redirected all http traffic to Squid which is why I added a nat rule for all traffic on lan1 port 80 and redirect it to 127.0.0.1:8080 provided dansguardian is listening on loopback making it transparent

    2. When squid is in transparent mode You may need squid listening on Lan.  If you had Havp set as parent for squid and listening on loopback what IP is shown in the Integrations section on the squid config page? You may need to change it from your lan address to 127.0.0.1

    3. For me it sorta worked with parent for Squid. But for me and a big But, after installing Dansguardian and restarting squid or rebooting the machine I had to re add the line

    acl all src 0.0.0.0/0.0.0.0

    to squid.conf before the lines

    never_direct allow allsrc
    cache_peer 127.0.0.1 parent 3125 0 name=havp no-query no-digest no-netdb-exchange default

    that Havp added when set to Parent for Squid and restart squid. Which again didnt survive a restart. To fix this I set Havp to S5.  I NAT'd Opt1 traffic to Opt1 interface IP address and LAN Subnet to the LAN interface IP address tandard and manually add  the above lines in order to the Custom Options section on the proxy conf page making sure the integrations section is empty. To get the integrations section to stay empty after Havp had been set to Parent for squid I had to uninstall and reinstall Havp (Possibly Squid aswell I cant remember). For testing purposes I just set my browsers proxy to squid also for testing I did enable Squid to listen on Lan aswell.

    4. see 1.

    5. What address/port where you listening on/redirecting to? provided Dansguardian is listening on loopback port 8080 you should be fine redirecting traffic from Lan/Opt1 port 80 to 127.0.0.1 port 8080

    7. All should well work well with all services running whilst listening on Lan but should a user know the ip and port numbers of the services running they could bypass vital parts of the chain by changing their browsers proxy settings.

    I used to run a similar set up but without Havp on a dual P-III 600Mhz with 750 MBs ram set up with IPCop a few years ago and that used to max out pretty easy. The setup I used in the howto is an old P4 1.7Ghz with 1GB ram. Still running now with that how to.



  • berrance,
    Great contribution.  :)

    Just some notes:

    • Dansguardian already has antivirus integration, you do not need HAVP.  ;)

    • Are you sure dansguardian blacklists can't deny ads?

    • passing traffic through a lot of daemons, may slow down internet access.

    • HTTPS access will not be filtered on this setup



  • Marcello,

    What is your recommended configuration?

    Thanks!



  • @KeltecRFB:

    What is your recommended configuratio?

    User -> dansguardian -> squid -> internet
                    (With clamav)



  • Any good documentation on just using Squid and Dansguardian or point me to the best thread because everything I have tried fails.



  • @KeltecRFB:

    Any good documentation on just using Squid and Dansguardian or point me to the best thread because everything I have tried fails.

    Did you tried this?

    Installing the Dansguardian package in PFSense - One user's experience



  • @marcelloc:

    Did you tried this?

    Installing the Dansguardian package in PFSense - One user's experience

    Thank you, I figured out what was wrong … :-)

    Now to figure out anti-virus.



  • @KeltecRFB:

    Now to figure out anti-virus.

    just select clamdscan on system -> dansguardian -> general and save config



  • @marcelloc:

    @KeltecRFB:

    Now to figure out anti-virus.

    just select clamdscan on system -> dansguardian -> general and save config

    I did just that but then Danguardian pops up with a message stating it is blocking the site because it can not scan it, Categories:  Content scan.  What else needs to be done; do I also need to disable a categories filter?

    Thanks!



  • Force a frashclam update on console and then restart dansguardian(or save config)



  • @marcelloc:

    Force a frashclam update on console and then restart dansguardian(or save config)

    Did that, same response - website blocked, unable to scan content…

    In addition, if I turn on clamdscan, and then turn it off because it is not working properly yet, I have to reboot the box for internet connectivity to be restored.

    Should I try to enable it and reboot?



  • Are you on latest dansguardian package version?



  • @marcelloc:

    Are you on latest dansguardian package version?

    2.12.0.3 pkg v.0.1.7



  • I see other people have had this problem, but nothing sticks out yet.



  • Marcello,

    I tried doing this and the download was successful, however, the deny page for Cisco is still displayed when clamdscan is activated.

    http://forum.pfsense.org/index.php/topic,52163.0.html



  • Problem resolved…

    I made a change thinking that was the problem for something else but it wasn't and caused a problem with clamdscan working properly.  Now everything is working.



  • Hi all,

    I followed this, and got it to work, but found that things eventually slowed to a crawl. I think my box may be under powered, but for me, blocking Ads network wide would be enough.

    Can privoxy work standalone?  I get an error in my browser when I try to NAT directly to 8118 :

    invalid header received from client

    Privoxy is listening on localhost:8118

    I would appreciate any tips.

    Phob



  • Did you tried only dansguarguardian (with clamav enabled) + squid?



  • No I didn't - I thought privoxy would be more light weight, if I only need to block ads…



  • You are having issues using only privoxy?



  • Sorry for the delay in my reply.

    Yes, I was receiving an error message for every page I tried to browse when I pointed the NAT rule directly to port 8118 (Privoxy):

    "invalid header received from client"

    I finally installed Squid3 again, and set 8118 as the parent.  I still wasn't having any luck until I changed Squid to transparent mode and eliminated the NAT rule to 8118.

    Now things seem to be working, but I'm running Squid when I don't think I should need to for a Privoxy only setup.



  • I get the following error when trying to start privoxy:
    Starting privoxy.
    /libexec/ld-elf.so.1: Shared object "libpcre.so.0" not found, required by "privoxy"

    Has anyone else had this?  What did you do to resolve the issue?

    thanks



  • When adding the privoxy package, here is a warning I get…  is there a privoxy version that uses pcre-8.32?

    Fetching ftp://ftp-archive.freebsd.org/pub/FreeBSD-Archive/ports/i386/packages-8.1-release/Latest/privoxy.tbz... Done.
    pkg_add: warning: package 'privoxy-3.0.16_1' requires 'pcre-8.02', but 'pcre-8.32' is installed



  • I follwed the steps on the first post and got everything working w/o a hitch.

    I also tried just the Dansguardian -> squid method which worked but for some reason it was much slower and didnt block any ads vs the dansquardian -> squid-> HAVP-> privoxy method (which makes no sense conceptually) and i could never get dansguardian's built in clamd to work properly (which also makes no sense)

    besure to see this on the webupload bug fix as well http://forum.pfsense.org/index.php/topic,58442.15.html (which works as this post was sucessful)



  • @krk:

    I get the following error when trying to start privoxy:
    Starting privoxy.
    /libexec/ld-elf.so.1: Shared object "libpcre.so.0" not found, required by "privoxy"

    Has anyone else had this?  What did you do to resolve the issue?

    thanks

    I had this issue w/ a brand new install on another box just now. do a "find / -name libpcre.so.0" and ls -l the files, you should see that they're linked to libpcre.so.3, create a sym link to libpcre.so.0 instead and it should solve the privoxy issue. I have no idea if the new symlink location will screw anything else up. so far it hasn't.

    Another tidbit is that you have to install dansguardian first before you install squid3 (if you want to use squid3). Dasguardian will install squid 2.7.x as a dependency which will cause pfsense to revert to using the older version if you have squid3 installed beforehand.



  • I was getting /libexec/ld-elf.so.1: Shared object "libpcre.so.0" not found, required by "privoxy" to and i think there is a fast way to fix it

    if your on a x64 box install the x64 software

    pkg_add -r http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/ports/amd64/packages-8.1-release/Latest/privoxy.tbz

    after i did that it work for me.



  • @ion9:

    I was getting /libexec/ld-elf.so.1: Shared object "libpcre.so.0" not found, required by "privoxy" to and i think there is a fast way to fix it

    if your on a x64 box install the x64 software

    pkg_add -r http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/ports/amd64/packages-8.1-release/Latest/privoxy.tbz

    after i did that it work for me.

    I tried this, both using …/amd64/packages-8.1-release/Latest/privoxy.tbz , .../amd64/packages-8.2-release/Latest/privoxy.tbz and .../amd64/packages-9.0-release/Latest/privoxy.tbz . I still get "libpcre.so.0 not found".

    2.1-RC0 (amd64)
    built on Sat Jun 8 09:20:03 EDT 2013
    FreeBSD 8.3-RELEASE-p8

    I've tried the same on 2.0.3 (amd64) without success.

    Anyone successfully installed privoxy on pfSense? What version of pfSense and privoxy are you using?



  • @mosfet:

    Anyone successfully installed privoxy on pfSense? What version of pfSense and privoxy are you using?

    Did you tried installing pcre package too? what version of pcre do you have on your system?



  • @marcelloc:

    @mosfet:

    Anyone successfully installed privoxy on pfSense? What version of pfSense and privoxy are you using?

    Did you tried installing pcre package too? what version of pcre do you have on your system?

    Well, I've not tried to install pcre as a package, but it is installed. The last attempt I tried, are using pcre-8.32 and privoxy-3.0.17.

    This leads to: "/libexec/ld-elf.se.1: Shared object "libpcre.so.0" not found, required by "privoxy"".



  • @mosfet:

    This leads to: "/libexec/ld-elf.se.1: Shared object "libpcre.so.0" not found, required by "privoxy"".

    what version of libpcre you get on your system?

    find / -name "libpcre.so*"



  • @marcelloc:

    @mosfet:

    This leads to: "/libexec/ld-elf.se.1: Shared object "libpcre.so.0" not found, required by "privoxy"".

    what version of libpcre you get on your system?

    find / -name "libpcre.so*"

    Output:
    /usr/local/lib/libpcre.so.3
    /usr/local/lib/libpcre.so



  • @mosfet:

    Output:
    /usr/local/lib/libpcre.so.3
    /usr/local/lib/libpcre.so

    Try to force it with a link

    ln -s /usr/local/lib/libpcre.so.3 /usr/local/lib/libpcre.so.0

    this is not the best way to workaround it but works.



  • @marcelloc:

    @mosfet:

    Output:
    /usr/local/lib/libpcre.so.3
    /usr/local/lib/libpcre.so

    Try to force it with a link

    ln -s /usr/local/lib/libpcre.so.3 /usr/local/lib/libpcre.so.0

    this is not the best way to workaround it but works.

    Wow - that worked. Thanks a million!



  • To answer Marcel's question from back in January (I know, I don't read the boards enough  ;))

    You can filter ads very effectively with DG using a Regexp filter. This gets most of them including popups, sidebars and even YouTube video embedded ads. Just add it under
    Dansguardian - Access Lists -> URL

    (^ads.|/ads/|/ads.|.ads.|adcycle|adserv|/ad.|.ad.|/ad/|.adview|/adview|pagead|adpopup|phpAds|.adscript|/adscript|/adframe|.adframe)
    (werbung|anzeige|banner|sponsors|advertising|adimg)
    (doubleclick)



  • followed the guide until my/its first error

    Select the /usr/local/etc/privoxy/config

    This file does not exist.  After I tried starting privoxy it was created. I could then edit it.

    I then encountered the libpcre error as above but could not create a link.

    putty ssh as admin

    same find output

    /usr/local/lib/libpcre.so.3
    /usr/local/lib/libpcre.so

    and 20ish error lines follow (I don't know how to copy in putty ssh / ubuntu)
    find: /var/spool/epilocks: Permission denied
    find: /var/log/squid
    find: /var  (12 more )
    find: /tmp/lighttpdcompress (4 of these)

    @marcelloc:

    Try to force it with a link

    ln -s /usr/local/lib/libpcre.so.3 /usr/local/lib/libpcre.so.0

    this is not the best way to workaround it but works.

    ln: /usr/local/lib/libpcre.so.0: Permission denied

    installed from CD a few days ago

    " 2.0.3-RELEASE (i386)
    built on Fri Apr 12 10:22:21 EDT 2013
    FreeBSD 8.1-RELEASE-p13"

    on an older Dell Optiplex "Intel(R) Core(TM)2 Duo CPU E4600 @ 2.40GHz " with one pci-e two port intel 1000 nic and an older but never before used SSD



  • Hi folks,

    I'm a newbie here who's been having a play with pfSense with two ADSL connections.  I've managed to get multi-wan working (running a traceroute to different servers goes over each WAN connection) and I've also got this guide working, but anything going over the proxy only seems to go over the one connection (I've tried whatismyipaddress.com and it always returns the same IP address).

    I just wondered if anyone could give any pointers on how I might be able to configure privoxy to go over the gateway group I've configured?  (I wasn't sure if it was possible to setup a firewall rule so anything over a certain port number goes to the gateway group?).

    Ta,

    Rob


Log in to reply