Tutorial: Using pfsense as a network wide adblocker.

bunny

I'm not quite sure if you guys have a tutorial section so mods forward this thread to its right place if it needs to be moved please. -thanks

This is a tutorial on how to get pfsense to act as your own personal adblocker using easylist: http://easylist.adblockplus.org/

Personal note:

This doesn't seem to work atm with flash based ads and text based ads(google ads). It does not collapse most ads either. In other words, if you're using firefox then using the adblock plus plugin will work better than this even if they filter the same ads. The reason it doesn't work for everything is because it is not editing the html sent over the net. It is just blocking urls for anything that tries to pass through squid which is pictures. Anything that is dynamic like most flash ads or websites like, for example, hulu.com do not pass through squid. If anyone knows how to block flash ads, block the text type ads, and collapse blocked ads then please let me know!
There really is no real reason to use squidGuard. Imho it kinda sucks. If I get around to updating this tutorial I'll alter it to block ads just with squid alone.

To do this I'm currently using:
pfSense 1.2.3-RC3
squid 2.6.21_10
squidGuard 1.3-2
Cron 0.2

If your versions are different then your paths might be different than this tutorial. In terminal using the command: find / | grep <filename or="" foldername="" you're="" looking="" for="">can help a lot for varying setups.

This tutorial does use some ssh (terminal) so if you're afraid of any sort of command prompt turn back now.

First, you want to enable ssh in pfsense if it isn't already. To do so go to System -> Advanced and make sure 'Enable Secure Shell' is checked and saved. In windows you probably will want to use putty (a freeware program. find it on google). Also note: in ssh the login username is admin and password is whatever your password is set to on the httpd.

Second, time to install squid proxy if you haven't already. Go to System -> Packages, find and install squid. I'm using the 2.6 version instead of the experimental one and it works fine.

After squid is done installing go to Interfaces -> Proxy Server to configure squid to your particular setup. My setup has LAN as my proxy interface and transparent proxy is checked. I also altered some settings in the cache management and traffic management pages.
*NOTE: If you can not get to the other squid http pages outside of general settings and lets say your router address is https://192.168.1.1/ then these would be the addresses for the other pages:
https://192.168.1.1/pkg_edit.php?xml=squid_upstream.xml&id=0
https://192.168.1.1/pkg_edit.php?xml=squid_cache.xml&id=0
https://192.168.1.1/pkg_edit.php?xml=squid_nac.xml&id=0
https://192.168.1.1/pkg_edit.php?xml=squid_traffic.xml&id=0
https://192.168.1.1/pkg_edit.php?xml=squid_auth.xml&id=0
https://192.168.1.1/pkg.php?xml=squid_users.xml
Adjust the links accordingly to your setup.

Optional: If you adjusted your cache management then you probably need to let squid apply those settings which can sometimes take a while. To do this login via ssh and type: squid -k shutdown
then wait about 10 seconds and then type: squid -z

Have you ever used Vi before? It is a powerful text editor in terminal. Personally, I hate it so instead I installed nano as a replacement which is a far simpler text editor. This is optional but recommended if you've never used Vi before. First, log in via ssh. Type 8 for shell and then type: pkg_add -r nano
after that logout since $PATH is not automatically updated.
*EDIT: I forgot about ee. If you do not want to use nano you can always use ee which is much simpler than vi. So either use ee, vi, or install nano. Use whichever text editor you want to.

Now go back to System -> Packages and install squidGuard.
After it is done installing go to Interfaces -> Proxy filter.
-Click the default tab.
-Click Destination ruleset.
-Click the access drop down on the right and set it to allow.
-Click save.
-Click the general settings tab.
-Check enabled. (top)
-Click save. (bottom)
-Click apply.

*From now on do not touch the 'Proxy filter' settings! If you touch it then it will either crash or reset everything!

Log back into the pfSense shell via ssh. Type the following:
cd /var/db/squidGuard/
rm -rf * (Optional: Deletes the clutter.)
mkdir adblock
cd adblock
nano

Now nano should load up as a blank document. Paste in this text:
/@@./d;
/^!./d;
/^[.]$/d;
s#http://##g;
s,[.?=&/|],\&,g;
s##.#g;
s,$.$,,g;

Press ctrl+x to save in nano (ee is ctrl+[). Name the file: regex.sed

Now continue typing into terminal:
fetch http://adblockplus.mozdev.org/easylist/easylist.txt
cat easylist.txt | sed -f regex.sed > expressions

cd /usr/local/etc/squidGuard/
rm blacklist.files (Optional: Deletes the clutter.)
rm squidguard_conf.xml (Optional: Deletes the clutter.)
nano squidGuard.conf

You should now see the squidGuard.conf file loaded up in nano to edit. Delete every line after 'dbhome /var/db/squidGuard'. You can do this quickly by pressing ctrl+k to remove an entire line.

Now below dbhome /var/db/squidGuard paste in:
src my_network {
ip 192.168.1.0/24
}

dest adblock {
expressionlist adblock/expressions
redirect https://192.168.1.1/sgerror.php?url=blank_img
}

acl {
my_network {
pass !adblock any
}
default {
pass none
redirect https://192.168.1.1/sgerror.php?url=blank_img
}
}

*My pfsense httpd is set to https which is not the default. You might need to change https://192.168.1.1/sgerror.php?url=blank to http://192.168.1.1/sgerror.php?url=blank or another path. To double check try loading the url in your browser. Does it load a blank page (good) or does the page time out (bad)?
*Also, ip 192.168.1.0/24 in my_network is allowed subnets to get online. You might need to change this as well.

Now type: squid -k reconfigure

Check a webpage. The adblock isn't perfect. For some sort of reason it isn't blocking google ads for me atm but for example, if I go to digg.com the ads have now disappeared! If the adblocker is not working backtrack and see where you might of typed something wrong or where your settings need to change from mine.

Congratulations if you've gotten this far successfully! There is only one more thing to do: Auto update the adblocker list weekly to keep everything up to date. This is optional but HIGHLY RECOMMENDED.

Go back to System -> Packages and install the cron package.
Go to Services -> Cron and add a new rule.

Lets say I want to update my adblock list every tuesday at 11:26 PM then I would do:
minute: 26
hour: 23
mday: *
month: *
wday: 2 (0 is sunday, 1 is monday, …)
who: root
command: fetch -o /var/db/squidGuard/adblock http://adblockplus.mozdev.org/easylist/easylist.txt ; cat /var/db/squidGuard/adblock/easylist.txt | sed -f /var/db/squidGuard/adblock/regex.sed > /var/db/squidGuard/adblock/expressions

The command will download a new adblock list and create a new expressions file from so squidGuard can use it.

If you have any questions, comments, feedback, or whatever then feel free to reply to this thread. :)</filename>

Hugovsky

I have a problem with the redirect to https. It keeps giving me this:

fwdNegotiateSSL: Error negotiating SSL connection on FD 79: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed (1/-1/0)

craibo

I have implemented this exactly as you have explained in the tutorial changing the information which relates to my system (eg. http and 192.168.10.0) how ever it now blocks all pages with "http://" still allows "https://".

So when trying to open any page it just loads a blank page.

When it didn't work the first time, I started from the beginning and tried again but still the same result.

If you have time, would you upload the tutorial letting me know how to use adblocking with squid rather then squidGuard.
Would you be able to include how you would add your own sites which also need to be bloacked but are not included in easylist.

Many Thanks

My Setup:
pfSense 1.2.3-RC3
squid 2.7.7
squidGuard 1.3-2
Cron 0.2

craibo

I have solved the problem as to why my system was crashing squid.
you need to fetch the easylist file from:

fetch http://easylist.adblockplus.org/easylist.txt

I have implemented the ad blocker using squid acl but it is not working completely still allows through some ads which it should not.
Will put up what I have done in due course

i_robot73

Found a little somthing-something on this internet thingy, might help out a few other peeps as well:

https://bugs.launchpad.net/ubuntu/+source/squidguard/+bug/316816

/@@./d;
/^!./d;
/^[.]$/d;
s#http://#^#g;
s,[.?=&/|()[],\&,g;
s##.#g;
s,$.$,,g;

@bunny:

Now nano should load up as a blank document. Paste in this text:
/@@./d;
/^!./d;
/^[.]$/d;
s#http://##g;
s,[.?=&/|],\&,g;
s##.#g;
s,$.$,,g;

joukahainen

What would his command be if one is using Shallalist? (http://www.shallalist.de/Downloads/shallalist.tar.gz)

"command: fetch -o /var/db/squidGuard/adblock http://adblockplus.mozdev.org/easylist/easylist.txt ; cat /var/db/squidGuard/adblock/easylist.txt | sed -f /var/db/squidGuard/adblock/regex.sed > /var/db/squidGuard/adblock/expressions"

DigitalJer

I've been looking for something such as this, as well.

When I had a Linksys WRT as my firewall (have long since outgrown it), the script in the link below worked exceptionally well - so well, that whenever anyone asks me about adblocking, I recommend a WRT54GL / Tomato (or DD-WRT), and the script mentioned here (so far, out of about 2 dozen WRT's deployed, there has been only one bit of negative feedback - someone couldn't play her games on the MSN Gaming Zone):

http://www.linksysinfo.org/forums/showthread.php?t=53133

It cleans up the Interpipes very well, no matter what browser used. Of course, if FF with ABP is used too, it's even cleaner yet.

…if there is some way to adapt this to pfsense, I'd be ecstatic.

atmospheres

@bunny:

This doesn't seem to work atm with flash based ads and text based ads(google ads). It does not collapse most ads either. In other words, if you're using firefox then using the adblock plus plugin will work better than this even if they filter the same ads. The reason it doesn't work for everything is because it is not editing the html sent over the net. It is just blocking urls for anything that tries to pass through squid which is pictures. Anything that is dynamic like most flash ads or websites like, for example, hulu.com do not pass through squid. If anyone knows how to block flash ads, block the text type ads, and collapse blocked ads then please let me know!

Thanks for the tutorial!
Would using this setup along with Adblock solve the problem? Especially the collapse issue?