HOWTO: Transparently block all ads on home / work network

Maxamoto

I'm posting this here in case anyone else has a violent aversion to ads on their overpriced Internet service like I do. If it's in the wrong area, please feel free to move it.

So, the endstate is that ads are completely removed, with no trace that they ever existed in the first place. This saves on bandwidth and obviously, results in a much cleaner web experience. I've had several friends comment on my home Internet service being much faster and web pages are much "cleaner" than their own services in terms of web serving, and I can only assume that this is due to the fact that I take great pains in getting rid of as many ads as possible. Yes, I am fully aware that there are several notable and worthy causes out there to support and that by blocking ads I may potentially be denying someone of an income source, but in all honesty, I pay a lot for my Internet service and in the end, it's mine to control in whichever way I see fit once the packets enter my network. With that said, on to the insanely easy tutorial.

Step 1: Install Squid package

Step 2: Configure Squid however you see fit

Step 3: On the Squid General page, add the following to both the Integrations and Custom Options field:

deny_info http://192.168.10.1/4x4.gif blacklist

Note that the IP address should be the internal interface on your pfSense firewall, which will probably be different than what I have.

Step 4: Grab the 4x4.gif attached to this post

Step 5: Put 4x4.gif in /usr/local/www on pfSense server

Step 6: Head over to http://pgl.yoyo.org/adservers/ and generate a complete list using dstdom_regex list. I have also attached a text file of all of the ad servers compiled up to date, called ads.txt

Step 7: Copy and paste the entire list of domains into the Blacklist field on the ACL tab of the Squid server.

If you did Squid in transparent mode, you're pretty much done. If you didn't configure your clients manually and watch the ads disappear. Head over to http://www.newgrounds.com/ for a test. If you see lots of ads, recheck your work. I know I've skimmed over a few things in order to get to the meat of the matter, but I'm sure the target audience will know exactly how to accomplish this. The benefits are immediately apparent, assuming you're not in the ad serving business :)

One last thing: As ad pushers continue to evolve, so have I. If you notice an ad slipping through, it's a simple thing to add the new dstdom_regex to the Blocklist field and apply. I've also skimmed through page code to get the domains to manually add. If you have flash ads that just won't seem to go away, you might find yourself getting into the .fla or .swf to get the domain.

I hate ads.

ads.txt

Mr. Jingles

This is indeed an insanely simple how-to, and thank you for that ;D

However, something isn't working per the screenshot. I also note I do not have a 'integrations' to put the deny line in(?) I have Squid, not Squid3.

Would you happen to have any idea?

2013-11-11_134900.jpg_thumb

Mr. Jingles

While the *.gif is uploaded to /www and put in the custom options per this pic (and of course I restarted Squid).

2.jpg_thumb

Maxamoto

I'm running squid3, and honestly I don't remember what the fields for v2 were, but it should still work in the custom options field. It might be a permissions issue? I think I ran into that at one point.

Unfortunately, I'm in the middle of a move and all of my networking equipment is packed up. Hopefully I'll be back up and running here in a few weeks. If it hasn't been resolved by then I'll take a look and see what I did to fix it. I seem to remember doing this with v2 just fine, but I ran into the same issue as you. I think I might have had Blacklist instead of blacklist. Case sensitive :)

Mr. Jingles

@Maxamoto:

I'm running squid3, and honestly I don't remember what the fields for v2 were, but it should still work in the custom options field. It might be a permissions issue? I think I ran into that at one point.

Unfortunately, I'm in the middle of a move and all of my networking equipment is packed up. Hopefully I'll be back up and running here in a few weeks. If it hasn't been resolved by then I'll take a look and see what I did to fix it. I seem to remember doing this with v2 just fine, but I ran into the same issue as you. I think I might have had Blacklist instead of blacklist. Case sensitive :)

Thank you for your reply, Maxamoto; appreciated ;D

The capital in 'blacklist' made sense, as this is also as it is shown in the GUI, but unfortunately it didn't work (I did clear the browser cache). So perhaps the permission thing then. But would you happen to know what the permissions of /www would need to be then?

Thank you ;D

Mr. Jingles

Hmmm :-[

I was searching for a solution and found this:

https://workaround.org/squid-acls

[quote]Custom error pages (deny_info)

By default when you deny access the user gets the error page that is stored in the ERR_ACCESS_DENIED file. But luckily you can define your own custom error pages and display them when you deny certain accesses. A simple example:

acl google dstdomain google.com
deny_info error-google google
http_access deny google

Put an error page into the directory where the HTML files are stored (look for error_directory in your squid.conf) and name it error-google. If the user tries to access www.google.com the access is denied and your error page is shown.

In squid.conf it says the error directory is /usr/local/etc/squid/errors/English, so I uploaded the 4x4.gif there, but nothing. I also removed the 192.168.x.x. from the config line, so it only saying:

deny_info 4x4.gif blacklist

but still not working :P

Maxamoto

Are you using SSL with your webGUI? If so, it won't work. That was the problem I had now that I recall. Not really an issue for me since my pfSense box is on my home network and my kids are too young to hack, for now :) Anyway, pfSense webGUI running SSL won't insert the 4x4.gif because the deny_info line is "redirecting" the ad to http://ip-address-of-pfsense, and there technically is no http://, only https://. I'm pretty sure I tried adding the https:// part in the deny_info line, but I remember it borking. I'm still in transit and all my gear is on a boat to Hawaii, so I'm working on pure memory here!

if you are running the webGUI on SSL, try switching it to regular HTTP, restarting squid and Ctrl+F5 on your client to force it to reload the page with the new config. Hopefully that will set you right!

Mr. Jingles

@Maxamoto:

Are you using SSL with your webGUI? If so, it won't work. That was the problem I had now that I recall. Not really an issue for me since my pfSense box is on my home network and my kids are too young to hack, for now :) Anyway, pfSense webGUI running SSL won't insert the 4x4.gif because the deny_info line is "redirecting" the ad to http://ip-address-of-pfsense, and there technically is no http://, only https://. I'm pretty sure I tried adding the https:// part in the deny_info line, but I remember it borking. I'm still in transit and all my gear is on a boat to Hawaii, so I'm working on pure memory here!

if you are running the webGUI on SSL, try switching it to regular HTTP, restarting squid and Ctrl+F5 on your client to force it to reload the page with the new config. Hopefully that will set you right!

Thank you very much for your reply, Maxamoto :P

So, after reading it, I thought: that's simple. And so I did that. And then I couldn't log in to the webgui anymore.

I restarted the webconfigurator from the CLI; nothing. I rebooted the box; nothing. No more access.

So I had no other choice but to do a complete reinstall of the box. Luckily I had made a config backup only last week. So I reinstalled Pfsense, and restored the config backup.

It went on and on installing and deinstalling and installing and deinstalling packages. Yes, it installed pfblocker, then deinstalled it, then reinstalled it again, and so on. I have no clue what it was thinking.

And when it finally was done after three hours? I still couldn't get into the GUI.

So I took another beer and reinstalled it again, after that manually installing the packages and doing all the configuration of all packages, the DHCP, the DNS, and so on. Which took this whole day.

I made a config backup, and finally installed Squid 2.7. I only set it up standard, so no block lists.

And I could visit every site on the internet, even pfsense.org. Except for one: forum.pfsense.org.

I got the Squid block page, even though I had not entered any restricted sites, of regular expressions, or whatever. Just SQUID transparent. Squidclient showed cache hits, I could go everywhere, except for this forum.

So I will first have to figure out what is wrong here, and then I can try if your solution might work. I will report back.

Stupid bug :'( :'(

Thank you & bye,

Mr. Jingles

Well, I rebooted the box and hoped for the best. It appears it is now running and I am allowed to visit this fine forum via Squid :-X

Yet, Maxamoto, the result is not what I expected when I visit newgrounds, per the attached screenshot.

I did notice that little 'alert generated by Opera', 'though. So I checked in firefox and internet explorer, and there it does not show that alert, but it leaves the space blank (second pic). But then you never know if there was an ad there, or something else 'important'.

Do you experience the same?

2013-11-11_134900.jpg_thumb

Internetexplorerandfirefox.jpg_thumb

Maxamoto

The second screenshot you posted is what I get in Linux Mint with FF or Chromium and on Windows running IE, FF or Chrome. I've never really used Opera so I don't know what it's doing there, or why. Seems kinda pointless.

Anyway, I would say you've achieved the result we were looking for with your second screenshot. If you're running into issues with pages / images being blocked that you don't want blocked just grep the blacklist and take it out :)

vincom

got this to work, yeh, my initial prob was dansguardian, i disabled it but i forgot to change the proxy interface to point to lan in the squid general page as it was set to loopback. its so simple once you figure out your mistakes, its nice and light and should not overwhelm and basic hw setup

Image3.jpg_thumb

SpankIt

Thanks for writing this how to. It worked great. I did go a step further and added some automation in order to update the definitions once a week using crontab and the following script I hacked together.


#!/bin/sh

PATH="/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/"

# Blacklist Removal
rm /var/squid/acl/blacklist.acl

# Blacklist Download
wget -O /var/squid/acl/blacklist.acl "http://pgl.yoyo.org/as/serverlist.php?hostformat=squid-dstdom-regex&showintro="

# Set Permissions
chown proxy:proxy /var/squid/acl/blacklist.acl

# Restarting Squid
killall -9 squid
squid

cplmayo

@SpankIt:

Thanks for writing this how to. It worked great. I did go a step further and added some automation in order to update the definitions once a week using crontab and the following script I hacked together.
#!/bin/sh

PATH="/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/"

# Blacklist Removal
rm /var/squid/acl/blacklist.acl

# Blacklist Download
wget -O /var/squid/acl/blacklist.acl "http://pgl.yoyo.org/as/serverlist.php?hostformat=squid-dstdom-regex&showintro="

# Set Permissions
chown proxy:proxy /var/squid/acl/blacklist.acl

# Restarting Squid
killall -9 squid
squid

I think I did the same thing using the cron package on pfsense. Is there anyway to double check this?

Reiner030

@Maxamoto:

if you are running the webGUI on SSL, try switching it to regular HTTP, restarting squid and Ctrl+F5 on your client to force it to reload the page with the new config. Hopefully that will set you right!

better:

deactivate https redirect rule in advanded section
optionally install package Filer if you have editable content
install package vHosts
create a vhost on port 80 for your static content which would be go to /usr/local/vhosts/
put with scp content in this folder
reference your content by url

so you have secured WebGUI for administration and "free" available content for your users

Exolon

@SpankIt:


# Blacklist Download
wget -O /var/squid/acl/blacklist.acl "http://pgl.yoyo.org/as/serverlist.php?hostformat=squid-dstdom-regex&showintro="

This is downloading an HTML file, if you look at the start and end, there are HTML tags, best to add the &mimetype=plaintext like this:


# Blacklist Download
wget -O /var/squid/acl/blacklist.acl "http://pgl.yoyo.org/as/serverlist.php?hostformat=squid-dstdom-regex&mimetype=plaintext&showintro="

Is there any need to kill Squid?

@SpankIt:


# Restarting Squid
killall -9 squid
squid

Could you not use the reconfigure parameter:


# Restarting Squid
squid -k reconfigure

angst

I can confirm this works with squid 2, just add the deny line in the custom config box, it works exactly the same as far as I can see as squid 3. Full Install v2.1. Thanks Maximoto, really clear, consise guide one of the best on this subject I have read.

nextear

Thanks so much for doing this! It works great! To automate the updates (per above)

Install package Cron
Install package Filer
In WebGui, select Diagnostics/Filer and create file "/usr/local/bin/update-blacklist", permissions of 744
In WebGui, Services/Cron create a new entry to run "/usr/local/bin/update-blacklist" periodically (I do once a week)

#!/bin/sh

PATH="/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/"

#mount as rw
/etc/rc.conf_mount_rw

# Blacklist Removal
rm /var/squid/acl/blacklist.acl

# Blacklist Download
fetch -qno /var/squid/acl/blacklist.acl "http://pgl.yoyo.org/as/serverlist.php?hostformat=squid-dstdom-regex&mimetype=plaintext&showintro="

# Set Permissions
chown proxy:proxy /var/squid/acl/blacklist.acl

#mount as ro
/etc/rc.conf_mount_ro

# Restarting Squid
squid -k reconfigure

Thanks again!

Escorpiom

Awesome, works on 2.2 beta!

So easy, yet effective.
Only one question:
I've got about 8 vlan's.
Do I have to put every vlan interface IP address in the "integration" and "acl" section?
Or is it possible to use a wildcard?

Cheers.

tong

@nextear:

Thanks so much for doing this! It works great! To automate the updates (per above)

Install package Cron

Install package Filer

In WebGui, select Diagnostics/Filer and create file "/usr/local/bin/update-blacklist", permissions of 744

In WebGui, Services/Cron create a new entry to run "/usr/local/bin/update-blacklist" periodically (I do once a week)
#!/bin/sh

PATH="/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/"

#mount as rw
/etc/rc.conf_mount_rw

# Blacklist Removal
rm /var/squid/acl/blacklist.acl

# Blacklist Download
fetch -qno /var/squid/acl/blacklist.acl "http://pgl.yoyo.org/as/serverlist.php?hostformat=squid-dstdom-regex&mimetype=plaintext&showintro="

# Set Permissions
chown proxy:proxy /var/squid/acl/blacklist.acl

#mount as ro
/etc/rc.conf_mount_ro

# Restarting Squid
squid -k reconfigure
Thanks again!

How do I add these:


S1="http://pgl.yoyo.org/as/serverlist.php?hostformat=nohtml"  #44K
S2="http://mirror1.malwaredomains.com/files/justdomains"    #189K
S3="http://www.malwaredomainlist.com/hostslist/hosts.txt"   #97K
S4="http://winhelp2002.mvps.org/hosts.txt"              #620K
S5="http://hosts-file.net/hphosts-partial.asp"              #460K
S6="http://hostsfile.mine.nu/Hosts"                         #2641K
S7="http://support.it-mate.co.uk/downloads/hosts.txt"       #3851K

to the update list so that it grabs all the sites, sifts through them and deletes doubles, then block all of them?

I used to use these when my router was a "dumber" device running tomato firmware, and this combined list is well into the 800,000 range of blocked add sites. I literally used to only see and add maybe once or twice a month, weather video or banner.

In case people are wondering these sites come from: http://www.linksysinfo.org/index.php?threads/all-u-need-ad-blocking.33191/
maybe it might be helpfull for some of the code too. Unfortunately I'm nowhere near skilled enough to figure this out myself.

kilobit

@tong:

@nextear:
Thanks so much for doing this! It works great! To automate the updates (per above)

Install package Cron

Install package Filer

In WebGui, select Diagnostics/Filer and create file "/usr/local/bin/update-blacklist", permissions of 744

In WebGui, Services/Cron create a new entry to run "/usr/local/bin/update-blacklist" periodically (I do once a week)
#!/bin/sh

PATH="/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/"

#mount as rw
/etc/rc.conf_mount_rw

# Blacklist Removal
rm /var/squid/acl/blacklist.acl

# Blacklist Download
fetch -qno /var/squid/acl/blacklist.acl "http://pgl.yoyo.org/as/serverlist.php?hostformat=squid-dstdom-regex&mimetype=plaintext&showintro="

# Set Permissions
chown proxy:proxy /var/squid/acl/blacklist.acl

#mount as ro
/etc/rc.conf_mount_ro

# Restarting Squid
squid -k reconfigure
Thanks again!
How do I add these:
S1="http://pgl.yoyo.org/as/serverlist.php?hostformat=nohtml"  #44K
S2="http://mirror1.malwaredomains.com/files/justdomains"    #189K
S3="http://www.malwaredomainlist.com/hostslist/hosts.txt"   #97K
S4="http://winhelp2002.mvps.org/hosts.txt"              #620K
S5="http://hosts-file.net/hphosts-partial.asp"              #460K
S6="http://hostsfile.mine.nu/Hosts"                         #2641K
S7="http://support.it-mate.co.uk/downloads/hosts.txt"       #3851K
to the update list so that it grabs all the sites, sifts through them and deletes doubles, then block all of them?

I used to use these when my router was a "dumber" device running tomato firmware, and this combined list is well into the 800,000 range of blocked add sites. I literally used to only see and add maybe once or twice a month, weather video or banner.

In case people are wondering these sites come from: http://www.linksysinfo.org/index.php?threads/all-u-need-ad-blocking.33191/
maybe it might be helpfull for some of the code too. Unfortunately I'm nowhere near skilled enough to figure this out myself.

Just wondering if you figured this out and if so if you wouldnt mind posting how. Thanks