HOWTO: Transparently block all ads on home / work network

cplmayo

Thanks for writing this how to. It worked great. I did go a step further and added some automation in order to update the definitions once a week using crontab and the following script I hacked together.
#!/bin/sh

PATH="/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/"

# Blacklist Removal
rm /var/squid/acl/blacklist.acl

# Blacklist Download
wget -O /var/squid/acl/blacklist.acl "http://pgl.yoyo.org/as/serverlist.php?hostformat=squid-dstdom-regex&showintro="

# Set Permissions
chown proxy:proxy /var/squid/acl/blacklist.acl

# Restarting Squid
killall -9 squid
squid

I think I did the same thing using the cron package on pfsense. Is there anyway to double check this?

Reiner030

@Maxamoto:

if you are running the webGUI on SSL, try switching it to regular HTTP, restarting squid and Ctrl+F5 on your client to force it to reload the page with the new config. Hopefully that will set you right!

better:

deactivate https redirect rule in advanded section
optionally install package Filer if you have editable content
install package vHosts
create a vhost on port 80 for your static content which would be go to /usr/local/vhosts/
put with scp content in this folder
reference your content by url

so you have secured WebGUI for administration and "free" available content for your users

Exolon

@SpankIt:


# Blacklist Download
wget -O /var/squid/acl/blacklist.acl "http://pgl.yoyo.org/as/serverlist.php?hostformat=squid-dstdom-regex&showintro="

This is downloading an HTML file, if you look at the start and end, there are HTML tags, best to add the &mimetype=plaintext like this:


# Blacklist Download
wget -O /var/squid/acl/blacklist.acl "http://pgl.yoyo.org/as/serverlist.php?hostformat=squid-dstdom-regex&mimetype=plaintext&showintro="

Is there any need to kill Squid?

@SpankIt:


# Restarting Squid
killall -9 squid
squid

Could you not use the reconfigure parameter:


# Restarting Squid
squid -k reconfigure

angst

I can confirm this works with squid 2, just add the deny line in the custom config box, it works exactly the same as far as I can see as squid 3. Full Install v2.1. Thanks Maximoto, really clear, consise guide one of the best on this subject I have read.

nextear

Thanks so much for doing this! It works great! To automate the updates (per above)

Install package Cron
Install package Filer
In WebGui, select Diagnostics/Filer and create file "/usr/local/bin/update-blacklist", permissions of 744
In WebGui, Services/Cron create a new entry to run "/usr/local/bin/update-blacklist" periodically (I do once a week)

#!/bin/sh

PATH="/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/"

#mount as rw
/etc/rc.conf_mount_rw

# Blacklist Removal
rm /var/squid/acl/blacklist.acl

# Blacklist Download
fetch -qno /var/squid/acl/blacklist.acl "http://pgl.yoyo.org/as/serverlist.php?hostformat=squid-dstdom-regex&mimetype=plaintext&showintro="

# Set Permissions
chown proxy:proxy /var/squid/acl/blacklist.acl

#mount as ro
/etc/rc.conf_mount_ro

# Restarting Squid
squid -k reconfigure

Thanks again!

Escorpiom

Awesome, works on 2.2 beta!

So easy, yet effective.
Only one question:
I've got about 8 vlan's.
Do I have to put every vlan interface IP address in the "integration" and "acl" section?
Or is it possible to use a wildcard?

Cheers.

tong

@nextear:

Thanks so much for doing this! It works great! To automate the updates (per above)

Install package Cron

Install package Filer

In WebGui, select Diagnostics/Filer and create file "/usr/local/bin/update-blacklist", permissions of 744

In WebGui, Services/Cron create a new entry to run "/usr/local/bin/update-blacklist" periodically (I do once a week)
#!/bin/sh

PATH="/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/"

#mount as rw
/etc/rc.conf_mount_rw

# Blacklist Removal
rm /var/squid/acl/blacklist.acl

# Blacklist Download
fetch -qno /var/squid/acl/blacklist.acl "http://pgl.yoyo.org/as/serverlist.php?hostformat=squid-dstdom-regex&mimetype=plaintext&showintro="

# Set Permissions
chown proxy:proxy /var/squid/acl/blacklist.acl

#mount as ro
/etc/rc.conf_mount_ro

# Restarting Squid
squid -k reconfigure
Thanks again!

How do I add these:


S1="http://pgl.yoyo.org/as/serverlist.php?hostformat=nohtml"  #44K
S2="http://mirror1.malwaredomains.com/files/justdomains"    #189K
S3="http://www.malwaredomainlist.com/hostslist/hosts.txt"   #97K
S4="http://winhelp2002.mvps.org/hosts.txt"              #620K
S5="http://hosts-file.net/hphosts-partial.asp"              #460K
S6="http://hostsfile.mine.nu/Hosts"                         #2641K
S7="http://support.it-mate.co.uk/downloads/hosts.txt"       #3851K

to the update list so that it grabs all the sites, sifts through them and deletes doubles, then block all of them?

I used to use these when my router was a "dumber" device running tomato firmware, and this combined list is well into the 800,000 range of blocked add sites. I literally used to only see and add maybe once or twice a month, weather video or banner.

In case people are wondering these sites come from: http://www.linksysinfo.org/index.php?threads/all-u-need-ad-blocking.33191/
maybe it might be helpfull for some of the code too. Unfortunately I'm nowhere near skilled enough to figure this out myself.

kilobit

@tong:

@nextear:
Thanks so much for doing this! It works great! To automate the updates (per above)

Install package Cron

Install package Filer

In WebGui, select Diagnostics/Filer and create file "/usr/local/bin/update-blacklist", permissions of 744

In WebGui, Services/Cron create a new entry to run "/usr/local/bin/update-blacklist" periodically (I do once a week)
#!/bin/sh

PATH="/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/"

#mount as rw
/etc/rc.conf_mount_rw

# Blacklist Removal
rm /var/squid/acl/blacklist.acl

# Blacklist Download
fetch -qno /var/squid/acl/blacklist.acl "http://pgl.yoyo.org/as/serverlist.php?hostformat=squid-dstdom-regex&mimetype=plaintext&showintro="

# Set Permissions
chown proxy:proxy /var/squid/acl/blacklist.acl

#mount as ro
/etc/rc.conf_mount_ro

# Restarting Squid
squid -k reconfigure
Thanks again!
How do I add these:
S1="http://pgl.yoyo.org/as/serverlist.php?hostformat=nohtml"  #44K
S2="http://mirror1.malwaredomains.com/files/justdomains"    #189K
S3="http://www.malwaredomainlist.com/hostslist/hosts.txt"   #97K
S4="http://winhelp2002.mvps.org/hosts.txt"              #620K
S5="http://hosts-file.net/hphosts-partial.asp"              #460K
S6="http://hostsfile.mine.nu/Hosts"                         #2641K
S7="http://support.it-mate.co.uk/downloads/hosts.txt"       #3851K
to the update list so that it grabs all the sites, sifts through them and deletes doubles, then block all of them?

I used to use these when my router was a "dumber" device running tomato firmware, and this combined list is well into the 800,000 range of blocked add sites. I literally used to only see and add maybe once or twice a month, weather video or banner.

In case people are wondering these sites come from: http://www.linksysinfo.org/index.php?threads/all-u-need-ad-blocking.33191/
maybe it might be helpfull for some of the code too. Unfortunately I'm nowhere near skilled enough to figure this out myself.

Just wondering if you figured this out and if so if you wouldnt mind posting how. Thanks

KOM

This is an old thread. Lately, you can get the same results using a blacklist along with squidGuard.

johnpoz

Or you could just use blacklist in unbound.. That same yoyo.org link they listed has the list in unbound.. Just load the file into unbound and all the ad domains just get redirect to 127.0.0.1

Maxamoto

I wasn't aware that unbound or SquidGuard offered the ability to replace ads with a transparent gif. Redirecting to 127.0.0.1 will leave errors all over the page where the ads used to be; replacing with a transparent gif at least keeps it looking clean. Aesthetics :)

C0RR0SIVE

Squidguard has an option at General Settings, under Miscellaneous for "Clean Advertising" Works pretty good for me.

KOM

Yes, it's amazing what you find when you take the time to look.

Mr. Jingles

@johnpoz:

Or you could just use blacklist in unbound.. That same yoyo.org link they listed has the list in unbound.. Just load the file into unbound and all the ad domains just get redirect to 127.0.0.1

This is very interesting. How would one load the list in unbound with a script, similar to the way we can load it into Squid with the script?

It would appear it needs to be merged with /var/unbound/access_lists.conf, but how?