HOWTO: Transparently block all ads on home / work network

Mr. Jingles

Are you using SSL with your webGUI? If so, it won't work. That was the problem I had now that I recall. Not really an issue for me since my pfSense box is on my home network and my kids are too young to hack, for now :) Anyway, pfSense webGUI running SSL won't insert the 4x4.gif because the deny_info line is "redirecting" the ad to http://ip-address-of-pfsense, and there technically is no http://, only https://. I'm pretty sure I tried adding the https:// part in the deny_info line, but I remember it borking. I'm still in transit and all my gear is on a boat to Hawaii, so I'm working on pure memory here!

if you are running the webGUI on SSL, try switching it to regular HTTP, restarting squid and Ctrl+F5 on your client to force it to reload the page with the new config. Hopefully that will set you right!

Thank you very much for your reply, Maxamoto :P

So, after reading it, I thought: that's simple. And so I did that. And then I couldn't log in to the webgui anymore.

I restarted the webconfigurator from the CLI; nothing. I rebooted the box; nothing. No more access.

So I had no other choice but to do a complete reinstall of the box. Luckily I had made a config backup only last week. So I reinstalled Pfsense, and restored the config backup.

It went on and on installing and deinstalling and installing and deinstalling packages. Yes, it installed pfblocker, then deinstalled it, then reinstalled it again, and so on. I have no clue what it was thinking.

And when it finally was done after three hours? I still couldn't get into the GUI.

So I took another beer and reinstalled it again, after that manually installing the packages and doing all the configuration of all packages, the DHCP, the DNS, and so on. Which took this whole day.

I made a config backup, and finally installed Squid 2.7. I only set it up standard, so no block lists.

And I could visit every site on the internet, even pfsense.org. Except for one: forum.pfsense.org.

I got the Squid block page, even though I had not entered any restricted sites, of regular expressions, or whatever. Just SQUID transparent. Squidclient showed cache hits, I could go everywhere, except for this forum.

So I will first have to figure out what is wrong here, and then I can try if your solution might work. I will report back.

Stupid bug :'( :'(

Thank you & bye,

Mr. Jingles

Well, I rebooted the box and hoped for the best. It appears it is now running and I am allowed to visit this fine forum via Squid :-X

Yet, Maxamoto, the result is not what I expected when I visit newgrounds, per the attached screenshot.

I did notice that little 'alert generated by Opera', 'though. So I checked in firefox and internet explorer, and there it does not show that alert, but it leaves the space blank (second pic). But then you never know if there was an ad there, or something else 'important'.

Do you experience the same?

2013-11-11_134900.jpg_thumb

Internetexplorerandfirefox.jpg_thumb

Maxamoto

The second screenshot you posted is what I get in Linux Mint with FF or Chromium and on Windows running IE, FF or Chrome. I've never really used Opera so I don't know what it's doing there, or why. Seems kinda pointless.

Anyway, I would say you've achieved the result we were looking for with your second screenshot. If you're running into issues with pages / images being blocked that you don't want blocked just grep the blacklist and take it out :)

vincom

got this to work, yeh, my initial prob was dansguardian, i disabled it but i forgot to change the proxy interface to point to lan in the squid general page as it was set to loopback. its so simple once you figure out your mistakes, its nice and light and should not overwhelm and basic hw setup

Image3.jpg_thumb

SpankIt

Thanks for writing this how to. It worked great. I did go a step further and added some automation in order to update the definitions once a week using crontab and the following script I hacked together.


#!/bin/sh

PATH="/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/"

# Blacklist Removal
rm /var/squid/acl/blacklist.acl

# Blacklist Download
wget -O /var/squid/acl/blacklist.acl "http://pgl.yoyo.org/as/serverlist.php?hostformat=squid-dstdom-regex&showintro="

# Set Permissions
chown proxy:proxy /var/squid/acl/blacklist.acl

# Restarting Squid
killall -9 squid
squid

cplmayo

@SpankIt:

Thanks for writing this how to. It worked great. I did go a step further and added some automation in order to update the definitions once a week using crontab and the following script I hacked together.
#!/bin/sh

PATH="/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/"

# Blacklist Removal
rm /var/squid/acl/blacklist.acl

# Blacklist Download
wget -O /var/squid/acl/blacklist.acl "http://pgl.yoyo.org/as/serverlist.php?hostformat=squid-dstdom-regex&showintro="

# Set Permissions
chown proxy:proxy /var/squid/acl/blacklist.acl

# Restarting Squid
killall -9 squid
squid

I think I did the same thing using the cron package on pfsense. Is there anyway to double check this?

Reiner030

@Maxamoto:

if you are running the webGUI on SSL, try switching it to regular HTTP, restarting squid and Ctrl+F5 on your client to force it to reload the page with the new config. Hopefully that will set you right!

better:

deactivate https redirect rule in advanded section
optionally install package Filer if you have editable content
install package vHosts
create a vhost on port 80 for your static content which would be go to /usr/local/vhosts/
put with scp content in this folder
reference your content by url

so you have secured WebGUI for administration and "free" available content for your users

Exolon

@SpankIt:


# Blacklist Download
wget -O /var/squid/acl/blacklist.acl "http://pgl.yoyo.org/as/serverlist.php?hostformat=squid-dstdom-regex&showintro="

This is downloading an HTML file, if you look at the start and end, there are HTML tags, best to add the &mimetype=plaintext like this:


# Blacklist Download
wget -O /var/squid/acl/blacklist.acl "http://pgl.yoyo.org/as/serverlist.php?hostformat=squid-dstdom-regex&mimetype=plaintext&showintro="

Is there any need to kill Squid?

@SpankIt:


# Restarting Squid
killall -9 squid
squid

Could you not use the reconfigure parameter:


# Restarting Squid
squid -k reconfigure

angst

I can confirm this works with squid 2, just add the deny line in the custom config box, it works exactly the same as far as I can see as squid 3. Full Install v2.1. Thanks Maximoto, really clear, consise guide one of the best on this subject I have read.

nextear

Thanks so much for doing this! It works great! To automate the updates (per above)

Install package Cron
Install package Filer
In WebGui, select Diagnostics/Filer and create file "/usr/local/bin/update-blacklist", permissions of 744
In WebGui, Services/Cron create a new entry to run "/usr/local/bin/update-blacklist" periodically (I do once a week)

#!/bin/sh

PATH="/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/"

#mount as rw
/etc/rc.conf_mount_rw

# Blacklist Removal
rm /var/squid/acl/blacklist.acl

# Blacklist Download
fetch -qno /var/squid/acl/blacklist.acl "http://pgl.yoyo.org/as/serverlist.php?hostformat=squid-dstdom-regex&mimetype=plaintext&showintro="

# Set Permissions
chown proxy:proxy /var/squid/acl/blacklist.acl

#mount as ro
/etc/rc.conf_mount_ro

# Restarting Squid
squid -k reconfigure

Thanks again!

Escorpiom

Awesome, works on 2.2 beta!

So easy, yet effective.
Only one question:
I've got about 8 vlan's.
Do I have to put every vlan interface IP address in the "integration" and "acl" section?
Or is it possible to use a wildcard?

Cheers.

tong

@nextear:

Thanks so much for doing this! It works great! To automate the updates (per above)

Install package Cron

Install package Filer

In WebGui, select Diagnostics/Filer and create file "/usr/local/bin/update-blacklist", permissions of 744

In WebGui, Services/Cron create a new entry to run "/usr/local/bin/update-blacklist" periodically (I do once a week)
#!/bin/sh

PATH="/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/"

#mount as rw
/etc/rc.conf_mount_rw

# Blacklist Removal
rm /var/squid/acl/blacklist.acl

# Blacklist Download
fetch -qno /var/squid/acl/blacklist.acl "http://pgl.yoyo.org/as/serverlist.php?hostformat=squid-dstdom-regex&mimetype=plaintext&showintro="

# Set Permissions
chown proxy:proxy /var/squid/acl/blacklist.acl

#mount as ro
/etc/rc.conf_mount_ro

# Restarting Squid
squid -k reconfigure
Thanks again!

How do I add these:


S1="http://pgl.yoyo.org/as/serverlist.php?hostformat=nohtml"  #44K
S2="http://mirror1.malwaredomains.com/files/justdomains"    #189K
S3="http://www.malwaredomainlist.com/hostslist/hosts.txt"   #97K
S4="http://winhelp2002.mvps.org/hosts.txt"              #620K
S5="http://hosts-file.net/hphosts-partial.asp"              #460K
S6="http://hostsfile.mine.nu/Hosts"                         #2641K
S7="http://support.it-mate.co.uk/downloads/hosts.txt"       #3851K

to the update list so that it grabs all the sites, sifts through them and deletes doubles, then block all of them?

I used to use these when my router was a "dumber" device running tomato firmware, and this combined list is well into the 800,000 range of blocked add sites. I literally used to only see and add maybe once or twice a month, weather video or banner.

In case people are wondering these sites come from: http://www.linksysinfo.org/index.php?threads/all-u-need-ad-blocking.33191/
maybe it might be helpfull for some of the code too. Unfortunately I'm nowhere near skilled enough to figure this out myself.

kilobit

@tong:

@nextear:
Thanks so much for doing this! It works great! To automate the updates (per above)

Install package Cron

Install package Filer

In WebGui, select Diagnostics/Filer and create file "/usr/local/bin/update-blacklist", permissions of 744

In WebGui, Services/Cron create a new entry to run "/usr/local/bin/update-blacklist" periodically (I do once a week)
#!/bin/sh

PATH="/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/"

#mount as rw
/etc/rc.conf_mount_rw

# Blacklist Removal
rm /var/squid/acl/blacklist.acl

# Blacklist Download
fetch -qno /var/squid/acl/blacklist.acl "http://pgl.yoyo.org/as/serverlist.php?hostformat=squid-dstdom-regex&mimetype=plaintext&showintro="

# Set Permissions
chown proxy:proxy /var/squid/acl/blacklist.acl

#mount as ro
/etc/rc.conf_mount_ro

# Restarting Squid
squid -k reconfigure
Thanks again!
How do I add these:
S1="http://pgl.yoyo.org/as/serverlist.php?hostformat=nohtml"  #44K
S2="http://mirror1.malwaredomains.com/files/justdomains"    #189K
S3="http://www.malwaredomainlist.com/hostslist/hosts.txt"   #97K
S4="http://winhelp2002.mvps.org/hosts.txt"              #620K
S5="http://hosts-file.net/hphosts-partial.asp"              #460K
S6="http://hostsfile.mine.nu/Hosts"                         #2641K
S7="http://support.it-mate.co.uk/downloads/hosts.txt"       #3851K
to the update list so that it grabs all the sites, sifts through them and deletes doubles, then block all of them?

I used to use these when my router was a "dumber" device running tomato firmware, and this combined list is well into the 800,000 range of blocked add sites. I literally used to only see and add maybe once or twice a month, weather video or banner.

In case people are wondering these sites come from: http://www.linksysinfo.org/index.php?threads/all-u-need-ad-blocking.33191/
maybe it might be helpfull for some of the code too. Unfortunately I'm nowhere near skilled enough to figure this out myself.

Just wondering if you figured this out and if so if you wouldnt mind posting how. Thanks

KOM

This is an old thread. Lately, you can get the same results using a blacklist along with squidGuard.

johnpoz

Or you could just use blacklist in unbound.. That same yoyo.org link they listed has the list in unbound.. Just load the file into unbound and all the ad domains just get redirect to 127.0.0.1

Maxamoto

I wasn't aware that unbound or SquidGuard offered the ability to replace ads with a transparent gif. Redirecting to 127.0.0.1 will leave errors all over the page where the ads used to be; replacing with a transparent gif at least keeps it looking clean. Aesthetics :)

C0RR0SIVE

Squidguard has an option at General Settings, under Miscellaneous for "Clean Advertising" Works pretty good for me.

KOM

Yes, it's amazing what you find when you take the time to look.

Mr. Jingles

@johnpoz:

Or you could just use blacklist in unbound.. That same yoyo.org link they listed has the list in unbound.. Just load the file into unbound and all the ad domains just get redirect to 127.0.0.1

This is very interesting. How would one load the list in unbound with a script, similar to the way we can load it into Squid with the script?

It would appear it needs to be merged with /var/unbound/access_lists.conf, but how?