Ad blocking with pfsense

fraglord

Yeah, you might shout out: squid / squidguard. That is indeed a good (the best?) solution but unfortunately not an option because i have a multi-WAN setup and squidk keeps using the default gateway for all connections and ignores the firewall rules to set specifis gateways.
pfBlockerNG 2.0 supposed to incorporate a functionality to block ads but the release date is set to "when it's done".
So what other options do I have to do adblocking on my pfsense box?

Cmellons

I believe that there are many ways to go about this. One of my favorites which apparently does not work anymore is HAVP because you could easily add an entire domain without having to worry about bs characters in the front where they constantly randomize. Another option that might work well is Dan's Guardian.

In quite a few cases Suricata has blocked some ads for me but in a lot of those cases the site would become nearly unusable because Suricata does not play.

I'm thinking that there is a way to do it with Squid as well if you are able to add a domain like this for instance if I didn't want google anymore, HAVP would let me type something like this ( .google.com/)

It's been so long though so I really don't remember the proper syntax but you can look it up or just install HAVP to have a look at it and get some ideas even if it doesn't work you can learn about some things.

Pfblocker is another option and just remember that you could get an entire ad list of ip addresses, but it may block too much so you really need to get in there and right click on the ads to find out where they are coming from and deal with the domains.

Who wants to do a thousand IP addresses when you can just kill it with the domain.

The other option is, if it's that bad I would just go ahead and use Adblock plus. I don't use it because it slows down my computer.

Also take a look at some of the threads that are already created if you can find them. There are many great ideas about this and more than likely you will find a method that is the most efficient for you just by using the search function at the top right. Good luck.

Nullity

If pfSense is your DNS, you could employ a modified /etc/hosts file, right? That is the simplest method I know. I use this method.

My only complaint is that I am unaware of a quick way to unblock a site, but this is only a problem when using some shady file-sharing website.

doktornotor

Or PM BBcan17 about pfBlockerNG-Dev testing.

firewalluser

@Cmellons:

The other option is, if it's that bad I would just go ahead and use Adblock plus. I don't use it because it slows down my computer.

NoScript in firefox is very fast and makes loading of websites even faster as I find the CDN's take for ever getting their act together.

You still need to tweak the settings for the first time when you download this addon though, eg in the options settings
General Tab,
Unticked - Automatically reload affected pages when permissions change
Ticked - Reload the current tab only.
If you have multiple tabs open for the same site but want to allow one or more domains hilighted on tab, stops the other tabs open on the same main domain from reloading so it saves you bandwith but also reduces the instance of a pattern forming which can show you are using NoScript in default mode.

Clear the whitelist down and build it up from scratch, it doesnt take much to whitelist your favourite websites.

The rest of the options/tabs can be left alone iirc, but using NoScript certainly speeds up the loading of webpages, gives you more privacy from online trackers and reduces your risk of getting a virus as I have seen plenty of new viruses delivered through adverts coming from Content Delivery Networks over the years. On the virus point, these CDN's still use the same popular AV tools you and I can use, so a virus writer just needs to come up with something new to bypass the existing checks et voila you can infect millions of people in the blink of an eye so to speak, hence the need for things like NoScript is valid.

fraglord

Unfortunately it is not that easy: HAVP does rely on squid as well which is, currently, considered to be broken for multi-wan environments as it keeps using the default gateway, no matter what firewall rules you configure.
A browser extension is nice - where it is available. On most mobile devices (Windows Phone & RT, Apple) it is not. Also there are systemwide ads within the apps etc.
I'll contact BBcan17 ;)

firewalluser

@fraglord:

Unfortunately it is not that easy: HAVP does rely on squid as well which is, currently, considered to be broken for multi-wan environments as it keeps using the default gateway, no matter what firewall rules you configure.
A browser extension is nice - where it is available. On most mobile devices (Windows Phone & RT, Apple) it is not. Also there are systemwide ads within the apps etc.
I'll contact BBcan17 ;)

Perhaps building a list of domains which are known trackers or sites you dont want interact with would be a start for blocking purposes, and then for mobile devices or where a browser extension doesnt exist, having the browser tunnel back to pfsense via a vpn may be a suitable workaround although not perfect.

I say not perfect because I dont allow youtube for most things or google for that matter so default is its blocked, but sometimes I want to watch something on youtube or google from another site so I enable it in those instances.

I guess something which would be closer to ideal would be to have a list of domain associated with themselves, eg take the TheGuardian.com you also have guardianapis.com, guardianapps.co.uk, & guim.co.uk which all need to be enabled before you can read and interact with the main TheGuardian.com domain. So having a system/app on pfsense which can recognise you are visiting the main domain and thus allow the other domains to work would be useful.

Edit.
Whilst there are these links which suggest ways to block by domain name, these will only be worthwhile for blocking trackers imo.

http://samkear.com/pfsense/how-to-configure-a-dns-blacklist-using-pfsense
https://ejnetwork.wordpress.com/2014/08/04/blocking-domains-with-pfsense-using-dns-forwarder/
https://doc.pfsense.org/index.php/Blocking_websites
https://forum.pfsense.org/index.php?topic=19434.0
http://hubpages.com/hub/URL-Filtering-How-To-Configure-SquidGuard-in-pfSense

YMMV

Wolf666

@doktornotor:

Or PM BBcan17 about pfBlockerNG-Dev testing.

Or wait for the release…...so far so good... ;D

syntactic.net

DEAR PEOPLE FROM THE FUTURE: Here's what we've figured out so far …

Sorry to bump an old thread, but while I've seen a number of ideas on this topic, I don't think that this ever got a satisfactory response.. Waiting for the next pfBlockerNG doesn't seem to be the right choice - several months after that suggestion, it's still not out. Using adblock in firefox is all very well for desktops, but ad blocking on mobile devices is sometimes itself blocked by the OS, and on devices like TVs and Roku or similar, you have no real control over the host.

If your ad blocking needs are mostly on HTTP then this solution might work for you: https://forum.pfsense.org/index.php?topic=94222.0. However, once advertisers start using HTTPS, using squid as a transparent proxy becomes noticeably less simple. In any case, for the vast majority of ads out there, DNS-based blocking does the trick - you'll still often lose screen real-estate to ad elements, but you won't see the actual ads.

The same site suggested in the thread referenced above has a dnsmasq-compatible list that you can get in plaintext form:
http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&mimetype=plaintext
There may be other sites that offer blacklists in dnsmasq form, or other options from this site that you want to use - YMMV.

To make this work, you'll need to tell dnsmasq to read additional configuration files from a config directory, with for example "conf_dir=/etc/dnsmasq" in the advanced config options. SSH into the machine, create that directory, and then download the file to it. Restart dnsmasq and you're good to go.

To automate the process for periodic updates you'll need to add a cron job, and you'll need either wget or curl installed on the pfsense box. SSH in, go to shell, run "pkg install curl" or wget, and eventually you'll have a working version. If that doesn't work, run "pkg" without any options first. Then your cron job can run one of these commands:

curl http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&mimetype=plaintext > /etc/dnsmasq/dnsmasq.blocklist
wget -O /etc/dnsmasq/dnsmasq.blocklist http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&mimetype=plaintext

Now, to restart dnsmasq you'll need the following file: /etc/phpshellsessions/restartdnsmasq

services_dnsmasq_configure();

Then run "pfSsh.php playback restartdnsmasq" within your cron script to restart the dnsmasq server.

alex_ncus

OK, here is what I am using (seems to be working for me).

Using dnsmasq guide @ http://thomasloughlin.com/pfsense-dnsmasq-advanced-setup

Created a new script (blackhole.sh) in /etc directory and then used CRON to update once a day.

Here is the script that I am using:

#!/bin/sh
## blackhole.sh
## Adapted for pfSense from Tomato WAN Up script v3.3 by haarp

TMPFILE="/tmp/dnsmasq.work"                          ## dnsmasq temporary file
GENFILE="/usr/local/etc/dnsmasq.d/dnsmasq.custom"    ## dnsmasq custom config

SOURCES=""
SOURCES="$SOURCES http://winhelp2002.mvps.org/hosts.txt"
SOURCES="$SOURCES http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext"
#SOURCES="$SOURCES http://hosts-file.net/ad_servers.asp"
SOURCES="$SOURCES http://hosts-file.net/.%5Cad_servers.txt"
##SOURCES="$SOURCES http://hostsfile.mine.nu/Hosts"
##SOURCES="$SOURCES http://sysctl.org/cameleon/hosts"
SOURCES="$SOURCES http://adaway.org/hosts.txt"
##SOURCES="$SOURCES http://hosts-file.net/download/hosts.txt"
#SOURCES="$SOURCES http://hosts-file.net/hphosts-partial.asp"
SOURCES="$SOURCES http://www.malwaredomainlist.com/hostslist/hosts.txt"

## Blacklist additional sites (add inside quotes, space-separated)
BLACKLIST="google-analytics.com"

## Whitelist sites from blocking (add inside quotes, space-separated)
WHITELIST=""

echo "Download starting"
until ping -q -c1 google.com >/dev/null; do
    echo "Waiting for internet"
    sleep 5
done

echo -n "" > $TMPFILE
for s in $SOURCES; do
    { (wget $s -O - || elog "Failed: $s") | \
      tr -d "\r" | \
      sed -e '/^[[:alnum:]:]/!d' | \
      awk '{print $2}' | \
      sed -e '/^localhost$/d' >> $TMPFILE
    } &
done

wait

if [ -s $TMPFILE ]; then
   echo "Download finished"
else
   echo "Failed: Download unsuccessful, aborting"
   rm $TMPFILE
   exit 1                                           
fi

echo "Generating $TMPFILE"
for b in $BLACKLIST; do
    echo "$b" >> $TMPFILE
done

for w in $WHITELIST; do
    sed -i -e "/$w/d" $TMPFILE
done

sort -u $TMPFILE -o $TMPFILE                                           ## Sort and remove duplicates
awk '{print "address=/"$0"/127.0.0.1/"}' $TMPFILE > $GENFILE           ## format file for dnsmasq ... address=/domain-name/127.0.0.1

echo "Config generated, $(wc -l < $GENFILE) unique hosts to block"
echo "Restarting dnsmasq"
service dnsmasq restart

echo "Deleting $TMPFILE to free memory"
rm $TMPFILE

f34rinc

If you take a look at the changes on github you'll see why it took so long for 2.0 to come out.

I encourage others to try Ad blocking\filtering with pfBlockerNG 2.0+ with its DNSBL using the unbound DNS resolver. If a domain is blocked I can use the alerts tab to suppress this domain from being blocked from future updates or manually enter the domain in the Custom Domain Suppression text box on the main DNSBL tab.

Below is some of the lists I am currently using myself

Ad Blocking List
http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext
http://hosts-file.net/download/hosts.zip
http://hosts-file.net/hphosts-partial.asp
http://hosts-file.net/.%5Cad_servers.txt
http://adaway.org/hosts.txt
http://someonewhocares.org/hosts/hosts
http://winhelp2002.mvps.org/hosts.txt
http://adblock.gjtech.net/?format=unix-hosts

Malware \ Phishing 
http://mirror1.malwaredomains.com/files/justdomains
http://www.malwaredomainlist.com/hostslist/hosts.txt
http://osint.bambenekconsulting.com/feeds/dga-feed.gz
http://data.phishtank.com/data/online-valid.csv.gz
https://dshield.org/feeds/suspiciousdomains_Medium.txt
https://dshield.org/feeds/suspiciousdomains_High.txt
https://www.openphish.com/feed.txt

alex_ncus

NICE!

One thing you might want to consider when deduplicating using domain hierarchy. For example, if blacklist consists of:

…
xyz.com
w1-1.xyz.com
w1-2.xyz.com
www.xyz.com
...

they could all be reduced to a single xyz.com entry (instead of 4) ...

this could have a dramatic impact of (substantially) reducing table size (and improving search performance). Perhaps down by 2/3 for large blacklists.

awk -F "." '{for(i=NF; i > 1; i--) printf "%s.", $i; print $1}' $TMPFILE | sort -u | awk -F "." '{for(i=NF; i > 1; i--) printf "%s.", $i; print $1}' | awk 'BEGIN {last = "foo"} /^$/ {next} $0 !~ last { print "address=/"$0"/127.0.0.1"; last = $0 }' > $GENFILE

I tried this in earlier blackhole example with dnsmasq and reduced 369799 unique entries down to 117386.

f34rinc

Stripping the URL down to the domain is going to cause a lot of false positives.

ads.xyz.com
turns into
xyz.com

what if I try to block ads.yahoo.com and end up blocking all of yahoo.com

Nullity

@f34rinc:

Stripping the URL down to the domain is going to cause a lot of false positives.

ads.xyz.com
turns into
xyz.com

what if I try to block ads.yahoo.com and end up blocking all of yahoo.com

I did not look through his script, but if it is accurate to his explanation, it should only consolidate when it will make no difference.

so, if
a.xyz.co
b.xyz.co
xyz.co
would consolidate to simply xyz.co, but not if only a.xyz.co and b.xyz.co were listed.

Honestly, I would expect pfblockerng, or any blocklist aggregator, to do that type of optimization.

bluepr0

EDIT: never mind, I got the pfblocker list working with DNSBL. One thing I'm noticing is that for some websites it takes a lot of time to resolve the DNS, sometimes it gets inside the web and sometimes it shows connection time out on the browser. Would be great to know a bit more in detail the overall configuration so pfblocker works fine! (for example, do I have to stop some other service?)

BBcan177

bluepr0,

Make sure that all of the LAN devices DNS settings, are pointing only to the DNS Resolver. You shouldn't be getting those timeout messages. Are you on a Multi-Segmented LAN setup? If yes, then in the DNBSL tab, there is a checkbox to auto-create a floating firewall rule to allow the traffic from those other LAN subnets to access the DNSBL VIP… You should be able to ping and browse to the DNSBL VIP (1x1 pixel).

bluepr0

Fixed it! Now it's working nicely!

I used to have a VM with pi-hole.net but if I can have ad filter directly on the router, much better

Now I will have to read more about easylist, so I can add Adblock lists!