Ad blocking with pfsense

fraglord

Unfortunately it is not that easy: HAVP does rely on squid as well which is, currently, considered to be broken for multi-wan environments as it keeps using the default gateway, no matter what firewall rules you configure.
A browser extension is nice - where it is available. On most mobile devices (Windows Phone & RT, Apple) it is not. Also there are systemwide ads within the apps etc.
I'll contact BBcan17 ;)

firewalluser

@fraglord:

Unfortunately it is not that easy: HAVP does rely on squid as well which is, currently, considered to be broken for multi-wan environments as it keeps using the default gateway, no matter what firewall rules you configure.
A browser extension is nice - where it is available. On most mobile devices (Windows Phone & RT, Apple) it is not. Also there are systemwide ads within the apps etc.
I'll contact BBcan17 ;)

Perhaps building a list of domains which are known trackers or sites you dont want interact with would be a start for blocking purposes, and then for mobile devices or where a browser extension doesnt exist, having the browser tunnel back to pfsense via a vpn may be a suitable workaround although not perfect.

I say not perfect because I dont allow youtube for most things or google for that matter so default is its blocked, but sometimes I want to watch something on youtube or google from another site so I enable it in those instances.

I guess something which would be closer to ideal would be to have a list of domain associated with themselves, eg take the TheGuardian.com you also have guardianapis.com, guardianapps.co.uk, & guim.co.uk which all need to be enabled before you can read and interact with the main TheGuardian.com domain. So having a system/app on pfsense which can recognise you are visiting the main domain and thus allow the other domains to work would be useful.

Edit.
Whilst there are these links which suggest ways to block by domain name, these will only be worthwhile for blocking trackers imo.

http://samkear.com/pfsense/how-to-configure-a-dns-blacklist-using-pfsense
https://ejnetwork.wordpress.com/2014/08/04/blocking-domains-with-pfsense-using-dns-forwarder/
https://doc.pfsense.org/index.php/Blocking_websites
https://forum.pfsense.org/index.php?topic=19434.0
http://hubpages.com/hub/URL-Filtering-How-To-Configure-SquidGuard-in-pfSense

YMMV

Wolf666

@doktornotor:

Or PM BBcan17 about pfBlockerNG-Dev testing.

Or wait for the release…...so far so good... ;D

syntactic.net

DEAR PEOPLE FROM THE FUTURE: Here's what we've figured out so far …

Sorry to bump an old thread, but while I've seen a number of ideas on this topic, I don't think that this ever got a satisfactory response.. Waiting for the next pfBlockerNG doesn't seem to be the right choice - several months after that suggestion, it's still not out. Using adblock in firefox is all very well for desktops, but ad blocking on mobile devices is sometimes itself blocked by the OS, and on devices like TVs and Roku or similar, you have no real control over the host.

If your ad blocking needs are mostly on HTTP then this solution might work for you: https://forum.pfsense.org/index.php?topic=94222.0. However, once advertisers start using HTTPS, using squid as a transparent proxy becomes noticeably less simple. In any case, for the vast majority of ads out there, DNS-based blocking does the trick - you'll still often lose screen real-estate to ad elements, but you won't see the actual ads.

The same site suggested in the thread referenced above has a dnsmasq-compatible list that you can get in plaintext form:
http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&mimetype=plaintext
There may be other sites that offer blacklists in dnsmasq form, or other options from this site that you want to use - YMMV.

To make this work, you'll need to tell dnsmasq to read additional configuration files from a config directory, with for example "conf_dir=/etc/dnsmasq" in the advanced config options. SSH into the machine, create that directory, and then download the file to it. Restart dnsmasq and you're good to go.

To automate the process for periodic updates you'll need to add a cron job, and you'll need either wget or curl installed on the pfsense box. SSH in, go to shell, run "pkg install curl" or wget, and eventually you'll have a working version. If that doesn't work, run "pkg" without any options first. Then your cron job can run one of these commands:

curl http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&mimetype=plaintext > /etc/dnsmasq/dnsmasq.blocklist
wget -O /etc/dnsmasq/dnsmasq.blocklist http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&mimetype=plaintext

Now, to restart dnsmasq you'll need the following file: /etc/phpshellsessions/restartdnsmasq

services_dnsmasq_configure();

Then run "pfSsh.php playback restartdnsmasq" within your cron script to restart the dnsmasq server.

alex_ncus

OK, here is what I am using (seems to be working for me).

Using dnsmasq guide @ http://thomasloughlin.com/pfsense-dnsmasq-advanced-setup

Created a new script (blackhole.sh) in /etc directory and then used CRON to update once a day.

Here is the script that I am using:

#!/bin/sh
## blackhole.sh
## Adapted for pfSense from Tomato WAN Up script v3.3 by haarp

TMPFILE="/tmp/dnsmasq.work"                          ## dnsmasq temporary file
GENFILE="/usr/local/etc/dnsmasq.d/dnsmasq.custom"    ## dnsmasq custom config

SOURCES=""
SOURCES="$SOURCES http://winhelp2002.mvps.org/hosts.txt"
SOURCES="$SOURCES http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext"
#SOURCES="$SOURCES http://hosts-file.net/ad_servers.asp"
SOURCES="$SOURCES http://hosts-file.net/.%5Cad_servers.txt"
##SOURCES="$SOURCES http://hostsfile.mine.nu/Hosts"
##SOURCES="$SOURCES http://sysctl.org/cameleon/hosts"
SOURCES="$SOURCES http://adaway.org/hosts.txt"
##SOURCES="$SOURCES http://hosts-file.net/download/hosts.txt"
#SOURCES="$SOURCES http://hosts-file.net/hphosts-partial.asp"
SOURCES="$SOURCES http://www.malwaredomainlist.com/hostslist/hosts.txt"

## Blacklist additional sites (add inside quotes, space-separated)
BLACKLIST="google-analytics.com"

## Whitelist sites from blocking (add inside quotes, space-separated)
WHITELIST=""

echo "Download starting"
until ping -q -c1 google.com >/dev/null; do
    echo "Waiting for internet"
    sleep 5
done

echo -n "" > $TMPFILE
for s in $SOURCES; do
    { (wget $s -O - || elog "Failed: $s") | \
      tr -d "\r" | \
      sed -e '/^[[:alnum:]:]/!d' | \
      awk '{print $2}' | \
      sed -e '/^localhost$/d' >> $TMPFILE
    } &
done

wait

if [ -s $TMPFILE ]; then
   echo "Download finished"
else
   echo "Failed: Download unsuccessful, aborting"
   rm $TMPFILE
   exit 1                                           
fi

echo "Generating $TMPFILE"
for b in $BLACKLIST; do
    echo "$b" >> $TMPFILE
done

for w in $WHITELIST; do
    sed -i -e "/$w/d" $TMPFILE
done

sort -u $TMPFILE -o $TMPFILE                                           ## Sort and remove duplicates
awk '{print "address=/"$0"/127.0.0.1/"}' $TMPFILE > $GENFILE           ## format file for dnsmasq ... address=/domain-name/127.0.0.1

echo "Config generated, $(wc -l < $GENFILE) unique hosts to block"
echo "Restarting dnsmasq"
service dnsmasq restart

echo "Deleting $TMPFILE to free memory"
rm $TMPFILE

f34rinc

If you take a look at the changes on github you'll see why it took so long for 2.0 to come out.

I encourage others to try Ad blocking\filtering with pfBlockerNG 2.0+ with its DNSBL using the unbound DNS resolver. If a domain is blocked I can use the alerts tab to suppress this domain from being blocked from future updates or manually enter the domain in the Custom Domain Suppression text box on the main DNSBL tab.

Below is some of the lists I am currently using myself

Ad Blocking List
http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext
http://hosts-file.net/download/hosts.zip
http://hosts-file.net/hphosts-partial.asp
http://hosts-file.net/.%5Cad_servers.txt
http://adaway.org/hosts.txt
http://someonewhocares.org/hosts/hosts
http://winhelp2002.mvps.org/hosts.txt
http://adblock.gjtech.net/?format=unix-hosts

Malware \ Phishing 
http://mirror1.malwaredomains.com/files/justdomains
http://www.malwaredomainlist.com/hostslist/hosts.txt
http://osint.bambenekconsulting.com/feeds/dga-feed.gz
http://data.phishtank.com/data/online-valid.csv.gz
https://dshield.org/feeds/suspiciousdomains_Medium.txt
https://dshield.org/feeds/suspiciousdomains_High.txt
https://www.openphish.com/feed.txt

alex_ncus

NICE!

One thing you might want to consider when deduplicating using domain hierarchy. For example, if blacklist consists of:

…
xyz.com
w1-1.xyz.com
w1-2.xyz.com
www.xyz.com
...

they could all be reduced to a single xyz.com entry (instead of 4) ...

this could have a dramatic impact of (substantially) reducing table size (and improving search performance). Perhaps down by 2/3 for large blacklists.

awk -F "." '{for(i=NF; i > 1; i--) printf "%s.", $i; print $1}' $TMPFILE | sort -u | awk -F "." '{for(i=NF; i > 1; i--) printf "%s.", $i; print $1}' | awk 'BEGIN {last = "foo"} /^$/ {next} $0 !~ last { print "address=/"$0"/127.0.0.1"; last = $0 }' > $GENFILE

I tried this in earlier blackhole example with dnsmasq and reduced 369799 unique entries down to 117386.

f34rinc

Stripping the URL down to the domain is going to cause a lot of false positives.

ads.xyz.com
turns into
xyz.com

what if I try to block ads.yahoo.com and end up blocking all of yahoo.com

Nullity

@f34rinc:

Stripping the URL down to the domain is going to cause a lot of false positives.

ads.xyz.com
turns into
xyz.com

what if I try to block ads.yahoo.com and end up blocking all of yahoo.com

I did not look through his script, but if it is accurate to his explanation, it should only consolidate when it will make no difference.

so, if
a.xyz.co
b.xyz.co
xyz.co
would consolidate to simply xyz.co, but not if only a.xyz.co and b.xyz.co were listed.

Honestly, I would expect pfblockerng, or any blocklist aggregator, to do that type of optimization.

bluepr0

EDIT: never mind, I got the pfblocker list working with DNSBL. One thing I'm noticing is that for some websites it takes a lot of time to resolve the DNS, sometimes it gets inside the web and sometimes it shows connection time out on the browser. Would be great to know a bit more in detail the overall configuration so pfblocker works fine! (for example, do I have to stop some other service?)

BBcan177

bluepr0,

Make sure that all of the LAN devices DNS settings, are pointing only to the DNS Resolver. You shouldn't be getting those timeout messages. Are you on a Multi-Segmented LAN setup? If yes, then in the DNBSL tab, there is a checkbox to auto-create a floating firewall rule to allow the traffic from those other LAN subnets to access the DNSBL VIP… You should be able to ping and browse to the DNSBL VIP (1x1 pixel).

bluepr0

Fixed it! Now it's working nicely!

I used to have a VM with pi-hole.net but if I can have ad filter directly on the router, much better

Now I will have to read more about easylist, so I can add Adblock lists!