Ad blocking with pfsense



  • Yeah, you might shout out: squid / squidguard. That is indeed a good (the best?) solution but unfortunately not an option because i have a multi-WAN setup and squidk keeps using the default gateway for all connections and ignores the firewall rules to set specifis gateways.
    pfBlockerNG 2.0 supposed to incorporate a functionality to block ads but the release date is set to "when it's done".
    So what other options do I have to do adblocking on my pfsense box?



  • I believe that there are many ways to go about this. One of my favorites which apparently does not work anymore is HAVP because you could easily add an entire domain without having to worry about bs characters in the front where they constantly randomize. Another option that might work well is Dan's Guardian.

    In quite a few cases Suricata has blocked some ads for me but in a lot of those cases the site would become nearly unusable because Suricata does not play.

    I'm thinking that there is a way to do it with Squid as well if you are able to add a domain like this for instance if I didn't want google anymore, HAVP would let me type something like this ( .google.com/)

    It's been so long though so I really don't remember the proper syntax but you can look it up or just install HAVP to have a look at it and get some ideas even if it doesn't work you can learn about some things.

    Pfblocker is another option and just remember that you could get an entire ad list of ip addresses, but it may block too much so you really need to get in there and right click on the ads to find out where they are coming from and deal with the domains.

    Who wants to do a thousand IP addresses when you can just kill it with the domain.

    The other option is, if it's that bad I would just go ahead and use Adblock plus. I don't use it because it slows down my computer.

    Also take a look at some of the threads that are already created if you can find them. There are many great ideas about this and more than likely you will find a method that is the most efficient for you just by using the search function at the top right. Good luck.



  • If pfSense is your DNS, you could employ a modified /etc/hosts file, right? That is the simplest method I know. I use this method.

    My only complaint is that I am unaware of a quick way to unblock a site, but this is only a problem when using some shady file-sharing website.


  • Banned

    Or PM BBcan17 about pfBlockerNG-Dev testing.



  • @Cmellons:

    The other option is, if it's that bad I would just go ahead and use Adblock plus. I don't use it because it slows down my computer.

    NoScript in firefox is very fast and makes loading of websites even faster as I find the CDN's take for ever getting their act together.

    You still need to tweak the settings for the first time when you download this addon though, eg in the options settings
    General Tab,
    Unticked - Automatically reload affected pages when permissions change
    Ticked - Reload the current tab only.
    If you have multiple tabs open for the same site but want to allow one or more domains hilighted on tab, stops the other tabs open on the same main domain from reloading so it saves you bandwith but also reduces the instance of a pattern forming which can show you are using NoScript in default mode.

    Clear the whitelist down and build it up from scratch, it doesnt take much to whitelist your favourite websites.

    The rest of the options/tabs can be left alone iirc, but using NoScript certainly speeds up the loading of webpages, gives you more privacy from online trackers and reduces your risk of getting a virus as I have seen plenty of new viruses delivered through adverts coming from Content Delivery Networks over the years. On the virus point, these CDN's still use the same popular AV tools you and I can use, so a virus writer just needs to come up with something new to bypass the existing checks et voila you can infect millions of people in the blink of an eye so to speak, hence the need for things like NoScript is valid.



  • Unfortunately it is not that easy: HAVP does rely on squid as well which is, currently, considered to be broken for multi-wan environments as it keeps using the default gateway, no matter what firewall rules you configure.
    A browser extension is nice - where it is available. On most mobile devices (Windows Phone & RT, Apple) it is not. Also there are systemwide ads within the apps etc.
    I'll contact BBcan17  ;)



  • @fraglord:

    Unfortunately it is not that easy: HAVP does rely on squid as well which is, currently, considered to be broken for multi-wan environments as it keeps using the default gateway, no matter what firewall rules you configure.
    A browser extension is nice - where it is available. On most mobile devices (Windows Phone & RT, Apple) it is not. Also there are systemwide ads within the apps etc.
    I'll contact BBcan17  ;)

    Perhaps building a list of domains which are known trackers or sites you dont want interact with would be a start for blocking purposes, and then for mobile devices or where a browser extension doesnt exist, having the browser tunnel back to pfsense via a vpn may be a suitable workaround although not perfect.

    I say not perfect because I dont allow youtube for most things or google for that matter so default is its blocked, but sometimes I want to watch something on youtube or google from another site so I enable it in those instances.

    I guess something which would be closer to ideal would be to have a list of domain associated with themselves, eg take the TheGuardian.com you also have guardianapis.com, guardianapps.co.uk, & guim.co.uk which all need to be enabled before you can read and interact with the main TheGuardian.com domain. So having a system/app on pfsense which can recognise you are visiting the main domain and thus allow the other domains to work would be useful.

    Edit.
    Whilst there are these links which suggest ways to block by domain name, these will only be worthwhile for blocking trackers imo.

    http://samkear.com/pfsense/how-to-configure-a-dns-blacklist-using-pfsense
    https://ejnetwork.wordpress.com/2014/08/04/blocking-domains-with-pfsense-using-dns-forwarder/
    https://doc.pfsense.org/index.php/Blocking_websites
    https://forum.pfsense.org/index.php?topic=19434.0
    http://hubpages.com/hub/URL-Filtering-How-To-Configure-SquidGuard-in-pfSense

    YMMV



  • @doktornotor:

    Or PM BBcan17 about pfBlockerNG-Dev testing.

    Or wait for the release…...so far so good... ;D



  • DEAR PEOPLE FROM THE FUTURE: Here's what we've figured out so far …

    Sorry to bump an old thread, but while I've seen a number of ideas on this topic, I don't think that this ever got a satisfactory response.. Waiting for the next pfBlockerNG doesn't seem to be the right choice - several months after that suggestion, it's still not out. Using adblock in firefox is all very well for desktops, but ad blocking on mobile devices is sometimes itself blocked by the OS, and on devices like TVs and Roku or similar, you have no real control over the host.

    If your ad blocking needs are mostly on HTTP then this solution might work for you: https://forum.pfsense.org/index.php?topic=94222.0. However, once advertisers start using HTTPS, using squid as a transparent proxy becomes noticeably less simple. In any case, for the vast majority of ads out there, DNS-based blocking does the trick - you'll still often lose screen real-estate to ad elements, but you won't see the actual ads.

    The same site suggested in the thread referenced above has a dnsmasq-compatible list that you can get in plaintext form:
    http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&mimetype=plaintext
    There may be other sites that offer blacklists in dnsmasq form, or other options from this site that you want to use - YMMV.

    To make this work, you'll need to tell dnsmasq to read additional configuration files from a config directory, with for example "conf_dir=/etc/dnsmasq" in the advanced config options. SSH into the machine, create that directory, and then download the file to it. Restart dnsmasq and you're good to go.

    To automate the process for periodic updates you'll need to add a cron job, and you'll need either wget or curl installed on the pfsense box. SSH in, go to shell, run "pkg install curl" or wget, and eventually you'll have a working version. If that doesn't work, run "pkg" without any options first. Then your cron job can run one of these commands:

    curl http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&mimetype=plaintext > /etc/dnsmasq/dnsmasq.blocklist
    wget -O /etc/dnsmasq/dnsmasq.blocklist http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&mimetype=plaintext
    

    Now, to restart dnsmasq you'll need the following file: /etc/phpshellsessions/restartdnsmasq

    services_dnsmasq_configure();
    

    Then run "pfSsh.php playback restartdnsmasq" within your cron script to restart the dnsmasq server.



  • OK, here is what I am using (seems to be working for me).

    Using dnsmasq guide @ http://thomasloughlin.com/pfsense-dnsmasq-advanced-setup

    Created a new script (blackhole.sh) in /etc directory and then used CRON to update once a day.

    Here is the script that I am using:

    #!/bin/sh
    ## blackhole.sh
    ## Adapted for pfSense from Tomato WAN Up script v3.3 by haarp
    
    TMPFILE="/tmp/dnsmasq.work"                          ## dnsmasq temporary file
    GENFILE="/usr/local/etc/dnsmasq.d/dnsmasq.custom"    ## dnsmasq custom config
    
    SOURCES=""
    SOURCES="$SOURCES http://winhelp2002.mvps.org/hosts.txt"
    SOURCES="$SOURCES http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext"
    #SOURCES="$SOURCES http://hosts-file.net/ad_servers.asp"
    SOURCES="$SOURCES http://hosts-file.net/.%5Cad_servers.txt"
    ##SOURCES="$SOURCES http://hostsfile.mine.nu/Hosts"
    ##SOURCES="$SOURCES http://sysctl.org/cameleon/hosts"
    SOURCES="$SOURCES http://adaway.org/hosts.txt"
    ##SOURCES="$SOURCES http://hosts-file.net/download/hosts.txt"
    #SOURCES="$SOURCES http://hosts-file.net/hphosts-partial.asp"
    SOURCES="$SOURCES http://www.malwaredomainlist.com/hostslist/hosts.txt"
    
    ## Blacklist additional sites (add inside quotes, space-separated)
    BLACKLIST="google-analytics.com"
    
    ## Whitelist sites from blocking (add inside quotes, space-separated)
    WHITELIST=""
    
    echo "Download starting"
    until ping -q -c1 google.com >/dev/null; do
        echo "Waiting for internet"
        sleep 5
    done
    
    echo -n "" > $TMPFILE
    for s in $SOURCES; do
        { (wget $s -O - || elog "Failed: $s") | \
          tr -d "\r" | \
          sed -e '/^[[:alnum:]:]/!d' | \
          awk '{print $2}' | \
          sed -e '/^localhost$/d' >> $TMPFILE
        } &
    done
    
    wait
    
    if [ -s $TMPFILE ]; then
       echo "Download finished"
    else
       echo "Failed: Download unsuccessful, aborting"
       rm $TMPFILE
       exit 1                                           
    fi
    
    echo "Generating $TMPFILE"
    for b in $BLACKLIST; do
        echo "$b" >> $TMPFILE
    done
    
    for w in $WHITELIST; do
        sed -i -e "/$w/d" $TMPFILE
    done
    
    sort -u $TMPFILE -o $TMPFILE                                           ## Sort and remove duplicates
    awk '{print "address=/"$0"/127.0.0.1/"}' $TMPFILE > $GENFILE           ## format file for dnsmasq ... address=/domain-name/127.0.0.1
    
    echo "Config generated, $(wc -l < $GENFILE) unique hosts to block"
    echo "Restarting dnsmasq"
    service dnsmasq restart
    
    echo "Deleting $TMPFILE to free memory"
    rm $TMPFILE
    


  • If you take a look at the changes on github you'll see why it took so long for 2.0 to come out.

    I encourage others to try Ad blocking\filtering with pfBlockerNG 2.0+ with its DNSBL using the unbound DNS resolver.  If a domain is blocked I can use the alerts tab to suppress this domain from being blocked from future updates or manually enter the domain in the Custom Domain Suppression text box on the main DNSBL tab.

    Below is some of the lists I am currently using myself

    Ad Blocking List
    http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext
    http://hosts-file.net/download/hosts.zip
    http://hosts-file.net/hphosts-partial.asp
    http://hosts-file.net/.%5Cad_servers.txt
    http://adaway.org/hosts.txt
    http://someonewhocares.org/hosts/hosts
    http://winhelp2002.mvps.org/hosts.txt
    http://adblock.gjtech.net/?format=unix-hosts
    
    
    Malware \ Phishing 
    http://mirror1.malwaredomains.com/files/justdomains
    http://www.malwaredomainlist.com/hostslist/hosts.txt
    http://osint.bambenekconsulting.com/feeds/dga-feed.gz
    http://data.phishtank.com/data/online-valid.csv.gz
    https://dshield.org/feeds/suspiciousdomains_Medium.txt
    https://dshield.org/feeds/suspiciousdomains_High.txt
    https://www.openphish.com/feed.txt
    
    


  • NICE!

    One thing you might want to consider when deduplicating using domain hierarchy. For example, if blacklist consists of:


    xyz.com
    w1-1.xyz.com
    w1-2.xyz.com
    www.xyz.com
    ...

    they could all be reduced to a single xyz.com entry (instead of 4) ...

    this could have a dramatic impact of (substantially) reducing table size (and improving search performance). Perhaps down by 2/3 for large blacklists.

    awk -F "." '{for(i=NF; i > 1; i--) printf "%s.", $i; print $1}' $TMPFILE | sort -u | awk -F "." '{for(i=NF; i > 1; i--) printf "%s.", $i; print $1}' | awk 'BEGIN {last = "foo"} /^$/ {next} $0 !~ last { print "address=/"$0"/127.0.0.1"; last = $0 }' > $GENFILE
    

    I tried this in earlier blackhole example with dnsmasq and reduced 369799 unique entries down to 117386.



  • Stripping the URL down to the domain is going to cause a lot of false positives.

    ads.xyz.com
    turns into
    xyz.com

    what if I try to block ads.yahoo.com and end up blocking all of yahoo.com



  • @f34rinc:

    Stripping the URL down to the domain is going to cause a lot of false positives.

    ads.xyz.com
    turns into
    xyz.com

    what if I try to block ads.yahoo.com and end up blocking all of yahoo.com

    I did not look through his script, but if it is accurate to his explanation, it should only consolidate when it will make no difference.

    so, if
    a.xyz.co
    b.xyz.co
    xyz.co
    would consolidate to simply xyz.co, but not if only a.xyz.co and b.xyz.co were listed.

    Honestly, I would expect pfblockerng, or any blocklist aggregator, to do that type of optimization.



  • EDIT: never mind, I got the pfblocker list working with DNSBL. One thing I'm noticing is that for some websites it takes a lot of time to resolve the DNS, sometimes it gets inside the web and sometimes it shows connection time out on the browser. Would be great to know a bit more in detail the overall configuration so pfblocker works fine! (for example, do I have to stop some other service?)


  • Moderator

    bluepr0,

    Make sure that all of the LAN devices DNS settings, are pointing only to the DNS Resolver. You shouldn't be getting those timeout messages. Are you on a Multi-Segmented LAN setup? If yes, then in the DNBSL tab, there is a checkbox to auto-create a floating firewall rule to allow the traffic from those other LAN subnets to access the DNSBL VIP… You should be able to ping and browse to the DNSBL VIP (1x1 pixel).



  • Fixed it! Now it's working nicely!

    I used to have a VM with pi-hole.net but if I can have ad filter directly on the router, much better

    Now I will have to read more about easylist, so I can add Adblock lists!


Log in to reply