• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

Ad blocking with pfsense

Scheduled Pinned Locked Moved General pfSense Questions
17 Posts 11 Posters 29.0k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • F
    fraglord
    last edited by May 21, 2015, 11:30 PM

    Yeah, you might shout out: squid / squidguard. That is indeed a good (the best?) solution but unfortunately not an option because i have a multi-WAN setup and squidk keeps using the default gateway for all connections and ignores the firewall rules to set specifis gateways.
    pfBlockerNG 2.0 supposed to incorporate a functionality to block ads but the release date is set to "when it's done".
    So what other options do I have to do adblocking on my pfsense box?

    pfSense 2.4.0 (amd64) running on IGEL H710C | 1G RAM | 8G SSD | INTEL PRO/1000 PT Dual NIC

    1 Reply Last reply Reply Quote 0
    • C
      Cmellons
      last edited by May 22, 2015, 1:29 AM May 22, 2015, 1:21 AM

      I believe that there are many ways to go about this. One of my favorites which apparently does not work anymore is HAVP because you could easily add an entire domain without having to worry about bs characters in the front where they constantly randomize. Another option that might work well is Dan's Guardian.

      In quite a few cases Suricata has blocked some ads for me but in a lot of those cases the site would become nearly unusable because Suricata does not play.

      I'm thinking that there is a way to do it with Squid as well if you are able to add a domain like this for instance if I didn't want google anymore, HAVP would let me type something like this ( .google.com/)

      It's been so long though so I really don't remember the proper syntax but you can look it up or just install HAVP to have a look at it and get some ideas even if it doesn't work you can learn about some things.

      Pfblocker is another option and just remember that you could get an entire ad list of ip addresses, but it may block too much so you really need to get in there and right click on the ads to find out where they are coming from and deal with the domains.

      Who wants to do a thousand IP addresses when you can just kill it with the domain.

      The other option is, if it's that bad I would just go ahead and use Adblock plus. I don't use it because it slows down my computer.

      Also take a look at some of the threads that are already created if you can find them. There are many great ideas about this and more than likely you will find a method that is the most efficient for you just by using the search function at the top right. Good luck.

      1 Reply Last reply Reply Quote 0
      • N
        Nullity
        last edited by May 22, 2015, 5:07 AM

        If pfSense is your DNS, you could employ a modified /etc/hosts file, right? That is the simplest method I know. I use this method.

        My only complaint is that I am unaware of a quick way to unblock a site, but this is only a problem when using some shady file-sharing website.

        Please correct any obvious misinformation in my posts.
        -Not a professional; an arrogant ignoramous.

        1 Reply Last reply Reply Quote 0
        • D
          doktornotor Banned
          last edited by May 22, 2015, 7:17 AM

          Or PM BBcan17 about pfBlockerNG-Dev testing.

          1 Reply Last reply Reply Quote 0
          • F
            firewalluser
            last edited by May 22, 2015, 9:35 AM

            @Cmellons:

            The other option is, if it's that bad I would just go ahead and use Adblock plus. I don't use it because it slows down my computer.

            NoScript in firefox is very fast and makes loading of websites even faster as I find the CDN's take for ever getting their act together.

            You still need to tweak the settings for the first time when you download this addon though, eg in the options settings
            General Tab,
            Unticked - Automatically reload affected pages when permissions change
            Ticked - Reload the current tab only.
            If you have multiple tabs open for the same site but want to allow one or more domains hilighted on tab, stops the other tabs open on the same main domain from reloading so it saves you bandwith but also reduces the instance of a pattern forming which can show you are using NoScript in default mode.

            Clear the whitelist down and build it up from scratch, it doesnt take much to whitelist your favourite websites.

            The rest of the options/tabs can be left alone iirc, but using NoScript certainly speeds up the loading of webpages, gives you more privacy from online trackers and reduces your risk of getting a virus as I have seen plenty of new viruses delivered through adverts coming from Content Delivery Networks over the years. On the virus point, these CDN's still use the same popular AV tools you and I can use, so a virus writer just needs to come up with something new to bypass the existing checks et voila you can infect millions of people in the blink of an eye so to speak, hence the need for things like NoScript is valid.

            Capitalism, currently The World's best Entertainment Control System and YOU cant buy it! But you can buy this, or some of this or some of these

            Asch Conformity, mainly the blind leading the blind.

            1 Reply Last reply Reply Quote 0
            • F
              fraglord
              last edited by May 22, 2015, 10:24 AM

              Unfortunately it is not that easy: HAVP does rely on squid as well which is, currently, considered to be broken for multi-wan environments as it keeps using the default gateway, no matter what firewall rules you configure.
              A browser extension is nice - where it is available. On most mobile devices (Windows Phone & RT, Apple) it is not. Also there are systemwide ads within the apps etc.
              I'll contact BBcan17  ;)

              pfSense 2.4.0 (amd64) running on IGEL H710C | 1G RAM | 8G SSD | INTEL PRO/1000 PT Dual NIC

              1 Reply Last reply Reply Quote 0
              • F
                firewalluser
                last edited by May 22, 2015, 5:34 PM May 22, 2015, 5:30 PM

                @fraglord:

                Unfortunately it is not that easy: HAVP does rely on squid as well which is, currently, considered to be broken for multi-wan environments as it keeps using the default gateway, no matter what firewall rules you configure.
                A browser extension is nice - where it is available. On most mobile devices (Windows Phone & RT, Apple) it is not. Also there are systemwide ads within the apps etc.
                I'll contact BBcan17  ;)

                Perhaps building a list of domains which are known trackers or sites you dont want interact with would be a start for blocking purposes, and then for mobile devices or where a browser extension doesnt exist, having the browser tunnel back to pfsense via a vpn may be a suitable workaround although not perfect.

                I say not perfect because I dont allow youtube for most things or google for that matter so default is its blocked, but sometimes I want to watch something on youtube or google from another site so I enable it in those instances.

                I guess something which would be closer to ideal would be to have a list of domain associated with themselves, eg take the TheGuardian.com you also have guardianapis.com, guardianapps.co.uk, & guim.co.uk which all need to be enabled before you can read and interact with the main TheGuardian.com domain. So having a system/app on pfsense which can recognise you are visiting the main domain and thus allow the other domains to work would be useful.

                Edit.
                Whilst there are these links which suggest ways to block by domain name, these will only be worthwhile for blocking trackers imo.

                http://samkear.com/pfsense/how-to-configure-a-dns-blacklist-using-pfsense
                https://ejnetwork.wordpress.com/2014/08/04/blocking-domains-with-pfsense-using-dns-forwarder/
                https://doc.pfsense.org/index.php/Blocking_websites
                https://forum.pfsense.org/index.php?topic=19434.0
                http://hubpages.com/hub/URL-Filtering-How-To-Configure-SquidGuard-in-pfSense

                YMMV

                Capitalism, currently The World's best Entertainment Control System and YOU cant buy it! But you can buy this, or some of this or some of these

                Asch Conformity, mainly the blind leading the blind.

                1 Reply Last reply Reply Quote 0
                • W
                  Wolf666
                  last edited by May 23, 2015, 7:33 PM

                  @doktornotor:

                  Or PM BBcan17 about pfBlockerNG-Dev testing.

                  Or wait for the release…...so far so good... ;D

                  Modem Draytek Vigor 130
                  pfSense 2.4 Supermicro A1SRi-2558 - 8GB ECC RAM - Intel S3500 SSD 80GB - M350 Case
                  Switch Cisco SG350-10
                  AP Netgear R7000 (Stock FW)
                  HTPC Intel NUC5i3RYH
                  NAS Synology DS1515+
                  NAS Synology DS213+

                  1 Reply Last reply Reply Quote 0
                  • S
                    syntactic.net
                    last edited by Sep 22, 2015, 2:11 AM

                    DEAR PEOPLE FROM THE FUTURE: Here's what we've figured out so far …

                    Sorry to bump an old thread, but while I've seen a number of ideas on this topic, I don't think that this ever got a satisfactory response.. Waiting for the next pfBlockerNG doesn't seem to be the right choice - several months after that suggestion, it's still not out. Using adblock in firefox is all very well for desktops, but ad blocking on mobile devices is sometimes itself blocked by the OS, and on devices like TVs and Roku or similar, you have no real control over the host.

                    If your ad blocking needs are mostly on HTTP then this solution might work for you: https://forum.pfsense.org/index.php?topic=94222.0. However, once advertisers start using HTTPS, using squid as a transparent proxy becomes noticeably less simple. In any case, for the vast majority of ads out there, DNS-based blocking does the trick - you'll still often lose screen real-estate to ad elements, but you won't see the actual ads.

                    The same site suggested in the thread referenced above has a dnsmasq-compatible list that you can get in plaintext form:
                    http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&mimetype=plaintext
                    There may be other sites that offer blacklists in dnsmasq form, or other options from this site that you want to use - YMMV.

                    To make this work, you'll need to tell dnsmasq to read additional configuration files from a config directory, with for example "conf_dir=/etc/dnsmasq" in the advanced config options. SSH into the machine, create that directory, and then download the file to it. Restart dnsmasq and you're good to go.

                    To automate the process for periodic updates you'll need to add a cron job, and you'll need either wget or curl installed on the pfsense box. SSH in, go to shell, run "pkg install curl" or wget, and eventually you'll have a working version. If that doesn't work, run "pkg" without any options first. Then your cron job can run one of these commands:

                    curl http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&mimetype=plaintext > /etc/dnsmasq/dnsmasq.blocklist
                    wget -O /etc/dnsmasq/dnsmasq.blocklist http://pgl.yoyo.org/adservers/serverlist.php?hostformat=dnsmasq&mimetype=plaintext
                    

                    Now, to restart dnsmasq you'll need the following file: /etc/phpshellsessions/restartdnsmasq

                    services_dnsmasq_configure();
                    

                    Then run "pfSsh.php playback restartdnsmasq" within your cron script to restart the dnsmasq server.

                    1 Reply Last reply Reply Quote 0
                    • A
                      alex_ncus
                      last edited by Dec 2, 2015, 11:27 PM Dec 2, 2015, 11:23 PM

                      OK, here is what I am using (seems to be working for me).

                      Using dnsmasq guide @ http://thomasloughlin.com/pfsense-dnsmasq-advanced-setup

                      Created a new script (blackhole.sh) in /etc directory and then used CRON to update once a day.

                      Here is the script that I am using:

                      #!/bin/sh
                      ## blackhole.sh
                      ## Adapted for pfSense from Tomato WAN Up script v3.3 by haarp
                      
                      TMPFILE="/tmp/dnsmasq.work"                          ## dnsmasq temporary file
                      GENFILE="/usr/local/etc/dnsmasq.d/dnsmasq.custom"    ## dnsmasq custom config
                      
                      SOURCES=""
                      SOURCES="$SOURCES http://winhelp2002.mvps.org/hosts.txt"
                      SOURCES="$SOURCES http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext"
                      #SOURCES="$SOURCES http://hosts-file.net/ad_servers.asp"
                      SOURCES="$SOURCES http://hosts-file.net/.%5Cad_servers.txt"
                      ##SOURCES="$SOURCES http://hostsfile.mine.nu/Hosts"
                      ##SOURCES="$SOURCES http://sysctl.org/cameleon/hosts"
                      SOURCES="$SOURCES http://adaway.org/hosts.txt"
                      ##SOURCES="$SOURCES http://hosts-file.net/download/hosts.txt"
                      #SOURCES="$SOURCES http://hosts-file.net/hphosts-partial.asp"
                      SOURCES="$SOURCES http://www.malwaredomainlist.com/hostslist/hosts.txt"
                      
                      ## Blacklist additional sites (add inside quotes, space-separated)
                      BLACKLIST="google-analytics.com"
                      
                      ## Whitelist sites from blocking (add inside quotes, space-separated)
                      WHITELIST=""
                      
                      echo "Download starting"
                      until ping -q -c1 google.com >/dev/null; do
                          echo "Waiting for internet"
                          sleep 5
                      done
                      
                      echo -n "" > $TMPFILE
                      for s in $SOURCES; do
                          { (wget $s -O - || elog "Failed: $s") | \
                            tr -d "\r" | \
                            sed -e '/^[[:alnum:]:]/!d' | \
                            awk '{print $2}' | \
                            sed -e '/^localhost$/d' >> $TMPFILE
                          } &
                      done
                      
                      wait
                      
                      if [ -s $TMPFILE ]; then
                         echo "Download finished"
                      else
                         echo "Failed: Download unsuccessful, aborting"
                         rm $TMPFILE
                         exit 1                                           
                      fi
                      
                      echo "Generating $TMPFILE"
                      for b in $BLACKLIST; do
                          echo "$b" >> $TMPFILE
                      done
                      
                      for w in $WHITELIST; do
                          sed -i -e "/$w/d" $TMPFILE
                      done
                      
                      sort -u $TMPFILE -o $TMPFILE                                           ## Sort and remove duplicates
                      awk '{print "address=/"$0"/127.0.0.1/"}' $TMPFILE > $GENFILE           ## format file for dnsmasq ... address=/domain-name/127.0.0.1
                      
                      echo "Config generated, $(wc -l < $GENFILE) unique hosts to block"
                      echo "Restarting dnsmasq"
                      service dnsmasq restart
                      
                      echo "Deleting $TMPFILE to free memory"
                      rm $TMPFILE
                      
                      1 Reply Last reply Reply Quote 0
                      • F
                        f34rinc
                        last edited by Dec 7, 2015, 5:16 PM Dec 3, 2015, 6:00 PM

                        If you take a look at the changes on github you'll see why it took so long for 2.0 to come out.

                        I encourage others to try Ad blocking\filtering with pfBlockerNG 2.0+ with its DNSBL using the unbound DNS resolver.  If a domain is blocked I can use the alerts tab to suppress this domain from being blocked from future updates or manually enter the domain in the Custom Domain Suppression text box on the main DNSBL tab.

                        Below is some of the lists I am currently using myself

                        Ad Blocking List
                        http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext
                        http://hosts-file.net/download/hosts.zip
                        http://hosts-file.net/hphosts-partial.asp
                        http://hosts-file.net/.%5Cad_servers.txt
                        http://adaway.org/hosts.txt
                        http://someonewhocares.org/hosts/hosts
                        http://winhelp2002.mvps.org/hosts.txt
                        http://adblock.gjtech.net/?format=unix-hosts
                        
                        
                        Malware \ Phishing 
                        http://mirror1.malwaredomains.com/files/justdomains
                        http://www.malwaredomainlist.com/hostslist/hosts.txt
                        http://osint.bambenekconsulting.com/feeds/dga-feed.gz
                        http://data.phishtank.com/data/online-valid.csv.gz
                        https://dshield.org/feeds/suspiciousdomains_Medium.txt
                        https://dshield.org/feeds/suspiciousdomains_High.txt
                        https://www.openphish.com/feed.txt
                        
                        
                        1 Reply Last reply Reply Quote 0
                        • A
                          alex_ncus
                          last edited by Dec 16, 2015, 8:13 PM

                          NICE!

                          One thing you might want to consider when deduplicating using domain hierarchy. For example, if blacklist consists of:

                          …
                          xyz.com
                          w1-1.xyz.com
                          w1-2.xyz.com
                          www.xyz.com
                          ...

                          they could all be reduced to a single xyz.com entry (instead of 4) ...

                          this could have a dramatic impact of (substantially) reducing table size (and improving search performance). Perhaps down by 2/3 for large blacklists.

                          awk -F "." '{for(i=NF; i > 1; i--) printf "%s.", $i; print $1}' $TMPFILE | sort -u | awk -F "." '{for(i=NF; i > 1; i--) printf "%s.", $i; print $1}' | awk 'BEGIN {last = "foo"} /^$/ {next} $0 !~ last { print "address=/"$0"/127.0.0.1"; last = $0 }' > $GENFILE
                          

                          I tried this in earlier blackhole example with dnsmasq and reduced 369799 unique entries down to 117386.

                          1 Reply Last reply Reply Quote 0
                          • F
                            f34rinc
                            last edited by Dec 17, 2015, 5:27 AM

                            Stripping the URL down to the domain is going to cause a lot of false positives.

                            ads.xyz.com
                            turns into
                            xyz.com

                            what if I try to block ads.yahoo.com and end up blocking all of yahoo.com

                            1 Reply Last reply Reply Quote 0
                            • N
                              Nullity
                              last edited by Dec 17, 2015, 10:35 AM

                              @f34rinc:

                              Stripping the URL down to the domain is going to cause a lot of false positives.

                              ads.xyz.com
                              turns into
                              xyz.com

                              what if I try to block ads.yahoo.com and end up blocking all of yahoo.com

                              I did not look through his script, but if it is accurate to his explanation, it should only consolidate when it will make no difference.

                              so, if
                              a.xyz.co
                              b.xyz.co
                              xyz.co
                              would consolidate to simply xyz.co, but not if only a.xyz.co and b.xyz.co were listed.

                              Honestly, I would expect pfblockerng, or any blocklist aggregator, to do that type of optimization.

                              Please correct any obvious misinformation in my posts.
                              -Not a professional; an arrogant ignoramous.

                              1 Reply Last reply Reply Quote 0
                              • B
                                bluepr0
                                last edited by Dec 17, 2015, 11:03 PM Dec 17, 2015, 10:41 PM

                                EDIT: never mind, I got the pfblocker list working with DNSBL. One thing I'm noticing is that for some websites it takes a lot of time to resolve the DNS, sometimes it gets inside the web and sometimes it shows connection time out on the browser. Would be great to know a bit more in detail the overall configuration so pfblocker works fine! (for example, do I have to stop some other service?)

                                1 Reply Last reply Reply Quote 0
                                • BBcan177B
                                  BBcan177 Moderator
                                  last edited by Dec 17, 2015, 11:41 PM

                                  bluepr0,

                                  Make sure that all of the LAN devices DNS settings, are pointing only to the DNS Resolver. You shouldn't be getting those timeout messages. Are you on a Multi-Segmented LAN setup? If yes, then in the DNBSL tab, there is a checkbox to auto-create a floating firewall rule to allow the traffic from those other LAN subnets to access the DNSBL VIP… You should be able to ping and browse to the DNSBL VIP (1x1 pixel).

                                  "Experience is something you don't get until just after you need it."

                                  Website: http://pfBlockerNG.com
                                  Twitter: @BBcan177  #pfBlockerNG
                                  Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                                  1 Reply Last reply Reply Quote 0
                                  • B
                                    bluepr0
                                    last edited by Dec 18, 2015, 6:32 AM

                                    Fixed it! Now it's working nicely!

                                    I used to have a VM with pi-hole.net but if I can have ad filter directly on the router, much better

                                    Now I will have to read more about easylist, so I can add Adblock lists!

                                    1 Reply Last reply Reply Quote 0
                                    • First post
                                      Last post
                                    Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
                                      [[user:consent.lead]]
                                      [[user:consent.not_received]]