Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Whitelist for uBlock?

    Scheduled Pinned Locked Moved pfBlockerNG
    16 Posts 5 Posters 6.9k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      MrGlasspoole
      last edited by

      Sorry if i don't read all the pfBlockerNG threads.

      I did installed Pi-hole yesterday and now found pfBlockerNG.

      With Pi-hole i first did wonder i see ads in YouTube videos and also allot of other ads on websites.
      I'm a uBlock Origin user in browsers and AdAway on Android and nowhere i have YouTube ads or other stuff.

      From reading Pi-hole can't block elements and does not work on iframes and does not do the cosmetic stuff that uBlock does.
      How is that with pfBlockerNG?

      How does pfBlockerNG handle the adblock formatted lists - removing all the rules and just leaving the domains?

      My idea would be doing some script that works together with uBlock.
      From looking at the 4 uBlock txt files it seems like they override/remove some rules that are in the EasyList and other lists?

      I'm not a bash or perl guy and don't know how to do it, but maybe:
      1. Download all the files
      2. Remove whats in our whitelist
      3. Apply/check the uBlock filters
      4. Create a file with the rules that only can be done by uBlock
      5. Create a file that is cleaned up from duplicates and the stuff that is done by uBlock (used then by pfBlockerNG)

      Then disabling every filter in uBlock and creating one that points to the one that was created by the script.
      So all devices have everything and browsers with uBlock can do there cosmetic stuff and have less work to do.
      Modify message

      1 Reply Last reply Reply Quote 0
      • BBcan177B
        BBcan177 Moderator
        last edited by

        @MrGlasspoole:

        I did installed Pi-hole yesterday and now found pfBlockerNG.

        Great news :)

        From reading Pi-hole can't block elements and does not work on iframes and does not do the cosmetic stuff that uBlock does. How is that with pfBlockerNG?

        Any DNS filter will only block based on Domain name… so DNS Filters cannot filter like UBlock origin which modifies the page elements etc...

        pfBlockerNG parses the EasyList feeds, and collects just the domains that can be blocked... The rest are discarded..

        If you still see an AD in a browser, right-click on it, and select "inspect"... Then if the element has a Domain name, you can add that domain to a customlist in DNSBL to block it...  otherwise, it can't be filtered with a DNS Filter...

        "Experience is something you don't get until just after you need it."

        Website: http://pfBlockerNG.com
        Twitter: @BBcan177  #pfBlockerNG
        Reddit: https://www.reddit.com/r/pfBlockerNG/new/

        1 Reply Last reply Reply Quote 0
        • M
          MrGlasspoole
          last edited by

          Hm, my idea is to fetch the uBlock filters:

          https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/filters.txt
          https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/badware.txt
          https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/privacy.txt
          https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/unbreak.txt

          Then compare this filters with the other lists (look for rules that need to be applied on domains in this list)
          and remove them and create a file for DNS.
          Create another (uBlock.txt) file with all the uBlock filters plus the stuff that was removed from the other lists.
          So stuff where the uBlock filters need to do its think go to your PC.

          I mean how much sense does it make to parse the EasyList feeds and collects just the domains if the rules are not working.
          I guess if you have all this:

          http://mirror1.malwaredomains.com/files/justdomains
          https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
          https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt
          http://malwaredomains.lehigh.edu/files/immortal_domains.txt
          https://s3.amazonaws.com/lists.disconnect.me/simple_malware.txt
          https://www.malwaredomainlist.com/hostslist/hosts.txt
          http://winhelp2002.mvps.org/hosts.txt
          https://adaway.org/hosts.txt
          https://hosts-file.net/ad_servers.txt
          https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=1&mimetype=plaintext

          then there are not much domains missing that the EasyLists would have.

          1 Reply Last reply Reply Quote 0
          • BBcan177B
            BBcan177 Moderator
            last edited by

            pfBlockerNG already has an EasyList tab… No need to convert anything....

            For these UBlock links... Seems only the first link (filters.txt) is somewhat usable... Typically look for lines that start with "||"... and then review if those domains are worth blocking...

            See the follow cmds... Found approx 129 domains, Most of these domains are already listed in the other DNSBL Feeds available...

            fetch -o /tmp/file "https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/filters.txt"
            grep "^||" /tmp/file | sed -e 's/$.*//' -e 's/\/.*//' -e 's/\^.*//' | tr -d '|' | grep -v '*' | sort | uniq
            

            FYI, DNSBL has deduplication.

            "Experience is something you don't get until just after you need it."

            Website: http://pfBlockerNG.com
            Twitter: @BBcan177  #pfBlockerNG
            Reddit: https://www.reddit.com/r/pfBlockerNG/new/

            1 Reply Last reply Reply Quote 0
            • W
              wcrowder
              last edited by

              BB' You work to hard… :) BTW, How you doing? Great work, keep it up! :)

              1 Reply Last reply Reply Quote 0
              • BBcan177B
                BBcan177 Moderator
                last edited by

                @wcrowder:

                BB' You work to hard… :) BTW, How you doing? Great work, keep it up! :)

                BC!! Where u been. Welcome back.  8) 8) 
                Next version will be even better. Just waiting for your final beta test approval :)

                "Experience is something you don't get until just after you need it."

                Website: http://pfBlockerNG.com
                Twitter: @BBcan177  #pfBlockerNG
                Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                1 Reply Last reply Reply Quote 0
                • M
                  MrGlasspoole
                  last edited by

                  Yes pfBlockerNG has an EasyList tab and you wrote:
                  @BBcan177:

                  pfBlockerNG parses the EasyList feeds, and collects just the domains that can be blocked… The rest are discarded..

                  I guess we talk past each other :)
                  If you look at the uBlock filters, there are not there to at more domains.
                  They fix stuff that are broken.
                  For example the first rule in: https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/unbreak.txt
                  Peter Lowe's list blocks google analytics so on some sites stuff does not work.

                  So the idea is to take out google analytics from all the other lists and let it pass through to the PCs.
                  Then on the PC uBlock can handle it.

                  1 Reply Last reply Reply Quote 0
                  • BBcan177B
                    BBcan177 Moderator
                    last edited by

                    @MrGlasspoole:

                    So the idea is to take out google analytics from all the other lists and let it pass through to the PCs.
                    Then on the PC uBlock can handle it.

                    Should have defined this thread as "UBlock whitelist" :)

                    I am not sure of the syntax used for the UBlock feeds… But if you parse these files, and collect the Domains to whitelist, you can add them to the DNSBL Whitelist customlist, and run a "Force Reload - DNSBL" for it to take effect... You can prefix a domain with a "dot" and it will also whitelist any sub-domains.

                    Everyones whitelist is going to be different, some of these exceptions from UBlock, are for a single website that might have issues with blocking a domain... It might be a domain that some users never browse to.... Unfortunately, there is no "bypass" mechanism in Unbound to allow some users to bypass a blocked domain. So its all or nothing at the moment... Unbound does have a Python integration, that may be used at some point, which would allow for improvements in this area.  If there are some LAN devices, or sub-nets, that you do not want to be filtered by DNSBL, you can also define a different DNS server for those devices...

                    Building a list of domains to whitelist is generally a good idea, as that can help others atleast choose which domains are causing issues, and they can decide what to whitelist...

                    "Experience is something you don't get until just after you need it."

                    Website: http://pfBlockerNG.com
                    Twitter: @BBcan177  #pfBlockerNG
                    Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                    1 Reply Last reply Reply Quote 0
                    • M
                      MrGlasspoole
                      last edited by

                      That would be the coolest thing if we could do it like uBlock: allow certain things for certain domains.
                      I also don't visit most of this sites but from this:

                      # To counter `google-analytics.com` in Peter Lowe's list
                      @@||google-analytics.com^$domain=avianca.com|jackbox.tv|newegg.com|bikstok.sonymusicshop.dk
                      
                      # Videos broken on weather.com
                      # To counter `tags.tiqcdn.com` in  MVPS
                      @@||tags.tiqcdn.com/utag/*.js$script,domain=weather.com
                      

                      newegg.com and weather.com are two popular sites.
                      And nobody wants to allow google-analytics ;D

                      How do i make a second DNS server and is it possible to point a single browser to this DNS or does it need a proxy then?

                      And sorry this brings up another question i never thought of:
                      If you block something doesn't it stress the browser? Because the browser is constantly trying to load…

                      1 Reply Last reply Reply Quote 0
                      • D
                        doktornotor Banned
                        last edited by

                        No, that's not doable with DNS and is exactly the problem here - using DNS instead of parsing the content.

                        1 Reply Last reply Reply Quote 0
                        • M
                          MrGlasspoole
                          last edited by

                          What do you mean by parsing? Without looking at the domain how can it be blocked?

                          If we can say don't allow "google-analytics.com" then it must also be possible to say block "google-analytics.com" but allow it for "newegg.com"?
                          If not how do you guys handle it?

                          As i said newegg.com and weather.com are popular.

                          1 Reply Last reply Reply Quote 0
                          • D
                            doktornotor Banned
                            last edited by

                            I mean L7. As said, this is NOT doable with DNS. It either resolves something or not, it doesn't care what code the request came from. So, it will always resolve to the same thing, and does not give a damn about whether you are browsing newegg.com or whatever.bar.baz.

                            1 Reply Last reply Reply Quote 0
                            • M
                              MrGlasspoole
                              last edited by

                              So the question remains: how do you guys handle it?

                              I can't tell others here in the house they can't use there favorite price comparison site anymore because i don't allow it ;D

                              Two years ago i tried SquidGuard and was not happy. Now i give pfBlockerNG a try and i see the same problems :(
                              I mean if so many people use it there must be an answer how they deal with the problems.

                              Edit:
                              In uBlock you have the option to disable cosmetic filtering. So if uBlock can detect what is cosmetic, then there must be a way to parse
                              all the EasyList/Fanboy lists and create a file with just the cosmetic filters.
                              Creating that file and then load it in uBlock in the browser would allow to block ads and bad domains in the whole LAN + having the cosmetics in the browser.

                              1 Reply Last reply Reply Quote 0
                              • M
                                MrGlasspoole
                                last edited by

                                Ok, maybe it helps somebody. I made a list.

                                –---------------
                                uBlock filters

                                uBlock filters:
                                https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/filters.txt
                                Cosmetic filters!

                                uBlock filters – Badware risks:
                                https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/badware.txt
                                Cosmetic filters!

                                uBlock filters – Privacy:
                                https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/privacy.txt
                                Cosmetic filters!

                                uBlock filters – Unbreak:
                                https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/unbreak.txt
                                Cosmetic filters!


                                With Rules

                                EasyList:
                                https://easylist.to/easylist/easyprivacy.txt
                                Cosmetic filters!

                                EasyList without element hiding rules:
                                https://easylist-downloads.adblockplus.org/easylist_noelemhide.txt
                                Cosmetic filters!

                                EasyPrivacy:
                                https://easylist.to/easylist/easyprivacy.txt

                                Fanboy Complete List (includes Easylist, EasyPrivacy, Fanboy Enhanced Trackers List):
                                https://secure.fanboy.co.nz/r/fanboy-complete.txt
                                Cosmetic filters!

                                Fanboy+Easylist-Merged Ultimate List (includes EasyList, EasyPrivacy, Fanboy Enhanced Trackers List & Annoyances List & Social List & Anti-Cookie Filters):
                                https://www.fanboy.co.nz/r/fanboy-ultimate.txt
                                Cosmetic filters!

                                Fanboy Annoyances List (includes Fanboy-Social List & Anti-Cookie Filters):
                                https://easylist-downloads.adblockplus.org/fanboy-annoyance.txt
                                Cosmetic filters!

                                Fanboy Enhanced Trackers List:
                                https://secure.fanboy.co.nz/enhancedstats.txt
                                Cosmetic filters!

                                Fanboy Anti-Facebook Filters (Anti-ThirdpartySocial):
                                https://www.fanboy.co.nz/fanboy-antifacebook.txt
                                Cosmetic filters!

                                Fanboy Anti-Cookie Filters (Fanboy's Cookiemonster List):
                                https://secure.fanboy.co.nz/fanboy-cookiemonster.txt
                                Cosmetic filters!

                                EasyList Germany:
                                https://easylist-downloads.adblockplus.org/easylistgermany.txt
                                Cosmetic filters!

                                Spam404:
                                https://raw.githubusercontent.com/Dawsey21/Lists/master/adblock-list.txt
                                Cosmetic filters!


                                Warning Removal

                                Adblock Warning Removal List:
                                https://easylist-downloads.adblockplus.org/antiadblockfilters.txt
                                Cosmetic filters!

                                Anti-Adblock Killer | Reek:
                                https://raw.githubusercontent.com/reek/anti-adblock-killer/master/anti-adblock-killer-filters.txt
                                Cosmetic filters!


                                Host sources with just domains

                                Malware domains:
                                http://mirror1.malwaredomains.com/files/justdomains

                                Basic tracking list by Disconnect​​​​​:
                                https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt

                                Malvertising filter list by Disconnect:
                                https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt

                                Malware domains (long-lived):
                                http://malwaredomains.lehigh.edu/files/immortal_domains.txt

                                Malware filter list by Disconnect:
                                https://s3.amazonaws.com/lists.disconnect.me/simple_malware.txt


                                Host sources with localhost in front of domains

                                Malware Domain List:
                                https://www.malwaredomainlist.com/hostslist/hosts.txt

                                MVPS HOSTS:
                                http://winhelp2002.mvps.org/hosts.txt

                                AdAway:
                                https://adaway.org/hosts.txt

                                hpHosts:
                                https://hosts-file.net/ad_servers.txt

                                Peter Lowe’s Ad and tracking server list:
                                https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext

                                1 Reply Last reply Reply Quote 0
                                • M
                                  MrGlasspoole
                                  last edited by

                                  To have the cosmetic filters i have enabled in uBlock:

                                  uBlock filters
                                  uBlock filters – Badware risks
                                  uBlock filters – Privacy
                                  uBlock filters – Unbreak
                                  Adblock Warning Removal List
                                  Anti-Adblock Killer | Reek
                                  EasyList
                                  Fanboy Enhanced Trackers List
                                  Spam404
                                  Fanboy Anti-Facebook Filters (Anti-ThirdpartySocial in uBlock)
                                  Fanboy Annoyances List (includes Fanboy Social List & Anti-Cookie Filters)
                                  EasyList "your language"

                                  In pfBlockerNG i have all the host sources + EasyList without element hiding rules and EasyPrivacy.

                                  So i guess this will take away some processing that was needed when all the host sources where in uBlock.

                                  The evil here is google analytics if some side does not work without it if we need to allow it for the whole LAN in pfBlockerNG.

                                  If we could have just the cosmetics in uBlock that would be cool.

                                  If you don't want the Fanboy Social List (Hide and block social content, social widgets, social scripts and social icons):
                                  You have to remove "Fanboy Annoyances List" and at by hand "Fanboy Anti-Cookie Filters"
                                  But this also removes hiding "In page Pop-Ups and other annoyances" that are in the "Fanboy Annoyances List".

                                  1 Reply Last reply Reply Quote 0
                                  • ExolonE
                                    Exolon
                                    last edited by

                                    In theory you could install Privoxy on your pfSense box and setup a cron entry to download the adblock2privoxy lists, say every week?

                                    adblock2privoxy
                                    downloads page

                                    1 Reply Last reply Reply Quote 0
                                    • First post
                                      Last post
                                    Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.