Whitelist for uBlock?

MrGlasspoole · Feb 20, 2017, 7:06 PM

Sorry if i don't read all the pfBlockerNG threads.

I did installed Pi-hole yesterday and now found pfBlockerNG.

With Pi-hole i first did wonder i see ads in YouTube videos and also allot of other ads on websites.
I'm a uBlock Origin user in browsers and AdAway on Android and nowhere i have YouTube ads or other stuff.

From reading Pi-hole can't block elements and does not work on iframes and does not do the cosmetic stuff that uBlock does.
How is that with pfBlockerNG?

How does pfBlockerNG handle the adblock formatted lists - removing all the rules and just leaving the domains?

My idea would be doing some script that works together with uBlock.
From looking at the 4 uBlock txt files it seems like they override/remove some rules that are in the EasyList and other lists?

I'm not a bash or perl guy and don't know how to do it, but maybe:
1. Download all the files
2. Remove whats in our whitelist
3. Apply/check the uBlock filters
4. Create a file with the rules that only can be done by uBlock
5. Create a file that is cleaned up from duplicates and the stuff that is done by uBlock (used then by pfBlockerNG)

Then disabling every filter in uBlock and creating one that points to the one that was created by the script.
So all devices have everything and browsers with uBlock can do there cosmetic stuff and have less work to do.
Modify message

BBcan177 · Feb 20, 2017, 8:12 PM

@MrGlasspoole:

I did installed Pi-hole yesterday and now found pfBlockerNG.

Great news :)

From reading Pi-hole can't block elements and does not work on iframes and does not do the cosmetic stuff that uBlock does. How is that with pfBlockerNG?

Any DNS filter will only block based on Domain name… so DNS Filters cannot filter like UBlock origin which modifies the page elements etc...

pfBlockerNG parses the EasyList feeds, and collects just the domains that can be blocked... The rest are discarded..

If you still see an AD in a browser, right-click on it, and select "inspect"... Then if the element has a Domain name, you can add that domain to a customlist in DNSBL to block it... otherwise, it can't be filtered with a DNS Filter...

MrGlasspoole · Feb 20, 2017, 11:33 PM

Hm, my idea is to fetch the uBlock filters:

https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/filters.txt
https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/badware.txt
https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/privacy.txt
https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/unbreak.txt

Then compare this filters with the other lists (look for rules that need to be applied on domains in this list)
and remove them and create a file for DNS.
Create another (uBlock.txt) file with all the uBlock filters plus the stuff that was removed from the other lists.
So stuff where the uBlock filters need to do its think go to your PC.

I mean how much sense does it make to parse the EasyList feeds and collects just the domains if the rules are not working.
I guess if you have all this:

http://mirror1.malwaredomains.com/files/justdomains
https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt
http://malwaredomains.lehigh.edu/files/immortal_domains.txt
https://s3.amazonaws.com/lists.disconnect.me/simple_malware.txt
https://www.malwaredomainlist.com/hostslist/hosts.txt
http://winhelp2002.mvps.org/hosts.txt
https://adaway.org/hosts.txt
https://hosts-file.net/ad_servers.txt
https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=1&mimetype=plaintext

then there are not much domains missing that the EasyLists would have.

BBcan177 · Feb 21, 2017, 12:11 AM

pfBlockerNG already has an EasyList tab… No need to convert anything....

For these UBlock links... Seems only the first link (filters.txt) is somewhat usable... Typically look for lines that start with "||"... and then review if those domains are worth blocking...

See the follow cmds... Found approx 129 domains, Most of these domains are already listed in the other DNSBL Feeds available...

fetch -o /tmp/file "https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/filters.txt"
grep "^||" /tmp/file | sed -e 's/$.*//' -e 's/\/.*//' -e 's/\^.*//' | tr -d '|' | grep -v '*' | sort | uniq

FYI, DNSBL has deduplication.

wcrowder · Feb 21, 2017, 3:17 AM

BB' You work to hard… :) BTW, How you doing? Great work, keep it up! :)

BBcan177 · Feb 21, 2017, 3:54 AM

@wcrowder:

BB' You work to hard… :) BTW, How you doing? Great work, keep it up! :)

BC!! Where u been. Welcome back. 8) 8)
Next version will be even better. Just waiting for your final beta test approval :)

MrGlasspoole · Feb 21, 2017, 10:25 AM

Yes pfBlockerNG has an EasyList tab and you wrote:
@BBcan177:

pfBlockerNG parses the EasyList feeds, and collects just the domains that can be blocked… The rest are discarded..

I guess we talk past each other :)
If you look at the uBlock filters, there are not there to at more domains.
They fix stuff that are broken.
For example the first rule in: https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/unbreak.txt
Peter Lowe's list blocks google analytics so on some sites stuff does not work.

So the idea is to take out google analytics from all the other lists and let it pass through to the PCs.
Then on the PC uBlock can handle it.

BBcan177 · Feb 21, 2017, 3:02 PM

@MrGlasspoole:

So the idea is to take out google analytics from all the other lists and let it pass through to the PCs.
Then on the PC uBlock can handle it.

Should have defined this thread as "UBlock whitelist" :)

I am not sure of the syntax used for the UBlock feeds… But if you parse these files, and collect the Domains to whitelist, you can add them to the DNSBL Whitelist customlist, and run a "Force Reload - DNSBL" for it to take effect... You can prefix a domain with a "dot" and it will also whitelist any sub-domains.

Everyones whitelist is going to be different, some of these exceptions from UBlock, are for a single website that might have issues with blocking a domain... It might be a domain that some users never browse to.... Unfortunately, there is no "bypass" mechanism in Unbound to allow some users to bypass a blocked domain. So its all or nothing at the moment... Unbound does have a Python integration, that may be used at some point, which would allow for improvements in this area. If there are some LAN devices, or sub-nets, that you do not want to be filtered by DNSBL, you can also define a different DNS server for those devices...

Building a list of domains to whitelist is generally a good idea, as that can help others atleast choose which domains are causing issues, and they can decide what to whitelist...

MrGlasspoole · Feb 23, 2017, 11:41 AM

That would be the coolest thing if we could do it like uBlock: allow certain things for certain domains.
I also don't visit most of this sites but from this:

# To counter `google-analytics.com` in Peter Lowe's list
@@||google-analytics.com^$domain=avianca.com|jackbox.tv|newegg.com|bikstok.sonymusicshop.dk

# Videos broken on weather.com
# To counter `tags.tiqcdn.com` in  MVPS
@@||tags.tiqcdn.com/utag/*.js$script,domain=weather.com

newegg.com and weather.com are two popular sites.
And nobody wants to allow google-analytics ;D

How do i make a second DNS server and is it possible to point a single browser to this DNS or does it need a proxy then?

And sorry this brings up another question i never thought of:
If you block something doesn't it stress the browser? Because the browser is constantly trying to load…

doktornotor · Feb 23, 2017, 12:05 PM

No, that's not doable with DNS and is exactly the problem here - using DNS instead of parsing the content.

MrGlasspoole · Feb 23, 2017, 12:54 PM

What do you mean by parsing? Without looking at the domain how can it be blocked?

If we can say don't allow "google-analytics.com" then it must also be possible to say block "google-analytics.com" but allow it for "newegg.com"?
If not how do you guys handle it?

As i said newegg.com and weather.com are popular.

doktornotor · Feb 23, 2017, 1:14 PM

I mean L7. As said, this is NOT doable with DNS. It either resolves something or not, it doesn't care what code the request came from. So, it will always resolve to the same thing, and does not give a damn about whether you are browsing newegg.com or whatever.bar.baz.

MrGlasspoole · Feb 23, 2017, 6:33 PM

So the question remains: how do you guys handle it?

I can't tell others here in the house they can't use there favorite price comparison site anymore because i don't allow it ;D

Two years ago i tried SquidGuard and was not happy. Now i give pfBlockerNG a try and i see the same problems :(
I mean if so many people use it there must be an answer how they deal with the problems.

Edit:
In uBlock you have the option to disable cosmetic filtering. So if uBlock can detect what is cosmetic, then there must be a way to parse
all the EasyList/Fanboy lists and create a file with just the cosmetic filters.
Creating that file and then load it in uBlock in the browser would allow to block ads and bad domains in the whole LAN + having the cosmetics in the browser.

MrGlasspoole · Feb 24, 2017, 12:59 PM

Ok, maybe it helps somebody. I made a list.

–---------------
uBlock filters

uBlock filters:
https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/filters.txt
Cosmetic filters!

uBlock filters – Badware risks:
https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/badware.txt
Cosmetic filters!

uBlock filters – Privacy:
https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/privacy.txt
Cosmetic filters!

uBlock filters – Unbreak:
https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/unbreak.txt
Cosmetic filters!

With Rules

EasyList:
https://easylist.to/easylist/easyprivacy.txt
Cosmetic filters!

EasyList without element hiding rules:
https://easylist-downloads.adblockplus.org/easylist_noelemhide.txt
Cosmetic filters!

EasyPrivacy:
https://easylist.to/easylist/easyprivacy.txt

Fanboy Complete List (includes Easylist, EasyPrivacy, Fanboy Enhanced Trackers List):
https://secure.fanboy.co.nz/r/fanboy-complete.txt
Cosmetic filters!

Fanboy+Easylist-Merged Ultimate List (includes EasyList, EasyPrivacy, Fanboy Enhanced Trackers List & Annoyances List & Social List & Anti-Cookie Filters):
https://www.fanboy.co.nz/r/fanboy-ultimate.txt
Cosmetic filters!

Fanboy Annoyances List (includes Fanboy-Social List & Anti-Cookie Filters):
https://easylist-downloads.adblockplus.org/fanboy-annoyance.txt
Cosmetic filters!

Fanboy Enhanced Trackers List:
https://secure.fanboy.co.nz/enhancedstats.txt
Cosmetic filters!

Fanboy Anti-Facebook Filters (Anti-ThirdpartySocial):
https://www.fanboy.co.nz/fanboy-antifacebook.txt
Cosmetic filters!

Fanboy Anti-Cookie Filters (Fanboy's Cookiemonster List):
https://secure.fanboy.co.nz/fanboy-cookiemonster.txt
Cosmetic filters!

EasyList Germany:
https://easylist-downloads.adblockplus.org/easylistgermany.txt
Cosmetic filters!

Spam404:
https://raw.githubusercontent.com/Dawsey21/Lists/master/adblock-list.txt
Cosmetic filters!

Warning Removal

Adblock Warning Removal List:
https://easylist-downloads.adblockplus.org/antiadblockfilters.txt
Cosmetic filters!

Anti-Adblock Killer | Reek:
https://raw.githubusercontent.com/reek/anti-adblock-killer/master/anti-adblock-killer-filters.txt
Cosmetic filters!

Host sources with just domains

Malware domains:
http://mirror1.malwaredomains.com/files/justdomains

Basic tracking list by Disconnect:
https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt

Malvertising filter list by Disconnect:
https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt

Malware domains (long-lived):
http://malwaredomains.lehigh.edu/files/immortal_domains.txt

Malware filter list by Disconnect:
https://s3.amazonaws.com/lists.disconnect.me/simple_malware.txt

Host sources with localhost in front of domains

Malware Domain List:
https://www.malwaredomainlist.com/hostslist/hosts.txt

MVPS HOSTS:
http://winhelp2002.mvps.org/hosts.txt

AdAway:
https://adaway.org/hosts.txt

hpHosts:
https://hosts-file.net/ad_servers.txt

Peter Lowe’s Ad and tracking server list:
https://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&showintro=0&mimetype=plaintext

MrGlasspoole · Feb 24, 2017, 1:21 PM

To have the cosmetic filters i have enabled in uBlock:

uBlock filters
uBlock filters – Badware risks
uBlock filters – Privacy
uBlock filters – Unbreak
Adblock Warning Removal List
Anti-Adblock Killer | Reek
EasyList
Fanboy Enhanced Trackers List
Spam404
Fanboy Anti-Facebook Filters (Anti-ThirdpartySocial in uBlock)
Fanboy Annoyances List (includes Fanboy Social List & Anti-Cookie Filters)
EasyList "your language"

In pfBlockerNG i have all the host sources + EasyList without element hiding rules and EasyPrivacy.

So i guess this will take away some processing that was needed when all the host sources where in uBlock.

The evil here is google analytics if some side does not work without it if we need to allow it for the whole LAN in pfBlockerNG.

If we could have just the cosmetics in uBlock that would be cool.

If you don't want the Fanboy Social List (Hide and block social content, social widgets, social scripts and social icons):
You have to remove "Fanboy Annoyances List" and at by hand "Fanboy Anti-Cookie Filters"
But this also removes hiding "In page Pop-Ups and other annoyances" that are in the "Fanboy Annoyances List".

Exolon · Feb 24, 2017, 8:19 PM

In theory you could install Privoxy on your pfSense box and setup a cron entry to download the adblock2privoxy lists, say every week?

adblock2privoxy
downloads page