Is there any way to load this list?
-
Is there any way to load this list?
(Original Source: https://github.com/notracking/hosts-blocklists )URL: https://raw.githubusercontent.com/notracking/hosts-blocklists/master/domains.txt
Sample:
################################[UPDATE]################################# # Source: https://github.com/notracking/hosts-blocklists # Last update: Thu Dec 27 05:36:10 CET 2018 # Filter supports ipv4 and ipv6 # Number of unique domains: 83050 #################################[INFO]################################## # Blocks ads, fake sites, shock sites, malware hosts and trackers # Removed verified false calls and added a large set of custom entries # This hostfiles aim is not to break any useful websites # Submit a issue on Github in case of any false or missing hosts ################################[INSTALL]################################ # Update your dnsmasq.conf file and add the following 2 lines: # conf-file=/path/to/domains.txt # addn-hosts=/path/to/hostnames.txt ################################[DOMAINS]################################ address=/000007.ru/0.0.0.0 address=/000007.ru/:: address=/000free.us/0.0.0.0 address=/000free.us/:: address=/000freexxx.com/0.0.0.0 address=/000freexxx.com/::===
The default parser doesn't get anything at all.
-
Which device?
For pfsense, the package "pfBlockerNG" can handle those kind of lists.
-
@mike69 said in Is there any way to load this list?:
Which device?
For pfsense, the package "pfBlockerNG" can handle those kind of lists.
I thought it could, but the current stable version doesn't find any domains.
-
Do you "really" need that list ? Those domains are already present in other tables.
grep 000007.ru /var/db/pfblockerng/dnsbl/*.txt /var/db/pfblockerng/dnsblorig/*.orig /var/unbound/pfb_dnsbl.conf /usr/local/pkg/pfblockerng/dnsbl_tld /var/db/pfblockerng/dnsbl/Malic_MDS_Immortal.txt:local-data: "000007.ru 60 IN A 10.10.10.1" /var/db/pfblockerng/dnsblorig/Malic_MDS_Immortal.orig:000007.ru grep 000free.us /var/db/pfblockerng/dnsbl/*.txt /var/db/pfblockerng/dnsblorig/*.orig /var/unbound/pfb_dnsbl.conf /usr/local/pkg/pfblockerng/dnsbl_tld /var/db/pfblockerng/dnsbl/Malic2_Krog_BD.txt:local-data: "000free.us 60 IN A 10.10.10.1" /var/db/pfblockerng/dnsblorig/Malic2_Krog_BD.orig:0.0.0.0 000free.us grep 000freexxx.com /var/db/pfblockerng/dnsbl/*.txt /var/db/pfblockerng/dnsblorig/*.orig /var/unbound/pfb_dnsbl.conf /usr/local/pkg/pfblockerng/dnsbl_tld /var/db/pfblockerng/dnsbl/Shallalist_adv.txt:local-data: "000freexxx.com 60 IN A 10.10.10.1" /var/db/pfblockerng/dnsblorig/Shallalist_adv.orig:000freexxx.com
-
Thanks @ronpfs for the reply
@ronpfs said in Is there any way to load this list?:Do you "really" need that list ? Those domains are already present in other tables.
I'm not sure... like a lot of the lists there may be a lot of dupes... I have lists with 50% or more dupes, but the ones that aren't dupes are potentially useful. Unless I can load the list I don't have any easy way to know.
-
Well there are many more domains in the list ...
-
@ronpfs IIRC there are about 1500 entries...
-
@guardian You could edit the file and only leave the domain names, create a table with this file as the last table in DNSBL and see how many Domain are not dupe.
-
@ronpfs it is a shame there isn't a hook to call a custom preparser script.
-
@guardian Write a parser and post it here
-
Try this feed instead:
https://raw.githubusercontent.com/notracking/hosts-blocklists/master/hostnames.txtIt will have many duplicates since they post two lines per domain (ipv4/6)
-
@ronpfs said in Is there any way to load this list?:
@guardian Write a parser and post it here
Hi @ronpfs... How about this... clean list with no dupes or extra crap:
curl https://raw.githubusercontent.com/notracking/hosts-blocklists/master/domains.txt | sed -ne 's|address=/(.*)/0.0.0.0|\1|p'@bbcan177 said in Is there any way to load this list?:
Try this feed instead:
https://raw.githubusercontent.com/notracking/hosts-blocklists/master/hostnames.txt
Hi @bbcan177... this is not the same list... I picked 2 or 3 from the original list and searched, and they weren't there. I actually want to use both lists.It will have many duplicates since they post two lines per domain (ipv4/6)
True enough... but does it really matter... you remove dupes anyway
-
This post is deleted!
-
This post is deleted!
-
@bbcan177 said in Is there any way to load this list?:
Try this feed instead:
https://raw.githubusercontent.com/notracking/hosts-blocklists/master/hostnames.txtis NOT the same list as:
https://raw.githubusercontent.com/notracking/hosts-blocklists/master/domains.txtI did a compare of both lists (I cleaned up the lists as required using sed to strip everything but the domain, and then used sort -u to remove any dupes). I then used comp to check for overlaps, and there were NONE. So both lists are very different.
It will have many duplicates since they post two lines per domain (ipv4/6)
Do duplicates really matter? I thought pfblocker removed dupes? I guess it wastes processing time, but in my application that isn't a problem.@ronpfs Here's is the corrected filter - it produces a totally clean list with no dupes or extra crap:
curl https://raw.githubusercontent.com/notracking/hosts-blocklists/master/domains.txt | sed -nr 's|address=/(.*)/0.0.0.0|\1|p'BTW @bbcan177 any plan to add the ability to do some sort of custom preprocessing on a list (such as I have illustrated above)?
-
Install Cron package and then you could run a cron shell job 15 min before PfblockerNG Cron Update that download and process the list to a local file to be ready for processing.
