Is there any way to load this list?

guardian

Is there any way to load this list?
(Original Source: https://github.com/notracking/hosts-blocklists )

URL: https://raw.githubusercontent.com/notracking/hosts-blocklists/master/domains.txt

Sample:

################################[UPDATE]#################################
# Source: https://github.com/notracking/hosts-blocklists
# Last update: Thu Dec 27 05:36:10 CET 2018
# Filter supports ipv4 and ipv6
# Number of unique domains: 83050
#################################[INFO]##################################
# Blocks ads, fake sites, shock sites, malware hosts and trackers
# Removed verified false calls and added a large set of custom entries
# This hostfiles aim is not to break any useful websites
# Submit a issue on Github in case of any false or missing hosts
################################[INSTALL]################################
# Update your dnsmasq.conf file and add the following 2 lines:
#  conf-file=/path/to/domains.txt
#  addn-hosts=/path/to/hostnames.txt
################################[DOMAINS]################################
address=/000007.ru/0.0.0.0
address=/000007.ru/::
address=/000free.us/0.0.0.0
address=/000free.us/::
address=/000freexxx.com/0.0.0.0
address=/000freexxx.com/::

===

The default parser doesn't get anything at all.

mike69

Which device?

For pfsense, the package "pfBlockerNG" can handle those kind of lists.

guardian

@mike69 said in Is there any way to load this list?:

Which device?

For pfsense, the package "pfBlockerNG" can handle those kind of lists.

I thought it could, but the current stable version doesn't find any domains.

RonpfS

Do you "really" need that list ? Those domains are already present in other tables.

grep 000007.ru /var/db/pfblockerng/dnsbl/*.txt /var/db/pfblockerng/dnsblorig/*.orig /var/unbound/pfb_dnsbl.conf /usr/local/pkg/pfblockerng/dnsbl_tld

/var/db/pfblockerng/dnsbl/Malic_MDS_Immortal.txt:local-data: "000007.ru 60 IN A 10.10.10.1"
/var/db/pfblockerng/dnsblorig/Malic_MDS_Immortal.orig:000007.ru

grep 000free.us /var/db/pfblockerng/dnsbl/*.txt /var/db/pfblockerng/dnsblorig/*.orig /var/unbound/pfb_dnsbl.conf /usr/local/pkg/pfblockerng/dnsbl_tld

/var/db/pfblockerng/dnsbl/Malic2_Krog_BD.txt:local-data: "000free.us 60 IN A 10.10.10.1"
/var/db/pfblockerng/dnsblorig/Malic2_Krog_BD.orig:0.0.0.0 000free.us

grep 000freexxx.com /var/db/pfblockerng/dnsbl/*.txt /var/db/pfblockerng/dnsblorig/*.orig /var/unbound/pfb_dnsbl.conf /usr/local/pkg/pfblockerng/dnsbl_tld

/var/db/pfblockerng/dnsbl/Shallalist_adv.txt:local-data: "000freexxx.com 60 IN A 10.10.10.1"
/var/db/pfblockerng/dnsblorig/Shallalist_adv.orig:000freexxx.com

guardian

Thanks @ronpfs for the reply
@ronpfs said in Is there any way to load this list?:

Do you "really" need that list ? Those domains are already present in other tables.

I'm not sure... like a lot of the lists there may be a lot of dupes... I have lists with 50% or more dupes, but the ones that aren't dupes are potentially useful. Unless I can load the list I don't have any easy way to know.

RonpfS

Well there are many more domains in the list ...

guardian

@ronpfs IIRC there are about 1500 entries...

RonpfS

@guardian You could edit the file and only leave the domain names, create a table with this file as the last table in DNSBL and see how many Domain are not dupe.

guardian

@ronpfs it is a shame there isn't a hook to call a custom preparser script.

RonpfS

@guardian Write a parser and post it here

BBcan177

@guardian

Try this feed instead:
https://raw.githubusercontent.com/notracking/hosts-blocklists/master/hostnames.txt

It will have many duplicates since they post two lines per domain (ipv4/6)

guardian

@ronpfs said in Is there any way to load this list?:

@guardian Write a parser and post it here

Hi @ronpfs... How about this... clean list with no dupes or extra crap:
curl https://raw.githubusercontent.com/notracking/hosts-blocklists/master/domains.txt | sed -ne 's|address=/(.*)/0.0.0.0|\1|p'

@bbcan177 said in Is there any way to load this list?:

@guardian

Try this feed instead:
https://raw.githubusercontent.com/notracking/hosts-blocklists/master/hostnames.txt
Hi @bbcan177... this is not the same list... I picked 2 or 3 from the original list and searched, and they weren't there. I actually want to use both lists.

It will have many duplicates since they post two lines per domain (ipv4/6)
True enough... but does it really matter... you remove dupes anyway

guardian

This post is deleted!

RonpfS

This post is deleted!

guardian

@bbcan177 said in Is there any way to load this list?:

@guardian

Try this feed instead:
https://raw.githubusercontent.com/notracking/hosts-blocklists/master/hostnames.txt

is NOT the same list as:
https://raw.githubusercontent.com/notracking/hosts-blocklists/master/domains.txt

I did a compare of both lists (I cleaned up the lists as required using sed to strip everything but the domain, and then used sort -u to remove any dupes). I then used comp to check for overlaps, and there were NONE. So both lists are very different.

It will have many duplicates since they post two lines per domain (ipv4/6)
Do duplicates really matter? I thought pfblocker removed dupes? I guess it wastes processing time, but in my application that isn't a problem.

@ronpfs Here's is the corrected filter - it produces a totally clean list with no dupes or extra crap:

curl https://raw.githubusercontent.com/notracking/hosts-blocklists/master/domains.txt | sed -nr 's|address=/(.*)/0.0.0.0|\1|p'

BTW @bbcan177 any plan to add the ability to do some sort of custom preprocessing on a list (such as I have illustrated above)?

RonpfS

Install Cron package and then you could run a cron shell job 15 min before PfblockerNG Cron Update that download and process the list to a local file to be ready for processing.