Is there any way to load this list?

guardian

Thanks @ronpfs for the reply
@ronpfs said in Is there any way to load this list?:

Do you "really" need that list ? Those domains are already present in other tables.

I'm not sure... like a lot of the lists there may be a lot of dupes... I have lists with 50% or more dupes, but the ones that aren't dupes are potentially useful. Unless I can load the list I don't have any easy way to know.

RonpfS

Well there are many more domains in the list ...

guardian

@ronpfs IIRC there are about 1500 entries...

RonpfS

@guardian You could edit the file and only leave the domain names, create a table with this file as the last table in DNSBL and see how many Domain are not dupe.

guardian

@ronpfs it is a shame there isn't a hook to call a custom preparser script.

RonpfS

@guardian Write a parser and post it here

BBcan177

@guardian

Try this feed instead:
https://raw.githubusercontent.com/notracking/hosts-blocklists/master/hostnames.txt

It will have many duplicates since they post two lines per domain (ipv4/6)

guardian

@ronpfs said in Is there any way to load this list?:

@guardian Write a parser and post it here

Hi @ronpfs... How about this... clean list with no dupes or extra crap:
curl https://raw.githubusercontent.com/notracking/hosts-blocklists/master/domains.txt | sed -ne 's|address=/(.*)/0.0.0.0|\1|p'

@bbcan177 said in Is there any way to load this list?:

@guardian

Try this feed instead:
https://raw.githubusercontent.com/notracking/hosts-blocklists/master/hostnames.txt
Hi @bbcan177... this is not the same list... I picked 2 or 3 from the original list and searched, and they weren't there. I actually want to use both lists.

It will have many duplicates since they post two lines per domain (ipv4/6)
True enough... but does it really matter... you remove dupes anyway

guardian

This post is deleted!

RonpfS

This post is deleted!

guardian

@bbcan177 said in Is there any way to load this list?:

@guardian

Try this feed instead:
https://raw.githubusercontent.com/notracking/hosts-blocklists/master/hostnames.txt

is NOT the same list as:
https://raw.githubusercontent.com/notracking/hosts-blocklists/master/domains.txt

I did a compare of both lists (I cleaned up the lists as required using sed to strip everything but the domain, and then used sort -u to remove any dupes). I then used comp to check for overlaps, and there were NONE. So both lists are very different.

It will have many duplicates since they post two lines per domain (ipv4/6)
Do duplicates really matter? I thought pfblocker removed dupes? I guess it wastes processing time, but in my application that isn't a problem.

@ronpfs Here's is the corrected filter - it produces a totally clean list with no dupes or extra crap:

curl https://raw.githubusercontent.com/notracking/hosts-blocklists/master/domains.txt | sed -nr 's|address=/(.*)/0.0.0.0|\1|p'

BTW @bbcan177 any plan to add the ability to do some sort of custom preprocessing on a list (such as I have illustrated above)?

RonpfS

Install Cron package and then you could run a cron shell job 15 min before PfblockerNG Cron Update that download and process the list to a local file to be ready for processing.