DNS Blacklist, New Package! Check it out.

xa0z · Oct 5, 2009, 5:09 PM

Thanks for the comments. The next version of this script does allow users to add their own custom blacklist entries. As of right now there is no way to specify the URL that a blacklisted item is sent to, that is why it is forwarding to a Google IP. Doing the method you are requesting requires a proxy. This is being looked into for the future.

As for being able to select your own blacklist database, I have already thought of this and think it would be a great idea but right now the scripting is ~~static~~ …it was static, it is dynamic now... to use preset categories/directories for the database. In the near future I would like to have this set to load the categories from a variable after reading the available directories containing the blacklists.

In a beta version of DNS Blacklist we had two separate categories… "Porn" and "Adult". That blacklist database is not what we're using right now so the categories are different and now contained under the same category. The configuration page still lists the two categories in the NOTE because I forgot to remove it, just a self error.

We're going to release a "fixed" version here soon, but a major release with additions will be a little later.

cybrsrfr · Oct 5, 2009, 11:23 PM

Did some improvements to the package today.
1. wording changes.
2. uninstall has been fixed
3. all categories can be deselected.

You can edit categories by using pfSense's command page:
Diagnostics -> Command

ls /usr/local/www/packages/dnsblacklist/blacklists

Lets say you wanted to edit the 'adult' list, you can download the list using:
File to download -> /usr/local/www/packages/dnsblacklist/blacklists/adult/domains

You would then edit the file using an editor that deals well with large files.
If you are on windows don't use the standard notepad.exe instead use pspad or notepad++

To upload the file from the command page use:
File to upload

The file will be uploaded to the /tmp directory.

Then use 'Execute Shell command' to cp the file to the correct directory.
cp /tmp/domains /usr/local/www/packages/dnsblacklist/blacklists/adult/domains

If you follow the folder structure consistently you can add additional categories.
You also would need to carefully edit the /usr/local/www/packages/dnsblacklist/blacklists/global_usage file which list the categories and provides the description for the categories. You can edit the global_usage file using Diagnostics -> Edit File

xa0z · Oct 6, 2009, 12:29 AM

Tomorrow I will begin working on our own Blacklist Database. It will work in the same manor as current… We will have a blacklist directory, and within that directory we will have a directory with the respective category name, and then within that directory we have the "domains" file which contains the domains that will be added to the active blacklist when that category is selected.

Today mcrane did a great job in updating the code to make sure everything was working as normal as we can. Remember we're still in BETA so give it some time, make your requests for addons, report any bugs you might see and we will do our best to make sure things go as smooth as possible.

As for right now the only concern I have is making sure we can get a really good, clean, host-database for the blacklisting.

cybrsrfr · Oct 6, 2009, 12:37 AM

The current list comes from here:
http://cri.univ-tlse1.fr/blacklists/index_en.php

Its just not as large as xa0z would like.

xa0z · Oct 6, 2009, 1:35 AM

The size isn't what bothers me… I think that a proper database should contain categories that are more limited to specific matching items, and better defined categories at that.

That was the whole point of me starting this project.

cybrsrfr · Oct 6, 2009, 7:34 AM

Here is the original blacklist that was going to be used.
http://urlblacklist.com/

I didn't use it in the final version of the package because I didn't feel that redistribution of that list was the ethical thing to do when the website that provides the list is trying to sell it. They provide the entire list for download and payment is on an honor basis as stated on their website. Their intent is really to sell it for a subscription. I did not notice this when I started building this package. When I noticed they were trying to sell the list they had compiled I could not with good conscious redistribute urlblacklist.com's list. If someone wants to use that list and pay for the subscription they can do so by editing the domain lists and adding the categories as I described in an earlier post.
http://urlblacklist.com/?sec=subscribe

Here is a few more choices of DNS blacklists.
http://www.squidguard.org/blacklists.html

Anyone can edit the list that the package comes with. In order for an alternative customized list to be used in the DNS blacklist package it must be compiled ethically and legally or I will not commit it to the package. For example it wouldn't be ethical to use lists from urlblacklist.com or other lists that don't offer the list in freely. Unless you obtain permission directly from them to use the list for such a purpose.

The list from http://cri.univ-tlse1.fr/blacklists/index_en.php says on their website 'can be used with many commercial or free software' therefore it is ethical to use their list for the DNS Blacklist package.

xa0z · Oct 6, 2009, 2:19 PM

Hmm, so I guess there isn't a need for me to continue on with the project. It is just pointless for me to try and do one thing and then have it all changed to not be the way it was when I started it.

I already said I was going to build my own database list but what's the point, I don't need to supply something that anyone can just grab from all over the net to use.

I'm glad for the help I got when I started writing the project but if I am being phased out, which it so seems I am, then I guess my input, or work is useless.

Supermule · Oct 6, 2009, 2:28 PM

If this isn't supported and maintained, pls. inform me how to delete it completely! It wont go away…..any way I do it.

xa0z · Oct 6, 2009, 3:10 PM

Okay, what you need to do is "Remove" the package like normal… then Re-Install it like normal. Then the new version has the fixes to take care of the problems you're having. You can then remove it to completely remove it, or you can keep using it.

My only problem is that this is starting to move a direction in which I don't seem to have anymore say so in the project. I'm not saying anything bad, and I'm not about to put mcrane down, he's a great guy and has helped me a great deal with stuff, but I just wish the project I started, and wrote part of... I had some control over.

Supermule · Oct 6, 2009, 3:12 PM

That is quite understandable…..:)

You have my support for the project, but it seems that the way the list is generated and the source of the data, is the big hurdle???

xa0z · Oct 6, 2009, 3:31 PM

I want to compile my own database. I want to get it as simple as possible where specific rated categories are available for certin things. I want to remove all the IPs in the database and use only hostnames (for now) to help keep it clean. Web Browsers don't do reverse lookups so having all those extra useless items is a waste of RAM.

I will try my best to keep with the program, just want to make sure it stays on track. I mean the whole point to me starting this was to help out, and if it's not helping me, how can it help others?

Supermule · Oct 6, 2009, 3:33 PM

Exactly. :)

Keep it up! You are on the right track….IP's change. Domainnames do not as often....So it is a good argument.

cybrsrfr · Oct 6, 2009, 3:47 PM

The problem is the list. As I stated before it has to be created ethically. That is compiled from free lists where the owners give their permission.

I will not commit something to pfSense that makes me an accessory to stealing. That is why I refused to commit the package with the previous list. If I did this I would expect to get my commit authority revoked.

It is not impossible to find lists that are free for any use and supplement them, and improve them with your own domains you and others find while searching.

xa0z · Oct 6, 2009, 3:51 PM

That is exactly what I'm doing. I'm not worried about the urlBlacklists database anymore. I am just saying that I'm getting the database taken care of on my own, and it will be greatly categorizied.

Davc · Oct 7, 2009, 1:27 AM

I think this is a good project, if the DNS Blacklist is good and effectively use. Surely there will be people supported and willing to donate some subscription fee to maintain the Blacklist and the project itself in this Pfsense forum/members.

I am no expert in here, but there are people who might (already) using OpenDNS to filter, as long as this project don't run the same track as the OpenDNS or able to offer more than OpenDNS. I think there are light on this projects.

Good Work !! ;)

jan.gestre · Oct 7, 2009, 6:02 AM

Hi Davc,

I've tried DNS blacklist in one of my pfSense box but it broke the dns forwarder service, after installation the dnsmasq service stopped and can't be restarted even after repeated tries and reboot. I uninstalled the aforementioned package but I still can't start the dnsmasq service,I had to examine other working boxes to see what files have been added or changed by the DNS blacklist package, it added the dnsmasq.conf which I've deleted and my dnsmasq service finally started.

Did I missed something?

Regards,

Jan

cybrsrfr · Oct 7, 2009, 6:13 AM

When you tried to start dnsmasq. If you would have looked at the Diagnostics: System logs: System and looked for errors inr regards to dnsmasq then you should be ablt to find a description why it refused to start.

jigpe · Oct 7, 2009, 8:24 AM

Good job :)

I have Q.. How to block HTTPS? (except legit HTTPS)?

jigp

Davc · Oct 7, 2009, 4:48 PM

Mine is running fine.

1.2.3-RC2
built on Sat Jul 18 19:19:52 EDT 2009
FreeBSD 7.2-RELEASE-p2 i386

DNS Blacklist 0.2.4

Yes, after the installation of DNS Blacklist. I have to manual restart the services.

My Box run in Bridge mode, I guess yours are different in NAT mode.

Davc

xa0z · Oct 7, 2009, 5:23 PM

After installing DNS Blacklist you shouldn't be required to restart dnsmasq as nothing is edited that pertains to dnsmasq at the time. DNS Blacklist adds the dnsmasq.conf, and dnsmasq.blacklist.conf files into /usr/local/etc/. When DNS Blacklist is enabled it adds a string into dnsmasq.conf to load the dnsmasq.blacklist.conf file, then restarts dnsmasq. Any of the categories you select are entered within the dnsmasq.blacklist.conf file and that is what allows us to filter dns querys to the local server.

I am in no way out to seek any money from anyone for the blacklist database I'm putting together. I can maintain a "main blacklist" but users would be free to add their own domains that aren't already listed. Here soon I'll work on adding a custom Blacklist/Whitelist text area for you to enter your own on the fly.

If you want to block all https, you need to put a block on dport:443, that isn't associated with DNS Blacklist.