DNS Blacklist, New Package! Check it out.

[xa0z]

Okay, so yesterday we released DNS Blacklist which is now added into the Packages list for pfSense.

The idea of the package is to use a freely available url/host/ip blacklist database to filter out sites you do not want visited from your network.  Using every category listed in the DNS Blacklist will require roughly 256mb free memory on its own.  This works by using dnsmasq and adding the categories you select into the dnsmasq config to reject the unwanted host and forward to Google.  This is somewhat similar to OpenDNS in a sense, and does work really well.

I started this project a little over a month ago and with the help of mcrane I was able to get it finished and submitted.  Right now you're limited to the hosts within the database but if the project continues on I will set it to allow custom entries from the DNS Blacklist control panel, and the option to edit the categories yourself.

Any and all feedback is welcome, or you can find me in ##pfsense or ##pfsense-dnsblacklist or just message me on freenode.   And a big thanks to mcrane for his help, and putting up with me :P

–-

Edit:  I do realize that since combining the porn/adult categories I forgot to edit the text on the control panel.  Also you are not required to restart dnsmasq manually like it says in the "savemsg"... That was fixed by adding services_dnsmasq_configure() into the source.  And one last thing I can think of… You do not need to hit "APPLY" in the savemsg dialog to make the changes stick.  I'll get all of this taken care of later today.   Just in a hurry to get the package released we didn't get the control panel cleaned up properly.

[mikesamo]

thank you for the package I will check this =)

[Supermule]

Cannot be disabled and uninstalled in webgui….

[xa0z]

@Supermule:

Cannot be disabled and uninstalled in webgui….

I noticed this earlier after the package was submitted.  Not sure what happened between the version I submitted and the version being supplied but it does seem to have an issue here and there that I'll take care of later today.  For a temporary fix, you can disable "DNS Blacklist", but you must leave a category selected while the "Enabled" checkbox is deselected.

I don't have access to the "DNS Blacklist" repo on pfSense to fix things so I have to wait for mcrane to come online, then submit the changes to him and then he will submit them to pfSense.

[Supermule]

Nice….Not superurgent, but a small feedback to you.

:)

I suppose one can update the package via an uninstall....

[jigpe]

Thank you xa0z! Great job :)

Thanks for the heads up!

jigp

[keeper 0]

thanks, more power to your team guys  :)

[vreid473]

Hello,

Thanks for implementing this package.  I'm trying it out , and I think it has a nice feature set and is super easy to configure.  I have a few suggestions for additions:

1.  The ability to add or remove domains and url's to a whitelist and to a blacklist.
2.  The ability to specify the web page url to which users are redirected for each category.
3.  The ability to specify the blacklist to use.  Or, alternatively, some documentation about what's on the existing blocklist.

I noticed that the dns blacklist package mentions porn in the notes at the top of the page, but there appears to be not porn category (at least on the embedded machine on which I'm doing my initial testing).  Is this an error or a feature to reduce the footprint on embedded devices?

I appreciate the time and effort that you have placed into creating this package.  I think it definitely fits the need for an additional filtering plugin for PFSense.  :)

Vaughn Reid III

[xa0z]

Thanks for the comments.   The next version of this script does allow users to add their own custom blacklist entries.  As of right now there is no way to specify the URL that a blacklisted item is sent to, that is why it is forwarding to a Google IP.  Doing the method you are requesting requires a proxy.  This is being looked into for the future.

As for being able to select your own blacklist database, I have already thought of this and think it would be a great idea but right now the scripting is static …it was static, it is dynamic now... to use preset categories/directories for the database.  In the near future I would like to have this set to load the categories from a variable after reading the available directories containing the blacklists.

In a beta version of DNS Blacklist we had two separate categories… "Porn" and "Adult".  That blacklist database is not what we're using right now so the categories are different and now contained under the same category.   The configuration page still lists the two categories in the NOTE because I forgot to remove it, just a self error.

We're going to release a "fixed" version here soon, but a major release with additions will be a little later.

[cybrsrfr]

Did some improvements to the package today.
1. wording changes.
2. uninstall has been fixed
3. all categories can be deselected.

You can edit categories by using pfSense's command page:
Diagnostics -> Command

ls /usr/local/www/packages/dnsblacklist/blacklists

Lets say you wanted to edit the 'adult' list, you can download the list using:
File to download -> /usr/local/www/packages/dnsblacklist/blacklists/adult/domains

You would then edit the file using an editor that deals well with large files.
If you are on windows don't use the standard notepad.exe instead use pspad or notepad++

To upload the file from the command page use:
File to upload

The file will be uploaded to the /tmp directory.

Then use 'Execute Shell command' to cp the file to the correct directory.
cp /tmp/domains /usr/local/www/packages/dnsblacklist/blacklists/adult/domains

If you follow the folder structure consistently you can add additional categories.
You also would need to carefully edit the /usr/local/www/packages/dnsblacklist/blacklists/global_usage file which list the categories and provides the description for the categories. You can edit the global_usage file using Diagnostics -> Edit File

[xa0z]

Tomorrow I will begin working on our own Blacklist Database.  It will work in the same manor as current…  We will have a blacklist directory, and within that directory we will have a directory with the respective category name, and then within that directory we have the "domains" file which contains the domains that will be added to the active blacklist when that category is selected.

Today mcrane did a great job in updating the code to make sure everything was working as normal as we can.  Remember we're still in BETA so give it some time, make your requests for addons, report any bugs you might see and we will do our best to make sure things go as smooth as possible.

As for right now the only concern I have is making sure we can get a really good, clean, host-database for the blacklisting.

[cybrsrfr]

The current list comes from here:
http://cri.univ-tlse1.fr/blacklists/index_en.php

Its just not as large as xa0z would like.

[xa0z]

The size isn't what bothers me… I think that a proper database should contain categories that are more limited to specific matching items, and better defined categories at that.

That was the whole point of me starting this project.

[cybrsrfr]

Here is the original blacklist that was going to be used.
http://urlblacklist.com/

I didn't use it in the final version of the package because I didn't feel that redistribution of that list was the ethical thing to do when the website that provides the list is trying to sell it. They provide the entire list for download and payment is on an honor basis as stated on their website. Their intent is really to sell it for a subscription. I did not notice this when I started building this package. When I noticed they were trying to sell the list they had compiled I could not with good conscious redistribute urlblacklist.com's list. If someone wants to use that list and pay for the subscription they can do so by editing the domain lists and adding the categories as I described in an earlier post.
http://urlblacklist.com/?sec=subscribe

Here is a few more choices of DNS blacklists.
http://www.squidguard.org/blacklists.html

Anyone can edit the list that the package comes with. In order for an alternative customized list to be used in the DNS blacklist package it must be compiled ethically and legally or I will not commit it to the package. For example it wouldn't be ethical to use lists from urlblacklist.com or other lists that don't offer the list in freely. Unless you obtain permission directly from them to use the list for such a purpose.

The list from http://cri.univ-tlse1.fr/blacklists/index_en.php says on their website 'can be used with many commercial or free software' therefore it is ethical to use their list for the DNS Blacklist package.

[xa0z]

Hmm, so I guess there isn't a need for me to continue on with the project.  It is just pointless for me to try and do one thing and then have it all changed to not be the way it was when I started it.

I already said I was going to build my own database list but what's the point, I don't need to supply something that anyone can just grab from all over the net to use.

I'm glad for the help I got when I started writing the project but if I am being phased out, which it so seems I am, then I guess my input, or work is useless.

[Supermule]

If this isn't supported and maintained, pls. inform me how to delete it completely! It wont go away…..any way I do it.

[xa0z]

Okay, what you need to do is "Remove" the package like normal… then Re-Install it like normal.  Then the new version has the fixes to take care of the problems you're having.  You can then remove it to completely remove it, or you can keep using it.

My only problem is that this is starting to move a direction in which I don't seem to have anymore say so in the project.  I'm not saying anything bad, and I'm not about to put mcrane down, he's a great guy and has helped me a great deal with stuff, but I just wish the project I started, and wrote part of... I had some control over.

[Supermule]

That is quite understandable…..:)

You have my support for the project, but it seems that the way the list is generated and the source of the data, is the big hurdle???

[xa0z]

I want to compile my own database.  I want to get it as simple as possible where specific rated categories are available for certin things.  I want to remove all the IPs in the database and use only hostnames (for now) to help keep it clean.  Web Browsers don't do reverse lookups so having all those extra useless items is a waste of RAM.

I will try my best to keep with the program, just want to make sure it stays on track.  I mean the whole point to me starting this was to help out, and if it's not helping me, how can it help others?

[Supermule]

Exactly. :)

Keep it up! You are on the right track….IP's change. Domainnames do not as often....So it is a good argument.