DNS Blacklist, New Package! Check it out.
Okay, so yesterday we released DNS Blacklist which is now added into the Packages list for pfSense.
The idea of the package is to use a freely available url/host/ip blacklist database to filter out sites you do not want visited from your network. Using every category listed in the DNS Blacklist will require roughly 256mb free memory on its own. This works by using dnsmasq and adding the categories you select into the dnsmasq config to reject the unwanted host and forward to Google. This is somewhat similar to OpenDNS in a sense, and does work really well.
I started this project a little over a month ago and with the help of mcrane I was able to get it finished and submitted. Right now you're limited to the hosts within the database but if the project continues on I will set it to allow custom entries from the DNS Blacklist control panel, and the option to edit the categories yourself.
Any and all feedback is welcome, or you can find me in ##pfsense or ##pfsense-dnsblacklist or just message me on freenode. And a big thanks to mcrane for his help, and putting up with me :P
Edit: I do realize that since combining the porn/adult categories I forgot to edit the text on the control panel. Also you are not required to restart dnsmasq manually like it says in the "savemsg"... That was fixed by adding services_dnsmasq_configure() into the source. And one last thing I can think of… You do not need to hit "APPLY" in the savemsg dialog to make the changes stick. I'll get all of this taken care of later today. Just in a hurry to get the package released we didn't get the control panel cleaned up properly.
thank you for the package I will check this =)
Cannot be disabled and uninstalled in webgui….
Cannot be disabled and uninstalled in webgui….
I noticed this earlier after the package was submitted. Not sure what happened between the version I submitted and the version being supplied but it does seem to have an issue here and there that I'll take care of later today. For a temporary fix, you can disable "DNS Blacklist", but you must leave a category selected while the "Enabled" checkbox is deselected.
I don't have access to the "DNS Blacklist" repo on pfSense to fix things so I have to wait for mcrane to come online, then submit the changes to him and then he will submit them to pfSense.
Nice….Not superurgent, but a small feedback to you.
I suppose one can update the package via an uninstall....
Thank you xa0z! Great job :)
Thanks for the heads up!
thanks, more power to your team guys :)
Thanks for implementing this package. I'm trying it out , and I think it has a nice feature set and is super easy to configure. I have a few suggestions for additions:
1. The ability to add or remove domains and url's to a whitelist and to a blacklist.
2. The ability to specify the web page url to which users are redirected for each category.
3. The ability to specify the blacklist to use. Or, alternatively, some documentation about what's on the existing blocklist.
I noticed that the dns blacklist package mentions porn in the notes at the top of the page, but there appears to be not porn category (at least on the embedded machine on which I'm doing my initial testing). Is this an error or a feature to reduce the footprint on embedded devices?
I appreciate the time and effort that you have placed into creating this package. I think it definitely fits the need for an additional filtering plugin for PFSense. :)
Vaughn Reid III
Thanks for the comments. The next version of this script does allow users to add their own custom blacklist entries. As of right now there is no way to specify the URL that a blacklisted item is sent to, that is why it is forwarding to a Google IP. Doing the method you are requesting requires a proxy. This is being looked into for the future.
As for being able to select your own blacklist database, I have already thought of this and think it would be a great idea but right now the scripting is
static…it was static, it is dynamic now... to use preset categories/directories for the database. In the near future I would like to have this set to load the categories from a variable after reading the available directories containing the blacklists.
In a beta version of DNS Blacklist we had two separate categories… "Porn" and "Adult". That blacklist database is not what we're using right now so the categories are different and now contained under the same category. The configuration page still lists the two categories in the NOTE because I forgot to remove it, just a self error.
We're going to release a "fixed" version here soon, but a major release with additions will be a little later.
Did some improvements to the package today.
1. wording changes.
2. uninstall has been fixed
3. all categories can be deselected.
You can edit categories by using pfSense's command page:
Diagnostics -> Command
Lets say you wanted to edit the 'adult' list, you can download the list using:
File to download -> /usr/local/www/packages/dnsblacklist/blacklists/adult/domains
You would then edit the file using an editor that deals well with large files.
If you are on windows don't use the standard notepad.exe instead use pspad or notepad++
To upload the file from the command page use:
File to upload
The file will be uploaded to the /tmp directory.
Then use 'Execute Shell command' to cp the file to the correct directory.
cp /tmp/domains /usr/local/www/packages/dnsblacklist/blacklists/adult/domains
If you follow the folder structure consistently you can add additional categories.
You also would need to carefully edit the /usr/local/www/packages/dnsblacklist/blacklists/global_usage file which list the categories and provides the description for the categories. You can edit the global_usage file using Diagnostics -> Edit File
Tomorrow I will begin working on our own Blacklist Database. It will work in the same manor as current… We will have a blacklist directory, and within that directory we will have a directory with the respective category name, and then within that directory we have the "domains" file which contains the domains that will be added to the active blacklist when that category is selected.
Today mcrane did a great job in updating the code to make sure everything was working as normal as we can. Remember we're still in BETA so give it some time, make your requests for addons, report any bugs you might see and we will do our best to make sure things go as smooth as possible.
As for right now the only concern I have is making sure we can get a really good, clean, host-database for the blacklisting.
The current list comes from here:
Its just not as large as xa0z would like.
The size isn't what bothers me… I think that a proper database should contain categories that are more limited to specific matching items, and better defined categories at that.
That was the whole point of me starting this project.
Here is the original blacklist that was going to be used.
I didn't use it in the final version of the package because I didn't feel that redistribution of that list was the ethical thing to do when the website that provides the list is trying to sell it. They provide the entire list for download and payment is on an honor basis as stated on their website. Their intent is really to sell it for a subscription. I did not notice this when I started building this package. When I noticed they were trying to sell the list they had compiled I could not with good conscious redistribute urlblacklist.com's list. If someone wants to use that list and pay for the subscription they can do so by editing the domain lists and adding the categories as I described in an earlier post.
Here is a few more choices of DNS blacklists.
Anyone can edit the list that the package comes with. In order for an alternative customized list to be used in the DNS blacklist package it must be compiled ethically and legally or I will not commit it to the package. For example it wouldn't be ethical to use lists from urlblacklist.com or other lists that don't offer the list in freely. Unless you obtain permission directly from them to use the list for such a purpose.
The list from http://cri.univ-tlse1.fr/blacklists/index_en.php says on their website 'can be used with many commercial or free software' therefore it is ethical to use their list for the DNS Blacklist package.
Hmm, so I guess there isn't a need for me to continue on with the project. It is just pointless for me to try and do one thing and then have it all changed to not be the way it was when I started it.
I already said I was going to build my own database list but what's the point, I don't need to supply something that anyone can just grab from all over the net to use.
I'm glad for the help I got when I started writing the project but if I am being phased out, which it so seems I am, then I guess my input, or work is useless.
If this isn't supported and maintained, pls. inform me how to delete it completely! It wont go away…..any way I do it.
Okay, what you need to do is "Remove" the package like normal… then Re-Install it like normal. Then the new version has the fixes to take care of the problems you're having. You can then remove it to completely remove it, or you can keep using it.
My only problem is that this is starting to move a direction in which I don't seem to have anymore say so in the project. I'm not saying anything bad, and I'm not about to put mcrane down, he's a great guy and has helped me a great deal with stuff, but I just wish the project I started, and wrote part of... I had some control over.
That is quite understandable…..:)
You have my support for the project, but it seems that the way the list is generated and the source of the data, is the big hurdle???
I want to compile my own database. I want to get it as simple as possible where specific rated categories are available for certin things. I want to remove all the IPs in the database and use only hostnames (for now) to help keep it clean. Web Browsers don't do reverse lookups so having all those extra useless items is a waste of RAM.
I will try my best to keep with the program, just want to make sure it stays on track. I mean the whole point to me starting this was to help out, and if it's not helping me, how can it help others?
Keep it up! You are on the right track….IP's change. Domainnames do not as often....So it is a good argument.
The problem is the list. As I stated before it has to be created ethically. That is compiled from free lists where the owners give their permission.
I will not commit something to pfSense that makes me an accessory to stealing. That is why I refused to commit the package with the previous list. If I did this I would expect to get my commit authority revoked.
It is not impossible to find lists that are free for any use and supplement them, and improve them with your own domains you and others find while searching.
That is exactly what I'm doing. I'm not worried about the urlBlacklists database anymore. I am just saying that I'm getting the database taken care of on my own, and it will be greatly categorizied.
I think this is a good project, if the DNS Blacklist is good and effectively use. Surely there will be people supported and willing to donate some subscription fee to maintain the Blacklist and the project itself in this Pfsense forum/members.
I am no expert in here, but there are people who might (already) using OpenDNS to filter, as long as this project don't run the same track as the OpenDNS or able to offer more than OpenDNS. I think there are light on this projects.
Good Work !! ;)
I've tried DNS blacklist in one of my pfSense box but it broke the dns forwarder service, after installation the dnsmasq service stopped and can't be restarted even after repeated tries and reboot. I uninstalled the aforementioned package but I still can't start the dnsmasq service,I had to examine other working boxes to see what files have been added or changed by the DNS blacklist package, it added the dnsmasq.conf which I've deleted and my dnsmasq service finally started.
Did I missed something?
When you tried to start dnsmasq. If you would have looked at the Diagnostics: System logs: System and looked for errors inr regards to dnsmasq then you should be ablt to find a description why it refused to start.
Good job :)
I have Q.. How to block HTTPS? (except legit HTTPS)?
Mine is running fine.
built on Sat Jul 18 19:19:52 EDT 2009
FreeBSD 7.2-RELEASE-p2 i386
DNS Blacklist 0.2.4
Yes, after the installation of DNS Blacklist. I have to manual restart the services.
My Box run in Bridge mode, I guess yours are different in NAT mode.
After installing DNS Blacklist you shouldn't be required to restart dnsmasq as nothing is edited that pertains to dnsmasq at the time. DNS Blacklist adds the dnsmasq.conf, and dnsmasq.blacklist.conf files into /usr/local/etc/. When DNS Blacklist is enabled it adds a string into dnsmasq.conf to load the dnsmasq.blacklist.conf file, then restarts dnsmasq. Any of the categories you select are entered within the dnsmasq.blacklist.conf file and that is what allows us to filter dns querys to the local server.
I am in no way out to seek any money from anyone for the blacklist database I'm putting together. I can maintain a "main blacklist" but users would be free to add their own domains that aren't already listed. Here soon I'll work on adding a custom Blacklist/Whitelist text area for you to enter your own on the fly.
If you want to block all https, you need to put a block on dport:443, that isn't associated with DNS Blacklist.
| If you want to block all https, you need to put a block on dport:443, that isn't associated with DNS Blacklist.
- I dont want to block 443 from Firewall LAn.. Some users needs to access legitimate https sites…. Kindly show us the right way xa0z? Thanks
I can't really thank you enough for putting in the effort for this package. This is exactly what my place of employment has been looking for to push us off of using WatchGuard Fireboxes and moving to a custom-built firewall running pfSense.
Thanks, and if you need any help, let me know!
Is that package the same as using squid with 0 cache + squidGuard using the urlblacklist.com filter?
Not really, but the same concept mainly. We're not using proxies, and are just making hostnames that you don't want allowed on your network to resolve to a specific IP rather than loading a proxy, etc.
So for instance, if you have facebook.com added to the category of denied hosts, then if anyone tried to resolve the forementioned host name then it would resolve to the IP I currently have set in the config, which is a Google IP. So all requests would be for example like so… http://www.facebook.com/games/mafiawars, would actually load http://www.google.com/games/mafiawars, which would fail, and alert you so.
I do have improvement ideas I'd like to implement in, but I won't be able to submit/commit them until mcrane is ready to do so and currently he has other projects he's working on.
As I understand the LAN DNS must be the pfsense DNS forwarder in order to make this package work?
10X for your effort
Yes, in order to use this you MUST do 1 of two things…
1: Make sure ALL clients on your LAN have the pfSense Gateway IP as their DNS IP.
2: Set any and all connections that pass through on port 53, to bind back to the router IP on port 53.
Are there any sense in doing this, if other DNS Services (OpenDNS) will override the settings on this??
In order for OpenDNS, and other DNS Services to work, you need to use their IP Address as your DNS Server IP.
The concept of OpenDNS and DNSBlacklist is about the same except the changes made to DNSBlacklist are local (on the system)
If you run DNS Blacklist, or other DNS Services like OpenDNS you can prevent people from loading other DNS Servers by forcing ALL outbound connections to port 53 to stop at the pfSense box. This way no matter where they try to resolve host names, it will always use the DNS Server on the pfSense box, be that the DNS Forwarder of OpenDNS, etc.
How do you specific prevent people from doing that???
How to in Pfsense???
Highlighted in RED.
I cant see anything….
heh, reload the page. it should show up now.