DNSBlacklist - script for more Ad-Filters
-
After i've changed back from a DD-WRT Router to pfSense yesterday i was missing the very good ADBlock function via hosts update from MVPS.
In a nutshell - wrote a little Script to get MVPS into DNSBlacklist ADS Filter.
The Script works basically, but i need a little help for the download command.
oh - and it's a n00b Script - my first one for a Unix System - so, if i did anything wrong let me know ;-)#!/bin/sh ############################################################################### # # moreHosts.sh # # Script to add additional Blacklists to DNSBlackList Package # created by o0M4dH4tter0o # ### Description ############################################################### # # the Script catches hosts files with "127.0.0.1 bad-address" format, # removes everything except the AD Server Adresses and writes them in # the DNSBlackList ADS domain file # ### Configuration ############################################################# # # there are 3 predefined hosts Server, # MVPS, HostFiles Project and Hostfiles NET # HF Project and HF NET are disabled because of the very restrictive # Filtering (for example, "megavideo") # To activate the Filters just uncomment them in the "download*" and # read "host file(s)*" Section # ### Version ################################################################### # # v0.1b - first Release # - File download doesn't work # ### created by OGrue ########################################################## ### first of all, backup the original host file; ############################## if [ -f /var/backups/dnsblacklist/domains ]; then echo " "domains" Backup exists" else mkdir /var/backups/dnsblacklist cp /usr/local/www/packages/dnsblacklist/blacklists/ads/domains /var/backups/dnsblacklist/ fi ### create temp folder ######################################################## mkdir /var/tmp/moreHosts ### download host files; ###################################################### # !!! i've no idea how this works in pfSense; !!! #http://winhelp2002.mvps.org/hosts.txt > /var/tmp/moreHosts/mvps #http://hostsfile.mine.nu/Hosts > /var/tmp/moreHosts/hfproject #http://www.it-mate.co.uk/downloads/hosts.txt > /var/tmp/moreHosts/hfnet ### copy "domain" from "DNSBlacklist ADS" to "hosts"; ######################### cp /usr/local/www/packages/dnsblacklist/blacklists/ads/domains /var/tmp/moreHosts/hosts ### read host file(s), remove "127.0.0.1", comments and "^M" (yes, i've ####### #### uploaded from a MS Client) and write in "hosts" ########################## more /var/tmp/moreHosts/mvps | grep -v localhost | grep 127.0.0.1 | awk '{print $2}' | tr -d '\r' >> /var/tmp/moreHosts/hosts #more /var/tmp/moreHosts/hfproject | grep -v localhost | grep 127.0.0.1 | awk '{print $2}' | tr -d '\r' >> /var/tmp/moreHosts/hosts #more /var/tmp/moreHosts/hfnet | grep -v localhost | grep 127.0.0.1 | awk '{print $2}' | tr -d '\r' >> /var/tmp/moreHosts/hosts ### filter double entries an write to DNSBlackList ADS "domains"; ############# sort -u /var/tmp/moreHosts/hosts > /usr/local/www/packages/dnsblacklist/blacklists/ads/domains ### *knock *knock - cleaner is here; ########################################## # as long as the download isn't working i'll delete only the hosts file # instead of the whole Directory #rm -r /var/tmp/moreHosts rm /var/tmp/moreHosts/hosts -
download host files;
!!! i've no idea how this works in pfSense; !!!
#http://winhelp2002.mvps.org/hosts.txt > /var/tmp/moreHosts/mvps
#http://hostsfile.mine.nu/Hosts > /var/tmp/moreHosts/hfproject
#http://www.it-mate.co.uk/downloads/hosts.txt > /var/tmp/moreHosts/hfnetuse fetch
fetch http://winhelp2002.mvps.org/hosts.txt
you must check all folders too.
-
use fetch
thx
you must check all folders too.
??? which folders should i check?
updated Code, works IMO fine (added a cron Job to update automatic) ..
#!/bin/sh ############################################################################### # moreHosts.sh # # Script to add additional Blacklists to DNSBlackList Package # created by o0M4dH4tter0o # ### Description ############################################################### # the Script catches hosts files with "127.0.0.1 bad-address" format, # removes everything except the AD Server Adresses and writes them in # the DNSBlackList ADS domain file # ### Configuration ############################################################# # there are 3 predefined hosts Server, # MVPS, HostFiles Project and Hostfiles NET # HF Project and HF NET are disabled because of the very restrictive # Filtering (for example, "megavideo") # To activate the Filters just uncomment them in the "download*" and # read "host file(s)*" Section # ### Version ################################################################### # v0.2b - 2nd Release // for me, the Script does what it should do # v0.1b - 1st Release // File download doesn't work # ### created by OGrue ########################################################## ### first of all, backup the original host file; ############################## if [ -f /var/backups/dnsblacklist/domains ]; then echo " "domains" Backup exists" else mkdir /var/backups/dnsblacklist cp /usr/local/www/packages/dnsblacklist/blacklists/ads/domains /var/backups/dnsblacklist/ fi ### create temp folder ######################################################## mkdir /var/tmp/moreHosts ### download host files; ###################################################### fetch -o /var/tmp/moreHosts/mvps http://winhelp2002.mvps.org/hosts.txt # fetch -o /var/tmp/moreHosts/hfproject http://hostsfile.mine.nu/Hosts # fetch -o /var/tmp/moreHosts/hfnet http://www.it-mate.co.uk/downloads/hosts.txt ### copy "domain" from "DNSBlacklist ADS" to "hosts"; ######################### cp /usr/local/www/packages/dnsblacklist/blacklists/ads/domains /var/tmp/moreHosts/hosts ### read host file(s), remove "127.0.0.1", comments and "^M" ################## more /var/tmp/moreHosts/mvps | grep -v localhost | grep 127.0.0.1 | awk '{print $2}' | tr -d '\r' >> /var/tmp/moreHosts/hosts # more /var/tmp/moreHosts/hfproject | grep -v localhost | grep 127.0.0.1 | awk '{print $2}' | tr -d '\r' >> /var/tmp/moreHosts/hosts # more /var/tmp/moreHosts/hfnet | grep -v localhost | grep 127.0.0.1 | awk '{print $2}' | tr -d '\r' >> /var/tmp/moreHosts/hosts ### filter double entries an write to DNSBlackList ADS "domains"; ############# sort -u /var/tmp/moreHosts/hosts > /usr/local/www/packages/dnsblacklist/blacklists/ads/domains ### *knock *knock - cleaner is here; ########################################## rm -r /var/tmp/moreHosts -
all folders you use in script.
/var/tmp to /tmp for example
-
to be a good net citizen, I would highly recommend either of the following with fetch. Of course that will require you to leave an unmodified copy of the file on your system, such as not deleting from /var/tmp/moreHosts.
from man fetch:
-i file If-Modified-Since mode: the remote file will only be
retrieved if it is newer than file on the local host. (HTTP
only)-m Mirror mode: if the file already exists locally and has the
same size and modification time as the remote file, it will
not be fetched. Note that the -m and -r flags are mutually
exclusive.Also from your script, your assuming /var/backups exists you might want to modify your script to check for its existence, and then use mkdir -p which will create all needed directories.
Just my $0.02.
