DNSBlacklist - script for more Ad-Filters



  • After i've changed back from a DD-WRT Router to pfSense yesterday i was missing the very good ADBlock function via hosts update from MVPS.
    In a nutshell - wrote a little Script to get MVPS into DNSBlacklist ADS Filter.
    The Script works basically, but i need a little help for the download command.
    oh - and it's a n00b Script - my first one for a Unix System - so, if i did anything wrong let me know ;-)

    #!/bin/sh
    ###############################################################################
    #
    # moreHosts.sh
    #
    # Script to add additional Blacklists to DNSBlackList Package
    # created by o0M4dH4tter0o
    # 
    ### Description ###############################################################
    #
    # the Script catches hosts files with "127.0.0.1   bad-address" format,
    # removes everything except the AD Server Adresses and writes them in
    # the DNSBlackList ADS domain file
    #
    ### Configuration #############################################################
    #
    # there are 3 predefined hosts Server,
    # MVPS, HostFiles Project and Hostfiles NET
    # HF Project and HF NET are disabled because of the very restrictive
    # Filtering (for example, "megavideo")
    # To activate the Filters just uncomment them in the "download*" and
    # read "host file(s)*" Section
    #
    ### Version ###################################################################
    #
    # v0.1b - first Release
    #   - File download doesn't work
    #
    ### created by OGrue ##########################################################
    
    ### first of all, backup the original host file; ##############################
    if [ -f /var/backups/dnsblacklist/domains ];
    then
       echo " "domains" Backup exists"
    else
       mkdir /var/backups/dnsblacklist
       cp /usr/local/www/packages/dnsblacklist/blacklists/ads/domains /var/backups/dnsblacklist/
    fi
    
    ### create temp folder ########################################################
    mkdir /var/tmp/moreHosts
    
    ### download host files; ######################################################
    # !!! i've no idea how this works in pfSense; !!!
    #http://winhelp2002.mvps.org/hosts.txt 		> /var/tmp/moreHosts/mvps
    #http://hostsfile.mine.nu/Hosts  		> /var/tmp/moreHosts/hfproject
    #http://www.it-mate.co.uk/downloads/hosts.txt	> /var/tmp/moreHosts/hfnet
    
    ### copy "domain" from "DNSBlacklist ADS" to "hosts"; #########################
    cp /usr/local/www/packages/dnsblacklist/blacklists/ads/domains /var/tmp/moreHosts/hosts
    
    ### read host file(s), remove "127.0.0.1", comments and "^M" (yes, i've #######
    #### uploaded from a MS Client) and write in "hosts" ##########################
    more /var/tmp/moreHosts/mvps | grep -v localhost | grep 127.0.0.1 | awk '{print $2}' | tr -d '\r' >> /var/tmp/moreHosts/hosts
    #more /var/tmp/moreHosts/hfproject | grep -v localhost | grep 127.0.0.1 | awk '{print $2}' | tr -d '\r' >> /var/tmp/moreHosts/hosts
    #more /var/tmp/moreHosts/hfnet | grep -v localhost | grep 127.0.0.1 | awk '{print $2}' | tr -d '\r' >> /var/tmp/moreHosts/hosts
    
    ### filter double entries an write to DNSBlackList ADS "domains"; #############
    sort -u /var/tmp/moreHosts/hosts > /usr/local/www/packages/dnsblacklist/blacklists/ads/domains
    
    ### *knock *knock - cleaner is here; ##########################################
    # as long as the download isn't working i'll delete only the hosts file
    # instead of the whole Directory
    #rm -r /var/tmp/moreHosts
    rm /var/tmp/moreHosts/hosts
    


  • @OGrue:

    download host files;

    !!! i've no idea how this works in pfSense; !!!

    #http://winhelp2002.mvps.org/hosts.txt > /var/tmp/moreHosts/mvps
    #http://hostsfile.mine.nu/Hosts  > /var/tmp/moreHosts/hfproject
    #http://www.it-mate.co.uk/downloads/hosts.txt > /var/tmp/moreHosts/hfnet

    use fetch

    fetch http://winhelp2002.mvps.org/hosts.txt

    you must check all folders too.



  • @marcelloc:

    use fetch

    thx

    @marcelloc:

    you must check all folders too.

    ??? which folders should i check?

    updated Code, works IMO fine (added a cron Job to update automatic) ..

    
    #!/bin/sh
    
    ###############################################################################
    # moreHosts.sh
    #
    # Script to add additional Blacklists to DNSBlackList Package
    # created by o0M4dH4tter0o
    # 
    ### Description ###############################################################
    # the Script catches hosts files with "127.0.0.1   bad-address" format,
    # removes everything except the AD Server Adresses and writes them in
    # the DNSBlackList ADS domain file
    #
    ### Configuration #############################################################
    # there are 3 predefined hosts Server,
    # MVPS, HostFiles Project and Hostfiles NET
    # HF Project and HF NET are disabled because of the very restrictive
    # Filtering (for example, "megavideo")
    # To activate the Filters just uncomment them in the "download*" and
    # read "host file(s)*" Section
    #
    ### Version ###################################################################
    # v0.2b - 2nd Release // for me, the Script does what it should do
    # v0.1b - 1st Release // File download doesn't work
    #
    ### created by OGrue ##########################################################
    
    ### first of all, backup the original host file; ##############################
    if [ -f /var/backups/dnsblacklist/domains ];
    then
       echo " "domains" Backup exists"
    else
       mkdir /var/backups/dnsblacklist
       cp /usr/local/www/packages/dnsblacklist/blacklists/ads/domains /var/backups/dnsblacklist/
    fi
    
    ### create temp folder ########################################################
    mkdir /var/tmp/moreHosts
    
    ### download host files; ######################################################
    fetch -o /var/tmp/moreHosts/mvps http://winhelp2002.mvps.org/hosts.txt
    # fetch -o /var/tmp/moreHosts/hfproject http://hostsfile.mine.nu/Hosts
    # fetch -o /var/tmp/moreHosts/hfnet http://www.it-mate.co.uk/downloads/hosts.txt
    
    ### copy "domain" from "DNSBlacklist ADS" to "hosts"; #########################
    cp /usr/local/www/packages/dnsblacklist/blacklists/ads/domains /var/tmp/moreHosts/hosts
    
    ### read host file(s), remove "127.0.0.1", comments and "^M" ##################
    more /var/tmp/moreHosts/mvps | grep -v localhost | grep 127.0.0.1 | awk '{print $2}' | tr -d '\r' >> /var/tmp/moreHosts/hosts
    # more /var/tmp/moreHosts/hfproject | grep -v localhost | grep 127.0.0.1 | awk '{print $2}' | tr -d '\r' >> /var/tmp/moreHosts/hosts
    # more /var/tmp/moreHosts/hfnet | grep -v localhost | grep 127.0.0.1 | awk '{print $2}' | tr -d '\r' >> /var/tmp/moreHosts/hosts
    
    ### filter double entries an write to DNSBlackList ADS "domains"; #############
    sort -u /var/tmp/moreHosts/hosts > /usr/local/www/packages/dnsblacklist/blacklists/ads/domains
    
    ### *knock *knock - cleaner is here; ##########################################
    rm -r /var/tmp/moreHosts
    
    


  • all folders you use in script.

    /var/tmp to /tmp for example



  • to be a good net citizen, I would highly recommend either of the following with fetch. Of course that will require you to leave an unmodified copy of the file on your system, such as not deleting from /var/tmp/moreHosts.

    from man fetch:

    -i file     If-Modified-Since mode: the remote file will only be
                    retrieved if it is newer than file on the local host.  (HTTP
                    only)

    -m          Mirror mode: if the file already exists locally and has the
                    same size and modification time as the remote file, it will
                    not be fetched.  Note that the -m and -r flags are mutually
                    exclusive.

    Also from your script, your assuming /var/backups exists you might want to modify your script to check for its existence, and then use mkdir -p which will create all needed directories.

    Just my $0.02.


Log in to reply