Shallalist2pfBlockerNG.php - script to import shallalit category to pfBlockerNG



  • Hi.

    shallalist2pfBlockerNG.php - script to import shallalit category to pfBlockerNG

    Download - download: http://www.javcasta.com/?smd_process_download=1&download_id=33336

    shallalist2pfBlockerNG.php is a php cli script to import the categories of shallalist to pfBlockerNG and make a cron to update them.

    Tested in pfSense with pfBlockerNG 2.1.1_4 2.3.2_1 amd64

    Necessary condition: Having installed in the pfBlockerNG pfSense package and configured and operating your DNSBL.

    The script is housed in a folder, for example /scripts, and run from shell or console:

    php /scripts/shallalist2pfBlockerNG.php

    Once executed and imported categories of shallalist the DNSBL feed shalllaist2pfBlockerNG, let pfSense GUI: Firewall> pfBlockerNG> DNSBL Feeds,

    and edit the feed shallalist2pfBlockerNG and put to ON (or fail to OFF) the categories you want.

    Will only make a Force Update / Reload the DNSBL pfblockerNG

    Code: shallalist2pfBlockerNG.php

    
    /*
    	shallalist2pfBlockerNG.php
    	Copyright (C) 2016 javier@javcasta.com Javier Castañón
            @javcasta - PIyMenta - https://javcasta.com/
    
    	Redistribution and use in source and binary forms, with or without
    	modification, are permitted provided that the following conditions are met:
    
    	1\. Redistributions of source code must retain the above copyright notice,
    	   this list of conditions and the following disclaimer.
    
    	2\. Redistributions in binary form must reproduce the above copyright
    	   notice, this list of conditions and the following disclaimer in the
    	   documentation and/or other materials provided with the distribution.
    
    	THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
    	INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
    	AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
    	AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY,
    	OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
    	SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
    	INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
    	CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
    	ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
    	POSSIBILITY OF SUCH DAMAGE.
    */
    
    require_once("config.inc");
    require_once("util.inc");
    require_once("functions.inc");
    require_once("pkg-utils.inc");
    require_once("pfsense-utils.inc");
    require_once("globals.inc");
    require_once("services.inc");
    
    global $config;
    $config = parse_config(true);
    
    //bash 
    //Remove directory
    print_r("Remove /tmp/BL dir \n");
    mwexec("/bin/rm -rf /tmp/BL");
    //Create directory
    print_r("Create /tmp/BL dir \n");
    mwexec("/bin/mkdir /tmp/BL");
    //Download list
    print_r("Download shallalist.tar.gz \n");
    mwexec("/usr/bin/fetch -o /tmp/BL http://www.shallalist.de/Downloads/shallalist.tar.gz");
    #Extract and remove archive
    print_r("Extract... \n");
    //mwexec("/usr/bin/cd /tmp/BL");
    mwexec("/usr/bin/tar -xf /tmp/BL/shallalist.tar.gz --strip 1 -C /tmp/BL");
    mwexec("/bin/rm -rf /tmp/BL/shallalist.tar.gz");
    
    function listFolders($dir)
    {
        //ref //http://stackoverflow.com/questions/14358346/how-to-build-a-array-that-contains-a-folder-structure
        $dh = scandir($dir);
        $resultado = array();
        $laruta = '';
    
        foreach ($dh as $folder) {
            if ($folder != '.' && $folder != '..' && $folder != 'COPYRIGHT' &&
                    $folder != 'global_usage' && $folder != 'shallalist.tar.gz') {
                $laruta = $dir . '/' . $folder;
                if (is_dir($dir . '/' . $folder)) {                
                    $resultado[] = listFolders($dir . '/' . $folder);               
                } else {
                    if (strpos($folder, 'domains') !== false) $resultado[] = $laruta;                
                }
            }
        }
        return $resultado;
    }
    //folders /tmp/BL to array
    $losdiraux = listFolders("/tmp/BL");
    $losdir = iterator_to_array(new RecursiveIteratorIterator(new RecursiveArrayIterator($losdiraux)), 0);
    
    //DNSBL feed shallalist2pfBlockerNG
    $j = 0;
    $existeDNSBLfeed = false;
    while (!empty($config['installedpackages']['pfblockerngdnsbl']['config'][$j]['aliasname'])) {
        $j++;
        if (strpos($config['installedpackages']['pfblockerngdnsbl']['config'][$j]['aliasname'], "shallalist2pfpBlockerNG") !== false) {
            $existeDNSBLfeed = true;
            print_r("Already exists DNSBL feed shallalist2pfpBlockerNG ... \n");
        }
    }
    
    if (!$existeDNSBLfeed) {
        $config['installedpackages']['pfblockerngdnsbl']['config'][$j]['aliasname'] = "shallalist2pfpBlockerNG";
        $config['installedpackages']['pfblockerngdnsbl']['config'][$j]['description'] = "shallalist to pfBlockerNG";
        $config['installedpackages']['pfblockerngdnsbl']['config'][$j]['infolist'] = "";
        //...
        $i = 0;
        foreach ($losdir as $elemento) {
            $config['installedpackages']['pfblockerngdnsbl']['config'][$j]['row'][$i]['format'] = 'auto';
            $config['installedpackages']['pfblockerngdnsbl']['config'][$j]['row'][$i]['state'] = 'Disabled'; //Enabled
            $config['installedpackages']['pfblockerngdnsbl']['config'][$j]['row'][$i]['url'] = "$elemento";
            $config['installedpackages']['pfblockerngdnsbl']['config'][$j]['row'][$i]['header'] = "sh2pfB_$i";
            $i++;
        }
        //...
        $config['installedpackages']['pfblockerngdnsbl']['config'][$j]['action'] = "unbound";
        $config['installedpackages']['pfblockerngdnsbl']['config'][$j]['cron'] = "Weekly";
        $config['installedpackages']['pfblockerngdnsbl']['config'][$j]['filter_alexa'] = "on";
        $config['installedpackages']['pfblockerngdnsbl']['config'][$j]['custom'] = "";
        $config['installedpackages']['pfblockerngdnsbl']['config'][$j]['custom_update'] = "disabled";
        print  "pfBlockerNG DNSBL feed List shallalist2pfpBlockerNG Import Completed...\n\n";
        write_config();
    }
    
    //Cron
    //si no existe /scripts lo creamos - if not exist /scripts create it
    if(!is_dir('/scripts')) {
      mkdir('/scripts');
    }
    //si no existe /scripts/cron-download-shallalist2pfpBlockerNG.sh lo creamos
    if (!file_exists('/scripts/cron-download-shallalist2pfpBlockerNG.sh')) {
     //bash
     $contenido = "#!/bin/sh\n"
            . "/bin/rm -rf /tmp/BL\n"
            . "/bin/mkdir /tmp/BL\n"
            . "/usr/bin/fetch -o /tmp/BL http://www.shallalist.de/Downloads/shallalist.tar.gz\n"
            . "/usr/bin/tar -xf /tmp/BL/shallalist.tar.gz --strip 1 -C /tmp/BL\n"
            . "/bin/rm -rf /tmp/BL/shallalist.tar.gz\n";
     file_put_contents("/scripts/cron-download-shallalist2pfpBlockerNG.sh", $contenido);
     print_r("Create /scripts/cron-download-shallalist2pfpBlockerNG.sh ... \n");
    
    } else print_r("Already exists /scripts/cron-download-shallalist2pfpBlockerNG.sh ... \n");
    //existe Cron?
    $j = 0;
    $existecron = false;
    while (!empty($config['cron']['item'][$j]['command'])) {
        $j++;
        //print_r($config['cron']['item'][$j]['command']."\n");
        if (strpos($config['cron']['item'][$j]['command'], "/bin/sh /scripts/cron-download-shallalist2pfpBlockerNG.sh") !== false) {
            $existecron = true;
            print_r("Already exists cron for /bin/sh /scripts/cron-download-shallalist2pfpBlockerNG.sh \n");
        }
    }
    //si no existe cron lo creamos a ejecutar como root cada domingo a las 06:06
    if (!$existecron) {
        $config['cron']['item'][$j]['minute'] = "6";
        $config['cron']['item'][$j]['hour'] = "6";
        //$config['cron']['item'][$j]['mday'] = "*";
        //$config['cron']['item'][$j]['month'] = "*";
        $config['cron']['item'][$j]['wday'] = "7";
        $config['cron']['item'][$j]['who'] = "root";
        $config['cron']['item'][$j]['command'] = "/bin/sh /scripts/cron-download-shallalist2pfpBlockerNG.sh";
        print_r("Create cron for /bin/sh /scripts/cron-download-shallalist2pfpBlockerNG.sh ... \n\n");
        write_config();
    }
    print_r("You have to go to https://YOUR-IP-PFSENSE/pkg.php?xml=/pfblockerng/pfblockerng_dnsbl_lists.xml \n"
            . " Firewall > pfBlockerNG > DNSBL Feeds: and edit/put ON/OFF the Categories you wish \n"
            . "and Force Update update/reload DNSBL pfblockerNG \n\n");
    print_r("The End :). By javcasta - htpps://javcasta.com \n");
    ?>
    
    

    Regards.

    References

    download: http://www.javcasta.com/?smd_process_download=1&download_id=33336

    https://www.javcasta.com/shallalist2pfblockerng-php-script-para-importar-categorias-shallalist-a-pfblockerng/

    https://forum.pfsense.org/index.php?topic=120068.0

    https://forum.pfsense.org/index.php?topic=119851.0



  • Hi.

    A new version for this script.

    I only change the path, because on a reboot the directory /tmp is not persistent.

    Now the path is: /usr/games/ , I have checked that this directory is persistent after a reboot.

    The code.

    
    /*
    	shallalist2pfBlockerNG.php
    	Copyright (C) 2016 javier@javcasta.com Javier Castañón
            @javcasta - PIyMenta - https://javcasta.com/
    
    	Redistribution and use in source and binary forms, with or without
    	modification, are permitted provided that the following conditions are met:
    
    	1\. Redistributions of source code must retain the above copyright notice,
    	   this list of conditions and the following disclaimer.
    
    	2\. Redistributions in binary form must reproduce the above copyright
    	   notice, this list of conditions and the following disclaimer in the
    	   documentation and/or other materials provided with the distribution.
    
    	THIS SOFTWARE IS PROVIDED ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES,
    	INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
    	AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
    	AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY,
    	OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
    	SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
    	INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
    	CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
    	ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
    	POSSIBILITY OF SUCH DAMAGE.
    */
    
    require_once("config.inc");
    require_once("util.inc");
    require_once("functions.inc");
    require_once("pkg-utils.inc");
    require_once("pfsense-utils.inc");
    require_once("globals.inc");
    require_once("services.inc");
    
    global $config;
    $config = parse_config(true);
    
    //bash 
    // /usr/games/
    //Remove directory
    print_r("Remove /usr/games/BL dir \n");
    mwexec("/bin/rm -rf /usr/games/BL");
    //Create directory
    print_r("Create /usr/games/BL dir \n");
    mwexec("/bin/mkdir /usr/games/BL");
    //Download list
    print_r("Download shallalist.tar.gz \n");
    mwexec("/usr/bin/fetch -o /usr/games/BL http://www.shallalist.de/Downloads/shallalist.tar.gz");
    #Extract and remove archive
    print_r("Extract... \n");
    //mwexec("/usr/bin/cd /usr/games/BL");
    mwexec("/usr/bin/tar -xf /usr/games/BL/shallalist.tar.gz --strip 1 -C /usr/games/BL");
    mwexec("/bin/rm -rf /usr/games/BL/shallalist.tar.gz");
    
    function listFolders($dir)
    {
        //ref //http://stackoverflow.com/questions/14358346/how-to-build-a-array-that-contains-a-folder-structure
        $dh = scandir($dir);
        $resultado = array();
        $laruta = '';
    
        foreach ($dh as $folder) {
            if ($folder != '.' && $folder != '..' && $folder != 'COPYRIGHT' &&
                    $folder != 'global_usage' && $folder != 'shallalist.tar.gz') {
                $laruta = $dir . '/' . $folder;
                if (is_dir($dir . '/' . $folder)) {                
                    $resultado[] = listFolders($dir . '/' . $folder);               
                } else {
                    if (strpos($folder, 'domains') !== false) $resultado[] = $laruta;                
                }
            }
        }
        return $resultado;
    }
    //folders /usr/games/BL to array
    $losdiraux = listFolders("/usr/games/BL");
    $losdir = iterator_to_array(new RecursiveIteratorIterator(new RecursiveArrayIterator($losdiraux)), 0);
    
    //DNSBL feed shallalist2pfBlockerNG
    $j = 0;
    $existeDNSBLfeed = false;
    while (!empty($config['installedpackages']['pfblockerngdnsbl']['config'][$j]['aliasname'])) {
        $j++;
        if (strpos($config['installedpackages']['pfblockerngdnsbl']['config'][$j]['aliasname'], "shallalist2pfpBlockerNG") !== false) {
            $existeDNSBLfeed = true;
            print_r("Already exists DNSBL feed shallalist2pfpBlockerNG ... \n");
        }
    }
    
    if (!$existeDNSBLfeed) {
        $config['installedpackages']['pfblockerngdnsbl']['config'][$j]['aliasname'] = "shallalist2pfpBlockerNG";
        $config['installedpackages']['pfblockerngdnsbl']['config'][$j]['description'] = "shallalist to pfBlockerNG";
        $config['installedpackages']['pfblockerngdnsbl']['config'][$j]['infolist'] = "";
        //...
        $i = 0;
        foreach ($losdir as $elemento) {
            $config['installedpackages']['pfblockerngdnsbl']['config'][$j]['row'][$i]['format'] = 'auto';
            $config['installedpackages']['pfblockerngdnsbl']['config'][$j]['row'][$i]['state'] = 'Disabled'; //Enabled
            $config['installedpackages']['pfblockerngdnsbl']['config'][$j]['row'][$i]['url'] = "$elemento";
            $config['installedpackages']['pfblockerngdnsbl']['config'][$j]['row'][$i]['header'] = "sh2pfB_$i";
            $i++;
        }
        //...
        $config['installedpackages']['pfblockerngdnsbl']['config'][$j]['action'] = "unbound";
        $config['installedpackages']['pfblockerngdnsbl']['config'][$j]['cron'] = "Weekly";
        $config['installedpackages']['pfblockerngdnsbl']['config'][$j]['filter_alexa'] = "on";
        $config['installedpackages']['pfblockerngdnsbl']['config'][$j]['custom'] = "";
        $config['installedpackages']['pfblockerngdnsbl']['config'][$j]['custom_update'] = "disabled";
        print  "pfBlockerNG DNSBL feed List shallalist2pfpBlockerNG Import Completed...\n\n";
        write_config();
    }
    
    //Cron
    //si no existe /usr/games/scripts lo creamos - if not exist /scripts create it
    if(!is_dir('/usr/games/scripts')) {
      mkdir('/usr/games/scripts');
    }
    //si no existe /usr/games/scripts/cron-download-shallalist2pfpBlockerNG.sh lo creamos
    if (!file_exists('/usr/games/scripts/cron-download-shallalist2pfpBlockerNG.sh')) {
     //bash
     $contenido = "#!/bin/sh\n"
            . "/bin/rm -rf /usr/games/BL\n"
            . "/bin/mkdir /usr/games/BL\n"
            . "/usr/bin/fetch -o /usr/games/BL http://www.shallalist.de/Downloads/shallalist.tar.gz\n"
            . "/usr/bin/tar -xf /usr/games/BL/shallalist.tar.gz --strip 1 -C /usr/games/BL\n"
            . "/bin/rm -rf /usr/games/BL/shallalist.tar.gz\n";
     file_put_contents("/usr/games/scripts/cron-download-shallalist2pfpBlockerNG.sh", $contenido);
     print_r("Create /usr/games/scripts/cron-download-shallalist2pfpBlockerNG.sh ... \n");
    
    } else print_r("Already exists /usr/games/scripts/cron-download-shallalist2pfpBlockerNG.sh ... \n");
    //existe Cron?
    $j = 0;
    $existecron = false;
    while (!empty($config['cron']['item'][$j]['command'])) {
        $j++;
        //print_r($config['cron']['item'][$j]['command']."\n");
        if (strpos($config['cron']['item'][$j]['command'], "/bin/sh /usr/games/scripts/cron-download-shallalist2pfpBlockerNG.sh") !== false) {
            $existecron = true;
            print_r("Already exists cron for /bin/sh /usr/games/scripts/cron-download-shallalist2pfpBlockerNG.sh \n");
        }
    }
    //si no existe cron lo creamos a ejecutar como root cada domingo a las 06:06
    if (!$existecron) {
        $config['cron']['item'][$j]['minute'] = "6";
        $config['cron']['item'][$j]['hour'] = "6";
        //$config['cron']['item'][$j]['mday'] = "*";
        //$config['cron']['item'][$j]['month'] = "*";
        $config['cron']['item'][$j]['wday'] = "7";
        $config['cron']['item'][$j]['who'] = "root";
        $config['cron']['item'][$j]['command'] = "/bin/sh /usr/games/scripts/cron-download-shallalist2pfpBlockerNG.sh";
        print_r("Create cron for /bin/sh /usr/games/scripts/cron-download-shallalist2pfpBlockerNG.sh ... \n\n");
        write_config();
    }
    print_r("You have to go to https://YOUR-IP-PFSENSE/pkg.php?xml=/pfblockerng/pfblockerng_dnsbl_lists.xml \n"
            . " Firewall > pfBlockerNG > DNSBL Feeds: and edit/put ON/OFF the Categories you wish \n"
            . "and Force Update update/reload DNSBL pfblockerNG \n\n");
    print_r("The End :). By javcasta - htpps://javcasta.com \n");
    ?>
    
    

    Regards.



  • Great will give this a test as i was looking for a way to stop using squid and squidguard



  • Beautiful, thank you very much, karma for you  :-*



  • Hi

    @Mr.:

    Beautiful, thank you very much, karma for you  :-*

    Pay attention to the next release of pfBlockerNG. It will bring many new features, including categories from the list of shallalist … and many more things :). Great tool!!!

    Regads.



  • Great stuff and I am about to give it a go.

    Why are you using /usr/games?  I understand that /tmp is transient but surely a better place would be /var/lib/BL for lists and /usr/local/BL for your code.

    You could put everything under /usr/local/shallalist, so /usr/local/shallalist/{bin, tmp, lib} for code, downloads and working, lists.

    Cheers
    Jon

    [edit]  Nice piece of work.  Just had a go and it works flawlessly.  Thank you.



  • Hi
    @Jon:

    Great stuff and I am about to give it a go.

    Why are you using /usr/games?  I understand that /tmp is transient but surely a better place would be /var/lib/BL for lists and /usr/local/BL for your code.

    You could put everything under /usr/local/shallalist, so /usr/local/shallalist/{bin, tmp, lib} for code, downloads and working, lists.

    Cheers
    Jon

    [edit]  Nice piece of work.  Just had a go and it works flawlessly.  Thank you.

    hi Jon, thanks :)

    The path /usr/games It's like a wink or a joke to the scripters and developers  ;D

    This script is obsolete or deprecated from the moment the next version of pfBlockerNG is released.

    We are waiting for pfBlockerNG 2.1.1_7 !!!!!!!!!!! (GREAT TOOL!!!)

    Regards



  • Me too. Waiting for godot  8)



  • @javcasta:

    hi Jon, thanks :)

    This script is obsolete or deprecated from the moment the next version of pfBlockerNG is released.

    We are waiting for pfBlockerNG 2.1.1_7 !!!!!!!!!!! (GREAT TOOL!!!)

    The 2.1.1_7 doesn't have this implemented yet  :(



  • cant tell, does this work or not , current pfBlockerNG 2.1.1_8



  • good morning all

    i have pfBlockerNG 2.1.1_10 shalla function is built in ?

    where can i find  it ?

    thanks Alberto



  • Hi,

    pfSense 2.4.0 Release with pfBlockerNG 2.1.2:

    I can not find any option to activate shallalist. :-(

    Any news? When will it be available?

    Regards



  • The latest updates to pfBlockerNG are bug fixes.

    I guess BBCan177 will change version number when the new features are implemented and tested.


Log in to reply