PfBlocker and .CSV File Format Solution


  • Moderator

    If anyone was looking for a way to have pfBlocker utilize .CSV files, here is one option-

    Stop Forum Spam (List is currently Listing approx 442,000 unique IPs addresses.

    We provide free access to our database of known forum and blog spammers. The database provides their
    email addresses, IP addresses, usernames, spamming frequency, and, in some cases, evidence of their spam.

    http://www.stopforumspam.com
    http://www.stopforumspam.com/downloads/bannedips.zip  (Which contain a .CSV file)

    Unfortunately pfBlocker only supports .TXT and .GZ files.

    My Solution:

    SSH into your pfSense box.

    cd /tmp

    (create this file with the following script)

    vi pffetch

    #!/bin/sh
          fetch http://www.stopforumspam.com/downloads/bannedips.zip
          tar -xOf bannedips.zip | tr , '\n' > bannedips.txt

    (save)

    chmod +x pffetch

    To test, type the following

    ./pffetch
        cat bannedips.txt

    If it displays the file with

    x.x.x.x
        x.x.x.x
        x.x.x.x
        x.x.x.x
        (will show all 442,000 ips)

    than the pffetch script works correctly.

    –----------------------

    From the pfSense GUI

    Goto SYSTEM:PACKAGES

    Add the "CRON" Package

    Once Cron is installed, go into CRON and add a new "Cron Task"

    ******* Be careful not to delete or edit any of the existing CRON jobs!**

    min/hour/mday/month/wday/who/command
    0 | 4,12,20 | * | * | * | root | /tmp/pffetch

    So this will run the script at 4,12,20 Military Time.
    StopForumSpam limits downloading to 3 times MAX per DAY!

    –---------------------

    Make sure you have a large enough "Firewall Maximum Table Entries" in SYSTEM:ADVANCED:FIREWALL/NAT
    It needs to be large enough to hold all of your ip lists together.

    "Maximum number of table entries for systems such as aliases, sshlockout, snort, etc, combined."

    –---------------------

    Goto pfBlocker:LISTS

    Create a new List

    Alias Name  : StopForumSpam
    List Desc.  : pfBlocker StopForumSpam
    txt        : /tmp/bannedips.txt

    List Action : Alias Only

    Update Freq : Every 4 Hours

    Save

    –---------------------

    Goto Firewall:Rules

    Add Rules to utilize the Alias "StopForumSpam" in WAN as "Block" and LAN as "Reject"

    –----------------------

    Another list that uses .CSV format

    http://www.autoshun.org/
    http://www.autoshun.org/files/shunlist.csv

    Same process as above. Use the same script unless you want to create multiple scripts with
    different Cron/pfBlocker tasks.

    #!/bin/sh
    fetch http://www.stopforumspam.com/downloads/bannedips.zip
    fetch http://www.autoshun.org/files/shunlist.csv
    tar -xOf bannedips.zip | tr , '\n' > bannedips.txt
    cat shunlist.csv | grep -o '^*[^[,]{0,15}' > shunlist.txt


  • Moderator

    Made some changes to the script. If the download fails for what ever reason, it can clear the previous downloaded data leaving your table blank.

    #!/bin/sh
    
    fetch http://www.stopforumspam.com/downloads/bannedips.zip
    returncode=$?
    echo $returncode
    
    if [ "$returncode" -eq "0" ]; then
             tar -xOf bannedips.zip | tr , '\n' > bannedips.txt
    fi
    
    fetch http://www.autoshun.org/files/shunlist.csv
    returncode=$?
    echo $returncode
    
    if [ "$returncode" -eq "0" ]; then
             cat shunlist.csv | grep -o '\^*[^[,]\{0,15\}' > shunlist.txt
    fi
    
    

    Also, moved the script to /home folder as the /tmp folder doesn't persist after reboot …

    Does anyone know if there is an upper limit on the total number of :

    "Firewall Maximum Table Entries" in SYSTEM:ADVANCED:FIREWALL/NAT



  • Thanks for the contribution.

    To keep this script on pfsense xml backup, I suggest you to install package filer instead of creating this script via console/ssl.


  • Moderator

    @marcelloc:

    Thanks for the contribution.

    To keep this script on pfsense xml backup, I suggest you to install package filer instead of creating this script via console/ssl.

    Thanks Marcelloc,

    I added the "Filer" Package. I had to rename the file to pffetch.sh to get Filer View/Edit working.

    To make the script executable, you will still need to go manually run "chmod +x"? Or is there a work around?

    How do you get the files to save in the xml backup file? I don't see any settings for that? Or is there a particular folder to create script files?

    Thanks for your help.

    ps - Next step is to get pfBlocker to pull spam ips from my local mail server… Spammustdie!!!


  • Moderator

    @marcelloc:

    Thanks for the contribution.

    To keep this script on pfsense xml backup, I suggest you to install package filer instead of creating this script via console/ssl.

    Hi Marcelloc,

    I installed "File manager" instead of the "Filer" package.

    I now see the correct settings to manage the script from the GUI.

    Much easier….. Thanks.

    Question - What would be the best way to get a .txt file from a local server to pfSense?


  • Moderator

    @marcelloc:

    Thanks for the contribution.

    To keep this script on pfsense xml backup, I suggest you to install package filer instead of creating this script via console/ssl.

    Marcelloc,

    I have a script that is pulling my spammers IP address and collecting them into a .TXT file (CIDR Format). It is creating a file every 5 minutes and sending it to a file in pfSense
    with SCP.

    Unfortunately pfBlocker is limited to once per hour.

    Can this code be edited to say 5mins, 1hr, 4hrs, 12hrs? Can I edited this php file directly?

     $cron=array('01hour' => 1,
                                    '04hours' => 4,
                                    '12hours' => 12,
                                    'EveryDay' => 23);
    
    

    Please let me know if you have any other suggestions on how to get it to update more frequently.

    Thanks.


  • Moderator

    The little script in my first post needs to be updated to change the path to the user directory. As is, the Fetch (downloads) are going into

    /etc/cron directory.

    Please add

    #!/bin/sh

    cd /home/USERDIRECTORY (Or where ever you would like the downloads to go to. And also where pfBlocker will lookup.


Log in to reply