PfBlocker and .CSV File Format Solution

[BBcan177]

If anyone was looking for a way to have pfBlocker utilize .CSV files, here is one option-

Stop Forum Spam (List is currently Listing approx 442,000 unique IPs addresses.

We provide free access to our database of known forum and blog spammers. The database provides their
email addresses, IP addresses, usernames, spamming frequency, and, in some cases, evidence of their spam.

http://www.stopforumspam.com
http://www.stopforumspam.com/downloads/bannedips.zip  (Which contain a .CSV file)

Unfortunately pfBlocker only supports .TXT and .GZ files.

My Solution:

SSH into your pfSense box.

cd /tmp

(create this file with the following script)

vi pffetch

#!/bin/sh
      fetch http://www.stopforumspam.com/downloads/bannedips.zip
      tar -xOf bannedips.zip | tr , '\n' > bannedips.txt

(save)

chmod +x pffetch

To test, type the following

./pffetch
    cat bannedips.txt

If it displays the file with

x.x.x.x
    x.x.x.x
    x.x.x.x
    x.x.x.x
    (will show all 442,000 ips)

than the pffetch script works correctly.

–----------------------

From the pfSense GUI

Goto SYSTEM:PACKAGES

Add the "CRON" Package

Once Cron is installed, go into CRON and add a new "Cron Task"

******* Be careful not to delete or edit any of the existing CRON jobs!**

min/hour/mday/month/wday/who/command
0 | 4,12,20 | * | * | * | root | /tmp/pffetch

So this will run the script at 4,12,20 Military Time.
StopForumSpam limits downloading to 3 times MAX per DAY!

–---------------------

Make sure you have a large enough "Firewall Maximum Table Entries" in SYSTEM:ADVANCED:FIREWALL/NAT
It needs to be large enough to hold all of your ip lists together.

"Maximum number of table entries for systems such as aliases, sshlockout, snort, etc, combined."

–---------------------

Goto pfBlocker:LISTS

Create a new List

Alias Name  : StopForumSpam
List Desc.  : pfBlocker StopForumSpam
txt        : /tmp/bannedips.txt

List Action : Alias Only

Update Freq : Every 4 Hours

Save

–---------------------

Goto Firewall:Rules

Add Rules to utilize the Alias "StopForumSpam" in WAN as "Block" and LAN as "Reject"

–----------------------

Another list that uses .CSV format

http://www.autoshun.org/
http://www.autoshun.org/files/shunlist.csv

Same process as above. Use the same script unless you want to create multiple scripts with
different Cron/pfBlocker tasks.

#!/bin/sh
fetch http://www.stopforumspam.com/downloads/bannedips.zip
fetch http://www.autoshun.org/files/shunlist.csv
tar -xOf bannedips.zip | tr , '\n' > bannedips.txt
cat shunlist.csv | grep -o '^*[^[,]{0,15}' > shunlist.txt

[BBcan177]

Made some changes to the script. If the download fails for what ever reason, it can clear the previous downloaded data leaving your table blank.

#!/bin/sh

fetch http://www.stopforumspam.com/downloads/bannedips.zip
returncode=$?
echo $returncode

if [ "$returncode" -eq "0" ]; then
         tar -xOf bannedips.zip | tr , '\n' > bannedips.txt
fi

fetch http://www.autoshun.org/files/shunlist.csv
returncode=$?
echo $returncode

if [ "$returncode" -eq "0" ]; then
         cat shunlist.csv | grep -o '\^*[^[,]\{0,15\}' > shunlist.txt
fi

Also, moved the script to /home folder as the /tmp folder doesn't persist after reboot …

Does anyone know if there is an upper limit on the total number of :

"Firewall Maximum Table Entries" in SYSTEM:ADVANCED:FIREWALL/NAT

[marcelloc]

Thanks for the contribution.

To keep this script on pfsense xml backup, I suggest you to install package filer instead of creating this script via console/ssl.

[BBcan177]

@marcelloc:

Thanks for the contribution.

To keep this script on pfsense xml backup, I suggest you to install package filer instead of creating this script via console/ssl.

Thanks Marcelloc,

I added the "Filer" Package. I had to rename the file to pffetch.sh to get Filer View/Edit working.

To make the script executable, you will still need to go manually run "chmod +x"? Or is there a work around?

How do you get the files to save in the xml backup file? I don't see any settings for that? Or is there a particular folder to create script files?

Thanks for your help.

ps - Next step is to get pfBlocker to pull spam ips from my local mail server… Spammustdie!!!

[BBcan177]

@marcelloc:

Thanks for the contribution.

To keep this script on pfsense xml backup, I suggest you to install package filer instead of creating this script via console/ssl.

Hi Marcelloc,

I installed "File manager" instead of the "Filer" package.

I now see the correct settings to manage the script from the GUI.

Much easier….. Thanks.

Question - What would be the best way to get a .txt file from a local server to pfSense?

[BBcan177]

@marcelloc:

Thanks for the contribution.

To keep this script on pfsense xml backup, I suggest you to install package filer instead of creating this script via console/ssl.

Marcelloc,

I have a script that is pulling my spammers IP address and collecting them into a .TXT file (CIDR Format). It is creating a file every 5 minutes and sending it to a file in pfSense
with SCP.

Unfortunately pfBlocker is limited to once per hour.

Can this code be edited to say 5mins, 1hr, 4hrs, 12hrs? Can I edited this php file directly?

 $cron=array('01hour' => 1,
                                '04hours' => 4,
                                '12hours' => 12,
                                'EveryDay' => 23);

Please let me know if you have any other suggestions on how to get it to update more frequently.

Thanks.

[BBcan177]

The little script in my first post needs to be updated to change the path to the user directory. As is, the Fetch (downloads) are going into

/etc/cron directory.

Please add

#!/bin/sh

cd /home/USERDIRECTORY (Or where ever you would like the downloads to go to. And also where pfBlocker will lookup.