PfBlocker and .CSV File Format Solution
-
If anyone was looking for a way to have pfBlocker utilize .CSV files, here is one option-
Stop Forum Spam (List is currently Listing approx 442,000 unique IPs addresses.
We provide free access to our database of known forum and blog spammers. The database provides their
email addresses, IP addresses, usernames, spamming frequency, and, in some cases, evidence of their spam.http://www.stopforumspam.com
http://www.stopforumspam.com/downloads/bannedips.zip (Which contain a .CSV file)Unfortunately pfBlocker only supports .TXT and .GZ files.
My Solution:
SSH into your pfSense box.
cd /tmp
(create this file with the following script)
vi pffetch
#!/bin/sh
fetch http://www.stopforumspam.com/downloads/bannedips.zip
tar -xOf bannedips.zip | tr , '\n' > bannedips.txt(save)
chmod +x pffetch
To test, type the following
./pffetch
cat bannedips.txtIf it displays the file with
x.x.x.x
x.x.x.x
x.x.x.x
x.x.x.x
(will show all 442,000 ips)than the pffetch script works correctly.
–----------------------
From the pfSense GUI
Goto SYSTEM:PACKAGES
Add the "CRON" Package
Once Cron is installed, go into CRON and add a new "Cron Task"
******* Be careful not to delete or edit any of the existing CRON jobs!**
min/hour/mday/month/wday/who/command
0 | 4,12,20 | * | * | * | root | /tmp/pffetchSo this will run the script at 4,12,20 Military Time.
StopForumSpam limits downloading to 3 times MAX per DAY!–---------------------
Make sure you have a large enough "Firewall Maximum Table Entries" in SYSTEM:ADVANCED:FIREWALL/NAT
It needs to be large enough to hold all of your ip lists together."Maximum number of table entries for systems such as aliases, sshlockout, snort, etc, combined."
–---------------------
Goto pfBlocker:LISTS
Create a new List
Alias Name : StopForumSpam
List Desc. : pfBlocker StopForumSpam
txt : /tmp/bannedips.txtList Action : Alias Only
Update Freq : Every 4 Hours
Save
–---------------------
Goto Firewall:Rules
Add Rules to utilize the Alias "StopForumSpam" in WAN as "Block" and LAN as "Reject"
–----------------------
Another list that uses .CSV format
http://www.autoshun.org/
http://www.autoshun.org/files/shunlist.csvSame process as above. Use the same script unless you want to create multiple scripts with
different Cron/pfBlocker tasks.#!/bin/sh
fetch http://www.stopforumspam.com/downloads/bannedips.zip
fetch http://www.autoshun.org/files/shunlist.csv
tar -xOf bannedips.zip | tr , '\n' > bannedips.txt
cat shunlist.csv | grep -o '^*[^[,]{0,15}' > shunlist.txt -
Made some changes to the script. If the download fails for what ever reason, it can clear the previous downloaded data leaving your table blank.
#!/bin/sh fetch http://www.stopforumspam.com/downloads/bannedips.zip returncode=$? echo $returncode if [ "$returncode" -eq "0" ]; then tar -xOf bannedips.zip | tr , '\n' > bannedips.txt fi fetch http://www.autoshun.org/files/shunlist.csv returncode=$? echo $returncode if [ "$returncode" -eq "0" ]; then cat shunlist.csv | grep -o '\^*[^[,]\{0,15\}' > shunlist.txt fi
Also, moved the script to /home folder as the /tmp folder doesn't persist after reboot …
Does anyone know if there is an upper limit on the total number of :
"Firewall Maximum Table Entries" in SYSTEM:ADVANCED:FIREWALL/NAT
-
Thanks for the contribution.
To keep this script on pfsense xml backup, I suggest you to install package filer instead of creating this script via console/ssl.
-
Thanks for the contribution.
To keep this script on pfsense xml backup, I suggest you to install package filer instead of creating this script via console/ssl.
Thanks Marcelloc,
I added the "Filer" Package. I had to rename the file to pffetch.sh to get Filer View/Edit working.
To make the script executable, you will still need to go manually run "chmod +x"? Or is there a work around?
How do you get the files to save in the xml backup file? I don't see any settings for that? Or is there a particular folder to create script files?
Thanks for your help.
ps - Next step is to get pfBlocker to pull spam ips from my local mail server… Spammustdie!!!
-
Thanks for the contribution.
To keep this script on pfsense xml backup, I suggest you to install package filer instead of creating this script via console/ssl.
Hi Marcelloc,
I installed "File manager" instead of the "Filer" package.
I now see the correct settings to manage the script from the GUI.
Much easier….. Thanks.
Question - What would be the best way to get a .txt file from a local server to pfSense?
-
Thanks for the contribution.
To keep this script on pfsense xml backup, I suggest you to install package filer instead of creating this script via console/ssl.
Marcelloc,
I have a script that is pulling my spammers IP address and collecting them into a .TXT file (CIDR Format). It is creating a file every 5 minutes and sending it to a file in pfSense
with SCP.Unfortunately pfBlocker is limited to once per hour.
Can this code be edited to say 5mins, 1hr, 4hrs, 12hrs? Can I edited this php file directly?
$cron=array('01hour' => 1, '04hours' => 4, '12hours' => 12, 'EveryDay' => 23);
Please let me know if you have any other suggestions on how to get it to update more frequently.
Thanks.
-
The little script in my first post needs to be updated to change the path to the user directory. As is, the Fetch (downloads) are going into
/etc/cron directory.
Please add
#!/bin/sh
cd /home/USERDIRECTORY (Or where ever you would like the downloads to go to. And also where pfBlocker will lookup.