IP-Blocklist
-
Hi tommyboy!
I am VERY happy to report that I am no longer seeing any issues between IPBlockList, CountryBlock and HAVP! THANK YOU!
Quick Q: will the URLs from iblocklist work in their regular format (http://list.iblocklist.com/?list=ynkdjqsjyfmilsgbogqf), or do we have to point to the .gz file directly? I ask because I had a bunch of direct links to the gz files, and IPBL showed 103000 blocked networks… I replaced them all with the /?listname links, and now I am showing only 1978 blocked IPs/Networks.
Also, how can I set up IPBL so it will automatically update the lists every night?
Thanks!
Glad to hear it!
You do have to directly link to the lists. Perhaps in the future I may find a way around this.
To have the package update your lists every night you can edit the cron executable which is located at /usr/local/etc/rc.d/IP-Blocklist.shJust comment out lines 3-14 and it should be good. Setup a cron job to run this executable whenever you like. Use the cron job package for that.
-
Thanks for the quick reply!
What is the best repository to get at the gz files directly? is it this? http://list11.iblocklist.com/files/
Do you have a preferred source, and also what are your favorite lists?
Glad to hear it!
You do have to directly link to the lists. Perhaps in the future I may find a way around this.
To have the package update your lists every night you can edit the cron executable which is located at /usr/local/etc/rc.d/IP-Blocklist.shJust comment out lines 3-14 and it should be good. Setup a cron job to run this executable whenever you like. Use the cron job package for that.
-
Thanks for the quick reply!
What is the best repository to get at the gz files directly? is it this? http://list11.iblocklist.com/files/
Do you have a preferred source, and also what are your favorite lists?
Glad to hear it!
You do have to directly link to the lists. Perhaps in the future I may find a way around this.
To have the package update your lists every night you can edit the cron executable which is located at /usr/local/etc/rc.d/IP-Blocklist.shJust comment out lines 3-14 and it should be good. Setup a cron job to run this executable whenever you like. Use the cron job package for that.
I didn't know that there was a directory listing! I usually download the list and go back to my download history and copy the address. I am working on a way to upload lists but I won't have that out for some time. I usually use iblocklist.com for most of my lists.
My favorite lists include the following:
-
http://iblocklist.dbnservers.net/files/bt_spyware.gz
-
http://withhorns.com/files/ficutxiwawokxlcyoeye.gz
-
http://iblocklist.dchubad.com/files/ghlzqtqxnzctvvajwwag.gz
-
http://list.iblocklist.com/files/sh_drop.gz
-
http://www.tomschaefer.org/temp/pfsense/IP-Blocklist-ForumSpam.txt
-
http://iblocklist.dchubad.com/files/bt_ads.gz
-
http://www.tomschaefer.org/temp/pfsense/MISC-Block.txt
The TomSchaefer.org ones are my custom lists that I made.
-
-
Whoa… check this out:
[12-31-10 16:12:19]root@/usr/local/www/packages/ipblocklist/lists# lh total 32776 -rw-r--r-- 1 root wheel 88K Dec 31 16:06 ?list=tor -rw-r--r-- 1 root wheel 131K Dec 30 17:00 bt_ads -rw-r--r-- 1 root wheel 969K Dec 30 17:00 bt_badpeers -rw-r--r-- 1 root wheel 6.8K Dec 30 17:00 bt_dshield -rw-r--r-- 1 root wheel 12M Dec 30 17:00 bt_level1 -rw-r--r-- 1 root wheel 19K Mar 21 2010 bt_webexploit-forumspam -rw-r--r-- 1 root wheel 1.7K Dec 30 17:00 dcha_faker -rw-r--r-- 1 root wheel 5.4K Dec 30 17:00 dcha_hacker -rw-r--r-- 1 root wheel 111K Dec 30 17:00 dcha_pedophiles -rw-r--r-- 1 root wheel 6.9K Dec 30 17:00 dcha_spammer -rw-r--r-- 1 root wheel 3.8M Dec 31 16:09 ipfw.ipfw -rw-r--r-- 1 root wheel 11K Dec 28 17:00 sh_drop -rw-r--r-- 1 root wheel 11K Dec 19 17:00 tbg_hijacked -rw-r--r-- 1 root wheel 15M Dec 28 17:00 tbg_primarythreats -rw-r--r-- 1 root wheel 6.6K Dec 30 17:00 ynkdjqsjyfmilsgbogqf [12-31-10 16:12:30]root@/usr/local/www/packages/ipblocklist/lists# cat ?list=tor # List distributed by iblocklist.com The Onion Router:2.36.33.51-2.36.33.51 The Onion Router:8.17.81.25-8.17.81.25 The Onion Router:8.24.61.246-8.24.61.246 ...Looks like the /?list= links DO somewhat work… weird, huh?
-
Whoa… check this out:
[12-31-10 16:12:19]root@/usr/local/www/packages/ipblocklist/lists# lh total 32776 -rw-r--r-- 1 root wheel 88K Dec 31 16:06 ?list=tor -rw-r--r-- 1 root wheel 131K Dec 30 17:00 bt_ads -rw-r--r-- 1 root wheel 969K Dec 30 17:00 bt_badpeers -rw-r--r-- 1 root wheel 6.8K Dec 30 17:00 bt_dshield -rw-r--r-- 1 root wheel 12M Dec 30 17:00 bt_level1 -rw-r--r-- 1 root wheel 19K Mar 21 2010 bt_webexploit-forumspam -rw-r--r-- 1 root wheel 1.7K Dec 30 17:00 dcha_faker -rw-r--r-- 1 root wheel 5.4K Dec 30 17:00 dcha_hacker -rw-r--r-- 1 root wheel 111K Dec 30 17:00 dcha_pedophiles -rw-r--r-- 1 root wheel 6.9K Dec 30 17:00 dcha_spammer -rw-r--r-- 1 root wheel 3.8M Dec 31 16:09 ipfw.ipfw -rw-r--r-- 1 root wheel 11K Dec 28 17:00 sh_drop -rw-r--r-- 1 root wheel 11K Dec 19 17:00 tbg_hijacked -rw-r--r-- 1 root wheel 15M Dec 28 17:00 tbg_primarythreats -rw-r--r-- 1 root wheel 6.6K Dec 30 17:00 ynkdjqsjyfmilsgbogqf [12-31-10 16:12:30]root@/usr/local/www/packages/ipblocklist/lists# cat ?list=tor # List distributed by iblocklist.com The Onion Router:2.36.33.51-2.36.33.51 The Onion Router:8.17.81.25-8.17.81.25 The Onion Router:8.24.61.246-8.24.61.246 ...Looks like the /?list= links DO somewhat work… weird, huh?
Yeah but inspect the file. Make sure they are real lists with IPs.
EDIT. I see you ran cat on one, so I guess you're right. I have found that it doesn't work on some links so just be careful. It's probably a good idea to go direct just to ensure you really are getting the right content. -
About the CRON job… are these the lines I have to comment out?
#check if ipblocklist running export resultr=`pfctl -s rules | grep -c ipblocklist` #echo $resultr if [ "$resultr" -gt "0" ]; then echo running exit 1 else echo not running /usr/bin/logger -s "IP-Blocklist was found not running" echo "IP-Blocklist not running" | /usr/local/bin/php /usr/local/www/packages/ipblocklist/email_send.php fiAfter they get commented, I make a copy of the script and schedule that one? (I am kinda UNIX n00b :P)
So, my script now looks like so:
#!/bin/sh ####check if ipblocklist running #export resultr=`pfctl -s rules | grep -c ipblocklist` ####echo $resultr #if [ "$resultr" -gt "0" ]; then # echo running # exit 1 #else # echo not running # /usr/bin/logger -s "IP-Blocklist was found not running" # echo "IP-Blocklist not running" | /usr/local/bin/php /usr/local/www/packages/ipblocklist/email_send.php #fi ...and I saved that one with a different name, (I don't know if commenting those line on the original script will break the package, so I made a copy and set it to executable)
[12-31-10 16:28:13]root@/usr/local/etc/rc.d# lh | grep IP -rwxr-xr-x 1 root wheel 4.6K Dec 31 09:27 IP-Blocklist.sh -rwxr-xr-x 1 root wheel 4.6K Dec 31 16:25 IP-Blocklist_cron.sh -
Yeah. Make a copy, remove each of the lines you quoted and schedule your copy.
What lists change every day? I'm curious.
-
One last question:
The CRONTAB format, if I want the lists to be updated every day @ 3:30 AM:
30 */03 * * * root /usr/local/etc/rc.d/IP-Blocklist_cron.shDoes this look OK to you, sir?
-
I would move the script out of rc.d so it doesn't run on startup. That way you don't have anything conflicting with the other startup script.
Other than that everything looks good! -
tommyboy180
Kill table and start over
Uncheck checkbox click save
pfctl -s rules | grep ipblocklist no output
Reload firewall filter
edit /usr/local/www/packages/ipblocklist/interfaces.txt "any"
/usr/local/www/packages/ipblocklist/convert-execute.sh no errors and warnings
Current Status = Running
You are blocking 0 Networks/IPs
yes running countryblock working
-
tommyboy180
Kill table and start over
Uncheck checkbox click save
pfctl -s rules | grep ipblocklist no output
Reload firewall filter
edit /usr/local/www/packages/ipblocklist/interfaces.txt "any"
/usr/local/www/packages/ipblocklist/convert-execute.sh no errors and warnings
Current Status = Running
You are blocking 0 Networks/IPs
yes running countryblock working
Without being able to inspect your system it looks like convert-execute.sh isn't running the commands correctly.
On countryblock when it runs a program it calls for it without direct path. For example a line might read, "pfctl -t countryblock -T kill"
However on ipblocklist when it runs a program it calls for it via its direct path. For example a line might read, "/sbin/pfctl -t ipblocklist -T kill"I think this is why you're not seeing any warnings or output. Attached is a replacement BASH script file that you should try. See if that fixes the issue.
Warning For other users downloading this replacement BASH script this may break your package. This is for troubleshooting only.
-
Using Multi pal Nat Routes,
WAN
LAN
OPT1 - Running a Hot Public WiFi zone, the previously attached replacement bash script fixed the issues for me.for the index.html?list=bt_level1, style files, I created a work around for my own use age, if this helps anyone:
I made a Sub directory on my web sever (Cloud hosted) and restricted this access down to my pfSense gateway IP.
created the following bash script.:cd /var/www/pg2
rm -rf bt_level1.gz && wget http://list.iblocklist.com/?list=bt_level1 && mv index.html?list=bt_level1 bt_level1.gz
rm -rf bt_level2.gz && wget http://list.iblocklist.com/?list=bt_level2 && mv index.html?list=bt_level2 bt_level2.gz
rm -rf bt_level3.gz && wget http://list.iblocklist.com/?list=bt_level3 && mv index.html?list=bt_level3 bt_level3.gzscheduled this to run on a weekly basis, and then have pfSense query the Remote Web server,
but there is no reason why this can't be changed to function local on the pfSense box.
Have it fetch the index.html?list=bt_level1 file, store it locally on the web-server, rename the file, and then re-query it's own web-sever from the ipblocklist interface, which should end up in the index.html?list=bt_level1 style links working without the need for the intermediate server, however for my needs I prefer to have the mirror pre-fetching occurring in a hosted environment.If there is any interest I'll happily attempt to create something to run this all locally on the pfSense Box.
-
If only pfsense came with wget to begin with then you wouldn't have to do this. Since fetch doesn't work well with these dynamic links it becomes a problem. Maybe I will consider adding wget after I get file uploads going.
-
I came across your forum after reviewing our server logs at Country IP Blocks. We noticed the discussion included Country IP Blocks Access Control Lists in a PG2 format. As a courtesy we have created these lists from our Country IP Blocks Database. The data is in a format that looks like this:
AZERBAIJAN:46.18.64.0-46.18.71.255
…
UNITED STATES:3.0.0.0-3.255.255.255
...
CHINA:1.12.0.0-1.15.255.255A reminder, our database is updated daily.
We can output our data in any format needed.
You can get your PG2 formatted list here: http://www.countryipblocks.net/networking/pg2-formatted-acess-control-lists/
Thanks,
Stewart White
-
@countryipblocks:
I came across your forum after reviewing our server logs at Country IP Blocks. We noticed the discussion included Country IP Blocks Access Control Lists in a PG2 format. As a courtesy we have created these lists from our Country IP Blocks Database. The data is in a format that looks like this:
AZERBAIJAN:46.18.64.0-46.18.71.255
…
UNITED STATES:3.0.0.0-3.255.255.255
...
CHINA:1.12.0.0-1.15.255.255A reminder, our database is updated daily.
We can output our data in any format needed.
You can get your PG2 formatted list here: http://www.countryipblocks.net/networking/pg2-formatted-acess-control-lists/
Thanks,
Stewart White
Excellent! Thank you for taking the time to post!
-
Excellent! Thank you for taking the time to post!
You're welcome. We'll keep the data flowing, just let us know your needs.
-
@countryipblocks:
Excellent! Thank you for taking the time to post!
You're welcome. We'll keep the data flowing, just let us know your needs.
I had a question for you. What do you think about the countryblock package for pfsense? It downloads countrylists from your site directly. A while ago we had a small discussion over the semi automated process and your rules within the FAQ. At the time we determined that it does not break your rules, however now that you're here you could comment and give me a final say.
-
I had a question for you. What do you think about the countryblock package for pfsense? It downloads countrylists from your site directly. A while ago we had a small discussion over the semi automated process and your rules within the FAQ. At the time we determined that it does not break your rules, however now that you're here you could comment and give me a final say.
I appreciate the question. I haven't seen your pfsense package yet. Could you provide me with a little info?
We arecurrently in the process of deciding how we want to deliver our, the formats, etc. One of the issues we are struggling with is the automated connections. When we started the website in 2007 we decided not to allow users to use automated tools to update. Our concern was an increase in traffic without the benefit of having the traffic originate through the site pages.
In 2008, I decided to change this rule and allow reasonable automation. We now get thousands of automated requests each day. I feel this is a good thing for those who use our data, but still leaves us paying exorbitant server and network costs in order to handle the growth.
We responded to these increased costs by placing some limited display ads on the website, but of course if the site is bypassed and the data download is automated the ads are missed.
In your case I think we can probably offer our assistance. We would appreciate credits and links back to the website.
We could probably collaborate on the project and create data expressly for your needs.
We recently added change logs to the site. This enables users to make daily modifications as opposed to downloading all or most of the countries.
Let's talk about what you would like to accomplish.
-
@countryipblocks:
I had a question for you. What do you think about the countryblock package for pfsense? It downloads countrylists from your site directly. A while ago we had a small discussion over the semi automated process and your rules within the FAQ. At the time we determined that it does not break your rules, however now that you're here you could comment and give me a final say.
I appreciate the question. I haven't seen your pfsense package yet. Could you provide me with a little info?
We arecurrently in the process of deciding how we want to deliver our, the formats, etc. One of the issues we are struggling with is the automated connections. When we started the website in 2007 we decided not to allow users to use automated tools to update. Our concern was an increase in traffic without the benefit of having the traffic originate through the site pages.
In 2008, I decided to change this rule and allow reasonable automation. We now get thousands of automated requests each day. I feel this is a good thing for those who use our data, but still leaves us paying exorbitant server and network costs in order to handle the growth.
We responded to these increased costs by placing some limited display ads on the website, but of course if the site is bypassed and the data download is automated the ads are missed.
In your case I think we can probably offer our assistance. We would appreciate credits and links back to the website.
We could probably collaborate on the project and create data expressly for your needs.
We recently added change logs to the site. This enables users to make daily modifications as opposed to downloading all or most of the countries.
Let's talk about what you would like to accomplish.
I sent you a PM with more info. Thank you!
-
Troubleshooting guide:
Some people have been having issues getting the package to run. Before you think you need to rebuild pfsense try a simple test first.
Go to Firewall -> Rules and pick a random entry. Click the edit button. Don't make any changes! Now click save.
Go back to your package. Ensure Enable is checked and Click save. Your package should be running now.