IP-Blocklist

tommyboy180

I got it installed with a couple of lists, when I check syslog i always get this message: "root: IP-Blocklist was found not running", if i goto Firewall > IP-Blocklist status says its running with a number of networks blocked, any idea what the issue here is?

Thank you

I think I may know what the issue is. I'm still doing work on the package but the binary files are no longer in the repo. This will be fixed very soon!

n0vnm

How does one create an importable whitelist? I play Guild Wars and NC interactive is blocked with Level-1 blacklist. I want to whitelist the subnets for Guild Wars, but have ended up beating my head on a brick wall trying to import. I don't have sufficient skills to edit within the Linux shell. Any other suggestions?

NC INTERACTIVE, INC:206.127.144.0-206.127.159.255

Thanks!

ToxIcon

pfsense 2.0 R3
tommy can you fix the log for ipblocklist the
Enable Logging? does not stay enable once you click
save/update.

Also was just wondering if you can make a
dashboard widget that can display the log files for
ipblocklist and country block so that we
can see what is being block thanks. something like to one for snort
and havp.

knight313

pfsense 2.0 Rc3
built on Tue Jun 21 16:50:25 EDT 2011
Package does not block any IP.
System log (root: IP-Blocklist was found not running)

ToxIcon

Pfsense 2.0-RC3

tommyboy180 ip-blocklist will not update the list form iblocklist is it because they made some changes to the update url
like this address no longer seem to work http://iblocklist.aodcommunity.com/files/wujnnzxrgppvpdujdetm/bt_ads.gz
but if enter into a into a browser it downloads.
their new update format looks like this
http://list.iblocklist.com/?list=bt_ads&fileformat=p2p&archiveformat=gz
but when added to ip-blocklist it also does not download
their are no errors in the log for ip-blocklist.

Thanks tommyboy180 for all the great work and great packages

tommyboy180

@ToxIcon:

Pfsense 2.0-RC3

tommyboy180 ip-blocklist will not update the list form iblocklist is it because they made some changes to the update url
like this address no longer seem to work http://iblocklist.aodcommunity.com/files/wujnnzxrgppvpdujdetm/bt_ads.gz
but if enter into a into a browser it downloads.
their new update format looks like this
http://list.iblocklist.com/?list=bt_ads&fileformat=p2p&archiveformat=gz
but when added to ip-blocklist it also does not download
their are no errors in the log for ip-blocklist.

Thanks tommyboy180 for all the great work and great packages

http://iblocklist.taltos.org/files/wujnnzxrgppvpdujdetm/bt_ads.gz is the url for that file. To find the real url just download the file like normal in your browser and go back into your downloads and look at the url in your browser history.

I'm not sure how often the url scheme changes on that site so checking the url or saving it to a website you own may help with stability.

Thanks for your support! If you know php then perhaps we could work together on the project!

wm408

Tommyboy,

I have yet to upgrade to 2.0.x, as I will probably wait for final release (however I did test it and it looks great). This means I am still running 1.2.3.

I have recently worked with your latest, 0.3.2. As we've talked before in this thread, I do use the nanobsd (embedded) version and I do have a SATA 160GB drive mounted separately of the OS in my lil Soekris box. With the debugging/output that I think is now available (versus previous versions), I was able to modify all the scripts that were getting an error, not being able to write to the embedded file system, ie: read-only. Now they're all writing to my SATA drive and the IPblocklist package has been running for 72 hours without a crash. This is a lot better than before where the service would crash, or it would act silly, or whatever.

The files that needed to be written were files like: ipfw.ipfw, whitelist, and wlists.txt (i think thats all of them). (again once IPblocklist could write somewhere, it worked fine… maybe for embedded systems you could write it all to tmp? And then just have it reload them to tmp if the system gets reset?, not sure.), or where in the scripts it needs to write these files, invoke the /etc/rc.conf_mount_rw and then back /etc/rc.conf_mount_ro when done? (just a thought)

I am using a custom list that I created which basically just blocks 98% of all countries that I load from an internal webserver.

Maybe by now this whole thread includes this kind of information, or you probably already know.

I just wanted to say thanks. :)

tommyboy180

@wm408:

Tommyboy,

I have yet to upgrade to 2.0.x, as I will probably wait for final release (however I did test it and it looks great). This means I am still running 1.2.3.

I have recently worked with your latest, 0.3.2. As we've talked before in this thread, I do use the nanobsd (embedded) version and I do have a SATA 160GB drive mounted separately of the OS in my lil Soekris box. With the debugging/output that I think is now available (versus previous versions), I was able to modify all the scripts that were getting an error, not being able to write to the embedded file system, ie: read-only. Now they're all writing to my SATA drive and the IPblocklist package has been running for 72 hours without a crash. This is a lot better than before where the service would crash, or it would act silly, or whatever.

The files that needed to be written were files like: ipfw.ipfw, whitelist, and wlists.txt (i think thats all of them). (again once IPblocklist could write somewhere, it worked fine… maybe for embedded systems you could write it all to tmp? And then just have it reload them to tmp if the system gets reset?, not sure.), or where in the scripts it needs to write these files, invoke the /etc/rc.conf_mount_rw and then back /etc/rc.conf_mount_ro when done? (just a thought)

I am using a custom list that I created which basically just blocks 98% of all countries that I load from an internal webserver.

Maybe by now this whole thread includes this kind of information, or you probably already know.

I just wanted to say thanks. :)

Excellent. Would you be interested in helping write code and take this package to the next level? I'm getting more busy every day with normal day-to-day things and my school work. I'm actually in an Oracle class right now away from home. If you want to help out and dedicate a couple hours a week to this project that would super awesome.

Have you considered using Countryblock to block just countries? You should find it more stable for embedded installs. There are a couple hosting companies that are using countryblock and they have found it to be quite stable.

When I am able to find the time I will release a new version that will detect if embedded and put the files that must be written to and read from in /tmp.

wm408

Tommyboy,

Thanks for the coding offer. I just did a few, little hacks in your scripts in response to debugging output that was provided. I am taking Linux admin and some programming classes, but I am not uber like you. ;D

Let me know what you have in mind and maybe I can help, (pm ?)

As far as writing to tmp, that'd be a cool option I think. Maybe we could rig a cron job or something to help the IPblocklist reload after a power cycle.

And maybe the country block would have been a better choice. (doh).

tommyboy180

@wm408:

Tommyboy,

Thanks for the coding offer. I just did a few, little hacks in your scripts in response to debugging output that was provided. I am taking Linux admin and some programming classes, but I am not uber like you. ;D

Let me know what you have in mind and maybe I can help, (pm ?)

As far as writing to tmp, that'd be a cool option I think. Maybe we could rig a cron job or something to help the IPblocklist reload after a power cycle.

And maybe the country block would have been a better choice. (doh).

The latest version of ipblocklist reloads after a filter reload. It's actually really nice because you don't have to setup email notifications or a cron job anymore. It's stable now.
I do set the file system to read/write if on embedded but I guess the files need the same permission so /tmp is an excellent place.

I will be able to get to this update within the next couple of weeks. Thank you so much and when you feel comfortable with PHP send me a PM. I don't want to leave the package DEV scene but it just takes me a little longer to get updates out than users want.

wm408

Ahhh OK I see. After a filter reload sounds better.

Thanks!

@tommyboy180:

@wm408:

Tommyboy,

Thanks for the coding offer. I just did a few, little hacks in your scripts in response to debugging output that was provided. I am taking Linux admin and some programming classes, but I am not uber like you. ;D

Let me know what you have in mind and maybe I can help, (pm ?)

As far as writing to tmp, that'd be a cool option I think. Maybe we could rig a cron job or something to help the IPblocklist reload after a power cycle.

And maybe the country block would have been a better choice. (doh).

The latest version of ipblocklist reloads after a filter reload. It's actually really nice because you don't have to setup email notifications or a cron job anymore. It's stable now.
I do set the file system to read/write if on embedded but I guess the files need the same permission so /tmp is an excellent place.

I will be able to get to this update within the next couple of weeks. Thank you so much and when you feel comfortable with PHP send me a PM. I don't want to leave the package DEV scene but it just takes me a little longer to get updates out than users want.

tommyboy180

New IP-Blocklist poll at http://forum.pfsense.org/index.php/topic,39649.0.html

Everyone please vote on the issue. This directly affects the next release of the package. Thank you.

Woger

Hi,
I am new to pfsense but not completely new to freebsd. I have pfsense 2.0RC3 latest svn running with ip-blocklist installed. IP-blocklist seems to work fine. The problem I have is in the list. When I install sh-drop.tgz it installs about 430 IP's. If I install bt_hijjacked.gz only I get about 370 IP's. When I install both, I get just 70 IP's It seems that the script crashes as soon as it sees double IP ranges. I did read earlier about this and that doubles ones are completely removed, but it seems that the script stops completely.

I also get these errors:

[2.0-RC3][admin@pfsense.localdomain]/usr/local/www/packages/ipblocklist(39): ./convert-execute.sh
1 table deleted.
1 table deleted.
Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 63, <infile>line 469.
Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 36, <infile>line 469.
Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 36, <infile>line 469.
Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 63, <infile>line 469.
Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 36, <infile>line 469.
Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 36, <infile>line 469.
Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 63, <infile>line 469.
Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 36, <infile>line 469.
Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 36, <infile>line 469.
Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 63, <infile>line 469.
Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 36, <infile>line 469.
Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 36, <infile>line 469.
rm: /usr/local/www/packages/ipblocklist/lists/ipfw.ipfwTEMP: No such file or directory
rm: Wlists/whitelistTEMP: No such file or directory
rm: /tmp/rules.debug.tmp: No such file or directory
rm: /tmp/rules.debug.tmp: No such file or directory
0
1
2
3
4

Besides this: I am planning to run a few web and mailservers behind this pfsense machine. Which lists do you recommend to use?

Thanks,

Woger</infile></infile></infile></infile></infile></infile></infile></infile></infile></infile></infile></infile>

tommyboy180

70 seems like a really low number. What normally happens is the IP range is converted into CIDR and duplicates are removed. The number 70 doesn't mean 70 IP addresses it means 70 Networks since it's all converted to CIDR.

It's possible that the reading is correct. One way to see is to look at the ipfw.ipfw file. You can then see the raw list and determine why that reading is so low.

The next release will be CIDR native. This will eliminate weird stuff like this. If you want vote on the discussion to help me get a better understanding of what my userbase needs are at http://forum.pfsense.org/index.php/topic,39649.0.html

As far as lists go I recommend the spam, drop, and the forum spam list. Also I highly recommend running countryblock and blocking the major spamming countries.

Woger

Well it's 137 to be exactly. :)
The last line in ipfw.ipfw is 95.64.9.0/24 which is the second last line in sh_drop. The last line in sh_drop is DROP:95.64.98.0-95.64.99.255 . Exactly the same range is in bt_hijacked somewhere in the middle.
A wc -l on all three files gives:

358 bt_hijacked
139 ipfw.ipfw
469 sh_drop
Looks like it drops most off the lines during installation of the list files.

Woger

OK,
Probably found the problem.
both lists use a lot off the same ranges which are removed. This is because you use uniq -u in your convert script. The -u removes all duplicates. If I remove the -u it leaves 1 behind and I get 570 networks instead of 137. Is there a reason why you use the -u ?

Greetings,
Woger

tommyboy180

@Woger:

OK,
Probably found the problem.
both lists use a lot off the same ranges which are removed. This is because you use uniq -u in your convert script. The -u removes all duplicates. If I remove the -u it leaves 1 behind and I get 570 networks instead of 137. Is there a reason why you use the -u ?

Greetings,
Woger

Why wouldn't you use the unique switch?

kyronax

Hey everybody, first post here ;)

I've put together my first pfSense box and it's pretty sweet. It's running on a Supermicro X7SPA-HF-D525 booting off a 2GB USB drive. I did a "hard drive" install to the USB (not embedded because it didn't boot properly). http://www.dropbox.com/gallery/9284282/1/pfSense?h=5dead0

Anyways, I have a little problem with IP-Blocklist. I have 2 lists installed and 1 whitelist (Steam). Maybe it's just my stupidity but whenever I reboot the router it would "forget" about the whitelist and continues blocking Steam. I have to hit the Save/Update button to get it working again. I searched but couldn't find anything regarding this.

Thanks!

2.0-RC3 (i386) built on Tue Aug 16 19:48:02 EDT 2011
IP-Blocklist Package 0.3.2

Woger

@tommyboy180:

@Woger:

OK,
Probably found the problem.
both lists use a lot off the same ranges which are removed. This is because you use uniq -u in your convert script. The -u removes all duplicates. If I remove the -u it leaves 1 behind and I get 570 networks instead of 137. Is there a reason why you use the -u ?

Greetings,
Woger

Why wouldn't you use the unique switch?

uniq -u gets only the uniq lines from a file. So if there are double lines in a file these aren't uniq and will be discarded (as mentioned earlier in this thread). But without the -u switch, uniq will leave one of the double lines behind. For the 2 lists I use 358 lines in bt_hijacked and 469 lines in sh_drop there are more than 300 identical lines. With uniq -u these are all removed which ends in 139 lines in ipfw.ipfw. If I remove the -u switch the 300 identical lines are made uniq again which leaves more than 400 uniq lines in ipfw.ipfw.

hmm.. not very explaining ???. I'll try it like this:

file "test" containg these lines:
ip-1
ip-1
ip-2
ip-3
ip-3

uniq -u test > test2 will result in a file which contains
ip-2

uniq test > test2 will result in a file which contains
ip-1
ip-2
ip-3

tommyboy180

Well that is a major problem. I will hopefully push an update ASAP. Thanks for taking the time to explain it!