IP-Blocklist

tommyboy180

New IP-Blocklist poll at http://forum.pfsense.org/index.php/topic,39649.0.html

Everyone please vote on the issue. This directly affects the next release of the package. Thank you.

Woger

Hi,
I am new to pfsense but not completely new to freebsd. I have pfsense 2.0RC3 latest svn running with ip-blocklist installed. IP-blocklist seems to work fine. The problem I have is in the list. When I install sh-drop.tgz it installs about 430 IP's. If I install bt_hijjacked.gz only I get about 370 IP's. When I install both, I get just 70 IP's It seems that the script crashes as soon as it sees double IP ranges. I did read earlier about this and that doubles ones are completely removed, but it seems that the script stops completely.

I also get these errors:

[2.0-RC3][admin@pfsense.localdomain]/usr/local/www/packages/ipblocklist(39): ./convert-execute.sh
1 table deleted.
1 table deleted.
Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 63, <infile>line 469.
Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 36, <infile>line 469.
Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 36, <infile>line 469.
Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 63, <infile>line 469.
Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 36, <infile>line 469.
Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 36, <infile>line 469.
Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 63, <infile>line 469.
Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 36, <infile>line 469.
Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 36, <infile>line 469.
Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 63, <infile>line 469.
Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 36, <infile>line 469.
Use of uninitialized value in split at /usr/local/www/packages/ipblocklist/convert.pl line 36, <infile>line 469.
rm: /usr/local/www/packages/ipblocklist/lists/ipfw.ipfwTEMP: No such file or directory
rm: Wlists/whitelistTEMP: No such file or directory
rm: /tmp/rules.debug.tmp: No such file or directory
rm: /tmp/rules.debug.tmp: No such file or directory
0
1
2
3
4

Besides this: I am planning to run a few web and mailservers behind this pfsense machine. Which lists do you recommend to use?

Thanks,

Woger</infile></infile></infile></infile></infile></infile></infile></infile></infile></infile></infile></infile>

tommyboy180

70 seems like a really low number. What normally happens is the IP range is converted into CIDR and duplicates are removed. The number 70 doesn't mean 70 IP addresses it means 70 Networks since it's all converted to CIDR.

It's possible that the reading is correct. One way to see is to look at the ipfw.ipfw file. You can then see the raw list and determine why that reading is so low.

The next release will be CIDR native. This will eliminate weird stuff like this. If you want vote on the discussion to help me get a better understanding of what my userbase needs are at http://forum.pfsense.org/index.php/topic,39649.0.html

As far as lists go I recommend the spam, drop, and the forum spam list. Also I highly recommend running countryblock and blocking the major spamming countries.

Woger

Well it's 137 to be exactly. :)
The last line in ipfw.ipfw is 95.64.9.0/24 which is the second last line in sh_drop. The last line in sh_drop is DROP:95.64.98.0-95.64.99.255 . Exactly the same range is in bt_hijacked somewhere in the middle.
A wc -l on all three files gives:

358 bt_hijacked
139 ipfw.ipfw
469 sh_drop
Looks like it drops most off the lines during installation of the list files.

Woger

OK,
Probably found the problem.
both lists use a lot off the same ranges which are removed. This is because you use uniq -u in your convert script. The -u removes all duplicates. If I remove the -u it leaves 1 behind and I get 570 networks instead of 137. Is there a reason why you use the -u ?

Greetings,
Woger

tommyboy180

@Woger:

OK,
Probably found the problem.
both lists use a lot off the same ranges which are removed. This is because you use uniq -u in your convert script. The -u removes all duplicates. If I remove the -u it leaves 1 behind and I get 570 networks instead of 137. Is there a reason why you use the -u ?

Greetings,
Woger

Why wouldn't you use the unique switch?

kyronax

Hey everybody, first post here ;)

I've put together my first pfSense box and it's pretty sweet. It's running on a Supermicro X7SPA-HF-D525 booting off a 2GB USB drive. I did a "hard drive" install to the USB (not embedded because it didn't boot properly). http://www.dropbox.com/gallery/9284282/1/pfSense?h=5dead0

Anyways, I have a little problem with IP-Blocklist. I have 2 lists installed and 1 whitelist (Steam). Maybe it's just my stupidity but whenever I reboot the router it would "forget" about the whitelist and continues blocking Steam. I have to hit the Save/Update button to get it working again. I searched but couldn't find anything regarding this.

Thanks!

2.0-RC3 (i386) built on Tue Aug 16 19:48:02 EDT 2011
IP-Blocklist Package 0.3.2

Woger

@tommyboy180:

@Woger:

OK,
Probably found the problem.
both lists use a lot off the same ranges which are removed. This is because you use uniq -u in your convert script. The -u removes all duplicates. If I remove the -u it leaves 1 behind and I get 570 networks instead of 137. Is there a reason why you use the -u ?

Greetings,
Woger

Why wouldn't you use the unique switch?

uniq -u gets only the uniq lines from a file. So if there are double lines in a file these aren't uniq and will be discarded (as mentioned earlier in this thread). But without the -u switch, uniq will leave one of the double lines behind. For the 2 lists I use 358 lines in bt_hijacked and 469 lines in sh_drop there are more than 300 identical lines. With uniq -u these are all removed which ends in 139 lines in ipfw.ipfw. If I remove the -u switch the 300 identical lines are made uniq again which leaves more than 400 uniq lines in ipfw.ipfw.

hmm.. not very explaining ???. I'll try it like this:

file "test" containg these lines:
ip-1
ip-1
ip-2
ip-3
ip-3

uniq -u test > test2 will result in a file which contains
ip-2

uniq test > test2 will result in a file which contains
ip-1
ip-2
ip-3

tommyboy180

Well that is a major problem. I will hopefully push an update ASAP. Thanks for taking the time to explain it!

tommyboy180

Version 3.3 Released.

Fixed uniq -u bug
Removed email tab - no longer needed

Thank you Woger!

Woger

No thanks,
Glad I could help :)
Keep up the good work!

Shuey

When I try turning this on with one list added, I get the error:

/tmp/rules.debug:86: cannot load "/usr/local/www/packages/countryblock/lists/countries.txt": No such file or directory

I am running pfSense 2.0-RC3 (i386) Aug 18, 2011
I have package IP-Blocklist 0.3.3
I'm trying to use this list: http://www.tomschaefer.org/temp/pfsense/IP-Blocklist-ForumSpam.txt

I don't have the "country block" package installed at all, so I don't get why I'm seeing this error…

tommyboy180

Did you have the countryblock package installed at any point?

Shuey

@tommyboy180:

Did you have the countryblock package installed at any point?

Yes, and I could never get it to work properly. But it was uninstalled and I also upgraded pfSense from RC2 to RC3. I figured those two things would've fixed the problem, but pf obviously still sees some junk left over from CB.

What can I do?

mliesenfelt

I'm new to the forum. First I'd like to say thank you to tommyboy180 for his work and effort on the project.

I can confirm that ip-blocklist and country block do seem to interfere with each other and are not install-configure-run-uninstall-reinstall safe as Shuey mentioned. Also, the option for logging results seems to have amnesia.

I think this is a great project and I'm interesting in putting some time in to make it stable. Where is the source code for your branches and 3.3 beta located?

-Michael

tommyboy180

@mliesenfelt:

I'm new to the forum. First I'd like to say thank you to tommyboy180 for his work and effort on the project.

I can confirm that ip-blocklist and country block do seem to interfere with each other and are not install-configure-run-uninstall-reinstall safe as Shuey mentioned. Also, the option for logging results seems to have amnesia.

I think this is a great project and I'm interesting in putting some time in to make it stable. Where is the source code for your branches and 3.3 beta located?

-Michael

Send me a PM.

tommyboy180

Version 3.4 Released!

Whitelists are persistent though reboots
Logging stays checked when it's supposed to
Removed old Warning from main page
Added lvl1 list by default
Changed description on package list
Fixed i-frame error on main page

EDIT: The package has also gained several performance tweaks. The package uses less memory and processing time now.

Shuey

@Shuey:

@tommyboy180:

Did you have the countryblock package installed at any point?

Yes, and I could never get it to work properly. But it was uninstalled and I also upgraded pfSense from RC2 to RC3. I figured those two things would've fixed the problem, but pf obviously still sees some junk left over from CB.

What can I do?

I pulled up a command line using the shell option and I navigated to the "/usr/local/www/packages/" directory - there is no subfolder named "countryblock". The info from my error is being stored somewhere, but I don't know how to wipe it out so I can start over fresh.

Any help would be appreciated - thank you!

tommyboy180

@Shuey:

@Shuey:

@tommyboy180:

Did you have the countryblock package installed at any point?

Yes, and I could never get it to work properly. But it was uninstalled and I also upgraded pfSense from RC2 to RC3. I figured those two things would've fixed the problem, but pf obviously still sees some junk left over from CB.

What can I do?

it's looking for it in /tmp/rules.debug but before you edit this file I want to ask if you have rebooted your system since you uninstalled countryblock. If you haven't then do so and your problem should be solved.
I pulled up a command line using the shell option and I navigated to the "/usr/local/www/packages/" directory - there is no subfolder named "countryblock". The info from my error is being stored somewhere, but I don't know how to wipe it out so I can start over fresh.

Any help would be appreciated - thank you!

Shuey

That's the crazy thing… I uninstalled CountryBlock, updated my pfSense build and rebooted the system. I too was hoping this would fix the problem, but it hasn't. So, to be safe, I just now uninstalled the IP-Blocklist package, verified that it and the CountryBlock packages are no longer showing up in my list of installed packages. I rebooted the system, went back into packages, verified I still do not see a package for CountryBlock or IP-Blocklist. I installed IP-Blocklist and then enabled it. As soon as I saved the change, I got the error again :(