Country Block
-
;DpfBlocker 0.1.1 is out with: ;D
-
Countryblock features
-
Ipblocklist features
-
network block to CIDR internal feature to maintain compatibility with I-Blocklist free lists
-
many new options to choose what you want to block in and block out
This new package is awesome in my opinion!
TODO list:
-
code 'Update frequency' option together with cron package to stay away from file hacks
-
create a new tab for stats on what is blocked(maybe)
-
wait fixed pkg_edit.php in pfsense 2.0.1 or 2.1 to do not need a checkbox for rowhelper description
-
-
We also need a dashboard widget. We can query how many IP ranges in a pf table to display stats.
-
How countryblock widget works?
-
in /usr/local/www/widgets/widgets
Basically you get the number if entries in the table to get the IP ranges
pfctl -T show -t countryblock |grep -v -c Z
Here is my working widget for version 2.4 and IPblocklist.
@require_once("guiconfig.inc"); @require_once("pfsense-utils.inc"); @require_once("functions.inc"); $ipblocklist_file = "/usr/local/www/packages/ipblocklist/ipblocklist.php"; $countryblock_file = "/usr/local/www/packages/countryblock/countryblock.php"; echo "echo" "; if (file_exists($ipblocklist_file)) { echo " "; } if (file_exists($countryblock_file)) { echo " "; } echo" "; echo" "; if (file_exists($ipblocklist_file)) { echo " "; } if (file_exists($countryblock_file)) { echo " "; } echo" "; echo" "; if (file_exists($ipblocklist_file)) { ob_start(); $resultsIP = exec("tail -r /usr/local/www/packages/ipblocklist/errorOUT.txt"); ob_end_clean(); if ($resultsIP == "") { echo " "; } else { echo " "; } } if (file_exists($countryblock_file)) { ob_start(); $resultsCB = exec("tail -r /usr/local/www/packages/countryblock/errorOUT.txt"); ob_end_clean(); if ($resultsCB == "") { echo " "; } else { echo " "; } } echo" "; //echo" "; //if (file_exists($ipblocklist_file)) echo " "; //if (file_exists($countryblock_file)) echo " "; //echo" "; echo" | IP-Blocklist"; ob_start(); $results = exec("/sbin/pfctl -s rules | grep -c ipblocklist"); ob_end_clean(); if ($results > '2') echo "![](/themes/{$g[)"; else echo "![](/themes/{$g[)"; echo " | Countryblock"; ob_start(); $results = exec("/sbin/pfctl -s rules | grep -c countryblock"); ob_end_clean(); if ($results > '0') echo "![](/themes/{$g[)"; else echo "![](/themes/{$g[)"; echo " | | "; ob_start(); $resultsIP = exec("/sbin/pfctl -T show -t ipblocklist |grep -v -c Z"); ob_end_clean(); echo $resultsIP; echo " Networks"; echo " | "; ob_start(); $resultsCB = exec("pfctl -T show -t countryblock |grep -v -c Z"); ob_end_clean(); echo $resultsCB; echo " Networks"; echo " | | | "; echo " "; echo $resultsIP; echo ""; echo " | | "; echo " "; echo $resultsCB; echo ""; echo " | | | | ";
It's very basic.
-
Cool,
I'll take a look.
Maybe widget will be better then stats gui.
-
Errors:
topspammers Tab -> /usr/local/pkg/pfblocker_topspammers.xml does not exists on line 175I think top pfblocker_topspammers.xml was left out of pfblocker.xml
<additional_files_needed>http://www.pfsense.org/packages/config/pf-blocker/pfblocker_topspammers.xml <prefix>/usr/local/pkg/</prefix> <chmod>0755</chmod></additional_files_needed>
Also do we want to limit the ipblocklist functionality to CIDR lists only? I took a poll on the package forums asking users if they wanted CIDR only or to have the normal PG2 lists. If we use CIDR only then we can eliminate future problems.
I was talking a look at your pfblocker_Range2CIDR function and it looks good but it's a long process for very large files like Level1 and Level2 lists.
-
Almost all free lists from ip-blocked are p2p, so I think we can suggest CIDR but keep p2p feature.
I will fix missing file and test rpc sync.
-
topspammers call fixed.
Can you test how many time does rules apply takes when you do not change urls?
-
iblocklist.com lets your download all lists in p2p, dat, and CIDR now. I talked with fakhir not that long ago about the ip-blocklist package and moving to CIDR.
-
when I choose cidr format, all lists become 'subscription'.
-
Copy the list URL and manually change the fileformat to "cidr". It works for all links.
If we can figure out how to get fetch to follow links then our users won't have to input the direct URL anymore. -
But there are other compatible lists, and custom lists can be done too.
I understand that CIDR is better.
I've tested level1 file and both(p2p and CIDR) gets errors while appying to pfsense.
The block2CIDR works very fine, the extra time you see is downloading time. I did a lot of tests before including it to pfBlocker.
-
I didn't think about that. You're right.
I'll see if I can figure out why large lists are failing. -
The large lists take up the default memory space. I was able to overcome this in IPblocklist version 3.5 by adding:
set limit table-entries 900000
I put this at the top of rules.debug. You can enter any amount that you think will be big enough but 900000 should cover all scenarios.
Hopefully there is a smooth way to add this option to the rules.debug file. -
The large lists take up the default memory space. I was able to overcome this in IPblocklist version 3.5 by adding:
set limit table-entries 900000
I put this at the top of rules.debug. You can enter any amount that you think will be big enough but 900000 should cover all scenarios.
Hopefully there is a smooth way to add this option to the rules.debug file.I have no idea how to do this, but I will take a look on it.
For a stand alone use, all my tests went fine. tomorrow I will check sync problems.All tests with pfBlocker, including sync between pfsenses boxes were sucessfull.
Did you liked the new options in gui? It was what you were thinking?
-
Great work!! I am having an issue tho: i'm getting this error with i click on the top spammers tab:
Warning: fopen(/usr/local/pkg/pfblocker_topspammers.xml): failed to open stream: No such file or directory in /etc/inc/xmlparse.inc on line 175
Warning: Invalid argument supplied for foreach() in /usr/local/www/pkg_edit.php on line 418
I did go thru and removed every entry of countryblock and pfblocker in my config and file system before doing installing the package again.. same results
running:
2.1-DEVELOPMENT (i386)
built on Tue Sep 13 17:28:43 EDT 2011
FreeBSD 8.1-RELEASE-p4P.S my last git sync was this morning
-
Reinstall the package, there was a missing file in pfBlocker.xml
I've fixed this few hours ago. Btw i'll test this in a vm.
-
I did another re-install. Same results.. Let me know how your test turns out… Wondering if its because i'm running 2.1
-
I did another re-install. Same results.. Let me know how your test turns out… Wondering if its because i'm running 2.1
Does 2.1 uses same pkg repo as 2.0?
-
I did another re-install. Same results.. Let me know how your test turns out… Wondering if its because i'm running 2.1
Does 2.1 uses same pkg repo as 2.0?
as far as i know, yes
-
I found the error.
I'm fixing it right now.
-
I found the error.
I'm fixing it right now.
that did it.. Thank you!!! Only using the country blocking right now, over the weekend, i'll give the ip-list side a tried
-
pfBlocker 0.1.2 is out.
Now with widget.
-
I we can't figure out how to increase the table size limits for large lists like level1 then perhaps we can break the lists into multiple tables.
Example:
table limit will be 10,000 entries. More entries will go into another table. So if Level1 has 35,000 lines the we will have 3 tables-
pfblocker - 10000 lines
-
pfblocker1 - 10000 lines
-
pfblocker2 - 10000 lines
-
pfblocker3 - 5000 lines
What do you think? I hope we can find a way to increase the memory limit.
-
-
Could be a solution for this problem.
I'm still looking for a way to do it without multiple tables or file hacks.
Does it happens on x64 versions with for example 4g ram?
And thanks for widget clue. :)
-
I've included all countries and levels 1,2,3 plus some ads, virus, etc
total:449418 Networks
no erros.
my system is x64 with more then 4g ram.
-
The problem is it isn't a RAM limitation. It's a memory max allocation variable that is small to begin with. So no matter how much RAM you have on x86 or x64 you will get that error once that limitation is hit.
This is why it is a problem. We need to be able to raise that limit other wise the functionality will be greatly limited, leaving users to fall back on IP-Blocklist.
-
449418 Networks seems very small for level1, 2, and 3 with others. Are you sure those lists were all successful?
-
Could we include in GUI an option to increase this value and advise users that its unsupported?
EDIT:
wc -l in level files downloaded in cidr from fetch:
245749 level1
85703 level2
18753 level3
350205 total -
kk. We could give the users the option. It would just be inserting that text every time that file gets recreated. I've already done most of the work in IP-Blocklist and Countryblock.
/usr/local/pkg/pf/pf-blocker.sh
#!/bin/sh export t=`grep -n 'User Aliases' /tmp/rules.debug |grep -o '[0-9]\{1,2\}'` t=$(($t+'1')) #Insert table-entry limit /usr/bin/sed -i -e '/900000/d' /tmp/rules.debug while read line do a=$(($a+1)); #echo $a; if [ "$a" = "$t" ]; then echo "" >> /tmp/rules.debug.tmp echo "set limit table-entries 900000" >> /tmp/rules.debug fi echo $line >> /tmp/rules.debug done < "/tmp/rules.debug"
This code isn't tested but it's on the right track. We will have to have it check if the user has enabled this feature. I do this is IP-Blocklist and countryblock. Basically if the user enables it the package will create a file. If it's disabled the the file will be deleted. The script above will check if the file exists```
if [ -e FILE ] ....code here -
ok, I think its good enough to prevent errors until some 'core developer' could help us.
I'll try to put it in php way for most code compatibility.
The last thing to finish pfBlocker is cron package integration.
-
Cron for the updates?
-
use cron package instead of editing /etc/cront for lists updates…
-
Great work so far on the pfblocker package. I currently have a problem with it. I had countryblock installed and working previously. I removed that package and then installed the pfblocker package. The widget says that it is blocking networks but it comes up with a fail to load list message in the syslog.
Oct 29 01:38:43 php: : The command '/sbin/pfctl -o basic -f /tmp/rules.debug' returned exit code '1', the output was 'no IP address found for 1139.82.0.0/16 /tmp/rules.debug:16: file "/var/db/aliastables/pfBlockerInbound.txt" contains bad data no IP address found for 1139.82.0.0/16 /tmp/rules.debug:18: file "/var/db/aliastables/pfBlockerOutbound.txt" contains bad data pfctl: Syntax error in config file: pf rules not loaded'
Oct 29 01:38:43 php: : New alert found: There were error(s) loading the rules: no IP address found for 1139.82.0.0/16 /tmp/rules.debug:16: file "/var/db/aliastables/pfBlockerInbound.txt" contains bad data no IP address found for 1139.82.0.0/16 /tmp/rules.debug:18: file "/var/db/aliastables/pfBlockerOutbound.txt" contains bad data pfctl: Syntax error in config file: pf rules not loaded The line in question reads [16]: table <pfblockerinbound>persist file "/var/db/aliastables/pfBlockerInbound.txt"
Oct 29 01:38:43 php: : There were error(s) loading the rules: no IP address found for 1139.82.0.0/16 /tmp/rules.debug:16: file "/var/db/aliastables/pfBlockerInbound.txt" contains bad data no IP address found for 1139.82.0.0/16 /tmp/rules.debug:18: file "/var/db/aliastables/pfBlockerOutbound.txt" contains bad data pfctl: Syntax error in config file: pf rules not loaded - The line in question reads [16]: table <pfblockerinbound>persist file "/var/db/aliastables/pfBlockerInbound.txt"This is what happens when i only try to block countries and set none on the list block. I can try inbound or outbound and still get same error. I have tried reinstalling it. Do you know what it might be?</pfblockerinbound></pfblockerinbound>
-
Post here the countries you selected.
I will check the script.
-
I tired various countries, I get the same result when I select whitelist. I get the same result with all the countries. I have selected one from top spammers and one from south america. I also noticed that when i select one country from a tab then i can not unselect it. I can only select other ones. Using Chrome.
Oct 29 02:07:23 php: : The command '/sbin/pfctl -o basic -f /tmp/rules.debug' returned exit code '1', the output was 'no IP address found for 192.168.81.0/2411.0.1.0/24 /tmp/rules.debug:16: file "/var/db/aliastables/pfBlockerWL.txt" contains bad data pfctl: Syntax error in config file: pf rules not loaded'
Oct 29 02:07:23 php: : New alert found: There were error(s) loading the rules: no IP address found for 192.168.81.0/2411.0.1.0/24 /tmp/rules.debug:16: file "/var/db/aliastables/pfBlockerWL.txt" contains bad data pfctl: Syntax error in config file: pf rules not loaded The line in question reads [16]: table <pfblockerwl>persist file "/var/db/aliastables/pfBlockerWL.txt"
Oct 29 02:07:23 php: : There were error(s) loading the rules: no IP address found for 192.168.81.0/2411.0.1.0/24 /tmp/rules.debug:16: file "/var/db/aliastables/pfBlockerWL.txt" contains bad data pfctl: Syntax error in config file: pf rules not loaded - The line in question reads [16]: table <pfblockerwl>persist file "/var/db/aliastables/pfBlockerWL.txt"The rules and alias appear to be created ok.</pfblockerwl></pfblockerwl>
-
I found the error.
It's fixed now.
-
tommyboy,
Instead of editing /tmp/rules.debug, i will 'patch' filter.inc file with table-entries when clients choose to increase table.
I will also write some code to alert user when table gets larger then default limit.
-
That sounds like a good solution. We can use sed to insert the line to 'patch' and remove the line to 'unpatch'
-
I don't believe this:
Look what i found in filter.inc after coding everything…
set limit table-entries {$config['system']['maximumtableentries']}
it was all time in front of our noses :D
I'll fix the code and relase a new version..