PfBlocker
-
It looks like I am having the "cannot allocate memory" issue as well, running a 2.0.1 (i386) release, nanobsd 2G and pfBlocker 1.0.1. I increased the max table size, read page 21 as well but it just wouldn't cut it with my small platform.
Since I REALLY wanted to get the Bluetack's level1 blocklist, I needed to try something else and eventually found another solution…
What I did is write a simple perl script with that will read the list, and output a new CIDR list splitted into many files, actually 100,000 entries per file.
I then created a small cron job to download the list and execute the script on a linux server running Apache. Finally, I configured three different blocklists (aliases) under pfblocker (not just a big one with three URLs...). And it works! Maybe having such a mechanism (splitting big files) built into pfblocker could be useful for some.
In case someone is interested by the perl script, it looks like this:
#!/usr/bin/perl use Net::CIDR::Lite; my $filenum = "0" x 4; # if you ++ a string, it keeps the padding sub new_file { $filenum ++; my $name = "splat_$filenum.lis"; open OUT, ">$name" or die "canne open $name cap'n:$!\n"; warn "writing to:$name\n"; } my $cidr = Net::CIDR::Lite->new; open (MYFILE, $ARGV[0]); while (<myfile>) { chomp; $_ =~ /[^:]+:(.*)/; my $range = $1; #extracted IP Range, verify it is IPv4 if ( $range =~ m/\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}\-\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}/i ) { $cidr->add_range($range); } } close (MYFILE); my $index = 0; my @cidr_list = $cidr->list; foreach my $block ( @cidr_list ) { if ( $index % 100000 == 0 ) { new_file; } print OUT $block,"\n"; $index++; } close (OUT);</myfile>It receives the unzipped list in input, and will output the files in the CWD. Simple as that.
Have a nice day.
-
Good contributon. Thank you. :)
-
Just updated pfBlocker to 1.0.2 with:
-
Fix on array check error at line 368 when there is no alias defined on pfSense
-
reduce duplicate cases on automatic rules when using multiple interfaces as inbound and/or outbound
-
Increase php memory limit to 250Mb when x64 pfSense is detected(DO AT YOUR OWN RISK PATCH applied to code ;))
-
Updated country ip lists
-
-
Awesome!
-
-
Thanks for the update: I am gonna give it a shot like right after I post this.
I noticed that my script would fail when some blocklist would include multiple colons on one line. Here is my updated script, which now accept a number of lines as the second argument.
#!/usr/bin/perl use Net::CIDR::Lite; my $filenum = "0" x 4; # if you ++ a string, it keeps the padding my $limit = 100000; # default max number of lines sub new_file { $filenum ++; my $name = "splat_$filenum.lis"; open OUT, ">$name" or die "canne open $name cap'n:$!\n"; warn "writing to:$name\n"; } my $cidr = Net::CIDR::Lite->new; open (MYFILE, $ARGV[0]); if ( defined($ARGV[1])) { if ( $ARGV[1] =~ m/^\d{2,6}$/ ) { $limit = int($ARGV[1]); } } while (<myfile>) { chomp; my @line = split(/:+/); my $range = $line[-1]; #get IP Range, verify it is IPv4 if ( $range =~ m/^\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}\-\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}$/ ) { $cidr->add_range($range); } } close (MYFILE); my $index = 0; my @cidr_list = $cidr->list; foreach my $block ( @cidr_list ) { if ( $index % $limit == 0 ) { close (OUT); new_file; } print OUT $block,"\n"; $index++; } close (OUT);</myfile>Since my platform is not x64, and only have 256MB of RAM, I am not sure the new patch will fix the memory allocation issue for me… I am running with 60% memory used on average. Right now I am using a 60,000 lines as my maximum. 100,000 would seem to fail on some occasions.
-
Vlurk,
The memory patch is only for am64.
As we try to avoid file hacks, I've applied a value that is defined on config.inc but not reflected on gui.
As you have few memory available, the best option is the way you are doing or of course a hardware upgrade.Thank's for the script update.
-
I know Rules ReOrdering after a pfBlocker change has been covered in this thread.
I'd like to bring it back up because it's making me crazy.Here's my situation.
I use the pfBlocker widget. I also have my rules customized and ordered a certain way.In the last pfBlocker ver., I'd set every Action to Deny Inbound.
Next I'd customize and reorder the auto-created rules. I'd be finished in 10 min or so.
I'm pretty sure pfBlocker automatically changed Action to Alias when I had adjusted the rules.
The end result was the rules wouldn't change after an update.In this latest pfBlocker ver., my last method doesn't work. I have to set action to Alias myself.
If I don't, my rule changes are wiped out after every update.So, I make any changes at all to pfBlocker, I'm re-writing my blocking rules totally from scratch.
It's the only way I can have Widget+CustomizedRules+CustomRulesOrder.It's doubled my time to restore settings after each pfBlocker config change.
Selecting a single country becomes a 20+min process, per machine.I'm to weary to come up with any helpful suggestions/workarounds right now.
I'll revisit the thread when my brain is working again.Thanks.
edit: I had another look at the Backup feature and discovered the option for FirewallRules.
I've make my copy and will try to restore from it after my next pfBlocker change. -
Linuxtracker,
After a update, as well as I know, you need just to enable pfBlocker to get all your settings working again.
Maybe I misundertood you but I did not coded an automatic action switch from deny to alias only.
The steps I do for rule reordering are:
Apply pfBlocker conf with action I want on rules.
Change alias description on created firewall rules and then customize it's order.
Back on pfBlocker and change action to alias only. -
- Apply pfBlocker conf with action I want on rules.
- Change alias description on created firewall rules and then customize it's order.
- Back on pfBlocker and change action to alias only.
I did #1 and #2 and had just started on #3.
The moment I set the first country-group to alias (S.America) it tosses that country group off the list.
The remaining rules - order and customizations - were all reset.As near as I can tell, any change at all in pfBlocker now mandates that I rewrite my rules from scratch.
It may be that every list update does the same.
I offer that because the rules table completely reset about 11:30pm today - I have to rewrite them again. -
Linuxtracker,
How are you renaming rule description before changing action to alias only?
I did a clean install and then:-
Installed pfblocker
-
denied inbound access to argentina and some countries on Oceania
-
Renamed the rule description from South America to block Argentina
-
saved firewall rules and applied changes
-
back to pfblocker, set action to alias only on South America tab
-
saved config
After this, both rules(South america and Oceania) are still there.
I'll do some tests with lists applied too.
-
-
Linuxtracker,
How are you renaming rule description before changing action to alias only?
I don't change the rule descriptions that are generated by pfBlocker.
I figured they were necessary for the widget to work.When I write the rules from scratch, the descriptions are identical to the pfBlocker generated ones.
ie:```
pfBlockerSouthAmerica auto ruleThanks -
I did a clean install and then:
-
Installed pfblocker
-
denied inbound access to Argentina and some countries on Oceania
-
Renamed the rule description from South America to block Argentina
-
saved firewall rules and applied changes
-
back to pfblocker, set action to alias only on South America tab
-
saved config
After this, both rules(South America and Oceania) are still there.
I'll do some tests with lists applied too.
I need to clarify something.
- Renamed the rule description from South America to block Argentina
You mean you changed the rule description from "South America", so that it read "block Argentina" - correct?
The last time I changed my rule descriptions, my pfBlocker widget quit working.
So, I've kept my rules descriptions identical to whatever pfBlocker created.But:
It seems we can rename the pfBlocker-generated alias name
as long as the new alias name is at the beginning of the rules description.That won't break the widget. Do I understand correctly?
-
-
Linuxtracker,
I've changed rule description to "pfBlockerSouthAmerica deny inbound" to do not break widget and also included a list with every hour update and rules are still there.
-
Linuxtracker,
I've changed rule description to "pfBlockerSouthAmerica deny inbound" to do not break widget and also included a list with every hour update and rules are still there.
OK Thanks for your time on this.
I'll uninstall the package tonight and see what a fresh start yields.
Question: How do I force a manual list update?
-
Question: How do I force a manual list update?
As I forgot to include this option, you can change update frequency to every hour and then run
/usr/local/bin/php -q /usr/local/www/pfblocker.php cron
on console.
-
Linuxtracker,
I've changed rule description to "pfBlockerSouthAmerica deny inbound" to do not break widget and also included a list with every hour update and rules are still there.
My custom lists weren't pulling new updates. I don't think the countries were updating either.
So I uninstalled the package and deleted the pgblocker*.xml files in /usr/local/pkg.
After reinstalling pfBlocker, both lists and countries updated correctly.After that, I followed your guide as before.
Once my rules were setup, I went back into pfBlocker and changed Oceana from Deny All to Alias
and all my rule changes and ordering were thrown out.That made me sad.
Update:
So with a heavy heart I set out to rewrite my rules from scratch.
I set the rest of the pfBlocker options to Alias and applied the settings.I next went to rules - and discovered that my rule settings and ordering - were restored back to where I wanted them.
I am no longer sad. Now I am confused.
-
My custom lists weren't pulling new updates. I don't think the countries were updating either.
Did you tried to run it on console the way I described to you?
I don't think the countries were updating either.
Country lists are updated on pfblocker releases, not via cron job.
So I uninstalled the package and deleted the pgblocker*.xml files in /usr/local/pkg.
After reinstalling pfBlocker, both lists and countries updated correctly.After that, I followed your guide as before.
Once my rules were setup, I went back into pfBlocker and changed Oceana from Deny All to Alias
and all my rule changes and ordering were thrown out.That made me sad.
Update:
So with a heavy heart I set out to rewrite my rules from scratch.
I set the rest of the pfBlocker options to Alias and applied the settings.I next went to rules - and discovered that my rule settings and ordering - were restored back to where I wanted them.
I am no longer sad. Now I am confused.
I'll keep trying to simulate this issue.
All tests I did, preserving the aliasname on firewall rule description were fine. -
Did you tried to run it on console the way I described to you?
Yes. That did work and helped me to find a misspelled list name.
It'll also come in handy in the future.
Country lists are updated on pfblocker releases, not via cron job.
After I uninstalled the package -> deleted the pfblocker*.xml files -> reinstalled it - the country lists updated normally.
I think the package handler was wonky and didn't update pfblocker properly the last time.
I'll keep trying to simulate this issue.
All tests I did, preserving the aliasname on firewall rule description were fine.That my rules would suddenly show correctly - after they were reset - seems really strange.
I have other pfSense boxes out there. I'll update one or two of them and see if any issues pop up.
I certainly appreciate your efforts.
For now I'll keep looking into things on my end. -
My exp for using pfBlocker ,
if I set max table size = 100000 ,
there still have error logged ,php: : New alert found: There were error(s) loading the rules: /tmp/rules.debug:23: cannot define table pfBlockerNorthAmerica: Cannot allocate memory pfctl: Syntax error in config file: pf rules not loaded The line in question reads [23]: table <pfblockernorthamerica>persist file "/var/db/aliastables/pfBlockerNorthAmerica.txt"
if I set max table size = 1000000
problem solve , just like to share .</pfblockernorthamerica>
