PfBlocker

vlurk · Feb 13, 2012, 5:51 AM

It looks like I am having the "cannot allocate memory" issue as well, running a 2.0.1 (i386) release, nanobsd 2G and pfBlocker 1.0.1. I increased the max table size, read page 21 as well but it just wouldn't cut it with my small platform.

Since I REALLY wanted to get the Bluetack's level1 blocklist, I needed to try something else and eventually found another solution…

What I did is write a simple perl script with that will read the list, and output a new CIDR list splitted into many files, actually 100,000 entries per file.

I then created a small cron job to download the list and execute the script on a linux server running Apache. Finally, I configured three different blocklists (aliases) under pfblocker (not just a big one with three URLs...). And it works! Maybe having such a mechanism (splitting big files) built into pfblocker could be useful for some.

In case someone is interested by the perl script, it looks like this:

#!/usr/bin/perl
use Net::CIDR::Lite;

my $filenum = "0" x 4; # if you ++ a string, it keeps the padding

sub new_file {
        $filenum ++;
        my $name = "splat_$filenum.lis";
        open OUT, ">$name" or die "canne open $name cap'n:$!\n";
        warn "writing to:$name\n";
}

my $cidr = Net::CIDR::Lite->new;
open (MYFILE, $ARGV[0]);
while (<myfile>) {
  chomp;
  $_ =~ /[^:]+:(.*)/;
  my $range = $1; #extracted IP Range, verify it is IPv4
  if ( $range =~ m/\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}\-\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}/i ) {
        $cidr->add_range($range);
  }
}
close (MYFILE);
my $index = 0;
my @cidr_list = $cidr->list;
foreach my $block ( @cidr_list ) {
        if ( $index % 100000 == 0 ) {
                new_file;
        }
        print OUT $block,"\n";
        $index++;
}
close (OUT);</myfile>

It receives the unzipped list in input, and will output the files in the CWD. Simple as that.

Have a nice day.

marcelloc · Feb 13, 2012, 10:30 AM

Good contributon. Thank you. :)

marcelloc · Feb 21, 2012, 7:41 PM

Just updated pfBlocker to 1.0.2 with:

Fix on array check error at line 368 when there is no alias defined on pfSense
reduce duplicate cases on automatic rules when using multiple interfaces as inbound and/or outbound
Increase php memory limit to 250Mb when x64 pfSense is detected(DO AT YOUR OWN RISK PATCH applied to code ;))
Updated country ip lists

tommyboy180 · Feb 21, 2012, 12:46 PM

Awesome!

taryezveb · Feb 21, 2012, 6:24 PM

@tommyboy180:

Awesome!

+1, Thanks

vlurk · Feb 22, 2012, 1:46 AM

Thanks for the update: I am gonna give it a shot like right after I post this.

I noticed that my script would fail when some blocklist would include multiple colons on one line. Here is my updated script, which now accept a number of lines as the second argument.

#!/usr/bin/perl
use Net::CIDR::Lite;

my $filenum = "0" x 4; # if you ++ a string, it keeps the padding
my $limit = 100000; # default max number of lines

sub new_file {
	$filenum ++;
	my $name = "splat_$filenum.lis";
	open OUT, ">$name" or die "canne open $name cap'n:$!\n";
        warn "writing to:$name\n";
}

my $cidr = Net::CIDR::Lite->new;
open (MYFILE, $ARGV[0]);
if ( defined($ARGV[1])) {
	if ( $ARGV[1] =~  m/^\d{2,6}$/ ) {
		$limit = int($ARGV[1]);
	}
}

while (<myfile>) {
  chomp;
  my @line = split(/:+/);
  my $range = $line[-1]; #get IP Range, verify it is IPv4
  if ( $range =~ m/^\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}\-\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}$/ ) {
	$cidr->add_range($range);
  }
}
close (MYFILE);
my $index = 0;
my @cidr_list = $cidr->list;
foreach my $block ( @cidr_list ) {
	if ( $index % $limit == 0 ) {
		close (OUT);
		new_file;
	}
	print OUT $block,"\n";
	$index++;
}
close (OUT);</myfile>

Since my platform is not x64, and only have 256MB of RAM, I am not sure the new patch will fix the memory allocation issue for me… I am running with 60% memory used on average. Right now I am using a 60,000 lines as my maximum. 100,000 would seem to fail on some occasions.

marcelloc · Feb 22, 2012, 1:58 AM

Vlurk,

The memory patch is only for am64.
As we try to avoid file hacks, I've applied a value that is defined on config.inc but not reflected on gui.
As you have few memory available, the best option is the way you are doing or of course a hardware upgrade.

Thank's for the script update.

LinuxTracker · Feb 22, 2012, 5:48 AM

I know Rules ReOrdering after a pfBlocker change has been covered in this thread.
I'd like to bring it back up because it's making me crazy.

Here's my situation.
I use the pfBlocker widget. I also have my rules customized and ordered a certain way.

In the last pfBlocker ver., I'd set every Action to Deny Inbound.
Next I'd customize and reorder the auto-created rules. I'd be finished in 10 min or so.
I'm pretty sure pfBlocker automatically changed Action to Alias when I had adjusted the rules.
The end result was the rules wouldn't change after an update.

In this latest pfBlocker ver., my last method doesn't work. I have to set action to Alias myself.
If I don't, my rule changes are wiped out after every update.

So, I make any changes at all to pfBlocker, I'm re-writing my blocking rules totally from scratch.
It's the only way I can have Widget+CustomizedRules+CustomRulesOrder.

It's doubled my time to restore settings after each pfBlocker config change.
Selecting a single country becomes a 20+min process, per machine.

I'm to weary to come up with any helpful suggestions/workarounds right now.
I'll revisit the thread when my brain is working again.

Thanks.

edit: I had another look at the Backup feature and discovered the option for FirewallRules.
I've make my copy and will try to restore from it after my next pfBlocker change.

marcelloc · Feb 22, 2012, 10:22 AM

Linuxtracker,

After a update, as well as I know, you need just to enable pfBlocker to get all your settings working again.

Maybe I misundertood you but I did not coded an automatic action switch from deny to alias only.

The steps I do for rule reordering are:

Apply pfBlocker conf with action I want on rules.
Change alias description on created firewall rules and then customize it's order.
Back on pfBlocker and change action to alias only.

LinuxTracker · Feb 23, 2012, 6:38 AM

@marcelloc:

Apply pfBlocker conf with action I want on rules.

Change alias description on created firewall rules and then customize it's order.

Back on pfBlocker and change action to alias only.

I did #1 and #2 and had just started on #3.
The moment I set the first country-group to alias (S.America) it tosses that country group off the list.
The remaining rules - order and customizations - were all reset.

As near as I can tell, any change at all in pfBlocker now mandates that I rewrite my rules from scratch.

It may be that every list update does the same.
I offer that because the rules table completely reset about 11:30pm today - I have to rewrite them again.

marcelloc · Feb 23, 2012, 2:01 PM

Linuxtracker,

How are you renaming rule description before changing action to alias only?
I did a clean install and then:

Installed pfblocker
denied inbound access to argentina and some countries on Oceania
Renamed the rule description from South America to block Argentina
saved firewall rules and applied changes
back to pfblocker, set action to alias only on South America tab
saved config

After this, both rules(South america and Oceania) are still there.

I'll do some tests with lists applied too.

LinuxTracker · Feb 23, 2012, 5:41 PM

@marcelloc:

Linuxtracker,

How are you renaming rule description before changing action to alias only?

I don't change the rule descriptions that are generated by pfBlocker.
I figured they were necessary for the widget to work.

When I write the rules from scratch, the descriptions are identical to the pfBlocker generated ones.
ie:```
pfBlockerSouthAmerica auto rule


Thanks

LinuxTracker · Feb 23, 2012, 5:46 PM

@marcelloc:

I did a clean install and then:

Installed pfblocker

denied inbound access to Argentina and some countries on Oceania

Renamed the rule description from South America to block Argentina

saved firewall rules and applied changes

back to pfblocker, set action to alias only on South America tab

saved config

After this, both rules(South America and Oceania) are still there.

I'll do some tests with lists applied too.

I need to clarify something.

Renamed the rule description from South America to block Argentina

You mean you changed the rule description from "South America", so that it read "block Argentina" - correct?

The last time I changed my rule descriptions, my pfBlocker widget quit working.
So, I've kept my rules descriptions identical to whatever pfBlocker created.

But:
It seems we can rename the pfBlocker-generated alias name
as long as the new alias name is at the beginning of the rules description.

That won't break the widget. Do I understand correctly?

marcelloc · Feb 23, 2012, 5:55 PM

Linuxtracker,

I've changed rule description to "pfBlockerSouthAmerica deny inbound" to do not break widget and also included a list with every hour update and rules are still there.

LinuxTracker · Feb 23, 2012, 6:58 PM

@marcelloc:

Linuxtracker,

I've changed rule description to "pfBlockerSouthAmerica deny inbound" to do not break widget and also included a list with every hour update and rules are still there.

OK Thanks for your time on this.

I'll uninstall the package tonight and see what a fresh start yields.

Question: How do I force a manual list update?

marcelloc · Feb 23, 2012, 7:30 PM

@LinuxTracker:

Question: How do I force a manual list update?

As I forgot to include this option, you can change update frequency to every hour and then run

/usr/local/bin/php -q /usr/local/www/pfblocker.php cron

on console.

LinuxTracker · Feb 24, 2012, 9:48 AM

@marcelloc:

Linuxtracker,

I've changed rule description to "pfBlockerSouthAmerica deny inbound" to do not break widget and also included a list with every hour update and rules are still there.

My custom lists weren't pulling new updates. I don't think the countries were updating either.

So I uninstalled the package and deleted the pgblocker*.xml files in /usr/local/pkg.
After reinstalling pfBlocker, both lists and countries updated correctly.

After that, I followed your guide as before.
Once my rules were setup, I went back into pfBlocker and changed Oceana from Deny All to Alias
and all my rule changes and ordering were thrown out.

That made me sad.

Update:
So with a heavy heart I set out to rewrite my rules from scratch.
I set the rest of the pfBlocker options to Alias and applied the settings.

I next went to rules - and discovered that my rule settings and ordering - were restored back to where I wanted them.

I am no longer sad. Now I am confused.

marcelloc · Feb 24, 2012, 1:57 PM

@LinuxTracker:

My custom lists weren't pulling new updates. I don't think the countries were updating either.

Did you tried to run it on console the way I described to you?

@LinuxTracker:

I don't think the countries were updating either.

Country lists are updated on pfblocker releases, not via cron job.

@LinuxTracker:

So I uninstalled the package and deleted the pgblocker*.xml files in /usr/local/pkg.
After reinstalling pfBlocker, both lists and countries updated correctly.

After that, I followed your guide as before.
Once my rules were setup, I went back into pfBlocker and changed Oceana from Deny All to Alias
and all my rule changes and ordering were thrown out.

That made me sad.

Update:
So with a heavy heart I set out to rewrite my rules from scratch.
I set the rest of the pfBlocker options to Alias and applied the settings.

I next went to rules - and discovered that my rule settings and ordering - were restored back to where I wanted them.

I am no longer sad. Now I am confused.

I'll keep trying to simulate this issue.
All tests I did, preserving the aliasname on firewall rule description were fine.

LinuxTracker · Feb 24, 2012, 8:18 PM

@marcelloc:

Did you tried to run it on console the way I described to you?

Yes. That did work and helped me to find a misspelled list name.

It'll also come in handy in the future.

@LinuxTracker:

Country lists are updated on pfblocker releases, not via cron job.

After I uninstalled the package -> deleted the pfblocker*.xml files -> reinstalled it - the country lists updated normally.

I think the package handler was wonky and didn't update pfblocker properly the last time.

@LinuxTracker:

I'll keep trying to simulate this issue.
All tests I did, preserving the aliasname on firewall rule description were fine.

That my rules would suddenly show correctly - after they were reset - seems really strange.

I have other pfSense boxes out there. I'll update one or two of them and see if any issues pop up.

I certainly appreciate your efforts.
For now I'll keep looking into things on my end.

archy · Feb 29, 2012, 6:06 PM

My exp for using pfBlocker ,
if I set max table size = 100000 ,
there still have error logged ,

php: : New alert found: There were error(s) loading the rules: /tmp/rules.debug:23: cannot define table pfBlockerNorthAmerica: Cannot allocate memory pfctl: Syntax error in config file: pf rules not loaded The line in question reads [23]: table <pfblockernorthamerica>persist file "/var/db/aliastables/pfBlockerNorthAmerica.txt"

if I set max table size = 1000000

problem solve , just like to share .</pfblockernorthamerica>