Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    PfBlocker

    Scheduled Pinned Locked Moved pfSense Packages
    896 Posts 143 Posters 1.3m Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • V
      vlurk
      last edited by

      It looks like I am having the "cannot allocate memory" issue as well, running a 2.0.1 (i386) release, nanobsd 2G and pfBlocker 1.0.1. I increased the max table size, read page 21 as well but it just wouldn't cut it with my small platform.

      Since I REALLY wanted to get the Bluetack's level1 blocklist, I needed to try something else and eventually found another solution…

      What I did is write a simple perl script with that will read the list, and output a new CIDR list splitted into many files, actually 100,000 entries per file.

      I then created a small cron job to download the list and execute the script on a linux server running Apache.  Finally, I configured three different blocklists (aliases) under pfblocker (not just a big one with three URLs...).  And it works!  Maybe having such a mechanism (splitting big files) built into pfblocker could be useful for some.

      In case someone is interested by the perl script, it looks like this:

      #!/usr/bin/perl
      use Net::CIDR::Lite;
      
      my $filenum = "0" x 4; # if you ++ a string, it keeps the padding
      
      sub new_file {
              $filenum ++;
              my $name = "splat_$filenum.lis";
              open OUT, ">$name" or die "canne open $name cap'n:$!\n";
              warn "writing to:$name\n";
      }
      
      my $cidr = Net::CIDR::Lite->new;
      open (MYFILE, $ARGV[0]);
      while (<myfile>) {
        chomp;
        $_ =~ /[^:]+:(.*)/;
        my $range = $1; #extracted IP Range, verify it is IPv4
        if ( $range =~ m/\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}\-\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}/i ) {
              $cidr->add_range($range);
        }
      }
      close (MYFILE);
      my $index = 0;
      my @cidr_list = $cidr->list;
      foreach my $block ( @cidr_list ) {
              if ( $index % 100000 == 0 ) {
                      new_file;
              }
              print OUT $block,"\n";
              $index++;
      }
      close (OUT);</myfile> 
      

      It receives the unzipped list in input, and will output the files in the CWD. Simple as that.

      Have a nice day.

      1 Reply Last reply Reply Quote 0
      • marcellocM
        marcelloc
        last edited by

        Good contributon. Thank you. :)

        Treinamentos de Elite: http://sys-squad.com

        Help a community developer! ;D

        1 Reply Last reply Reply Quote 0
        • marcellocM
          marcelloc
          last edited by

          Just updated pfBlocker to 1.0.2 with:

          • Fix on array check error at line 368 when there is no alias defined on pfSense

          • reduce duplicate cases on automatic rules when using multiple interfaces as inbound and/or outbound

          • Increase php memory limit to 250Mb when x64 pfSense is detected(DO AT YOUR OWN RISK PATCH applied to code ;))

          • Updated country ip lists

          Treinamentos de Elite: http://sys-squad.com

          Help a community developer! ;D

          1 Reply Last reply Reply Quote 0
          • T
            tommyboy180
            last edited by

            Awesome!

            -Tom Schaefer
            SuperMicro 1U 2X Intel pro/1000 Dual Core Intel 2.2 Ghz - 2 Gig RAM

            Please support pfBlocker | File Browser | Strikeback

            1 Reply Last reply Reply Quote 0
            • T
              taryezveb
              last edited by

              @tommyboy180:

              Awesome!

              +1, Thanks

              1 Reply Last reply Reply Quote 0
              • V
                vlurk
                last edited by

                Thanks for the update: I am gonna give it a shot like right after I post this.

                I noticed that my script would fail when some blocklist would include multiple colons on one line. Here is my updated script, which now accept a number of lines as the second argument.

                #!/usr/bin/perl
                use Net::CIDR::Lite;
                
                my $filenum = "0" x 4; # if you ++ a string, it keeps the padding
                my $limit = 100000; # default max number of lines
                
                sub new_file {
                	$filenum ++;
                	my $name = "splat_$filenum.lis";
                	open OUT, ">$name" or die "canne open $name cap'n:$!\n";
                        warn "writing to:$name\n";
                }
                
                my $cidr = Net::CIDR::Lite->new;
                open (MYFILE, $ARGV[0]);
                if ( defined($ARGV[1])) {
                	if ( $ARGV[1] =~  m/^\d{2,6}$/ ) {
                		$limit = int($ARGV[1]);
                	}
                }
                
                while (<myfile>) {
                  chomp;
                  my @line = split(/:+/);
                  my $range = $line[-1]; #get IP Range, verify it is IPv4
                  if ( $range =~ m/^\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}\-\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3}$/ ) {
                	$cidr->add_range($range);
                  }
                }
                close (MYFILE);
                my $index = 0;
                my @cidr_list = $cidr->list;
                foreach my $block ( @cidr_list ) {
                	if ( $index % $limit == 0 ) {
                		close (OUT);
                		new_file;
                	}
                	print OUT $block,"\n";
                	$index++;
                }
                close (OUT);</myfile> 
                

                Since my platform is not x64, and only have 256MB of RAM, I am not sure the new patch will fix the memory allocation issue for me… I am running with 60% memory used on average. Right now I am using a 60,000 lines as my maximum. 100,000 would seem to fail on some occasions.

                1 Reply Last reply Reply Quote 0
                • marcellocM
                  marcelloc
                  last edited by

                  Vlurk,

                  The memory patch is only for am64.
                  As we try to avoid file hacks, I've applied a value that is defined on config.inc but not reflected on gui.
                  As you have few memory available, the best option is the way you are doing or of course a hardware upgrade.

                  Thank's for the script update.

                  Treinamentos de Elite: http://sys-squad.com

                  Help a community developer! ;D

                  1 Reply Last reply Reply Quote 0
                  • L
                    LinuxTracker
                    last edited by

                    I know Rules ReOrdering after a pfBlocker change has been covered in this thread.
                    I'd like to bring it back up because it's making me crazy.

                    Here's my situation.
                    I use the pfBlocker widget.  I also have my rules customized and ordered a certain way.

                    In the last pfBlocker ver., I'd set every Action to Deny Inbound.
                    Next I'd customize and reorder the auto-created rules. I'd be finished in 10 min or so.
                    I'm pretty sure pfBlocker automatically changed Action to Alias when I had adjusted the rules.
                    The end result was the rules wouldn't change after an update.

                    In this latest pfBlocker ver., my last method doesn't work.  I have to set action to Alias myself.
                    If I don't, my rule changes are wiped out after every update.

                    So, I make any changes at all to pfBlocker, I'm re-writing my blocking rules totally from scratch.
                    It's the only way I can have Widget+CustomizedRules+CustomRulesOrder.

                    It's doubled my time to restore settings after each pfBlocker config change.
                    Selecting a single country becomes a 20+min process, per machine.

                    I'm to weary to come up with any helpful suggestions/workarounds right now.
                    I'll revisit the thread when my brain is working again.

                    Thanks.

                    edit: I had another look at the Backup feature and discovered the option for FirewallRules.
                    I've make my copy and will try to restore from it after my next pfBlocker change.

                    1 Reply Last reply Reply Quote 0
                    • marcellocM
                      marcelloc
                      last edited by

                      Linuxtracker,

                      After a update, as well as I know, you need just to enable pfBlocker to get all your settings working again.

                      Maybe I misundertood you but I did not coded an automatic action switch from deny to alias only.

                      The steps I do for rule reordering are:

                      Apply pfBlocker conf with action I want on rules.
                      Change alias description on created firewall rules and then customize it's order.
                      Back on pfBlocker and change action to alias only.

                      Treinamentos de Elite: http://sys-squad.com

                      Help a community developer! ;D

                      1 Reply Last reply Reply Quote 0
                      • L
                        LinuxTracker
                        last edited by

                        @marcelloc:

                        1. Apply pfBlocker conf with action I want on rules.
                        2. Change alias description on created firewall rules and then customize it's order.
                        3. Back on pfBlocker and change action to alias only.

                        I did #1 and #2 and had just started on #3.
                        The moment I set the first country-group to alias (S.America) it tosses that country group off the list.
                        The remaining rules - order and customizations - were all reset.

                        As near as I can tell, any change at all in pfBlocker now mandates that I rewrite my rules from scratch.

                        It may be that every list update does the same.  
                        I offer that because the rules table completely reset about 11:30pm today - I have to rewrite them again.

                        1 Reply Last reply Reply Quote 0
                        • marcellocM
                          marcelloc
                          last edited by

                          Linuxtracker,

                          How are you renaming rule description before changing action to alias only?
                          I did a clean install and then:

                          • Installed pfblocker

                          • denied inbound access to argentina and some countries on Oceania

                          • Renamed the rule description from South America to block Argentina

                          • saved firewall rules and applied changes

                          • back to pfblocker, set action to alias only on South America tab

                          • saved config

                          After this, both rules(South america and Oceania) are still there.

                          I'll do some tests with lists applied too.

                          Treinamentos de Elite: http://sys-squad.com

                          Help a community developer! ;D

                          1 Reply Last reply Reply Quote 0
                          • L
                            LinuxTracker
                            last edited by

                            @marcelloc:

                            Linuxtracker,

                            How are you renaming rule description before changing action to alias only?

                            I don't change the rule descriptions that are generated by pfBlocker.
                            I figured they were necessary for the widget to work.

                            When I write the rules from scratch, the descriptions are identical to the pfBlocker generated ones.
                            ie:```
                            pfBlockerSouthAmerica auto rule

                            
                            Thanks
                            1 Reply Last reply Reply Quote 0
                            • L
                              LinuxTracker
                              last edited by

                              @marcelloc:

                              I did a clean install and then:

                              • Installed pfblocker

                              • denied inbound access to Argentina and some countries on Oceania

                              • Renamed the rule description from South America to block Argentina

                              • saved firewall rules and applied changes

                              • back to pfblocker, set action to alias only on South America tab

                              • saved config

                              After this, both rules(South America and Oceania) are still there.

                              I'll do some tests with lists applied too.

                              I need to clarify something.

                              • Renamed the rule description from South America to block Argentina

                              You mean you changed the rule description from "South America", so that it read "block Argentina" - correct?

                              The last time I changed my rule descriptions, my pfBlocker widget quit working.
                              So, I've kept my rules descriptions identical to whatever pfBlocker created.

                              But:
                              It seems we can rename the pfBlocker-generated alias name
                              as long as the new alias name is at the beginning of the rules description.

                              That won't break the widget.  Do I understand correctly?

                              1 Reply Last reply Reply Quote 0
                              • marcellocM
                                marcelloc
                                last edited by

                                Linuxtracker,

                                I've changed rule description to "pfBlockerSouthAmerica deny inbound" to do not break widget and also included a list with every hour update and rules are still there.

                                Treinamentos de Elite: http://sys-squad.com

                                Help a community developer! ;D

                                1 Reply Last reply Reply Quote 0
                                • L
                                  LinuxTracker
                                  last edited by

                                  @marcelloc:

                                  Linuxtracker,

                                  I've changed rule description to "pfBlockerSouthAmerica deny inbound" to do not break widget and also included a list with every hour update and rules are still there.

                                  OK Thanks for your time on this.

                                  I'll uninstall the package tonight and see what a fresh start yields.

                                  Question: How do I force a manual list update?

                                  1 Reply Last reply Reply Quote 0
                                  • marcellocM
                                    marcelloc
                                    last edited by

                                    @LinuxTracker:

                                    Question: How do I force a manual list update?

                                    As I forgot to include this option, you can change update frequency to every hour and then run

                                    /usr/local/bin/php -q /usr/local/www/pfblocker.php cron

                                    on console.

                                    Treinamentos de Elite: http://sys-squad.com

                                    Help a community developer! ;D

                                    1 Reply Last reply Reply Quote 0
                                    • L
                                      LinuxTracker
                                      last edited by

                                      @marcelloc:

                                      Linuxtracker,

                                      I've changed rule description to "pfBlockerSouthAmerica deny inbound" to do not break widget and also included a list with every hour update and rules are still there.

                                      My custom lists weren't pulling new updates.  I don't think the countries were updating either.

                                      So I uninstalled the package and deleted the pgblocker*.xml files in /usr/local/pkg.
                                      After reinstalling pfBlocker, both lists and countries updated correctly.

                                      After that, I followed your guide as before.
                                      Once my rules were setup, I went back into pfBlocker and changed Oceana from Deny All to Alias
                                      and all my rule changes and ordering were thrown out.

                                      That made me sad.

                                      Update:
                                      So with a heavy heart I set out to rewrite my rules from scratch.
                                      I set the rest of the pfBlocker options to Alias and applied the settings.

                                      I next went to rules - and discovered that my rule settings and ordering - were restored back to where I wanted them.

                                      I am no longer sad.  Now I am confused.

                                      1 Reply Last reply Reply Quote 0
                                      • marcellocM
                                        marcelloc
                                        last edited by

                                        @LinuxTracker:

                                        My custom lists weren't pulling new updates.  I don't think the countries were updating either.

                                        Did you tried to run it on console the way I described to you?

                                        @LinuxTracker:

                                        I don't think the countries were updating either.

                                        Country lists are updated on pfblocker releases, not via cron job.

                                        @LinuxTracker:

                                        So I uninstalled the package and deleted the pgblocker*.xml files in /usr/local/pkg.
                                        After reinstalling pfBlocker, both lists and countries updated correctly.

                                        After that, I followed your guide as before.
                                        Once my rules were setup, I went back into pfBlocker and changed Oceana from Deny All to Alias
                                        and all my rule changes and ordering were thrown out.

                                        That made me sad.

                                        Update:
                                        So with a heavy heart I set out to rewrite my rules from scratch.
                                        I set the rest of the pfBlocker options to Alias and applied the settings.

                                        I next went to rules - and discovered that my rule settings and ordering - were restored back to where I wanted them.

                                        I am no longer sad.  Now I am confused.

                                        I'll keep trying to simulate this issue.
                                        All tests I did, preserving the aliasname on firewall rule description were fine.

                                        Treinamentos de Elite: http://sys-squad.com

                                        Help a community developer! ;D

                                        1 Reply Last reply Reply Quote 0
                                        • L
                                          LinuxTracker
                                          last edited by

                                          @marcelloc:

                                          Did you tried to run it on console the way I described to you?

                                          Yes.  That did work and helped me to find a misspelled list name.

                                          It'll also come in handy in the future.

                                          @LinuxTracker:

                                          Country lists are updated on pfblocker releases, not via cron job.

                                          After I uninstalled the package -> deleted the pfblocker*.xml files -> reinstalled it - the country lists updated normally.

                                          I think the package handler was wonky and didn't update pfblocker properly the last time.

                                          @LinuxTracker:

                                          I'll keep trying to simulate this issue.
                                          All tests I did, preserving the aliasname on firewall rule description were fine.

                                          That my rules would suddenly show correctly - after they were reset - seems really strange.

                                          I have other pfSense boxes out there.  I'll update one or two of them and see if any issues pop up.

                                          I certainly appreciate your efforts. 
                                          For now I'll keep looking into things on my end.

                                          1 Reply Last reply Reply Quote 0
                                          • A
                                            archy
                                            last edited by

                                            My exp for using pfBlocker ,
                                            if I set max table size = 100000 ,
                                            there still have error logged ,

                                            php: : New alert found: There were error(s) loading the rules: /tmp/rules.debug:23: cannot define table pfBlockerNorthAmerica: Cannot allocate memory pfctl: Syntax error in config file: pf rules not loaded The line in question reads [23]: table <pfblockernorthamerica>persist file "/var/db/aliastables/pfBlockerNorthAmerica.txt"

                                            if I set max table size = 1000000

                                            problem solve , just like to share .</pfblockernorthamerica>

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.