Navigation

    Netgate Discussion Forum
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search

    PFBlockerNG 2.1.1_2 Memory Errors

    pfBlockerNG
    17
    61
    20222
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • R
      rajl last edited by

      I'm having constant "out of memory" errors related to the latest release of PFBlockerNG.  An example error message reads:

      
      /usr/local/www/pfblockerng/pfblockerng.php: PHP ERROR: Type: 1, File: /usr/local/pkg/pfblockerng/pfblockerng.inc, Line: 3874, Message: Allowed memory size of 536870912 bytes exhausted (tried to allocate 72 bytes)
      
      

      I have a really simple PFBlockerNG ruleset – block all incoming connections except for IPv4 and IPv6 connections from the United States + the ad-blocker built into PFBlocker.

      I've seen other people on this forum report similar problems, but their solutions don't seem to work for me.  For example, I have already increased the setting for Firewall Maximum Table Entries from 2000000 to 4000000 (see e.g. https://forum.pfsense.org/index.php?topic=102470.msg643960#msg643960) but continue to get memory errors.

      Any other ideas?

      1 Reply Last reply Reply Quote 0
      • Qinn
        Qinn last edited by

        You are not the only one https://forum.pfsense.org/index.php?topic=102470.750 see Reply #763 and Reply #760….

        1 Reply Last reply Reply Quote 0
        • RonpfS
          RonpfS last edited by

          @rajl:

          I'm having constant "out of memory" errors related to the latest release of PFBlockerNG.  An example error message reads:

          
          /usr/local/www/pfblockerng/pfblockerng.php: PHP ERROR: Type: 1, File: /usr/local/pkg/pfblockerng/pfblockerng.inc, Line: 3874, Message: Allowed memory size of 536870912 bytes exhausted (tried to allocate 72 bytes)
          
          

          I have a really simple PFBlockerNG ruleset – block all incoming connections except for IPv4 and IPv6 connections from the United States + the ad-blocker built into PFBlocker.

          I've seen other people on this forum report similar problems, but their solutions don't seem to work for me.  For example, I have already increased the setting for Firewall Maximum Table Entries from 2000000 to 4000000 (see e.g. https://forum.pfsense.org/index.php?topic=102470.msg643960#msg643960) but continue to get memory errors.

          Any other ideas?

          This is also related to the MaxMind db issue. But fixing this in that part of the code isn't so simple as BBcan177 is away at the moment.

          Check the selection of countries in the GeoIP tabs, someone reported that the selection was inverted at some point.

          1 Reply Last reply Reply Quote 0
          • R
            rajl last edited by

            What should I be looking for?  I only have one country selected (United States) and I did invert the rule (i.e. block all inbound connections except from U.S.).  Are there any solutions other than to change the maximum number of table entries?  I have  8GB of RAM installed in my PFSense box, so the idea of running out of memory is a bit flumoxing as by all measures, I should have more than enough.

            Thanks

            1 Reply Last reply Reply Quote 0
            • RonpfS
              RonpfS last edited by

              That was just to make sure that nothing got crooked in the config. One users reported that the selection was inverted somehow on his setup.

              The only fix will come from BBcan177, but for now is doesn't have affordable Internet access somewhere in the middle of the sea. :(

              1 Reply Last reply Reply Quote 0
              • RonpfS
                RonpfS last edited by

                The problem seems to be related to i386 installation.
                The fix is to edit /usr/local/pkg/pfblockerng/pfblockerng.inc  (Diagnostics/Edit File)
                go to line 236

                change```
                pfb_global();

                // DNSBL Lighttpd HTTPS Daemon (Scans Lighttpd dnsbl_error.log for requested https domain names)
                if (isset($argv[1]) && $argv[1] == 'dnsbl') {

                to```
                pfb_global();
                ini_set('memory_limit', '300M');
                
                // DNSBL Lighttpd HTTPS Daemon (Scans Lighttpd dnsbl_error.log for requested https domain names)
                if (isset($argv[1]) && $argv[1] == 'dnsbl') {
                

                I tested with 128M, 200M and it fails
                256M succeeds, but BBcan177 recommends using 300M. You might need more depending on your system.

                So for i386, the default is 128M so very low, it should be 256M or more depending on you setup.
                For amd64 system, the default is 512M, so you might need to bump it to 640M or 768M.

                after the modifying the file, go to shell and run

                php /usr/local/www/pfblockerng/pfblockerng.php dc

                Once this succeed, you do not need to re-run the dc command. The GeoIP db will change next month.

                1 Reply Last reply Reply Quote 0
                • G
                  Gerard64 last edited by

                  Thanks for the tip and help!
                  I changed the file like you wrote and i run the command in a putty terminal on the router now it hangs like 5 minutes on:

                  [2.3.2-RELEASE][root@router.xxxxx.xxx]/root: php /usr/local/www/pfblockerng/pfblockerng.php dc
                  Country code update Start [ 08/03/16 22:51:59 ]
                  Converting MaxMind Country databases for pfBlockerNG.
                  Processing ISO IPv4 Continent/Country Data
                  Processing ISO IPv6 Continent/Country Data

                  The prompt is gone it seems its waiting for something.

                  1 Reply Last reply Reply Quote 0
                  • G
                    Gerard64 last edited by

                    After a long wait it outputs:

                    [2.3.2-RELEASE][root@router.xxxxx.xxx]/root: php /usr/local/www/pfblockerng/pfblockerng.php dc
                    Country code update Start [ 08/03/16 22:51:59 ]
                    Converting MaxMind Country databases for pfBlockerNG.
                    Processing ISO IPv4 Continent/Country Data
                    Processing ISO IPv6 Continent/Country Data
                    Creating pfBlockerNG Continent XML files
                    IPv4 Africa
                    IPv6 Africa
                    IPv4 Antarctica
                    IPv6 Antarctica
                    IPv4 Asia
                    IPv6 Asia
                    IPv4 Europe
                    IPv6 Europe

                    Fatal error: Allowed memory size of 314572800 bytes exhausted (tried to allocate 72 bytes) in /usr/local/www/pfblockerng/pfblockerng.php on line 728

                    Call Stack:
                        0.0075    422488  1. {main}() /usr/local/www/pfblockerng/pfblockerng.php:0
                      603.9766  12588288  2. pfblockerng_get_countries() /usr/local/www/pfblockerng/pfblockerng.php:105
                      612.1411  44556872  3. explode() /usr/local/www/pfblockerng/pfblockerng.php:728

                    PHP ERROR: Type: 1, File: /usr/local/www/pfblockerng/pfblockerng.php, Line: 728, Message: Allowed memory size of 314572800 bytes exhausted (tried to allocate 72 bytes)[2.3.2-RELEASE][root@router.xxxxxxx.xxx]/root:

                    1 Reply Last reply Reply Quote 0
                    • RonpfS
                      RonpfS last edited by

                      Yes it takes 5+minutes to complete

                      Try increasing to 400M, 500M up to 750M

                      1 Reply Last reply Reply Quote 0
                      • G
                        Gerard64 last edited by

                        Thanks  RonpfS ..!
                        After I set it to 500M it worked.

                        1 Reply Last reply Reply Quote 0
                        • Qinn
                          Qinn last edited by

                          @Gé:

                          Thanks  RonpfS ..!
                          After I set it to 500M it worked.

                          Thanks @RonpfS setting the memory_limit on line 236 to 300M fixed it, strange that @Gé needed 500M?

                          1 Reply Last reply Reply Quote 0
                          • Perforado
                            Perforado Rebel Alliance last edited by

                            For me i needed even 500M didn't work :/ 640M worked for me.

                            1 Reply Last reply Reply Quote 0
                            • RonpfS
                              RonpfS last edited by

                              @Perforado:

                              For me i needed even more as 500M didn't work :/ 768M worked for me.

                              Which version of pfsense?
                              x386 or amd64 ?
                              how much ram ?

                              1 Reply Last reply Reply Quote 0
                              • Perforado
                                Perforado Rebel Alliance last edited by

                                CPU: Intel(R) Atom(TM) CPU  C2558  @ 2.40GHz (2400.06-MHz K8-class CPU)
                                RAM: 8192MB

                                pfSense 2.3.3.a.20160803.1621
                                pfBlockerNG 2.1.1_2

                                1 Reply Last reply Reply Quote 0
                                • RonpfS
                                  RonpfS last edited by

                                  Well on amd64 the default is 512MB.

                                  /var in Ramdisk ? getting full ?

                                  1 Reply Last reply Reply Quote 0
                                  • RonpfS
                                    RonpfS last edited by

                                    @oswoldy:

                                    Ok, while the php /usr/local/www/pfblockerng/pfblockerng.php dc command works, the cron jobs are still crashing and giving errors, I am currently at 2GB limit and climbing.

                                    If you run the dc command, it changes the GeoIP database, you have to run a force reload after the dc complete.

                                    Also check diagnostic system activity to see if there isn't something unusual.
                                    Look at the System log for hints on the failure mode.
                                    An can you post the portion of the pfblockerng.log file where you have failure.

                                    1 Reply Last reply Reply Quote 0
                                    • RonpfS
                                      RonpfS last edited by

                                      @oswoldy:

                                      @RonpfS:

                                      @oswoldy:

                                      Ok, while the php /usr/local/www/pfblockerng/pfblockerng.php dc command works, the cron jobs are still crashing and giving errors, I am currently at 2GB limit and climbing.

                                      If you run the dc command, it changes the GeoIP database, you have to run a force reload after the dc complete.

                                      Also check diagnostic system activity to see if there isn't something unusual.
                                      Look at the System log for hints on the failure mode.
                                      An can you post the portion of the pfblockerng.log file where you have failure.

                                      Ok, I ran the dc command, followed by a force reload, no different. If I remove pfB_NAmerica_v4 then it works fine, crash report is:

                                      Crash report begins.  Anonymous machine information:

                                      amd64
                                      10.3-RELEASE-p5
                                      FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016    root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense

                                      Crash report details:

                                      PHP Errors:
                                      [04-Aug-2016 10:29:24 Europe/London] PHP Fatal error:  Allowed memory size of 524288000 bytes exhausted (tried to allocate 20 bytes) in /usr/local/pkg/pfblockerng/pfblockerng.inc on line 3875
                                      [04-Aug-2016 10:29:24 Europe/London] PHP Stack trace:
                                      [04-Aug-2016 10:29:24 Europe/London] PHP  1. {main}() /usr/local/www/pfblockerng/pfblockerng.php:0
                                      [04-Aug-2016 10:29:24 Europe/London] PHP  2. sync_package_pfblockerng() /usr/local/www/pfblockerng/pfblockerng.php:87
                                      [04-Aug-2016 10:29:24 Europe/London] PHP  3. file() /usr/local/pkg/pfblockerng/pfblockerng.inc:3875

                                      I have reduced the limit back to 500MB as increasing it wasnt making a difference, the dc command still completes but not a force reload or CRON.

                                      Once the dc command complete, the MaxMind database is created, so you do not need to re run it.

                                      What about the pfblockerng.log ? What are the symptoms ?
                                      Maybe raise it to 640M or 768M?

                                      1 Reply Last reply Reply Quote 0
                                      • W
                                        wiz561 last edited by

                                        I started a thread up over in the pfblocker posting and just letting everybody know that I'm also experiencing very similar memory issues.

                                        Thismorning when I logged in, I also had a pfsense crash report with the following:

                                        					Crash report begins.  Anonymous machine information:
                                        
                                        amd64
                                        10.3-RELEASE-p5
                                        FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016     root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense
                                        
                                        Crash report details:
                                        
                                        PHP Errors:
                                        [04-Aug-2016 00:18:40 America/Chicago] PHP Fatal error:  Allowed memory size of 402653184 bytes exhausted (tried to allocate 72 bytes) in /usr/local/pkg/pfblockerng/pfblockerng.inc on line 3868
                                        [04-Aug-2016 00:18:40 America/Chicago] PHP Stack trace:
                                        [04-Aug-2016 00:18:40 America/Chicago] PHP   1\. {main}() /etc/rc.start_packages:0
                                        [04-Aug-2016 00:18:40 America/Chicago] PHP   2\. sync_package() /etc/rc.start_packages:90
                                        [04-Aug-2016 00:18:40 America/Chicago] PHP   3\. eval() /etc/inc/pkg-utils.inc:631
                                        [04-Aug-2016 00:18:40 America/Chicago] PHP   4\. sync_package_pfblockerng() /etc/inc/pkg-utils.inc(631) : eval()'d code:3
                                        [04-Aug-2016 00:18:40 America/Chicago] PHP   5\. array_merge() /usr/local/pkg/pfblockerng/pfblockerng.inc:3868
                                        
                                        Filename: /var/crash/minfree
                                        2048
                                        
                                        1 Reply Last reply Reply Quote 0
                                        • L
                                          lucasrca last edited by

                                          How I solved my problem:

                                          My pfSense config:

                                          • Version: 2.3.2 (amd64), running on VMWare 6
                                            – Snort
                                            -- pfBlockerNG
                                            -- OpenVPN
                                            -- Open-vm-tools
                                            -- DHCP Relay
                                            -- Quagga OSPFd with another 2 pfSense.
                                          • ~3000 users simultaneously
                                          • 2 x 100 Mbit uplinks
                                          • 16 GB RAM
                                          • 80 GB SAS
                                          • CPU Type: Intel(R) Xeon(R) CPU E5-4620 v2 @ 2.60GHz
                                          • 16 CPUs: 8 package(s) x 2 core(s)
                                          • 8 Interfaces, including WAN
                                          • routing, filtering and relaying dhcp to 16 branches over MPLS, WiMax and fiber
                                          • Load balance and failover
                                          • QoS with Traffic shaper
                                          1. Updated Firewall Maximum Table Entries: 4000000 -> 8000000
                                            1.1) Reboot
                                          2. Edited /usr/local/pkg/pfblockerng/pfblockerng.inc and set memory limit to 500M
                                          3. Executed php /usr/local/www/pfblockerng/pfblockerng.php dc
                                          4. It's alive.

                                          Thanks to all involved.

                                          1 Reply Last reply Reply Quote 0
                                          • M
                                            Mithrondil last edited by

                                            POST-INSTALL script failed
                                            Message from GeoIP-1.6.9:
                                            GeoIP does not ship with the actual data files. You must download
                                            them yourself! Please run:

                                            /usr/local/bin/geoipupdate.sh

                                            1 Reply Last reply Reply Quote 0
                                            • RonpfS
                                              RonpfS last edited by

                                              @wiz561:

                                              I started a thread up over in the pfblocker posting and just letting everybody know that I'm also experiencing very similar memory issues.

                                              Thismorning when I logged in, I also had a pfsense crash report with the following:

                                              					Crash report begins.  Anonymous machine information:
                                              
                                              amd64
                                              10.3-RELEASE-p5
                                              FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016     root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense
                                              
                                              Crash report details:
                                              
                                              PHP Errors:
                                              [04-Aug-2016 00:18:40 America/Chicago] PHP Fatal error:  Allowed memory size of 402653184 bytes exhausted (tried to allocate 72 bytes) in /usr/local/pkg/pfblockerng/pfblockerng.inc on line 3868
                                              [04-Aug-2016 00:18:40 America/Chicago] PHP Stack trace:
                                              [04-Aug-2016 00:18:40 America/Chicago] PHP   1\. {main}() /etc/rc.start_packages:0
                                              [04-Aug-2016 00:18:40 America/Chicago] PHP   2\. sync_package() /etc/rc.start_packages:90
                                              [04-Aug-2016 00:18:40 America/Chicago] PHP   3\. eval() /etc/inc/pkg-utils.inc:631
                                              [04-Aug-2016 00:18:40 America/Chicago] PHP   4\. sync_package_pfblockerng() /etc/inc/pkg-utils.inc(631) : eval()'d code:3
                                              [04-Aug-2016 00:18:40 America/Chicago] PHP   5\. array_merge() /usr/local/pkg/pfblockerng/pfblockerng.inc:3868
                                              
                                              Filename: /var/crash/minfree
                                              2048
                                              

                                              The php memory should be 512M by default, so setting it in the inc file to 400M might not help.
                                              Did it fail without any fix to the inc file? Can you try setting the limit to 640M or 768M?
                                              What do you have for Firewall Maximum Table Entries?

                                              1 Reply Last reply Reply Quote 0
                                              • RonpfS
                                                RonpfS last edited by

                                                @Mithrondil:

                                                POST-INSTALL script failed
                                                Message from GeoIP-1.6.9:
                                                GeoIP does not ship with the actual data files. You must download
                                                them yourself! Please run:

                                                /usr/local/bin/geoipupdate.sh

                                                You will need to run  php /usr/local/www/pfblockerng/pfblockerng.php dc from the shell.
                                                Once it succeeds, you should be able to install, it may need a reboot.

                                                1 Reply Last reply Reply Quote 0
                                                • RonpfS
                                                  RonpfS last edited by

                                                  For those with failed installation. Verify that /var isn't full. The MaxMind database is huge so if you are using a RAM Disk, it might eat up memory that is needed for the pfblockerng.php.

                                                  If disk space is running low, BBcan177 suggest to delete the /var/db/pfblockerng/deny and /var/db/pfblockerng/original folders before installation to free some disk space. This means it will need to redownload all IP feeds after installation.

                                                  Check pfblockerng.log, the system log, Dashboard for crash report, Status Monitoring System Memory.

                                                  Post relevant debug info here.

                                                  1 Reply Last reply Reply Quote 0
                                                  • W
                                                    wiz561 last edited by

                                                    @RonpfS:

                                                    @wiz561:

                                                    I started a thread up over in the pfblocker posting and just letting everybody know that I'm also experiencing very similar memory issues.

                                                    Thismorning when I logged in, I also had a pfsense crash report with the following:

                                                    					Crash report begins.  Anonymous machine information:
                                                    
                                                    amd64
                                                    10.3-RELEASE-p5
                                                    FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016     root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense
                                                    
                                                    Crash report details:
                                                    
                                                    PHP Errors:
                                                    [04-Aug-2016 00:18:40 America/Chicago] PHP Fatal error:  Allowed memory size of 402653184 bytes exhausted (tried to allocate 72 bytes) in /usr/local/pkg/pfblockerng/pfblockerng.inc on line 3868
                                                    [04-Aug-2016 00:18:40 America/Chicago] PHP Stack trace:
                                                    [04-Aug-2016 00:18:40 America/Chicago] PHP   1\. {main}() /etc/rc.start_packages:0
                                                    [04-Aug-2016 00:18:40 America/Chicago] PHP   2\. sync_package() /etc/rc.start_packages:90
                                                    [04-Aug-2016 00:18:40 America/Chicago] PHP   3\. eval() /etc/inc/pkg-utils.inc:631
                                                    [04-Aug-2016 00:18:40 America/Chicago] PHP   4\. sync_package_pfblockerng() /etc/inc/pkg-utils.inc(631) : eval()'d code:3
                                                    [04-Aug-2016 00:18:40 America/Chicago] PHP   5\. array_merge() /usr/local/pkg/pfblockerng/pfblockerng.inc:3868
                                                    
                                                    Filename: /var/crash/minfree
                                                    2048
                                                    

                                                    The php memory should be 512M by default, so setting it in the inc file to 400M might not help.
                                                    Did it fail without any fix to the inc file? Can you try setting the limit to 640M or 768M?
                                                    What do you have for Firewall Maximum Table Entries?

                                                    I'm going to have to do a bit of work this weekend on this and see more.  I can try it and see what happens, but I'm getting concerned about reaching my 2gb memory limit.  I can add more memory, but that requires me to go to the store and I'm kinda lazy…..and cheap.  :)

                                                    Interesting that I too have a similar setup to the one user above.  I'm running it on vmware esxi with Snort (disabled) and OpenVPN.  Granted, I don't have nearly the hardware or setup, but it's interesting that we're both running vmware.

                                                    1 Reply Last reply Reply Quote 0
                                                    • W
                                                      wiz561 last edited by

                                                      OK…  Now maybe I'll try what others suggested...  :)

                                                      I wiped and reinstalled pfsense tonight and pfblockng is still coming back with that crash and memory errors.  I know others said to adjust the memory but I thought I would give this a try.  Unfortunately, it failed.

                                                      I also had issues with php-fpm having high utilization....so I'm hoping that the wipe/reinstall fixed the issue with that.  Time will tell.

                                                      1 Reply Last reply Reply Quote 0
                                                      • RonpfS
                                                        RonpfS last edited by

                                                        What about posting pfblockerNG, system log, crash report, screen shot of system activity, etc, so we can see what is happening on your setup?

                                                        The crash report you posted earlier tells me you have under 400MB defined.

                                                        PHP Fatal error:  Allowed memory size of 402653184 bytes exhausted

                                                        Did you raise the Firewall Maximum Table Entries ?

                                                        1 Reply Last reply Reply Quote 0
                                                        • Perforado
                                                          Perforado Rebel Alliance last edited by

                                                          Temporary Fix for

                                                          php /usr/local/www/pfblockerng/pfblockerng.php update

                                                          Failing with memory exhaustion:

                                                          edit /usr/local/pkg/pfblockerng/pfblockerng.inc as discussed above:
                                                          …
                                                          pfb_global();
                                                          ini_set('memory_limit', '640M');
                                                          ...

                                                          cp /etc/rc.php_ini_setup /etc/rc.php_ini_setup.BACKUP
                                                          cp /usr/local/etc/php.ini /usr/local/etc/php.ini.BACKUP
                                                          perl -pi -e 's/536870912/671088640/g' /etc/rc.php_ini_setup /usr/local/etc/php.ini

                                                          512 * 1024 * 1024 -> 536870912
                                                          640 * 1024 * 1024 -> 671088640 works for me. maybe your setup needs more :)

                                                          1 Reply Last reply Reply Quote 0
                                                          • RonpfS
                                                            RonpfS last edited by

                                                            @Perforado:

                                                            Temporary Fix for

                                                            php /usr/local/www/pfblockerng/pfblockerng.php update

                                                            Failing with memory exhaustion:

                                                            cp /etc/rc.php_ini_setup /etc/rc.php_ini_setup.BACKUP
                                                            cp /usr/local/etc/php.ini /usr/local/etc/php.ini.BACKUP
                                                            perl -pi -e 's/536870912/671088640/g' /etc/rc.php_ini_setup /usr/local/etc/php.ini

                                                            512 * 1024 * 1024 -> 536870912
                                                            640 * 1024 * 1024 -> 671088640 works for me. maybe your setup needs more :)

                                                            There is probably a setting (memory_limit?) we can configure in the System/Advanced/System Tunables that will do that for you. But it may require a reboot.

                                                            The fix for the inc file is specific to pfblocker and shouldn't be needed once BBcan177 change the code.

                                                            1 Reply Last reply Reply Quote 0
                                                            • Perforado
                                                              Perforado Rebel Alliance last edited by

                                                              /var/db/aliastables:
                                                              -rw-r–r--  1 root  wheel    351450 Aug  5 10:56 pfB_Top_v4.txt
                                                              -rw-r--r--  1 root  wheel  30690970 Aug  5 11:00 pfB_Top_v6.txt

                                                              cat pfB_Top_v6.txt | wc -l
                                                              1421351

                                                              1.4mio entries? That can't be right?

                                                              cat pfB_Top_v4.txt | wc -l
                                                                22410

                                                              Could this be the the root cause of all this?

                                                              1 Reply Last reply Reply Quote 0
                                                              • RonpfS
                                                                RonpfS last edited by

                                                                @Perforado:

                                                                /var/db/aliastables:
                                                                -rw-r–r--  1 root  wheel    351450 Aug  5 10:56 pfB_Top_v4.txt
                                                                -rw-r--r--  1 root  wheel  30690970 Aug  5 11:00 pfB_Top_v6.txt

                                                                cat pfB_Top_v6.txt | wc -l
                                                                1421351

                                                                1.4mio entries? That can't be right?

                                                                cat pfB_Top_v4.txt | wc -l
                                                                  22410

                                                                Could this be the the root cause of all this?

                                                                What was the size of pfB_Top_v6.txt before the MaxMind db change ?  I do not use these table, so I can't compare.

                                                                On option BBcan177 mentioned was that he might need to aggregate the table  to shrink them.

                                                                1 Reply Last reply Reply Quote 0
                                                                • Perforado
                                                                  Perforado Rebel Alliance last edited by

                                                                  pfB_Top_v6 was about 13000-ish before as far as i recall.

                                                                  1 Reply Last reply Reply Quote 0
                                                                  • W
                                                                    wiz561 last edited by

                                                                    @RonpfS:

                                                                    What about posting pfblockerNG, system log, crash report, screen shot of system activity, etc, so we can see what is happening on your setup?

                                                                    The crash report you posted earlier tells me you have under 400MB defined.

                                                                    PHP Fatal error:  Allowed memory size of 402653184 bytes exhausted

                                                                    Did you raise the Firewall Maximum Table Entries ?

                                                                    I've been busy the past couple of days and blowing it away and restoring the config was pretty simple.  I'm going to try to work on this a bit more this week and take the suggestions of what others have posted to see if it fixes it.  I am going to guess that changing the memory settings around will help, but I also need to buy more memory for my system.

                                                                    1 Reply Last reply Reply Quote 0
                                                                    • R
                                                                      Rickinfl last edited by

                                                                      Hi,

                                                                      I've been reading this forum and trying to figure out if there is a fix for this or not. I really didn't see anyone say "This is the fix" with instructions.

                                                                      Can someone point me in the right direction?
                                                                      Has anyone contacted the package creator?
                                                                      Why hasn't anyone pulled this package from being install if there is issues with it?

                                                                      They should pull this package if its not working. It pretty much killed my pfsense box and I had to remove it.

                                                                      Sorry for being so direct. But I just had to shut down my firewall ports to my websites and I'm trying to get this fixed as soon as possible so I can bring them back online.

                                                                      Thanks,
                                                                      Rick

                                                                      1 Reply Last reply Reply Quote 0
                                                                      • D
                                                                        duanes last edited by

                                                                        Me Too…..
                                                                        (NOTE - malloc failure still shows 512mb of ram.  My mem limit seems to be ignored)

                                                                        I'm using
                                                                        pfb_global();
                                                                        ini_set('memory_limit', '4096M');

                                                                        Still fails when updating with....

                                                                        amd64
                                                                        10.3-RELEASE-p5
                                                                        FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016    root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense

                                                                        Crash report details:

                                                                        PHP Errors:
                                                                        [08-Aug-2016 07:47:51 America/Chicago] PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 72 bytes) in /usr/local/pkg/pfblockerng/pfblockerng.inc on line 3876
                                                                        [08-Aug-2016 07:47:51 America/Chicago] PHP Stack trace:
                                                                        [08-Aug-2016 07:47:51 America/Chicago] PHP  1. {main}() /usr/local/www/pfblockerng/pfblockerng.php:0
                                                                        [08-Aug-2016 07:47:51 America/Chicago] PHP  2. sync_package_pfblockerng() /usr/local/www/pfblockerng/pfblockerng.php:87
                                                                        [08-Aug-2016 07:47:51 America/Chicago] PHP  3. file() /usr/local/pkg/pfblockerng/pfblockerng.inc:3876

                                                                        1 Reply Last reply Reply Quote 0
                                                                        • Perforado
                                                                          Perforado Rebel Alliance last edited by

                                                                          All the php-scripts spawned by the gui are constrained by the memory-limit set by suhosin. Which is 512MB.

                                                                          That's what my perl-one-liner was for. I even increased the limit to 4GB for testing today …

                                                                          1 Reply Last reply Reply Quote 0
                                                                          • P
                                                                            paftdunk last edited by

                                                                            @lucasrca:

                                                                            How I solved my problem:…

                                                                            1. Updated Firewall Maximum Table Entries: 4000000 -> 8000000

                                                                            This ended up being the missing link for me. My default was 2M. When Perforado mentioned the count in  /var/db/aliastables/ I checked mine: 4.4M in those lists alone. I bumped my max table entries in System / Advanced / Firewall & NAT from 2M to 10M and pfblockerng started working again.

                                                                            1 Reply Last reply Reply Quote 0
                                                                            • RonpfS
                                                                              RonpfS last edited by

                                                                              @Perforado:

                                                                              All the php-scripts spawned by the gui are constrained by the memory-limit set by suhosin. Which is 512MB.

                                                                              That's what my perl-one-liner was for. I even increased the limit to 4GB for testing today …

                                                                              For those wandering about suhosin, it is defined in /usr/local/etc/php.ini

                                                                              ; File generated from /etc/rc.php_ini_setup
                                                                              output_buffering = "0"
                                                                              expose_php = Off
                                                                              implicit_flush = true
                                                                              magic_quotes_gpc = Off
                                                                              max_execution_time = 900
                                                                              request_terminate_timeout = 900
                                                                              max_input_time = 1800
                                                                              max_input_vars = 5000
                                                                              register_argc_argv = On
                                                                              register_long_arrays = Off
                                                                              variables_order = "GPCS"
                                                                              file_uploads = On
                                                                              upload_tmp_dir = /tmp
                                                                              upload_max_filesize = 200M
                                                                              post_max_size = 200M
                                                                              html_errors = Off
                                                                              zlib.output_compression = Off
                                                                              zlib.output_compression_level = 1
                                                                              include_path = ".:/etc/inc:/usr/local/www:/usr/local/captiveportal:/usr/local/pkg:/usr/local/www/classes:/usr/local/www/classes/Form"
                                                                              display_startup_errors=on
                                                                              display_errors=on
                                                                              log_errors=on
                                                                              error_log=/tmp/PHP_errors.log
                                                                              extension_dir=/usr/local/lib/php/20131226/
                                                                              date.timezone="America/New_York"
                                                                              session.hash_bits_per_character = 5
                                                                              session.hash_function = 1
                                                                              
                                                                              ; Extensions
                                                                              
                                                                              ; opcache Settings
                                                                              opcache.enabled="1"
                                                                              opcache.enable_cli="0"
                                                                              opcache.memory_consumption="50"
                                                                              
                                                                              [suhosin]
                                                                              suhosin.get.max_array_index_length = 256
                                                                              suhosin.get.max_vars = 5000
                                                                              suhosin.get.max_value_length = 500000
                                                                              suhosin.post.max_array_index_length = 256
                                                                              suhosin.post.max_vars = 5000
                                                                              suhosin.post.max_value_length = 500000
                                                                              suhosin.request.max_array_index_length = 256
                                                                              suhosin.request.max_vars = 5000
                                                                              suhosin.request.max_value_length = 500000
                                                                              suhosin.memory_limit = 536870912
                                                                              
                                                                              

                                                                              Setting 'memory_limit' in the inc file is maxed by the suhosin.memory_limit.

                                                                              The memory_limit is defined in /etc/inc/config.inc

                                                                              // Set memory limit to 512M on amd64.
                                                                              if ($ARCH == "amd64") {
                                                                              	ini_set("memory_limit", "512M");
                                                                              } else {
                                                                              	ini_set("memory_limit", "128M");
                                                                              }
                                                                              

                                                                              So for those using many IPV6 GeoIP table on amd64, they probably have to change /usr/local/etc/php.ini, /etc/rc.php_ini_setup, pfblockerng.inc as well as the Firewall Maximum Table Entries

                                                                              The php memory issues are not specific to pfBlockerNG, backup crashes when backup size is too big, the Diagnostics/Tables will crash when viewing huge table, etc.

                                                                              1 Reply Last reply Reply Quote 0
                                                                              • R
                                                                                richcj10gmail.com last edited by

                                                                                I am having similar issues + NAT / routing was not working at all.

                                                                                error:

                                                                                
                                                                                					Crash report begins.  Anonymous machine information:
                                                                                
                                                                                amd64
                                                                                10.3-RELEASE-p5
                                                                                FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016     root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense
                                                                                
                                                                                Crash report details:
                                                                                
                                                                                PHP Errors:
                                                                                [09-Aug-2016 11:27:29 America/New_York] PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 72 bytes) in /usr/local/pkg/pfblockerng/pfblockerng.inc on line 3875
                                                                                [09-Aug-2016 11:27:29 America/New_York] PHP Stack trace:
                                                                                [09-Aug-2016 11:27:29 America/New_York] PHP   1\. {main}() /etc/rc.start_packages:0
                                                                                [09-Aug-2016 11:27:29 America/New_York] PHP   2\. sync_package() /etc/rc.start_packages:90
                                                                                [09-Aug-2016 11:27:29 America/New_York] PHP   3\. eval() /etc/inc/pkg-utils.inc:631
                                                                                [09-Aug-2016 11:27:29 America/New_York] PHP   4\. sync_package_pfblockerng() /etc/inc/pkg-utils.inc(631) : eval()'d code:3
                                                                                [09-Aug-2016 11:27:29 America/New_York] PHP   5\. file() /usr/local/pkg/pfblockerng/pfblockerng.inc:3875
                                                                                
                                                                                

                                                                                I changed the fire wall rule # to 8000000 and added  ini_set("memory_limit", "768M");
                                                                                I still see the error above. But I at lest have routing back.

                                                                                1 Reply Last reply Reply Quote 0
                                                                                • RonpfS
                                                                                  RonpfS last edited by

                                                                                  @richcj10@gmail.com:

                                                                                  I am having similar issues + NAT / routing was not working at all.

                                                                                  error:

                                                                                  
                                                                                  					Crash report begins.  Anonymous machine information:
                                                                                  
                                                                                  amd64
                                                                                  10.3-RELEASE-p5
                                                                                  FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016     root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense
                                                                                  
                                                                                  Crash report details:
                                                                                  
                                                                                  PHP Errors:
                                                                                  [09-Aug-2016 11:27:29 America/New_York] PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 72 bytes) in /usr/local/pkg/pfblockerng/pfblockerng.inc on line 3875
                                                                                  [09-Aug-2016 11:27:29 America/New_York] PHP Stack trace:
                                                                                  [09-Aug-2016 11:27:29 America/New_York] PHP   1\. {main}() /etc/rc.start_packages:0
                                                                                  [09-Aug-2016 11:27:29 America/New_York] PHP   2\. sync_package() /etc/rc.start_packages:90
                                                                                  [09-Aug-2016 11:27:29 America/New_York] PHP   3\. eval() /etc/inc/pkg-utils.inc:631
                                                                                  [09-Aug-2016 11:27:29 America/New_York] PHP   4\. sync_package_pfblockerng() /etc/inc/pkg-utils.inc(631) : eval()'d code:3
                                                                                  [09-Aug-2016 11:27:29 America/New_York] PHP   5\. file() /usr/local/pkg/pfblockerng/pfblockerng.inc:3875
                                                                                  
                                                                                  

                                                                                  I changed the fire wall rule # to 8000000 and added  ini_set("memory_limit", "768M");
                                                                                  I still see the error above. But I at lest have routing back.

                                                                                  Did you fix  /usr/local/etc/php.ini, /etc/rc.php_ini_setup ?

                                                                                  1 Reply Last reply Reply Quote 0
                                                                                  • M
                                                                                    marian78 last edited by

                                                                                    same problem:

                                                                                    Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 8388608 bytes) in /usr/local/pkg/pfblockerng/pfblockerng.inc on line 3867 Call Stack: 0.0001 245984 1\. {main}() /usr/local/www/pkg_edit.php:0 0.4857 2721392 2\. eval('global $pfb; $pfb['save'] = TRUE; sync_package_pfblockerng();') /usr/local/www/pkg_edit.php:253 0.4857 2722328 3\. sync_package_pfblockerng() /usr/local/www/pkg_edit.php(253) : eval()'d code:3 3.9159 5449944 4\. file() /usr/local/pkg/pfblockerng/pfblockerng.inc:3867 PHP ERROR: Type: 1, File: /usr/local/pkg/pfblockerng/pfblockerng.inc, Line: 3867, Message: Allowed memory size of 134217728 bytes exhausted (tried to allocate 8388608 bytes)
                                                                                    

                                                                                    Crash report begins.  Anonymous machine information:

                                                                                    amd64
                                                                                    10.3-RELEASE-p5
                                                                                    FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016    root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense

                                                                                    Crash report details:

                                                                                    PHP Errors:
                                                                                    [10-Aug-2016 12:50:00 Europe/Bratislava] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 8388608 bytes) in /usr/local/pkg/pfblockerng/pfblockerng.inc on line 3867
                                                                                    [10-Aug-2016 12:50:00 Europe/Bratislava] PHP Stack trace:
                                                                                    [10-Aug-2016 12:50:00 Europe/Bratislava] PHP  1. {main}() /usr/local/www/pkg_edit.php:0
                                                                                    [10-Aug-2016 12:50:00 Europe/Bratislava] PHP  2. eval() /usr/local/www/pkg_edit.php:253
                                                                                    [10-Aug-2016 12:50:00 Europe/Bratislava] PHP  3. sync_package_pfblockerng() /usr/local/www/pkg_edit.php(253) : eval()'d code:3
                                                                                    [10-Aug-2016 12:50:00 Europe/Bratislava] PHP  4. file() /usr/local/pkg/pfblockerng/pfblockerng.inc:3867
                                                                                    [10-Aug-2016 12:54:33 Europe/Bratislava] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 8388608 bytes) in /usr/local/pkg/pfblockerng/pfblockerng.inc on line 3867
                                                                                    [10-Aug-2016 12:54:33 Europe/Bratislava] PHP Stack trace:
                                                                                    [10-Aug-2016 12:54:33 Europe/Bratislava] PHP  1. {main}() /usr/local/www/pkg_edit.php:0
                                                                                    [10-Aug-2016 12:54:33 Europe/Bratislava] PHP  2. eval() /usr/local/www/pkg_edit.php:253
                                                                                    [10-Aug-2016 12:54:33 Europe/Bratislava] PHP  3. sync_package_pfblockerng() /usr/local/www/pkg_edit.php(253) : eval()'d code:3
                                                                                    [10-Aug-2016 12:54:33 Europe/Bratislava] PHP  4. file() /usr/local/pkg/pfblockerng/pfblockerng.inc:3867

                                                                                    1 Reply Last reply Reply Quote 0
                                                                                    • First post
                                                                                      Last post

                                                                                    Products

                                                                                    • Platform Overview
                                                                                    • TNSR
                                                                                    • pfSense
                                                                                    • Appliances

                                                                                    Services

                                                                                    • Training
                                                                                    • Professional Services

                                                                                    Support

                                                                                    • Subscription Plans
                                                                                    • Contact Support
                                                                                    • Product Lifecycle
                                                                                    • Documentation

                                                                                    News

                                                                                    • Media Coverage
                                                                                    • Press
                                                                                    • Events

                                                                                    Resources

                                                                                    • Blog
                                                                                    • FAQ
                                                                                    • Find a Partner
                                                                                    • Resource Library
                                                                                    • Security Information

                                                                                    Company

                                                                                    • About Us
                                                                                    • Careers
                                                                                    • Partners
                                                                                    • Contact Us
                                                                                    • Legal
                                                                                    Our Mission

                                                                                    We provide leading-edge network security at a fair price - regardless of organizational size or network sophistication. We believe that an open-source security model offers disruptive pricing along with the agility required to quickly address emerging threats.

                                                                                    Subscribe to our Newsletter

                                                                                    Product information, software announcements, and special offers. See our newsletter archive to sign up for future newsletters and to read past announcements.

                                                                                    © 2021 Rubicon Communications, LLC | Privacy Policy