PFBlockerNG 2.1.1_2 Memory Errors
-
Temporary Fix for
php /usr/local/www/pfblockerng/pfblockerng.php update
Failing with memory exhaustion:
cp /etc/rc.php_ini_setup /etc/rc.php_ini_setup.BACKUP
cp /usr/local/etc/php.ini /usr/local/etc/php.ini.BACKUP
perl -pi -e 's/536870912/671088640/g' /etc/rc.php_ini_setup /usr/local/etc/php.ini512 * 1024 * 1024 -> 536870912
640 * 1024 * 1024 -> 671088640 works for me. maybe your setup needs more :)There is probably a setting (memory_limit?) we can configure in the System/Advanced/System Tunables that will do that for you. But it may require a reboot.
The fix for the inc file is specific to pfblocker and shouldn't be needed once BBcan177 change the code.
-
/var/db/aliastables:
-rw-r–r-- 1 root wheel 351450 Aug 5 10:56 pfB_Top_v4.txt
-rw-r--r-- 1 root wheel 30690970 Aug 5 11:00 pfB_Top_v6.txtcat pfB_Top_v6.txt | wc -l
14213511.4mio entries? That can't be right?
cat pfB_Top_v4.txt | wc -l
22410Could this be the the root cause of all this?
-
/var/db/aliastables:
-rw-r–r-- 1 root wheel 351450 Aug 5 10:56 pfB_Top_v4.txt
-rw-r--r-- 1 root wheel 30690970 Aug 5 11:00 pfB_Top_v6.txtcat pfB_Top_v6.txt | wc -l
14213511.4mio entries? That can't be right?
cat pfB_Top_v4.txt | wc -l
22410Could this be the the root cause of all this?
What was the size of pfB_Top_v6.txt before the MaxMind db change ? I do not use these table, so I can't compare.
On option BBcan177 mentioned was that he might need to aggregate the table to shrink them.
-
pfB_Top_v6 was about 13000-ish before as far as i recall.
-
What about posting pfblockerNG, system log, crash report, screen shot of system activity, etc, so we can see what is happening on your setup?
The crash report you posted earlier tells me you have under 400MB defined.
PHP Fatal error: Allowed memory size of 402653184 bytes exhausted
Did you raise the Firewall Maximum Table Entries ?
I've been busy the past couple of days and blowing it away and restoring the config was pretty simple. I'm going to try to work on this a bit more this week and take the suggestions of what others have posted to see if it fixes it. I am going to guess that changing the memory settings around will help, but I also need to buy more memory for my system.
-
Hi,
I've been reading this forum and trying to figure out if there is a fix for this or not. I really didn't see anyone say "This is the fix" with instructions.
Can someone point me in the right direction?
Has anyone contacted the package creator?
Why hasn't anyone pulled this package from being install if there is issues with it?They should pull this package if its not working. It pretty much killed my pfsense box and I had to remove it.
Sorry for being so direct. But I just had to shut down my firewall ports to my websites and I'm trying to get this fixed as soon as possible so I can bring them back online.
Thanks,
Rick -
Me Too…..
(NOTE - malloc failure still shows 512mb of ram. My mem limit seems to be ignored)I'm using
pfb_global();
ini_set('memory_limit', '4096M');Still fails when updating with....
amd64
10.3-RELEASE-p5
FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016 root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSenseCrash report details:
PHP Errors:
[08-Aug-2016 07:47:51 America/Chicago] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 72 bytes) in /usr/local/pkg/pfblockerng/pfblockerng.inc on line 3876
[08-Aug-2016 07:47:51 America/Chicago] PHP Stack trace:
[08-Aug-2016 07:47:51 America/Chicago] PHP 1. {main}() /usr/local/www/pfblockerng/pfblockerng.php:0
[08-Aug-2016 07:47:51 America/Chicago] PHP 2. sync_package_pfblockerng() /usr/local/www/pfblockerng/pfblockerng.php:87
[08-Aug-2016 07:47:51 America/Chicago] PHP 3. file() /usr/local/pkg/pfblockerng/pfblockerng.inc:3876 -
All the php-scripts spawned by the gui are constrained by the memory-limit set by suhosin. Which is 512MB.
That's what my perl-one-liner was for. I even increased the limit to 4GB for testing today …
-
How I solved my problem:…
- Updated Firewall Maximum Table Entries: 4000000 -> 8000000
This ended up being the missing link for me. My default was 2M. When Perforado mentioned the count in /var/db/aliastables/ I checked mine: 4.4M in those lists alone. I bumped my max table entries in System / Advanced / Firewall & NAT from 2M to 10M and pfblockerng started working again.
-
All the php-scripts spawned by the gui are constrained by the memory-limit set by suhosin. Which is 512MB.
That's what my perl-one-liner was for. I even increased the limit to 4GB for testing today …
For those wandering about suhosin, it is defined in /usr/local/etc/php.ini
; File generated from /etc/rc.php_ini_setup output_buffering = "0" expose_php = Off implicit_flush = true magic_quotes_gpc = Off max_execution_time = 900 request_terminate_timeout = 900 max_input_time = 1800 max_input_vars = 5000 register_argc_argv = On register_long_arrays = Off variables_order = "GPCS" file_uploads = On upload_tmp_dir = /tmp upload_max_filesize = 200M post_max_size = 200M html_errors = Off zlib.output_compression = Off zlib.output_compression_level = 1 include_path = ".:/etc/inc:/usr/local/www:/usr/local/captiveportal:/usr/local/pkg:/usr/local/www/classes:/usr/local/www/classes/Form" display_startup_errors=on display_errors=on log_errors=on error_log=/tmp/PHP_errors.log extension_dir=/usr/local/lib/php/20131226/ date.timezone="America/New_York" session.hash_bits_per_character = 5 session.hash_function = 1 ; Extensions ; opcache Settings opcache.enabled="1" opcache.enable_cli="0" opcache.memory_consumption="50" [suhosin] suhosin.get.max_array_index_length = 256 suhosin.get.max_vars = 5000 suhosin.get.max_value_length = 500000 suhosin.post.max_array_index_length = 256 suhosin.post.max_vars = 5000 suhosin.post.max_value_length = 500000 suhosin.request.max_array_index_length = 256 suhosin.request.max_vars = 5000 suhosin.request.max_value_length = 500000 suhosin.memory_limit = 536870912
Setting 'memory_limit' in the inc file is maxed by the suhosin.memory_limit.
The memory_limit is defined in /etc/inc/config.inc
// Set memory limit to 512M on amd64. if ($ARCH == "amd64") { ini_set("memory_limit", "512M"); } else { ini_set("memory_limit", "128M"); }
So for those using many IPV6 GeoIP table on amd64, they probably have to change /usr/local/etc/php.ini, /etc/rc.php_ini_setup, pfblockerng.inc as well as the Firewall Maximum Table Entries
The php memory issues are not specific to pfBlockerNG, backup crashes when backup size is too big, the Diagnostics/Tables will crash when viewing huge table, etc.
-
I am having similar issues + NAT / routing was not working at all.
error:
Crash report begins. Anonymous machine information: amd64 10.3-RELEASE-p5 FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016 root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense Crash report details: PHP Errors: [09-Aug-2016 11:27:29 America/New_York] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 72 bytes) in /usr/local/pkg/pfblockerng/pfblockerng.inc on line 3875 [09-Aug-2016 11:27:29 America/New_York] PHP Stack trace: [09-Aug-2016 11:27:29 America/New_York] PHP 1\. {main}() /etc/rc.start_packages:0 [09-Aug-2016 11:27:29 America/New_York] PHP 2\. sync_package() /etc/rc.start_packages:90 [09-Aug-2016 11:27:29 America/New_York] PHP 3\. eval() /etc/inc/pkg-utils.inc:631 [09-Aug-2016 11:27:29 America/New_York] PHP 4\. sync_package_pfblockerng() /etc/inc/pkg-utils.inc(631) : eval()'d code:3 [09-Aug-2016 11:27:29 America/New_York] PHP 5\. file() /usr/local/pkg/pfblockerng/pfblockerng.inc:3875
I changed the fire wall rule # to 8000000 and added ini_set("memory_limit", "768M");
I still see the error above. But I at lest have routing back. -
@richcj10@gmail.com:
I am having similar issues + NAT / routing was not working at all.
error:
Crash report begins. Anonymous machine information: amd64 10.3-RELEASE-p5 FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016 root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense Crash report details: PHP Errors: [09-Aug-2016 11:27:29 America/New_York] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 72 bytes) in /usr/local/pkg/pfblockerng/pfblockerng.inc on line 3875 [09-Aug-2016 11:27:29 America/New_York] PHP Stack trace: [09-Aug-2016 11:27:29 America/New_York] PHP 1\. {main}() /etc/rc.start_packages:0 [09-Aug-2016 11:27:29 America/New_York] PHP 2\. sync_package() /etc/rc.start_packages:90 [09-Aug-2016 11:27:29 America/New_York] PHP 3\. eval() /etc/inc/pkg-utils.inc:631 [09-Aug-2016 11:27:29 America/New_York] PHP 4\. sync_package_pfblockerng() /etc/inc/pkg-utils.inc(631) : eval()'d code:3 [09-Aug-2016 11:27:29 America/New_York] PHP 5\. file() /usr/local/pkg/pfblockerng/pfblockerng.inc:3875
I changed the fire wall rule # to 8000000 and added ini_set("memory_limit", "768M");
I still see the error above. But I at lest have routing back.Did you fix /usr/local/etc/php.ini, /etc/rc.php_ini_setup ?
-
same problem:
Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 8388608 bytes) in /usr/local/pkg/pfblockerng/pfblockerng.inc on line 3867 Call Stack: 0.0001 245984 1\. {main}() /usr/local/www/pkg_edit.php:0 0.4857 2721392 2\. eval('global $pfb; $pfb['save'] = TRUE; sync_package_pfblockerng();') /usr/local/www/pkg_edit.php:253 0.4857 2722328 3\. sync_package_pfblockerng() /usr/local/www/pkg_edit.php(253) : eval()'d code:3 3.9159 5449944 4\. file() /usr/local/pkg/pfblockerng/pfblockerng.inc:3867 PHP ERROR: Type: 1, File: /usr/local/pkg/pfblockerng/pfblockerng.inc, Line: 3867, Message: Allowed memory size of 134217728 bytes exhausted (tried to allocate 8388608 bytes)
Crash report begins. Anonymous machine information:
amd64
10.3-RELEASE-p5
FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016 root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSenseCrash report details:
PHP Errors:
[10-Aug-2016 12:50:00 Europe/Bratislava] PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 8388608 bytes) in /usr/local/pkg/pfblockerng/pfblockerng.inc on line 3867
[10-Aug-2016 12:50:00 Europe/Bratislava] PHP Stack trace:
[10-Aug-2016 12:50:00 Europe/Bratislava] PHP 1. {main}() /usr/local/www/pkg_edit.php:0
[10-Aug-2016 12:50:00 Europe/Bratislava] PHP 2. eval() /usr/local/www/pkg_edit.php:253
[10-Aug-2016 12:50:00 Europe/Bratislava] PHP 3. sync_package_pfblockerng() /usr/local/www/pkg_edit.php(253) : eval()'d code:3
[10-Aug-2016 12:50:00 Europe/Bratislava] PHP 4. file() /usr/local/pkg/pfblockerng/pfblockerng.inc:3867
[10-Aug-2016 12:54:33 Europe/Bratislava] PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 8388608 bytes) in /usr/local/pkg/pfblockerng/pfblockerng.inc on line 3867
[10-Aug-2016 12:54:33 Europe/Bratislava] PHP Stack trace:
[10-Aug-2016 12:54:33 Europe/Bratislava] PHP 1. {main}() /usr/local/www/pkg_edit.php:0
[10-Aug-2016 12:54:33 Europe/Bratislava] PHP 2. eval() /usr/local/www/pkg_edit.php:253
[10-Aug-2016 12:54:33 Europe/Bratislava] PHP 3. sync_package_pfblockerng() /usr/local/www/pkg_edit.php(253) : eval()'d code:3
[10-Aug-2016 12:54:33 Europe/Bratislava] PHP 4. file() /usr/local/pkg/pfblockerng/pfblockerng.inc:3867 -
@richcj10@gmail.com:
I am having similar issues + NAT / routing was not working at all.
error:
Crash report begins. Anonymous machine information: amd64 10.3-RELEASE-p5 FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016 root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense Crash report details: PHP Errors: [09-Aug-2016 11:27:29 America/New_York] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 72 bytes) in /usr/local/pkg/pfblockerng/pfblockerng.inc on line 3875 [09-Aug-2016 11:27:29 America/New_York] PHP Stack trace: [09-Aug-2016 11:27:29 America/New_York] PHP 1\. {main}() /etc/rc.start_packages:0 [09-Aug-2016 11:27:29 America/New_York] PHP 2\. sync_package() /etc/rc.start_packages:90 [09-Aug-2016 11:27:29 America/New_York] PHP 3\. eval() /etc/inc/pkg-utils.inc:631 [09-Aug-2016 11:27:29 America/New_York] PHP 4\. sync_package_pfblockerng() /etc/inc/pkg-utils.inc(631) : eval()'d code:3 [09-Aug-2016 11:27:29 America/New_York] PHP 5\. file() /usr/local/pkg/pfblockerng/pfblockerng.inc:3875
I changed the fire wall rule # to 8000000 and added ini_set("memory_limit", "768M");
I still see the error above. But I at lest have routing back.Did you fix /usr/local/etc/php.ini, /etc/rc.php_ini_setup ?
What fix?
-
@richcj10@gmail.com:
@richcj10@gmail.com:
I am having similar issues + NAT / routing was not working at all.
error:
Crash report begins. Anonymous machine information: amd64 10.3-RELEASE-p5 FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016 root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense Crash report details: PHP Errors: [09-Aug-2016 11:27:29 America/New_York] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 72 bytes) in /usr/local/pkg/pfblockerng/pfblockerng.inc on line 3875 [09-Aug-2016 11:27:29 America/New_York] PHP Stack trace: [09-Aug-2016 11:27:29 America/New_York] PHP 1\. {main}() /etc/rc.start_packages:0 [09-Aug-2016 11:27:29 America/New_York] PHP 2\. sync_package() /etc/rc.start_packages:90 [09-Aug-2016 11:27:29 America/New_York] PHP 3\. eval() /etc/inc/pkg-utils.inc:631 [09-Aug-2016 11:27:29 America/New_York] PHP 4\. sync_package_pfblockerng() /etc/inc/pkg-utils.inc(631) : eval()'d code:3 [09-Aug-2016 11:27:29 America/New_York] PHP 5\. file() /usr/local/pkg/pfblockerng/pfblockerng.inc:3875
I changed the fire wall rule # to 8000000 and added ini_set("memory_limit", "768M");
I still see the error above. But I at lest have routing back.Did you fix /usr/local/etc/php.ini, /etc/rc.php_ini_setup ?
What fix?
Temporary Fix for
php /usr/local/www/pfblockerng/pfblockerng.php update
Failing with memory exhaustion:
edit /usr/local/pkg/pfblockerng/pfblockerng.inc as discussed above:
…
pfb_global();
ini_set('memory_limit', '640M');
...cp /etc/rc.php_ini_setup /etc/rc.php_ini_setup.BACKUP
cp /usr/local/etc/php.ini /usr/local/etc/php.ini.BACKUP
perl -pi -e 's/536870912/671088640/g' /etc/rc.php_ini_setup /usr/local/etc/php.ini512 * 1024 * 1024 -> 536870912
640 * 1024 * 1024 -> 671088640 works for me. maybe your setup needs more :)You can edit the files or do the perl command
-
Hello!
I also had this problem. Editing the files as described earlier (including The memory_limit defined in /etc/inc/config.inc where I increased the amd64 limit to 640M) was part of the solution for me.
The last thing for me was to increase the Firewall Maximum Table Entries: 2000000 -> 10000000. When using pfBlockerNG the total of firewall rules are about 4500000 rows now. Reloading those rules requires about 9000000 table entries since that the new rules are loaded before the old ones are deleted ending up with temporary 9000000 million rows in the table.
If you have lesser than 9000000 you will get an out-of-memory error message plus that some rules are not loaded correctly. -
https://forum.pfsense.org/index.php?topic=102470.msg645219#msg645219
-
I've submitted a PR to pfSense's GitHub repo:
https://github.com/pfsense/pfsense/pull/3101After that, the following changes suggested by @Perforado on the package could be implemented and should work.
@Perforado:Temporary Fix for
php /usr/local/www/pfblockerng/pfblockerng.php update
Failing with memory exhaustion:
edit /usr/local/pkg/pfblockerng/pfblockerng.inc as discussed above:
…
pfb_global();
ini_set('memory_limit', '640M');
... -
I don't think you need that PR, if you are following the pfBlockerNG v2.0 w/DNSBL thread, you should have known that we are working on a fix that shouldn't require modifying memory_limit or php.ini.
-
I don't think you need that PR, if you are following the pfBlockerNG v2.0 w/DNSBL thread, you should have known that we are working on a fix that shouldn't require modifying memory_limit or php.ini.
A 55 page, nine month old thread introducing the feature seems like a weird place to have active commentary on fixing the bug reported in this one.