PFBlockerNG 2.1.1_2 Memory Errors
-
All the php-scripts spawned by the gui are constrained by the memory-limit set by suhosin. Which is 512MB.
That's what my perl-one-liner was for. I even increased the limit to 4GB for testing today …
For those wandering about suhosin, it is defined in /usr/local/etc/php.ini
; File generated from /etc/rc.php_ini_setup output_buffering = "0" expose_php = Off implicit_flush = true magic_quotes_gpc = Off max_execution_time = 900 request_terminate_timeout = 900 max_input_time = 1800 max_input_vars = 5000 register_argc_argv = On register_long_arrays = Off variables_order = "GPCS" file_uploads = On upload_tmp_dir = /tmp upload_max_filesize = 200M post_max_size = 200M html_errors = Off zlib.output_compression = Off zlib.output_compression_level = 1 include_path = ".:/etc/inc:/usr/local/www:/usr/local/captiveportal:/usr/local/pkg:/usr/local/www/classes:/usr/local/www/classes/Form" display_startup_errors=on display_errors=on log_errors=on error_log=/tmp/PHP_errors.log extension_dir=/usr/local/lib/php/20131226/ date.timezone="America/New_York" session.hash_bits_per_character = 5 session.hash_function = 1 ; Extensions ; opcache Settings opcache.enabled="1" opcache.enable_cli="0" opcache.memory_consumption="50" [suhosin] suhosin.get.max_array_index_length = 256 suhosin.get.max_vars = 5000 suhosin.get.max_value_length = 500000 suhosin.post.max_array_index_length = 256 suhosin.post.max_vars = 5000 suhosin.post.max_value_length = 500000 suhosin.request.max_array_index_length = 256 suhosin.request.max_vars = 5000 suhosin.request.max_value_length = 500000 suhosin.memory_limit = 536870912Setting 'memory_limit' in the inc file is maxed by the suhosin.memory_limit.
The memory_limit is defined in /etc/inc/config.inc
// Set memory limit to 512M on amd64. if ($ARCH == "amd64") { ini_set("memory_limit", "512M"); } else { ini_set("memory_limit", "128M"); }So for those using many IPV6 GeoIP table on amd64, they probably have to change /usr/local/etc/php.ini, /etc/rc.php_ini_setup, pfblockerng.inc as well as the Firewall Maximum Table Entries
The php memory issues are not specific to pfBlockerNG, backup crashes when backup size is too big, the Diagnostics/Tables will crash when viewing huge table, etc.
-
I am having similar issues + NAT / routing was not working at all.
error:
Crash report begins. Anonymous machine information: amd64 10.3-RELEASE-p5 FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016 root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense Crash report details: PHP Errors: [09-Aug-2016 11:27:29 America/New_York] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 72 bytes) in /usr/local/pkg/pfblockerng/pfblockerng.inc on line 3875 [09-Aug-2016 11:27:29 America/New_York] PHP Stack trace: [09-Aug-2016 11:27:29 America/New_York] PHP 1\. {main}() /etc/rc.start_packages:0 [09-Aug-2016 11:27:29 America/New_York] PHP 2\. sync_package() /etc/rc.start_packages:90 [09-Aug-2016 11:27:29 America/New_York] PHP 3\. eval() /etc/inc/pkg-utils.inc:631 [09-Aug-2016 11:27:29 America/New_York] PHP 4\. sync_package_pfblockerng() /etc/inc/pkg-utils.inc(631) : eval()'d code:3 [09-Aug-2016 11:27:29 America/New_York] PHP 5\. file() /usr/local/pkg/pfblockerng/pfblockerng.inc:3875I changed the fire wall rule # to 8000000 and added ini_set("memory_limit", "768M");
I still see the error above. But I at lest have routing back. -
@richcj10@gmail.com:
I am having similar issues + NAT / routing was not working at all.
error:
Crash report begins. Anonymous machine information: amd64 10.3-RELEASE-p5 FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016 root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense Crash report details: PHP Errors: [09-Aug-2016 11:27:29 America/New_York] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 72 bytes) in /usr/local/pkg/pfblockerng/pfblockerng.inc on line 3875 [09-Aug-2016 11:27:29 America/New_York] PHP Stack trace: [09-Aug-2016 11:27:29 America/New_York] PHP 1\. {main}() /etc/rc.start_packages:0 [09-Aug-2016 11:27:29 America/New_York] PHP 2\. sync_package() /etc/rc.start_packages:90 [09-Aug-2016 11:27:29 America/New_York] PHP 3\. eval() /etc/inc/pkg-utils.inc:631 [09-Aug-2016 11:27:29 America/New_York] PHP 4\. sync_package_pfblockerng() /etc/inc/pkg-utils.inc(631) : eval()'d code:3 [09-Aug-2016 11:27:29 America/New_York] PHP 5\. file() /usr/local/pkg/pfblockerng/pfblockerng.inc:3875I changed the fire wall rule # to 8000000 and added ini_set("memory_limit", "768M");
I still see the error above. But I at lest have routing back.Did you fix /usr/local/etc/php.ini, /etc/rc.php_ini_setup ?
-
same problem:
Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 8388608 bytes) in /usr/local/pkg/pfblockerng/pfblockerng.inc on line 3867 Call Stack: 0.0001 245984 1\. {main}() /usr/local/www/pkg_edit.php:0 0.4857 2721392 2\. eval('global $pfb; $pfb['save'] = TRUE; sync_package_pfblockerng();') /usr/local/www/pkg_edit.php:253 0.4857 2722328 3\. sync_package_pfblockerng() /usr/local/www/pkg_edit.php(253) : eval()'d code:3 3.9159 5449944 4\. file() /usr/local/pkg/pfblockerng/pfblockerng.inc:3867 PHP ERROR: Type: 1, File: /usr/local/pkg/pfblockerng/pfblockerng.inc, Line: 3867, Message: Allowed memory size of 134217728 bytes exhausted (tried to allocate 8388608 bytes)Crash report begins. Anonymous machine information:
amd64
10.3-RELEASE-p5
FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016 root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSenseCrash report details:
PHP Errors:
[10-Aug-2016 12:50:00 Europe/Bratislava] PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 8388608 bytes) in /usr/local/pkg/pfblockerng/pfblockerng.inc on line 3867
[10-Aug-2016 12:50:00 Europe/Bratislava] PHP Stack trace:
[10-Aug-2016 12:50:00 Europe/Bratislava] PHP 1. {main}() /usr/local/www/pkg_edit.php:0
[10-Aug-2016 12:50:00 Europe/Bratislava] PHP 2. eval() /usr/local/www/pkg_edit.php:253
[10-Aug-2016 12:50:00 Europe/Bratislava] PHP 3. sync_package_pfblockerng() /usr/local/www/pkg_edit.php(253) : eval()'d code:3
[10-Aug-2016 12:50:00 Europe/Bratislava] PHP 4. file() /usr/local/pkg/pfblockerng/pfblockerng.inc:3867
[10-Aug-2016 12:54:33 Europe/Bratislava] PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 8388608 bytes) in /usr/local/pkg/pfblockerng/pfblockerng.inc on line 3867
[10-Aug-2016 12:54:33 Europe/Bratislava] PHP Stack trace:
[10-Aug-2016 12:54:33 Europe/Bratislava] PHP 1. {main}() /usr/local/www/pkg_edit.php:0
[10-Aug-2016 12:54:33 Europe/Bratislava] PHP 2. eval() /usr/local/www/pkg_edit.php:253
[10-Aug-2016 12:54:33 Europe/Bratislava] PHP 3. sync_package_pfblockerng() /usr/local/www/pkg_edit.php(253) : eval()'d code:3
[10-Aug-2016 12:54:33 Europe/Bratislava] PHP 4. file() /usr/local/pkg/pfblockerng/pfblockerng.inc:3867 -
@richcj10@gmail.com:
I am having similar issues + NAT / routing was not working at all.
error:
Crash report begins. Anonymous machine information: amd64 10.3-RELEASE-p5 FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016 root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense Crash report details: PHP Errors: [09-Aug-2016 11:27:29 America/New_York] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 72 bytes) in /usr/local/pkg/pfblockerng/pfblockerng.inc on line 3875 [09-Aug-2016 11:27:29 America/New_York] PHP Stack trace: [09-Aug-2016 11:27:29 America/New_York] PHP 1\. {main}() /etc/rc.start_packages:0 [09-Aug-2016 11:27:29 America/New_York] PHP 2\. sync_package() /etc/rc.start_packages:90 [09-Aug-2016 11:27:29 America/New_York] PHP 3\. eval() /etc/inc/pkg-utils.inc:631 [09-Aug-2016 11:27:29 America/New_York] PHP 4\. sync_package_pfblockerng() /etc/inc/pkg-utils.inc(631) : eval()'d code:3 [09-Aug-2016 11:27:29 America/New_York] PHP 5\. file() /usr/local/pkg/pfblockerng/pfblockerng.inc:3875I changed the fire wall rule # to 8000000 and added ini_set("memory_limit", "768M");
I still see the error above. But I at lest have routing back.Did you fix /usr/local/etc/php.ini, /etc/rc.php_ini_setup ?
What fix?
-
@richcj10@gmail.com:
@richcj10@gmail.com:
I am having similar issues + NAT / routing was not working at all.
error:
Crash report begins. Anonymous machine information: amd64 10.3-RELEASE-p5 FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016 root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense Crash report details: PHP Errors: [09-Aug-2016 11:27:29 America/New_York] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 72 bytes) in /usr/local/pkg/pfblockerng/pfblockerng.inc on line 3875 [09-Aug-2016 11:27:29 America/New_York] PHP Stack trace: [09-Aug-2016 11:27:29 America/New_York] PHP 1\. {main}() /etc/rc.start_packages:0 [09-Aug-2016 11:27:29 America/New_York] PHP 2\. sync_package() /etc/rc.start_packages:90 [09-Aug-2016 11:27:29 America/New_York] PHP 3\. eval() /etc/inc/pkg-utils.inc:631 [09-Aug-2016 11:27:29 America/New_York] PHP 4\. sync_package_pfblockerng() /etc/inc/pkg-utils.inc(631) : eval()'d code:3 [09-Aug-2016 11:27:29 America/New_York] PHP 5\. file() /usr/local/pkg/pfblockerng/pfblockerng.inc:3875I changed the fire wall rule # to 8000000 and added ini_set("memory_limit", "768M");
I still see the error above. But I at lest have routing back.Did you fix /usr/local/etc/php.ini, /etc/rc.php_ini_setup ?
What fix?
Temporary Fix for
php /usr/local/www/pfblockerng/pfblockerng.php update
Failing with memory exhaustion:
edit /usr/local/pkg/pfblockerng/pfblockerng.inc as discussed above:
…
pfb_global();
ini_set('memory_limit', '640M');
...cp /etc/rc.php_ini_setup /etc/rc.php_ini_setup.BACKUP
cp /usr/local/etc/php.ini /usr/local/etc/php.ini.BACKUP
perl -pi -e 's/536870912/671088640/g' /etc/rc.php_ini_setup /usr/local/etc/php.ini512 * 1024 * 1024 -> 536870912
640 * 1024 * 1024 -> 671088640 works for me. maybe your setup needs more :)You can edit the files or do the perl command
-
Hello!
I also had this problem. Editing the files as described earlier (including The memory_limit defined in /etc/inc/config.inc where I increased the amd64 limit to 640M) was part of the solution for me.
The last thing for me was to increase the Firewall Maximum Table Entries: 2000000 -> 10000000. When using pfBlockerNG the total of firewall rules are about 4500000 rows now. Reloading those rules requires about 9000000 table entries since that the new rules are loaded before the old ones are deleted ending up with temporary 9000000 million rows in the table.
If you have lesser than 9000000 you will get an out-of-memory error message plus that some rules are not loaded correctly. -
https://forum.pfsense.org/index.php?topic=102470.msg645219#msg645219
-
I've submitted a PR to pfSense's GitHub repo:
https://github.com/pfsense/pfsense/pull/3101After that, the following changes suggested by @Perforado on the package could be implemented and should work.
@Perforado:Temporary Fix for
php /usr/local/www/pfblockerng/pfblockerng.php update
Failing with memory exhaustion:
edit /usr/local/pkg/pfblockerng/pfblockerng.inc as discussed above:
…
pfb_global();
ini_set('memory_limit', '640M');
... -
I don't think you need that PR, if you are following the pfBlockerNG v2.0 w/DNSBL thread, you should have known that we are working on a fix that shouldn't require modifying memory_limit or php.ini.
-
I don't think you need that PR, if you are following the pfBlockerNG v2.0 w/DNSBL thread, you should have known that we are working on a fix that shouldn't require modifying memory_limit or php.ini.
A 55 page, nine month old thread introducing the feature seems like a weird place to have active commentary on fixing the bug reported in this one.
-
This thread is about fixing memory issues introduced by MaxMind database changes.
The post about "Blocking the world" is about people configuration that trigger the bug.
It is normally useless to Block the world and the comment is in the right place in the pfBlockerNG w/DNSBL thread.The thread has 55 pages, because pfBlockerNG is not trivial to configure.
Since when the size of a thread make it irrelevant to read it? -
Following the following instructions I have trashed my pfSense install and receive the following stack trace on boot:
Fatal Error: Allowed memory size of 262144 bytes exhausted (tried to allocate 49152 bytes) in /etc/inc/interfaces.inc on line 568
Call Stack:
0.0002 219464 1. {main}() /etc/rc.conf_mount_ro:0
0.0012 259568 2: require_once('etc/inc/config.inc') /etc/rc.conf_mount_ro:55
0.0043 366712 3. require_once('/etc/inc/notices.inc') /etc/inc/config.inc:56
0.0050 396632 4. require_once('/etc/inc/functions.inc') /etc/inc/notices.inc:56No options work
I followed the post below, and then ran the geoip download which completed successfully. Then while running the update from pfBlocker it froze, and the network went down. Now the DHCP does not assign addresses to the network and I cannot access the box except going directly to it with KBB and monitor.
Reverting changes and rebooting does not resolve. Running any option from the terminal returns the above error. If only the devs hadn't removed the script for manual backups and recovery….
All the php-scripts spawned by the gui are constrained by the memory-limit set by suhosin. Which is 512MB.
That's what my perl-one-liner was for. I even increased the limit to 4GB for testing today …
For those wandering about suhosin, it is defined in /usr/local/etc/php.ini
; File generated from /etc/rc.php_ini_setup output_buffering = "0" expose_php = Off implicit_flush = true magic_quotes_gpc = Off max_execution_time = 900 request_terminate_timeout = 900 max_input_time = 1800 max_input_vars = 5000 register_argc_argv = On register_long_arrays = Off variables_order = "GPCS" file_uploads = On upload_tmp_dir = /tmp upload_max_filesize = 200M post_max_size = 200M html_errors = Off zlib.output_compression = Off zlib.output_compression_level = 1 include_path = ".:/etc/inc:/usr/local/www:/usr/local/captiveportal:/usr/local/pkg:/usr/local/www/classes:/usr/local/www/classes/Form" display_startup_errors=on display_errors=on log_errors=on error_log=/tmp/PHP_errors.log extension_dir=/usr/local/lib/php/20131226/ date.timezone="America/New_York" session.hash_bits_per_character = 5 session.hash_function = 1 ; Extensions ; opcache Settings opcache.enabled="1" opcache.enable_cli="0" opcache.memory_consumption="50" [suhosin] suhosin.get.max_array_index_length = 256 suhosin.get.max_vars = 5000 suhosin.get.max_value_length = 500000 suhosin.post.max_array_index_length = 256 suhosin.post.max_vars = 5000 suhosin.post.max_value_length = 500000 suhosin.request.max_array_index_length = 256 suhosin.request.max_vars = 5000 suhosin.request.max_value_length = 500000 suhosin.memory_limit = 536870912Setting 'memory_limit' in the inc file is maxed by the suhosin.memory_limit.
The memory_limit is defined in /etc/inc/config.inc
// Set memory limit to 512M on amd64. if ($ARCH == "amd64") { ini_set("memory_limit", "512M"); } else { ini_set("memory_limit", "128M"); }So for those using many IPV6 GeoIP table on amd64, they probably have to change /usr/local/etc/php.ini, /etc/rc.php_ini_setup, pfblockerng.inc as well as the Firewall Maximum Table Entries
The php memory issues are not specific to pfBlockerNG, backup crashes when backup size is too big, the Diagnostics/Tables will crash when viewing huge table, etc.
This is a AsRock Q1900M w/quad core celeron J1900, 8GB RAM, 1TB HDD, 1 intel dual GbE, 1 intel single GbE cards.
-
So what modification did you do.
On i386, only the pfblockerng.inc fix is normally required.The config.inc was just stated for referenced.If you modified it, maybe you forgot to put the "M" in 256M? (262144 bytes exhausted)The /etc/rc.php_ini_setup & /usr/local/etc/php.ini fix is for amd64 that need more than 512MB.
-
Thanks Ron…a little M goes a long way to making the system work. 2048 instead of 2048M.
-
Thanks Ron…a little M goes a long way to making the system work. 2048 instead of 2048M.
And where did you made the modifications ? 2048M in config.inc might break something else somewhere.
-
// Set memory limit to 512M on amd64.
if ($ARCH == "amd64") {
ini_set("memory_limit", "512M");
} else {
ini_set("memory_limit", "128M");
}changed to
// Set memory limit to 512M on amd64.
if ($ARCH == "amd64") {
ini_set("memory_limit", "2048M");
} else {
ini_set("memory_limit", "128M");
}reverting to default shouldn't hurt as pfBlocker is uninstalled until a real fix is in place.
-
I would put it back to 512MB.
Modify the pfblockerng.inc instead, this will only affect the pfblocker pkg.
However, the php.ini limit is 512M, so if you need more than 512M, you will need both fixes, the one for php and the on for pfblockerng.inc. -
Will do, thanks.
-
Hi All,
I think i have a better solution to this although feel free to shoot it down if i've messed up.Basically, say you only want the UK to be permitted, but not the ROTW…
If you deny everything then it takes a a shed load of memory and causes the problems above. However my solution is to only PERMIT what you want and then DENY everything that isnt permitted. You dont need the entire Maxmind database, only the IP's of what you want to permit.
To do this, go into PfblockerNG, GeoIP, Top20 and select only UK (or your country /countries) and then in List action select Permit Inbound.
This will create a floating rule in Firewall tab that is called pfB_Top_v4 that contains all the IP's of the countries you select to be permitted from that tab.
Now create another rule that is EXACTLY the same as the above one in floating rules, except tick the "Invert Match" box and change the rule to a block or reject rule then add it before the auto generated pfB_Top_v4 rule and name it !pfB_Top_v4 or something similar.
Now all IP's that aren't UK based (in my example) will be denied and whenever you update the pfB_Top_v4 alias, it will also update your deny rule automatically too.
You may need to duplicate multiple rules if your permits appear in other Geo Locations like Africa or Antartica for example as this will create additional Floating rules.
This also has the massive benefit of having an order of magnitude less rules in your Firewall table as you only checking a specific permitted countries IP range as opposed to the ROTW's IP ranges.
