PFBlockerNG 2.1.1_2 Memory Errors

lucasrca · Aug 4, 2016, 2:02 PM

How I solved my problem:

My pfSense config:

Version: 2.3.2 (amd64), running on VMWare 6
– Snort
-- pfBlockerNG
-- OpenVPN
-- Open-vm-tools
-- DHCP Relay
-- Quagga OSPFd with another 2 pfSense.
~3000 users simultaneously
2 x 100 Mbit uplinks
16 GB RAM
80 GB SAS
CPU Type: Intel(R) Xeon(R) CPU E5-4620 v2 @ 2.60GHz
16 CPUs: 8 package(s) x 2 core(s)
8 Interfaces, including WAN
routing, filtering and relaying dhcp to 16 branches over MPLS, WiMax and fiber
Load balance and failover
QoS with Traffic shaper

Updated Firewall Maximum Table Entries: 4000000 -> 8000000
1.1) Reboot
Edited /usr/local/pkg/pfblockerng/pfblockerng.inc and set memory limit to 500M
Executed php /usr/local/www/pfblockerng/pfblockerng.php dc
It's alive.

Thanks to all involved.

Mithrondil · Aug 4, 2016, 2:34 PM

POST-INSTALL script failed
Message from GeoIP-1.6.9:
GeoIP does not ship with the actual data files. You must download
them yourself! Please run:

/usr/local/bin/geoipupdate.sh

RonpfS · Aug 4, 2016, 4:21 PM

@wiz561:

I started a thread up over in the pfblocker posting and just letting everybody know that I'm also experiencing very similar memory issues.

Thismorning when I logged in, I also had a pfsense crash report with the following:

					Crash report begins.  Anonymous machine information:

amd64
10.3-RELEASE-p5
FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016     root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense

Crash report details:

PHP Errors:
[04-Aug-2016 00:18:40 America/Chicago] PHP Fatal error:  Allowed memory size of 402653184 bytes exhausted (tried to allocate 72 bytes) in /usr/local/pkg/pfblockerng/pfblockerng.inc on line 3868
[04-Aug-2016 00:18:40 America/Chicago] PHP Stack trace:
[04-Aug-2016 00:18:40 America/Chicago] PHP   1\. {main}() /etc/rc.start_packages:0
[04-Aug-2016 00:18:40 America/Chicago] PHP   2\. sync_package() /etc/rc.start_packages:90
[04-Aug-2016 00:18:40 America/Chicago] PHP   3\. eval() /etc/inc/pkg-utils.inc:631
[04-Aug-2016 00:18:40 America/Chicago] PHP   4\. sync_package_pfblockerng() /etc/inc/pkg-utils.inc(631) : eval()'d code:3
[04-Aug-2016 00:18:40 America/Chicago] PHP   5\. array_merge() /usr/local/pkg/pfblockerng/pfblockerng.inc:3868

Filename: /var/crash/minfree
2048

The php memory should be 512M by default, so setting it in the inc file to 400M might not help.
Did it fail without any fix to the inc file? Can you try setting the limit to 640M or 768M?
What do you have for Firewall Maximum Table Entries?

RonpfS · Aug 4, 2016, 4:24 PM

@Mithrondil:

POST-INSTALL script failed
Message from GeoIP-1.6.9:
GeoIP does not ship with the actual data files. You must download
them yourself! Please run:

/usr/local/bin/geoipupdate.sh

You will need to run php /usr/local/www/pfblockerng/pfblockerng.php dc from the shell.
Once it succeeds, you should be able to install, it may need a reboot.

RonpfS · Aug 5, 2016, 7:39 AM

For those with failed installation. Verify that /var isn't full. The MaxMind database is huge so if you are using a RAM Disk, it might eat up memory that is needed for the pfblockerng.php.

If disk space is running low, BBcan177 suggest to delete the /var/db/pfblockerng/deny and /var/db/pfblockerng/original folders before installation to free some disk space. This means it will need to redownload all IP feeds after installation.

Check pfblockerng.log, the system log, Dashboard for crash report, Status Monitoring System Memory.

Post relevant debug info here.

wiz561 · Aug 4, 2016, 10:35 PM

@RonpfS:

@wiz561:

I started a thread up over in the pfblocker posting and just letting everybody know that I'm also experiencing very similar memory issues.

Thismorning when I logged in, I also had a pfsense crash report with the following:

					Crash report begins.  Anonymous machine information:

amd64
10.3-RELEASE-p5
FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016     root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense

Crash report details:

PHP Errors:
[04-Aug-2016 00:18:40 America/Chicago] PHP Fatal error:  Allowed memory size of 402653184 bytes exhausted (tried to allocate 72 bytes) in /usr/local/pkg/pfblockerng/pfblockerng.inc on line 3868
[04-Aug-2016 00:18:40 America/Chicago] PHP Stack trace:
[04-Aug-2016 00:18:40 America/Chicago] PHP   1\. {main}() /etc/rc.start_packages:0
[04-Aug-2016 00:18:40 America/Chicago] PHP   2\. sync_package() /etc/rc.start_packages:90
[04-Aug-2016 00:18:40 America/Chicago] PHP   3\. eval() /etc/inc/pkg-utils.inc:631
[04-Aug-2016 00:18:40 America/Chicago] PHP   4\. sync_package_pfblockerng() /etc/inc/pkg-utils.inc(631) : eval()'d code:3
[04-Aug-2016 00:18:40 America/Chicago] PHP   5\. array_merge() /usr/local/pkg/pfblockerng/pfblockerng.inc:3868

Filename: /var/crash/minfree
2048

The php memory should be 512M by default, so setting it in the inc file to 400M might not help.
Did it fail without any fix to the inc file? Can you try setting the limit to 640M or 768M?
What do you have for Firewall Maximum Table Entries?

I'm going to have to do a bit of work this weekend on this and see more. I can try it and see what happens, but I'm getting concerned about reaching my 2gb memory limit. I can add more memory, but that requires me to go to the store and I'm kinda lazy…..and cheap. :)

Interesting that I too have a similar setup to the one user above. I'm running it on vmware esxi with Snort (disabled) and OpenVPN. Granted, I don't have nearly the hardware or setup, but it's interesting that we're both running vmware.

wiz561 · Aug 5, 2016, 3:34 AM

OK… Now maybe I'll try what others suggested... :)

I wiped and reinstalled pfsense tonight and pfblockng is still coming back with that crash and memory errors. I know others said to adjust the memory but I thought I would give this a try. Unfortunately, it failed.

I also had issues with php-fpm having high utilization....so I'm hoping that the wipe/reinstall fixed the issue with that. Time will tell.

RonpfS · Aug 5, 2016, 6:55 AM

What about posting pfblockerNG, system log, crash report, screen shot of system activity, etc, so we can see what is happening on your setup?

The crash report you posted earlier tells me you have under 400MB defined.

PHP Fatal error: Allowed memory size of 402653184 bytes exhausted

Did you raise the Firewall Maximum Table Entries ?

Perforado · Aug 5, 2016, 8:31 AM

Temporary Fix for

php /usr/local/www/pfblockerng/pfblockerng.php update

Failing with memory exhaustion:

edit /usr/local/pkg/pfblockerng/pfblockerng.inc as discussed above:
…
pfb_global();
ini_set('memory_limit', '640M');
...

cp /etc/rc.php_ini_setup /etc/rc.php_ini_setup.BACKUP
cp /usr/local/etc/php.ini /usr/local/etc/php.ini.BACKUP
perl -pi -e 's/536870912/671088640/g' /etc/rc.php_ini_setup /usr/local/etc/php.ini

512 * 1024 * 1024 -> 536870912
640 * 1024 * 1024 -> 671088640 works for me. maybe your setup needs more :)

RonpfS · Aug 5, 2016, 9:20 AM

@Perforado:

Temporary Fix for

php /usr/local/www/pfblockerng/pfblockerng.php update

Failing with memory exhaustion:

cp /etc/rc.php_ini_setup /etc/rc.php_ini_setup.BACKUP
cp /usr/local/etc/php.ini /usr/local/etc/php.ini.BACKUP
perl -pi -e 's/536870912/671088640/g' /etc/rc.php_ini_setup /usr/local/etc/php.ini

512 * 1024 * 1024 -> 536870912
640 * 1024 * 1024 -> 671088640 works for me. maybe your setup needs more :)

There is probably a setting (memory_limit?) we can configure in the System/Advanced/System Tunables that will do that for you. But it may require a reboot.

The fix for the inc file is specific to pfblocker and shouldn't be needed once BBcan177 change the code.

Perforado · Aug 5, 2016, 9:24 AM

/var/db/aliastables:
-rw-r–r-- 1 root wheel 351450 Aug 5 10:56 pfB_Top_v4.txt
-rw-r--r-- 1 root wheel 30690970 Aug 5 11:00 pfB_Top_v6.txt

cat pfB_Top_v6.txt | wc -l
1421351

1.4mio entries? That can't be right?

cat pfB_Top_v4.txt | wc -l
22410

Could this be the the root cause of all this?

RonpfS · Aug 5, 2016, 5:22 PM

@Perforado:

/var/db/aliastables:
-rw-r–r-- 1 root wheel 351450 Aug 5 10:56 pfB_Top_v4.txt
-rw-r--r-- 1 root wheel 30690970 Aug 5 11:00 pfB_Top_v6.txt

cat pfB_Top_v6.txt | wc -l
1421351

1.4mio entries? That can't be right?

cat pfB_Top_v4.txt | wc -l
22410

Could this be the the root cause of all this?

What was the size of pfB_Top_v6.txt before the MaxMind db change ? I do not use these table, so I can't compare.

On option BBcan177 mentioned was that he might need to aggregate the table to shrink them.

Perforado · Aug 6, 2016, 6:48 AM

pfB_Top_v6 was about 13000-ish before as far as i recall.

wiz561 · Aug 7, 2016, 1:16 PM

@RonpfS:

What about posting pfblockerNG, system log, crash report, screen shot of system activity, etc, so we can see what is happening on your setup?

The crash report you posted earlier tells me you have under 400MB defined.

PHP Fatal error: Allowed memory size of 402653184 bytes exhausted

Did you raise the Firewall Maximum Table Entries ?

I've been busy the past couple of days and blowing it away and restoring the config was pretty simple. I'm going to try to work on this a bit more this week and take the suggestions of what others have posted to see if it fixes it. I am going to guess that changing the memory settings around will help, but I also need to buy more memory for my system.

Rickinfl · Aug 8, 2016, 12:26 PM

Hi,

I've been reading this forum and trying to figure out if there is a fix for this or not. I really didn't see anyone say "This is the fix" with instructions.

Can someone point me in the right direction?
Has anyone contacted the package creator?
Why hasn't anyone pulled this package from being install if there is issues with it?

They should pull this package if its not working. It pretty much killed my pfsense box and I had to remove it.

Sorry for being so direct. But I just had to shut down my firewall ports to my websites and I'm trying to get this fixed as soon as possible so I can bring them back online.

Thanks,
Rick

duanes · Aug 8, 2016, 1:18 PM

Me Too…..
(NOTE - malloc failure still shows 512mb of ram. My mem limit seems to be ignored)

I'm using
pfb_global();
ini_set('memory_limit', '4096M');

Still fails when updating with....

amd64
10.3-RELEASE-p5
FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016 root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense

Crash report details:

PHP Errors:
[08-Aug-2016 07:47:51 America/Chicago] PHP Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 72 bytes) in /usr/local/pkg/pfblockerng/pfblockerng.inc on line 3876
[08-Aug-2016 07:47:51 America/Chicago] PHP Stack trace:
[08-Aug-2016 07:47:51 America/Chicago] PHP 1. {main}() /usr/local/www/pfblockerng/pfblockerng.php:0
[08-Aug-2016 07:47:51 America/Chicago] PHP 2. sync_package_pfblockerng() /usr/local/www/pfblockerng/pfblockerng.php:87
[08-Aug-2016 07:47:51 America/Chicago] PHP 3. file() /usr/local/pkg/pfblockerng/pfblockerng.inc:3876

Perforado · Aug 8, 2016, 4:33 PM

All the php-scripts spawned by the gui are constrained by the memory-limit set by suhosin. Which is 512MB.

That's what my perl-one-liner was for. I even increased the limit to 4GB for testing today …

paftdunk · Aug 8, 2016, 5:20 PM

@lucasrca:

How I solved my problem:…

Updated Firewall Maximum Table Entries: 4000000 -> 8000000

This ended up being the missing link for me. My default was 2M. When Perforado mentioned the count in /var/db/aliastables/ I checked mine: 4.4M in those lists alone. I bumped my max table entries in System / Advanced / Firewall & NAT from 2M to 10M and pfblockerng started working again.

RonpfS · Aug 8, 2016, 6:24 PM

@Perforado:

All the php-scripts spawned by the gui are constrained by the memory-limit set by suhosin. Which is 512MB.

That's what my perl-one-liner was for. I even increased the limit to 4GB for testing today …

For those wandering about suhosin, it is defined in /usr/local/etc/php.ini

; File generated from /etc/rc.php_ini_setup
output_buffering = "0"
expose_php = Off
implicit_flush = true
magic_quotes_gpc = Off
max_execution_time = 900
request_terminate_timeout = 900
max_input_time = 1800
max_input_vars = 5000
register_argc_argv = On
register_long_arrays = Off
variables_order = "GPCS"
file_uploads = On
upload_tmp_dir = /tmp
upload_max_filesize = 200M
post_max_size = 200M
html_errors = Off
zlib.output_compression = Off
zlib.output_compression_level = 1
include_path = ".:/etc/inc:/usr/local/www:/usr/local/captiveportal:/usr/local/pkg:/usr/local/www/classes:/usr/local/www/classes/Form"
display_startup_errors=on
display_errors=on
log_errors=on
error_log=/tmp/PHP_errors.log
extension_dir=/usr/local/lib/php/20131226/
date.timezone="America/New_York"
session.hash_bits_per_character = 5
session.hash_function = 1

; Extensions

; opcache Settings
opcache.enabled="1"
opcache.enable_cli="0"
opcache.memory_consumption="50"

[suhosin]
suhosin.get.max_array_index_length = 256
suhosin.get.max_vars = 5000
suhosin.get.max_value_length = 500000
suhosin.post.max_array_index_length = 256
suhosin.post.max_vars = 5000
suhosin.post.max_value_length = 500000
suhosin.request.max_array_index_length = 256
suhosin.request.max_vars = 5000
suhosin.request.max_value_length = 500000
suhosin.memory_limit = 536870912

Setting 'memory_limit' in the inc file is maxed by the suhosin.memory_limit.

The memory_limit is defined in /etc/inc/config.inc

// Set memory limit to 512M on amd64.
if ($ARCH == "amd64") {
	ini_set("memory_limit", "512M");
} else {
	ini_set("memory_limit", "128M");
}

So for those using many IPV6 GeoIP table on amd64, they probably have to change /usr/local/etc/php.ini, /etc/rc.php_ini_setup, pfblockerng.inc as well as the Firewall Maximum Table Entries

The php memory issues are not specific to pfBlockerNG, backup crashes when backup size is too big, the Diagnostics/Tables will crash when viewing huge table, etc.

richcj10gmail.com · Aug 9, 2016, 3:48 PM

I am having similar issues + NAT / routing was not working at all.

error:


					Crash report begins.  Anonymous machine information:

amd64
10.3-RELEASE-p5
FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016     root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense

Crash report details:

PHP Errors:
[09-Aug-2016 11:27:29 America/New_York] PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 72 bytes) in /usr/local/pkg/pfblockerng/pfblockerng.inc on line 3875
[09-Aug-2016 11:27:29 America/New_York] PHP Stack trace:
[09-Aug-2016 11:27:29 America/New_York] PHP   1\. {main}() /etc/rc.start_packages:0
[09-Aug-2016 11:27:29 America/New_York] PHP   2\. sync_package() /etc/rc.start_packages:90
[09-Aug-2016 11:27:29 America/New_York] PHP   3\. eval() /etc/inc/pkg-utils.inc:631
[09-Aug-2016 11:27:29 America/New_York] PHP   4\. sync_package_pfblockerng() /etc/inc/pkg-utils.inc(631) : eval()'d code:3
[09-Aug-2016 11:27:29 America/New_York] PHP   5\. file() /usr/local/pkg/pfblockerng/pfblockerng.inc:3875

I changed the fire wall rule # to 8000000 and added ini_set("memory_limit", "768M");
I still see the error above. But I at lest have routing back.