Hard Crashing - Out of Memory

stephenw10

@xxgbhxx said in Hard Crashing - Out of Memory:

536870912

It's exhausting the 512MB limit for php processes. It appears to be trying to allocate 375MB:

	printf("<a href=\"diag_dump_states.php?ruleid=%s\" data-toggle=\"popover\" data-trigger=\"hover focus\" title=\"%s\" ",
	    $rulesid, gettext("States details"));

So when you are mousing over something in the ruleset? Do you have huge aliases? A very large number of states?

Steve

xxGBHxx

I don't have a large number of states - I've not noticed it goes over 1000 total for the entire firewall. I'm not sure what you'd call huge aliases but within Pf none of the aliases are over 20 IP's. In PfBlockerNG I've got a number (17) of blocklists defined and the largest one when I do a refresh is about 80,000 IP's so tiny on the scale of lists.

I've just gone and cycled through the interfaces it seems to be my OpenVPN interface that's causing the issue. Even when I just click on it to browse to it that's when it locks up and it's now giving me a 504 error. Thing is I can't get into the GUI to see what rules are there that might be causing the problems. How can I get to the rules and disable/delete to see if that solves the problem?

G

xxGBHxx

Ok this time now that it's re-started I get

PHP errors

    PHP ERROR: Type: 1, File: /usr/local/www/firewall_rules.php, Line: 88, Message: Allowed memory size of 536870912 bytes exhausted (tried to allocate 375427072 bytes) @ 2019-03-13 18:43:03
    PHP ERROR: Type: 1, File: /usr/local/www/firewall_rules.php, Line: 491, Message: Allowed memory size of 536870912 bytes exhausted (tried to allocate 532692992 bytes) @ 2019-03-13 20:31:43

General

    Unresolvable source alias 'Internal_SUbnets' for rule 'OpenVPN wizard' @ 2019-03-13 23:00:09
    Unresolvable source alias 'Internal_SUbnets' for rule 'OpenVPN wizard' @ 2019-03-13 23:00:46

So it's clearly those last two lines. As I mentioned when I click on the tab it crashed the firewall so how do I get in there to delete the offending rule. The only range under "Internal_Subnets" is 172.17.10.0/24 so I don't understand why it's causing these problems.

Any ideas?

G

xxGBHxx

Hi,

@stephenw10 any ideas?

Thanks

Gary

Gertjan

@xxgbhxx said in Hard Crashing - Out of Memory:

The only range under "Internal_Subnets" is 172.17.10.0/24 so I don't understand why it's causing these problems.

This particular alias is just the drip that overflows the bucket.

A radical test to check if you are on the right track : tell PHP it can use more memory. Do this by changing the php.ini file.

Edit : etc/rc.php_ini_setup - line 181 - add " memory_limit = 256M "

....
; File generated from /etc/rc.php_ini_setup
memory_limit = 256M
...

and restart PHP using console option 16.

Now test.

xxGBHxx

@gertjan said in Hard Crashing - Out of Memory:

etc/rc.php_ini_setup

Hi thanks for that.

So I tried 256M and it had no effect. I increased it to 1024M restarting PHP with option 16 both times and it had no effect.

PHP ERROR: Type: 1, File: /usr/local/www/firewall_rules.php, Line: 708, Message: Allowed memory size of 536870912 bytes exhausted (tried to allocate 375398400 bytes) @ 2019-03-14 14:04:56

It doesn't seem to increase the allowed memory size at all either way.

Any other ideas?

G

Gertjan

Place this 'code' in a file called whatever.php into the GUI webroot (/usr/local/www/) :

<?php
phpinfo();
?>

and open the file like
http://pfsense.yourlan.tld/whatever.php

Check the several occurrences of 'memory' : I can't find 512M (!!) or 536870912 bytes ....
"memory_limit" is the value that should have been changed. With 6 Gb you could even make that value bigger.

Btw :

@xxgbhxx said in Hard Crashing - Out of Memory:

(only about 20 lists and none of them more than half a million IP's)

Serious ??
You really ask unbound, the DNS server, to restart with lists close to half a million of IP's ??
Woooow.

xxGBHxx

@gertjan

memory_limit = 1024M for both the local and master value

Not sure what your reference to DNS means. The lists are all IP's so wouldn't go anywhere near DNS.

Thanks again though

G

KOM

He might have meant pfBlocker, not Unbound.

Gertjan

@kom said in Hard Crashing - Out of Memory:

He might have meant pfBlocker, not Unbound.

Or both.
pfBlocker populate lists parsed by unbound.

(I guess it's time that I replay with pfBlocker - but don't do what to block, or who ...)

stephenw10

Or more likely that pfBlocker can add lists to both but DNSBL does not have to be used.

I believe this is the only line that would need changing:
https://github.com/pfsense/pfsense/blob/64d2dd619973fa986389de2b5db43b0043c5ddd6/src/etc/inc/config.inc#L43
But I have not tried it and you should only do so as a test. It should not be trying to allocate that ammount of RAM to open the page.

Steve

xxGBHxx

@kom

Ah

pfBlocker handles lists many millions of IP's in size - well according to the 100's of posts I've read. Many people use much larger IP lists than I do.

Either way it's not going anywhere near Unbound as the lists I use are all IP address so there is no resolution to be done unless I'm missing something obvious.

Either way the firewall is no where near ram, disk or CPU limits.

G

xxGBHxx

@stephenw10

Do I need to do a firewall re-start for it to pick up the change?

G

stephenw10

You would likely need to restart php and the webgui from the console menu at least. It might require a reboot.

Steve

xxGBHxx

@stephenw10

Increased that value to 2048 and restarted PHP/WEB. Also rebooted. Hasn't allowed me to get into that tab in the GUI as I still get a 504 error however it has now stopped giving me out of memory errors.

One step forwards but I sense no nearer!

Thanks again.

Any other ideas?

G

stephenw10

2048 is huge, is it exhausting the system RAM?

It probably isn't because I would expect you might hit this before that:
https://github.com/pfsense/pfsense/blob/RELENG_2_4_4/src/etc/rc.php_ini_setup#L271

Check /tmp/php_errors.txt

Check the main system log.

I assume restarting php and the wbegui from the menu does not allow you back in?

Steve

xxGBHxx

@stephenw10

I did it to ensure that exhausting memory wasn't the problem - appreciate it's way more than needed. I have plenty of system RAM (6GB) allocated to the VM so 2GB isn't anywhere near the limit for the system.

I'm certain it's something that's corrupted with the OpenVPN rule I have as it's referencing a table that doesn't exist (assuming it's case sensitive)

php_errors is a zero byte file.

Main system log looks to be corrupted I think but there are errors noted there from 3 days ago. Can't see anything more recent though.

Nothing I do allows me to click on the OpenVPN tab under "Rules". The rest of the GUI works fine and does not crash.

I will have a fiddle.

G

Gertjan

@xxgbhxx said in Hard Crashing - Out of Memory:

Main system log looks to be corrupted I think but there are errors noted there from 3 days ago. Can't see anything more recent though.

What are you doing to check this ?

xxGBHxx

@gertjan @stephenw10

I'm just catting the file like I would any other log file so

cat /var/log/system.log

I've renamed it to system.log.old and restarted. Log is now logging fine again.

The only error/issue I can see is

Mar 14 18:20:20 pfsense2 php-fpm[340]: /rc.filter_configure_sync: New alert found: Unresolvable source alias 'Internal_SUbnets' for rule 'OpenVPN  wizard'

Which is the same error I get if I click on the OpenVPN tab in the GUI (which then goes on to crash the FW)

I need some way of directly editing the rules for that tab to remove this reference (assuming that's where it is) so that the GUI stops crashing.

Apart from not being able to get into this part of the GUI the FW seems to be performing as I'd expect.

G

RonpfS

@xxgbhxx said in Hard Crashing - Out of Memory:

I'm just catting the file like I would any other log file so
cat /var/log/system.log

Those files are circular files, the have to be read with

 clog /var/log/system.log

They are managed with Status / System Logs / Settings. The drawback is that resetting will reset all log files.