Hard Crashing - Out of Memory
-
2048 is huge, is it exhausting the system RAM?
It probably isn't because I would expect you might hit this before that:
https://github.com/pfsense/pfsense/blob/RELENG_2_4_4/src/etc/rc.php_ini_setup#L271Check
/tmp/php_errors.txtCheck the main system log.
I assume restarting php and the wbegui from the menu does not allow you back in?
Steve
-
I did it to ensure that exhausting memory wasn't the problem - appreciate it's way more than needed. I have plenty of system RAM (6GB) allocated to the VM so 2GB isn't anywhere near the limit for the system.
I'm certain it's something that's corrupted with the OpenVPN rule I have as it's referencing a table that doesn't exist (assuming it's case sensitive)
php_errors is a zero byte file.
Main system log looks to be corrupted I think but there are errors noted there from 3 days ago. Can't see anything more recent though.
Nothing I do allows me to click on the OpenVPN tab under "Rules". The rest of the GUI works fine and does not crash.
I will have a fiddle.
G
-
@xxgbhxx said in Hard Crashing - Out of Memory:
Main system log looks to be corrupted I think but there are errors noted there from 3 days ago. Can't see anything more recent though.
What are you doing to check this ?
-
I'm just catting the file like I would any other log file so
cat /var/log/system.logI've renamed it to system.log.old and restarted. Log is now logging fine again.
The only error/issue I can see is
Mar 14 18:20:20 pfsense2 php-fpm[340]: /rc.filter_configure_sync: New alert found: Unresolvable source alias 'Internal_SUbnets' for rule 'OpenVPN wizard'Which is the same error I get if I click on the OpenVPN tab in the GUI (which then goes on to crash the FW)
I need some way of directly editing the rules for that tab to remove this reference (assuming that's where it is) so that the GUI stops crashing.
Apart from not being able to get into this part of the GUI the FW seems to be performing as I'd expect.
G
-
@xxgbhxx said in Hard Crashing - Out of Memory:
I'm just catting the file like I would any other log file so
cat /var/log/system.logThose files are circular files, the have to be read with
clog /var/log/system.logThey are managed with Status / System Logs / Settings. The drawback is that resetting will reset all log files.
-
Check the ruleset in
/tmp/rules.debug.If you made a change that that you think is causing this you can roll back to a previous config from the console menu.
Steve
-
OK we have progress!
There are probably 500+ lines in there that say
pass in log quick on $OpenVPN inet from $Internal_Subnets to any tracker 1424387567 keep state label "USER_RULE: OpenVPN wizard"That is clearly the issue.
I have NO idea where they've come from - I've not run the OpenVPN wizard as far as I can remember. That said of course I might have and not remembered though why would it have just randomly created 100's of entries in that ruleset?
If I did run it I have absolutely no way of knowing when. Is there absolutely no way I can just remove the offending lines from the OpenVPN rules from the command line?
G
-
Ok, then is becomes more clear.
@xxgbhxx said in Hard Crashing - Out of Memory:
I've just gone and cycled through the interfaces it seems to be my OpenVPN interface that's causing the issue. Even when I just click on it to browse to it that's when it locks up and it's now giving me a 504 error.
There is something with the rules on that interface. The GUI throws 504.
When rules are generated, PHP runs out of memory.Export a backup of your config, and remove all firewall rules. Import back in again, rebuild your rules.
-
OK sorted.
Thank you very much.
When I exported the rules it created a 12MB 500,000+ line file. What was extraordinarily bizarre was that when I saved the file from the FW gui, the OS couldn't see the file so I couldn't open it to edit it. Even more bizarre was that if I went to save in the GUI, the FW save dialog could see the files but the OS and any other app couldn't.
In the end I opened the file "live" into Notepad++ but even then I couldn't actually edit the file. I had to save the file in Notepad++ and then re-open it to actually be able to edit it. First time I've ever had that.
Anyway, I deleted over 500,000 lines of config added by the OpenVPN Wizard and that took the file from 12MB and 550,000 lines to 61k and 2700 lines. Reloaded that back into the FW and with a bit more tidying up everything is working again and not crashing.
Thank you very much to everyone who chipped in. Was a "simple" fix I really should have come up with myself in the end but thanks for the help. If nothing else it's massively increased my knowledge.
G
-
Wow! Never seen anything even remotely like that. Weird.
I also noticed there are two aliases there
Internal_SUbnetsandInternal_Subnets. That seems suspicious, you might want to just check your config file now to see if both are still present.Steve
-
Hi,
I saw that too and checked for "SU" and "Su" and only "Internal_Subnets " exists so I have no idea where that came from either.
I clearly must have accidentally clicked on on the Wizard at some point in the last few days and not noticed leading it to get very confused.
It all seems good now though.
Thanks again for your help
G
