Problems 2.3.4_1 with Gui and pfctl
-
Hello ppl,
yesterday I had to reinstall from scratch our office pfsense since the old instal was not booting any more (init process wasn't starting).
So I deciced to make a fresh install and use the saved config to restore everything.
Since the server (Dell 2970 dual CPU, 8gb Ram, RAID 10 138GB hd 6 nics) was the very same machine everything went as planned and I had it back online in 40 minutes whith all services back online (HaProxy, NUT, PFBlockerNG, two OpenVPN tun +tap and zabbix agent).
This morning we had a brief internet outage and I was not able to log into the WebGui anymore.
After logging on the console using SSH I found :- /var/run full and the problem was the filter_reload_status file that ws 3.4 MB;
- a lot of PHP processes eating CPU and filling the logs with "pfsense kernel: pid 97435 (php-fpm), uid 0 inumber 21 on /var/run: filesystem full";
So I: - shortened the filter_reload_satus file by some thousands lines in order to free space;
- issued a "restart webconfigurator" form console ;
The PHP processes were still there with aloadvarage of 16.xx.
So I killed the PHP processes by hand and issued a "Restart PHP-FPM" from console.
Now I can login into the webconsole (extremely slow) but every time I try to change config I get a PHP out fo memory error: "PHP ERROR: Type: 1, File: /etc/inc/xmlparse.inc, Line: 297, Message: Allowed memory size of 536870912 bytes exhausted (tried to allocate 24652530 bytes) @ 2017-07-25 11:22:03"
If I use top on cli while trying to use the web interface I can see that from 1 to 4 pfctl processes are eating away all the CPU.
Anyone have a clue on what the hell is happening?
I would like not to reboot the FW since this is simply not supposed to happen and I would like to understand what the hell happened.
Thanks for your time and patience.
Alberto
*** UPDATE ***
After rebooting the Firewall, still same problem as yesterday: no init .
I waited 60 secs for init to run but since it did not come up I decided to reinstall again.
Can someone please help? This is getting frustrating since I have no clue on what's going on.
Thanks -
Do you have an unusually large number of rules, NAT entries, interfaces, or other entries somewhere in the GUI? Any packages installed that might have created them (like pfBlocker_ng)?
The filter_reload_status file is not usually that large, but it does get a couple entries per rule when it's generating the ruleset.
And if the XML Parser is running out of RAM, the config.xml must be massive.
-
Hello Jiimp,
thanks for your answer and taking interest.
Actually I have almost no rules but I did have installed pfBlocker_NG (now I removed it) .
I only added one ipv4 blocklist (Pedopoorn) from i-blocklist.
My backup config is 549 KB since it has some certs in it, but I would not consider it massive.
I also removed freeradius2 …
ThanksAlberto
-
If you can get it running again, try to copy off /var/run/filter_reload_status and post it somewhere so we can see what is inside it when it's that large.
Given the system specs you quoted I would not normally expect it to run the way you are seeing.
FreeRADIUS2 (or 3) wouldn't take up that much in terms of resources either.
-
Hi Jimp,
now the firewall is running fine, I plan on trying a reboot soon in order to verify if the init is screwed again.
Should the firewall perform a correct reboot I will:- upgrade it to 2.3.4_1;
- install pfblockerNG and add one IP blocklist per day.
I cannot post the filter_reload_status file since it's long gone…
Might it be a problem with the ip-blocklist? I left the type to "auto".
Thanks
Alberto
-
First update.
I upgraded to 2.3.4_1 and, till now, all is fine.
I'll keep the fw as it is for a few days and I'll activate again pfBlockerNG
ThanksAlberto
-
Hello,
reistalled pfBlockerNG and I'm having issues with aggregate process, it just sucks one core al 100% for MANY minutes (I killed it after 7:54 at 100%) with just ONE blocklist.
I tried downloading it in P2P format and CIDR format, no change: agregate process seemes totally stuck.
List is I-Blocklist level1.
I know it's big, but in CIDR format it's just 4.2 MB and pfBlockerNG should be able to handle it.
Any help?
ThanksAlberto Tarantino
