Problems 2.3.4_1 with Gui and pfctl
yesterday I had to reinstall from scratch our office pfsense since the old instal was not booting any more (init process wasn't starting).
So I deciced to make a fresh install and use the saved config to restore everything.
Since the server (Dell 2970 dual CPU, 8gb Ram, RAID 10 138GB hd 6 nics) was the very same machine everything went as planned and I had it back online in 40 minutes whith all services back online (HaProxy, NUT, PFBlockerNG, two OpenVPN tun +tap and zabbix agent).
This morning we had a brief internet outage and I was not able to log into the WebGui anymore.
After logging on the console using SSH I found :
- /var/run full and the problem was the filter_reload_status file that ws 3.4 MB;
- a lot of PHP processes eating CPU and filling the logs with "pfsense kernel: pid 97435 (php-fpm), uid 0 inumber 21 on /var/run: filesystem full";
- shortened the filter_reload_satus file by some thousands lines in order to free space;
- issued a "restart webconfigurator" form console ;
The PHP processes were still there with aloadvarage of 16.xx.
So I killed the PHP processes by hand and issued a "Restart PHP-FPM" from console.
Now I can login into the webconsole (extremely slow) but every time I try to change config I get a PHP out fo memory error: "PHP ERROR: Type: 1, File: /etc/inc/xmlparse.inc, Line: 297, Message: Allowed memory size of 536870912 bytes exhausted (tried to allocate 24652530 bytes) @ 2017-07-25 11:22:03"
If I use top on cli while trying to use the web interface I can see that from 1 to 4 pfctl processes are eating away all the CPU.
Anyone have a clue on what the hell is happening?
I would like not to reboot the FW since this is simply not supposed to happen and I would like to understand what the hell happened.
Thanks for your time and patience.
*** UPDATE ***
After rebooting the Firewall, still same problem as yesterday: no init .
I waited 60 secs for init to run but since it did not come up I decided to reinstall again.
Can someone please help? This is getting frustrating since I have no clue on what's going on.
Do you have an unusually large number of rules, NAT entries, interfaces, or other entries somewhere in the GUI? Any packages installed that might have created them (like pfBlocker_ng)?
The filter_reload_status file is not usually that large, but it does get a couple entries per rule when it's generating the ruleset.
And if the XML Parser is running out of RAM, the config.xml must be massive.
thanks for your answer and taking interest.
Actually I have almost no rules but I did have installed pfBlocker_NG (now I removed it) .
I only added one ipv4 blocklist (Pedopoorn) from i-blocklist.
My backup config is 549 KB since it has some certs in it, but I would not consider it massive.
I also removed freeradius2 …
If you can get it running again, try to copy off /var/run/filter_reload_status and post it somewhere so we can see what is inside it when it's that large.
Given the system specs you quoted I would not normally expect it to run the way you are seeing.
FreeRADIUS2 (or 3) wouldn't take up that much in terms of resources either.
now the firewall is running fine, I plan on trying a reboot soon in order to verify if the init is screwed again.
Should the firewall perform a correct reboot I will:
- upgrade it to 2.3.4_1;
- install pfblockerNG and add one IP blocklist per day.
I cannot post the filter_reload_status file since it's long gone…
Might it be a problem with the ip-blocklist? I left the type to "auto".
I upgraded to 2.3.4_1 and, till now, all is fine.
I'll keep the fw as it is for a few days and I'll activate again pfBlockerNG
reistalled pfBlockerNG and I'm having issues with aggregate process, it just sucks one core al 100% for MANY minutes (I killed it after 7:54 at 100%) with just ONE blocklist.
I tried downloading it in P2P format and CIDR format, no change: agregate process seemes totally stuck.
List is I-Blocklist level1.
I know it's big, but in CIDR format it's just 4.2 MB and pfBlockerNG should be able to handle it.