Disk usage /var/run/ over 100% - big php-fpm.core
-
I encountered something similar upon having 2.3 running for a few days. At one point, Unbound would start going crazy servicing DNS requests, and WAN interface would drop off the network. All my internal systems were constantly hammering the firewall trying to renew DHCP as well.
At one point it dawned on me that /var/run was full. At this point, rebooting seemed to fix it but not for long. I eventually moved /var/run to run in a RAM disk (configured under System > Advanced > Miscellaneous) by checking "Use RAM Disks" and then I proceeded to increase /var RAM Disk Size setting to 600MiB.
At the time, I too had Squid proxy running with Clam Anti-virus as well (main reason for Squid) but I have since removed this package and everything has been stable since, and I have been monitoring that partition to see if it is growing, and it does not appear to be any longer.
-
Hi all
Since a few days, I'm having the same problem with my system. The disk space of /var/run is running out of space …
2.3.1-RELEASE-p1 (amd64)
built on Wed May 25 14:53:06 CDT 2016
FreeBSD 10.3-RELEASE-p3Disk usage ( /var/run ) 108% of 3.4MiB - ufs in RAM
[2.3.1-RELEASE][admin@---]/var/run: ls -lh total 3468 drwxrwxr-x 2 root operator 512B May 30 21:11 .snap srw-rw-rw- 1 root wheel 0B May 30 21:11 check_reload_status -rw------- 1 root wheel 5B May 30 21:11 cron.pid -rw------- 1 root wheel 3B May 30 21:11 devd.pid srw-rw-rw- 1 root wheel 0B May 30 21:11 devd.pipe srw-rw-rw- 1 root wheel 0B May 30 21:11 devd.seqpacket.pipe ---------- 1 root wheel 5B May 31 21:48 dhcpleases.pid -rw-r--r-- 1 root wheel 5B May 31 21:48 dnsmasq.pid -rw-r--r-- 1 root wheel 6B May 30 21:11 dpinger_GW_WAN_192.168.254.2_192.168.254.1.pid srw-rw-rw- 1 root wheel 0B May 30 21:11 dpinger_GW_WAN_192.168.254.2_192.168.254.1.sock -rw-r--r-- 1 root wheel 6B May 30 21:11 expire_accounts.pid -rw-r--r-- 1 root wheel 4B May 31 22:15 filter_reload_status -rw-r--r-- 1 root wheel 6B May 30 21:11 filterdns.pid -rw-r--r-- 1 root wheel 6B May 30 21:11 filterlog.pid -r--r--r-- 1 root wheel 215B May 30 21:11 ld-elf.so.hints -r--r--r-- 1 root wheel 139B May 30 21:11 ld-elf32.so.hints srw-rw-rw- 1 root wheel 0B May 30 21:11 log srw------- 1 root wheel 0B May 30 21:11 logpriv -rw-r--r-- 1 root wheel 6B May 30 21:11 miniupnpd.pid -rw-r--r-- 1 root wheel 6B May 30 21:11 nginx-webConfigurator.pid -rw-r--r-- 1 root wheel 5B May 30 21:11 ntpd.pid -rw-r--r-- 1 root wheel 6B May 30 21:11 openvpn_server1.pid -rw-r--r-- 1 root wheel 6B May 30 21:11 openvpn_server2.pid -rw------- 1 root wheel 3.3M May 30 21:53 php-fpm.core -rw-r--r-- 1 root wheel 3B May 30 21:11 php-fpm.pid srw------- 1 root wheel 0B May 30 21:11 php-fpm.socket -rw-r--r-- 1 root wheel 6B May 30 21:11 ping_hosts.pid -rw------- 1 root wheel 5B May 30 21:12 powerd.pid -rw------- 1 root wheel 6B May 31 12:05 snort_igb040327.pid -rw-r--r-- 1 root wheel 6B May 31 21:48 sshd.pid -rw------- 1 root wheel 4B May 30 21:12 syslog.pid -rw-r--r-- 1 root wheel 6B May 30 21:11 update_alias_url_data.pid -rw-r--r-- 1 root wheel 6B May 30 21:11 updaterrd.sh.pid -rw-r--r-- 1 root wheel 0B May 30 21:11 utmp -rw-r--r-- 1 root wheel 394B May 31 21:48 utx.active -rw-r--r-- 1 root wheel 6B May 30 21:11 xinetd.pidI already rebooted the system, but without the hoping result. Before 2.3.1-RELEASE-p1 I never had this problem.
 -
I'm seeing the same thing on a fresh install built around 7PM CDT 6/20/2016. Not upgraded.
Version 2.3.1-RELEASE-p5 (amd64) built on Thu Jun 16 12:53:15 CDT 2016Reboot seemed to fix the issue, but I got curious and came here to see if any one else was reporting this.
Now I'm concerned it will come back.It's running on a Dell 9020 Corei5 with 8G RAM, 250G HDD, 4 port Intel NIC. Pretty basic system.
There are no addition packages installed on this box yet. Configuring Interfaces, Captive Portal, DHCP Server, and Firewall Rules are the only things I have touched on this box.
I noticed the high disk usage after I had finished making firewall rules for the 5 vlans I have. I was dragging the rules around to get them in the order I needed them.
When I returned to the main page disk usage /var/run was at 108% This was at around 8PM CDT. So it happened in less than an hour for me, with no traffic on the box besides the one test PC I was using to configure everything. -
Add a tunable to set kern.corefile=/root/%N.core and see if the behavior improves. That won't stop whatever made PHP quit in the first place, but it will prevent it from overrunning /var/run/
-
Forgive my ignorance here, but I'm not exactly sure how you intended me to create this tunable. If I make a new one should I name it kern.corefile with a value of /root/%N.core? I assumed that is what you suggested, but when I tried it I was given an error for using unacceptable characters. If you get a chance, could you please elaborate.
Also just to make sure, will I need to reboot to have the tunable take effect?
And just to give as much info as I can, I'm also seeing this same message in my log. https://forum.pfsense.org/index.php?topic=111274.15 -
We just made that the default value for the next update/release, so you won't have to worry about it much longer. I forgot the tunables page might not allow % there, normally what you said is what it should do.
You can apply that change manually using the system patches package:
https://github.com/pfsense/pfsense/commit/4a304fbf40eaa1a5a77ae1360f87914989c1b8efYou don't have to reboot, just run this from the shell or Diag > Command in the shell exec box:
sysctl kern.corefile=/root/%N.core -
I also pushed a change to allow % and / in the sysctl values since they are valid for that oid (and presumably others)
-
I entered the following code a few weeks ago and I have since rebooted the SG 2440.
[2.3.1-RELEASE][admin@pfSense.home]/root: sysctl kern.corefile=/root/%N.core kern.corefile: %N.core -> /root/%N.core [2.3.1-RELEASE][admin@pfSense.home]/root:I thought this had resolved, but this morning I noticed I have 106% of 3.4MiB under the Disk Usage (/var/run) section of the dashboard.
My version is:
2.3.1-RELEASE-p5 (amd64)
built on Wed Jun 15 13:58:09 CDT 2016
FreeBSD 10.3-RELEASE-p3The system is on the latest version.
Any additional suggestions to help me with this problem?
Thank you very much.
-
That setting would not stick across reboots. If you rebooted since you added it, it would not be active.
-
So I reentered the code, but I notice my Disk Usage is still 106%. Here is a copy of me entering the code followed by a look at the /var/run directory. I'm not sure what I need to do to fix this.
Thanks,
Jerold[2.3.1-RELEASE][admin@pfSense.home]/var/run: sysctl kern.corefile=/root/%N.core kern.corefile: /root/%N.core -> /root/%N.core [2.3.1-RELEASE][admin@pfSense.home]/var/run: ls -l total 3412 drwxrwxr-x 2 root operator 512 Jul 17 06:24 .snap srw-rw-rw- 1 root wheel 0 Jul 17 06:24 check_reload_status -rw------- 1 root wheel 5 Jul 17 06:24 cron.pid -rw------- 1 root wheel 3 Jul 17 06:24 devd.pid srw-rw-rw- 1 root wheel 0 Jul 17 06:24 devd.pipe srw-rw-rw- 1 root wheel 0 Jul 17 06:24 devd.seqpacket.pipe -rw------- 1 root wheel 5 Jul 19 14:57 dhclient.igb0.pid -rw-r--r-- 1 root wheel 6 Jul 19 14:57 dhcp6c_igb0.pid ---------- 1 root wheel 6 Jul 19 14:57 dhcpleases.pid -rw-r--r-- 1 root wheel 6 Jul 23 14:56 dpinger_WAN_DHCP6_fe80::12dd:b1ff:fea5:220e%igb0_fe80::201:xxx:xxx:a846%igb0.pid srw-rw-rw- 1 root wheel 0 Jul 23 14:56 dpinger_WAN_DHCP6_fe80::12dd:b1ff:fea5:220e%igb0_fe80::201:xxx:xxx:a846%igb0.sock -rw-r--r-- 1 root wheel 6 Jul 23 14:56 dpinger_WAN_DHCP_71.196.xxx.xxx_71.196.xxx.xxx.pid srw-rw-rw- 1 root wheel 0 Jul 23 14:56 dpinger_WAN_DHCP_71.196.xxx.xxx_71.196.xxx.xxx.sock -rw-r--r-- 1 root wheel 6 Jul 17 06:24 expire_accounts.pid -rw-r--r-- 1 root wheel 4 Jul 23 20:23 filter_reload_status -rw-r--r-- 1 root wheel 5 Jul 17 06:24 filterlog.pid -r--r--r-- 1 root wheel 194 Jul 17 06:24 ld-elf.so.hints -r--r--r-- 1 root wheel 139 Jul 17 06:24 ld-elf32.so.hints srw-rw-rw- 1 root wheel 0 Jul 17 06:24 log srw------- 1 root wheel 0 Jul 17 06:24 logpriv -rw-r--r-- 1 root wheel 6 Jul 17 06:24 miniupnpd.pid -rw-r--r-- 1 root wheel 6 Jul 17 06:24 nginx-webConfigurator.pid -rw-r--r-- 1 root wheel 5 Jul 17 06:24 ntpd.pid -rw-r--r-- 1 root wheel 6 Jul 19 14:57 openvpn_server1.pid -rw------- 1 root wheel 3342336 Jul 19 14:55 php-fpm.core -rw-r--r-- 1 root wheel 3 Jul 17 06:24 php-fpm.pid srw------- 1 root wheel 0 Jul 17 06:24 php-fpm.socket -rw-r--r-- 1 root wheel 6 Jul 17 06:24 ping_hosts.pid -rw------- 1 root wheel 5 Jul 17 06:25 powerd.pid drwxr-xr-x 2 squid wheel 512 Jul 17 06:24 squid -rw-r--r-- 1 root wheel 5 Jul 17 06:24 sshd.pid -rw------- 1 root wheel 5 Jul 17 06:25 syslog.pid -rw-r--r-- 1 root wheel 6 Jul 19 14:57 unbound.pid -rw-r--r-- 1 root wheel 6 Jul 17 06:24 update_alias_url_data.pid -rw-r--r-- 1 root wheel 6 Jul 19 14:57 updaterrd.sh.pid -rw-r--r-- 1 root wheel 0 Jul 17 06:24 utmp -rw-r--r-- 1 root wheel 394 Jul 23 06:06 utx.active -rw------- 1 root wheel 5 Jul 17 06:25 watchdogd.pid -rw-r--r-- 1 root wheel 6 Jul 17 06:24 xinetd.pid -
So I reentered the code, but I notice my Disk Usage is still 106%. Here is a copy of me entering the code followed by a look at the /var/run directory. I'm not sure what I need to do to fix this.
Run on shell:
rm /var/run/php-fpm.core