Disk usage /var/run/ over 100% - big php-fpm.core
-
Hi all
Since a few days, I'm having the same problem with my system. The disk space of /var/run is running out of space …
2.3.1-RELEASE-p1 (amd64)
built on Wed May 25 14:53:06 CDT 2016
FreeBSD 10.3-RELEASE-p3Disk usage ( /var/run ) 108% of 3.4MiB - ufs in RAM
[2.3.1-RELEASE][admin@---]/var/run: ls -lh total 3468 drwxrwxr-x 2 root operator 512B May 30 21:11 .snap srw-rw-rw- 1 root wheel 0B May 30 21:11 check_reload_status -rw------- 1 root wheel 5B May 30 21:11 cron.pid -rw------- 1 root wheel 3B May 30 21:11 devd.pid srw-rw-rw- 1 root wheel 0B May 30 21:11 devd.pipe srw-rw-rw- 1 root wheel 0B May 30 21:11 devd.seqpacket.pipe ---------- 1 root wheel 5B May 31 21:48 dhcpleases.pid -rw-r--r-- 1 root wheel 5B May 31 21:48 dnsmasq.pid -rw-r--r-- 1 root wheel 6B May 30 21:11 dpinger_GW_WAN_192.168.254.2_192.168.254.1.pid srw-rw-rw- 1 root wheel 0B May 30 21:11 dpinger_GW_WAN_192.168.254.2_192.168.254.1.sock -rw-r--r-- 1 root wheel 6B May 30 21:11 expire_accounts.pid -rw-r--r-- 1 root wheel 4B May 31 22:15 filter_reload_status -rw-r--r-- 1 root wheel 6B May 30 21:11 filterdns.pid -rw-r--r-- 1 root wheel 6B May 30 21:11 filterlog.pid -r--r--r-- 1 root wheel 215B May 30 21:11 ld-elf.so.hints -r--r--r-- 1 root wheel 139B May 30 21:11 ld-elf32.so.hints srw-rw-rw- 1 root wheel 0B May 30 21:11 log srw------- 1 root wheel 0B May 30 21:11 logpriv -rw-r--r-- 1 root wheel 6B May 30 21:11 miniupnpd.pid -rw-r--r-- 1 root wheel 6B May 30 21:11 nginx-webConfigurator.pid -rw-r--r-- 1 root wheel 5B May 30 21:11 ntpd.pid -rw-r--r-- 1 root wheel 6B May 30 21:11 openvpn_server1.pid -rw-r--r-- 1 root wheel 6B May 30 21:11 openvpn_server2.pid -rw------- 1 root wheel 3.3M May 30 21:53 php-fpm.core -rw-r--r-- 1 root wheel 3B May 30 21:11 php-fpm.pid srw------- 1 root wheel 0B May 30 21:11 php-fpm.socket -rw-r--r-- 1 root wheel 6B May 30 21:11 ping_hosts.pid -rw------- 1 root wheel 5B May 30 21:12 powerd.pid -rw------- 1 root wheel 6B May 31 12:05 snort_igb040327.pid -rw-r--r-- 1 root wheel 6B May 31 21:48 sshd.pid -rw------- 1 root wheel 4B May 30 21:12 syslog.pid -rw-r--r-- 1 root wheel 6B May 30 21:11 update_alias_url_data.pid -rw-r--r-- 1 root wheel 6B May 30 21:11 updaterrd.sh.pid -rw-r--r-- 1 root wheel 0B May 30 21:11 utmp -rw-r--r-- 1 root wheel 394B May 31 21:48 utx.active -rw-r--r-- 1 root wheel 6B May 30 21:11 xinetd.pidI already rebooted the system, but without the hoping result. Before 2.3.1-RELEASE-p1 I never had this problem.
 -
I'm seeing the same thing on a fresh install built around 7PM CDT 6/20/2016. Not upgraded.
Version 2.3.1-RELEASE-p5 (amd64) built on Thu Jun 16 12:53:15 CDT 2016Reboot seemed to fix the issue, but I got curious and came here to see if any one else was reporting this.
Now I'm concerned it will come back.It's running on a Dell 9020 Corei5 with 8G RAM, 250G HDD, 4 port Intel NIC. Pretty basic system.
There are no addition packages installed on this box yet. Configuring Interfaces, Captive Portal, DHCP Server, and Firewall Rules are the only things I have touched on this box.
I noticed the high disk usage after I had finished making firewall rules for the 5 vlans I have. I was dragging the rules around to get them in the order I needed them.
When I returned to the main page disk usage /var/run was at 108% This was at around 8PM CDT. So it happened in less than an hour for me, with no traffic on the box besides the one test PC I was using to configure everything. -
Add a tunable to set kern.corefile=/root/%N.core and see if the behavior improves. That won't stop whatever made PHP quit in the first place, but it will prevent it from overrunning /var/run/
-
Forgive my ignorance here, but I'm not exactly sure how you intended me to create this tunable. If I make a new one should I name it kern.corefile with a value of /root/%N.core? I assumed that is what you suggested, but when I tried it I was given an error for using unacceptable characters. If you get a chance, could you please elaborate.
Also just to make sure, will I need to reboot to have the tunable take effect?
And just to give as much info as I can, I'm also seeing this same message in my log. https://forum.pfsense.org/index.php?topic=111274.15 -
We just made that the default value for the next update/release, so you won't have to worry about it much longer. I forgot the tunables page might not allow % there, normally what you said is what it should do.
You can apply that change manually using the system patches package:
https://github.com/pfsense/pfsense/commit/4a304fbf40eaa1a5a77ae1360f87914989c1b8efYou don't have to reboot, just run this from the shell or Diag > Command in the shell exec box:
sysctl kern.corefile=/root/%N.core -
I also pushed a change to allow % and / in the sysctl values since they are valid for that oid (and presumably others)
-
I entered the following code a few weeks ago and I have since rebooted the SG 2440.
[2.3.1-RELEASE][admin@pfSense.home]/root: sysctl kern.corefile=/root/%N.core kern.corefile: %N.core -> /root/%N.core [2.3.1-RELEASE][admin@pfSense.home]/root:I thought this had resolved, but this morning I noticed I have 106% of 3.4MiB under the Disk Usage (/var/run) section of the dashboard.
My version is:
2.3.1-RELEASE-p5 (amd64)
built on Wed Jun 15 13:58:09 CDT 2016
FreeBSD 10.3-RELEASE-p3The system is on the latest version.
Any additional suggestions to help me with this problem?
Thank you very much.
-
That setting would not stick across reboots. If you rebooted since you added it, it would not be active.
-
So I reentered the code, but I notice my Disk Usage is still 106%. Here is a copy of me entering the code followed by a look at the /var/run directory. I'm not sure what I need to do to fix this.
Thanks,
Jerold[2.3.1-RELEASE][admin@pfSense.home]/var/run: sysctl kern.corefile=/root/%N.core kern.corefile: /root/%N.core -> /root/%N.core [2.3.1-RELEASE][admin@pfSense.home]/var/run: ls -l total 3412 drwxrwxr-x 2 root operator 512 Jul 17 06:24 .snap srw-rw-rw- 1 root wheel 0 Jul 17 06:24 check_reload_status -rw------- 1 root wheel 5 Jul 17 06:24 cron.pid -rw------- 1 root wheel 3 Jul 17 06:24 devd.pid srw-rw-rw- 1 root wheel 0 Jul 17 06:24 devd.pipe srw-rw-rw- 1 root wheel 0 Jul 17 06:24 devd.seqpacket.pipe -rw------- 1 root wheel 5 Jul 19 14:57 dhclient.igb0.pid -rw-r--r-- 1 root wheel 6 Jul 19 14:57 dhcp6c_igb0.pid ---------- 1 root wheel 6 Jul 19 14:57 dhcpleases.pid -rw-r--r-- 1 root wheel 6 Jul 23 14:56 dpinger_WAN_DHCP6_fe80::12dd:b1ff:fea5:220e%igb0_fe80::201:xxx:xxx:a846%igb0.pid srw-rw-rw- 1 root wheel 0 Jul 23 14:56 dpinger_WAN_DHCP6_fe80::12dd:b1ff:fea5:220e%igb0_fe80::201:xxx:xxx:a846%igb0.sock -rw-r--r-- 1 root wheel 6 Jul 23 14:56 dpinger_WAN_DHCP_71.196.xxx.xxx_71.196.xxx.xxx.pid srw-rw-rw- 1 root wheel 0 Jul 23 14:56 dpinger_WAN_DHCP_71.196.xxx.xxx_71.196.xxx.xxx.sock -rw-r--r-- 1 root wheel 6 Jul 17 06:24 expire_accounts.pid -rw-r--r-- 1 root wheel 4 Jul 23 20:23 filter_reload_status -rw-r--r-- 1 root wheel 5 Jul 17 06:24 filterlog.pid -r--r--r-- 1 root wheel 194 Jul 17 06:24 ld-elf.so.hints -r--r--r-- 1 root wheel 139 Jul 17 06:24 ld-elf32.so.hints srw-rw-rw- 1 root wheel 0 Jul 17 06:24 log srw------- 1 root wheel 0 Jul 17 06:24 logpriv -rw-r--r-- 1 root wheel 6 Jul 17 06:24 miniupnpd.pid -rw-r--r-- 1 root wheel 6 Jul 17 06:24 nginx-webConfigurator.pid -rw-r--r-- 1 root wheel 5 Jul 17 06:24 ntpd.pid -rw-r--r-- 1 root wheel 6 Jul 19 14:57 openvpn_server1.pid -rw------- 1 root wheel 3342336 Jul 19 14:55 php-fpm.core -rw-r--r-- 1 root wheel 3 Jul 17 06:24 php-fpm.pid srw------- 1 root wheel 0 Jul 17 06:24 php-fpm.socket -rw-r--r-- 1 root wheel 6 Jul 17 06:24 ping_hosts.pid -rw------- 1 root wheel 5 Jul 17 06:25 powerd.pid drwxr-xr-x 2 squid wheel 512 Jul 17 06:24 squid -rw-r--r-- 1 root wheel 5 Jul 17 06:24 sshd.pid -rw------- 1 root wheel 5 Jul 17 06:25 syslog.pid -rw-r--r-- 1 root wheel 6 Jul 19 14:57 unbound.pid -rw-r--r-- 1 root wheel 6 Jul 17 06:24 update_alias_url_data.pid -rw-r--r-- 1 root wheel 6 Jul 19 14:57 updaterrd.sh.pid -rw-r--r-- 1 root wheel 0 Jul 17 06:24 utmp -rw-r--r-- 1 root wheel 394 Jul 23 06:06 utx.active -rw------- 1 root wheel 5 Jul 17 06:25 watchdogd.pid -rw-r--r-- 1 root wheel 6 Jul 17 06:24 xinetd.pid -
So I reentered the code, but I notice my Disk Usage is still 106%. Here is a copy of me entering the code followed by a look at the /var/run directory. I'm not sure what I need to do to fix this.
Run on shell:
rm /var/run/php-fpm.core