Disk usage /var/run/ over 100% - big php-fpm.core



  • Hi!

    After the upgrade to 2.3 I get disk usage in /var/run/ over 100%
    and when I look in the folder it´s php-fpm.core that is about 3.4 MB in size.
    After restarting the system it´s back to normal for a while before it happens again.

    It seems that the php can be configured differently, but I´m not sure how and where.
    I found a general solution for BSD saying that you could change the settings
    for pm from dynamic to on-demand.

    Anyway, I´m a bit stuck here, and yes I´ve searched for an answer, but nothing
    comes close to my problem.

    So I need a bit of encouragement from you kind souls, please
    give me some hints or tricks I can try.

    Thanks in advance!

    Sven



  • Hi,

    I'd appreciate if you could clarify which version you are running. On your post you mention 2.3 but this forum is for 2.3.1. Are you running 2.3 (original version from April 12th), 2.3_1 (2.3 Update 1) or 2.3.1 (Development snapshots) ?

    By the looks of what you are telling, this seems to be a serious problem, unfortunately with the little detail you given, no tips can be given to workaround/fix it.

    It would help to know the specs of the hardware on which you are running pfSense. Additionally, this php-fpm.core, may have insights on what is happening to cause its dumping.

    Hopefully one of the developers can reply, analyze this in detail for more clues, and shed some light on this.

    Thanks.

    Regards,
    Jorge M. Oliveira



  • Yes, of course.

    It´s the 2.3 update 1 version.
    Sorry if I´m in the wrong topic?

    It´s running on Intel(R) Core(TM)2 CPU 6700 @ 2.66GHz and there was no problem with the 2.2 version.

    I think I see a connection with Squid 0.4.16_2, that seems to be stopping frequently.

    I have sent an error report a few times in order for the developers to know what is going on.

    Do you need a log entry to get more data?

    I don´t like to not being able to solve this on my own, but now I really frustrated.
    I like this version, but of course I can downgrade, but thats a defeat…

    Sven



  • I encountered something similar upon having 2.3 running for a few days. At one point, Unbound would start going crazy servicing DNS requests, and WAN interface would drop off the network. All my internal systems were constantly hammering the firewall trying to renew DHCP as well.

    At one point it dawned on me that /var/run was full. At this point, rebooting seemed to fix it but not for long. I eventually moved /var/run to run in a RAM disk (configured under System > Advanced > Miscellaneous) by checking "Use RAM Disks" and then I proceeded to increase /var RAM Disk Size setting to 600MiB.

    At the time, I too had Squid proxy running with Clam Anti-virus as well (main reason for Squid) but I have since removed this package and everything has been stable since, and I have been monitoring that partition to see if it is growing, and it does not appear to be any longer.



  • Hi all

    Since a few days, I'm having the same problem with my system. The disk space of /var/run is running out of space …

    2.3.1-RELEASE-p1 (amd64)
    built on Wed May 25 14:53:06 CDT 2016
    FreeBSD 10.3-RELEASE-p3

    Disk usage ( /var/run ) 108% of 3.4MiB - ufs in RAM

    [2.3.1-RELEASE][admin@---]/var/run: ls -lh
    total 3468
    drwxrwxr-x  2 root  operator   512B May 30 21:11 .snap
    srw-rw-rw-  1 root  wheel        0B May 30 21:11 check_reload_status
    -rw-------  1 root  wheel        5B May 30 21:11 cron.pid
    -rw-------  1 root  wheel        3B May 30 21:11 devd.pid
    srw-rw-rw-  1 root  wheel        0B May 30 21:11 devd.pipe
    srw-rw-rw-  1 root  wheel        0B May 30 21:11 devd.seqpacket.pipe
    ----------  1 root  wheel        5B May 31 21:48 dhcpleases.pid
    -rw-r--r--  1 root  wheel        5B May 31 21:48 dnsmasq.pid
    -rw-r--r--  1 root  wheel        6B May 30 21:11 dpinger_GW_WAN_192.168.254.2_192.168.254.1.pid
    srw-rw-rw-  1 root  wheel        0B May 30 21:11 dpinger_GW_WAN_192.168.254.2_192.168.254.1.sock
    -rw-r--r--  1 root  wheel        6B May 30 21:11 expire_accounts.pid
    -rw-r--r--  1 root  wheel        4B May 31 22:15 filter_reload_status
    -rw-r--r--  1 root  wheel        6B May 30 21:11 filterdns.pid
    -rw-r--r--  1 root  wheel        6B May 30 21:11 filterlog.pid
    -r--r--r--  1 root  wheel      215B May 30 21:11 ld-elf.so.hints
    -r--r--r--  1 root  wheel      139B May 30 21:11 ld-elf32.so.hints
    srw-rw-rw-  1 root  wheel        0B May 30 21:11 log
    srw-------  1 root  wheel        0B May 30 21:11 logpriv
    -rw-r--r--  1 root  wheel        6B May 30 21:11 miniupnpd.pid
    -rw-r--r--  1 root  wheel        6B May 30 21:11 nginx-webConfigurator.pid
    -rw-r--r--  1 root  wheel        5B May 30 21:11 ntpd.pid
    -rw-r--r--  1 root  wheel        6B May 30 21:11 openvpn_server1.pid
    -rw-r--r--  1 root  wheel        6B May 30 21:11 openvpn_server2.pid
    -rw-------  1 root  wheel      3.3M May 30 21:53 php-fpm.core
    -rw-r--r--  1 root  wheel        3B May 30 21:11 php-fpm.pid
    srw-------  1 root  wheel        0B May 30 21:11 php-fpm.socket
    -rw-r--r--  1 root  wheel        6B May 30 21:11 ping_hosts.pid
    -rw-------  1 root  wheel        5B May 30 21:12 powerd.pid
    -rw-------  1 root  wheel        6B May 31 12:05 snort_igb040327.pid
    -rw-r--r--  1 root  wheel        6B May 31 21:48 sshd.pid
    -rw-------  1 root  wheel        4B May 30 21:12 syslog.pid
    -rw-r--r--  1 root  wheel        6B May 30 21:11 update_alias_url_data.pid
    -rw-r--r--  1 root  wheel        6B May 30 21:11 updaterrd.sh.pid
    -rw-r--r--  1 root  wheel        0B May 30 21:11 utmp
    -rw-r--r--  1 root  wheel      394B May 31 21:48 utx.active
    -rw-r--r--  1 root  wheel        6B May 30 21:11 xinetd.pid
    

    I already rebooted the system, but without the hoping result. Before 2.3.1-RELEASE-p1 I never had this problem.

    ![2016-05-31 22_12_41-fw01.lowet-willems.be - Status_ Dashboard.jpg](/public/imported_attachments/1/2016-05-31 22_12_41-fw01.lowet-willems.be - Status_ Dashboard.jpg)
    ![2016-05-31 22_12_41-fw01.lowet-willems.be - Status_ Dashboard.jpg_thumb](/public/imported_attachments/1/2016-05-31 22_12_41-fw01.lowet-willems.be - Status_ Dashboard.jpg_thumb)



  • I'm seeing the same thing on a fresh install built around 7PM CDT 6/20/2016. Not upgraded.
    Version 2.3.1-RELEASE-p5 (amd64) built on Thu Jun 16 12:53:15 CDT 2016

    Reboot seemed to fix the issue, but I got curious and came here to see if any one else was reporting this.
    Now I'm concerned it will come back.

    It's running on a Dell 9020 Corei5 with 8G RAM, 250G HDD, 4 port Intel NIC.  Pretty basic system.

    There are no addition packages installed on this box yet.  Configuring Interfaces, Captive Portal, DHCP Server, and Firewall Rules are the only things I have touched on this box.
    I noticed the high disk usage after I had finished making firewall rules for the 5 vlans I have.  I was dragging the rules around to get them in the order I needed them.
    When I returned to the main page disk usage /var/run was at 108%  This was at around 8PM CDT. So it happened in less than an hour for me, with no traffic on the box besides the one test PC I was using to configure everything.


  • Rebel Alliance Developer Netgate

    Add a tunable to set kern.corefile=/root/%N.core and see if the behavior improves. That won't stop whatever made PHP quit in the first place, but it will prevent it from overrunning /var/run/



  • Forgive my ignorance here, but I'm not exactly sure how you intended me to create this tunable.  If I make a new one should I name it kern.corefile with a value of /root/%N.core?  I assumed that is what you suggested, but when I tried it I was given an error for using unacceptable characters.  If you get a chance, could you please elaborate.
    Also just to make sure, will I need to reboot to have the tunable take effect?
    And just to give as much info as I can, I'm also seeing this same message in my log.  https://forum.pfsense.org/index.php?topic=111274.15


  • Rebel Alliance Developer Netgate

    We just made that the default value for the next update/release, so you won't have to worry about it much longer. I forgot the tunables page might not allow % there, normally what you said is what it should do.

    You can apply that change manually using the system patches package:
    https://github.com/pfsense/pfsense/commit/4a304fbf40eaa1a5a77ae1360f87914989c1b8ef

    You don't have to reboot, just run this from the shell or Diag > Command in the shell exec box:

    sysctl kern.corefile=/root/%N.core
    

  • Rebel Alliance Developer Netgate

    I also pushed a change to allow % and / in the sysctl values since they are valid for that oid (and presumably others)



  • I entered the following code a few weeks ago and I have since rebooted the SG 2440.

    [2.3.1-RELEASE][admin@pfSense.home]/root: sysctl kern.corefile=/root/%N.core
    kern.corefile: %N.core -> /root/%N.core
    [2.3.1-RELEASE][admin@pfSense.home]/root:
    

    I thought this had resolved, but this morning I noticed I have 106% of 3.4MiB under the Disk Usage (/var/run) section of the dashboard.

    My version is:

    2.3.1-RELEASE-p5 (amd64)
    built on Wed Jun 15 13:58:09 CDT 2016
    FreeBSD 10.3-RELEASE-p3

    The system is on the latest version.

    Any additional suggestions to help me with this problem?

    Thank you very much.


  • Rebel Alliance Developer Netgate

    That setting would not stick across reboots. If you rebooted since you added it, it would not be active.



  • So I reentered the code, but I notice my Disk Usage is still 106%.  Here is a copy of me entering the code followed by a look at the /var/run directory.    I'm not sure what I need to do to fix this.

    Thanks,
    Jerold

    
    [2.3.1-RELEASE][admin@pfSense.home]/var/run: sysctl kern.corefile=/root/%N.core
    kern.corefile: /root/%N.core -> /root/%N.core
    [2.3.1-RELEASE][admin@pfSense.home]/var/run: ls -l
    total 3412
    drwxrwxr-x  2 root   operator      512 Jul 17 06:24 .snap
    srw-rw-rw-  1 root   wheel           0 Jul 17 06:24 check_reload_status
    -rw-------  1 root   wheel           5 Jul 17 06:24 cron.pid
    -rw-------  1 root   wheel           3 Jul 17 06:24 devd.pid
    srw-rw-rw-  1 root   wheel           0 Jul 17 06:24 devd.pipe
    srw-rw-rw-  1 root   wheel           0 Jul 17 06:24 devd.seqpacket.pipe
    -rw-------  1 root   wheel           5 Jul 19 14:57 dhclient.igb0.pid
    -rw-r--r--  1 root   wheel           6 Jul 19 14:57 dhcp6c_igb0.pid
    ----------  1 root   wheel           6 Jul 19 14:57 dhcpleases.pid
    -rw-r--r--  1 root   wheel           6 Jul 23 14:56 dpinger_WAN_DHCP6_fe80::12dd:b1ff:fea5:220e%igb0_fe80::201:xxx:xxx:a846%igb0.pid
    srw-rw-rw-  1 root   wheel           0 Jul 23 14:56 dpinger_WAN_DHCP6_fe80::12dd:b1ff:fea5:220e%igb0_fe80::201:xxx:xxx:a846%igb0.sock
    -rw-r--r--  1 root   wheel           6 Jul 23 14:56 dpinger_WAN_DHCP_71.196.xxx.xxx_71.196.xxx.xxx.pid
    srw-rw-rw-  1 root   wheel           0 Jul 23 14:56 dpinger_WAN_DHCP_71.196.xxx.xxx_71.196.xxx.xxx.sock
    -rw-r--r--  1 root   wheel           6 Jul 17 06:24 expire_accounts.pid
    -rw-r--r--  1 root   wheel           4 Jul 23 20:23 filter_reload_status
    -rw-r--r--  1 root   wheel           5 Jul 17 06:24 filterlog.pid
    -r--r--r--  1 root   wheel         194 Jul 17 06:24 ld-elf.so.hints
    -r--r--r--  1 root   wheel         139 Jul 17 06:24 ld-elf32.so.hints
    srw-rw-rw-  1 root   wheel           0 Jul 17 06:24 log
    srw-------  1 root   wheel           0 Jul 17 06:24 logpriv
    -rw-r--r--  1 root   wheel           6 Jul 17 06:24 miniupnpd.pid
    -rw-r--r--  1 root   wheel           6 Jul 17 06:24 nginx-webConfigurator.pid
    -rw-r--r--  1 root   wheel           5 Jul 17 06:24 ntpd.pid
    -rw-r--r--  1 root   wheel           6 Jul 19 14:57 openvpn_server1.pid
    -rw-------  1 root   wheel     3342336 Jul 19 14:55 php-fpm.core
    -rw-r--r--  1 root   wheel           3 Jul 17 06:24 php-fpm.pid
    srw-------  1 root   wheel           0 Jul 17 06:24 php-fpm.socket
    -rw-r--r--  1 root   wheel           6 Jul 17 06:24 ping_hosts.pid
    -rw-------  1 root   wheel           5 Jul 17 06:25 powerd.pid
    drwxr-xr-x  2 squid  wheel         512 Jul 17 06:24 squid
    -rw-r--r--  1 root   wheel           5 Jul 17 06:24 sshd.pid
    -rw-------  1 root   wheel           5 Jul 17 06:25 syslog.pid
    -rw-r--r--  1 root   wheel           6 Jul 19 14:57 unbound.pid
    -rw-r--r--  1 root   wheel           6 Jul 17 06:24 update_alias_url_data.pid
    -rw-r--r--  1 root   wheel           6 Jul 19 14:57 updaterrd.sh.pid
    -rw-r--r--  1 root   wheel           0 Jul 17 06:24 utmp
    -rw-r--r--  1 root   wheel         394 Jul 23 06:06 utx.active
    -rw-------  1 root   wheel           5 Jul 17 06:25 watchdogd.pid
    -rw-r--r--  1 root   wheel           6 Jul 17 06:24 xinetd.pid
    


  • @jpvonhemel:

    So I reentered the code, but I notice my Disk Usage is still 106%. Here is a copy of me entering the code followed by a look at the /var/run directory. I'm not sure what I need to do to fix this.

    Run on shell:

    rm /var/run/php-fpm.core
    

Log in to reply