Warning: fopen(/tmp/config.lock): failed to open stream



  • I have just tried installation on a new router,
    things are fine and the links are up.

    But after few days I cannot enter the web config page and getting below error.
    I typed the default password and it then give the below error and it can't login.

    Warning: fopen(/tmp/config.lock): failed to open stream: Device not configured in /etc/inc/util.inc on line 123 Warning: flock() expects parameter 1 to be resource, null given in /etc/inc/util.inc on line 134 Warning: fclose(): supplied argument is not a valid stream resource in /etc/inc/util.inc on line 135 Warning: session_start(): open(/var/tmp//sess_fd59264b4e5cf3b6ae1121f1ccf4762a, O_RDWR) failed: No space left on device (28) in /etc/inc/auth.inc on line 1254

    We are using Super Micro Server Model 502-2 P/N 5015A-EHF-D525 with 32 GB crucial hdd.

    Please suggest


  • Netgate Administrator

    What are you booting from?

    Steve



  • Hi Steve, basically we are running pfSense 2.1.5-RELEASE (i386) on supermicro box and booting from a hard drive.

    When I have logged in via console on pfsense box then found below error, can you please suggest how to resolve this issue without reinstalling the firewall.


    Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.ccoOV4, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
    Fatal error: PHP Startup: apc_fcntl_lock failed: in Unknown on line 0
    Fatal error: PHP Startup: apc_fcntl_unlock failed: in Unknown on line 0
    Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.QLoRQD, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
    Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.aWv6Ra, O_RDWR|O_CREA
    Fatal error: PHP Startup: apc_fcntl_lock failed: in Unknown on line 0
    Fatal error: PHP Startup: apc_fcntl_unlock failed: in Unknown on line 0
    Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.sgK0ch, O_RDWR|O_CREA
    Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.BEhNIf, O_RDWR|O_CREA
    Fatal error: Unknown: apc_fcntl_rdlock failed: in Unknown on line 0

    1. Logout (SSH only)                  8 ) Shell
    2. Assign Interfaces                  9) pfTop
    3. Set interface(s) IP address      10) Filter Logs
    4. Reset webConfigurator password    11) Restart webConfigurator
    5. Reset to factory defaults        12) pfSense Developer Shell
    6. Reboot system                    13) Upgrade from console
    7. Halt system                      14) Disable Secure Shell (sshd)
    8. Ping host                        15) Restore recent configuration

    Enter an option: 8

    /etc/rc.initial: /bin/tcsh: Device not configured
    /: write failed, filesystem is full
    Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.PA8GQF, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
    Fatal error: PHP Startup: apc_fcntl_lock failed: in Unknown on line 0
    Fatal error: PHP Startup: apc_fcntl_unlock failed: in Unknown on line 0
    Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.LMIIbh, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
    Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.I7p46p, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
    Fatal error: PHP Startup: apc_fcntl_lock failed: in Unknown on line 0
    Fatal error: PHP Startup: apc_fcntl_unlock failed: in Unknown on line 0
    Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.UPJxwV, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
    Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.tZet8w, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
    Fatal error: Unknown: apc_fcntl_rdlock failed: in Unknown on line 0


  • Netgate Administrator

    /: write failed, filesystem is full
    

    This looks like a boot device failure of some sort. Re-reading your first post I see you're booting from a hard drive. How did you install to the HD?

    Have you moved /tmp to a ram drive? (an advanced  gui option)

    If it were me I would re-install faced with errors like that and run some low level disc check utility. It's not hard to re-install and restore the config.

    Steve



  • i am getting the same error i left the office with everthing fine but in the morning i get the same error image is attached. according to my experiance it is HD error now i will try with new harddisk than update you soon




  • Hi Steve, Thanks for your reply.

    We have fresh installed pfsense with cd and this is a brand new hardware with new HDD, also i would like to update you that we have 8 locations where we installed pfsense firewall and this is 3rd time on 3rd location where I get same error. Its seems some bug which is creating this issue.

    If I reboot the firewall with error, its got crash and then it needs to reinstall again.

    Please let me know if there is any solution for this issue.

    Also please find the attached Screen shot or error.






  • i tried with other HD 500 gb sata but still get the same error next morning.
    now i install old 40gb ide let ce what happen now.




  • again failed after 30 minute of online.



  • anybody is there for help please

    every morning i amgetting the pfsense box down. i have tried 5 different harddisk assuming that it is hd error.



  • Netgate Administrator

    Hmm. It could be complete coincidence but there seem to be an unusually large number of reports of similar failures on the forum at the moment.

    If you've tried multiple hard drives perhaps it's a cabling or disk controller issue?

    Steve



  • The "no inodes free" message in your console screenshot makes me think something is creating many many little files, probably in /tmp or /var somewhere.
    Use console menu option 8 to get a command prompt. Then look around /var/ and /tmp and see if some folder has many many little files. Report back what they are and we can then think about why it is happening.


  • Netgate Administrator

    Ah, well spotted I missed that. That's on Tarun's screenshot?

    Steve



  • @stephenw10:

    Ah, well spotted I missed that. That's on Tarun's screenshot?

    Yes, Tarun's screenshot has a "create symlink failed - no inodes free"

    I guess something is creating many many little files or?


  • Netgate Administrator

    Yep. I don't see any inode warnings from ishtiaqaj though, here or anywhere else.
    The fstat command referenced here looks appropriate for this:

    [2.2-RC][root@xtm5.localdomain]/root: fstat | awk '\!/CMD/{print $2} END{printf("* Total files open: %d", NR)}' | sort | uniq -c | sort -n
       1 * Total files open: 511
       2 kernel
       2 md0
       2 md1
       3 init
       4 getty
       6 rrdtool
       7 apinger
       7 awk
       7 cron
       7 sort
       7 uniq
       8 fstat
       8 login
       8 sshlockout_pf
      12 sleep
      13 powerd
      14 dnsmasq
      15 LCDd
      15 filterlog
      15 lcdproc
      15 lighttpd
      16 check_reload_status
      16 devd
      16 tcsh
      17 inetd
      20 dhcpd
      22 sshd
      23 ntpd
      36 dhclient
      36 minicron
      42 php-fpm
      42 syslogd
      47 sh
    
    

    Nothing too extreme on my test 2.2 box.

    Steve


  • Netgate

    what does df -i show?

    ![Screen Shot 2014-12-28 at 12.09.15 PM.png](/public/imported_attachments/1/Screen Shot 2014-12-28 at 12.09.15 PM.png)
    ![Screen Shot 2014-12-28 at 12.09.15 PM.png_thumb](/public/imported_attachments/1/Screen Shot 2014-12-28 at 12.09.15 PM.png_thumb)



  • Hi phil,

    Thanks for your reply, actually when I get this error, i tried to use console menu option 8 to get a command prompt but its unable me to get into the command prompt. Its giving below error.

    1. Logout (SSH only)                  8 ) Shell
    2. Assign Interfaces                  9) pfTop
    3. Set interface(s) IP address      10) Filter Logs
    4. Reset webConfigurator password    11) Restart webConfigurator
    5. Reset to factory defaults        12) pfSense Developer Shell
    6. Reboot system                    13) Upgrade from console
    7. Halt system                      14) Disable Secure Shell (sshd)
    8. Ping host                        15) Restore recent configuration

    Enter an option: 8

    /etc/rc.initial: /bin/tcsh: Device not configured
    /: write failed, filesystem is full
    Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.PA8GQF, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
    Fatal error: PHP Startup: apc_fcntl_lock failed: in Unknown on line 0
    Fatal error: PHP Startup: apc_fcntl_unlock failed: in Unknown on line 0
    Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.LMIIbh, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
    Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.I7p46p, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
    Fatal error: PHP Startup: apc_fcntl_lock failed: in Unknown on line 0
    Fatal error: PHP Startup: apc_fcntl_unlock failed: in Unknown on line 0
    Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.UPJxwV, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
    Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.tZet8w, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
    Fatal error: Unknown: apc_fcntl_rdlock failed: in Unknown on line 0



  • As per the error shown on GUI page its clear that its some where a issue with hdd full in tmp and var directory but this issue keep coming up again and I have also read on forum some guys reported the same issue even they used 500 GB hdd. Its seems some bug which is creating this issue.

    When I tried to get command prompt access it given me php error as shown in my post. If I tried to restart the pfsense, its got crash and then I need to fresh install. One more thing if we tried to install pfSense on same hdd it never install on it and say I/O error, even the hdd is brand new. Please find the screen shot.

    This same thing was happen on 3 of our pfsense, which have brand new hardware with brand new hdd.



  • Netgate

    If it's brand new hardware, why are you running i386?  https://forum.pfsense.org/index.php?topic=84679.0

    What image are you installing?  How?



  • We are installing i386 from CD image downloaded from https://www.pfsense.org/download/mirror.php?section=downloads.

    Its Supermicro Model 502-2 P/N 5015A-EHF-D525 with 32 GB crucial hdd.

    I hope you have read all error posted by me from starting.

    Error Post 1

    after few days I cannot enter the web config page and getting below error.
    I typed the default password and it then give the below error and it can't login.

    Warning: fopen(/tmp/config.lock): failed to open stream: Device not configured in /etc/inc/util.inc on line 123 Warning: flock() expects parameter 1 to be resource, null given in /etc/inc/util.inc on line 134 Warning: fclose(): supplied argument is not a valid stream resource in /etc/inc/util.inc on line 135 Warning: session_start(): open(/var/tmp//sess_fd59264b4e5cf3b6ae1121f1ccf4762a, O_RDWR) failed: No space left on device (28) in /etc/inc/auth.inc on line 1254

    We are using Super Micro Server with 32 GB crucial hdd.

    Error Post 2

    Tried to use console menu option 8 to get a command prompt but its unable me to get into the command prompt. Its giving below error.

    1. Logout (SSH only)                  8 ) Shell
    2. Assign Interfaces                  9) pfTop
    3. Set interface(s) IP address      10) Filter Logs
    4. Reset webConfigurator password    11) Restart webConfigurator
    5. Reset to factory defaults        12) pfSense Developer Shell
    6. Reboot system                    13) Upgrade from console
    7. Halt system                      14) Disable Secure Shell (sshd)
    8. Ping host                        15) Restore recent configuration

    Enter an option: 8

    /etc/rc.initial: /bin/tcsh: Device not configured
    /: write failed, filesystem is full
    Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.PA8GQF, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
    Fatal error: PHP Startup: apc_fcntl_lock failed: in Unknown on line 0
    Fatal error: PHP Startup: apc_fcntl_unlock failed: in Unknown on line 0
    Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.LMIIbh, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
    Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.I7p46p, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
    Fatal error: PHP Startup: apc_fcntl_lock failed: in Unknown on line 0
    Fatal error: PHP Startup: apc_fcntl_unlock failed: in Unknown on line 0
    Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.UPJxwV, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
    Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.tZet8w, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
    Fatal error: Unknown: apc_fcntl_rdlock failed: in Unknown on line 0

    Error Post 3

    As per the error shown on GUI page its clear that its some where a issue with hdd full in tmp and var directory but this issue keep coming up again and I have also read on forum some guys reported the same issue even they used 500 GB hdd. Its seems some bug which is creating this issue.

    When I tried to get command prompt access it given me php error as shown in my post. If I tried to restart the pfsense, its got crash and then I need to fresh install. One more thing if we tried to install pfSense on same hdd it never install on it and say I/O error, even the hdd is brand new.

    When we format hdd then its able to reinstalled the pfsense.


  • Netgate

    Why i386?  Why not amd64?  All those I/O errors make me think the filesystem full messages are sort of a false indicator.

    Looks like failing disk from here.


  • Netgate

    All modern intel processors are amd64, bro.  AMD won that one.  Install the 64-bit version. https://forum.pfsense.org/index.php?topic=84679.0

    I don't think the file systems are full.  I think something on a failing disk or a weird install case is reporting a disk full.



  • The console screen picture shows disk errors. So I expect the disk is not really full, the disk writes or reads are failing.
    If it was running nicely for some time and the problem has only now started, then I guess a failing disk.
    If the problem has always been there, then some issue between the hardware and the FreeBSD disk drivers.


  • Netgate Administrator

    The picture here is getting muddied by multiple reports across different systems etc.
    @tarun:

    I have just concerned with one of my colleague, he has installed AMD64 but faced the same issue.

    Is it actually exactly the same symptoms or just similar? Did he try to replicate your setup? Same hardware?

    Tarun, you posted one screenshot showing an out of inodes warning. That is potentially a solvable problem. If something in your setup is creating hundreds of files it should be possible to see what is doing it and find out why.

    The other errors shown here and on other threads usually point to a failing harddrive.  ishtiaqaj has tried multiple hard drives with the same result which then leads me to suspect a bad cable or disk controller or some badly supported hardware. Tarun, have you tried more than one hard drive?

    Although the problems here look similar anything that prevents writing to the drive could throw these same errors. The fact that the vast majority of users are running the same version as you and not seeing any errors implies that your install is different somehow. Which Supermicro box are you using? They are well tested though it would be surprising to find something unsupported.

    Steve



  • Yes, we have tried different hdd, but the issue same. We have face this issue on 3 of our location.

    We are using Supermicro 502-2 1U Server 5015A-EHF-D525 w/Intel Atom D525 CPU /4GB RAM/ 32 GB Crucial hdd.

    I have just concerned with one of my colleague, he has installed AMD64 but faced the same issue.

    Warning: fopen(/tmp/config.lock): failed to open stream: Device not configured in /etc/inc/util.inc on line 123 Warning: flock() expects parameter 1 to be resource, null given in /etc/inc/util.inc on line 134 Warning: fclose(): supplied argument is not a valid stream resource in /etc/inc/util.inc on line 135 Warning: session_start(): open(/var/tmp//sess_fd59264b4e5cf3b6ae1121f1ccf4762a, O_RDWR) failed: No space left on device (28) in /etc/inc/auth.inc on line 1254


  • Netgate Administrator

    Ok,  that's fairly standard hardware, nothing unusual there. Is it identical on all three sites? Are those Crucial SSDs or really old/small hard drives? How old are they? Checking the SMART stats on them would probably be good idea.
    If it's none of those then i'd look at something unusual in your config that's common across the sites.

    Steve


  • Rebel Alliance Developer Netgate

    @tarun:

    I have just concerned with one of my colleague, he has installed AMD64 but faced the same issue.

    Warning: fopen(/tmp/config.lock): failed to open stream: Device not configured in /etc/inc/util.inc on line 123 Warning: flock() expects parameter 1 to be resource, null given in /etc/inc/util.inc on line 134 Warning: fclose(): supplied argument is not a valid stream resource in /etc/inc/util.inc on line 135 Warning: session_start(): open(/var/tmp//sess_fd59264b4e5cf3b6ae1121f1ccf4762a, O_RDWR) failed: No space left on device (28) in /etc/inc/auth.inc on line 1254

    More likely than not, the disk is dead. If you have multiple units doing it, then it wouldn't be the first time someone got a bad batch of SSDs. We've seen similar bad batches with both Kingston and Crucial in the past.

    It comes down to one of three possibilities:
    1. The drive really is full, or at least /var and /tmp both are (unlikely)
    2. The disk is dead, in which case it's a coin toss as to whether the system will come back after a reboot
    3. The disk has stopped responding to the operating system due to a controller issue, which may be OK after a reboot. Sometimes there are firmware updates for SSDs to help with such problems.

    More info here:
    https://doc.pfsense.org/index.php/Filesystem_Full_/_Out_of_Inode_Errors



  • Hi Steve,

    Yes it is identical on all three sites,

    These Crucial SSDs are brand new hard drives.

    Thanks


  • Netgate Administrator

    Well, as Jim said, they could potentially be from a bad batch of drives if you bought them all together or maybe they have bad firmware that can be updated.  :-\ Have you tried using any other drives?

    Steve



  • the same probelm i was facing and it is with SATA harddisk now i used ide harddisk my luck i found 40 gb IDE hard disk and than i chang from bios hard disk detection auto to LBA(Logical Block Addressing) now it is working since two days hope it has solved my probelm.



  • Hi Guys,

    We have faced the issue again on one of our site and we need to rebuilt the firewal as it was crash after we reboot.

    Please let me know if any one has solution for this.

    Thanks



  • Netgate Administrator

    Have you checked the SMART parameters from these SSDs? Firmware updates?

    Steve



  • Hi Steve,

    Yes, we have run SMART parameter on SSD its seems fine and yesterday one more pfsense failed with the same error.

    Please have a look on SMART report.

    Thanks




  • the same issue was with me. before i was using sata 500GB and i also check 320GB but now i am using 40GB IDE but first you check from bios go to device information page there press enter by selecting HD chnage the reading method from auto to LBA . this trick work for me now pfsense is working fine since two weeks.



  • @ishtiaqaj:

    the same issue was with me. before i was using sata 500GB and i also check 320GB but now i am using 40GB IDE but first you check from bios go to device information page there press enter by selecting HD chnage the reading method from auto to LBA . this trick work for me now pfsense is working fine since two weeks.

    ishtiaqaj, we are not using IDE hard disk, we are using SSD hard disks and for few weeks our firewall is also working fine but its gives error in between after few weeks.



  • @tarun:

    @ishtiaqaj:

    the same issue was with me. before i was using sata 500GB and i also check 320GB but now i am using 40GB IDE but first you check from bios go to device information page there press enter by selecting HD chnage the reading method from auto to LBA . this trick work for me now pfsense is working fine since two weeks.

    ishtiaqaj, we are not using IDE hard disk, we are using SSD hard disks and for few weeks our firewall is also working fine but its gives error in between after few weeks.

    dear tarun, you should also check bios hd option there are three option lbs chs and large you should try large mai b it solve your probelm. yes the firewall work someday and than down…


  • Netgate Administrator

    The smart parameters all look reasonable.
    You said you tried multiple drives but were they all identical SSDs? As Jim suggested here and Chris in your other thread it's not unheard of for a whole batch of drives to have firmware bug that your particular setup is triggering.
    Have you tried running from a standard SATA drive as a test?

    Steve



  • I have encountered the same problem, when I activated

    System: Advanced: Miscellaneous
    Use RAM Disks with default /tmp (40 mb) /var (60 mb) ram disk sizes and
    with 3hrs periodic frequencies

    I had to restore configuration to an earlier state without these settings activated via ssh and reboot. (uninstalled havp package & dashboard widget, but I don't think that was the issue)

    I can finally log back in via web interface.

    here are some info without these settings set up

    __________________________________
    2.1.5-RELEASE (amd64) 
    built on Mon Aug 25 07:44:45 EDT 2014 
    FreeBSD 8.3-RELEASE-p16
    Intel(R) Core(TM)2 Duo CPU E7400 @ 2.80GHz
    2 CPUs: 1 package(s) x 2 core(s)
    Memory usage 
    18% of 4073 MB
    SWAP usage 
    0% of 8192 MB
    Disk usage
    1% of 443G
    ____________________________
    Installed packages:
    
    Cron 0.1.9
    Lightsquid 2.41
    pfBlocker 1.0.2
    Sarg 2.3.6_2 pkg v.0.6.3
    squid3 3.1.20 pkg 2.1.2
    squidGuard-squid3
    1.4_4 pkg v.1.9.12
    _________________________________
    output of pkg_info
    
    bsdinstaller-2.0.2014.0430 BSD Installer mega-package
    compat6x-amd64-6.4.604000.200810_3 A convenience package to install the compat6x libraries
    gettext-0.18.1.1    GNU gettext package
    libexecinfo-1.1_3   A library for inspecting program's backtrace
    libiconv-1.14       A character set conversion library
    p5-Digest-HMAC-1.03 Perl5 interface to HMAC Message-Digest Algorithms
    p5-IO-Socket-SSL-1.52 Perl5 interface to SSL sockets
    p5-Net-SMTP-TLS-0.12_1 An SMTP client supporting TLS and AUTH
    p5-Net-SSLeay-1.42  Perl5 interface to SSL
    perl-5.12.4_3       Practical Extraction and Report Language
    pkg_info: the package info for package 'pkg' is corrupt
    pkgconf-0.8.9       Utility to help to configure compiler and linker flags
    python27-2.7.3_3    An interpreted object-oriented programming language
    samba36-smbclient-3.6.7 Samba "ftp-like" client
    talloc-2.0.7        Hierarchical pool based memory allocator
    tdb-1.2.9,1         Trivial Database
    
    

  • Netgate Administrator

    There's a good chance the HAVP package could have filled /tmp there. That's normally only applied in embedded (Nano) installs where running HAVP is not recomended.
    Increase the size of /tmp if that's the case.

    Steve