Warning: fopen(/tmp/config.lock): failed to open stream

tarun · Jan 15, 2015, 9:31 AM

I have just tried installation on a new router,
things are fine and the links are up.

But after few days I cannot enter the web config page and getting below error.
I typed the default password and it then give the below error and it can't login.

Warning: fopen(/tmp/config.lock): failed to open stream: Device not configured in /etc/inc/util.inc on line 123 Warning: flock() expects parameter 1 to be resource, null given in /etc/inc/util.inc on line 134 Warning: fclose(): supplied argument is not a valid stream resource in /etc/inc/util.inc on line 135 Warning: session_start(): open(/var/tmp//sess_fd59264b4e5cf3b6ae1121f1ccf4762a, O_RDWR) failed: No space left on device (28) in /etc/inc/auth.inc on line 1254

We are using Super Micro Server Model 502-2 P/N 5015A-EHF-D525 with 32 GB crucial hdd.

Please suggest

stephenw10 · Dec 23, 2014, 9:31 PM

What are you booting from?

Steve

tarun · Jan 15, 2015, 9:34 AM

Hi Steve, basically we are running pfSense 2.1.5-RELEASE (i386) on supermicro box and booting from a hard drive.

When I have logged in via console on pfsense box then found below error, can you please suggest how to resolve this issue without reinstalling the firewall.

Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.ccoOV4, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_lock failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_unlock failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.QLoRQD, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.aWv6Ra, O_RDWR|O_CREA
Fatal error: PHP Startup: apc_fcntl_lock failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_unlock failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.sgK0ch, O_RDWR|O_CREA
Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.BEhNIf, O_RDWR|O_CREA
Fatal error: Unknown: apc_fcntl_rdlock failed: in Unknown on line 0

Logout (SSH only) 8 ) Shell
Assign Interfaces 9) pfTop
Set interface(s) IP address 10) Filter Logs
Reset webConfigurator password 11) Restart webConfigurator
Reset to factory defaults 12) pfSense Developer Shell
Reboot system 13) Upgrade from console
Halt system 14) Disable Secure Shell (sshd)
Ping host 15) Restore recent configuration

Enter an option: 8

/etc/rc.initial: /bin/tcsh: Device not configured
/: write failed, filesystem is full
Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.PA8GQF, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_lock failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_unlock failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.LMIIbh, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.I7p46p, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_lock failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_unlock failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.UPJxwV, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.tZet8w, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
Fatal error: Unknown: apc_fcntl_rdlock failed: in Unknown on line 0

stephenw10 · Dec 24, 2014, 3:38 PM

/: write failed, filesystem is full

This looks like a boot device failure of some sort. Re-reading your first post I see you're booting from a hard drive. How did you install to the HD?

Have you moved /tmp to a ram drive? (an advanced gui option)

If it were me I would re-install faced with errors like that and run some low level disc check utility. It's not hard to re-install and restore the config.

Steve

Iahmad · Dec 24, 2014, 11:48 PM

i am getting the same error i left the office with everthing fine but in the morning i get the same error image is attached. according to my experiance it is HD error now i will try with new harddisk than update you soon

tarun · Dec 25, 2014, 6:57 AM

Hi Steve, Thanks for your reply.

We have fresh installed pfsense with cd and this is a brand new hardware with new HDD, also i would like to update you that we have 8 locations where we installed pfsense firewall and this is 3rd time on 3rd location where I get same error. Its seems some bug which is creating this issue.

If I reboot the firewall with error, its got crash and then it needs to reinstall again.

Please let me know if there is any solution for this issue.

Also please find the attached Screen shot or error.

Iahmad · Dec 25, 2014, 6:10 AM

i tried with other HD 500 gb sata but still get the same error next morning.
now i install old 40gb ide let ce what happen now.

Iahmad · Dec 25, 2014, 7:25 AM

again failed after 30 minute of online.

Iahmad · Dec 26, 2014, 6:35 PM

anybody is there for help please

every morning i amgetting the pfsense box down. i have tried 5 different harddisk assuming that it is hd error.

stephenw10 · Dec 27, 2014, 4:03 PM

Hmm. It could be complete coincidence but there seem to be an unusually large number of reports of similar failures on the forum at the moment.

If you've tried multiple hard drives perhaps it's a cabling or disk controller issue?

Steve

phil.davis · Dec 28, 2014, 3:44 PM

The "no inodes free" message in your console screenshot makes me think something is creating many many little files, probably in /tmp or /var somewhere.
Use console menu option 8 to get a command prompt. Then look around /var/ and /tmp and see if some folder has many many little files. Report back what they are and we can then think about why it is happening.

stephenw10 · Dec 28, 2014, 4:11 PM

Ah, well spotted I missed that. That's on Tarun's screenshot?

Steve

phil.davis · Dec 28, 2014, 4:52 PM

@stephenw10:

Ah, well spotted I missed that. That's on Tarun's screenshot?

Yes, Tarun's screenshot has a "create symlink failed - no inodes free"

I guess something is creating many many little files or?

stephenw10 · Dec 28, 2014, 8:04 PM

Yep. I don't see any inode warnings from ishtiaqaj though, here or anywhere else.
The fstat command referenced here looks appropriate for this:

[2.2-RC][root@xtm5.localdomain]/root: fstat | awk '\!/CMD/{print $2} END{printf("* Total files open: %d", NR)}' | sort | uniq -c | sort -n
   1 * Total files open: 511
   2 kernel
   2 md0
   2 md1
   3 init
   4 getty
   6 rrdtool
   7 apinger
   7 awk
   7 cron
   7 sort
   7 uniq
   8 fstat
   8 login
   8 sshlockout_pf
  12 sleep
  13 powerd
  14 dnsmasq
  15 LCDd
  15 filterlog
  15 lcdproc
  15 lighttpd
  16 check_reload_status
  16 devd
  16 tcsh
  17 inetd
  20 dhcpd
  22 sshd
  23 ntpd
  36 dhclient
  36 minicron
  42 php-fpm
  42 syslogd
  47 sh

Nothing too extreme on my test 2.2 box.

Steve

Derelict · Dec 28, 2014, 8:12 PM

what does df -i show?

![Screen Shot 2014-12-28 at 12.09.15 PM.png](/public/imported_attachments/1/Screen Shot 2014-12-28 at 12.09.15 PM.png)
![Screen Shot 2014-12-28 at 12.09.15 PM.png_thumb](/public/imported_attachments/1/Screen Shot 2014-12-28 at 12.09.15 PM.png_thumb)

tarun · Dec 29, 2014, 4:41 AM

Hi phil,

Thanks for your reply, actually when I get this error, i tried to use console menu option 8 to get a command prompt but its unable me to get into the command prompt. Its giving below error.

Logout (SSH only) 8 ) Shell
Assign Interfaces 9) pfTop
Set interface(s) IP address 10) Filter Logs
Reset webConfigurator password 11) Restart webConfigurator
Reset to factory defaults 12) pfSense Developer Shell
Reboot system 13) Upgrade from console
Halt system 14) Disable Secure Shell (sshd)
Ping host 15) Restore recent configuration

Enter an option: 8

/etc/rc.initial: /bin/tcsh: Device not configured
/: write failed, filesystem is full
Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.PA8GQF, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_lock failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_unlock failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.LMIIbh, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.I7p46p, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_lock failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_unlock failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.UPJxwV, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.tZet8w, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
Fatal error: Unknown: apc_fcntl_rdlock failed: in Unknown on line 0

tarun · Dec 29, 2014, 5:40 AM

As per the error shown on GUI page its clear that its some where a issue with hdd full in tmp and var directory but this issue keep coming up again and I have also read on forum some guys reported the same issue even they used 500 GB hdd. Its seems some bug which is creating this issue.

When I tried to get command prompt access it given me php error as shown in my post. If I tried to restart the pfsense, its got crash and then I need to fresh install. One more thing if we tried to install pfSense on same hdd it never install on it and say I/O error, even the hdd is brand new. Please find the screen shot.

This same thing was happen on 3 of our pfsense, which have brand new hardware with brand new hdd.

Derelict · Dec 29, 2014, 6:49 AM

If it's brand new hardware, why are you running i386? https://forum.pfsense.org/index.php?topic=84679.0

What image are you installing? How?

tarun · Dec 29, 2014, 7:20 AM

We are installing i386 from CD image downloaded from https://www.pfsense.org/download/mirror.php?section=downloads.

Its Supermicro Model 502-2 P/N 5015A-EHF-D525 with 32 GB crucial hdd.

I hope you have read all error posted by me from starting.

Error Post 1

after few days I cannot enter the web config page and getting below error.
I typed the default password and it then give the below error and it can't login.

Warning: fopen(/tmp/config.lock): failed to open stream: Device not configured in /etc/inc/util.inc on line 123 Warning: flock() expects parameter 1 to be resource, null given in /etc/inc/util.inc on line 134 Warning: fclose(): supplied argument is not a valid stream resource in /etc/inc/util.inc on line 135 Warning: session_start(): open(/var/tmp//sess_fd59264b4e5cf3b6ae1121f1ccf4762a, O_RDWR) failed: No space left on device (28) in /etc/inc/auth.inc on line 1254

We are using Super Micro Server with 32 GB crucial hdd.

Error Post 2

Tried to use console menu option 8 to get a command prompt but its unable me to get into the command prompt. Its giving below error.

Logout (SSH only) 8 ) Shell
Assign Interfaces 9) pfTop
Set interface(s) IP address 10) Filter Logs
Reset webConfigurator password 11) Restart webConfigurator
Reset to factory defaults 12) pfSense Developer Shell
Reboot system 13) Upgrade from console
Halt system 14) Disable Secure Shell (sshd)
Ping host 15) Restore recent configuration

Enter an option: 8

/etc/rc.initial: /bin/tcsh: Device not configured
/: write failed, filesystem is full
Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.PA8GQF, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_lock failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_unlock failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.LMIIbh, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.I7p46p, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_lock failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_unlock failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.UPJxwV, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.tZet8w, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
Fatal error: Unknown: apc_fcntl_rdlock failed: in Unknown on line 0

Error Post 3

As per the error shown on GUI page its clear that its some where a issue with hdd full in tmp and var directory but this issue keep coming up again and I have also read on forum some guys reported the same issue even they used 500 GB hdd. Its seems some bug which is creating this issue.

When I tried to get command prompt access it given me php error as shown in my post. If I tried to restart the pfsense, its got crash and then I need to fresh install. One more thing if we tried to install pfSense on same hdd it never install on it and say I/O error, even the hdd is brand new.

When we format hdd then its able to reinstalled the pfsense.

Derelict · Dec 29, 2014, 7:35 AM

Why i386? Why not amd64? All those I/O errors make me think the filesystem full messages are sort of a false indicator.

Looks like failing disk from here.