Warning: fopen(/tmp/config.lock): failed to open stream

[phil.davis]

The "no inodes free" message in your console screenshot makes me think something is creating many many little files, probably in /tmp or /var somewhere.
Use console menu option 8 to get a command prompt. Then look around /var/ and /tmp and see if some folder has many many little files. Report back what they are and we can then think about why it is happening.

[stephenw10]

Ah, well spotted I missed that. That's on Tarun's screenshot?

Steve

[phil.davis]

@stephenw10:

Ah, well spotted I missed that. That's on Tarun's screenshot?

Yes, Tarun's screenshot has a "create symlink failed - no inodes free"

I guess something is creating many many little files or?

[stephenw10]

Yep. I don't see any inode warnings from ishtiaqaj though, here or anywhere else.
The fstat command referenced here looks appropriate for this:

[2.2-RC][root@xtm5.localdomain]/root: fstat | awk '\!/CMD/{print $2} END{printf("* Total files open: %d", NR)}' | sort | uniq -c | sort -n
   1 * Total files open: 511
   2 kernel
   2 md0
   2 md1
   3 init
   4 getty
   6 rrdtool
   7 apinger
   7 awk
   7 cron
   7 sort
   7 uniq
   8 fstat
   8 login
   8 sshlockout_pf
  12 sleep
  13 powerd
  14 dnsmasq
  15 LCDd
  15 filterlog
  15 lcdproc
  15 lighttpd
  16 check_reload_status
  16 devd
  16 tcsh
  17 inetd
  20 dhcpd
  22 sshd
  23 ntpd
  36 dhclient
  36 minicron
  42 php-fpm
  42 syslogd
  47 sh

Nothing too extreme on my test 2.2 box.

Steve

[Derelict]

what does df -i show?


[tarun]

Hi phil,

Thanks for your reply, actually when I get this error, i tried to use console menu option 8 to get a command prompt but its unable me to get into the command prompt. Its giving below error.

0. Logout (SSH only)                  8 ) Shell
1. Assign Interfaces                  9) pfTop
2. Set interface(s) IP address      10) Filter Logs
3. Reset webConfigurator password    11) Restart webConfigurator
4. Reset to factory defaults        12) pfSense Developer Shell
5. Reboot system                    13) Upgrade from console
6. Halt system                      14) Disable Secure Shell (sshd)
7. Ping host                        15) Restore recent configuration

Enter an option: 8

/etc/rc.initial: /bin/tcsh: Device not configured
/: write failed, filesystem is full
Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.PA8GQF, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_lock failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_unlock failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.LMIIbh, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.I7p46p, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_lock failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_unlock failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.UPJxwV, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.tZet8w, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
Fatal error: Unknown: apc_fcntl_rdlock failed: in Unknown on line 0

[tarun]

As per the error shown on GUI page its clear that its some where a issue with hdd full in tmp and var directory but this issue keep coming up again and I have also read on forum some guys reported the same issue even they used 500 GB hdd. Its seems some bug which is creating this issue.

When I tried to get command prompt access it given me php error as shown in my post. If I tried to restart the pfsense, its got crash and then I need to fresh install. One more thing if we tried to install pfSense on same hdd it never install on it and say I/O error, even the hdd is brand new. Please find the screen shot.

This same thing was happen on 3 of our pfsense, which have brand new hardware with brand new hdd.

[Derelict]

If it's brand new hardware, why are you running i386?  https://forum.pfsense.org/index.php?topic=84679.0

What image are you installing?  How?

[tarun]

We are installing i386 from CD image downloaded from https://www.pfsense.org/download/mirror.php?section=downloads.

Its Supermicro Model 502-2 P/N 5015A-EHF-D525 with 32 GB crucial hdd.

I hope you have read all error posted by me from starting.

Error Post 1

after few days I cannot enter the web config page and getting below error.
I typed the default password and it then give the below error and it can't login.

Warning: fopen(/tmp/config.lock): failed to open stream: Device not configured in /etc/inc/util.inc on line 123 Warning: flock() expects parameter 1 to be resource, null given in /etc/inc/util.inc on line 134 Warning: fclose(): supplied argument is not a valid stream resource in /etc/inc/util.inc on line 135 Warning: session_start(): open(/var/tmp//sess_fd59264b4e5cf3b6ae1121f1ccf4762a, O_RDWR) failed: No space left on device (28) in /etc/inc/auth.inc on line 1254

We are using Super Micro Server with 32 GB crucial hdd.

Error Post 2

Tried to use console menu option 8 to get a command prompt but its unable me to get into the command prompt. Its giving below error.

0. Logout (SSH only)                  8 ) Shell
1. Assign Interfaces                  9) pfTop
2. Set interface(s) IP address      10) Filter Logs
3. Reset webConfigurator password    11) Restart webConfigurator
4. Reset to factory defaults        12) pfSense Developer Shell
5. Reboot system                    13) Upgrade from console
6. Halt system                      14) Disable Secure Shell (sshd)
7. Ping host                        15) Restore recent configuration

Enter an option: 8

/etc/rc.initial: /bin/tcsh: Device not configured
/: write failed, filesystem is full
Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.PA8GQF, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_lock failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_unlock failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.LMIIbh, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.I7p46p, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_lock failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_unlock failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.UPJxwV, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
Fatal error: PHP Startup: apc_fcntl_create: open(/tmp/.apc.tZet8w, O_RDWR|O_CREAT, 0666) failed: in Unknown on line 0
Fatal error: Unknown: apc_fcntl_rdlock failed: in Unknown on line 0

Error Post 3

As per the error shown on GUI page its clear that its some where a issue with hdd full in tmp and var directory but this issue keep coming up again and I have also read on forum some guys reported the same issue even they used 500 GB hdd. Its seems some bug which is creating this issue.

When I tried to get command prompt access it given me php error as shown in my post. If I tried to restart the pfsense, its got crash and then I need to fresh install. One more thing if we tried to install pfSense on same hdd it never install on it and say I/O error, even the hdd is brand new.

When we format hdd then its able to reinstalled the pfsense.

[Derelict]

Why i386?  Why not amd64?  All those I/O errors make me think the filesystem full messages are sort of a false indicator.

Looks like failing disk from here.

[Derelict]

All modern intel processors are amd64, bro.  AMD won that one.  Install the 64-bit version. https://forum.pfsense.org/index.php?topic=84679.0

I don't think the file systems are full.  I think something on a failing disk or a weird install case is reporting a disk full.

[phil.davis]

The console screen picture shows disk errors. So I expect the disk is not really full, the disk writes or reads are failing.
If it was running nicely for some time and the problem has only now started, then I guess a failing disk.
If the problem has always been there, then some issue between the hardware and the FreeBSD disk drivers.

[stephenw10]

The picture here is getting muddied by multiple reports across different systems etc.
@tarun:

I have just concerned with one of my colleague, he has installed AMD64 but faced the same issue.

Is it actually exactly the same symptoms or just similar? Did he try to replicate your setup? Same hardware?

Tarun, you posted one screenshot showing an out of inodes warning. That is potentially a solvable problem. If something in your setup is creating hundreds of files it should be possible to see what is doing it and find out why.

The other errors shown here and on other threads usually point to a failing harddrive.  ishtiaqaj has tried multiple hard drives with the same result which then leads me to suspect a bad cable or disk controller or some badly supported hardware. Tarun, have you tried more than one hard drive?

Although the problems here look similar anything that prevents writing to the drive could throw these same errors. The fact that the vast majority of users are running the same version as you and not seeing any errors implies that your install is different somehow. Which Supermicro box are you using? They are well tested though it would be surprising to find something unsupported.

Steve

[tarun]

Yes, we have tried different hdd, but the issue same. We have face this issue on 3 of our location.

We are using Supermicro 502-2 1U Server 5015A-EHF-D525 w/Intel Atom D525 CPU /4GB RAM/ 32 GB Crucial hdd.

I have just concerned with one of my colleague, he has installed AMD64 but faced the same issue.

Warning: fopen(/tmp/config.lock): failed to open stream: Device not configured in /etc/inc/util.inc on line 123 Warning: flock() expects parameter 1 to be resource, null given in /etc/inc/util.inc on line 134 Warning: fclose(): supplied argument is not a valid stream resource in /etc/inc/util.inc on line 135 Warning: session_start(): open(/var/tmp//sess_fd59264b4e5cf3b6ae1121f1ccf4762a, O_RDWR) failed: No space left on device (28) in /etc/inc/auth.inc on line 1254

[stephenw10]

Ok,  that's fairly standard hardware, nothing unusual there. Is it identical on all three sites? Are those Crucial SSDs or really old/small hard drives? How old are they? Checking the SMART stats on them would probably be good idea.
If it's none of those then i'd look at something unusual in your config that's common across the sites.

Steve

[jimp]

@tarun:

I have just concerned with one of my colleague, he has installed AMD64 but faced the same issue.

Warning: fopen(/tmp/config.lock): failed to open stream: Device not configured in /etc/inc/util.inc on line 123 Warning: flock() expects parameter 1 to be resource, null given in /etc/inc/util.inc on line 134 Warning: fclose(): supplied argument is not a valid stream resource in /etc/inc/util.inc on line 135 Warning: session_start(): open(/var/tmp//sess_fd59264b4e5cf3b6ae1121f1ccf4762a, O_RDWR) failed: No space left on device (28) in /etc/inc/auth.inc on line 1254

More likely than not, the disk is dead. If you have multiple units doing it, then it wouldn't be the first time someone got a bad batch of SSDs. We've seen similar bad batches with both Kingston and Crucial in the past.

It comes down to one of three possibilities:
1. The drive really is full, or at least /var and /tmp both are (unlikely)
2. The disk is dead, in which case it's a coin toss as to whether the system will come back after a reboot
3. The disk has stopped responding to the operating system due to a controller issue, which may be OK after a reboot. Sometimes there are firmware updates for SSDs to help with such problems.

More info here:
https://doc.pfsense.org/index.php/Filesystem_Full_/_Out_of_Inode_Errors

[tarun]

Hi Steve,

Yes it is identical on all three sites,

These Crucial SSDs are brand new hard drives.

Thanks

[stephenw10]

Well, as Jim said, they could potentially be from a bad batch of drives if you bought them all together or maybe they have bad firmware that can be updated.  :-\ Have you tried using any other drives?

Steve

[Iahmad]

the same probelm i was facing and it is with SATA harddisk now i used ide harddisk my luck i found 40 gb IDE hard disk and than i chang from bios hard disk detection auto to LBA(Logical Block Addressing) now it is working since two days hope it has solved my probelm.

[tarun]

Hi Guys,

We have faced the issue again on one of our site and we need to rebuilt the firewal as it was crash after we reboot.

Please let me know if any one has solution for this.

Thanks