Fatal Error Unable to create lock file: No space left on device

[bdjackson]

Hello,

I am receiving an error on the console that reads "Fatal Error Unable to create lock file: No space left on device" I recently updated the firmware on a SG-2100 using the latest firmware version (I believe 24.x). I thought the firmware took just fine because that was three days ago. Today, one of my locations was down because of this error.

Not sure what to do with it next, however this seemed like a bug in the latest software or quite possibly it is unrelated to anything that I have done. I would speculate that one of the log files was not getting purged.

Any thoughts or ideas on this problem?

Thanks,

Brent Jackson

[SteveITS]

@bdjackson check for deleteable boot environments:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/filesystem-shrink.html

Or
https://docs.netgate.com/pfsense/en/latest/troubleshooting/filesystem-usage.html

[bdjackson]

Neither worked. Does not complete a boot. I can get to the Marvell> prompt and that is about it.

Brent

[SteveITS]

@bdjackson Hmm that makes it difficult to do much I suppose. Reinstalling is probably the quickest way back. Then restore from a backup config file.

https://docs.netgate.com/pfsense/en/latest/solutions/netgate-2100/reinstall-pfsense.html

The lock file message is showing at boot then?

[bdjackson]

Yes. Lock file at boot.

Brent

[jlrobinson2171]

I also received this error on my SG2100 after attempting to update to 24.03. I can't do anything except get into the shell via console. Every other option gives me the error:

Fatal Error Unable to create lock file: No space left on device (28)

I looked at the documents posted by @SteveITS but not very helpful. Any other options than reinstalling?

I was able to get into Single User Mode and attempted to run fsck -fy / but I received this error message:

fsck: cannot open `pfSense/ROOT/default_20240426094800': No such file or directory

Any ideas?

[stephenw10]

Run: bectl list

[jlrobinson2171]

@stephenw10

BE Active Mountpoint Space Created
auto-default-20230628094454 - - 589M 2023-06-28 09:44
auto-default-20230628102402 - - 829M 2023-06-28 10:24
auto-default-20240223110046 - - 885M 2024-02-23 11:00
auto-default-20240407101025 - - 563M 2024-04-07 10:10
default - - 1.10G 2024-04-26 09:48
default_20240426094800 NR / 4.98G 2022-06-22 15:27

[SteveITS]

@jlrobinson2171 Delete some of those, for instance the 2022 and 2023 ones.

bectl destroy default_20240426094800

or auto-default-20230628094454, etc.

If it isn't a 2100 max then it's only an 8 GB disk.

[stephenw10]

Yup I would delete those first two BEs and then reboot.

The 2022 BE is the one marked for next boot though so I would not delete that.

If it still fails you can try activating the most recent BE (2024-04-07). That is probably the one created at the last upgrade/attempt.

[jlrobinson2171]

@SteveITS and @stephenw10

Thank you for your assistance. After removing just one, I was able to reboot and log back in.

[Barbapappa]

Found this through Google after I ran into the same issue with my SG-2100. Managed to resolve by following the instructions here, thanks!

Maybe the installer should do a check for sufficient space before continuing and when necessary provide an option to cleanup old instances to prevent this happening in the future?

[Barbapappa]

Edit: ah I see that this has already been suggested in other topics (1 2

[soportemk]

@SteveITS Thank you very much for the response. You rescued us.

[FoolCoconut]

4100 - Upgraded from 24.03 to 24.11 - whole network died, really fascinating how there's no automatic check to see if there's fucking free space before committing to an update/reboot??

[stephenw10]

Were you running ZFS? If drive space was exhausted it would usually just fail to complete the update before the reboot. But in some edge case it would fail to complete the boot check and just roll back to the last good BE.

[Gertjan]

@FoolCoconut said in Fatal Error Unable to create lock file: No space left on device:

really fascinating how there's no automatic check to see if there's fucking free space before committing to an update/reboot??

That's part of the admin's job :

• Have a backup of (pfSense) config file ?
• Disk space looks good ?
• A manual reboot has been executed to handle all outstanding file system issues ?
  and also :
• The USB key with the actual firmware is in good state ?
• If something goes wrong, do I have enough spare time in front of me ?
• Do I have a spare and working Internet access method ?
• .....

Very IMHO of course.
The same list is also valid for your coffee machine, car, alarm, NAS, printer and even phone.
After all, it's ok to trust yourself.
Just don't trust your hardware. Things break all the time. Discovering an issue during a update/upgrade boils down to : bye bye the disk, your data, at best.

If you positively checked all these items, you're good to go.
Checking the list : 5 minutes or so ?
And remember, Murphy's law applies : if you have checked them all, you'll probably need use them. The other way around also applies ^^

[stephenw10]

To be fair here pfSense's handling of filled disks has never been good. But the new ZFS upgrade process is way better in that respect.