filesystem full (previous title devs full)
-
Hello,
It's a brand new Netgate XG-7100, installed yesterday.
Today I can see that devfs is full
df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/ufsid/5c33f6d457277de3 27G 23G 2.2G 91% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/md0 3.4M 100K 3.0M 3% /var/run
devfs 1.0K 1.0K 0B 100% /var/dhcpd/devdu -k -d 1 /dev | sort -nr
4 /dev
1 /dev/usb
1 /dev/ufsid
1 /dev/reroot
1 /dev/label
1 /dev/iov
1 /dev/fdMy situation is quite critical, and I hope someone can help.
- file config.xml is saved truncated (I guess because of disk being full)
- I still have web gui access, but no ssh access. However, I can still enter shell commands with the web gui
- I do not want to reboot as with half config.xml file, I afraid I will no longer be able to connect again
- I tried to restore a good config but it failed with error 500
My idea would be to free space with the web gui, restore a safe config and reboot
Can someone suggest what I can safely delete in /dev ?
or any other ideas to restire a running device
All suggestions are very welcome
My next question is what can be the cause for devfs becoming full ?
Thank you very much
Phil
-
It's normal, nothing to see here
https://forums.freebsd.org/threads/whats-devfs.1742/
[2.4.5-RELEASE][admin@pfsense]/root: df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/ufsid/5cac5dc6ba6d28e2 111G 2.1G 100G 2% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/md0 3.4M 224K 2.9M 7% /var/run
devfs 1.0K 1.0K 0B 100% /var/dhcpd/dev
[2.4.5-RELEASE][admin@pfsense]/root:$ uname -a
FreeBSD sftp 12.0-RELEASE FreeBSD 12.0-RELEASE r341666 GENERIC arm
$ df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/ufs/rootfs 58G 1.6G 51G 3% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/msdosfs/MSDOSBOOT 50M 16M 34M 32% /boot/msdos
tmpfs 50M 4.0K 50M 0% /tmp
$ -
Thank you for your post. I understand your point, so devfs is not the problem.
I saw 100% and made a conclusion too fast.
However, my NetGate no longer works and I do get these errors:
Jun 5 12:20:01 kernel pid 22652 (ntopng), uid 0 inumber 3373971 on /: filesystem full
Jun 5 12:20:00 kernel pid 22652 (ntopng), uid 0 inumber 3370921 on /: filesystem full
Jun 5 12:19:56 kernel pid 22652 (ntopng), uid 0 inumber 3371556 on /: filesystem full
Jun 5 12:19:41 kernel pid 22652 (ntopng), uid 0 inumber 3371556 on /: filesystem fulland a few "No space left on device"
"/var/tmp/nginx/client_body_temp/0000000002" failed (28: No space left on device), client: 83.159.49.9, server: , request: "POST /diag_backup.php HTTP/2.0", host: "82.210.41.178", referrer: "https://82.210.41.178/diag_backup.php"
df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/ufsid/5c33f6d457277de3 27G 23G 2.2G 91% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/md0 3.4M 100K 3.0M 3% /var/run
devfs 1.0K 1.0K 0B 100% /var/dhcpd/devI am surprised since / has 91%, not 100%
Thank you for all help
Phil
-
I'd be tempted to look for core dumps in the /root directory, IMO ntopng is core dump happy.
-
Thank you for your suggest.
The device is now off line, and I am not on site,
there I can't check immediately.However, it doesn't seem to be the problem,
before I lost connection I could run a couple du commandSpace seem used in /var/db
du -k -d 1 /var | sort -nr
23841779 /
22250676 /var
1028844 /usr
330388 /root
161856 /boot
11868 /lib
9972 /rescuedu -k -d 1 /var | sort -nr
22251092 /var
22069252 /var/db
167844 /var/cache
9888 /var/log
3756 /var/dhcpd
100 /var/runThanks
-
My next question is what can be the cause for devfs becoming full ?
No packages installed ? Just the classic config and your file system fills up to 91 % ?
You still have 2,2 Gbytes left ...This (example) :
devfs 1.0K 1.0K 0B 100% /dev /dev/md0 3.4M 184K 3.0M 6% /var/run devfs 1.0K 1.0K 0B 100% /var/unbound/dev /usr/local/bin 285G 2.6G 259G 1% /var/unbound/usr/local/bin /usr/local/lib 285G 2.6G 259G 1% /var/unbound/usr/local/lib
is normal.
edit : always activate the SSH access. It's not a gadget, but a live-saver tool.
Plan B : you have the console access. Works always.
GUI access works well ... if all goes well. -
I installed package ntopng, nothing else.
Again, it was installed yesterday, so only 24h ntopng datas and logs.
Is it normal to get "filesystem full" errors with 91% ?
Maybe the 9% left are somehow reserved and not available ?I am quite sur I did enable SSH access, for sure I will double check next time.
I hope you are rigth whe you say console always works, I will try as soon as possible
Thanks
Phil
-
I'd bet my .02 on ntopng (/var/db/ntopng). After updating to latest version, I need to delete ntopng data twice a month.
-
Solved. Thanks to all for the help.
I case someone else has this same problem, here is what I did:
I used the USB Cable to connect to the console
I could see all the disk space was used in /var/db/ntopng, as psp said.
I deleted a few logs file, then I could restore a valid config file with the console menu
After a reboot, I could login with the Webgui and delete all datas from ntopng in the settings
and disable the package.Phil
-
See also https://forum.netgate.com/topic/142004/dhcp-config-is-apparently-not-updated-in-a-safe-fashion