filesystem full (previous title devs full)

[Phil Ten]

Hello,

It's a brand new Netgate XG-7100, installed yesterday.

Today I can see that devfs is full

df -h

Filesystem Size Used Avail Capacity Mounted on
/dev/ufsid/5c33f6d457277de3 27G 23G 2.2G 91% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/md0 3.4M 100K 3.0M 3% /var/run
devfs 1.0K 1.0K 0B 100% /var/dhcpd/dev

du -k -d 1 /dev | sort -nr

4 /dev
1 /dev/usb
1 /dev/ufsid
1 /dev/reroot
1 /dev/label
1 /dev/iov
1 /dev/fd

My situation is quite critical, and I hope someone can help.

• file config.xml is saved truncated (I guess because of disk being full)
• I still have web gui access, but no ssh access. However, I can still enter shell commands with the web gui
• I do not want to reboot as with half config.xml file, I afraid I will no longer be able to connect again
• I tried to restore a good config but it failed with error 500

My idea would be to free space with the web gui, restore a safe config and reboot

Can someone suggest what I can safely delete in /dev ?

or any other ideas to restire a running device

All suggestions are very welcome

My next question is what can be the cause for devfs becoming full ?

Thank you very much

Phil

[NogBadTheBad]

It's normal, nothing to see here

https://forums.freebsd.org/threads/whats-devfs.1742/

[2.4.5-RELEASE][admin@pfsense]/root: df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/ufsid/5cac5dc6ba6d28e2 111G 2.1G 100G 2% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/md0 3.4M 224K 2.9M 7% /var/run
devfs 1.0K 1.0K 0B 100% /var/dhcpd/dev
[2.4.5-RELEASE][admin@pfsense]/root:

$ uname -a
FreeBSD sftp 12.0-RELEASE FreeBSD 12.0-RELEASE r341666 GENERIC arm
$ df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/ufs/rootfs 58G 1.6G 51G 3% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/msdosfs/MSDOSBOOT 50M 16M 34M 32% /boot/msdos
tmpfs 50M 4.0K 50M 0% /tmp
$

[Phil Ten]

Thank you for your post. I understand your point, so devfs is not the problem.

I saw 100% and made a conclusion too fast.

However, my NetGate no longer works and I do get these errors:

Jun 5 12:20:01 kernel pid 22652 (ntopng), uid 0 inumber 3373971 on /: filesystem full
Jun 5 12:20:00 kernel pid 22652 (ntopng), uid 0 inumber 3370921 on /: filesystem full
Jun 5 12:19:56 kernel pid 22652 (ntopng), uid 0 inumber 3371556 on /: filesystem full
Jun 5 12:19:41 kernel pid 22652 (ntopng), uid 0 inumber 3371556 on /: filesystem full

and a few "No space left on device"

"/var/tmp/nginx/client_body_temp/0000000002" failed (28: No space left on device), client: 83.159.49.9, server: , request: "POST /diag_backup.php HTTP/2.0", host: "82.210.41.178", referrer: "https://82.210.41.178/diag_backup.php"

df -h

Filesystem Size Used Avail Capacity Mounted on
/dev/ufsid/5c33f6d457277de3 27G 23G 2.2G 91% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/md0 3.4M 100K 3.0M 3% /var/run
devfs 1.0K 1.0K 0B 100% /var/dhcpd/dev

I am surprised since / has 91%, not 100%

Thank you for all help

Phil

[NogBadTheBad]

I'd be tempted to look for core dumps in the /root directory, IMO ntopng is core dump happy.

[Phil Ten]

Thank you for your suggest.

The device is now off line, and I am not on site,
there I can't check immediately.

However, it doesn't seem to be the problem,
before I lost connection I could run a couple du command

Space seem used in /var/db

du -k -d 1 /var | sort -nr

23841779 /
22250676 /var
1028844 /usr
330388 /root
161856 /boot
11868 /lib
9972 /rescue

du -k -d 1 /var | sort -nr

22251092 /var
22069252 /var/db
167844 /var/cache
9888 /var/log
3756 /var/dhcpd
100 /var/run

Thanks

[Gertjan]

@Phil-Ten said in devs full:

My next question is what can be the cause for devfs becoming full ?

No packages installed ? Just the classic config and your file system fills up to 91 % ?
You still have 2,2 Gbytes left ...

This (example) :

devfs                          1.0K    1.0K      0B   100%    /dev
/dev/md0                       3.4M    184K    3.0M     6%    /var/run
devfs                          1.0K    1.0K      0B   100%    /var/unbound/dev
/usr/local/bin                 285G    2.6G    259G     1%    /var/unbound/usr/local/bin
/usr/local/lib                 285G    2.6G    259G     1%    /var/unbound/usr/local/lib

is normal.

edit : always activate the SSH access. It's not a gadget, but a live-saver tool.
Plan B : you have the console access. Works always.
GUI access works well ... if all goes well.

[Phil Ten]

I installed package ntopng, nothing else.

Again, it was installed yesterday, so only 24h ntopng datas and logs.

Is it normal to get "filesystem full" errors with 91% ?
Maybe the 9% left are somehow reserved and not available ?

I am quite sur I did enable SSH access, for sure I will double check next time.

I hope you are rigth whe you say console always works, I will try as soon as possible

Thanks

Phil

[psp]

I'd bet my .02 on ntopng (/var/db/ntopng). After updating to latest version, I need to delete ntopng data twice a month.

[Phil Ten]

Solved. Thanks to all for the help.

I case someone else has this same problem, here is what I did:

I used the USB Cable to connect to the console
I could see all the disk space was used in /var/db/ntopng, as psp said.
I deleted a few logs file, then I could restore a valid config file with the console menu
After a reboot, I could login with the Webgui and delete all datas from ntopng in the settings
and disable the package.

Phil

[Gertjan]

See also https://forum.netgate.com/topic/142004/dhcp-config-is-apparently-not-updated-in-a-safe-fashion