Does this look dodgy?

Gertjan

@dal1980 said in Does this look dodgy?:

Apparently the Drive is full too

Probably the best way to blow up a system.
First check : console/SSH in, go to /var/log/ and sub directories.

Btw : your pfSense version ?
Packages installed ?

Dal1980

@gertjan thanks for replying

Apparently I have no packaged installed
"There are no packages currently installed."

PfSense ver: 2.6.0-RELEASE
built on Mon Jan 31 19:57:53 UTC 2022
FreeBSD 12.3-STABLE

I'll go check directories

Dal1980

Looks like lots of large log files

Do I just delete these or is there a gui option to clear these out?

alt text

I'll keep an eye on it in future

johnpoz

@dal1980 said in Does this look dodgy?:

Looks like lots of large log files

Those are not very big.. So unless you have a really small disk those shouldn't be the problem.

Dal1980

@johnpoz

Hmm

I'm not sure what I'm looking for then. Any more tips?

alt text

johnpoz

@dal1980 well your looking for files a lot bigger, or a lot more of them than that if you have 100G drive like your system shows.

Those are only a few MBs in total.

Dal1980

@johnpoz

alt text

Sorry for all the images but I've got no internet and I have to remain connected to my pfSense network.

Unless that command doesn't do what I think it does there doesn't seem to be any large directories

Gertjan

@dal1980 said in Does this look dodgy?:

Apparently I have no packaged installed

But :

Doesn't look like 'pfSense' to me, I never saw that file.
It's a small file, that's not the issue.

Btw : I presume you didn't asked for the right tools ^^
Let me help you : here you go : Google : pfsense check disk space

A couple of seconds later you will find many solutions, like:

du -Pshx /*

and you'll get a list with folders and their sizes.
Pick the biggest, probably a xx G bytes size folder, and repeat :
(Let's says it was /var/ that is huge) :

du -Pshx /var/*

etc.

You will wind up in the folder with many or huge or both files.
Now start backup these files (better get SFTP ready now, easy if you have SSH already set up).
And then it 'rm' time.

Dal1980

@gertjan

Thanks. I googled a few ways to find file sizes but I don't understand what I'm looking for exactly since there isn't a directory that is that big. Here's two other ways (one of which you supplied thank you)

alt text

I also had a look in that file you mentioned.
alt text

I noticed the path /usr/share/bsdconfig/dialog.subr so went poking around there too... Seems there's a bunch of scripts in there wrote by Devin Taske 2006-2015. Could this be left over from a previous version ‍️ ?

alt text

Dal1980

I'm now downloading everything over SFTP to see if I can find any large files that way (or find out where the many files live)

Update: looks like /Dev/ada0p3 is 3.79GB also looks like ada0p2 is still downloading at 10GB. What are these files? I just cancelled the /Dev folder downloads because I'm not sure what they are.

Update2: I also found them ada files in /var/dhcpd/dev/ folder too which I just cancelled.

Everything downloaded and it seems to be only them ada files that seem to be huge. I looked through everything else and found nothing of any real size.

Gertjan

@dal1980

/dev/ files are your devices like 'ada' is your hard drive. Don't 'touch' these.
Remember : everything for Unix and FreeBSD is files.

Do also this : How to Run a pfSense Software File System Check

Dal1980

@gertjan

Thanks very much. That filechecker solved whatever was going on with the filesystem. Ran it 4 times for good measure, rebooted and the drive space went from 102GB/104GB used to 2GB/104GB used.

Gertjan

@dal1980

Humm. Good.
It's still on my 'need to understand that ones and for all' list : the disk system is marked a full, but no big or many files that occupy the place.
It's probably an 'inode' thing, like files are de allocated, but the free clusters (inodes ?) are not given back to the free drive space pool. Over time this creates what you just experienced.
That why the Netgate video exists. It's even not a pfSense thing, it's a FreeBSD thing.
The file system is pretty resilient, but people still tend to 'remove the power cord' without shutting down the system first (do that with your PC x times, and you will loose your PC (disk) !).
That's also why on newer systems pfSense prefers now the ZFS as a file system.

rcoleman-netgate

@dal1980 said in Does this look dodgy?:

Ran it 4 times for good measure

Good. In TAC we recommend 5-10. Thankfully you can queue them up easily with the up arrow and enter.

rcoleman-netgate

@gertjan said in Does this look dodgy?:

Over time this creates what you just experienced.

{professional aside}
This is a similar thing to what happens when you only delete photos from a SD/CF card and never re-format. In order to spare the writes across the system it just marks the formerly used blocks as... well, still used but doesn't move the write point because of fragmentation. Seen this many times with department photogs in my last job because my supervisor told them never to format, but then the card stopped working because they ran out of the next set of available blocks.

Similar things happen with APFS on the latest releases of macOS but eventually the system catches up and shuffles the data it needs to and 'frees' up the missing space for writing.

Dal1980

Thanks guys.

I normally shutdown properly but had an issue with power loss at home which was pretty regular. I have a UPS but it's just my Unraid that's plugged into that but I'll look at getting my pfSense hooked up to it too maybe. Power cuts were down to my Lava Lamp as it happens which I've now stopped using anyway.

Interesting stuff though.