/var is low on disk space
-
@SteveITS I don't know if I'm running a ram disk. It's basically a default install from Netgate, it's their disk layout. Thanks for boot environments suggestion. I forgot about that one. And I failed to see that filesystem-shrink in the doc's but did notice an impressive amount of info in that section.
#du -h -d 1 /var 512B /var/games 512B /var/account 3.1M /var/dhcpd 512B /var/empty 512B /var/msgs 4.7M /var/log 512B /var/preserve 14K /var/mail 39K /var/tmp 495M /var/unbound 12K /var/spool 512B /var/authpf 198M /var/cache 6.2M /var/syslog-ng 80M /var/db 1.0K /var/lib 512B /var/crash 512B /var/yp 9.8M /var/backups 40K /var/etc 135K /var/run 1.5K /var/audit 512B /var/heimdal 512B /var/rwho 1.5K /var/at 1.0K /var/cron 797M /var -
@JonH You’d have to set a RAM disk in System/Advanced/Miscellaneous. Plus it would be tmpfs not ZFS. It just seems odd it’s 51 MB. But I don’t have access to a 5100.
Did you reinstall, and change partitioning maybe? With ZFS I think it will reduce the “disk capacity” a la that doc page.
My 2100 has /var/db and /var/cache/pkg at 4GB.
-
@SteveITS said in /var is low on disk space:
Did you reinstall, and change partitioning maybe? With ZFS I think it will reduce the “disk capacity” a la that doc page.
There is no RAM disk active in advanced->misc.
Last year I requested from Netgate a download to update my system to zfs. Unfortunately I cannot locate the details but suffice to say there have been a number of updates from Netgate since then and other than awhile ago having the DNS issue that some had, everything has been hunky-dorie until now.
A few days after 23.05.1 was released I installed it. A few days after that I found my Apple mobile devices (also recently updated) unable to connect (no route to host). Checking my resolver.log shows no activity although I still have connectivity via wired desktop.
I still have not clearly identified the culprit, be it a switch, an AP, pfSense, or pfBlocker or something else.
I'll read that doc you linked to before I fool with removing boot environment files. The sizes of those files seem insignificant to my problem unless you are saying to remove the default environment also. I presume it is needed for booting but maybe this is essentially a backup to easily reinstall the system? I won't touch that until I get more info.
I am planning to force a reinstall of pfBlocker before I do anything else.
My AP's are 2 Unifi (and 1 older Asus). Unifi has a separate web interface with a dashboard that shows reliability. That indicates a loss of 'wifi experience' at 0015 hrs, the exact time pfBlocker runs an update. I cannot discount that perhaps the AP is about to croak and I'll fool around with that later in the week. My unifi also sends me a msg that a 'rogue AP is near my Unifi' (yes, that would be the Asus). All of my AP's have the same SSID to allow easy roaming throughout my property. Today I disabled the Asus radio's so I'll see if the situation changes in the next 24 hrs. So it's possible that my problem is not with pfSense but I'm still concerned by the very low free space showing in /var.
-
@JonH The boot environments may not help, they are part of ZFS and not in var. just a quick way to free space in general. I haven’t had many with ZFS as most of our clients are older but yesterday after upgrading at home I deleted a 23.01 line showing “32k” and it freed about 200 MB space per the dashboard widget. Don’t delete the one you’re using…I would hope it stops you.
-
@SteveITS said in /var is low on disk space:
after upgrading at home I deleted a 23.01 line showing “32k” and it freed about 200 MB space per the dashboard widget.
Thanks! That helped. It's like taking off my socks and I can wiggle my toes again
I have not gotten to reinstalling pfBlockerNG (maybe it's not a problem).
I removed a 23.01 from March (152k) and 22.05 from Feb (24k) from the environment.
It removed all my usage warnings and I my low disk space issue is solved.Now to see what happens with my mobile devices over the next few days.
-
@JonH Nice. Does the dashboard still show /var at 51 MB?
-
@SteveITS 975MB, nearly 20x increase. No more red bars
-
@JonH We do recommend checking the eMMC health in our devices over time to make sure you do not exhaust the writes allowed on the storage before you've had a chance to add an SSD. Waiting until the eMMC has failed, on this model, may result in a system that cannot be booted at all when you get an SSD.
https://docs.netgate.com/pfsense/en/latest/troubleshooting/disk-lifetime.html#emmc
-
@rcoleman-netgate said in /var is low on disk space:
We do recommend checking the eMMC health in our devices
Thanks so much for this reminder. I looked at s.m.a.r.t. status but I guess it applies to scsi drive types.
I'll check out the link.
-
@JonH interesting. That implies the issue could be 1) completely hidden by a large RAM disk, and 2) the RAM disk might not be able to copy 200MB of data to a “51MB” partition in the background.
Perhaps the self-cleaning of old BEs should happen sooner if /var is under a certain amount of total space. Or available space.
-
@rcoleman-netgate said in /var is low on disk space:
Waiting until the eMMC has failed, on this model, may result in a system that cannot be booted at all when you get an SSD
My results are:
eMMC Firmware Version: ?
eMMC Life Time Estimation A [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_A]: 0x0b
eMMC Life Time Estimation B [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_B]: 0x0b
eMMC Pre EOL information [EXT_CSD_PRE_EOL_INFO]: 0x01So 0x0b is "The disk has used 100%-110% of its estimated life time" and somewhat concerning.
and 0x01 is "The disk has consumed less than 80% of its reserved blocks" which is somewhat less concerning.At any rate, I was planning on Monday checking with old-store.netgate to inquire about the feasibility of installing an M.2 2242 in my 5100, mainly because I want to run ntopng without continually removing the data and also have more logging storage. This 5100 replaced a failed 2440 which had a 20GB (I think) SSD and I never had to worry about space.
Although I was aware that SDD had an eventual fail possibility I never thought it could be so soon. All mechanical drives had an eventual fail possibility and I've yet to have that happen in 40+ years of various systems. So I probably have been too shortsighted in this possibility. The test I just ran shows me that the 8GB with logging, especially the filter.log, probably shortens the life considerably. In fact, maybe that is what happened to my 2440 and my previous 5100 that lasted 3 yrs (2440) and only 18 mo (first 5100). They both simply do not boot nor provide an output via the serial cable. I always ran a lot of logging & ntopng.
Thanks again for this heads-up.
-
@JonH depends heavily on logging and packages, see
https://www.netgate.com/supported-pfsense-plus-packages
Edit: yeah ntopng, there you go, didn’t read down that far yet :)Also:
https://docs.netgate.com/pfsense/en/latest/solutions/sg-5100/m-2-sata-installation.html -
@SteveITS said in /var is low on disk space:
Perhaps the self-cleaning of old BEs should happen sooner
Indeed. I have been quite negligent in reviewing various changes and feature additions.
I did see that BEs existed but never gave it much thought in the past. Now I also see that my negligence has poured over to SDD lifetime.I'm in my mid-70's now and keeping up with tech has become quite difficult. And with the amount of security patches for various devices coming fast and furious of late I now see that it will be difficult if not impossible to keep a system running for my own expected lifetime without needing to upgrade the hardware (I mostly use an Apple Intel machine which is going to be unsupported in the very near future).
-
@JonH said in /var is low on disk space:
So 0x0b is "The disk has used 100%-110% of its estimated life time" and somewhat concerning.
and 0x01 is "The disk has consumed less than 80% of its reserved blocks" which is somewhat less concerning.Get an m.2 SSD ASAP.
Any mSATA m.2 SSD will work. https://docs.netgate.com/pfsense/en/latest/solutions/sg-5100/m-2-sata-installation.html
Do not wait. if the eMMC on this goes read-only the only method you have (that I have determined so far) is to use a re-soldering station to heat the eMMC and remove it from the board. it's messy. I haven't gotten a second 5100 with a read-only eMMC yet to see if there's another option to block the BIOS from seeing the chip.
-
-
@JonH said in /var is low on disk space:
@rcoleman-netgate said in /var is low on disk space:
Get an m.2 SSD ASAP.I strongly agree with that advice. My SG-5100 exhibited this same behavior when I upgraded to 23.01 back in March. I installed a compatible SSD within a couple days but unfortunately it was too late. The eMMC had completely failed at that point and prevented the unit from booting. My beloved SG-5100 was a boat anchor after only 20 months of use!
And the sad thing is, it really wasn't used that hard. Just a simple home environment with 40 or so client devices. The only write-heavy package installed was pfBlockerng.
I would highly recommend that you get a backup plan together in case your unit suffers the same fate. I had to scramble and go to Best Buy to get a POS consumer-grade Asus router just to get back online. Needless to say, the whole ordeal was pretty annoying an stressful. I work from home and this happened in the middle of a busy week. The wife wasn't too happy either...
Best of luck.
-
@azdeltawye said in /var is low on disk space:
I strongly agree with that advice. My SG-5100 exhibited this same behavior when I upgraded to 23.01 back in March. I installed a compatible SSD within a couple days but unfortunately it was too late. The eMMC had completely failed at that point and prevented the unit from booting.
Unfortunately for @azdeltawye, but (and I think I speak for both of us here) we hope to be fortune for others, this very same 5100 is the one I mentioned above where I extracted the eMMC from the board and it immediately started perform normally.
It's not a process I recommend everyone to do -- but for those who have the skills it can be done.
This is the eMMC on the board (the bottom side)
The board removed using a re-soldering station (a programmable heat gun)
The empty space (partially cleaned up)
The bottom of the eMMC.
-
@azdeltawye said in /var is low on disk space:
My beloved SG-5100 was a boat anchor after only 20 months of use!
Indeed. I got 16 mo's out of my 1st 5100, this one is going on 27 mo.
Same as you, my biggest write-hvy is pfBlockerNG. I am also simple home w/4 active devices but up to 30 because of smart devices.I was thinking of getting a SSD just to have more room, I had no idea how fragile the SSD was until now. I knew it was limited but didn't think it was this limited. A few minutes ago I cut my pfBlocker updates from 4x/day to once.
Thanks
-
@rcoleman-netgate said in /var is low on disk space:
use a re-soldering station to heat the eMMC and remove it from the board.
Referring to the later post with the photos of the removed eMMC can you clarify for me:
If an mSATA is installed it will supersede the eMMC, essentially making it electronically 'disappear'?But if it goes south then the mSATA won't do any good?
That is what I understand from what you posted, correct?
And the 32 GB SATAIII M.2 2242 SSD would be a suitable replacement?
As an aside, my 1st 5100 is still laying around, but if I read correctly, adding a mSATA probably won't help because if the eMMC is the cause then it's too late. Correct?
Is this likely to be a similar situation with my old 2440 w/ 20GB drive? Or is that a totally different situation? I know this may be impossible to answer, just fishing on the off chance I can get a working emergency spare. I'm ok with soldering but I don't have a re-soldering station, just a solder sucker and copper braid. And noticeably shakier hands than in the 'good old days'.
Neither of those units will POST and the only signs of life is the power light.
-
@JonH said in /var is low on disk space:
my biggest write-hvy is pfBlockerNG
Note NtopNG is marked "Requires SSD/HDD" in the linked package list above...
@rcoleman-netgate "...do not boot nor provide an output via the serial cable." Would that be similar for the 4860 model we discussed elsewhere? IIRC that doesn't have eMMC or at least can't use the "write life" software. Plus I'd expect the backup router to have way less disk writes.
