/var is low on disk space
-
@SteveITS 975MB, nearly 20x increase. No more red bars
-
@JonH We do recommend checking the eMMC health in our devices over time to make sure you do not exhaust the writes allowed on the storage before you've had a chance to add an SSD. Waiting until the eMMC has failed, on this model, may result in a system that cannot be booted at all when you get an SSD.
https://docs.netgate.com/pfsense/en/latest/troubleshooting/disk-lifetime.html#emmc
-
@rcoleman-netgate said in /var is low on disk space:
We do recommend checking the eMMC health in our devices
Thanks so much for this reminder. I looked at s.m.a.r.t. status but I guess it applies to scsi drive types.
I'll check out the link.
-
@JonH interesting. That implies the issue could be 1) completely hidden by a large RAM disk, and 2) the RAM disk might not be able to copy 200MB of data to a “51MB” partition in the background.
Perhaps the self-cleaning of old BEs should happen sooner if /var is under a certain amount of total space. Or available space.
-
@rcoleman-netgate said in /var is low on disk space:
Waiting until the eMMC has failed, on this model, may result in a system that cannot be booted at all when you get an SSD
My results are:
eMMC Firmware Version: ?
eMMC Life Time Estimation A [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_A]: 0x0b
eMMC Life Time Estimation B [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_B]: 0x0b
eMMC Pre EOL information [EXT_CSD_PRE_EOL_INFO]: 0x01So 0x0b is "The disk has used 100%-110% of its estimated life time" and somewhat concerning.
and 0x01 is "The disk has consumed less than 80% of its reserved blocks" which is somewhat less concerning.At any rate, I was planning on Monday checking with old-store.netgate to inquire about the feasibility of installing an M.2 2242 in my 5100, mainly because I want to run ntopng without continually removing the data and also have more logging storage. This 5100 replaced a failed 2440 which had a 20GB (I think) SSD and I never had to worry about space.
Although I was aware that SDD had an eventual fail possibility I never thought it could be so soon. All mechanical drives had an eventual fail possibility and I've yet to have that happen in 40+ years of various systems. So I probably have been too shortsighted in this possibility. The test I just ran shows me that the 8GB with logging, especially the filter.log, probably shortens the life considerably. In fact, maybe that is what happened to my 2440 and my previous 5100 that lasted 3 yrs (2440) and only 18 mo (first 5100). They both simply do not boot nor provide an output via the serial cable. I always ran a lot of logging & ntopng.
Thanks again for this heads-up.
-
@JonH depends heavily on logging and packages, see
https://www.netgate.com/supported-pfsense-plus-packages
Edit: yeah ntopng, there you go, didn’t read down that far yet :)Also:
https://docs.netgate.com/pfsense/en/latest/solutions/sg-5100/m-2-sata-installation.html -
@SteveITS said in /var is low on disk space:
Perhaps the self-cleaning of old BEs should happen sooner
Indeed. I have been quite negligent in reviewing various changes and feature additions.
I did see that BEs existed but never gave it much thought in the past. Now I also see that my negligence has poured over to SDD lifetime.I'm in my mid-70's now and keeping up with tech has become quite difficult. And with the amount of security patches for various devices coming fast and furious of late I now see that it will be difficult if not impossible to keep a system running for my own expected lifetime without needing to upgrade the hardware (I mostly use an Apple Intel machine which is going to be unsupported in the very near future).
-
@JonH said in /var is low on disk space:
So 0x0b is "The disk has used 100%-110% of its estimated life time" and somewhat concerning.
and 0x01 is "The disk has consumed less than 80% of its reserved blocks" which is somewhat less concerning.Get an m.2 SSD ASAP.
Any mSATA m.2 SSD will work. https://docs.netgate.com/pfsense/en/latest/solutions/sg-5100/m-2-sata-installation.html
Do not wait. if the eMMC on this goes read-only the only method you have (that I have determined so far) is to use a re-soldering station to heat the eMMC and remove it from the board. it's messy. I haven't gotten a second 5100 with a read-only eMMC yet to see if there's another option to block the BIOS from seeing the chip.
-
-
@JonH said in /var is low on disk space:
@rcoleman-netgate said in /var is low on disk space:
Get an m.2 SSD ASAP.I strongly agree with that advice. My SG-5100 exhibited this same behavior when I upgraded to 23.01 back in March. I installed a compatible SSD within a couple days but unfortunately it was too late. The eMMC had completely failed at that point and prevented the unit from booting. My beloved SG-5100 was a boat anchor after only 20 months of use!
And the sad thing is, it really wasn't used that hard. Just a simple home environment with 40 or so client devices. The only write-heavy package installed was pfBlockerng.
I would highly recommend that you get a backup plan together in case your unit suffers the same fate. I had to scramble and go to Best Buy to get a POS consumer-grade Asus router just to get back online. Needless to say, the whole ordeal was pretty annoying an stressful. I work from home and this happened in the middle of a busy week. The wife wasn't too happy either...
Best of luck.
-
@azdeltawye said in /var is low on disk space:
I strongly agree with that advice. My SG-5100 exhibited this same behavior when I upgraded to 23.01 back in March. I installed a compatible SSD within a couple days but unfortunately it was too late. The eMMC had completely failed at that point and prevented the unit from booting.
Unfortunately for @azdeltawye, but (and I think I speak for both of us here) we hope to be fortune for others, this very same 5100 is the one I mentioned above where I extracted the eMMC from the board and it immediately started perform normally.
It's not a process I recommend everyone to do -- but for those who have the skills it can be done.
This is the eMMC on the board (the bottom side)
The board removed using a re-soldering station (a programmable heat gun)
The empty space (partially cleaned up)
The bottom of the eMMC.
-
@azdeltawye said in /var is low on disk space:
My beloved SG-5100 was a boat anchor after only 20 months of use!
Indeed. I got 16 mo's out of my 1st 5100, this one is going on 27 mo.
Same as you, my biggest write-hvy is pfBlockerNG. I am also simple home w/4 active devices but up to 30 because of smart devices.I was thinking of getting a SSD just to have more room, I had no idea how fragile the SSD was until now. I knew it was limited but didn't think it was this limited. A few minutes ago I cut my pfBlocker updates from 4x/day to once.
Thanks
-
@rcoleman-netgate said in /var is low on disk space:
use a re-soldering station to heat the eMMC and remove it from the board.
Referring to the later post with the photos of the removed eMMC can you clarify for me:
If an mSATA is installed it will supersede the eMMC, essentially making it electronically 'disappear'?But if it goes south then the mSATA won't do any good?
That is what I understand from what you posted, correct?
And the 32 GB SATAIII M.2 2242 SSD would be a suitable replacement?
As an aside, my 1st 5100 is still laying around, but if I read correctly, adding a mSATA probably won't help because if the eMMC is the cause then it's too late. Correct?
Is this likely to be a similar situation with my old 2440 w/ 20GB drive? Or is that a totally different situation? I know this may be impossible to answer, just fishing on the off chance I can get a working emergency spare. I'm ok with soldering but I don't have a re-soldering station, just a solder sucker and copper braid. And noticeably shakier hands than in the 'good old days'.
Neither of those units will POST and the only signs of life is the power light.
-
@JonH said in /var is low on disk space:
my biggest write-hvy is pfBlockerNG
Note NtopNG is marked "Requires SSD/HDD" in the linked package list above...
@rcoleman-netgate "...do not boot nor provide an output via the serial cable." Would that be similar for the 4860 model we discussed elsewhere? IIRC that doesn't have eMMC or at least can't use the "write life" software. Plus I'd expect the backup router to have way less disk writes.
-
@SteveITS said in /var is low on disk space:
Note NtopNG is marked "Requires SSD/HDD" in the linked package list above...
When I purchased my first 5100 in Nov 2019, to replace my dead 2440 w/ 20gb drive, I asked about boosting the product above the 8gb because I did not feel it was enough. The sales person said 8 was enough and they didn't have any upgrades for 5100.
I don't know when the "marked "Requires SSD/HDD"" was posted but at that time I probably would not have seen it even if it was posted. I stopped using NtopNG about 6 mo's ago but would like to run it again.
That linked list also mentions DarkStat. I do run that.
Thanks for this info.
-
@rcoleman-netgate said in /var is low on disk space:
Unfortunately for @azdeltawye, but (and I think I speak for both of us here) we hope to be fortune for others, this very same 5100 is the one I mentioned above where I extracted the eMMC from the board and it immediately started perform normally.
I am glad that my old 5100 has a chance at a second life.
I'm happy to help the cause!! -
@JonH said in /var is low on disk space:
Referring to the later post with the photos of the removed eMMC can you clarify for me:
If an mSATA is installed it will supersede the eMMC, essentially making it electronically 'disappear'?No. The BIOS POST has a step we haven't been able to find a way around -- a read-only eMMC will cause the system to boot very slowly (15-30 minutes to POST) or not at all.
And the 32 GB SATAIII M.2 2242 SSD would be a suitable replacement?
Probably.
As an aside, my 1st 5100 is still laying around, but if I read correctly, adding a mSATA probably won't help because if the eMMC is the cause then it's too late. Correct?
If, yes. If this is the case, and you are in the United States, and you are interested in having it used for a testing/trial process LMK and I can open DMs for you.
Is this likely to be a similar situation with my old 2440 w/ 20GB drive? Or is that a totally different situation? I know this may be impossible to answer, just fishing on the off chance I can get a working emergency spare. I'm ok with soldering but I don't have a re-soldering station, just a solder sucker and copper braid. And noticeably shakier hands than in the 'good old days'.
The RED LED of death? That's a CPU issue on the C2000 systems. The 5100 is not a C2000.
-
@SteveITS said in /var is low on disk space:
@rcoleman-netgate "...do not boot nor provide an output via the serial cable." Would that be similar for the 4860 model we discussed elsewhere? IIRC that doesn't have eMMC or at least can't use the "write life" software. Plus I'd expect the backup router to have way less disk writes.
The red LED of death is a C2K CPU issue. In theory you could use the same technique to replace the CPU but ... I am not about to do that experiment.
-
@rcoleman-netgate Hmm, no LED for me, just fan noise. Just curious. Never mind. :)
-
@rcoleman-netgate said in /var is low on disk space:
In theory you could use the same technique to replace the CPU but ... I am not about to do that experiment.
Hahaha. After replacing my wife's iPad battery I swore I'd never dive into these things again. And I didn't have to use my soldering iron for that one.
thanks