SG-1100 update from 23.09.1 to 24.03 keeps failing

SteveITS

@mrneutron

https://docs.netgate.com/pfsense/en/latest/troubleshooting/disk-lifetime.html#emmc

:)

mrneutron

@SteveITS said in SG-1100 update from 23.09.1 to 24.03 keeps failing:

@mrneutron

https://docs.netgate.com/pfsense/en/latest/troubleshooting/disk-lifetime.html#emmc

:)

I think you posted before you saw my edits in the above message.

stephenw10

You look to be missing the last lines of the output where the Life Time is shown like:

Packed command status [PACKED_COMMAND_STATUS]: 0x00
Packed command failure index [PACKED_FAILURE_INDEX]: 0x00
Power Off Notification [POWER_OFF_NOTIFICATION]: 0x00
Control to turn the Cache ON/OFF [CACHE_CTRL]: 0x01
Control to turn the Cache Barrier ON/OFF [BARRIER_CTRL]: 0x00
eMMC Firmware Version: 09111317
eMMC Life Time Estimation A [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_A]: 0x02
eMMC Life Time Estimation B [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_B]: 0x05
eMMC Pre EOL information [EXT_CSD_PRE_EOL_INFO]: 0x01
Secure Removal Type [SECURE_REMOVAL_TYPE]: 0x08
 information is configured to be removed by an erase of the physical memory
 Supported Secure Removal Type:
  information removed using a vendor defined
Command Queue Support [CMDQ_SUPPORT]: 0x01
Command Queue Depth [CMDQ_DEPTH]: 32
Command Enabled [CMDQ_MODE_EN]: 0x00

mrneutron

@stephenw10 said in SG-1100 update from 23.09.1 to 24.03 keeps failing:

You look to be missing the last lines of the output where the Life Time is shown like:

I posted all the output there was from the "mmc extcsd read /dev/mmcsd0rpmb" command.

The text on the page
https://docs.netgate.com/pfsense/en/latest/troubleshooting/disk-lifetime.html#emmc
specifically says: "Not all disks support all of these fields."
It would appear the flash memory on the board is one of those.

stephenw10

Yup you have an early 1100 there. Later devices show:

[24.03-RELEASE][admin@1100-3.stevew.lan]/root: mmc extcsd read /dev/mmcsd0rpmb
=============================================
  Extended CSD rev 1.8 (MMC 5.1)
=============================================

I would try a clean install. If the eMMC really has failed the initial part of that will also fail to clear the drive.

domnado

I finally got my SG-1100 to update. I had to serial console in and clear some disk space as it was failing with only three packages remaining due to the disk being full. I reset the log files, pkg cache and removed '/boot/kernel.old'.

However now that I'm on 24.03, on the status page I am now seeing an extremely full disk. says its 78% full, 1.1G used out of 1.3G. Whereas before the update it was .9G used out of 2G.

df -h

Filesystem                            Size    Used   Avail Capacity  Mounted on
pfSense/ROOT/default                  1.3G    1.1G    302M    78%    /
devfs                                 1.0K      0B    1.0K     0%    /dev
pfSense/var                           313M     11M    302M     3%    /var
pfSense/tmp                           302M    240K    302M     0%    /tmp
pfSense/home                          302M     96K    302M     0%    /home
pfSense/var/db                        305M    2.8M    302M     1%    /var/db
pfSense/var/log                       305M    3.3M    302M     1%    /var/log
pfSense/var/cache                     302M    104K    302M     0%    /var/cache
pfSense/var/tmp                       302M    112K    302M     0%    /var/tmp
pfSense/ROOT/default/cf               305M    2.5M    302M     1%    /cf
pfSense/ROOT/default/var_cache_pkg    302M    240K    302M     0%    /var/cache/pkg
pfSense/ROOT/default/var_db_pkg       311M    9.0M    302M     3%    /var/db/pkg
tmpfs                                 4.0M    116K    3.9M     3%    /var/run

du -d1 -h /

2.4M    ./cf
4.0K    ./dev
512B    ./proc
 33K    ./root
 25M    ./var
 16M    ./rescue
 93K    ./tmp
5.0K    ./conf.default
512B    ./mnt
104M    ./boot
113K    ./libexec
 12M    ./lib
2.9M    ./sbin
512B    ./media
941M    ./usr
873K    ./bin
512B    ./home
512B    ./net
3.6M    ./etc
1.1G    .

mmc extcsd read /dev/mmcsd0rpmb | egrep 'LIFE|EOL'

eMMC Life Time Estimation A [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_A]: 0x06
eMMC Life Time Estimation B [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_B]: 0x0b
eMMC Pre EOL information [EXT_CSD_PRE_EOL_INFO]: 0x01

stephenw10

And there are no BE snapshots you can remove?

That 0x0bvalue for MLC estimation isn't great.

domnado

@stephenw10
Yeah I didn't like reading that value either. I will check on the BE snapshots.

EDIT:
After checking the Boot Environments I remember looking at the five entries other than 24.03 and thinking "32K, that's too small to worry about". But removing just one of them gained me about 700M of space, so yeah that was my disk full issue

stephenw10

Yes that value is misleading. We hope to replace that with the actual drive space the BE uses.

mrneutron

@stephenw10 said in SG-1100 update from 23.09.1 to 24.03 keeps failing:

Yup you have an early 1100 there. Later devices show:
[24.03-RELEASE][admin@1100-3.stevew.lan]/root: mmc extcsd read /dev/mmcsd0rpmb
=============================================
  Extended CSD rev 1.8 (MMC 5.1)
=============================================
I would try a clean install. If the eMMC really has failed the initial part of that will also fail to clear the drive.

@stephenw10
I bought 2 SG-1100s in fall 2019.

I'm going to try a reload of ver. 24.03 with the Network Installer USB stick tonight.
Netgate Support told me I can use a GPT partition with ZFS in an SG-1100.

If my emmc memory is failing, is there a replacement/repair option?
Perhaps a discount on a purchase of a new SG-1100?
Do the newer models of SG-1100 have an emmc memory problem?

stephenw10

The eMMC write life has not changed significantly as far as I know.

Minimum drive writes can be achieved by using UFS, disabling SWAP and enabling RAM disks.

mrneutron

@stephenw10 said in SG-1100 update from 23.09.1 to 24.03 keeps failing:

The eMMC write life has not changed significantly as far as I know.

Minimum drive writes can be achieved by using UFS, disabling SWAP and enabling RAM disks.

I've just used the SG-1100 as-is out of the box - no exotic configurations with the command line, etc.
It was MBR file system when it arrived in 2019.
ZFS file system was only possible with with 23.09 upgrade, as I recall.

If my emmc memory is failing, is there a replacement/repair option?
Perhaps a discount on a purchase of a new SG-1100?

mrneutron

OK. I tried to use the Online Network Installer USB drive image.
FAIL
It booted and asked if I wanted to install...yes.
It asked where to find the config.xml file.../config/config.xml on the USB drive.
It asked which interface to use to connect to Netgate Servers...WAN.
It asked which file system...ZFS.
It asked which partition system...GPT.
It asked to choose the ZFS Configuration...stripe (is the default, for a single disk).
It asked to select a disk to include in the ZFS configuration...mmcsd0 7.3G MMCHC
It started to partition....then "Installation aborted" (click OK)
"Failed to commit the file system changes...RESTART / EXIT"

I told it to RESTART and I tried again with ZFS and MBR...."Installation aborted" again.

PXL_20240508_013706798 (Custom).jpg PXL_20240508_013743939 (Custom).jpg PXL_20240508_013807623 (Custom).jpg PXL_20240508_014040114 (Custom).jpg PXL_20240508_013954488 (Custom).jpg

stephenw10

If you choose to drop to the shell at that point you can review the install logs in /var/log and /tmp. That should tell you why it aborted.

There's no way to replace the eMMC on the 1100, it's on-board.

mrneutron

@stephenw10 said in SG-1100 update from 23.09.1 to 24.03 keeps failing:

If you choose to drop to the shell at that point you can review the install logs in /var/log and /tmp. That should tell you why it aborted.

There's no way to replace the eMMC on the 1100, it's on-board.

Which specific files will show the evidence? /var/log/system.log? /var/log/messages?

IF the emmc is failing on early SG-1100 units, does Netgate offer a discount on a purchase of a new SG-1100?

SteveITS

@mrneutron said in SG-1100 update from 23.09.1 to 24.03 keeps failing:

SG-1100s in fall 2019

That's 4.5 years ago... I think you're confusing "failing" with "reached end of write life" meaning "too much has been written to it." Do you have any of the "SSD" packages listed at https://www.netgate.com/supported-pfsense-plus-packages installed?

Solid state storage has a finite write life. eMMC is much shorter than SSD, and SSDs are far larger so can spread writes out more.

mrneutron

@SteveITS said in SG-1100 update from 23.09.1 to 24.03 keeps failing:

@mrneutron said in SG-1100 update from 23.09.1 to 24.03 keeps failing:

SG-1100s in fall 2019

That's 4.5 years ago...

So, are you saying that Netgate hardware cannot be expected to last more than 4 years?

Do you have any of the "SSD" packages listed at https://www.netgate.com/supported-pfsense-plus-packages installed?

Solid state storage has a finite write life. eMMC is much shorter than SSD, and SSDs are far larger so can spread writes out more.

I have never loaded any SSD packages.
The SG-1100 does not have an SSD, or have an option to add an SSD.

SteveITS

@mrneutron said in SG-1100 update from 23.09.1 to 24.03 keeps failing:

SG-1100 does not have an SSD

I'm aware. I meant, a package like NtopNG or HAProxy or Snort/Suricata, as listed in the "Storage Requirements" column. I believe Squid used to be listed there while it was supported. I'm not trying to argue with you, just asking.

As for product life time, we've had clients with routers in service a long time. Disk life depends on what is written to disk. Everyone's use case varies. We typically turn off the logging of default block rules, and set a RAM disk, among other things. Some people like to log everything, and watch the dashboard all day which logs every web server request.

stephenw10

@mrneutron said in SG-1100 update from 23.09.1 to 24.03 keeps failing:

Which specific files will show the evidence?

root@pfSense-install:/tmp # ls /tmp
.ICE-unix               .font-unix              bsdinstall_log          recovered_config
.X11-unix               bsdinstall_boot         install-log.txt         tmp.GEB2CtRhSn
.XIM-unix               bsdinstall_etc          mnt_recovery

Check install-log.txt and bsdinstall_log from /tmp.

mrneutron

I just saw this thread on longevity of the emmc in the 1100.
https://forum.netgate.com/topic/170128/emmc-write-endurance

It looks like ZFS creates more disk writes than UFS.
But, we needed to move to ZFS in order to upgrade to the 2023 versions of firmware so we have ZFS now, and need to deal with it, or watch them get eaten alive.

It sure seems like Netgate needs to roll in some kind of endurance measures into the stock firmware (like a RAMDISK?) to prevent ZFS from wearing out the emmc up prematurely.

It looks like the SG-2100 has an option for an SSD, so maybe it would be able to avoid getting eaten alive by the ZFS and the OS?
"Storage: 8 GB eMMC Flash onboard (or optional 128 GB M.2 SATA 2242 SSD)"
https://shop.netgate.com/products/2100-max-pfsense