SG-1100 update from 23.09.1 to 24.03 keeps failing

[mrneutron]

@stephenw10
I deleted the big BE files and they are not coming back, so I don't think the system can be booting from those old BE files.
There are some 24k files that won't stay deleted. They come back upon a reboot.
It's like there's some kind of write protection in place right now??

After deleting the fig BE files, I think there's plenty of flash memory, available.

[23.09.1-RELEASE][admin@redacted]/root: zfs list
NAME                                                     USED  AVAIL  REFER  MOUNTPOINT
pfSense                                                 3.63G  2.91G    96K  none
pfSense/ROOT                                            2.96G  2.91G    96K  none
pfSense/ROOT/auto-default-20231106232605                  24K  2.91G   887M  /
pfSense/ROOT/auto-default-20231106232605/cf                0B  2.91G  1.09M  /cf
pfSense/ROOT/auto-default-20231106232605/var_cache_pkg     8K  2.91G  15.7M  /var/cache/pkg
pfSense/ROOT/auto-default-20231106232605/var_db_pkg        8K  2.91G  4.02M  /var/db/pkg
pfSense/ROOT/auto-default-20231212212909                  24K  2.91G  1.13G  /
pfSense/ROOT/auto-default-20231212212909/cf                0B  2.91G  2.68M  /cf
pfSense/ROOT/auto-default-20231212212909/var_cache_pkg     8K  2.91G   344M  /var/cache/pkg
pfSense/ROOT/auto-default-20231212212909/var_db_pkg        8K  2.91G  6.30M  /var/db/pkg
pfSense/ROOT/default                                    2.96G  2.91G  1.06G  /
pfSense/ROOT/default/cf                                 5.58M  2.91G  2.61M  /cf
pfSense/ROOT/default/var_cache_pkg                       535M  2.91G   190M  /var/cache/pkg
pfSense/ROOT/default/var_db_pkg                         14.4M  2.91G  6.31M  /var/db/pkg
pfSense/home                                             120K  2.91G   120K  /home
pfSense/reservation                                       96K  3.56G    96K  none
pfSense/tmp                                             1.30M  2.91G  1.30M  /tmp
pfSense/var                                             7.37M  2.91G  3.21M  /var
pfSense/var/cache                                         96K  2.91G    96K  /var/cache
pfSense/var/db                                          1.56M  2.91G  1.56M  /var/db
pfSense/var/log                                         2.40M  2.91G  2.40M  /var/log
pfSense/var/tmp                                          112K  2.91G   112K  /var/tmp

I've been doing all the advised procedures, and updateing is still not working.
After reading current posts about other failed 24.03 updates (also with Unlocking package pkg...failed), it sure seems that the update process is currently broken for the SG-1100! I wonder if version 23.09.1 has some flaw which prevents updating?

If the Netgate support will send me the giant 24.03 update package file to write to a USB flash drive, I'll take it!

[mrneutron]

@mrneutron
from this page:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/upgrades.html
I tried these recommended commands:
pkg-static clean -ay; pkg-static install -fy pkg pfSense-repo pfSense-upgrade

It removed a bunch of packages and "upated the updater", which sounds helpful...

pkg-static: Warning: Major OS version upgrade detected.  Running "pkg bootstrap -f" recommended
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
New version of pkg detected; it needs to be installed first.
The following 1 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
        pkg: 1.20.8_2 -> 1.20.9_1 [pfSense]

Number of packages to be upgraded: 1

...lots of lines of updating...

The following 3 package(s) will be affected (of 0 checked):

Installed packages to be UPGRADED:
        pfSense-repo: 23.09.1 -> 24.03 [pfSense]
        pfSense-upgrade: 1.2.1 -> 1.2.20 [pfSense]

Installed packages to be REINSTALLED:
        pkg-1.20.9_1 [pfSense]

I'll try a reboot and update again...

Nope...same failure!!!

[183/183] Fetching 7-zip-23.01.pkg: .......... done
Checking integrity... Unlocking package pkg...failed.
Failed

Of note, I discovered that every time the upgrade fails, the SG-1100 reboots.

[SteveITS]

@mrneutron said in SG-1100 update from 23.09.1 to 24.03 keeps failing:

There are some 24k files that won't stay deleted. They come back upon a reboot.

1. the size in the GUI is wildly wrong and always a few K...deleting will gain a gig, give or take.

2. is your disk read only/EOL? I seem to recall reading ZFS will "write" but can't actually save it to disk so writes disappear on a boot. Try creating or editing a file without upgrading, and restart.

https://docs.netgate.com/pfsense/en/latest/troubleshooting/disk-lifetime.html#emmc

[mrneutron]

@SteveITS

I just installed the mmc-utils and ran the recommended commands:

[23.09.1-RELEASE][admin@redacted]/: mmc extcsd read /dev/mmcsd0rpmb
=============================================
  Extended CSD rev 1.6 (MMC 4.5)
=============================================

Card Supported Command sets [S_CMD_SET: 0x01]
HPI Features [HPI_FEATURE: 0x01]: implementation based on CMD13
Background operations support [BKOPS_SUPPORT: 0x01]
Max Packet Read Cmd [MAX_PACKED_READS: 0x3f]
Max Packet Write Cmd [MAX_PACKED_WRITES: 0x3f]
Data TAG support [DATA_TAG_SUPPORT: 0x01]
Data TAG Unit Size [TAG_UNIT_SIZE: 0x03]
Tag Resources Size [TAG_RES_SIZE: 0x03]
Context Management Capabilities [CONTEXT_CAPABILITIES: 0x05]
Large Unit Size [LARGE_UNIT_SIZE_M1: 0x00]
Extended partition attribute support [EXT_SUPPORT: 0x00]
Generic CMD6 Timer [GENERIC_CMD6_TIME: 0x19]
Power off notification [POWER_OFF_LONG_TIME: 0x64]
Cache Size [CACHE_SIZE] is 0 KiB
Background operations status [BKOPS_STATUS: 0x00]
1st Initialisation Time after programmed sector [INI_TIMEOUT_AP: 0x0a]
Power class for 52MHz, DDR at 3.6V [PWR_CL_DDR_52_360: 0x22]
Power class for 52MHz, DDR at 1.95V [PWR_CL_DDR_52_195: 0x00]
Power class for 200MHz at 3.6V [PWR_CL_200_360: 0x00]
Power class for 200MHz, at 1.95V [PWR_CL_200_195: 0x00]
Minimum Performance for 8bit at 52MHz in DDR mode:
 [MIN_PERF_DDR_W_8_52: 0x00]
 [MIN_PERF_DDR_R_8_52: 0x00]
TRIM Multiplier [TRIM_MULT: 0x0e]
Secure Feature support [SEC_FEATURE_SUPPORT: 0x55]
Boot Information [BOOT_INFO: 0x07]
 Device supports alternative boot method
 Device supports dual data rate during boot
 Device supports high speed timing during boot
Boot partition size [BOOT_SIZE_MULTI: 0x10]
Access size [ACC_SIZE: 0x08]
High-capacity erase unit size [HC_ERASE_GRP_SIZE: 0x04]
 i.e. 2048 KiB
High-capacity erase timeout [ERASE_TIMEOUT_MULT: 0x0e]
Reliable write sector count [REL_WR_SEC_C: 0x01]
High-capacity W protect group size [HC_WP_GRP_SIZE: 0x08]
 i.e. 16384 KiB
Sleep current (VCC) [S_C_VCC: 0x08]
Sleep current (VCCQ) [S_C_VCCQ: 0x07]
Sleep/awake timeout [S_A_TIMEOUT: 0x13]
Sector Count [SEC_COUNT: 0x00e90000]
 Device is block-addressed
Minimum Write Performance for 8bit:
 [MIN_PERF_W_8_52: 0x0a]
 [MIN_PERF_R_8_52: 0x0a]
 [MIN_PERF_W_8_26_4_52: 0x0a]
 [MIN_PERF_R_8_26_4_52: 0x0a]
Minimum Write Performance for 4bit:
 [MIN_PERF_W_4_26: 0x0a]
 [MIN_PERF_R_4_26: 0x0a]
Power classes registers:
 [PWR_CL_26_360: 0x22]
 [PWR_CL_52_360: 0x22]
 [PWR_CL_26_195: 0x00]
 [PWR_CL_52_195: 0x00]
Partition switching timing [PARTITION_SWITCH_TIME: 0x03]
Out-of-interrupt busy timing [OUT_OF_INTERRUPT_TIME: 0x19]
I/O Driver Strength [DRIVER_STRENGTH: 0x01]
Card Type [CARD_TYPE: 0x07]
 HS Dual Data Rate eMMC @52MHz 1.8V or 3VI/O
 HS eMMC @52MHz - at rated device voltage(s)
 HS eMMC @26MHz - at rated device voltage(s)
CSD structure version [CSD_STRUCTURE: 0x02]
Command set [CMD_SET: 0x00]
Command set revision [CMD_SET_REV: 0x00]
Power class [POWER_CLASS: 0x02]
High-speed interface timing [HS_TIMING: 0x01]
Erased memory content [ERASED_MEM_CONT: 0x00]
Boot configuration bytes [PARTITION_CONFIG: 0x03]
 Not boot enable
 R/W Replay Protected Memory Block (RPMB)
Boot config protection [BOOT_CONFIG_PROT: 0x00]
Boot bus Conditions [BOOT_BUS_CONDITIONS: 0x00]
High-density erase group definition [ERASE_GROUP_DEF: 0x01]
Boot write protection status registers [BOOT_WP_STATUS]: 0x00
Boot Area Write protection [BOOT_WP]: 0x00
 Power ro locking: possible
 Permanent ro locking: possible
 partition 0 ro lock status: not locked
 partition 1 ro lock status: not locked
User area write protection register [USER_WP]: 0x00
FW configuration [FW_CONFIG]: 0x00
RPMB Size [RPMB_SIZE_MULT]: 0x10
Write reliability setting register [WR_REL_SET]: 0x1f
 user area: the device protects existing data if a power failure occurs during a write operation
 partition 1: the device protects existing data if a power failure occurs during a write operation
 partition 2: the device protects existing data if a power failure occurs during a write operation
 partition 3: the device protects existing data if a power failure occurs during a write operation
 partition 4: the device protects existing data if a power failure occurs during a write operation
Write reliability parameter register [WR_REL_PARAM]: 0x05
 Device supports writing EXT_CSD_WR_REL_SET
 Device supports the enhanced def. of reliable write
Enable background operations handshake [BKOPS_EN]: 0x00
H/W reset function [RST_N_FUNCTION]: 0x00
HPI management [HPI_MGMT]: 0x00
Partitioning Support [PARTITIONING_SUPPORT]: 0x01
 Device support partitioning feature
 Device cannot have enhanced tech.
Max Enhanced Area Size [MAX_ENH_SIZE_MULT]: 0x0000e9
 i.e. 3817472 KiB
Partitions attribute [PARTITIONS_ATTRIBUTE]: 0x00
Partitioning Setting [PARTITION_SETTING_COMPLETED]: 0x00
 Device partition setting NOT complete
General Purpose Partition Size
 [GP_SIZE_MULT_4]: 0x000000
 [GP_SIZE_MULT_3]: 0x000000
 [GP_SIZE_MULT_2]: 0x000000
 [GP_SIZE_MULT_1]: 0x000000
Enhanced User Data Area Size [ENH_SIZE_MULT]: 0x000000
 i.e. 0 KiB
Enhanced User Data Start Address [ENH_START_ADDR]: 0x00000000
 i.e. 0 bytes offset
Bad Block Management mode [SEC_BAD_BLK_MGMNT]: 0x00
Periodic Wake-up [PERIODIC_WAKEUP]: 0x00
Program CID/CSD in DDR mode support [PROGRAM_CID_CSD_DDR_SUPPORT]: 0x01
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[127]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[126]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[125]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[124]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[123]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[122]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[121]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[120]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[119]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[118]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[117]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[116]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[115]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[114]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[113]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[112]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[111]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[110]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[109]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[108]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[107]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[106]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[105]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[104]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[103]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[102]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[101]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[100]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[99]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[98]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[97]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[96]]: 0x03
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[95]]: 0x02
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[94]]: 0x0b
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[93]]: 0x04
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[92]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[91]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[90]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[89]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[88]]: 0x0b
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[87]]: 0x0b
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[86]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[85]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[84]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[83]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[82]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[81]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[80]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[79]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[78]]: 0x72
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[77]]: 0x35
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[76]]: 0x35
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[75]]: 0x69
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[74]]: 0x32
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[73]]: 0x32
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[72]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[71]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[70]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[69]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[68]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[67]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[66]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[65]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[64]]: 0x00
Native sector size [NATIVE_SECTOR_SIZE]: 0x00
Sector size emulation [USE_NATIVE_SECTOR]: 0x00
Sector size [DATA_SECTOR_SIZE]: 0x00
1st initialization after disabling sector size emulation [INI_TIMEOUT_EMU]: 0x00
Class 6 commands control [CLASS_6_CTRL]: 0x00
Number of addressed group to be Released[DYNCAP_NEEDED]: 0x00
Exception events control [EXCEPTION_EVENTS_CTRL]: 0x0000
Exception events status[EXCEPTION_EVENTS_STATUS]: 0x0000
Extended Partitions Attribute [EXT_PARTITIONS_ATTRIBUTE]: 0x0000
Context configuration [CONTEXT_CONF[51]]: 0x00
Context configuration [CONTEXT_CONF[50]]: 0x00
Context configuration [CONTEXT_CONF[49]]: 0x00
Context configuration [CONTEXT_CONF[48]]: 0x00
Context configuration [CONTEXT_CONF[47]]: 0x00
Context configuration [CONTEXT_CONF[46]]: 0x00
Context configuration [CONTEXT_CONF[45]]: 0x00
Context configuration [CONTEXT_CONF[44]]: 0x00
Context configuration [CONTEXT_CONF[43]]: 0x00
Context configuration [CONTEXT_CONF[42]]: 0x00
Context configuration [CONTEXT_CONF[41]]: 0x00
Context configuration [CONTEXT_CONF[40]]: 0x00
Context configuration [CONTEXT_CONF[39]]: 0x00
Context configuration [CONTEXT_CONF[38]]: 0x00
Context configuration [CONTEXT_CONF[37]]: 0x00
Packed command status [PACKED_COMMAND_STATUS]: 0x00
Packed command failure index [PACKED_FAILURE_INDEX]: 0x00
Power Off Notification [POWER_OFF_NOTIFICATION]: 0x00
Control to turn the Cache ON/OFF [CACHE_CTRL]: 0x00
Control to turn the Cache Barrier ON/OFF [BARRIER_CTRL]: 0x00

Those big BE files reside in the root directory right? (mountpoint /)
They aren't there any more.

[23.09.1-RELEASE][admin@redacted]/: ls -la
total 166
drwxr-xr-x  21 root wheel   27 May  4 11:18 .
drwxr-xr-x  21 root wheel   27 May  4 11:18 ..
-rw-r--r--   2 root wheel 1011 Dec  6 15:20 .cshrc
-rw-r--r--   2 root wheel 1199 Apr  6 05:54 .profile
-rw-------   1 root wheel 1024 Feb 10  2023 .rnd
-r--r--r--   1 root wheel 6109 Dec  6 15:20 COPYRIGHT
drwxr-xr-x   2 root wheel   48 Dec 12 21:32 bin
drwxr-xr-x  16 root wheel   50 May  4 11:18 boot
-rw-r--r--   1 root wheel    9 May  4 11:18 boot.config
drwxr-xr-x   3 root wheel    3 Feb 10  2023 cf
lrwxr-xr-x   1 root wheel    8 Dec 31  1969 conf -> /cf/conf
drwxr-xr-x   2 root wheel    3 Dec 12 21:33 conf.default
dr-xr-xr-x  10 root wheel  512 May  4 11:17 dev
drwxr-xr-x  34 root wheel  193 May  4 11:18 etc
drwxr-xr-x   4 root wheel    4 Feb 10  2023 home
drwxr-xr-x   5 root wheel   81 Dec 12 21:32 lib
drwxr-xr-x   3 root wheel    4 Dec 12 21:32 libexec
drwxr-xr-x   2 root wheel    2 Feb 10  2023 media
drwxr-xr-x   2 root wheel    2 Feb 10  2023 mnt
drwxr-xr-x   2 root wheel    2 Dec  6 15:19 net
dr-xr-xr-x   2 root wheel    2 Feb 10  2023 proc
drwxr-xr-x   2 root wheel  145 Dec 12 21:31 rescue
drwxr-xr-x   2 root wheel   10 Mar  3 13:16 root
drwxr-xr-x   2 root wheel  137 Dec 12 21:32 sbin
drwxrwxrwt   6 root wheel   49 May  4 11:37 tmp
drwxr-xr-x  13 root wheel   13 Dec  6 15:23 usr
drwxr-xr-x  27 root wheel   27 Dec  6 15:19 var

[23.09.1-RELEASE][admin@redacted]/: df
Filesystem                         1K-blocks    Used   Avail Capacity  Mounted on
pfSense/ROOT/default                 4158325 1107404 3050921    27%    /
devfs                                      1       0       1     0%    /dev
pfSense/tmp                          3051249     328 3050921     0%    /tmp
pfSense/home                         3051041     120 3050921     0%    /home
pfSense/var                          3054213    3292 3050921     0%    /var
pfSense/var/log                      3053453    2532 3050921     0%    /var/log
pfSense/var/cache                    3051017      96 3050921     0%    /var/cache
pfSense/var/db                       3052237    1316 3050921     0%    /var/db
pfSense/var/tmp                      3051033     112 3050921     0%    /var/tmp
pfSense/ROOT/default/cf              3053609    2688 3050921     0%    /cf
pfSense/ROOT/default/var_cache_pkg   3245633  194712 3050921     6%    /var/cache/pkg
pfSense/ROOT/default/var_db_pkg      3057369    6448 3050921     0%    /var/db/pkg
tmpfs                                   4096     116    3980     3%    /var/run
devfs                                      1       0       1     0%    /var/dhcpd/dev

[SteveITS]

@mrneutron

https://docs.netgate.com/pfsense/en/latest/troubleshooting/disk-lifetime.html#emmc

:)

[mrneutron]

@SteveITS said in SG-1100 update from 23.09.1 to 24.03 keeps failing:

@mrneutron

https://docs.netgate.com/pfsense/en/latest/troubleshooting/disk-lifetime.html#emmc

:)

I think you posted before you saw my edits in the above message.

[stephenw10]

You look to be missing the last lines of the output where the Life Time is shown like:

Packed command status [PACKED_COMMAND_STATUS]: 0x00
Packed command failure index [PACKED_FAILURE_INDEX]: 0x00
Power Off Notification [POWER_OFF_NOTIFICATION]: 0x00
Control to turn the Cache ON/OFF [CACHE_CTRL]: 0x01
Control to turn the Cache Barrier ON/OFF [BARRIER_CTRL]: 0x00
eMMC Firmware Version: 09111317
eMMC Life Time Estimation A [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_A]: 0x02
eMMC Life Time Estimation B [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_B]: 0x05
eMMC Pre EOL information [EXT_CSD_PRE_EOL_INFO]: 0x01
Secure Removal Type [SECURE_REMOVAL_TYPE]: 0x08
 information is configured to be removed by an erase of the physical memory
 Supported Secure Removal Type:
  information removed using a vendor defined
Command Queue Support [CMDQ_SUPPORT]: 0x01
Command Queue Depth [CMDQ_DEPTH]: 32
Command Enabled [CMDQ_MODE_EN]: 0x00

[mrneutron]

@stephenw10 said in SG-1100 update from 23.09.1 to 24.03 keeps failing:

You look to be missing the last lines of the output where the Life Time is shown like:

I posted all the output there was from the "mmc extcsd read /dev/mmcsd0rpmb" command.

The text on the page
https://docs.netgate.com/pfsense/en/latest/troubleshooting/disk-lifetime.html#emmc
specifically says: "Not all disks support all of these fields."
It would appear the flash memory on the board is one of those.

[stephenw10]

Yup you have an early 1100 there. Later devices show:

[24.03-RELEASE][admin@1100-3.stevew.lan]/root: mmc extcsd read /dev/mmcsd0rpmb
=============================================
  Extended CSD rev 1.8 (MMC 5.1)
=============================================

I would try a clean install. If the eMMC really has failed the initial part of that will also fail to clear the drive.

[domnado]

I finally got my SG-1100 to update. I had to serial console in and clear some disk space as it was failing with only three packages remaining due to the disk being full. I reset the log files, pkg cache and removed '/boot/kernel.old'.

However now that I'm on 24.03, on the status page I am now seeing an extremely full disk. says its 78% full, 1.1G used out of 1.3G. Whereas before the update it was .9G used out of 2G.

df -h

Filesystem                            Size    Used   Avail Capacity  Mounted on
pfSense/ROOT/default                  1.3G    1.1G    302M    78%    /
devfs                                 1.0K      0B    1.0K     0%    /dev
pfSense/var                           313M     11M    302M     3%    /var
pfSense/tmp                           302M    240K    302M     0%    /tmp
pfSense/home                          302M     96K    302M     0%    /home
pfSense/var/db                        305M    2.8M    302M     1%    /var/db
pfSense/var/log                       305M    3.3M    302M     1%    /var/log
pfSense/var/cache                     302M    104K    302M     0%    /var/cache
pfSense/var/tmp                       302M    112K    302M     0%    /var/tmp
pfSense/ROOT/default/cf               305M    2.5M    302M     1%    /cf
pfSense/ROOT/default/var_cache_pkg    302M    240K    302M     0%    /var/cache/pkg
pfSense/ROOT/default/var_db_pkg       311M    9.0M    302M     3%    /var/db/pkg
tmpfs                                 4.0M    116K    3.9M     3%    /var/run

du -d1 -h /

2.4M    ./cf
4.0K    ./dev
512B    ./proc
 33K    ./root
 25M    ./var
 16M    ./rescue
 93K    ./tmp
5.0K    ./conf.default
512B    ./mnt
104M    ./boot
113K    ./libexec
 12M    ./lib
2.9M    ./sbin
512B    ./media
941M    ./usr
873K    ./bin
512B    ./home
512B    ./net
3.6M    ./etc
1.1G    .

mmc extcsd read /dev/mmcsd0rpmb | egrep 'LIFE|EOL'

eMMC Life Time Estimation A [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_A]: 0x06
eMMC Life Time Estimation B [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_B]: 0x0b
eMMC Pre EOL information [EXT_CSD_PRE_EOL_INFO]: 0x01

[stephenw10]

And there are no BE snapshots you can remove?

That 0x0bvalue for MLC estimation isn't great.

[domnado]

@stephenw10
Yeah I didn't like reading that value either. I will check on the BE snapshots.

EDIT:
After checking the Boot Environments I remember looking at the five entries other than 24.03 and thinking "32K, that's too small to worry about". But removing just one of them gained me about 700M of space, so yeah that was my disk full issue

[stephenw10]

Yes that value is misleading. We hope to replace that with the actual drive space the BE uses.

[mrneutron]

@stephenw10 said in SG-1100 update from 23.09.1 to 24.03 keeps failing:

Yup you have an early 1100 there. Later devices show:

[24.03-RELEASE][admin@1100-3.stevew.lan]/root: mmc extcsd read /dev/mmcsd0rpmb
=============================================
  Extended CSD rev 1.8 (MMC 5.1)
=============================================

I would try a clean install. If the eMMC really has failed the initial part of that will also fail to clear the drive.

@stephenw10
I bought 2 SG-1100s in fall 2019.

I'm going to try a reload of ver. 24.03 with the Network Installer USB stick tonight.
Netgate Support told me I can use a GPT partition with ZFS in an SG-1100.

If my emmc memory is failing, is there a replacement/repair option?
Perhaps a discount on a purchase of a new SG-1100?
Do the newer models of SG-1100 have an emmc memory problem?

[stephenw10]

The eMMC write life has not changed significantly as far as I know.

Minimum drive writes can be achieved by using UFS, disabling SWAP and enabling RAM disks.

[mrneutron]

@stephenw10 said in SG-1100 update from 23.09.1 to 24.03 keeps failing:

The eMMC write life has not changed significantly as far as I know.

Minimum drive writes can be achieved by using UFS, disabling SWAP and enabling RAM disks.

I've just used the SG-1100 as-is out of the box - no exotic configurations with the command line, etc.
It was MBR file system when it arrived in 2019.
ZFS file system was only possible with with 23.09 upgrade, as I recall.

If my emmc memory is failing, is there a replacement/repair option?
Perhaps a discount on a purchase of a new SG-1100?

[mrneutron]

OK. I tried to use the Online Network Installer USB drive image.
FAIL
It booted and asked if I wanted to install...yes.
It asked where to find the config.xml file.../config/config.xml on the USB drive.
It asked which interface to use to connect to Netgate Servers...WAN.
It asked which file system...ZFS.
It asked which partition system...GPT.
It asked to choose the ZFS Configuration...stripe (is the default, for a single disk).
It asked to select a disk to include in the ZFS configuration...mmcsd0 7.3G MMCHC
It started to partition....then "Installation aborted" (click OK)
"Failed to commit the file system changes...RESTART / EXIT"

I told it to RESTART and I tried again with ZFS and MBR...."Installation aborted" again.

[stephenw10]

If you choose to drop to the shell at that point you can review the install logs in /var/log and /tmp. That should tell you why it aborted.

There's no way to replace the eMMC on the 1100, it's on-board.

[mrneutron]

@stephenw10 said in SG-1100 update from 23.09.1 to 24.03 keeps failing:

If you choose to drop to the shell at that point you can review the install logs in /var/log and /tmp. That should tell you why it aborted.

There's no way to replace the eMMC on the 1100, it's on-board.

Which specific files will show the evidence? /var/log/system.log? /var/log/messages?

IF the emmc is failing on early SG-1100 units, does Netgate offer a discount on a purchase of a new SG-1100?

[SteveITS]

@mrneutron said in SG-1100 update from 23.09.1 to 24.03 keeps failing:

SG-1100s in fall 2019

That's 4.5 years ago... I think you're confusing "failing" with "reached end of write life" meaning "too much has been written to it." Do you have any of the "SSD" packages listed at https://www.netgate.com/supported-pfsense-plus-packages installed?

Solid state storage has a finite write life. eMMC is much shorter than SSD, and SSDs are far larger so can spread writes out more.