Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    SG-1100 eMMC failing? Solaris: WARNING: Pool 'pfSense" has encountered an uncorrectable I/O failure [...]

    Scheduled Pinned Locked Moved Official Netgate® Hardware
    21 Posts 2 Posters 3.4k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C
      Cabledude
      last edited by Cabledude

      Here is the other topic I mentioned, where you and @jimp contributed. Jim eventually concluded that that other SG-1100 had a failing / failed eMMC. Could this unit be the same?

      @stephenw10 said in SG-1100 possible failed storage? Can I run it from USB?:

      Sometimes the USB controller in the 1100 just doesn't like a USB drive for some reason.

      I have used this USB thumb drive many times on several SG-1100's, also on this one. Could be gone bad, but the new one doesn't work either.

      Usually it will work in the other slot because one is USB2 and the other USB3.

      Check the boot logs after running 'usbrecovery' for the number of devices reported before the kernel loads. It should show 5 there. If it shows 8 that indicates it did not wipe the eMMC as expected.

      Steve

      I will see if I can find the number of devices 5 or 8

      Pete
      Home: SG-2100 + UniFi + Synology. SG-1100 retired
      Parents: SG-1100 + UniFi + Synology
      Testing: SG-1100 w/ 120GB SSD via ext USB (eMMC dead). Works great

      C 1 Reply Last reply Reply Quote 0
      • C
        Cabledude @Cabledude
        last edited by

        @stephenw10
        Do you mean this:
        0ec31ae3-0963-4411-af1d-f168f0b562c1-image.png

        Ok right after this I get a bunch of these, must be at least 50 lines, probably more:
        0ad3a75b-cbdf-42de-ac33-ed509c044f52-image.png

        Then this:
        9a226c6a-fd03-41be-b99f-695414d242f0-image.png

        Do these tell you anything?

        Pete
        Home: SG-2100 + UniFi + Synology. SG-1100 retired
        Parents: SG-1100 + UniFi + Synology
        Testing: SG-1100 w/ 120GB SSD via ext USB (eMMC dead). Works great

        1 Reply Last reply Reply Quote 0
        • stephenw10S
          stephenw10 Netgate Administrator
          last edited by

          Yeah, that error is in reading the USB drive, not the eMMC.

          But if it reports 8 devices after it shows it erased the eMMC that's almost certainly an eMMC issue.

          C 1 Reply Last reply Reply Quote 0
          • C
            Cabledude @stephenw10
            last edited by Cabledude

            @stephenw10
            I'm sorry I don't know what to look for exactly. I took a screen capture of the console output because once it restarts the log is gone. But I cannot post a video here I think. Could I send it to your personal account?

            Here is another part of that output that I was lucky to freeze as it flashy by extremely fast. Any clues here?

            b24da391-dfe0-403d-9e79-037db813ec6a-image.png

            Then the bad MBR sector warnings keep rolling and rolling and then this is the next output that appears:
            d2b2c2be-9e0d-456d-a29e-7c783071d8aa-image.png

            Pete
            Home: SG-2100 + UniFi + Synology. SG-1100 retired
            Parents: SG-1100 + UniFi + Synology
            Testing: SG-1100 w/ 120GB SSD via ext USB (eMMC dead). Works great

            1 Reply Last reply Reply Quote 0
            • jimpJ jimp moved this topic from Problems Installing or Upgrading pfSense Software on
            • stephenw10S
              stephenw10 Netgate Administrator
              last edited by

              No the number of detected devices is show very early in the boot just after it finishes erasing the eMMC after you run usbrecovcery:

              Marvell>> run usbrecovery
switch to partitions #0, OK
mmc1(part 0) is current device

MMC erase: dev # 1, block # 0, count 4194304 ... 4194304 blocks erased: OK
starting USB...
USB0:   Register 2000104 NbrPorts 2
Starting the controller
USB XHCI 1.00
USB1:   USB EHCI 1.00
scanning bus 0 for devices... 2 USB Device(s) found
scanning bus 1 for devices... 1 USB Device(s) found
       scanning usb for storage devices... 1 Storage Device(s) found
reading efi/boot/bootaa64.efi
393216 bytes read in 36 ms (10.4 MiB/s)
reading armada-3720-sg1100.dtb
13323 bytes read in 20 ms (650.4 KiB/s)
## Starting EFI application at 05000000 ...
Card did not respond to voltage select!
mmc_init: -95, time 41
Scanning disk sdhci@d8000.blk...
bad MBR sector signature 0x0000
Scanning disk usb_mass_storage.lun0...
Found 5 disks

              If, after erasing the eMMC, it still shows 8 disks there that's a problem.

              Steve

              C 1 Reply Last reply Reply Quote 0
              • C
                Cabledude @stephenw10
                last edited by

                @stephenw10
                Thank you for your sample output. The point is I don't get the same lines as you provide.

                After the MMC erase, you get:
                starting USB...
                I get:
                resetting USB...

                Then this part is the same:

                Starting the controller
USB XHCI 1.00
USB1:   USB EHCI 1.00
scanning bus 0 for devices... 2 USB Device(s) found
scanning bus 1 for devices... 1 USB Device(s) found
       scanning usb for storage devices... 1 Storage Device(s) found

                And then your output reads

                reading efi/boot/bootaa64.efi

                I don't see that at all. Instead, I get:
                ebd81205-d55f-443b-939f-1f3ed4e44bf8-image.png

                Then you get:

                ## Starting EFI application at 05000000 ...
Card did not respond to voltage select!
mmc_init: -95, time 41
Scanning disk sdhci@d8000.blk...
bad MBR sector signature 0x0000
Scanning disk usb_mass_storage.lun0...
Found 5 disks

                What I see is:
                bf9815ec-6340-4326-82b6-954b434d2d63-image.png

                Then after tens of "bad MBR sector signature 0x0000" it just proceeds directly to
                cb02420c-08e0-430c-a030-37ffcfffdeb6-image.png

                And it never gets to "Found 5 disks". Or any different number of disks. The entire line is not there.

                Pete
                Home: SG-2100 + UniFi + Synology. SG-1100 retired
                Parents: SG-1100 + UniFi + Synology
                Testing: SG-1100 w/ 120GB SSD via ext USB (eMMC dead). Works great

                C 1 Reply Last reply Reply Quote 0
                • C
                  Cabledude @Cabledude
                  last edited by

                  @stephenw10

                  I tried flashing the firmware 3 more times with different USB thumb drives. Eventually the third one passed right through until the end and appears normal. Above output is from that third try.

                  But the device still gets very hot, much hotter than my other SG-1100, which is running my network now, including pfBlocker IP and DNSBL blocklists.

                  I tried to read the eMMC health by following these Netgate instructions but it doesn't provide the

                  eMMC Life Time Estimation A [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_A]: 0x01
eMMC Life Time Estimation B [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_B]: 0x02

                  information. Instead I get:

                  Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[79]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[78]]: 0x72
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[77]]: 0x35
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[76]]: 0x35
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[75]]: 0x69
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[74]]: 0x32
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[73]]: 0x32
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[72]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[71]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[70]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[69]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[68]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[67]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[66]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[65]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[64]]: 0x00
Native sector size [NATIVE_SECTOR_SIZE]: 0x00
Sector size emulation [USE_NATIVE_SECTOR]: 0x00
Sector size [DATA_SECTOR_SIZE]: 0x00
1st initialization after disabling sector size emulation [INI_TIMEOUT_EMU]: 0x00
Class 6 commands control [CLASS_6_CTRL]: 0x00
Number of addressed group to be Released[DYNCAP_NEEDED]: 0x00
Exception events control [EXCEPTION_EVENTS_CTRL]: 0x0000
Exception events status[EXCEPTION_EVENTS_STATUS]: 0x0000
Extended Partitions Attribute [EXT_PARTITIONS_ATTRIBUTE]: 0x0000
Context configuration [CONTEXT_CONF[51]]: 0x00
Context configuration [CONTEXT_CONF[50]]: 0x00
Context configuration [CONTEXT_CONF[49]]: 0x00
Context configuration [CONTEXT_CONF[48]]: 0x00
Context configuration [CONTEXT_CONF[47]]: 0x00
Context configuration [CONTEXT_CONF[46]]: 0x00
Context configuration [CONTEXT_CONF[45]]: 0x00
Context configuration [CONTEXT_CONF[44]]: 0x00
Context configuration [CONTEXT_CONF[43]]: 0x00
Context configuration [CONTEXT_CONF[42]]: 0x00
Context configuration [CONTEXT_CONF[41]]: 0x00
Context configuration [CONTEXT_CONF[40]]: 0x00
Context configuration [CONTEXT_CONF[39]]: 0x00
Context configuration [CONTEXT_CONF[38]]: 0x00
Context configuration [CONTEXT_CONF[37]]: 0x00
Packed command status [PACKED_COMMAND_STATUS]: 0x00
Packed command failure index [PACKED_FAILURE_INDEX]: 0x00
Power Off Notification [POWER_OFF_NOTIFICATION]: 0x00
Control to turn the Cache ON/OFF [CACHE_CTRL]: 0x00
[23.05.1-RELEASE][root@pfSense-test-8sep23.home.arpa]/root: pkg install -y mmc-utils ; rehash

                  Can you make anything out of that? Or is this eMMC not supported?

                  Pete
                  Home: SG-2100 + UniFi + Synology. SG-1100 retired
                  Parents: SG-1100 + UniFi + Synology
                  Testing: SG-1100 w/ 120GB SSD via ext USB (eMMC dead). Works great

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by

                    That's only a fraction of the output, you should see:

                    [23.05.1-RELEASE][admin@1100-3.stevew.lan]/root: mmc extcsd read /dev/mmcsd0rpmb 
=============================================
  Extended CSD rev 1.8 (MMC 5.1)
=============================================

Card Supported Command sets [S_CMD_SET: 0x01]
HPI Features [HPI_FEATURE: 0x01]: implementation based on CMD13
Background operations support [BKOPS_SUPPORT: 0x01]
Max Packet Read Cmd [MAX_PACKED_READS: 0x3f]
Max Packet Write Cmd [MAX_PACKED_WRITES: 0x3f]
Data TAG support [DATA_TAG_SUPPORT: 0x01]
Data TAG Unit Size [TAG_UNIT_SIZE: 0x03]
Tag Resources Size [TAG_RES_SIZE: 0x03]
Context Management Capabilities [CONTEXT_CAPABILITIES: 0x05]
Large Unit Size [LARGE_UNIT_SIZE_M1: 0x00]
Extended partition attribute support [EXT_SUPPORT: 0x03]
Generic CMD6 Timer [GENERIC_CMD6_TIME: 0x19]
Power off notification [POWER_OFF_LONG_TIME: 0x19]
Cache Size [CACHE_SIZE] is 512 KiB
Background operations status [BKOPS_STATUS: 0x03]
1st Initialisation Time after programmed sector [INI_TIMEOUT_AP: 0x5a]
Power class for 52MHz, DDR at 3.6V [PWR_CL_DDR_52_360: 0x00]
Power class for 52MHz, DDR at 1.95V [PWR_CL_DDR_52_195: 0xdd]
Power class for 200MHz at 3.6V [PWR_CL_200_360: 0xdd]
Power class for 200MHz, at 1.95V [PWR_CL_200_195: 0x00]
Minimum Performance for 8bit at 52MHz in DDR mode:
 [MIN_PERF_DDR_W_8_52: 0x00]
 [MIN_PERF_DDR_R_8_52: 0x00]
TRIM Multiplier [TRIM_MULT: 0x03]
Secure Feature support [SEC_FEATURE_SUPPORT: 0x55]
Boot Information [BOOT_INFO: 0x07]
 Device supports alternative boot method
 Device supports dual data rate during boot
 Device supports high speed timing during boot
Boot partition size [BOOT_SIZE_MULTI: 0x20]
Access size [ACC_SIZE: 0x08]
High-capacity erase unit size [HC_ERASE_GRP_SIZE: 0x01]
 i.e. 512 KiB
High-capacity erase timeout [ERASE_TIMEOUT_MULT: 0x03]
Reliable write sector count [REL_WR_SEC_C: 0x01]
High-capacity W protect group size [HC_WP_GRP_SIZE: 0x10]
 i.e. 8192 KiB
Sleep current (VCC) [S_C_VCC: 0x05]
Sleep current (VCCQ) [S_C_VCCQ: 0x07]
Sleep/awake timeout [S_A_TIMEOUT: 0x12]
Sector Count [SEC_COUNT: 0x00e90e80]
 Device is block-addressed
Minimum Write Performance for 8bit:
 [MIN_PERF_W_8_52: 0x0a]
 [MIN_PERF_R_8_52: 0x0a]
 [MIN_PERF_W_8_26_4_52: 0x0a]
 [MIN_PERF_R_8_26_4_52: 0x0a]
Minimum Write Performance for 4bit:
 [MIN_PERF_W_4_26: 0x0a]
 [MIN_PERF_R_4_26: 0x0a]
Power classes registers:
 [PWR_CL_26_360: 0x00]
 [PWR_CL_52_360: 0x00]
 [PWR_CL_26_195: 0xdd]
 [PWR_CL_52_195: 0xdd]
Partition switching timing [PARTITION_SWITCH_TIME: 0x03]
Out-of-interrupt busy timing [OUT_OF_INTERRUPT_TIME: 0x0a]
I/O Driver Strength [DRIVER_STRENGTH: 0x1f]
Enhanced Strobe mode [STROBE_SUPPORT: 0x01]
Card Type [CARD_TYPE: 0x57]
 HS400 Dual Data Rate eMMC @200MHz 1.8VI/O
 HS200 Single Data Rate eMMC @200MHz 1.8VI/O
 HS Dual Data Rate eMMC @52MHz 1.8V or 3VI/O
 HS eMMC @52MHz - at rated device voltage(s)
 HS eMMC @26MHz - at rated device voltage(s)
CSD structure version [CSD_STRUCTURE: 0x02]
Command set [CMD_SET: 0x00]
Command set revision [CMD_SET_REV: 0x00]
Power class [POWER_CLASS: 0x0d]
High-speed interface timing [HS_TIMING: 0x01]
Erased memory content [ERASED_MEM_CONT: 0x00]
Boot configuration bytes [PARTITION_CONFIG: 0x03]
 Not boot enable
 R/W Replay Protected Memory Block (RPMB)
Boot config protection [BOOT_CONFIG_PROT: 0x00]
Boot bus Conditions [BOOT_BUS_CONDITIONS: 0x00]
High-density erase group definition [ERASE_GROUP_DEF: 0x01]
Boot write protection status registers [BOOT_WP_STATUS]: 0x00
Boot Area Write protection [BOOT_WP]: 0x00
 Power ro locking: possible
 Permanent ro locking: possible
 partition 0 ro lock status: not locked
 partition 1 ro lock status: not locked
User area write protection register [USER_WP]: 0x00
FW configuration [FW_CONFIG]: 0x00
RPMB Size [RPMB_SIZE_MULT]: 0x20
Write reliability setting register [WR_REL_SET]: 0x1f
 user area: the device protects existing data if a power failure occurs during a write operation
 partition 1: the device protects existing data if a power failure occurs during a write operation
 partition 2: the device protects existing data if a power failure occurs during a write operation
 partition 3: the device protects existing data if a power failure occurs during a write operation
 partition 4: the device protects existing data if a power failure occurs during a write operation
Write reliability parameter register [WR_REL_PARAM]: 0x15
 Device supports writing EXT_CSD_WR_REL_SET
 Device supports the enhanced def. of reliable write
Enable background operations handshake [BKOPS_EN]: 0x02
H/W reset function [RST_N_FUNCTION]: 0x00
HPI management [HPI_MGMT]: 0x00
Partitioning Support [PARTITIONING_SUPPORT]: 0x07
 Device support partitioning feature
 Device can have enhanced tech.
Max Enhanced Area Size [MAX_ENH_SIZE_MULT]: 0x0001b5
 i.e. 3579904 KiB
Partitions attribute [PARTITIONS_ATTRIBUTE]: 0x00
Partitioning Setting [PARTITION_SETTING_COMPLETED]: 0x00
 Device partition setting NOT complete
General Purpose Partition Size
 [GP_SIZE_MULT_4]: 0x000000
 [GP_SIZE_MULT_3]: 0x000000
 [GP_SIZE_MULT_2]: 0x000000
 [GP_SIZE_MULT_1]: 0x000000
Enhanced User Data Area Size [ENH_SIZE_MULT]: 0x000000
 i.e. 0 KiB
Enhanced User Data Start Address [ENH_START_ADDR]: 0x00000000
 i.e. 0 bytes offset
Bad Block Management mode [SEC_BAD_BLK_MGMNT]: 0x00
Periodic Wake-up [PERIODIC_WAKEUP]: 0x00
Program CID/CSD in DDR mode support [PROGRAM_CID_CSD_DDR_SUPPORT]: 0x01
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[127]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[126]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[125]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[124]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[123]]: 0x01
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[122]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[121]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[120]]: 0x01
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[119]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[118]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[117]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[116]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[115]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[114]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[113]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[112]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[111]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[110]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[109]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[108]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[107]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[106]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[105]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[104]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[103]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[102]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[101]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[100]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[99]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[98]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[97]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[96]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[95]]: 0x02
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[94]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[93]]: 0x01
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[92]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[91]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[90]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[89]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[88]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[87]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[86]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[85]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[84]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[83]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[82]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[81]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[80]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[79]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[78]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[77]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[76]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[75]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[74]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[73]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[72]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[71]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[70]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[69]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[68]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[67]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[66]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[65]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[64]]: 0x00
Native sector size [NATIVE_SECTOR_SIZE]: 0x00
Sector size emulation [USE_NATIVE_SECTOR]: 0x00
Sector size [DATA_SECTOR_SIZE]: 0x00
1st initialization after disabling sector size emulation [INI_TIMEOUT_EMU]: 0x0a
Class 6 commands control [CLASS_6_CTRL]: 0x00
Number of addressed group to be Released[DYNCAP_NEEDED]: 0x00
Exception events control [EXCEPTION_EVENTS_CTRL]: 0x0000
Exception events status[EXCEPTION_EVENTS_STATUS]: 0x0000
Extended Partitions Attribute [EXT_PARTITIONS_ATTRIBUTE]: 0x0000
Context configuration [CONTEXT_CONF[51]]: 0x00
Context configuration [CONTEXT_CONF[50]]: 0x00
Context configuration [CONTEXT_CONF[49]]: 0x00
Context configuration [CONTEXT_CONF[48]]: 0x00
Context configuration [CONTEXT_CONF[47]]: 0x00
Context configuration [CONTEXT_CONF[46]]: 0x00
Context configuration [CONTEXT_CONF[45]]: 0x00
Context configuration [CONTEXT_CONF[44]]: 0x00
Context configuration [CONTEXT_CONF[43]]: 0x00
Context configuration [CONTEXT_CONF[42]]: 0x00
Context configuration [CONTEXT_CONF[41]]: 0x00
Context configuration [CONTEXT_CONF[40]]: 0x00
Context configuration [CONTEXT_CONF[39]]: 0x00
Context configuration [CONTEXT_CONF[38]]: 0x00
Context configuration [CONTEXT_CONF[37]]: 0x00
Packed command status [PACKED_COMMAND_STATUS]: 0x00
Packed command failure index [PACKED_FAILURE_INDEX]: 0x00
Power Off Notification [POWER_OFF_NOTIFICATION]: 0x00
Control to turn the Cache ON/OFF [CACHE_CTRL]: 0x01
eMMC Firmware Version: 09111317
eMMC Life Time Estimation A [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_A]: 0x02
eMMC Life Time Estimation B [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_B]: 0x04
eMMC Pre EOL information [EXT_CSD_PRE_EOL_INFO]: 0x01
Secure Removal Type [SECURE_REMOVAL_TYPE]: 0x08
 information is configured to be removed by an erase of the physical memory
 Supported Secure Removal Type:
  information removed using a vendor defined
Command Queue Support [CMDQ_SUPPORT]: 0x01
Command Queue Depth [CMDQ_DEPTH]: 32
Command Enabled [CMDQ_MODE_EN]: 0x00
                    C 1 Reply Last reply Reply Quote 0
                    • C
                      Cabledude @stephenw10
                      last edited by Cabledude

                      @stephenw10
                      Yes I saw most of that when doing a screen recording, which is the only way for me to see what's flashing by because it goes too fast.

                      But my output stops at

                      Control to turn the Cache ON/OFF [CACHE_CTRL]: 0x00

                      ...so it does not show:

                      eMMC Firmware Version: 09111317
eMMC Life Time Estimation A [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_A]: 0x02
eMMC Life Time Estimation B [EXT_CSD_DEVICE_LIFE_TIME_EST_TYP_B]: 0x04
eMMC Pre EOL information [EXT_CSD_PRE_EOL_INFO]: 0x01

                      I notice you have the output in text. Is there an output file I can open?

                      Thanks,

                      Pete
                      Home: SG-2100 + UniFi + Synology. SG-1100 retired
                      Parents: SG-1100 + UniFi + Synology
                      Testing: SG-1100 w/ 120GB SSD via ext USB (eMMC dead). Works great

                      1 Reply Last reply Reply Quote 0
                      • C
                        Cabledude
                        last edited by Cabledude

                        Also tried the mmc command using other search strings:

                        [23.05.1-RELEASE][root@pfSense-test-8sep23.home.arpa]/root: mmc extcsd read /dev/mmcsd0rpmb | egrep 'PACKED'
Max Packet Read Cmd [MAX_PACKED_READS: 0x3f]
Max Packet Write Cmd [MAX_PACKED_WRITES: 0x3f]
Packed command status [PACKED_COMMAND_STATUS]: 0x00
Packed command failure index [PACKED_FAILURE_INDEX]: 0x00
[23.05.1-RELEASE][root@pfSense-test-8sep23.home.arpa]/root: mmc extcsd read /dev/mmcsd0rpmb | egrep 'Estimation'
[23.05.1-RELEASE][root@pfSense-test-8sep23.home.arpa]/root: mmc extcsd read /dev/mmcsd0rpmb | egrep 'EXT_CSD'
 Device supports writing EXT_CSD_WR_REL_SET
[23.05.1-RELEASE][root@pfSense-test-8sep23.home.arpa]/root: 

[23.05.1-RELEASE][root@pfSense-test-8sep23.home.arpa]/root: mmc extcsd read /dev/mmcsd0rpmb | egrep 'LIFE|EOL'
[23.05.1-RELEASE][root@pfSense-test-8sep23.home.arpa]/root:

                        The output simply doesn't contain the Life Time Estimation info.

                        Pete
                        Home: SG-2100 + UniFi + Synology. SG-1100 retired
                        Parents: SG-1100 + UniFi + Synology
                        Testing: SG-1100 w/ 120GB SSD via ext USB (eMMC dead). Works great

                        1 Reply Last reply Reply Quote 0
                        • stephenw10S
                          stephenw10 Netgate Administrator
                          last edited by

                          I just connected to it using SSH and copy and pasted it here. You can also scroll back through it and read exactly what is shown.

                          You can also copy/paste out of putty (or any terminal emulator) if you're uisng the console directly. Putty can also be set to log all output to a file directly if needed.

                          C 1 Reply Last reply Reply Quote 0
                          • C
                            Cabledude @stephenw10
                            last edited by

                            @stephenw10
                            Okay I haven’t tried SSH. But I’m on macOS, not windows. So I use terminal and it won’t let me select all + copy the entire output, just the last screen.

                            Bottom line is I don’t get the eMMC health output
                            And I’m still puzzled why my usbrecovery output is so much different from what you showed, not 5 or 8 devices etc.
                            And the heat and power draw also bothers me from a reliability point of view.

                            Would you think, based on all of the information and output I provided above, that this unit is okay, or otherwise?

                            Pete
                            Home: SG-2100 + UniFi + Synology. SG-1100 retired
                            Parents: SG-1100 + UniFi + Synology
                            Testing: SG-1100 w/ 120GB SSD via ext USB (eMMC dead). Works great

                            1 Reply Last reply Reply Quote 0
                            • stephenw10S
                              stephenw10 Netgate Administrator
                              last edited by

                              Mmm, try SSH. The fact you are seeing unexpected output seems suspicious. You might not be seeing everything or it might getting overwritten.

                              C 2 Replies Last reply Reply Quote 0
                              • C
                                Cabledude @stephenw10
                                last edited by Cabledude

                                @stephenw10
                                Connected via SSH now. Exact same output, so no health est info, but now I can indeed copy/paste all output which is progress.

                                Suspicious is not just the lack of eMMC health info, but also the other missing output during usb recovery.

                                I can’t re-flash the firmware using SSH. All I could try is find a windows machine somewhere at a friends office to compare putty output. Maybe tomorrow, it’s 2:30 am.

                                In my head I have written off this unit, from the high power usage and the extreme heat it produces alone. The reason I am still trying to estimate if the eMMC is bad, is that this circumstance might explain some or all of the issues I’ve been having with it. And by such conclusion may restore my confidence in netgate.

                                Maybe you could bear with me a little longer and help me troubleshoot the output issues so as to determine if this unit is good or bad. That would really help.

                                Pete
                                Home: SG-2100 + UniFi + Synology. SG-1100 retired
                                Parents: SG-1100 + UniFi + Synology
                                Testing: SG-1100 w/ 120GB SSD via ext USB (eMMC dead). Works great

                                1 Reply Last reply Reply Quote 0
                                • C
                                  Cabledude @stephenw10
                                  last edited by

                                  @stephenw10 said in SG-1100 eMMC failing? Solaris: WARNING: Pool 'pfSense" has encountered an uncorrectable I/O failure [...]:

                                  Mmm, try SSH. The fact you are seeing unexpected output seems suspicious. You might not be seeing everything or it might getting overwritten.

                                  I installed a "putty - equivalent" for macOS, the "Serial.app". Provides good output, can scroll back, copy, paste, etc. Here is the eMMC output, though it lacks the same essential parts as before:

                                  [23.05.1-RELEASE][root@pfSense-test-8sep23.home.arpa]/root: mmc extcsd read /dev/mmcsd0rpmb
=============================================
 Extended CSD rev 1.6 (MMC 4.5)
=============================================

Card Supported Command sets [S_CMD_SET: 0x01]
HPI Features [HPI_FEATURE: 0x01]: implementation based on CMD13
Background operations support [BKOPS_SUPPORT: 0x01]
Max Packet Read Cmd [MAX_PACKED_READS: 0x3f]
Max Packet Write Cmd [MAX_PACKED_WRITES: 0x3f]
Data TAG support [DATA_TAG_SUPPORT: 0x01]
Data TAG Unit Size [TAG_UNIT_SIZE: 0x03]
Tag Resources Size [TAG_RES_SIZE: 0x03]
Context Management Capabilities [CONTEXT_CAPABILITIES: 0x05]
Large Unit Size [LARGE_UNIT_SIZE_M1: 0x00]
Extended partition attribute support [EXT_SUPPORT: 0x00]
Generic CMD6 Timer [GENERIC_CMD6_TIME: 0x19]
Power off notification [POWER_OFF_LONG_TIME: 0x64]
Cache Size [CACHE_SIZE] is 0 KiB
Background operations status [BKOPS_STATUS: 0x00]
1st Initialisation Time after programmed sector [INI_TIMEOUT_AP: 0x0a]
Power class for 52MHz, DDR at 3.6V [PWR_CL_DDR_52_360: 0x22]
Power class for 52MHz, DDR at 1.95V [PWR_CL_DDR_52_195: 0x00]
Power class for 200MHz at 3.6V [PWR_CL_200_360: 0x00]
Power class for 200MHz, at 1.95V [PWR_CL_200_195: 0x00]
Minimum Performance for 8bit at 52MHz in DDR mode:
[MIN_PERF_DDR_W_8_52: 0x00]
[MIN_PERF_DDR_R_8_52: 0x00]
TRIM Multiplier [TRIM_MULT: 0x0e]
Secure Feature support [SEC_FEATURE_SUPPORT: 0x55]
Boot Information [BOOT_INFO: 0x07]
Device supports alternative boot method
Device supports dual data rate during boot
Device supports high speed timing during boot
Boot partition size [BOOT_SIZE_MULTI: 0x10]
Access size [ACC_SIZE: 0x08]
High-capacity erase unit size [HC_ERASE_GRP_SIZE: 0x04]
i.e. 2048 KiB
High-capacity erase timeout [ERASE_TIMEOUT_MULT: 0x0e]
Reliable write sector count [REL_WR_SEC_C: 0x01]
High-capacity W protect group size [HC_WP_GRP_SIZE: 0x08]
i.e. 16384 KiB
Sleep current (VCC) [S_C_VCC: 0x08]
Sleep current (VCCQ) [S_C_VCCQ: 0x07]
Sleep/awake timeout [S_A_TIMEOUT: 0x13]
Sector Count [SEC_COUNT: 0x00e90000]
Device is block-addressed
Minimum Write Performance for 8bit:
[MIN_PERF_W_8_52: 0x0a]
[MIN_PERF_R_8_52: 0x0a]
[MIN_PERF_W_8_26_4_52: 0x0a]
[MIN_PERF_R_8_26_4_52: 0x0a]
Minimum Write Performance for 4bit:
[MIN_PERF_W_4_26: 0x0a]
[MIN_PERF_R_4_26: 0x0a]
Power classes registers:
[PWR_CL_26_360: 0x22]
[PWR_CL_52_360: 0x22]
[PWR_CL_26_195: 0x00]
[PWR_CL_52_195: 0x00]
Partition switching timing [PARTITION_SWITCH_TIME: 0x03]
Out-of-interrupt busy timing [OUT_OF_INTERRUPT_TIME: 0x19]
I/O Driver Strength [DRIVER_STRENGTH: 0x01]
Enhanced Strobe mode [STROBE_SUPPORT: 0x00]
Card Type [CARD_TYPE: 0x07]
HS Dual Data Rate eMMC @52MHz 1.8V or 3VI/O
HS eMMC @52MHz - at rated device voltage(s)
HS eMMC @26MHz - at rated device voltage(s)
CSD structure version [CSD_STRUCTURE: 0x02]
Command set [CMD_SET: 0x00]
Command set revision [CMD_SET_REV: 0x00]
Power class [POWER_CLASS: 0x02]
High-speed interface timing [HS_TIMING: 0x01]
Erased memory content [ERASED_MEM_CONT: 0x00]
Boot configuration bytes [PARTITION_CONFIG: 0x03]
Not boot enable
R/W Replay Protected Memory Block (RPMB)
Boot config protection [BOOT_CONFIG_PROT: 0x00]
Boot bus Conditions [BOOT_BUS_CONDITIONS: 0x00]
High-density erase group definition [ERASE_GROUP_DEF: 0x01]
Boot write protection status registers [BOOT_WP_STATUS]: 0x00
Boot Area Write protection [BOOT_WP]: 0x00
Power ro locking: possible
Permanent ro locking: possible
partition 0 ro lock status: not locked
partition 1 ro lock status: not locked
User area write protection register [USER_WP]: 0x00
FW configuration [FW_CONFIG]: 0x00
RPMB Size [RPMB_SIZE_MULT]: 0x10
Write reliability setting register [WR_REL_SET]: 0x1f
user area: the device protects existing data if a power failure occurs during a write operation
partition 1: the device protects existing data if a power failure occurs during a write operation
partition 2: the device protects existing data if a power failure occurs during a write operation
partition 3: the device protects existing data if a power failure occurs during a write operation
partition 4: the device protects existing data if a power failure occurs during a write operation
Write reliability parameter register [WR_REL_PARAM]: 0x05
Device supports writing EXT_CSD_WR_REL_SET
Device supports the enhanced def. of reliable write
Enable background operations handshake [BKOPS_EN]: 0x00
H/W reset function [RST_N_FUNCTION]: 0x00
HPI management [HPI_MGMT]: 0x00
Partitioning Support [PARTITIONING_SUPPORT]: 0x01
Device support partitioning feature
Device cannot have enhanced tech.
Max Enhanced Area Size [MAX_ENH_SIZE_MULT]: 0x0000e9
i.e. 3817472 KiB
Partitions attribute [PARTITIONS_ATTRIBUTE]: 0x00
Partitioning Setting [PARTITION_SETTING_COMPLETED]: 0x00
Device partition setting NOT complete
General Purpose Partition Size
[GP_SIZE_MULT_4]: 0x000000
[GP_SIZE_MULT_3]: 0x000000
[GP_SIZE_MULT_2]: 0x000000
[GP_SIZE_MULT_1]: 0x000000
Enhanced User Data Area Size [ENH_SIZE_MULT]: 0x000000
i.e. 0 KiB
Enhanced User Data Start Address [ENH_START_ADDR]: 0x00000000
i.e. 0 bytes offset
Bad Block Management mode [SEC_BAD_BLK_MGMNT]: 0x00
Periodic Wake-up [PERIODIC_WAKEUP]: 0x00
Program CID/CSD in DDR mode support [PROGRAM_CID_CSD_DDR_SUPPORT]: 0x01
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[127]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[126]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[125]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[124]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[123]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[122]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[121]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[120]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[119]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[118]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[117]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[116]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[115]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[114]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[113]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[112]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[111]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[110]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[109]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[108]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[107]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[106]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[105]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[104]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[103]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[102]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[101]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[100]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[99]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[98]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[97]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[96]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[95]]: 0x02
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[94]]: 0x09
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[93]]: 0x04
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[92]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[91]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[90]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[89]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[88]]: 0x09
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[87]]: 0x08
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[86]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[85]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[84]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[83]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[82]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[81]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[80]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[79]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[78]]: 0x72
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[77]]: 0x35
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[76]]: 0x35
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[75]]: 0x69
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[74]]: 0x32
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[73]]: 0x32
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[72]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[71]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[70]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[69]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[68]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[67]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[66]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[65]]: 0x00
Vendor Specific Fields [VENDOR_SPECIFIC_FIELD[64]]: 0x00
Native sector size [NATIVE_SECTOR_SIZE]: 0x00
Sector size emulation [USE_NATIVE_SECTOR]: 0x00
Sector size [DATA_SECTOR_SIZE]: 0x00
1st initialization after disabling sector size emulation [INI_TIMEOUT_EMU]: 0x00
Class 6 commands control [CLASS_6_CTRL]: 0x00
Number of addressed group to be Released[DYNCAP_NEEDED]: 0x00
Exception events control [EXCEPTION_EVENTS_CTRL]: 0x0000
Exception events status[EXCEPTION_EVENTS_STATUS]: 0x0000
Extended Partitions Attribute [EXT_PARTITIONS_ATTRIBUTE]: 0x0000
Context configuration [CONTEXT_CONF[51]]: 0x00
Context configuration [CONTEXT_CONF[50]]: 0x00
Context configuration [CONTEXT_CONF[49]]: 0x00
Context configuration [CONTEXT_CONF[48]]: 0x00
Context configuration [CONTEXT_CONF[47]]: 0x00
Context configuration [CONTEXT_CONF[46]]: 0x00
Context configuration [CONTEXT_CONF[45]]: 0x00
Context configuration [CONTEXT_CONF[44]]: 0x00
Context configuration [CONTEXT_CONF[43]]: 0x00
Context configuration [CONTEXT_CONF[42]]: 0x00
Context configuration [CONTEXT_CONF[41]]: 0x00
Context configuration [CONTEXT_CONF[40]]: 0x00
Context configuration [CONTEXT_CONF[39]]: 0x00
Context configuration [CONTEXT_CONF[38]]: 0x00
Context configuration [CONTEXT_CONF[37]]: 0x00
Packed command status [PACKED_COMMAND_STATUS]: 0x00
Packed command failure index [PACKED_FAILURE_INDEX]: 0x00
Power Off Notification [POWER_OFF_NOTIFICATION]: 0x00
Control to turn the Cache ON/OFF [CACHE_CTRL]: 0x00
[23.05.1-RELEASE][root@pfSense-test-8sep23.home.arpa]/root: mmc extcsd read /dev/mmcsd0rpmb | egrep "EOL"
[23.05.1-RELEASE][root@pfSense-test-8sep23.home.arpa]/root: mmc extcsd read /dev/mmcsd0rpmb | egrep "LIFE"
[23.05.1-RELEASE][root@pfSense-test-8sep23.home.arpa]/root:

                                  Pete
                                  Home: SG-2100 + UniFi + Synology. SG-1100 retired
                                  Parents: SG-1100 + UniFi + Synology
                                  Testing: SG-1100 w/ 120GB SSD via ext USB (eMMC dead). Works great

                                  1 Reply Last reply Reply Quote 0
                                  • stephenw10S
                                    stephenw10 Netgate Administrator
                                    last edited by

                                    I can certainly try. That is odd that it doesn't show all the output from mmcuntils. Though there have been several different eMMC chips used.
                                    What does yours report for the MMC version at the top?

                                    Extended CSD rev 1.8 (MMC 5.1)
                                    1 Reply Last reply Reply Quote 0
                                    • stephenw10S
                                      stephenw10 Netgate Administrator
                                      last edited by

                                      Oh cross posted. Yes you have an early version with the MMC 4.5 chip. That doesn't report lifetime data.

                                      1 Reply Last reply Reply Quote 0
                                      • First post
                                        Last post
                                      Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.