SG-2220 boot issues

scottlindner

This is what I ordered for delivery tomorrow. The docs say it can't be NVME. As far as I can tell, this isn't NVME so I'm hoping this will be OK.
https://www.amazon.com/Kingdata-M-2-2260-512GB-High-Performance/dp/B08J2KQZHD/

And just for giggles, I'm posting here using a USB WiFi card in my desktop that is connected via the console to the SG-2220 using the mobile hotspot on my phone. Total suck! But it is somewhat functional. LOL

akuma1x

@scottlindner That amount of storage might be too large for the Netgate box to recognize, but I'm not sure. You could install it and see what happens.

Not sure if you are aware, but here are the instructions:

https://docs.netgate.com/pfsense/en/latest/solutions/sg-2220/m-2-sata-installation.html

This is the one I bought for my SG-2220, middle of 2020, so I could update my box while working from home during COVID:

https://www.amazon.com/gp/product/B077H276GQ?th=1

scottlindner

@akuma1x

Thank you for that. I just ordered the exact one because I just want it to work. Both are returnable so I'll try the big one first and if that doesn't work I'll try the one you already know works.

From what you know, does my console output suggest my eMMC is toast and I need this M.2 SSD? I really just want it working now and I'll still do the M.2 SSD when it arrives.

akuma1x

@scottlindner I'm a total novice when it comes to FreeBSD, I know just enough to get myself into trouble. But, some of the stuff near the end of that console dump looks like the drive has failed. Maybe one of the experts could chime in here and tell you for sure.

Do you have a backup of your config? If you do, and you can get your hands on a restore image of pfsense, it is possible that you can re-format the internal storage. It might just be messed up enough that it won't allow a successful boot of the operating system, maybe...

viragomann

@scottlindner said in SG-2220 boot issues:

Trying to mount root from ufs:/dev/ufsid/5911c7e65e0911e0 [rw]...

Do you have a backup?

If yes, reinstall pfSense using ZFS and import the backup.

If you really want to recover the UFS filesystem boot pfSense into the single user mode and enter the command prompt and try

/sbin/fsck -y -t ufs

scottlindner

@akuma1x Yes. I have always feared this situation so every time I make a config change I save the file. Restoring should be trivial once I have the pfSense installed on the new SSD.

viragomann

@scottlindner
Seems not to me that the device is damaged, but the filesystem is not clean.

ERROR: Impossible to mount filesystem, use interactive shell to attempt to recover it

scottlindner

@viragomann good catch. I have been trying to get to a shell so I can run fsck and it isn't finding a file system. When I try to boot into single user mode it prompts me with mountroot> and when I enter the device name it fails. But maybe I don't really know what I'm doing. Or maybe the device is so failed it isn't even reporting as a device anymore?

Let me get more console output for you if you don't mind looking at that.

SteveITS

@scottlindner Have you looked at https://docs.netgate.com/pfsense/en/latest/troubleshooting/filesystem-check.html?

scottlindner

@steveits said in SG-2220 boot issues:

@scottlindner Have you looked at https://docs.netgate.com/pfsense/en/latest/troubleshooting/filesystem-check.html?

I have. I get stuck at mountroot> It doesn't seem to find the eMMC device. And now I can't even get to the pfSense menu via the console anymore. I wonder if the whole thing is dying and not just the storage.

EDIT: I bet the eMMC device just completely died now that I think about it.

This is what I get now from the console and the #'s just keep tallying up forever.

SeaBIOS (version SageBIOS-20170207_214234-localhost.localdomain)


iPXE (http://ipxe.org) 00:14.0 C100 PCI2.10 PnP PMM+7FB7D480+7FADD480 C100





iPXE (http://ipxe.org) 00:14.1 C200 PCI2.10 PnP PMM 7FB7D480 7FADD480 C200


iPXE (http://ipxe.org) 00:14.2 C300 PCI2.10 PnP PMM 7FB7D480 7FADD480 C300


iPXE (http://ipxe.org) 00:14.3 C400 PCI2.10 PnP PMM 7FB7D480 7FADD480 C400

Press F12 for boot menu.

Select boot device:

1. USB MSC Drive Generic Ultra HS-COMBO 1.98
2. iPXE (PCI 00:14.0)
3. iPXE (PCI 00:14.1)
4. iPXE (PCI 00:14.2)
5. iPXE (PCI 00:14.3)
6. Payload [setup]


F1  pfSense

F6 PXE
Boot:  F1 ##

scottlindner

@viragomann said in SG-2220 boot issues:

@scottlindner said in SG-2220 boot issues:

Trying to mount root from ufs:/dev/ufsid/5911c7e65e0911e0 [rw]...

Do you have a backup?

If yes, reinstall pfSense using ZFS and import the backup.

If you really want to recover the UFS filesystem boot pfSense into the single user mode and enter the command prompt and try
/sbin/fsck -y -t ufs

I'm not able to boot into signle user mode. I put the pfSense installation on a USB stick following a guide cited in this discussion and booted into single user mode with that installer and ran the following:

/sbin/fsck -y -t ufs /dev/ufsid/5911c7e65e0911e0

And it is spewing tons of errors just like this (already filled the Putty console buffer so I can't go back to read any of the beginning stuff that was slightly different).

CANNOT READ BLK: 1311232
UNEXPECTED SOFT UPDATE INCONSISTENCY

CONTINUE? yes

THE FOLLOWING DISK SECTORS COULD NOT BE READ: 1311232, 1311233, 1311234, 1311235, 1311236, 1311237, 1311238, 1311239, 1311240, 1311241, 1311242, 1311243, 1311244, 1311245, 1311246, 1311247, 1311248, 1311249, 1311250, 1311251, 1311252, 1311253, 1311254, 1311255, 1311256, 1311257, 1311258, 1311259, 1311260, 1311261, 1311262, 1311263, 1311264, 1311265, 1311266, 1311267, 1311268, 1311269, 1311270, 1311271, 1311272, 1311273, 1311274, 1311275, 1311276, 1311277, 1311278, 1311279, 1311280, 1311281, 1311282, 1311283, 1311284, 1311285, 1311286, 1311287, 1311288, 1311289, 1311290, 1311291, 1311292, 1311293, 1311294, 1311295, 1311296, 1311297, 1311298, 1311299, 1311300, 1311301, 1311302, 1311303, 1311304, 1311305, 1311306, 1311307, 1311308, 1311309, 1311310, 1311311, 1311312, 1311313, 1311314, 1311315, 1311316, 1311317, 1311318, 1311319, 1311320, 1311321, 1311322, 1311323, 1311324, 1311325, 1311326, 1311327, 1311328, 1311329, 1311330, 1311331, 1311332, 1311333, 1311334, 1311335, 1311336, 1311337, 1311338, 1311339, 1311340, 1311341, 1311342, 1311343, 1311344, 1311345, 1311346, 1311347, 1311348, 1311349, 1311350, 1311351, 1311352, 1311353, 1311354, 1311355, 1311356, 1311357, 1311358, 1311359,

CANNOT READ BLK: 1311360
UNEXPECTED SOFT UPDATE INCONSISTENCY

CONTINUE? yes

THE FOLLOWING DISK SECTORS COULD NOT BE READ: 1311360, 1311361, 1311362, 1311363, 1311364, 1311365, 1311366, 1311367, 1311368, 1311369, 1311370, 1311371, 1311372, 1311373, 1311374, 1311375, 1311376, 1311377, 1311378, 1311379, 1311380, 1311381, 1311382, 1311383, 1311384, 1311385, 1311386, 1311387, 1311388, 1311389, 1311390, 1311391, 1311392, 1311393, 1311394, 1311395, 1311396, 1311397, 1311398, 1311399, 1311400, 1311401, 1311402, 1311403, 1311404, 1311405, 1311406, 1311407, 1311408, 1311409, 1311410, 1311411, 1311412, 1311413, 1311414, 1311415, 1311416, 1311417, 1311418, 1311419, 1311420, 1311421, 1311422, 1311423, 1311424, 1311425, 1311426, 1311427, 1311428, 1311429, 1311430, 1311431, 1311432, 1311433, 1311434, 1311435, 1311436, 1311437, 1311438, 1311439, 1311440, 1311441, 1311442, 1311443, 1311444, 1311445, 1311446, 1311447, 1311448, 1311449, 1311450, 1311451, 1311452, 1311453, 1311454, 1311455, 1311456, 1311457, 1311458, 1311459, 1311460, 1311461, 1311462, 1311463, 1311464, 1311465, 1311466, 1311467, 1311468, 1311469, 1311470, 1311471, 1311472, 1311473, 1311474, 1311475, 1311476, 1311477, 1311478, 1311479, 1311480, 1311481, 1311482, 1311483, 1311484, 1311485, 1311486, 1311487,

CANNOT READ BLK: 1311488
UNEXPECTED SOFT UPDATE INCONSISTENCY

CONTINUE? yes

I think this confirms the eMMC is dead, right?

scottlindner

I was able to reinstall pfSense+ onto a 32GB USB stick and restore the last config I had saved off from about 3 weeks ago which has all of my latest changes in it. I'm still going to add the M.2 SATA SSD when it arrives tomorrow and install again. I could not get it to install to the built in eMMC (/dev/da0). I think the eMMC is just completely toast.

I'll update again tomorrow after I get the new storage installed and see how well it all works out.

stephenw10

I agree it looks like the eMMC is damaged, those read errors do not look like just a filesystem problem.
Since you were able to install to and run from another USB drive though I would expect an m.2 drive to work fine.

Steve

scottlindner

@stephenw10 it appears every sector was marked bad. The eMMC max size is 0.

I'm happy with the fix. I didn't even know a USB install was possible until I started thinking back to the drive selection. I had to use a hub since it has one USB port which is fine. Looking forward to the M.2 to show up.

So glad I feared this situation years ago and got in the habit to save off my config after every change.

akuma1x

@scottlindner If it's possible at this point, you could do what I did with the additional storage I added to my SG-2220. Erase the internal EMMC storage, and leave it empty. That way, it won't attempt to boot from that volume. Then, simply format and install pfsense on your new internal M.2 drive.

scottlindner

@akuma1x I like the suggestion. I don't think it can attempt to boot from the eMMC because the size is now 0 after the fsck. I'm not 100% certain I know how this works but I think it was MBR and at 0 max size it can't have a boot record. Assuming I understand that is how it works. My firewall seems to be booting faster than it ever had in the past and that is off the USB. I'm hopeful it'll be even faster when the M.2 SATA SSD gets here later today.

I appreciate everyone's engagement on this. This is one of those ones that is so specific I didn't expect much interest in it.

scottlindner

@akuma1x I just installed the 512GB M.2 SATA and reinstalled pfSense using ZFS and it appears to work just fine and all of the storage is available. I don't know why I bought such a large drive. I guess price point stupidity. But this thing boots so much faster now I love it.