Installed pfSense to new mSATA, but device always boots from eMMC. (NetGate 4860 Desktop)

cyphonsqr

Hello,

I recently installed a 256GB mSATA to my Netgate 4860 Desktop device (for ntopng/traffic monitoring and logging).
I followed these instructions from the Netgate site. The drive is detected successfully and found by the BIOS/installer.
I then followed the NetGate instructions for a full install of 22.05 (file from TAC Lite) from a USB and installed to the SATA drive. (mode: AutoZFS (BIOS))
Install was a success and I rebooted after removing the USB (and avoiding a kernel panic).

Everything appeared to work, but when the device boots, pfSense is running from the original 23GB eMMC drive.

Could you please tell me:

How to install to the mSATA so that pfSense runs from that drive (as master each boot).
How to install a known stable version (i.e.: 22.01 + config.xml) on the eMMC as a backup, in case I break the system/config on the SATA drive (e.g.: doing a future upgrade) and need an urgent fallback.

I thought I knew how to do each of those (by simply picking the correct boot drive in BIOS/Bootloader) but clearly I've missed a trick.
Once I get help on item 1, I suspect item 2 will fall into place.

I have SSH and USB console access.
Thanks for any help.

jimp

It might not be a matter of what the BIOS is booting but conflicting ZFS pools. The safest thing to do is probably to wipe the MMC (or both disks) and only install to the mSATA.

If you boot the installation memstick you can drop to a shell before starting the installation and then use dd to wipe the MMC, and again for the SSD. Then I'd reboot after for good measure and proceed through the install using only the SSD.

There may be some half measures you can take, like clearing the ZFS label from the MMC, but wiping it is safer and more likely to have a positive result.

If you need model-specific info on wiping the MMC, shoot a message to TAC and they can tell you what to do on that box.

cyphonsqr

@jimp Thanks!

I agree that a full wipe is best :)

I'm happy to be given general guidance on using the dd command. I can't imagine it's too complicated (to fully wipe each disk).
And then I guess I'd boot from scratch and start the install anew.

jimp

In general you can do dd if=/dev/zero of=/dev/<disk name> bs=1M status=progress

The exact value of the disk device ID will vary by platform but you can see it in the boot log or other places, like the output of sysctl kern.disks shows all the detected drives.

cyphonsqr

@jimp Cheers.

From TAC, the command for my 4860 desktop is dd if=/dev/zero of=/dev/ada0 bs=8k count=16

I had to clear the boot record data (using gpart destroy -F ada0) because one of the boot records was reported as 'corrupt'.
From Googling, this 'corrupt' error occurs if you use the dd command on a ZFS filesystem (as ZFS doesn't clear both boot records or something like that). Overall, the dd (and gpart) commands did the trick.

I had trouble getting the system to boot. I suspect it was due to the install method.
Do you know the best/correct installation method for this device? GPT(BIOS), GPT(UEFI), MBR(BIOS), etc.?

jimp

There are a few other commands that clear metadata from the end of the disk as well as the beginning (e.g. gpart destroy, zpool labelclear, and potentially others), so using dd with count can be trickier. If you note my command didn't use count so it would have erased the entire disk, along with all of the metadata at the start and end.

As for your other question, IIRC that system can't do UEFI, so the best choice would be GPT (BIOS)

cyphonsqr

@jimp
Excellent. Thanks again for both those bits of advice. :)
I tried the wipe and install tonight, but it didn't go as planned :/ ... so I installed back what I had and will retry on the weekend.

cyphonsqr

@jimp Thanks for your help.
Just wanted to close this out. I finally got it installed.
Had to completely format both drives (eMMC and mSATA) and remove the ZFS partition data.
Wasn't able to install pfSense onto the eMMC as a backup (as every time the eMMC device had any ZFS boot label/data on it, it seemed to take over as primary boot device). In the end, I left the eMMC completely formatted (and wiped ZFS labels) and just installed onto the mSATA. That did the trick.

I installed pfSense using ZFS and GPT(BIOS) (as UEFI didn't work).

Thanks again.

rcoleman-netgate

@cyphonsqr said in Installed pfSense to new mSATA, but device always boots from eMMC. (NetGate 4860 Desktop):

Had to completely format both drives (eMMC and mSATA) and remove the ZFS partition data.

Yes, when ZFS is involved the only solution to installing to a SATA device involves reformatting/zeroing out the eMMC or it will attempt to boot from the eMMC instead.