6100 effectively dead after electricians were toggling breakers trying to find a circuit.

nvwasher

Netgate 6100 MAX — Repeated Install Failures After Power Event

Device

Netgate 6100 MAX, ~3 years old, pfSense Plus
128GB M.2 NVMe SSD (factory) + internal eMMC
Never had a paid TAC subscription; requesting help under TAC Lite (free-with-hardware)

Trigger

Local power event (electricians flipping breakers rapidly). Afterward the unit's front LED showed solid orange (Standby) instead of the normal blue "Ready" state. Power-cycling via the front panel button eventually got it POSTing again, but pfSense would not boot.

Initial diagnosis (via USB serial console)

UEFI/BlinkBoot POST completes normally and the FreeBSD loader successfully loads /EFI/freebsd/loader.efi from the NVMe's ESP partition. However, mounting the root ZFS pool fails:

Setting currdev to zfs:pfSense/ROOT/default:
zio_read error: 5
zio_read error: 5
ZFS: i/o error - all block copies unavailable

This repeats across multiple partitions/regions of the NVMe (both the primary root dataset and a secondary attempt), then drops to the loader prompt (ERROR: cannot open /boot/lua/loader.lua).

Reinstall attempts (chronological)

All installs performed via the official Netgate Installer (netgate-installer-amd64.img.gz/equivalent), written to USB via dd on macOS, with correct target-drive selection each time (nda0 for NVMe, mmcsd0 for eMMC — never both).

#	Release	Target	USB stick	Result
1	26.03.1	NVMe (`nda0`)	Stick A	Segfault at package 204/209 during extraction of `pfSense-system-26.03.1` (`Child process pid=3777 terminated abnormally: Segmentation fault`)
2	26.03	eMMC (`mmcsd0`)	Stick B (different physical stick)	Same segfault, same package (204/209)
3	26.03	eMMC (`mmcsd0`)	Stick B, freshly re-downloaded image (different file hash than attempt 1/2)	Same segfault, same package
3b	26.03	NVMe (`nda0`), same session	Stick B	`nvme0: waiting for reset` / `aborting outstanding i/o` looped during the attempt
4	25.11.1	NVMe (`nda0`)	Stick B	Package install completed without segfault. However, first full boot later showed `nvme0: aborting outstanding i/o` repeatedly, plus live `Input/output error` on basic reads (`sleep`, etc.) during an `rc.shutdown` sequence, ending in Standby.
5	25.11.1	eMMC (`mmcsd0`)	Stick B	Package install completed cleanly. First boot failed: `ERROR: It was not possible to identify which pfSense kernel is installed` → `* SYSTEM BOOT FAILURE *`. `zpool status` at that point showed a healthy single-disk pool (`pfSense` ONLINE, member `mmcsd0p4` ONLINE, 0 errors) — ruling out a multi-disk ZFS pool conflict. `pkg info` failed with: `sqlite error ... table files has no column named uname` / `Package database is busy while closing!`
6	25.11.1	eMMC (`mmcsd0`), disk properly wiped via a fresh installer run (not manual `dd`)	Stick B	Identical failure to attempt 5 — same kernel-identification error, same sqlite schema-corruption signature, on an independently clean install.

Why this looks board-level rather than disk/release/media-specific

The identical segfault (package 204/209) reproduced across two different target disks (NVMe and eMMC), three different USB installer sticks, and two independently downloaded installer images with different file hashes — ruling out a bad USB stick or a one-off bad download.
Both storage devices show write-integrity failures, just manifesting differently depending on the exact operation: bulk package extraction segfaults on 26.x, pkg database (SQLite) corruption on 25.x, reproduced twice independently on eMMC.
25.x completed the install step cleanly on both disks (no segfault), but both times developed a corrupted package database detectable only after reboot — consistent with data being corrupted in flight during writes (RAM or storage controller/power delivery) rather than a release-specific software bug.
NVMe independently shows a second, distinct failure mode under load: nvme0 reset/abort loops during a live, running OS session, not just at idle/boot.

Current status

A replacement Netgate 6100 is already ordered and expected to arrive tomorrow. Plan is to run the identical install (same USB stick, same release) on the replacement unit as a control test. If it completes cleanly there, that would confirm the original unit's board is at fault.

Request

Is this a known failure signature, or is there additional diagnostic data (e.g., memory test, NVMe SMART/media-error counters) worth capturing on the original unit before it's set aside?
Guidance on RMA path for a ~3-year-old unit under TAC Lite given this reproducible failure pattern.

tedquade

@nvwasher Place the new one on a UPS to avoid this sort of problem.

Ted

stephenw10

That's not a failure mode I've ever seen certainly. Did you try removing the NVMe drive? Even if you're installing to eMMC the NVMe is still active and can cause problems.

nvwasher

@tedquade UPS was present but non-functioning. New unit arriving soon

nvwasher

@stephenw10 said in 6100 effectively dead after electricians were toggling breakers trying to find a circuit.:

That's not a failure mode I've ever seen certainly. Did you try removing the NVMe drive? Even if you're installing to eMMC the NVMe is still active and can cause problems.

I got it working (limping along) by installing 25.07 to the eMMC and then updating to 26.03.1 and restoring my backup. New unit will arrive tomorrow. I'll get that one installed and then play with this one.

Gertjan

@nvwasher

So it was a 'drive' problem ?

nvwasher

@Gertjan So it seems. New drive is here but I've not had time to install it. Too many other fires.