Netgate 1100 EMMC failure, subsequent USB flash drive issues

TigerFox57

Hi.

In common with quite a few recent posts it seems, the emmc storage on my 1100 has started to fail. After a bit of googling and reading this forum the issue became clear so I don't need to go over all that again. I followed the instructions in another post about how to get the device to work from a USB stick, and everything seemed to go smoothly, the image was written to the stick, the boot order changed, and the 1100 boots into the latest version of the software.

However, it seems that there is now another problem. The firewall is unable to actually load any rules, even though they are showing in the GUI. Each time a rule is added or modified, messages like this appear in the log.

/rc.filter_configure_sync: The command '/sbin/pfctl -Of /tmp/rules.limits' returned exit code '1', the output was 'pfctl: DIOCSETTIMEOUT pfctl: DIOCSETTIMEOUT pfctl: DIOCSETTIMEOUT pfctl: DIOCSETTIMEOUT pfctl: DIOCSETTIMEOUT'

Reading the various posts on this forum that mention issues like this, my guess is that there is a kernel mismatch somehow. The GUI says I'm on version

24.03-RELEASE (arm64)
built on Fri Jan 10 15:05:00 UTC 2025
FreeBSD 14.0-CURRENT

and uname -a gives

FreeBSD pfSense.robertwatts.com 14.0-CURRENT FreeBSD 14.0-CURRENT aarch64 1400094 #1 plus-RELENG_23_09_1-n256200-3de1e293f3a: Wed Dec 6 20:59:18 UTC 2023

The image was obtained from Netgate support via their online store.

My question is, where is this mismatch coming from? Is it the fact that the failing emmc storage is still present and causing confusion in the boot process?

I'm not an expert in these things, but can follow instructions, so I'd be grateful if someone could give me a hint on how to fix this, assuming that it can be fixed at all.

Regards, Rob.

SteveITS

@TigerFox57 Had you found this : https://docs.netgate.com/pfsense/en/latest/troubleshooting/multiple-disks.html

stephenw10

What version did you choose to install? What filesystem type?

I'd guess it's mounting root as ZFS from eMMC still. So, yes, if you can clear the zpool labels from the eMMC that would prevent it.

If you can't clear that another option would be to install as UFS to the USB.

TigerFox57

@SteveITS Thanks for this, no I had not seen this.

As a first go, I went into the console and tried clearing the labels from the 5 emmc drives listed, mmcsd0s1,s2,s3,boot0 and boot1, but the command returned that the label had been failed to cleat. I then tried the gpart destroy -F mmcsd0 command, which returned success. dd if=/dev/zero of=/dev/mmcsd0 bs=1M count=1 status=progress seemed to progress instantly, but I'm not sure what it means.
1+0 records in
1+0 records out
1048576 bytes transferred in 0.111966 secs (9365111 bytes/sec)

I then rebooted. Everything looked OK on the console as it booted, I didn't see any error messages, but logging into the GUI showed similar error messages as before.

There were error(s) loading the rules: pfctl: DIOCADDRULENV: Device not configured - The line in question reads [0]: @ 2025-03-24 12:45:27

Also, in the console I now see this:

pid 72986 (pfctl), jid 0, uid 0: exited on signal 6 (core dumped)

continuously scrolling about once a second.

I imagine that the emmc has become read only, so the wipe actually failed. I guess my next trick is to try removing the emmc chip entirely and see where that leads.

Regards, Rob.

TigerFox57

@stephenw10 I'll try UFS first, before getting out my screwdriver...

TigerFox57

@stephenw10 OK, I reflashed the USB stick using ufs, and the errors have disappeared, so it seems that the 1100 is back alive.

uname -a FreeBSD pfSense.robertwatts.com 15.0-CURRENT FreeBSD 15.0-CURRENT #0 plus-RELENG_24_11-n256407-1bbb3194162: Fri Nov 22 05:12:13 UTC 2024

It took a very long time to actually do the reflash, not sure why, but it eventually completed.

Thanks for your help with this, it's just a pity that I didn't know about the shortcomings of emmc before, I might have saved myself some head scratching. Since I bought a different device to replace it while I figured out what was going on, I now have a spare for future emergencies.

Regards, Rob.

stephenw10

The new installer version will have options to set the zpool name which also avoids that.