Errors on fsck ... how to correct? Manual not helpful

Jax · Jan 16, 2021, 6:53 PM

fsck in console shell reports errors (shown at end of message).
Manual suggests Gui Diagnostics->Reboot and select reboot type but no such option for Reboot on SG-2100.
Edited /boot/loader.conf to force fsck on reboot and it runs but still errors.
What's the problem here?

[2.4.5-RELEASE][root@pfSense.localdomain]/root: fsck
** /dev/ufsid/5ed6cabbfa4b9f89 (NO WRITE)
** SU+J Recovering /dev/ufsid/5ed6cabbfa4b9f89

USE JOURNAL? no

** Skipping journal, falling through to full fsck

SETTING DIRTY FLAG IN READ_ONLY MODE

UNEXPECTED SOFT UPDATE INCONSISTENCY
** Last Mounted on /
** Root file system
** Phase 1 - Check Blocks and Sizes
INCORRECT BLOCK COUNT I=1385 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=1459 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=1516 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=46148 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=46490 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=58235 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=58315 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=95673 (24 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=95674 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=98456 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=98459 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=98461 (24 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=98467 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=98473 (144 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=98479 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=98480 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=98487 (8 should be 0)
CORRECT? no

bmeeks · Jan 16, 2021, 7:13 PM

Have you checked the official pfSense documentation here: https://docs.netgate.com/pfsense/en/latest/troubleshooting/filesystem-check.html?

Note that you must give some additional command-line arguments to fsck and you generally need to repeat the command at least five times to fully repair the filesystem if running the check manually. The link I provided first gives instructions for starting an automatic repair from the GUI (assuming your machine boots that far, but it may be halting at the boot up). If you can't run the automatic fix, the bottom of the linked page gives instructions for starting the fix manually.

Jax · Jan 16, 2021, 8:11 PM

@bmeeks After finding the manual was wrong about this:

GUI

Navigate to Diagnostics > Reboot

Set Reboot Method to Reboot with Filesystem Check

Click fa-wrench Submit

(No such option to "Set Reboot Method" in the GUI) I turned to general FreeBSD help and chose to trigger an fsck via a kernel var in /boot/loader.conf.

I will try opt 5 from the Console next, thank you.

Jax · Jan 16, 2021, 8:42 PM

This post is deleted!

Jax · Jan 16, 2021, 9:26 PM

@bmeeks Tried 6 times with an F reboot from console and still fsck shows errors. What now?

Jax · Jan 16, 2021, 9:43 PM

This after 7 "F" reboots:;

[2.4.5-RELEASE][root@pfSense.localdomain]/root: fsck
** /dev/ufsid/5ed6cabbfa4b9f89 (NO WRITE)
** SU+J Recovering /dev/ufsid/5ed6cabbfa4b9f89

USE JOURNAL? no

** Skipping journal, falling through to full fsck

SETTING DIRTY FLAG IN READ_ONLY MODE

UNEXPECTED SOFT UPDATE INCONSISTENCY
** Last Mounted on /
** Root file system
** Phase 1 - Check Blocks and Sizes
INCORRECT BLOCK COUNT I=1385 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=1459 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=1516 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=46148 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=46490 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=58235 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=58315 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=95672 (24 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=95673 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=98456 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=98459 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=98461 (24 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=98479 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=98480 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=98483 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=98484 (136 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=98487 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=98489 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=141112 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=141113 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=183304 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=183305 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=183306 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=183307 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=183308 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=183321 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=183333 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=183335 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=183337 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=183339 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=183352 (16 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=183357 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=183369 (8 should be 0)
CORRECT? no

INCORRECT BLOCK COUNT I=183376 (8 should be 0)
CORRECT? no

** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
UNREF FILE I=95665  OWNER=root MODE=100666
SIZE=0 MTIME=Jan 16 21:37 2021 
CLEAR? no

UNREF FILE  I=95675  OWNER=root MODE=100600
SIZE=0 MTIME=Jan 16 21:03 2021 
RECONNECT? no


CLEAR? no

UNREF FILE  I=98464  OWNER=root MODE=100644
SIZE=0 MTIME=Jan 16 21:38 2021 
RECONNECT? no


CLEAR? no

LINK COUNT FILE I=98482  OWNER=unbound MODE=100644
SIZE=758 MTIME=Jan 16 21:38 2021  COUNT 2 SHOULD BE 1
ADJUST? no

LINK COUNT DIR I=183296  OWNER=root MODE=40755
SIZE=512 MTIME=Jan 16 21:38 2021  COUNT 5 SHOULD BE 2
ADJUST? no

UNREF FILE  I=183322  OWNER=dhcpd MODE=100644
SIZE=0 MTIME=Jan 16 21:38 2021 
RECONNECT? no


CLEAR? no

UNREF FILE  I=183335  OWNER=root MODE=100644
SIZE=0 MTIME=Jan 16 21:38 2021 
RECONNECT? no


CLEAR? no

UNREF FILE  I=183339  OWNER=root MODE=100644
SIZE=0 MTIME=Jan 16 21:38 2021 
RECONNECT? no


CLEAR? no

UNREF FILE  I=183354  OWNER=root MODE=100644
SIZE=0 MTIME=Jan 16 21:38 2021 
RECONNECT? no


CLEAR? no

ZERO LENGTH DIR I=183357  OWNER=root MODE=40755
SIZE=0 MTIME=Jan 16 21:38 2021 
CLEAR? no

UNREF FILE  I=183358  OWNER=root MODE=140666
SIZE=0 MTIME=Jan 16 21:38 2021 
RECONNECT? no


CLEAR? no

ZERO LENGTH DIR I=183369  OWNER=root MODE=40750
SIZE=0 MTIME=Jan 16 21:38 2021 
CLEAR? no

ZERO LENGTH DIR I=183376  OWNER=root MODE=40750
SIZE=0 MTIME=Jan 16 21:38 2021 
CLEAR? no

** Phase 5 - Check Cyl groups
FREE BLK COUNT(S) WRONG IN SUPERBLK
SALVAGE? no

SUMMARY INFORMATION BAD
SALVAGE? no

BLK(S) MISSING IN BIT MAPS
SALVAGE? no

25284 files, 286147 used, 1553541 free (197 frags, 194168 blocks, 0.0% fragmentation)

Jax · Jan 16, 2021, 9:47 PM

I've never seen anything like this with a BSD.
If repeated fsck at boot can't fix storage, this is totally bogus.
Perhaps storage is defective, or the build of FreeBSD is bad.
Thinking of contacting Netgate for an RMA.

bmeeks · Jan 17, 2021, 3:45 AM

@jax said in Errors on fsck ... how to correct? Manual not helpful:

I've never seen anything like this with a BSD.
If repeated fsck at boot can't fix storage, this is totally bogus.
Perhaps storage is defective, or the build of FreeBSD is bad.
Thinking of contacting Netgate for an RMA.

Yes, since this is a Netgate appliance they should be able to help you with the hardware. I understand they respond to tickets very quickly, even after hours. There is a link on their website for opening a support ticket. I understand that hardware support (especially for under warranty products) does not require a paid support contract.

Jax · Jan 17, 2021, 3:45 AM

@bmeeks Okay, I had my head in a dark place :)
fsckon FreeBSD is invalid even in read-only mode running on a mounted partition.
At least I learned how to install pfSense from USB stick in trying to "debug" my non-problem!
Thanks for the help!

stephenw10 · Jan 18, 2021, 12:10 AM

For reference you would want to boot to single user mode so root is not mounted and then run:
fsck -y /

To avoid having to manually answer yes to all those questions.

And then, crucially, repeat that at least three times. The fsck command can incorrectly report clean on the first pass.

Steve

Jax · Jan 18, 2021, 12:43 AM

@stephenw10 Thank you. Do you recommend doing this for "good luck"? I'm not sure I ever had a problem. I was taking the output of fsck in read-only mode as valid indicating there were fs errors, but apparently on modern FreeBSD fsck on a mounted partition simply doesn't work.

stephenw10 · Jan 18, 2021, 4:38 PM

It doesn't hurt to do it but it shouldn't be necessary unless you have seen a power failure that might potentially have damaged the filesystem. Even then it shouldn't be because fsck gets run at every boot anyway so most issues just get corrected.

Steve

Jax · Jan 18, 2021, 4:38 PM

@stephenw10 Thanks, I ran a few times for good luck. No issues. HappyHappyHappyJoyJoyJoy :)

Thanks for help.