Fsck now not run at all, box stuck in RO mode and unreachable via network
-
So we had a power outage…
Trying to mount root from ufs:/dev/ufs/pfsense0 [ro,sync,noatime]... WARNING: / was not properly dismounted Configuring crash dumps... ** SU+J Recovering /dev/ufs/pfsense0 ** Reading 7565312 byte journal from inode 4. ** Building recovery table. ** Resolving unreferenced inode list. ** Processing journal entries. /dev/ufs/pfsense0: Block missing from ino 8254 at lbn 18 /dWARNING: /cf was not properly dismounted ev/ufs/pfsense0: UNEXPECTED SU+J INCONSISTENCY /dev/ufs/pfsense0: INTERNAL ERROR: GOT TO reply() /dev/ufs/pfsense0: UNEXPECTED SOFT UPDATE INCONSISTENCY; RUN fsck MANUALLY. Filesystems are clean, continuing... Mounting filesystems... mount: /dev/ufs/pfsense0: R/W mount of / denied. Filesystem is not clean - run fsck. Forced mount will invalidate journal contents: Operation not permitted mount: /dev/ufs/cf: R/W mount of /cf denied. Filesystem is not clean - run fsck.: Operation not permitted Setting up memory disks... done. Disabling APM on /dev/ad0 pwd_mkdb: /etc//pwd.db.tmp: Read-only file system ___ ___/ f \ / p \___/ Sense \___/ \ \___/ Welcome to pfSense 2.2.4-DEVELOPMENT ... cp: /var/db/pbi/keys/pfSense-BETA.ssl: Read-only file system Creating symlinks......ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout done. cap_mkdb: /etc/login.conf.db: Read-only file system >>> Under 512 megabytes of ram detected. Not enabling APC.Now, you can wait forever and nothing will happen. Until you CTRL+C and let fsck run manually.
^CJul 9 11:04:50 init: /bin/sh on /etc/rc terminated abnormally, going to single user mode # /sbin/fsck -p -y /dev/ufs/pfsense0 ** SU+J Recovering /dev/ufs/pfsense0 ** Reading 7565312 byte journal from inode 4. ** Building recovery table. ** Resolving unreferenced inode list. ** Processing journal entries. /dev/ufs/pfsense0: Block missing from ino 8254 at lbn 18 /dev/ufs/pfsense0: UNEXPECTED SU+J INCONSISTENCY /dev/ufs/pfsense0: INTERNAL ERROR: GOT TO reply() /dev/ufs/pfsense0: UNEXPECTED SOFT UPDATE INCONSISTENCY; RUN fsck MANUALLY. # /sbin/fsck -y /dev/ufs/pfsense0 ** /dev/ufs/pfsense0 USE JOURNAL? yes ** SU+J Recovering /dev/ufs/pfsense0 ** Reading 7565312 byte journal from inode 4. RECOVER? yes ** Building recovery table. ** Resolving unreferenced inode list. ** Processing journal entries. Block missing from ino 8254 at lbn 18 UNEXPECTED SU+J INCONSISTENCY FALLBACK TO FULL FSCK? yes ** Skipping journal, falling through to full fsck ** Last Mounted on / ** Root file system ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 11510 files, 732309 used, 1127049 free (2049 frags, 140625 blocks, 0.1% fragmentation) ***** FILE SYSTEM MARKED CLEAN ***** ***** FILE SYSTEM WAS MODIFIED ***** # /sbin/rebootHanged. Wonderful. Now, imagine this box is couple hundred miles away.
Kindly put the fsck back the way it was. And I'd also prefer to get rid of the journal crap, since it's apparently apparently broken and buggy and generally causes issues I've never ever seen to happen before. >:( >:( >:(
EDIT: Filed @ https://redmine.pfsense.org/issues/4822
-
Now, imagine this box is couple hundred miles away.
See the RedMine issue. I fully expect that the one I replaced a few hundred miles away in Jumla (mountain flight or 20 hours in a vehicle on this road https://www.youtube.com/watch?v=W_Su32QOYJQ - yes, I mountain-biked this road a few months ago, do come and visit and cycle with me some day) will come back to me with with some permutation of this problem.
Prior to 2.2.3 I have done remote updates of Alix systems and never had an issue. They had always survived unexpected power outages in multiple remote sites and always booted back clean.
I am away from home this week, but when I get back Friday night, assuming 2.2.4-DEVELOPMENT thinks the issue is fixed, I will upgrade my home Alix and try to mess it up with power-cycling. -
I cannot press Enter twice remotely. If you suggest that it's just the console showing no output, then no… this has lasted overnight with no progress anywhere. The box is simply unreachable over network. And this whole 2.2.3+ thing is a total disaster regarding remote boxes. IMNSHO almost all those random "fixes" for the filesystem corruption should be reverted (undo the fsck script changes, definitely get rid of SU+J shit, get rid of sync on full, stick back the vfs patch on nano and nothing like that should ever be done again on bugfix releases.)
P.S. 2.2.4-DEVELOPMENT is what my output is from - so no, definitely not fixed. Going to switch all the remote nano boxes back to the 2.2.2 slice ASAP.
-
How to get rid of SU+J shit on 2.2.3+ nanobsd:
@Ian:
1/ In Diagnostics - NanoBSD, duplicate the current slice:
2/ Check which slice is currently in use:
# mount -o rw / # mount | grep pfsense /dev/ufs/pfsense0 on / (ufs, local, noatime, synchronous, journaled soft-updates)3/ Disable the harmful SU+J stuff on the other slice – i.e., if the output above shows pfsense0, use pfsense1 for the rest of commands below. If it shows pfsense1, use pfsense0 instead.
# tunefs -j disable /dev/ufs/pfsense1 # tunefs -n disable /dev/ufs/pfsense1 # mount /dev/ufs/pfsense1 /mnt # rm -f /mnt/.sujournal # umount /mnt4/ Check that it worked:
# tunefs -p /dev/ufs/pfsense1 ... tunefs: soft updates: (-n) disabled tunefs: soft update journaling: (-j) disabled ...5/ In Diagnostics - NanoBSD, switch the boot slice:
6/ Reboot
7/ Double-check that it still works:
# mount -o rw / # mount | grep pfsense /dev/ufs/pfsense1 on / (ufs, local, noatime, synchronous)# tunefs -p /dev/ufs/pfsense1 ... tunefs: soft updates: (-n) disabled tunefs: soft update journaling: (-j) disabled ...Done!
FAQ:
Q: Will this survive upgrades?
A: Not on nanobsd. Wash, rinse, repeat the above until pfSense devs get rid of SU+J.Q: How can I do this on full install?
A: Perhaps boot from LiveCD and try there; tunefs doesn't work on currently mounted filesystem, even if remounted read-only, despite the fine manual claiming otherwise (plus you'd have to move /tmp and /var to ramdisk first.) -
There are no indications this has anything to do with SU+J. All the issues here seem to revolve around poorly performing/behaving CF cards/other flash media minus the forcesync patch. It's being worked on.
For full installs, sync has already been removed from the installer and will be removed from fstab upon upgrade to 2.2.4. That has no relation to the issues here, which are with nano. It doesn't cause any problems to have it, but is being removed since it's no longer necessary with pw being fixed.
-
@cmb:
There are no indications this has anything to do with SU+J.
What? You serious? :o
/dev/ufs/pfsense0: UNEXPECTED SU+J INCONSISTENCY /dev/ufs/pfsense0: INTERNAL ERROR: GOT TO reply() /dev/ufs/pfsense0: UNEXPECTED SOFT UPDATE INCONSISTENCY; RUN fsck MANUALLY.There are strong indications that SU+J is a steaming pile of shit, such as:
https://forums.freebsd.org/threads/softupdate-with-journaling-decrease-reliability.39125/
http://lists.freebsd.org/pipermail/freebsd-fs/2014-April/019253.htmlPretty much summarised here: https://marc.info/?l=freebsd-arm&m=142526289401012&w=2 and I honestly have no idea what was the idea behind switching to it regarding filesystem corruption, since it's been noted multiple times on the bug that neither SU, nor SU+J solves anything there regarding the /etc files corruption. If fact, the soft updates thing does cause silent filesystem corruption/inconsistency, as linked above.
For years, never seen a power failure to cause any of those loads of nano boxes to explode in similar way. N.B., this is not limited to nano. The damned fdisk thing just worked behind the scenes, instead of telling you to run it manually and getting stuck on boot, rendering the system inaccessible. Can we undo the huge regresssion here, and while at it, undo other completely pointless and harmful changes (such as the SU+J thing) with it, since it apparently is not production ready (and besides that, it is generally NOT wanted on CF and similar media)?
P.S. Special link for gonzo, he's gonna love it… :P Soft updates, hard problems
-
@cmb:
There are no indications this has anything to do with SU+J.
What? You serious? :o
After about a thousand power cycles on nanobsd systems today, I qualify that with - "on full installs". :) It's definitely a problem on nano with some flash. It's definitely not on full installs, we've been through many thousands of cycles on SG-* systems with full installs, SU+J. The problems you're referencing re: journaling are largely dated.
SU+J makes slow flash very susceptible to issues that fsck needs to run a few more times to fix, if it's kept permanent rw mounted. You can power cycle a ro nano SU+J system non stop at least 100 straight times with no issues. Make it rw, and on some slow flash media, you have about a 50% shot of fsck not fixing issues without running it manually ~3 times.
I have 3 systems in a power cycle loop at the moment. Letting that run overnight, then we'll have a better idea of how resilient that is.
What I'm testing is basically what doktornotor posted above to remove SU+J, plus the most recent 2.2.4 snapshot which has a fsync fix to include all of /cf/conf/.
-
P.S. Special link for gonzo, he's gonna love it… :P Soft updates, hard problems
gonzo said that you have opinions with which he disagrees.
And some more choice words that he'll have to re-sign up to the forum to share. :)
-
@cmb:
The problems you're referencing re: journaling are largely dated.
Yeah, like the March 2015 thread. Shhhhhhhhhhh, the issue doesn't exist until we admit it. ::)
@cmb:
SU+J makes slow flash very susceptible to issues that fsck needs to run a few more times to fix, if it's kept permanent rw mounted. You can power cycle a ro nano SU+J system non stop at least 100 straight times with no issues. Make it rw, and on some slow flash media, you have about a 50% shot of fsck not fixing issues without running it manually ~3 times.
I'd LOVE to keep nano read-only like it was for years without any problem. It's just not workable at all any more, due to the removal of the "harmful" patch that never harmed anyone… unless you put that patch back, you can completely scratch read-only, because things are written with assumption that RO remount takes a couple of seconds at worst. Not minutes.
And yeah, people treat these things like it's perfectly safe to pull the plug pretty much at any time. These boxes are used by people who paid to get a replacement for shitty CPEs supplied by their ISPs. Not by IT staff. You generally do NOT even want to give out GUI access to them to reboot things gracefully, or explain such things to them. When they do some cleaning, they pull the plug if needed. When they have internet issues, they pull the plug to see if it fixes them, before calling anyone... Etc.
Spent half of the night remotely doing the "kill SU+J" drill on some 30 boxes. Thank you very much. Plus, the neighbour brought his broken Alix here this morning. Guess what I've seen when I plugged the null modem cable:
/dev/ufs/pfsense1: UNEXPECTED SU+J INCONSISTENCY /dev/ufs/pfsense1: INTERNAL ERROR: GOT TO reply() /dev/ufs/pfsense1: UNEXPECTED SOFT UPDATE INCONSISTENCY; RUN fsck MANUALLY.Journal my ass. >:( >:( >:( Plus, what's up with the useless fsck that shows you the middle-finger salute and tells you to run it manually instead? >:( >:( >:(
-
Just to note that this is just happened on Alix nano fresh install. filesystem was mounted rw at the time of power failure.
Trying to mount root from ufs:/dev/ufs/pfsense0 [ro,sync,noatime]... WARNING: / was not properly dismounted Configuring crash dumps... /dev/ufs/pfsense0: 2389817 DUP I=150631 /dev/ufs/pfsense0: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY. Filesystems are clean, continuing... Mounting filesystems... WARNING: /cf was not properly dismounted mount: /dev/ufs/pfsense0: R/W mount of / denied. Filesystem is not clean - run fsck.: Operation not permitted mount: /dev/ufs/cf: R/W mount of /cf denied. Filesystem is not clean - run fsck.: Operation not permitted Setting up memory disks... done. pwd_mkdb: /etc//pwd.db.tmp: Read-only file system ___ ___/ f \ / p \___/ Sense \___/ \ \___/ Welcome to pfSense 2.2.4-RELEASE ... cp: /var/db/pbi/keys/pfSense-BETA.ssl: Read-only file system Creating symlinks......ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib a.out ldconfig path: /usr/lib/aout /usr/lib/compat/aout done. cap_mkdb: /etc/login.conf.db: Read-only file system >>> Under 512 megabytes of ram detected. Not enabling APC. ^CAug 26 13:50:57 init: /bin/sh on /etc/rc terminated abnormally, going to single user mode Enter full pathname of shell or RETURN for /bin/sh: # sbin/fsck -p -y /dev/ufs/pfsense0 /dev/ufs/pfsense0: 2389817 DUP I=150631 /dev/ufs/pfsense0: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY. # /sbin/fsck -y /dev/ufs/pfsense0 ** /dev/ufs/pfsense0 ** Last Mounted on / ** Root file system ** Phase 1 - Check Blocks and Sizes 2389817 DUP I=150631 2389818 DUP I=150631 INCORRECT BLOCK COUNT I=191239 (80 should be 0) CORRECT? yes ** Phase 1b - Rescan For More DUPS 2389817 DUP I=150236 2389818 DUP I=150236 ** Phase 2 - Check Pathnames DUP/BAD I=150631 OWNER=root MODE=100644 SIZE=685 MTIME=Aug 26 12:19 2015 FILE=/usr/local/etc/php.ini REMOVE? yes DUP/BAD I=150236 OWNER=root MODE=100755 SIZE=483100 MTIME=Jul 26 13:34 2015 FILE=/usr/local/lib/libpcre.so.1 REMOVE? yes ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts UNREF FILE I=9804 OWNER=root MODE=100600 SIZE=0 MTIME=Aug 26 12:19 2015 RECONNECT? yes NO lost+found DIRECTORY CREATE? yes BAD/DUP FILE I=150236 OWNER=root MODE=100755 SIZE=483100 MTIME=Jul 26 13:34 2015 CLEAR? yes BAD/DUP FILE I=150631 OWNER=root MODE=100644 SIZE=685 MTIME=Aug 26 12:19 2015 CLEAR? yes LINK COUNT FILE I=191019 OWNER=root MODE=100600 SIZE=40960 MTIME=Aug 26 12:19 2015 COUNT 2 SHOULD BE 1 ADJUST? yes UNREF FILE I=191239 OWNER=root MODE=100600 SIZE=0 MTIME=Aug 26 12:19 2015 CLEAR? yes ** Phase 5 - Check Cyl groups FREE BLK COUNT(S) WRONG IN SUPERBLK SALVAGE? yes SUMMARY INFORMATION BAD SALVAGE? yes BLK(S) MISSING IN BIT MAPS SALVAGE? yes 10600 files, 508691 used, 3271337 free (1161 frags, 408772 blocks, 0.0% fragmentation) ***** FILE SYSTEM STILL DIRTY ***** ***** FILE SYSTEM WAS MODIFIED ***** ***** PLEASE RERUN FSCK ***** # /sbin/fsck -y /dev/ufs/pfsense0 ** /dev/ufs/pfsense0 ** Last Mounted on / ** Root file system ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups FREE BLK COUNT(S) WRONG IN SUPERBLK SALVAGE? yes SUMMARY INFORMATION BAD SALVAGE? yes BLK(S) MISSING IN BIT MAPS SALVAGE? yes 10600 files, 508691 used, 3271337 free (1161 frags, 408772 blocks, 0.0% fragmentation) ***** FILE SYSTEM MARKED CLEAN ***** ***** FILE SYSTEM WAS MODIFIED ***** -
The fsck is a pile of garbage. You can move to https://bugs.freebsd.org/bugzilla/, file a bug and wait a couple of years. Don't expect any fixes here.