Force fsck on a secondary drive installed on the system every time at boot
-
Hi all,
I understand most people here will have a single disk setup. Therefore, when the system hard crashes (e.g. due to power failure), pfSense's default repair behaviour would go without issue (ufs is subject to fsck, and zfs (almost) does not corrupt so no issue there).
I have a double disk setup where the secondary disk is used for, 1) log storage, 2) squid cache.
The problem I have now is that, when the system hard crashes, I always need to intervene during the reboot process (i.e. enter single user mode, then run fsck -fy /(path to secondary disk), before the system will reboot normally). Otherwise, the system will tell me "ERROR: Impossible to mount filesystem, use interactive shell to attempt to recover it".
This is quite annoying so I am trying to figure out how to force fsck -fy /(path to secondary disk) as part of an autoexec/autorun script early enough in the boot process (it is quite fast anyway as it is a small drive).
Shellcmd does not work as the error is triggered during the kernel/driver loading process long before the userland is started.
I have looked generally at the freebsd forums but it appears that the files that were edited (e.g. rc.local, rc.d) don't operate in the same way as they do in pfsense.
I have read the fsck man page and seems like it is invoking in /etc/rc, but /etc/rc points to /etc/pfSense-rc. I am presuming this is the file to amend?
If so, where should I insert the proposed fsck line (i.e. fsck -fy /(path to secondary disk))? I have flagged a couple of lines below in bold, is it amongst one of those two?
# Set it to 0 if it's empty fsck_fix_count=${fsck_fix_count:-0} FSCK_ACTION_NEEDED=0 /sbin/fsck -p case $? in 0) echo "Filesystems are clean, continuing..." echo "Mounting filesystems..." ;; 8|16) echo "Preen mode recommended running a check that will be performed now." FSCK_ACTION_NEEDED=1 ;; *) echo "Stopping boot is recommended because filesystem manual action is needed, nevertheless automated repair of the filesystem will be attempted." FSCK_ACTION_NEEDED=1 ;; esac if [ ${FSCK_ACTION_NEEDED} = 1 ]; then echo "WARNING: Trying to recover filesystem from inconsistency..." **/sbin/fsck -fy -t ufs** fi /sbin/mount -a 2>/dev/null mount_rc=$? attempts=0 while [ "${mount_rc}" -ne 0 ] && [ "${attempts}" -lt 10 ]; do **/sbin/fsck -fy -t ufs** /sbin/mount -a 2>/dev/null mount_rc=$? attempts=$((attempts+1)) done if [ ${mount_rc} -ne 0 ]; then echo "ERROR: Impossible to mount filesystem, use interactive shell to attempt to recover it" /bin/sh /sbin/reboot fi -
@it_geek Maybe you should be spending your time trying to figure out why your firewall is crashing so frequently that you need to do this in the first place. A small UPS isn't that expensive.
-
You have the 2nd drive defined in the fstab? Then 'mount -a' should fail and fsck will be run. So yes in that loop somewhere is where you should enter it.
But, yes, why is it hard rebooting? And also, yes, that's unsupported!
-
@KOM said in Force fsck on a secondary drive installed on the system every time at boot:
@it_geek Maybe you should be spending your time trying to figure out why your firewall is crashing so frequently that you need to do this in the first place. A small UPS isn't that expensive.
I live in a home where people appear to take labels on power plugs as... suggestions. Been educating them so I hope they will stop turning off the wrong switch.
Recently I also have had NIC driver issues which prevent a clean shutdown, so a UPS won't solve this either.
@stephenw10 said in Force fsck on a secondary drive installed on the system every time at boot:
You have the 2nd drive defined in the fstab? Then 'mount -a' should fail and fsck will be run. So yes in that loop somewhere is where you should enter it.
But, yes, why is it hard rebooting? And also, yes, that's unsupported!
Yes, the drive in defined in /etc/fstab. It works because Squid has not had any issue running or it would have been throwing a bunch of errors.
Hard rebooting for the reasons I specified in the reply to KOM's post.
I was thinking of putting that line above the loop so that it will run regardless of whether the condition of the loop is met, will try it later when i get home.
Sorry I don't understand what you were referring to when you said that it was unsupported.
-
@it_geek said in Force fsck on a secondary drive installed on the system every time at boot:
I live in a home where people appear to take labels on power plugs as... suggestions.
Ha, well that's an issue.
In unsupported in that pfSense has nothing included to support multiple drives other than in a mirror.
Interesting. The fsck command run by pfSense does not specify a mount point so I'd expect it to run on everything in the fstab. I may be that it is but needs to run multiple times.
But, yes, you should be able to add a specific fsck command there for the 2nd drive.In the past other users have done this by using their own script to mount the drive at boot. That can then run fsck on it. There are some threads detailing that here somewhere.
-
@it_geek I live in a home where people appear to take labels on power plugs as... suggestions.
I feel for you, brother. With some people, carrots do not work and so the stick is what you have left to deal with them -- they require pain in order to learn. If they ignore you and cause the firewall to crash, maybe they can do without Internet for the next day or so until they agree to stop being so stupid.
-
@stephenw10 said in Force fsck on a secondary drive installed on the system every time at boot:
@it_geek said in Force fsck on a secondary drive installed on the system every time at boot:
I live in a home where people appear to take labels on power plugs as... suggestions.
Ha, well that's an issue.
In unsupported in that pfSense has nothing included to support multiple drives other than in a mirror.
Interesting. The fsck command run by pfSense does not specify a mount point so I'd expect it to run on everything in the fstab. I may be that it is but needs to run multiple times.
But, yes, you should be able to add a specific fsck command there for the 2nd drive.In the past other users have done this by using their own script to mount the drive at boot. That can then run fsck on it. There are some threads detailing that here somewhere.
@KOM said in Force fsck on a secondary drive installed on the system every time at boot:
@it_geek I live in a home where people appear to take labels on power plugs as... suggestions.
I feel for you, brother. With some people, carrots do not work and so the stick is what you have left to deal with them -- they require pain in order to learn. If they ignore you and cause the firewall to crash, maybe they can do without Internet for the next day or so until they agree to stop being so stupid.
Haha. We will see.
Anyway, I managed to find the fix. Edit this section in /etc/pfSense-rc: Do some basic search in the code and insert the code i wrote in lines 4 - 6.
fsck will always run on the secondary disk on boot regardless of whether the boot is clean or if resuming from some other system failure.
# Set it to 0 if it's empty fsck_fix_count=${fsck_fix_count:-0} # Force a fsck on the secondary disk, regardless of whether it is clean or not echo STARTING CHECK ON SECONDARY DISK, JUST IN CASE. PLEASE WAIT. /sbin/fsck -fy /(mount point to second disk) FSCK_ACTION_NEEDED=0 /sbin/fsck -p case $? in 0) echo "Filesystems are clean, continuing..." echo "Mounting filesystems..." ;; 8|16) echo "Preen mode recommended running a check that will be performed now." FSCK_ACTION_NEEDED=1 ;; *) echo "Stopping boot is recommended because filesystem manual action is needed, nevertheless automated repair of the filesystem will be attempted." FSCK_ACTION_NEEDED=1 ;; esac if [ ${FSCK_ACTION_NEEDED} = 1 ]; then echo "WARNING: Trying to recover filesystem from inconsistency..." /sbin/fsck -fy -t ufs fi -
@it_geek said in Force fsck on a secondary drive installed on the system every time at boot:
people appear to take labels on power plugs as... suggestions.
One way a good UPS solves this is by the constant "power out beep".. Just keep the display hard to reach so they can't silence it.
