Using "find /" brings pfSense down

NickJH

I was having big problems with pfSense yesterday and managed to get ssh access. I went in search of the backups so I could try to download them. I used the command find / -name config-pfSense.howitts.co.uk-20250317222358.xml but ran into serious problems. pfSense is on an SSD so should be quick, but the command never returned. I could not interrupt it with a ^c and I could not open any other ssh sessions to terminate the process. After a while (perhaps 2 hours) pfSense stops routing and requires a full reboot.

What am I doing wrong and how should I be looking for files?

patient0

@NickJH said in Using "find /" brings pfSense down:

What am I doing wrong and how should I be looking for files?
The current config is stored in the /conf/ directory and the previous versions in /conf/backup.

How did you get the name? That is a name I would expect if you save the config from the GUI to your computer.

In /conf/backup the files on my pfSense are name /conf/backup/config-<Unix time>.xml.

Nevertheless the find should never lockup the system. Is the filesystem clean (UFS or ZFS?) ?

NickJH

@patient0 The name was from a download I had, but it was just representative. I tried other names that I was looking for yesterday and find also locked up.

I an using zfs. I am not familiar with zfs, but I've just tried:

[2.7.2-RELEASE][root@pfSense.howitts.co.uk]/root: zpool scrub pfSense
[2.7.2-RELEASE][root@pfSense.howitts.co.uk]/root: zpool status -v pfSense
  pool: pfSense
 state: SUSPENDED
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool clear'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-HC
  scan: scrub in progress since Fri Apr 18 10:57:11 2025
        1.64G / 1.64G scanned, 1007M / 1.64G issued at 67.1M/s
        0B repaired, 59.92% done, 00:00:10 to go
config:

        NAME        STATE     READ WRITE CKSUM
        pfSense     ONLINE       0     0     0
          ada0p4    ONLINE     207    33   410

errors: List of errors unavailable: pool I/O is currently suspended

How should I proceed?

NickJH

@NickJH Hmm. Now:

[2.7.2-RELEASE][root@pfSense.howitts.co.uk]/root: zpool status -v pfSense
  pool: pfSense
 state: SUSPENDED
status: One or more devices are faulted in response to IO failures.
action: Make sure the affected devices are connected, then run 'zpool clear'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-HC
  scan: scrub in progress since Fri Apr 18 10:57:11 2025
        1.64G / 1.64G scanned, 1007M / 1.64G issued at 1.98M/s
        0B repaired, 59.92% done, no estimated completion time
config:

        NAME        STATE     READ WRITE CKSUM
        pfSense     ONLINE       0     0     0
          ada0p4    ONLINE     207    33   410

errors: List of errors unavailable: pool I/O is currently suspended

Stuck at 59.92%?

patient0

@NickJH I don't have experience with ZFS repair. Hopefully someone else can chime in.

If you search the internet for 'pfSense zfs suspended` it returns quite a few results.

What does SMART pfSense Doc: S.M.A.R.T. Hard Disk Status report for your disk? ( reddit: How can I check the health, run diagnostics on the SSD in my pfSense machine? )? Is it in a health state?

NickJH

@patient0 SMART data looks OK. I've performed a Short Test and it was fine:

SMART Extended Self-test Log Version: 1 (1 sectors)
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      8769         -

is it worth doing a long test?

I did a forced power-off/power-on (as the console was unresponsive) then a controlled reboot before doing those tests.

Zpool status doesn't look good:

[2.7.2-RELEASE][root@pfSense.howitts.co.uk]/root: zpool status -v pfSense
  pool: pfSense
 state: ONLINE
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub repaired 0B in 00:41:55 with 29 errors on Fri Apr 18 11:39:06 2025
config:

        NAME        STATE     READ WRITE CKSUM
        pfSense     ONLINE       0     0     0
          ada0p4    ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        /var/cache/pkg/pfSense-base-2.7.2~0135c9e036.pkg
        //usr/local/share/pfSense/base.txz
        //usr/sbin/wpa_supplicant
        //lib/libzpool.so.2
        //usr/local/include/boost/fusion/algorithm/transformation/detail/preprocessed/zip50.hpp
        //usr/local/include/php/Zend/zend_weakrefs.h
        //usr/bin/ztest
        //usr/lib/clang/15.0.7/lib/freebsd/libclang_rt.fuzzer-x86_64.a
        //usr/lib/libpmc.so.5
        //usr/local/lib/libpkg.so.4
        //usr/local/share/examples/dnsmasq/dnslist/dnslist.pl
        //usr/local/lib/libboost_test_exec_monitor.a
        //usr/share/syscons/fonts/koi8-rb-8x16.fnt
        //usr/local/lib/perl5/5.34/mach/CORE/libperl.so.5.34.1
        //usr/lib/clang/16/lib/freebsd/libclang_rt.stats-x86_64.a
        //usr/local/include/boost/fusion/container/vector/detail/cpp03/vector40.hpp
        //usr/lib/libc.a
        //usr/local/lib/python3.11/xml/dom/__pycache__/expatbuilder.cpython-311.opt-2.pyc
        //usr/lib/librpcsvc.a
        //usr/include/xlocale/_stdlib.h

If I can't find a way of repairing it, I'll investigate a reload and restore. I have a recent configuration backup.

NickJH

@NickJH Hmm. This is my gateway but pfSense only have an online installer. With some searching, I've found a link to the old offline installer - https://forum.netgate.com/topic/188105/ce-edition-download/9. I hope it is good.

Could I install onto another system and copy the corrupt files across? I am not sure it is very safe to do that.

patient0

@NickJH said in Using "find /" brings pfSense down:

With some searching, I've found a link to the old offline installer - https://forum.netgate.com/topic/188105/ce-edition-download/9. I hope it is good.

They are still good, the offline 2.7.2 installer, yes.

Could I install onto another system and copy the corrupt files across?

A reinstall with the offline installer will be faster I'd assume then copying file over from another installation (never tried that).

Have you had power cuts or multiple forced reboot? If yes, then that is a reason why the file system gets corrupted.
If no then you really have to low-level check of the SSD.

NickJH

@patient0 I have had multiple forced reboots. I am using a Chinese N100 system, but I don't know how relevant that is. I have just changed from Static to DHCP and that didn't go well yesterday. I did a few hard resets in order to try to get online, but I think I had the problem before that. Some of the hard resets hung and had to be reset again.

What is the best way of doing a low level check of the SSD (mSATA)? A long SMART test?

SteveITS

@NickJH FYI in the online installer one configures WAN and it downloads what it needs.

If you think the disk is fine I’d reinstall. However I’d also check for memory problems because it sounds like the data on disk was corrupted somehow.

As https://docs.netgate.com/pfsense/en/latest/troubleshooting/filesystem-check.html notes there isn’t a file system repair for ZFS as it’s not typically needed.

If pfSense is running it has SMART tests in https://docs.netgate.com/pfsense/en/latest/monitoring/status/smart.html.

stephenw10

Yeah that does seem like a drive issue to me, find should never lock like that. Or take very long at all:

[2.7.2-RELEASE][admin@cedev-3.stevew.lan]/root: time find / -name config-pfSense.howitts.co.uk-20250317222358.xml
0.029u 0.313s 0:00.95 34.7%	39+184k 3143+0io 2pf+0w

NickJH

@stephenw10 Thanks.

The long SMART test was clean, so, once my son is back at school next week, I'll try a reinstall, bootstrapping it with my latest config.xml - https://docs.netgate.com/pfsense/en/latest/backup/restore-during-install.html.

When I take it down I'll check the seating of the mSATA before reloading the software. If it fails again, I'll look at replacing the disk.

NickJH

@stephenw10 A reload of the firewall seems to have done the trick. Now finds are fast.

One thing. The reload failed to restore the config, perhaps because I removed the USB pen too early, so I rebooted with the config in /config/config.xml and that restored fine. I then removed the USB pen and pfSense crashed. I could not find the USB pen mounted anywhere so I'd have thought it safe to remove, but it seems I was wrong. What should I have done to remove it correctly?

[edit]
Found it. usbconfig
I was used to the Linux way of working.
[/edit]

stephenw10

Hmm, odd. Normally it would not be mounted. And even if it was it shouldn't really care unless it was actually running from USB.

NickJH

@stephenw10 I wonder if it was me putting the router back into place which disturbed something rather than removing the USB. I only noticed when I went back downstairs. A mystery and I don't want to try repeating it.

stephenw10

If it ain't broke....