Odd sudden kernel panic

Anaerin

Just had a crash and kernel panic, my first after having used pfSense for years on a machine that's been deployed and running without issue for at least 3 years now.

Crash report begins.  Anonymous machine information:

amd64
15.0-CURRENT
FreeBSD 15.0-CURRENT #1 RELENG_2_8_0-n256081-401ec5f685b9: Wed May 21 23:53:51 UTC 2025     root@freebsd:/var/jenkins/workspace/pfSense-CE-snapshots-2_8_0-main/obj/amd64/0q9vjGjc/var/jenkins/workspace/pfSense-CE-snapshots-2_8_0-main/sources/FreeBSD-src-RE

Crash report details:

No PHP errors found.

Filename: /var/crash/info.0
Dump header from device: /dev/label/swap0
  Architecture: amd64
  Architecture Version: 4
  Dump Length: 77312
  Blocksize: 512
  Compression: none
  Dumptime: 2025-07-14 14:35:50 -0600
  Hostname: router.localdomain
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 15.0-CURRENT #1 RELENG_2_8_0-n256081-401ec5f685b9: Wed May 21 23:53:51 UTC 2025
    root@freebsd:/var/jenkins/workspace/pfSense-CE-snapshots-2_8_0-main/obj/amd64/0q9vjGjc/var/jenkins
  Panic String: ffs_blkfree_cg: freeing free frag
  Dump Parity: 588138274
  Bounds: 0
  Dump Status: good

I have the full crash dumps as saved, anything I can do to help solve or report this? Looks like it's a filesystem issue?

stephenw10

We need the backtrace to know more really. But ffs_blkfree_cg does look like a filesystem or disk issue.

I'd run a SMART test at a minimum.

Can you upload the full crash report? https://nc.netgate.com/nextcloud/s/QEEnzs5dws2Rr9G

Anaerin

@stephenw10 Crash report uploaded. The drive itself is a RAID1 of 2 disks in mirror configuration, handled by the Dell RAID controller, so SMART information isn't available unfortunately.

stephenw10

OK since you're running UFS the first thing I would do is run a full filesystem check. It's probably just a filesystem error:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/filesystem-check.html#manual-filesystem-check

But I don't like the look of these logs at boot:

(probe48:mpt0:1:1:0): INQUIRY. CDB: 12 00 00 00 24 00 
(probe48:mpt0:1:1:0): CAM status: Unrecoverable Host Bus Adapter Error
(probe48:mpt0:1:1:0): Retrying command, 3 more tries remain
(probe0:mpt0:0:0:0): REPORT LUNS. CDB: a0 00 00 00 00 00 00 00 00 10 00 00 
(probe0:mpt0:0:0:0): CAM status: SCSI Status Error
(probe0:mpt0:0:0:0): SCSI status: Check Condition
(probe0:mpt0:0:0:0): SCSI sense: ILLEGAL REQUEST asc:ffffffff,ffffffff (Reserved ASC/ASCQ pair)
(probe0:mpt0:0:0:0): Error 22, Unretryable error
(probe48:mpt0:1:1:0): INQUIRY. CDB: 12 00 00 00 24 00 
(probe48:mpt0:1:1:0): CAM status: Unrecoverable Host Bus Adapter Error
(probe48:mpt0:1:1:0): Retrying command, 2 more tries remain
(probe48:mpt0:1:1:0): INQUIRY. CDB: 12 00 00 00 24 00 
(probe48:mpt0:1:1:0): CAM status: Unrecoverable Host Bus Adapter Error
(probe48:mpt0:1:1:0): Retrying command, 1 more tries remain
(probe48:mpt0:1:1:0): INQUIRY. CDB: 12 00 00 00 24 00 
(probe48:mpt0:1:1:0): CAM status: Unrecoverable Host Bus Adapter Error
(probe48:mpt0:1:1:0): Retrying command, 0 more tries remain
Trying to mount root from ufs:/dev/ufsid/65de6ac365342243 [rw,noatime]...
(probe48:mpt0:1:1:0): INQUIRY. CDB: 12 00 00 00 24 00 
(probe48:mpt0:1:1:0): CAM status: Unrecoverable Host Bus Adapter Error
(probe48:mpt0:1:1:0): Error 5, Retries exhausted

That's all output from the drive controller driver. It does still successfully mount root after that though.

I would also consider reinstalling as ZFS. And potentially just passing the drives through individually and using a ZFS mirror if that's possible.

Anaerin

@stephenw10 I believe that is mpt attempting to talk to the RAID card as if it was in IT mode, trying to count the individual drives ("REPORT LUNS"), and the card replying "No, this is RAID, you can't talk to the drives directly" ("ILLEGAL REQUEST").

I'll run a fs check next time it's convenient to take down the entire network. Probably this evening.