BOOT problem ZFS / NVME SSD - nvme0: System interrupt issues?

stephenw10

Oh it actually shows: nvme0: System interrupt issues? at the warm boot? That seems like a huge clue if so.
https://man.freebsd.org/cgi/man.cgi?query=nvme&sektion=4&manpath=FreeBSD+15.0-CURRENT#DIAGNOSTICS

Try booting verbose at the reboot see if that gives you more. Interrupt boot at the loader menu then at the OK> prompt enter: boot -v

Sometime the extra delay caused by booting verbose can actually pass whatever error is present. Which is inconvenient for troubleshooting!

mer

@ramup possibly. A reset line not physically routed to the right place. But again, sometimes this can be worked around by have software reinitializing registers.

stephenw10

Mmm I could believe something missing in the adapter could present like this.

I'd try forcing legacy interrupts for nvme.

ramup

@stephenw10
Thank you for the hint to FreeBSD man page. I did not even find that with Google with search "system interrupt issues?"

Can you tell me what you technically mean by "try forcing legacy interrupts for nvme"?

@mer
Thank you also for your input, but I also do not understand technically how to do "software reinitializing registers"?

ramup

@stephenw10
Do you mean:

https://man.freebsd.org/cgi/man.cgi?nvme(4)

setting:

hw.nvme.force_intx=1

in in /boot/loader.conf.local ?

stephenw10

Create the file /boot/loader.conf.local

Add the following line to it:
hw.nvme.force_intx=1

Reboot.

I've never had to try that so you may find you have to disable that again at the loader prompt if it fails to boot.

mer

@ramup said in BOOT problem ZFS / NVME SSD - Root mount waiting for CAM:

Thank you also for your input, but I also do not understand technically how to do "software reinitializing registers"?

That would be the OS/driver level. OS as it goes through boot sequence, OS detects a piece of hardware (ram, usb, pci, whatever), OS knows (because of magic) warm/cold boot, decides to set registers or not. Values in sysctls can affect this behavior.
So not a "pfSense problem" but maybe a FreeBSD issue (pfSense is built on top of FreeBSD), but reporting it here because this is where you ran across it here lets the pfSense team figure out what to do.

ramup

@stephenw10 + @mer

Thank you for your help and guidance. I will start to try with the extra boot command and if that does not work I will search for the possibilities to reinitialize registers via sysctl.

I will report back!

ramup

Feedback:

hw.nvme.force_intx=1

in

/boot/loader.conf.local

solved the issue!

Many thanks for your help!

I renamed the thread in case someone searches for it.

stephenw10

Nice, good result!

mer

@stephenw10 I agree @ramup thanks for keeping everyone in the loop