BOOT problem ZFS / NVME SSD - nvme0: System interrupt issues?
-
Hello,
I have a problem with booting pfSense that I am unable to solve.
I use a Fujitsu FUTRO S9010 ThinClient for pfsense equipped with Intel i350 quad port card + 1 LAN onboard, latest BIOS installed.
On the WiFi M.2 Slot is a WD SN520 NVMe SSD installed via PCIE Adapter -> A-E-KEY to M-Key.
On this NVMe SSD pfSense is installed as ZFS Single Disk.
The problem only occurs on warm reboots, not when the PC is powered off and turned on again.
This is dmesg from a cold reboot (where boot finishes):
---<<BOOT>>--- Copyright (c) 1992-2024 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 15.0-CURRENT #21 RELENG_2_8_1-n256095-47c932dcc0e9: Thu Aug 28 16:27:48 UTC 2025 root@pfsense-build-release-amd64-1.eng.atx.netgate.com:/var/jenkins/workspac e/pfSense-CE-snapshots-2_8_1-main/obj/amd64/AupY3aTL/var/jenkins/workspace/pfSen se-CE-snapshots-2_8_1-main/sources/FreeBSD-src-RELENG_2_8_1/amd64.amd64/sys/pfSe nse amd64 FreeBSD clang version 19.1.5 (https://github.com/llvm/llvm-project.git llvmorg-1 9.1.5-0-gab4b5a2db582) VT(vga): resolution 640x480 CPU: Intel(R) Pentium(R) Silver J5040 CPU @ 2.00GHz (1996.80-MHz K8-class CPU) Origin="GenuineIntel" Id=0x706a8 Family=0x6 Model=0x7a Stepping=8 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,C MOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x4ff8ebbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,C X16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,RDRAN D> AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM> AMD Features2=0x101<LAHF,Prefetch> Structured Extended Features=0x2294e287<FSGSBASE,TSCADJ,SGX,SMEP,ERMS,NFPUSG,M PX,PQE,RDSEED,SMAP,CLFLUSHOPT,PROCTRACE,SHA> Structured Extended Features2=0x40400004<UMIP,RDPID,SGXLC> Structured Extended Features3=0xac000400<MD_CLEAR,IBPB,STIBP,ARCH_CAP,SSBD> XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES> IA32_ARCH_CAPS=0x14000c6b<RDCL_NO,IBRS_ALL,SKIP_L1DFL_VME,MDS_NO> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr TSC: P-state invariant, performance statistics real memory = 17179869184 (16384 MB) avail memory = 16287608832 (15533 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: <INTEL GLK-SOC > WARNING: L1 data cache covers fewer APIC IDs than a core (0 < 1) FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) random: registering fast source Intel Secure Key RNG random: fast provider: "Intel Secure Key RNG" random: unblocking device. ioapic0 <Version 2.0> irqs 0-119 Launching APs: 2 3 1 TCP_ratelimit: Is now initialized ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LIC ENSE. ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /bo ot/loader.conf. module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff80758a00, 0) error 1 ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw.LI CENSE. ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /b oot/loader.conf. module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff80758ab0, 0) error 1 ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw .LICENSE. ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff80758b60, 0) error 1 iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LIC ENSE. iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /bo ot/loader.conf. module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff807781d0, 0) error 1 iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi.LI CENSE. iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /b oot/loader.conf. module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff80778280, 0) error 1 iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi .LICENSE. iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff80778330, 0) error 1 random: entropy device external interface wlan: mac acl policy registered kbd1 at kbdmux0 WARNING: Device "spkr" is Giant locked and may be deleted before FreeBSD 15.0. efirtc0: <EFI Realtime Clock> efirtc0: registered as a time-of-day clock, resolution 1.000000s netgate0: <unknown hardware> netgate0: version: 0.1 smbios0: <System Management BIOS> at iomem 0x6ad98000-0x6ad98017 smbios0: Version: 3.2 acpi0: <FUJ D3543-A2> acpi0: Power Button (fixed) unknown: I/O range not supported cpu0: <ACPI CPU> on acpi0 attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: <AT realtime clock> port 0x70-0x77 on acpi0 atrtc0: Warning: Couldn't map I/O. atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 8 on acpi0 Timecounter "HPET" frequency 19200000 Hz quality 950 Event timer "HPET" frequency 19200000 Hz quality 550 Event timer "HPET1" frequency 19200000 Hz quality 440 Event timer "HPET2" frequency 19200000 Hz quality 440 Event timer "HPET3" frequency 19200000 Hz quality 440 Event timer "HPET4" frequency 19200000 Hz quality 440 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 pcib0: no PRT entry for 0.0.INTA vgapci0: <VGA-compatible display> port 0xf000-0xf03f mem 0xa0000000-0xa0ffffff,0 x90000000-0x9fffffff at device 2.0 on pci0 vgapci0: Boot video device pci0: <simple comms> at device 15.0 (no driver attached) ahci0: <Intel Gemini Lake AHCI SATA controller> port 0xf090-0xf097,0xf080-0xf083 ,0xf060-0xf07f mem 0xa1710000-0xa1711fff,0xa172a000-0xa172a0ff,0xa1729000-0xa172 97ff at device 18.0 on pci0 ahci0: AHCI v1.31 with 2 6Gbps ports, Port Multiplier supported ahcich0: <AHCI channel> at channel 0 on ahci0 ahcich1: <AHCI channel> at channel 1 on ahci0 pcib1: <ACPI PCI-PCI bridge> at device 19.0 on pci0 pci1: <ACPI PCI bus> on pcib1 igb0: <Intel(R) I350 (Copper)> mem 0xa1300000-0xa13fffff,0xa140c000-0xa140ffff a t device 0.0 on pci1 igb0: EEPROM V1.63-0 eTrack 0x80000c52 igb0: Using 1024 TX descriptors and 1024 RX descriptors igb0: Using 4 RX queues 4 TX queues igb0: Using MSI-X interrupts with 5 vectors igb0: Ethernet address: a0:36:9f:7f:7d:c8 igb0: netmap queues/slots: TX 4/1024, RX 4/1024 igb1: <Intel(R) I350 (Copper)> mem 0xa1200000-0xa12fffff,0xa1408000-0xa140bfff a t device 0.1 on pci1 igb1: EEPROM V1.63-0 eTrack 0x80000c52 igb1: Using 1024 TX descriptors and 1024 RX descriptors igb1: Using 4 RX queues 4 TX queues igb1: Using MSI-X interrupts with 5 vectors igb1: Ethernet address: a0:36:9f:7f:7d:c9 igb1: netmap queues/slots: TX 4/1024, RX 4/1024 igb2: <Intel(R) I350 (Copper)> mem 0xa1100000-0xa11fffff,0xa1404000-0xa1407fff a t device 0.2 on pci1 igb2: EEPROM V1.63-0 eTrack 0x80000c52 igb2: Using 1024 TX descriptors and 1024 RX descriptors igb2: Using 4 RX queues 4 TX queues igb2: Using MSI-X interrupts with 5 vectors igb2: Ethernet address: a0:36:9f:7f:7d:ca igb2: netmap queues/slots: TX 4/1024, RX 4/1024 igb3: <Intel(R) I350 (Copper)> mem 0xa1000000-0xa10fffff,0xa1400000-0xa1403fff a t device 0.3 on pci1 igb3: EEPROM V1.63-0 eTrack 0x80000c52 igb3: Using 1024 TX descriptors and 1024 RX descriptors igb3: Using 4 RX queues 4 TX queues igb3: Using MSI-X interrupts with 5 vectors igb3: Ethernet address: a0:36:9f:7f:7d:cb igb3: netmap queues/slots: TX 4/1024, RX 4/1024 pcib2: <ACPI PCI-PCI bridge> at device 19.1 on pci0 pci2: <ACPI PCI bus> on pcib2 nvme0: <Generic NVMe Device> mem 0xa1600000-0xa1603fff at device 0.0 on pci2 pcib3: <ACPI PCI-PCI bridge> at device 19.2 on pci0 pci3: <ACPI PCI bus> on pcib3 re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xe000-0xe 0ff mem 0xa1500000-0xa1500fff,0x80100000-0x80103fff at device 0.0 on pci3 re0: Using 1 MSI-X message re0: ASPM disabled re0: Chip rev. 0x4c000000 re0: MAC rev. 0x00000000 miibus0: <MII bus> on re0 rgephy0: <RTL8251/8153 1000BASE-T media interface> PHY 1 on miibus0 rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX , 100baseTX-FDX-flow, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1 000baseT-FDX-flow-master, auto, auto-flow re0: Using defaults for TSO: 65518/35/2048 re0: Ethernet address: 68:84:7e:a4:02:69 re0: netmap queues/slots: TX 1/256, RX 1/256 pcib4: <ACPI PCI-PCI bridge> at device 19.3 on pci0 pci4: <ACPI PCI bus> on pcib4 pcib5: <ACPI PCI-PCI bridge> at device 20.0 on pci0 pci5: <ACPI PCI bus> on pcib5 pcib6: <ACPI PCI-PCI bridge> at device 20.1 on pci0 pci6: <ACPI PCI bus> on pcib6 xhci0: <Intel Gemini Lake USB 3.0 controller> mem 0xa1700000-0xa170ffff irq 17 a t device 21.0 on pci0 xhci0: 32 bytes context size, 64-bit DMA usbus0 on xhci0 usbus0: 5.0Gbps Super Speed USB v3.0 isab0: <PCI-ISA bridge> at device 31.0 on pci0 isa0: <ISA bus> on isab0 acpi_button0: <Power Button> on acpi0 acpi_tz0: <Thermal Zone> on acpi0 atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] est0: <Enhanced SpeedStep Frequency Control> on cpu0 cpufreq0: <CPU frequency control> on cpu0 cpufreq1: <CPU frequency control> on cpu1 cpufreq2: <CPU frequency control> on cpu2 cpufreq3: <CPU frequency control> on cpu3 Timecounter "TSC" frequency 1996800831 Hz quality 1000 Timecounters tick every 1.000 msec ugen0.1: <Intel XHCI root HUB> at usbus0 uhub0 on usbus0 uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Trying to mount root from zfs:pfSense/ROOT/default []... uhub0: 16 ports with 16 removable, self powered Root mount waiting for: CAM usbus0 ugen0.2: <vendor 0x0424 product 0x2514> at usbus0 uhub1 on uhub0 uhub1: <vendor 0x0424 product 0x2514, class 9/0, rev 2.00/b.b3, addr 1> on usbus 0 uhub1: MTT enabled uhub1: 4 ports with 4 removable, self powered Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM nda0 at nvme0 bus 0 scbus2 target 0 lun 1 nda0: <PC SN520 NVMe WDC 512GB 20220012 19490A800614> nda0: Serial Number 19490A800614 nda0: nvme version 1.3 nda0: 488386MB (1000215216 512 byte sectors) CPU: Intel(R) Pentium(R) Silver J5040 CPU @ 2.00GHz (1996.80-MHz K8-class CPU) Origin="GenuineIntel" Id=0x706a8 Family=0x6 Model=0x7a Stepping=8 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,C MOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x4ff8ebbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,C X16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,RDRAN D> AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM> AMD Features2=0x101<LAHF,Prefetch> Structured Extended Features=0x2294e287<FSGSBASE,TSCADJ,SGX,SMEP,ERMS,NFPUSG,M PX,PQE,RDSEED,SMAP,CLFLUSHOPT,PROCTRACE,SHA> Structured Extended Features2=0x40400004<UMIP,RDPID,SGXLC> Structured Extended Features3=0xac000400<MD_CLEAR,IBPB,STIBP,ARCH_CAP,SSBD> XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES> IA32_ARCH_CAPS=0x14000c6b<RDCL_NO,IBRS_ALL,SKIP_L1DFL_VME,MDS_NO> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr TSC: P-state invariant, performance statistics ig4iic0: <Intel Gemini Lake I2C Controller-0> mem 0xa1728000-0xa1728fff,0xa17270 00-0xa1727fff irq 27 at device 22.0 on pci0 ig4iic0: Using MSI iicbus0: <Philips I2C bus (ACPI-hinted)> on ig4iic0 ig4iic1: <Intel Gemini Lake I2C Controller-1> mem 0xa1726000-0xa1726fff,0xa17250 00-0xa1725fff irq 28 at device 22.1 on pci0 ig4iic1: Using MSI iicbus1: <Philips I2C bus (ACPI-hinted)> on ig4iic1 ig4iic2: <Intel Gemini Lake I2C Controller-2> mem 0xa1724000-0xa1724fff,0xa17230 00-0xa1723fff irq 29 at device 22.2 on pci0 ig4iic2: Using MSI iicbus2: <Philips I2C bus (ACPI-hinted)> on ig4iic2 ig4iic3: <Intel Gemini Lake I2C Controller-3> mem 0xa1722000-0xa1722fff,0xa17210 00-0xa1721fff irq 30 at device 22.3 on pci0 ig4iic3: Using MSI iicbus3: <Philips I2C bus (ACPI-hinted)> on ig4iic3 ig4iic4: <Intel Gemini Lake I2C Controller-4> mem 0xa1720000-0xa1720fff,0xa171f0 00-0xa171ffff irq 31 at device 23.0 on pci0 ig4iic4: Using MSI iicbus4: <Philips I2C bus (ACPI-hinted)> on ig4iic4 ig4iic5: <Intel Gemini Lake I2C Controller-5> mem 0xa171e000-0xa171efff,0xa171d0 00-0xa171dfff irq 32 at device 23.1 on pci0 ig4iic5: Using MSI iicbus5: <Philips I2C bus (ACPI-hinted)> on ig4iic5 ig4iic6: <Intel Gemini Lake I2C Controller-6> mem 0xa171c000-0xa171cfff,0xa171b0 00-0xa171bfff irq 33 at device 23.2 on pci0 ig4iic6: Using MSI iicbus6: <Philips I2C bus (ACPI-hinted)> on ig4iic6 ig4iic7: <Intel Gemini Lake I2C Controller-7> mem 0xa171a000-0xa171afff,0xa17190 00-0xa1719fff irq 34 at device 23.3 on pci0 ig4iic7: Using MSI iicbus7: <Philips I2C bus (ACPI-hinted)> on ig4iic7 ichsmb0: <Intel Gemini Lake SMBus controller> port 0xf040-0xf05f mem 0xa1712000- 0xa17120ff at device 31.1 on pci0 smbus0: <System Management Bus> on ichsmb0 aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS,SHA1,SHA256> wg0: changing name to 'tun_wg0' tun_wg0: link state changed to UP igb0: link state changed to UP igb1: link state changed to UP igb2: link state changed to UP igb3: link state changed to UP re0: link state changed to DOWN coretemp0: <CPU On-Die Thermal Sensors> on cpu0 lo0: link state changed to UP igb1: link state changed to DOWN vlan0: changing name to 'igb1.7' vlan1: changing name to 'igb0.10' igb0: link state changed to DOWN if_pppoe version fa94ad71dfbc275900de44b9a673f5c048968dfb igb3: link state changed to DOWN igb2: link state changed to DOWN ng0: changing name to 'pppoe0' igb1: link state changed to UP igb1.7: link state changed to UP igb2: link state changed to UP igb0: link state changed to UP igb0.10: link state changed to UP igb3: link state changed to UP pflog0: promiscuous mode enabled tun_wg0: link state changed to DOWN wg0: changing name to 'tun_wg0' tun_wg0: link state changed to UPOn a warm reboot the message:
Root mount waiting for: CAMspins forever without finishing booting.
What have I tried?
https://docs.netgate.com/pfsense/en/latest/troubleshooting/boot-issues.html
Although not being a USB boot device, because of the CAM messages I tried to play with the value:
kern.cam.boot_delay="10000" kern.cam.scsi_delay="10000"and even
kern.cam.boot_delay="30000" kern.cam.scsi_delay="30000"in /boot/loader.conf.local
but no differences.
camcontrol devlistshows when pfSense is running:
<PC SN520 NVMe WDC 512GB 20220012> at scbus2 target 0 lun 1 (pass0,nda0)In pfSense web interface S.M.A.R.T. Diagnostics, information is:
smartctl 7.4 2023-08-01 r5530 [FreeBSD 15.0-CURRENT amd64] (local build) Copyright (C) 2002-23, Bruce Allen, Christian Franke, www.smartmontools.org === START OF SMART DATA SECTION === Error Information (NVMe Log 0x01, 16 of 256 entries) No Errors Loggedand
=== START OF SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED SMART/Health Information (NVMe Log 0x02) Critical Warning: 0x00 Temperature: 49 Celsius Available Spare: 100% Available Spare Threshold: 50% Percentage Used: 11% Data Units Read: 37,726,245 [19.3 TB] Data Units Written: 36,642,209 [18.7 TB] Host Read Commands: 625,482,420 Host Write Commands: 1,129,156,302 Controller Busy Time: 2,188 Power Cycles: 1,020 Power On Hours: 7,905 Unsafe Shutdowns: 87 Media and Data Integrity Errors: 0 Error Information Log Entries: 0 Warning Comp. Temperature Time: 0 Critical Comp. Temperature Time: 0 Error Information (NVMe Log 0x01, 16 of 256 entries) No Errors Logged Self-test Log (NVMe Log 0x06) Self-test status: No self-test in progress Num Test_Description Status Power_on_Hours Failing_LBA NSID Seg SCT Code 0 Short Completed without error 7905 - - - - - 1 Short Completed without error 6688 - - - - - 2 Short Aborted: Self-test command 5079 - - - - - 3 Short Completed without error 24 - - - - - 4 Short Completed without error 10 - - - - -Does anyone have an idea how to solve this issue?
-
@ramup said in BOOT problem ZFS / NVME SSD - Root mount waiting for CAM:
nvme0: <Generic NVMe Device> mem 0xa1600000-0xa1603fff at device 0.0 on pci2
Do you still see that message in the boot log when it fails to mount root after rebooting?
How cold is the cold boot? Actually full power cycle? Or shutdown from the gui and boot with the power button?
-
My experience has been if there is a difference between warm reboot and cold (power cycle) reboot is "Device initialization difference". Warm reboot typically triggers a reset signal (hardware signal) that should get routed to any physical device (controllers) to cause them to reset. Sometimes vendors don't do that and you get problems.
Cold reboot takes power away and the software reinitializes everything it needs to.The check that @stephenw10 asks can tell you what "isn't" happening.
It can be worked around if the software always reinitializes everything but you don't always want that. -
Thanks for your input.
I just rebooted once again testwise and yes, the messages are almost the same (and in same order) but I now noticed (wasn`t aware before) that after (in respect to boot disk):
nvme0: <Generic NVMe Device> mem 0xa1600000-0xa1603fff at device 0.0 on pci2 .... ZFS filesystem version: 5 ZFS storage pool version: features support (5000)It says instead of (in case of successful boot)
Trying to mount root from zfs:pfSense/ROOT/default []... Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM ...in the order of this line:
nvme0: System interrupt issues? Root mount waiting for: CAM Root mount waiting for: CAM Root mount waiting for: CAM .... (endless repeat)Cold boot is e.g. when the warm boot fails I shut down the pc via power button (long press). When I turn the PC on again 10 seconds later it boots fine (like the example given in my first post).
-
Thanks for your input. According to your experience does the type of messages appear I posted above when vendors do not implement it the way it shoud be on warm reboots?
-
Oh it actually shows:
nvme0: System interrupt issues?at the warm boot? That seems like a huge clue if so.
https://man.freebsd.org/cgi/man.cgi?query=nvme&sektion=4&manpath=FreeBSD+15.0-CURRENT#DIAGNOSTICSTry booting verbose at the reboot see if that gives you more. Interrupt boot at the loader menu then at the OK> prompt enter:
boot -vSometime the extra delay caused by booting verbose can actually pass whatever error is present. Which is inconvenient for troubleshooting!
-
@ramup possibly. A reset line not physically routed to the right place. But again, sometimes this can be worked around by have software reinitializing registers.
-
Mmm I could believe something missing in the adapter could present like this.
I'd try forcing legacy interrupts for nvme.
-
@stephenw10
Thank you for the hint to FreeBSD man page. I did not even find that with Google with search "system interrupt issues?"Can you tell me what you technically mean by "try forcing legacy interrupts for nvme"?
@mer
Thank you also for your input, but I also do not understand technically how to do "software reinitializing registers"? -
@stephenw10
Do you mean:https://man.freebsd.org/cgi/man.cgi?nvme(4)
setting:
hw.nvme.force_intx=1
in in /boot/loader.conf.local ?
-
Create the file /boot/loader.conf.local
Add the following line to it:
hw.nvme.force_intx=1Reboot.
I've never had to try that so you may find you have to disable that again at the loader prompt if it fails to boot.
-
@ramup said in BOOT problem ZFS / NVME SSD - Root mount waiting for CAM:
Thank you also for your input, but I also do not understand technically how to do "software reinitializing registers"?
That would be the OS/driver level. OS as it goes through boot sequence, OS detects a piece of hardware (ram, usb, pci, whatever), OS knows (because of magic) warm/cold boot, decides to set registers or not. Values in sysctls can affect this behavior.
So not a "pfSense problem" but maybe a FreeBSD issue (pfSense is built on top of FreeBSD), but reporting it here because this is where you ran across it here lets the pfSense team figure out what to do. -
Thank you for your help and guidance. I will start to try with the extra boot command and if that does not work I will search for the possibilities to reinitialize registers via sysctl.
I will report back!
-
Feedback:
hw.nvme.force_intx=1
in
/boot/loader.conf.local
solved the issue!
Many thanks for your help!
I renamed the thread in case someone searches for it.
-
Nice, good result!
-
@stephenw10 I agree @ramup thanks for keeping everyone in the loop
