Kernel Panics on Intel Apollo Lake Processors
-
Well let's see how it goes without disabling the C States first as the ZimaBoard is a passively cooled SBC and i don't know how well it will take that :P however at the moment it is running at aprox 36 C
-
Check the sysctls for dev.cpu.0. It will show you what the lowest C state available is and what percentage of time the CPU has spent in each state:
[2.7.0-DEVELOPMENT][admin@t70.stevew.lan]/root: sysctl -a | grep cx_ hw.acpi.cpu.cx_lowest: C1 dev.cpu.3.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc dev.cpu.3.cx_usage_counters: 10875352 2826431 46137948 dev.cpu.3.cx_usage: 18.17% 4.72% 77.10% last 229us dev.cpu.3.cx_lowest: C3 dev.cpu.3.cx_supported: C1/1/1 C2/2/500 C3/3/1000 dev.cpu.2.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc dev.cpu.2.cx_usage_counters: 10937005 2854991 46338850 dev.cpu.2.cx_usage: 18.18% 4.74% 77.06% last 13us dev.cpu.2.cx_lowest: C3 dev.cpu.2.cx_supported: C1/1/1 C2/2/500 C3/3/1000 dev.cpu.1.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc dev.cpu.1.cx_usage_counters: 9982605 2917088 45710826 dev.cpu.1.cx_usage: 17.03% 4.97% 77.99% last 16us dev.cpu.1.cx_lowest: C3 dev.cpu.1.cx_supported: C1/1/1 C2/2/500 C3/3/1000 dev.cpu.0.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc dev.cpu.0.cx_usage_counters: 897566084 392 576 dev.cpu.0.cx_usage: 99.99% 0.00% 0.00% last 127us dev.cpu.0.cx_lowest: C3 dev.cpu.0.cx_supported: C1/1/1 C2/2/500 C3/3/1000I've never tried but it looks like you might be able to set the method used there.
-
Well it seems that at least the setting i changed on the BIOS side did indeed take effect as you can see that the method for the states is C1/hlt instead of mwait that you get on your end
hw.acpi.cpu.cx_lowest: C1 dev.cpu.3.cx_method: C1/hlt C2/io C3/io dev.cpu.3.cx_usage_counters: 9348450 0 0 dev.cpu.3.cx_usage: 100.00% 0.00% 0.00% last 93us dev.cpu.3.cx_lowest: C1 dev.cpu.3.cx_supported: C1/1/1 C2/2/50 C3/3/150 dev.cpu.2.cx_method: C1/hlt C2/io C3/io dev.cpu.2.cx_usage_counters: 27785106 0 0 dev.cpu.2.cx_usage: 100.00% 0.00% 0.00% last 491us dev.cpu.2.cx_lowest: C1 dev.cpu.2.cx_supported: C1/1/1 C2/2/50 C3/3/150 dev.cpu.1.cx_method: C1/hlt C2/io C3/io dev.cpu.1.cx_usage_counters: 26451744 0 0 dev.cpu.1.cx_usage: 100.00% 0.00% 0.00% last 236us dev.cpu.1.cx_lowest: C1 dev.cpu.1.cx_supported: C1/1/1 C2/2/50 C3/3/150 dev.cpu.0.cx_method: C1/hlt C2/io C3/io dev.cpu.0.cx_usage_counters: 13222448 0 0 dev.cpu.0.cx_usage: 100.00% 0.00% 0.00% last 12us dev.cpu.0.cx_lowest: C1 dev.cpu.0.cx_supported: C1/1/1 C2/2/50 C3/3/150So from what i can see my cores stay at C1 which is the halt state all the time while sleeping i suppose as it shows 0 time sepent on other states
I do not exactly know what that means in terms of thermals and stuff though as i am not that familiar with the states
-
I have also updated the cpu microcode as suggested on the article regarding the VM issues on Jasper Lake
It experienced a panic twice since yesterday and both times it was stuck it did not even reboot after the crash but again it has to do with the C1 state
-
It has been a little more than 24 hours since i have updated the cpu microcode and i haven't had any crashes yet which never happened before as i used to get 1-2 crashes per day
I will test it for a couple more days like this and then i will try enabling the monitor mwait functionality in bios as well to check if it will work like that
-
@sidegr You got my attention there are a lot of stepping issues with the N5105 and Proxmox virtualizations. Just letting you know that from what I read microcode #24 fixes a lot.
https://forum.proxmox.com/threads/vm-freezes-irregularly.111494/page-33
Thinking about testing 23.05 RC on my N5105 tonight.
PS: to get C3 states I got this implemented.
-
I can confirm that it crashed again even with the updated microcode and the crashes again are consistent this time crashing at sched_pickcpu() almost every single time or on cpu_search_highest() again while the system makes those calls the cores are in acpi_cpu_c1() state so the issue seems to be with cpu states in general
From the data i collected it mostly has to do with when some or most of the cores are in sleep state and the scheduler tries to wake them up to assign some task to them and this is when the panic occurs
I will also try your suggestion to add those flags in my tunables as from what i gathered so far my cores doesn't ever go into C3 they only reach C1 as it is now
-
Enabling the C3 state made the issue way worse that it was before so i guess i will stay with C1 for now although i am getting slightly higher temperatures while only allowing it to reach C1
It seems there is a big issue with the kernel and this generation's processors
Let's hope it will get fixed on the final release of 23.05
-
@sidegr said in Kernel Panics on Intel Apollo Lake Processors:
Let's hope it will get fixed on the final release of 23.05
Perhaps I am wrong with, but could it also be the Intel SpeedStep technology in that case? There were also an interesting thread I was sadly not able to find again
here in the forum, where someone was setting up
in the "tune ables" something based on the
SpeedStep technology and that solved that
problem really good. -
I will try and find the original thread or find something on that on the internet in general, i know for sure that i can disable speed step on my BIOS but if there is another solution i would prefer to not do that
-
I am also wondering if it has anything to do with PowerD that i have enabled and set to Hiadaptive
-
@sidegr said in Kernel Panics on Intel Apollo Lake Processors:
I am also wondering if it has anything to do with PowerD that i have enabled and set to Hiadaptive
-
@sidegr said in Kernel Panics on Intel Apollo Lake Processors:
I am also wondering if it has anything to do with PowerD that i have enabled and set to Hiadaptive
-
The speedstep/speedshift difference only affects p-states not c-states just for clarity. But you might find speedshift is enabled on a system where speedstep wasn't.
-
So i suppose PowerD doesnt really affect my issue, anyway the current status is that since i have disabled monitor mwait in BIOS and also included some parameters in the loader.conf file to update the cpu microcode on boot the panics significantly reduced in frequency but they are still happening a bit less than ones a day or so
(Edit) Those are the lines i have added to loader.conf
cpu_microcode_load="YES" cpu_microcode_name="/boot/firmware/intel-ucode.bin"And this is the output i get from dmesg
CPU microcode: updated from 0x3c to 0x48 -
You should check to see if Speedshift is attaching at boot though. It's enabled by default in 23.01 so it could be doing a bunch of CPU speed scaling that wouldn't have happened in older versions.
-
I see nothing related to speedshift in dmesg the only thing I see is in regards to speedstep and it’s this line
est0: <Enhanced SpeedStep Frequency Control> on cpu0However it only shows up for core 0 some posts I have seen online it shows up for all cores that the cpu exposes to the kernel like 0, 1, 2 and 3 in my case
Not sure if that could be an issue as well
-
This post is deleted! -
That is probably fine. Speedstep can only set the P-states all the same anyway so it only really needs to use core 0. Speedshift can set each core individually and does so at much faster rates. But your CPU doesn't support it so not a problem.
-
Oh ok then it makes sense for speedshift to not get attached at boot
Well for now it seems that the only solution is to just keep monitor mwait off on the BIOS side and only let the cores enter at most a C1 state cause anything more than C1 is extremely unstable, it would be nice to see that fixed tough, also it might have to do with the fact the freebsd 14 is still in a fairly early beta stage and might have issues
