Enhanced Intel SpeedStep / Speed Shift - Are they fully supported?
-
Figured it out! There as a BIOS option "Hardware P-States" that was set to "Disable". I enabled this and after that the hwpstate_intel driver loaded and Speed Shift settings were there in pfSense.
@RobbieTT - if you are using a Supermicro system, look for the "Hardware P-States" setting under Advanced Power Management Configuration. I set mine to from "Disable" to "Native Mode" and things started working.
-
Yep, the Intel naming conventions are beyond known science and using the same letters in an acronym for a near identical and related term is just bonkers. Maybe cracking this will be second task for quantum computers, just after they tackle Cisco's pricing structure.
I did load Ubuntu on this machine just to test it out of the box and Speed Shift (or later) seemed to work as expected. It was only well after that did I realise that not all OS kernels supported it and just happened to pick one that did.
I guess I could put an additional SSD into it and install Ubuntu again for a proper look around but I already think that I am way out of my depth just reading the Intel documents.
️ -
@tman222 said in Enhanced Intel SpeedStep / Speed Shift - Are they fully supported?:
Figured it out! There as a BIOS option "Hardware P-States" that was set to "Disable". I enabled this and after that the hwpstate_intel driver loaded and Speed Shift settings were there in pfSense.
@RobbieTT - if you are using a Supermicro system, look for the "Hardware P-States" setting under Advanced Power Management Configuration. I set mine to from "Disable" to "Native Mode" and things started working.
You are right! Great spot! There are 2 different Hardware P-State settings, the first is enabled by default and has additional setting beyond it but this second one is different and disabled by default:
Hardware PM State Control
Hardware P-States
This setting allows you to select between OS and hardware-controlled P-states. Selecting Native Mode allows the OS to choose a P-state. Selecting Out of Band Mode allows the hardware to autonomously choose a P-state without OS guidance. Selecting Native Mode with No Legacy Support functions as Native Mode with no support for older hardware.The options are Disable, Native Mode, Out of Band Mode, and Native Mode with No Legacy Support.
Do we collectively think that Native (allowing the OS to choose a state) is ok with BSD (I think it is) or is Out of Band Mode a better option?
️ -
@RobbieTT said in Enhanced Intel SpeedStep / Speed Shift - Are they fully supported?:
Do we collectively think that Native (allowing the OS to choose a state) is ok with BSD (I think it is) or is Out of Band Mode a better option?
️I was wondering this myself - if Out of Band mode is chosen does the hwpstate_intel driver still attach during boot? I would think that Native Mode is required for the driver to load, but I could be wrong.
-
The driver only configures the parameters for the hardware to use it doesn't actually set P-states directly. So.... unclear!
-
@stephenw10 said in Enhanced Intel SpeedStep / Speed Shift - Are they fully supported?:
The driver only configures the parameters for the hardware to use it doesn't actually set P-states directly. So.... unclear!
Tried "Out of Band Mode" and the
hwpstate_inteldriver no longer loaded / attached during boot and Speed Shift settings were also gone in pfSense. Looks like it needs to be set to Native Mode to work properly. -
@tman222 mine is set to native and seems to be functioning. I have a few days worth of 23.09 SNMP data illustrating the delta between 2.7 hiadaptive and 23.09 ‘80’ I’ll share when I get back to my desk.
-
@q54e3w said in Enhanced Intel SpeedStep / Speed Shift - Are they fully supported?:
@tman222 mine is set to native and seems to be functioning. I have a few days worth of 23.09 SNMP data illustrating the delta between 2.7 hiadaptive and 23.09 ‘80’ I’ll share when I get back to my desk.
Thanks @q54e3w - I look forward to seeing those statistics. I currently have mine set to 60 and the CPU is already running 2-3 degrees C cooler than before (when it was using EIST).
-
Set mine this evening and set the GUI slider to 80 for now and the recommended 'per core' although it defaulted to 'package'. Not sure if you needed to un-tick the PowerD box but did so anyway:
Noticed a couple of other changes on the Dashboard. The max frequency now shows as 2700 MHz (vice 2701) and the minimum freq displays 799 MHz (vice 800):
(The above shows 8-cores as I don't have HT enabled)
Turbo still seems to work though:
load 31%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 30%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 39%, current freq 799 MHz ( 0), wanted freq 5400 MHz load 65%, current freq 799 MHz ( 0), wanted freq 5400 MHzShame I don't get any power data from this Supermicro PSU:
According to my UPS my system may have reduced by a few watts but too early to tell if that is really the case.
️ -
On a related topic, have any of you made adjustments to the C-States on your firewalls? From what I can tell FreeBSD / pfSense show C1 and C2 as supported on the Xeon D-1718T CPU in my system:
dev.cpu.0.cx_supported: C1/1/1 C2/2/41Looking at the BIOS, I see that a C3 state is also supported and can be enabled. Unfortunately, even after enabling the additional C-State the OS still only seeds C1 and C2. Does anyone have any idea how to get the OS to recognize C3 as well?
Moving on, I saw that the
hw.acpi.cpu.cx_lowestsysctl tunable was set to C1 but allowed me to switch it over to C2.After ~24 hours of run time, I checked C state usage and it looks like that the system is very lightly utilized overall, i.e. spends the majority of time in the C2 state:
dev.cpu.7.cx_usage: 0.25% 99.74% last 98us dev.cpu.5.cx_usage: 0.29% 99.70% last 994us dev.cpu.3.cx_usage: 0.26% 99.73% last 618us dev.cpu.1.cx_usage: 0.29% 99.70% last 495us dev.cpu.6.cx_usage: 0.20% 99.79% last 25us dev.cpu.4.cx_usage: 0.19% 99.80% last 284us dev.cpu.2.cx_usage: 0.59% 99.40% last 131us dev.cpu.0.cx_usage: 0.32% 99.67% last 81usI'm not quite sure yet how much incremental power savings there are allowing C2 over just C1, but the impact to performance appears to be negligible (if any impact at all). Having said that, exit latency on C-States is definitely something to keep mind, especially when dealing with networking equipment. Thankfully in this case the exit latency on the C2 state on my system is quite low - only 41 microseconds.
-
@tman222 said in Enhanced Intel SpeedStep / Speed Shift - Are they fully supported?:
cx_supported
I keep mine in the default C1 as C2 seemed unusual for a router firewall and I didn't change the BIOS setting and I think only C1 & C2 are supported by pfSense.
There is constant background traffic on my network so I never imagined the C2 would be anything but an encumbrance. Hopefully Speed Shift does the clever stuff by core but perhaps I am mistaken for the wider system? Here are my stats for the D-1736NT:
dev.cpu.7.cx_usage: 100.00% 0.00% last 32us dev.cpu.6.cx_usage: 100.00% 0.00% last 557us dev.cpu.5.cx_usage: 100.00% 0.00% last 33us dev.cpu.4.cx_usage: 100.00% 0.00% last 70us dev.cpu.3.cx_usage: 100.00% 0.00% last 1290us dev.cpu.2.cx_usage: 100.00% 0.00% last 317us dev.cpu.1.cx_usage: 100.00% 0.00% last 1033us dev.cpu.0.cx_usage: 100.00% 0.00% last 756us dev.cpu.7.cx_supported: C1/1/1 C2/2/41 dev.cpu.6.cx_supported: C1/1/1 C2/2/41 dev.cpu.5.cx_supported: C1/1/1 C2/2/41 dev.cpu.4.cx_supported: C1/1/1 C2/2/41 dev.cpu.3.cx_supported: C1/1/1 C2/2/41 dev.cpu.2.cx_supported: C1/1/1 C2/2/41 dev.cpu.1.cx_supported: C1/1/1 C2/2/41 dev.cpu.0.cx_supported: C1/1/1 C2/2/41Having already disabled HT and setting Speed Shift to 80 I think the only extra power measures I could take is disabling unused interfaces. Perhaps I should allow the SSD to sleep but that may not make any real difference to power/heat.
️ -
pfSense supports higher (lower?) C states if the BIOS passes them:
dev.cpu.0.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc dev.cpu.0.cx_usage_counters: 9066 5086 8391 dev.cpu.0.cx_usage: 40.21% 22.56% 37.22% last 2204us dev.cpu.0.cx_lowest: C8 dev.cpu.0.cx_supported: C1/1/1 C2/2/127 C3/3/1048 -
@stephenw10
That's good to know. Typically, how deep down into the C-states would you expect for a router/firewall, without overly impacting performance?️ -
For best performance I would disable C states deeper than C1. I have enabled a deep as it could go on some boxes that mostly sit idle and it makes no significant difference to either performance or power consumption beyond C3 as far as I could tell.
-
@tman222 Note vertical scale only goes to 30%, its an overpowered system
~24 hours pre/post upgrade
Pre/current
Edit: I took the opportunity to upgrade the motherboard firmware at the same time as update pfSense too which may have some impact.
-
@q54e3w
Am I reading this correctly - that Speed Shift seems to work the CPU harder?I have noticed that the CPU tends to run at a higher load and frequency but drops pretty quickly, at least on my own system.
️ -
@RobbieTT Yup, it looks like it although how the operating system and pfSense has changed will also impact. Its not an apples to apples comparison exactly because its not on the same operating system and use of IPSec-MB etc. I also upgraded the BIOS at the same time given the box was out of service anyway.
I'm currently trying '60' to see if it makes a noticeable difference to temps, power consumption or performance. -
Speedshift reacts much faster so you will the CPU at higher frequencies more often. Quite how that translates into apparent loading though.... unclear!
-
@stephenw10
I think we may miss stuff with the reporting rate vs actual rate of change. With different frequencies on different cores the values we see may not be truly representative.With my previous 'adaptive' profile I didn't observe any turbo frequencies (does not mean it wasn't happening I guess) but with Speed Shift on 80 it sticks at 799 MHz under varying demand and then rapidly turbos when things get busy:
[23.09-RELEASE][admin@Router-7.redacted.me]/root: powerd -v powerd: unable to determine AC line status CPU frequency is below user-defined minimum; changing frequency to 2700 MHz load 0%, current freq 799 MHz ( 0), wanted freq 4754 MHz load 4%, current freq 799 MHz ( 0), wanted freq 4605 MHz load 94%, current freq 799 MHz ( 0), wanted freq 5400 MHz load 163%, current freq 799 MHz ( 0), wanted freq 5400 MHz load 156%, current freq 799 MHz ( 0), wanted freq 5400 MHz load 144%, current freq 799 MHz ( 0), wanted freq 5400 MHz load 190%, current freq 799 MHz ( 0), wanted freq 5400 MHz load 110%, current freq 799 MHz ( 0), wanted freq 5400 MHz load 136%, current freq 799 MHz ( 0), wanted freq 5400 MHz load 265%, current freq 799 MHz ( 0), wanted freq 5400 MHz load 194%, current freq 799 MHz ( 0), wanted freq 5400 MHz load 143%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 162%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 138%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 125%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 113%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 150%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 135%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 98%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 179%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 169%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 113%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 108%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 125%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 149%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 165%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 234%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 159%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 156%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 99%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 80%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 143%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 176%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 116%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 113%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 107%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 141%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 177%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 101%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 146%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 166%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 105%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 78%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 116%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 184%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 138%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 162%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 129%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 200%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 163%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 175%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 31%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 3%, current freq 3199 MHz ( 0), wanted freq 5231 MHz load 0%, current freq 3199 MHz ( 0), wanted freq 5067 MHz load 0%, current freq 3199 MHz ( 0), wanted freq 4908 MHz load 54%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 41%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 39%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 37%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 89%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 37%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 31%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 47%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 31%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 30%, current freq 3199 MHz ( 0), wanted freq 5400 MHz load 39%, current freq 799 MHz ( 0), wanted freq 5400 MHz load 65%, current freq 799 MHz ( 0), wanted freq 5400 MHz load 75%, current freq 799 MHz ( 0), wanted freq 5400 MHz load 22%, current freq 799 MHz ( 0), wanted freq 5231 MHz load 25%, current freq 799 MHz ( 0), wanted freq 5231 MHz load 44%, current freq 799 MHz ( 0), wanted freq 5400 MHz load 110%, current freq 799 MHz ( 0), wanted freq 5400 MHz load 127%, current freq 799 MHz ( 0), wanted freq 5400 MHz load 176%, current freq 799 MHz ( 0), wanted freq 5400 MHz load 138%, current freq 799 MHz ( 0), wanted freq 5400 MHz load 36%, current freq 799 MHz ( 0), wanted freq 5400 MHz load 11%, current freq 799 MHz ( 0), wanted freq 5231 MHz load 81%, current freq 799 MHz ( 0), wanted freq 5400 MHz load 185%, current freq 799 MHz ( 0), wanted freq 5400 MHz load 210%, current freq 799 MHz ( 0), wanted freq 5400 MHz load 197%, current freq 799 MHz ( 0), wanted freq 5400 MHz load 38%, current freq 799 MHz ( 0), wanted freq 5400 MHz load 44%, current freq 799 MHz ( 0), wanted freq 5400 MHz load 19%, current freq 799 MHz ( 0), wanted freq 5231 MHz load 22%, current freq 799 MHz ( 0), wanted freq 5067 MHz load 21%, current freq 799 MHz ( 0), wanted freq 4908 MHz load 58%, current freq 799 MHz ( 0), wanted freq 5400 MHz load 138%, current freq 799 MHz ( 0), wanted freq 5400 MHz load 34%, current freq 799 MHz ( 0), wanted freq 5400 MHz load 25%, current freq 799 MHz ( 0), wanted freq 5400 MHz load 52%, current freq 799 MHz ( 0), wanted freq 5400 MHz load 119%, current freq 799 MHz ( 0), wanted freq 5400 MHz load 17%, current freq 799 MHz ( 0), wanted freq 5231 MHzI presume the above is just the fastest core though.
️ -
I agree. When I was first testing Speedshift I had a very hard time determining if my changes were actually doing anything. I came to the conclusion it's because simply running sysctl to read the cpu state causes it to bump the frequency. For example running
sysctl dev.cpu.0.freq dev.cpu.1.freq dev.cpu.2.freq dev.cpu.3.freqgives the same result as runningsysctl dev.cpu.3.freq dev.cpu.2.freq dev.cpu.1.freq dev.cpu.0.freq. I.e. whichever core you query first gives the lowest result and each subsequent core is running faster.
