Enhanced Intel SpeedStep / Speed Shift - Are they fully supported?

stephenw10

Yeah, I'm reading up to reach that point!

tman222

I have a system with the same processor family (Xeon D-1718T vs. 1736NT), and from what I can tell Enhanced Intel SpeedStep is enabled and working. I don't think this particular CPU family supports Intel Speed Shift though.

Based on my reading, I see that Enhanced Intel SpeedStep is enabled through cpufreq via the est interface:

https://man.freebsd.org/cgi/man.cgi?est
https://man.freebsd.org/cgi/man.cgi?cpufreq

As mentioned in the documentation links, you should see a line similar to this in your boot logs if est is enabled: est%d: <Enhanced SpeedStep Frequency Control> on cpu%d

I have powerd set to hiadaptive and can see the frequency shift around between the min and max as defined by the SpeedStep dev.est.%d.freq_settings. sysctl variable.

RobbieTT

@tman222

The EIST works just fine and I have mine set on adaptive (ie down from hi-adaptive) and the frequency varies as you would expect. The bonus of Speed Shift is that it reacts much faster than EIST, enabling a larger power saving whilst preserving performance for when you need it.

The Intel docs do stipulate that Speed Shift is included in this family, same for Speed Select Technology. If it can be achieved though is a different matter and still open for debate. We may be missing part of the jigsaw puzzle and the documents are not written with much clarity when it comes to derivatives of the main CPU line. Moreover, there is no mention of BSD in the Intel documents but there is kernel support in Linux v5.3 onwards.

️

edit: This popped-up on the FreeBSD forum in the last few days:
https://forums.freebsd.org/threads/shell-script-to-adjust-intel-speed-shift-technology-on-the-fly.90841/

RobbieTT

The Linux kernel support for Intel Speed Select Technology is covered here:

https://docs.kernel.org/admin-guide/pm/intel-speed-select.html

tman222

Thanks for clarifying. From your initial post it sounded like you were also asking whether EIST was working / enabled, hence my response.

Regarding Speed Shift - are you saying you have an option in the BIOS to enable it but then it's not exposed / supported by the OS? I just looked and I don't see the hwpstate_intel sysctl variables on my end either. I do see that machdep.hwpstate_pkg_ctrl = 1. Have you tried setting that 0 to see if it changes anything? Also, there is a debug.hwpstate_verbose tunable - have you tried enabling that to see if it spits out anything interesting during boot up that could provide a further clue why it might not be getting enabled in the OS?

stephenw10

Yeah it soon becomes unclear exactly what 'EIST' entails after some reading. In FreeBSD however EIST is SpeedStep and not SpeedShift as far as I know. And I've yet to see anything about Speed Select Technology.

It seems likely the SpeedShift driver may just not support that CPU yet. However it could be just using the data passed via ACPI in which case maybe that system is just not passing it? Or not in a form the driver expects.

If you boot Linux on that hardware can you see it using SpeedShift?

tman222

Figured it out! There as a BIOS option "Hardware P-States" that was set to "Disable". I enabled this and after that the hwpstate_intel driver loaded and Speed Shift settings were there in pfSense.

@RobbieTT - if you are using a Supermicro system, look for the "Hardware P-States" setting under Advanced Power Management Configuration. I set mine to from "Disable" to "Native Mode" and things started working.

RobbieTT

@stephenw10

Yep, the Intel naming conventions are beyond known science and using the same letters in an acronym for a near identical and related term is just bonkers. Maybe cracking this will be second task for quantum computers, just after they tackle Cisco's pricing structure.

I did load Ubuntu on this machine just to test it out of the box and Speed Shift (or later) seemed to work as expected. It was only well after that did I realise that not all OS kernels supported it and just happened to pick one that did.

I guess I could put an additional SSD into it and install Ubuntu again for a proper look around but I already think that I am way out of my depth just reading the Intel documents.

️

RobbieTT

@tman222 said in Enhanced Intel SpeedStep / Speed Shift - Are they fully supported?:

Figured it out! There as a BIOS option "Hardware P-States" that was set to "Disable". I enabled this and after that the hwpstate_intel driver loaded and Speed Shift settings were there in pfSense.

@RobbieTT - if you are using a Supermicro system, look for the "Hardware P-States" setting under Advanced Power Management Configuration. I set mine to from "Disable" to "Native Mode" and things started working.

You are right! Great spot! There are 2 different Hardware P-State settings, the first is enabled by default and has additional setting beyond it but this second one is different and disabled by default:

Hardware PM State Control
Hardware P-States
This setting allows you to select between OS and hardware-controlled P-states. Selecting Native Mode allows the OS to choose a P-state. Selecting Out of Band Mode allows the hardware to autonomously choose a P-state without OS guidance. Selecting Native Mode with No Legacy Support functions as Native Mode with no support for older hardware.

The options are Disable, Native Mode, Out of Band Mode, and Native Mode with No Legacy Support.

Do we collectively think that Native (allowing the OS to choose a state) is ok with BSD (I think it is) or is Out of Band Mode a better option?

️

tman222

@RobbieTT said in Enhanced Intel SpeedStep / Speed Shift - Are they fully supported?:

Do we collectively think that Native (allowing the OS to choose a state) is ok with BSD (I think it is) or is Out of Band Mode a better option?

️

I was wondering this myself - if Out of Band mode is chosen does the hwpstate_intel driver still attach during boot? I would think that Native Mode is required for the driver to load, but I could be wrong.

stephenw10

The driver only configures the parameters for the hardware to use it doesn't actually set P-states directly. So.... unclear!

tman222

@stephenw10 said in Enhanced Intel SpeedStep / Speed Shift - Are they fully supported?:

The driver only configures the parameters for the hardware to use it doesn't actually set P-states directly. So.... unclear!

Tried "Out of Band Mode" and the hwpstate_intel driver no longer loaded / attached during boot and Speed Shift settings were also gone in pfSense. Looks like it needs to be set to Native Mode to work properly.

q54e3w

@tman222 mine is set to native and seems to be functioning. I have a few days worth of 23.09 SNMP data illustrating the delta between 2.7 hiadaptive and 23.09 ‘80’ I’ll share when I get back to my desk.

tman222

@q54e3w said in Enhanced Intel SpeedStep / Speed Shift - Are they fully supported?:

@tman222 mine is set to native and seems to be functioning. I have a few days worth of 23.09 SNMP data illustrating the delta between 2.7 hiadaptive and 23.09 ‘80’ I’ll share when I get back to my desk.

Thanks @q54e3w - I look forward to seeing those statistics. I currently have mine set to 60 and the CPU is already running 2-3 degrees C cooler than before (when it was using EIST).

RobbieTT

@tman222

Set mine this evening and set the GUI slider to 80 for now and the recommended 'per core' although it defaulted to 'package'. Not sure if you needed to un-tick the PowerD box but did so anyway:

20231110-Router-7-pfSense-Speed Step Enabled at 80.png

Noticed a couple of other changes on the Dashboard. The max frequency now shows as 2700 MHz (vice 2701) and the minimum freq displays 799 MHz (vice 800):

20231110-Router-7-BIOS to Intel Speed Shift-80-No turbo 2700 MHz question.png

(The above shows 8-cores as I don't have HT enabled)

Turbo still seems to work though:

load  31%, current freq 3199 MHz ( 0), wanted freq 5400 MHz
load  30%, current freq 3199 MHz ( 0), wanted freq 5400 MHz
load  39%, current freq  799 MHz ( 0), wanted freq 5400 MHz
load  65%, current freq  799 MHz ( 0), wanted freq 5400 MHz

Shame I don't get any power data from this Supermicro PSU:

20231110-Supermicro-No Power Consumption Displayed.png

According to my UPS my system may have reduced by a few watts but too early to tell if that is really the case.

️

tman222

On a related topic, have any of you made adjustments to the C-States on your firewalls? From what I can tell FreeBSD / pfSense show C1 and C2 as supported on the Xeon D-1718T CPU in my system:

 dev.cpu.0.cx_supported: C1/1/1 C2/2/41

Looking at the BIOS, I see that a C3 state is also supported and can be enabled. Unfortunately, even after enabling the additional C-State the OS still only seeds C1 and C2. Does anyone have any idea how to get the OS to recognize C3 as well?

Moving on, I saw that the hw.acpi.cpu.cx_lowest sysctl tunable was set to C1 but allowed me to switch it over to C2.

After ~24 hours of run time, I checked C state usage and it looks like that the system is very lightly utilized overall, i.e. spends the majority of time in the C2 state:

 dev.cpu.7.cx_usage: 0.25% 99.74% last 98us
 dev.cpu.5.cx_usage: 0.29% 99.70% last 994us
 dev.cpu.3.cx_usage: 0.26% 99.73% last 618us
 dev.cpu.1.cx_usage: 0.29% 99.70% last 495us
 dev.cpu.6.cx_usage: 0.20% 99.79% last 25us
 dev.cpu.4.cx_usage: 0.19% 99.80% last 284us
 dev.cpu.2.cx_usage: 0.59% 99.40% last 131us
 dev.cpu.0.cx_usage: 0.32% 99.67% last 81us

I'm not quite sure yet how much incremental power savings there are allowing C2 over just C1, but the impact to performance appears to be negligible (if any impact at all). Having said that, exit latency on C-States is definitely something to keep mind, especially when dealing with networking equipment. Thankfully in this case the exit latency on the C2 state on my system is quite low - only 41 microseconds.

RobbieTT

@tman222 said in Enhanced Intel SpeedStep / Speed Shift - Are they fully supported?:

cx_supported

I keep mine in the default C1 as C2 seemed unusual for a router firewall and I didn't change the BIOS setting and I think only C1 & C2 are supported by pfSense.

There is constant background traffic on my network so I never imagined the C2 would be anything but an encumbrance. Hopefully Speed Shift does the clever stuff by core but perhaps I am mistaken for the wider system? Here are my stats for the D-1736NT:

dev.cpu.7.cx_usage: 100.00% 0.00% last 32us
dev.cpu.6.cx_usage: 100.00% 0.00% last 557us
dev.cpu.5.cx_usage: 100.00% 0.00% last 33us
dev.cpu.4.cx_usage: 100.00% 0.00% last 70us
dev.cpu.3.cx_usage: 100.00% 0.00% last 1290us
dev.cpu.2.cx_usage: 100.00% 0.00% last 317us
dev.cpu.1.cx_usage: 100.00% 0.00% last 1033us
dev.cpu.0.cx_usage: 100.00% 0.00% last 756us
dev.cpu.7.cx_supported: C1/1/1 C2/2/41
dev.cpu.6.cx_supported: C1/1/1 C2/2/41
dev.cpu.5.cx_supported: C1/1/1 C2/2/41
dev.cpu.4.cx_supported: C1/1/1 C2/2/41
dev.cpu.3.cx_supported: C1/1/1 C2/2/41
dev.cpu.2.cx_supported: C1/1/1 C2/2/41
dev.cpu.1.cx_supported: C1/1/1 C2/2/41
dev.cpu.0.cx_supported: C1/1/1 C2/2/41

Having already disabled HT and setting Speed Shift to 80 I think the only extra power measures I could take is disabling unused interfaces. Perhaps I should allow the SSD to sleep but that may not make any real difference to power/heat.

️

stephenw10

pfSense supports higher (lower?) C states if the BIOS passes them:

dev.cpu.0.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc
dev.cpu.0.cx_usage_counters: 9066 5086 8391
dev.cpu.0.cx_usage: 40.21% 22.56% 37.22% last 2204us
dev.cpu.0.cx_lowest: C8
dev.cpu.0.cx_supported: C1/1/1 C2/2/127 C3/3/1048

RobbieTT

@stephenw10
That's good to know. Typically, how deep down into the C-states would you expect for a router/firewall, without overly impacting performance?

️

stephenw10

For best performance I would disable C states deeper than C1. I have enabled a deep as it could go on some boxes that mostly sit idle and it makes no significant difference to either performance or power consumption beyond C3 as far as I could tell.