Enhanced Intel SpeedStep / Speed Shift - Are they fully supported?

[stephenw10]

Mmm, I'm not actually sure how SpeedShift (or Speedstep) are exposed. The system I have is much older than that:

hwpstate_intel0: <Intel Speed Shift> on cpu0
hwpstate_intel1: <Intel Speed Shift> on cpu1
hwpstate_intel2: <Intel Speed Shift> on cpu2
hwpstate_intel3: <Intel Speed Shift> on cpu3
Timecounter "TSC-low" frequency 1595996829 Hz quality 1000
Timecounters tick every 1.000 msec
Trying to mount root from ufs:/dev/ufsid/5d40918dbf1f4964 [rw]...
ugen0.1: <Intel XHCI root HUB> at usbus0
uhub0 on usbus0
uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
Root mount waiting for: usbus0 CAM
uhub0: 26 ports with 26 removable, self powered
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
ses0 at ahciem0 bus 0 scbus3 target 0 lun 0
ses0: <AHCI SGPIO Enclosure 2.00 0001> SEMB S-E-S 2.00 device
ses0: SEMB SES Device
ada0 at ahcich1 bus 0 scbus0 target 0 lun 0
ada0: <TS16EPTMM1600L O1225G> ACS-2 ATA SATA 3.x device
ada0: Serial Number D720661699
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 1024bytes)
ada0: Command Queueing enabled
ada0: 15272MB (31277232 512 byte sectors)
ses0: ada0,pass0 in 'Slot 01', SATA Slot: scbus0 target 0
Dual Console: Serial Primary, Video Secondary
CPU: Intel(R) Core(TM) i3-6100T CPU @ 3.20GHz (3191.99-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x506e3  Family=0x6  Model=0x5e  Stepping=3
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x7ffafbbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0x29c67af<FSGSBASE,TSCADJ,SGX,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,NFPUSG,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PROCTRACE>
  Structured Extended Features3=0xbc002e00<MCUOPT,MD_CLEAR,TSXFA,IBPB,STIBP,L1DFL,ARCH_CAP,SSBD>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  IA32_ARCH_CAPS=0xc04<RSBA>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
pchtherm0: <Skylake PCH 100 Thermal Subsystem> mem 0xdff1b000-0xdff1bfff irq 18 at device 20.2 on pci0

[RobbieTT]

@stephenw10

I'm wading through the Intel stuff for Intel Speed Select Technology. It launched with Ice Lake 3rd Gen Xeon Scalable and Xeon-D, so reasonably new, and the dev docs only reference Linux support from more recent kernels.

I'm wondering if pfSense / FreeBSD is capable of supporting the more advanced performance tuning and management of these more-modern CPUs yet?

I'm out of my depth I think.

️

[stephenw10]

Hmm, if that replaced SpeedShift then maybe not:
https://github.com/pfsense/FreeBSD-src/blob/devel-main/sys/x86/cpufreq/hwpstate_intel.c

[RobbieTT]

@stephenw10

I know enough about this topic to know I have no understanding at all.

️

[stephenw10]

Yeah, I'm reading up to reach that point!

[tman222]

I have a system with the same processor family (Xeon D-1718T vs. 1736NT), and from what I can tell Enhanced Intel SpeedStep is enabled and working. I don't think this particular CPU family supports Intel Speed Shift though.

Based on my reading, I see that Enhanced Intel SpeedStep is enabled through cpufreq via the est interface:

https://man.freebsd.org/cgi/man.cgi?est
https://man.freebsd.org/cgi/man.cgi?cpufreq

As mentioned in the documentation links, you should see a line similar to this in your boot logs if est is enabled: est%d: <Enhanced SpeedStep Frequency Control> on cpu%d

I have powerd set to hiadaptive and can see the frequency shift around between the min and max as defined by the SpeedStep dev.est.%d.freq_settings. sysctl variable.

[RobbieTT]

@tman222

The EIST works just fine and I have mine set on adaptive (ie down from hi-adaptive) and the frequency varies as you would expect. The bonus of Speed Shift is that it reacts much faster than EIST, enabling a larger power saving whilst preserving performance for when you need it.

The Intel docs do stipulate that Speed Shift is included in this family, same for Speed Select Technology. If it can be achieved though is a different matter and still open for debate. We may be missing part of the jigsaw puzzle and the documents are not written with much clarity when it comes to derivatives of the main CPU line. Moreover, there is no mention of BSD in the Intel documents but there is kernel support in Linux v5.3 onwards.

️

edit: This popped-up on the FreeBSD forum in the last few days:
https://forums.freebsd.org/threads/shell-script-to-adjust-intel-speed-shift-technology-on-the-fly.90841/

[RobbieTT]

The Linux kernel support for Intel Speed Select Technology is covered here:

https://docs.kernel.org/admin-guide/pm/intel-speed-select.html

[tman222]

Thanks for clarifying. From your initial post it sounded like you were also asking whether EIST was working / enabled, hence my response.

Regarding Speed Shift - are you saying you have an option in the BIOS to enable it but then it's not exposed / supported by the OS? I just looked and I don't see the hwpstate_intel sysctl variables on my end either. I do see that machdep.hwpstate_pkg_ctrl = 1. Have you tried setting that 0 to see if it changes anything? Also, there is a debug.hwpstate_verbose tunable - have you tried enabling that to see if it spits out anything interesting during boot up that could provide a further clue why it might not be getting enabled in the OS?

[stephenw10]

Yeah it soon becomes unclear exactly what 'EIST' entails after some reading. In FreeBSD however EIST is SpeedStep and not SpeedShift as far as I know. And I've yet to see anything about Speed Select Technology.

It seems likely the SpeedShift driver may just not support that CPU yet. However it could be just using the data passed via ACPI in which case maybe that system is just not passing it? Or not in a form the driver expects.

If you boot Linux on that hardware can you see it using SpeedShift?

[tman222]

Figured it out! There as a BIOS option "Hardware P-States" that was set to "Disable". I enabled this and after that the hwpstate_intel driver loaded and Speed Shift settings were there in pfSense.

@RobbieTT - if you are using a Supermicro system, look for the "Hardware P-States" setting under Advanced Power Management Configuration. I set mine to from "Disable" to "Native Mode" and things started working.

[RobbieTT]

@stephenw10

Yep, the Intel naming conventions are beyond known science and using the same letters in an acronym for a near identical and related term is just bonkers. Maybe cracking this will be second task for quantum computers, just after they tackle Cisco's pricing structure.

I did load Ubuntu on this machine just to test it out of the box and Speed Shift (or later) seemed to work as expected. It was only well after that did I realise that not all OS kernels supported it and just happened to pick one that did.

I guess I could put an additional SSD into it and install Ubuntu again for a proper look around but I already think that I am way out of my depth just reading the Intel documents.

️

[RobbieTT]

@tman222 said in Enhanced Intel SpeedStep / Speed Shift - Are they fully supported?:

Figured it out! There as a BIOS option "Hardware P-States" that was set to "Disable". I enabled this and after that the hwpstate_intel driver loaded and Speed Shift settings were there in pfSense.

@RobbieTT - if you are using a Supermicro system, look for the "Hardware P-States" setting under Advanced Power Management Configuration. I set mine to from "Disable" to "Native Mode" and things started working.

You are right! Great spot! There are 2 different Hardware P-State settings, the first is enabled by default and has additional setting beyond it but this second one is different and disabled by default:

Hardware PM State Control
Hardware P-States
This setting allows you to select between OS and hardware-controlled P-states. Selecting Native Mode allows the OS to choose a P-state. Selecting Out of Band Mode allows the hardware to autonomously choose a P-state without OS guidance. Selecting Native Mode with No Legacy Support functions as Native Mode with no support for older hardware.

The options are Disable, Native Mode, Out of Band Mode, and Native Mode with No Legacy Support.

Do we collectively think that Native (allowing the OS to choose a state) is ok with BSD (I think it is) or is Out of Band Mode a better option?

️

[tman222]

@RobbieTT said in Enhanced Intel SpeedStep / Speed Shift - Are they fully supported?:

Do we collectively think that Native (allowing the OS to choose a state) is ok with BSD (I think it is) or is Out of Band Mode a better option?

️

I was wondering this myself - if Out of Band mode is chosen does the hwpstate_intel driver still attach during boot? I would think that Native Mode is required for the driver to load, but I could be wrong.

[stephenw10]

The driver only configures the parameters for the hardware to use it doesn't actually set P-states directly. So.... unclear!

[tman222]

@stephenw10 said in Enhanced Intel SpeedStep / Speed Shift - Are they fully supported?:

The driver only configures the parameters for the hardware to use it doesn't actually set P-states directly. So.... unclear!

Tried "Out of Band Mode" and the hwpstate_intel driver no longer loaded / attached during boot and Speed Shift settings were also gone in pfSense. Looks like it needs to be set to Native Mode to work properly.

[q54e3w]

@tman222 mine is set to native and seems to be functioning. I have a few days worth of 23.09 SNMP data illustrating the delta between 2.7 hiadaptive and 23.09 ‘80’ I’ll share when I get back to my desk.

[tman222]

@q54e3w said in Enhanced Intel SpeedStep / Speed Shift - Are they fully supported?:

@tman222 mine is set to native and seems to be functioning. I have a few days worth of 23.09 SNMP data illustrating the delta between 2.7 hiadaptive and 23.09 ‘80’ I’ll share when I get back to my desk.

Thanks @q54e3w - I look forward to seeing those statistics. I currently have mine set to 60 and the CPU is already running 2-3 degrees C cooler than before (when it was using EIST).

[RobbieTT]

@tman222

Set mine this evening and set the GUI slider to 80 for now and the recommended 'per core' although it defaulted to 'package'. Not sure if you needed to un-tick the PowerD box but did so anyway:

Noticed a couple of other changes on the Dashboard. The max frequency now shows as 2700 MHz (vice 2701) and the minimum freq displays 799 MHz (vice 800):

(The above shows 8-cores as I don't have HT enabled)

Turbo still seems to work though:

load  31%, current freq 3199 MHz ( 0), wanted freq 5400 MHz
load  30%, current freq 3199 MHz ( 0), wanted freq 5400 MHz
load  39%, current freq  799 MHz ( 0), wanted freq 5400 MHz
load  65%, current freq  799 MHz ( 0), wanted freq 5400 MHz

Shame I don't get any power data from this Supermicro PSU:

According to my UPS my system may have reduced by a few watts but too early to tell if that is really the case.

️

[tman222]

On a related topic, have any of you made adjustments to the C-States on your firewalls? From what I can tell FreeBSD / pfSense show C1 and C2 as supported on the Xeon D-1718T CPU in my system:

 dev.cpu.0.cx_supported: C1/1/1 C2/2/41

Looking at the BIOS, I see that a C3 state is also supported and can be enabled. Unfortunately, even after enabling the additional C-State the OS still only seeds C1 and C2. Does anyone have any idea how to get the OS to recognize C3 as well?

Moving on, I saw that the hw.acpi.cpu.cx_lowest sysctl tunable was set to C1 but allowed me to switch it over to C2.

After ~24 hours of run time, I checked C state usage and it looks like that the system is very lightly utilized overall, i.e. spends the majority of time in the C2 state:

 dev.cpu.7.cx_usage: 0.25% 99.74% last 98us
 dev.cpu.5.cx_usage: 0.29% 99.70% last 994us
 dev.cpu.3.cx_usage: 0.26% 99.73% last 618us
 dev.cpu.1.cx_usage: 0.29% 99.70% last 495us
 dev.cpu.6.cx_usage: 0.20% 99.79% last 25us
 dev.cpu.4.cx_usage: 0.19% 99.80% last 284us
 dev.cpu.2.cx_usage: 0.59% 99.40% last 131us
 dev.cpu.0.cx_usage: 0.32% 99.67% last 81us

I'm not quite sure yet how much incremental power savings there are allowing C2 over just C1, but the impact to performance appears to be negligible (if any impact at all). Having said that, exit latency on C-States is definitely something to keep mind, especially when dealing with networking equipment. Thankfully in this case the exit latency on the C2 state on my system is quite low - only 41 microseconds.