Enhanced Intel SpeedStep / Speed Shift - Are they fully supported?

RobbieTT

Thanks Steve - The (Ice Lake) Xeon seems to have the latest gizmos but I don't definitively know what is exposed to BSD.

It also has the split core thing, Intel SST, EIST, Resource Director Technology and the Time Coordinated Computing, which all fish in the same pond.

The microcode exposed at boot:

<118>Launching the init system...Updating CPU Microcode...
CPU: Intel(R) Xeon(R) D-1736NT CPU @ 2.70GHz (2700.00-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x606c1  Family=0x6  Model=0x6c  Stepping=1
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x7ffefbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0xf3bfbfff<FSGSBASE,TSCADJ,SGX,BMI1,HLE,AVX2,FDPEXC,SMEP,BMI2,ERMS,INVPCID,RTM,PQM,NFPUSG,PQE,AVX512F,AVX512DQ,RDSEED,ADX,SMAP,AVX512IFMA,CLFLUSHOPT,CLWB,PROCTRACE,AVX512CD,SHA,AVX512BW,AVX512VL>
  Structured Extended Features2=0x40417f5e<AVX512VBMI,UMIP,PKU,OSPKE,AVX512VBMI2,GFNI,VAES,VPCLMULQDQ,AVX512VNNI,AVX512BITALG,TME,AVX512VPOPCNTDQ,LA57,RDPID,SGXLC>
  Structured Extended Features3=0xbc040412<FSRM,MD_CLEAR,PCONFIG,IBPB,STIBP,L1DFL,ARCH_CAP,SSBD>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  IA32_ARCH_CAPS=0x6bdeb<RDCL_NO,IBRS_ALL,SKIP_L1DFL_VME,MDS_NO,TSX_CTRL,TAA_NO>
  AMD Extended Feature Extensions ID EBX=0x200<WBNOINVD>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
  TSC: P-state invariant, performance statistics
<118>Done.
<118> done.
<118>Initializing................... done.
<118>Starting device manager (devd)...
urndis0 numa-domain 0 on uhub1
urndis0: <RNDIS Communications Control> on usbus0
<6>ue0: <USB Ethernet> on urndis0
<6>ue0: Ethernet address: b0:3a:f2:b6:05:9f
<118>2023-11-04T16:44:55.571149+00:00 - php-fpm 420 - - /rc.linkup: Ignoring link event during boot sequence.
<118>2023-11-04T16:44:55.572411+00:00 - php-fpm 419 - - /rc.linkup: Ignoring link event during boot sequence.
<118>done.
<118>Loading configuration....2023-11-04T16:44:56.426117+00:00 - php-fpm 420 - - /rc.linkup: Ignoring link event during boot sequence.
<118>done.
<118>Updating configuration...done.
<118>Checking config backups consistency.................................done.
<118>Loading cryptographic accelerator drivers...
qat0: <Intel 200xx QuickAssist> mem 0xb3540000-0xb357ffff,0xb3500000-0xb353ffff at device 0.0 numa-domain 0 on pci1
qat0: qat_dev0 started 6 acceleration engines
qat0: FW version: 4.18.0
qat0: Excessive clock measure delay
qat_ocf0: <QAT engine>
<118>done.

️

stephenw10

Mmm, I'm not actually sure how SpeedShift (or Speedstep) are exposed. The system I have is much older than that:

hwpstate_intel0: <Intel Speed Shift> on cpu0
hwpstate_intel1: <Intel Speed Shift> on cpu1
hwpstate_intel2: <Intel Speed Shift> on cpu2
hwpstate_intel3: <Intel Speed Shift> on cpu3
Timecounter "TSC-low" frequency 1595996829 Hz quality 1000
Timecounters tick every 1.000 msec
Trying to mount root from ufs:/dev/ufsid/5d40918dbf1f4964 [rw]...
ugen0.1: <Intel XHCI root HUB> at usbus0
uhub0 on usbus0
uhub0: <Intel XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
Root mount waiting for: usbus0 CAM
uhub0: 26 ports with 26 removable, self powered
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
Root mount waiting for: CAM
ses0 at ahciem0 bus 0 scbus3 target 0 lun 0
ses0: <AHCI SGPIO Enclosure 2.00 0001> SEMB S-E-S 2.00 device
ses0: SEMB SES Device
ada0 at ahcich1 bus 0 scbus0 target 0 lun 0
ada0: <TS16EPTMM1600L O1225G> ACS-2 ATA SATA 3.x device
ada0: Serial Number D720661699
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 1024bytes)
ada0: Command Queueing enabled
ada0: 15272MB (31277232 512 byte sectors)
ses0: ada0,pass0 in 'Slot 01', SATA Slot: scbus0 target 0
Dual Console: Serial Primary, Video Secondary
CPU: Intel(R) Core(TM) i3-6100T CPU @ 3.20GHz (3191.99-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x506e3  Family=0x6  Model=0x5e  Stepping=3
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x7ffafbbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0x29c67af<FSGSBASE,TSCADJ,SGX,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,NFPUSG,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PROCTRACE>
  Structured Extended Features3=0xbc002e00<MCUOPT,MD_CLEAR,TSXFA,IBPB,STIBP,L1DFL,ARCH_CAP,SSBD>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  IA32_ARCH_CAPS=0xc04<RSBA>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
pchtherm0: <Skylake PCH 100 Thermal Subsystem> mem 0xdff1b000-0xdff1bfff irq 18 at device 20.2 on pci0

RobbieTT

@stephenw10

I'm wading through the Intel stuff for Intel Speed Select Technology. It launched with Ice Lake 3rd Gen Xeon Scalable and Xeon-D, so reasonably new, and the dev docs only reference Linux support from more recent kernels.

I'm wondering if pfSense / FreeBSD is capable of supporting the more advanced performance tuning and management of these more-modern CPUs yet?

I'm out of my depth I think.

️

stephenw10

Hmm, if that replaced SpeedShift then maybe not:
https://github.com/pfsense/FreeBSD-src/blob/devel-main/sys/x86/cpufreq/hwpstate_intel.c

RobbieTT

@stephenw10

I know enough about this topic to know I have no understanding at all.

️

stephenw10

Yeah, I'm reading up to reach that point!

tman222

I have a system with the same processor family (Xeon D-1718T vs. 1736NT), and from what I can tell Enhanced Intel SpeedStep is enabled and working. I don't think this particular CPU family supports Intel Speed Shift though.

Based on my reading, I see that Enhanced Intel SpeedStep is enabled through cpufreq via the est interface:

https://man.freebsd.org/cgi/man.cgi?est
https://man.freebsd.org/cgi/man.cgi?cpufreq

As mentioned in the documentation links, you should see a line similar to this in your boot logs if est is enabled: est%d: <Enhanced SpeedStep Frequency Control> on cpu%d

I have powerd set to hiadaptive and can see the frequency shift around between the min and max as defined by the SpeedStep dev.est.%d.freq_settings. sysctl variable.

RobbieTT

@tman222

The EIST works just fine and I have mine set on adaptive (ie down from hi-adaptive) and the frequency varies as you would expect. The bonus of Speed Shift is that it reacts much faster than EIST, enabling a larger power saving whilst preserving performance for when you need it.

The Intel docs do stipulate that Speed Shift is included in this family, same for Speed Select Technology. If it can be achieved though is a different matter and still open for debate. We may be missing part of the jigsaw puzzle and the documents are not written with much clarity when it comes to derivatives of the main CPU line. Moreover, there is no mention of BSD in the Intel documents but there is kernel support in Linux v5.3 onwards.

️

edit: This popped-up on the FreeBSD forum in the last few days:
https://forums.freebsd.org/threads/shell-script-to-adjust-intel-speed-shift-technology-on-the-fly.90841/

RobbieTT

The Linux kernel support for Intel Speed Select Technology is covered here:

https://docs.kernel.org/admin-guide/pm/intel-speed-select.html

tman222

Thanks for clarifying. From your initial post it sounded like you were also asking whether EIST was working / enabled, hence my response.

Regarding Speed Shift - are you saying you have an option in the BIOS to enable it but then it's not exposed / supported by the OS? I just looked and I don't see the hwpstate_intel sysctl variables on my end either. I do see that machdep.hwpstate_pkg_ctrl = 1. Have you tried setting that 0 to see if it changes anything? Also, there is a debug.hwpstate_verbose tunable - have you tried enabling that to see if it spits out anything interesting during boot up that could provide a further clue why it might not be getting enabled in the OS?

stephenw10

Yeah it soon becomes unclear exactly what 'EIST' entails after some reading. In FreeBSD however EIST is SpeedStep and not SpeedShift as far as I know. And I've yet to see anything about Speed Select Technology.

It seems likely the SpeedShift driver may just not support that CPU yet. However it could be just using the data passed via ACPI in which case maybe that system is just not passing it? Or not in a form the driver expects.

If you boot Linux on that hardware can you see it using SpeedShift?

tman222

Figured it out! There as a BIOS option "Hardware P-States" that was set to "Disable". I enabled this and after that the hwpstate_intel driver loaded and Speed Shift settings were there in pfSense.

@RobbieTT - if you are using a Supermicro system, look for the "Hardware P-States" setting under Advanced Power Management Configuration. I set mine to from "Disable" to "Native Mode" and things started working.

RobbieTT

@stephenw10

Yep, the Intel naming conventions are beyond known science and using the same letters in an acronym for a near identical and related term is just bonkers. Maybe cracking this will be second task for quantum computers, just after they tackle Cisco's pricing structure.

I did load Ubuntu on this machine just to test it out of the box and Speed Shift (or later) seemed to work as expected. It was only well after that did I realise that not all OS kernels supported it and just happened to pick one that did.

I guess I could put an additional SSD into it and install Ubuntu again for a proper look around but I already think that I am way out of my depth just reading the Intel documents.

️

RobbieTT

@tman222 said in Enhanced Intel SpeedStep / Speed Shift - Are they fully supported?:

Figured it out! There as a BIOS option "Hardware P-States" that was set to "Disable". I enabled this and after that the hwpstate_intel driver loaded and Speed Shift settings were there in pfSense.

@RobbieTT - if you are using a Supermicro system, look for the "Hardware P-States" setting under Advanced Power Management Configuration. I set mine to from "Disable" to "Native Mode" and things started working.

You are right! Great spot! There are 2 different Hardware P-State settings, the first is enabled by default and has additional setting beyond it but this second one is different and disabled by default:

Hardware PM State Control
Hardware P-States
This setting allows you to select between OS and hardware-controlled P-states. Selecting Native Mode allows the OS to choose a P-state. Selecting Out of Band Mode allows the hardware to autonomously choose a P-state without OS guidance. Selecting Native Mode with No Legacy Support functions as Native Mode with no support for older hardware.

The options are Disable, Native Mode, Out of Band Mode, and Native Mode with No Legacy Support.

Do we collectively think that Native (allowing the OS to choose a state) is ok with BSD (I think it is) or is Out of Band Mode a better option?

️

tman222

@RobbieTT said in Enhanced Intel SpeedStep / Speed Shift - Are they fully supported?:

Do we collectively think that Native (allowing the OS to choose a state) is ok with BSD (I think it is) or is Out of Band Mode a better option?

️

I was wondering this myself - if Out of Band mode is chosen does the hwpstate_intel driver still attach during boot? I would think that Native Mode is required for the driver to load, but I could be wrong.

stephenw10

The driver only configures the parameters for the hardware to use it doesn't actually set P-states directly. So.... unclear!

tman222

@stephenw10 said in Enhanced Intel SpeedStep / Speed Shift - Are they fully supported?:

The driver only configures the parameters for the hardware to use it doesn't actually set P-states directly. So.... unclear!

Tried "Out of Band Mode" and the hwpstate_intel driver no longer loaded / attached during boot and Speed Shift settings were also gone in pfSense. Looks like it needs to be set to Native Mode to work properly.

q54e3w

@tman222 mine is set to native and seems to be functioning. I have a few days worth of 23.09 SNMP data illustrating the delta between 2.7 hiadaptive and 23.09 ‘80’ I’ll share when I get back to my desk.

tman222

@q54e3w said in Enhanced Intel SpeedStep / Speed Shift - Are they fully supported?:

@tman222 mine is set to native and seems to be functioning. I have a few days worth of 23.09 SNMP data illustrating the delta between 2.7 hiadaptive and 23.09 ‘80’ I’ll share when I get back to my desk.

Thanks @q54e3w - I look forward to seeing those statistics. I currently have mine set to 60 and the CPU is already running 2-3 degrees C cooler than before (when it was using EIST).

RobbieTT

@tman222

Set mine this evening and set the GUI slider to 80 for now and the recommended 'per core' although it defaulted to 'package'. Not sure if you needed to un-tick the PowerD box but did so anyway:

20231110-Router-7-pfSense-Speed Step Enabled at 80.png

Noticed a couple of other changes on the Dashboard. The max frequency now shows as 2700 MHz (vice 2701) and the minimum freq displays 799 MHz (vice 800):

20231110-Router-7-BIOS to Intel Speed Shift-80-No turbo 2700 MHz question.png

(The above shows 8-cores as I don't have HT enabled)

Turbo still seems to work though:

load  31%, current freq 3199 MHz ( 0), wanted freq 5400 MHz
load  30%, current freq 3199 MHz ( 0), wanted freq 5400 MHz
load  39%, current freq  799 MHz ( 0), wanted freq 5400 MHz
load  65%, current freq  799 MHz ( 0), wanted freq 5400 MHz

Shame I don't get any power data from this Supermicro PSU:

20231110-Supermicro-No Power Consumption Displayed.png

According to my UPS my system may have reduced by a few watts but too early to tell if that is really the case.

️