Can't reach max turbo CPU frequency

Conjurer · Jul 15, 2023, 10:46 AM

Hi, I'm using a CWWK i7-1265u unit with pfSense 2.7 (and now pfSense+ 23.05.1) and I can't reach the P-cores specified maximum turbo speed of 4.8GHz and the E-cores turbo of 3.6GHz.

Background info
The Intel i7-1265u CPU has 2 P-cores which allows up to 4.8GHz, and 8 E-cores up to 3.6GHz. It supports and is using SpeedShift:

cat /var/run/dmesg.boot | grep "Speed Shift"
hwpstate_intel0: <Intel Speed Shift> on cpu0
hwpstate_intel1: <Intel Speed Shift> on cpu1
hwpstate_intel2: <Intel Speed Shift> on cpu2
hwpstate_intel3: <Intel Speed Shift> on cpu3
hwpstate_intel4: <Intel Speed Shift> on cpu4
hwpstate_intel5: <Intel Speed Shift> on cpu5
hwpstate_intel6: <Intel Speed Shift> on cpu6
hwpstate_intel7: <Intel Speed Shift> on cpu7
hwpstate_intel8: <Intel Speed Shift> on cpu8
hwpstate_intel9: <Intel Speed Shift> on cpu9
hwpstate_intel10: <Intel Speed Shift> on cpu10
hwpstate_intel11: <Intel Speed Shift> on cpu11

I've set tunables dev.hwpstate_intel.[0..11].epp all to 0. I've also set hw.acpi.cpu.cx_lowest, performance_cx_lowest and economy_cx_lowest to Cmax. Which translates to C8 being set for all cores:

sysctl -a dev.cpu | grep cx
dev.cpu.11.cx_method: C1/hlt
dev.cpu.11.cx_usage_counters: 3665
dev.cpu.11.cx_usage: 100.00% last 9497us
dev.cpu.11.cx_lowest: C8
dev.cpu.11.cx_supported: C1/1/0
dev.cpu.10.cx_method: C1/hlt
dev.cpu.10.cx_usage_counters: 3878
dev.cpu.10.cx_usage: 100.00% last 5673us
dev.cpu.10.cx_lowest: C8
dev.cpu.10.cx_supported: C1/1/0
(etc., cores 0..9 are the same as above)
...

(BTW, why is it showing support for only C1? I know it supports deeper C-states than C1. I would expect at minimum C-states C1, C2 and C3 shown)

When idle it gives me:

sysctl -a dev.cpu | grep 'freq:'
dev.cpu.11.freq: 3483
dev.cpu.10.freq: 3483
dev.cpu.9.freq: 3483
dev.cpu.8.freq: 3483
dev.cpu.7.freq: 3483
dev.cpu.6.freq: 3384
dev.cpu.5.freq: 3384
dev.cpu.4.freq: 3384
dev.cpu.3.freq: 4180
dev.cpu.2.freq: 4180
dev.cpu.1.freq: 4180
dev.cpu.0.freq: 4180

As you can see, cores 0..3 are the P-cores (hyperthreading) which are at a small 4.2GHz, all others are the E-cores are at ~3.5GHz.
When performing cpuset -l0 yes > /dev/null for performing a loadtest on core 0 the frequencies do not change, they stay at 4180MHz for the P-cores and 3384MHz for the E-cores. There is no thermal throttling involved due to decent cooling. Temperatures while performing the loadtest:

sysctl -a dev.cpu | grep 'temperature:'
dev.cpu.11.temperature: 36.0C
dev.cpu.10.temperature: 37.0C
dev.cpu.9.temperature: 39.0C
dev.cpu.8.temperature: 37.0C
dev.cpu.7.temperature: 49.0C
dev.cpu.6.temperature: 49.0C
dev.cpu.5.temperature: 47.0C
dev.cpu.4.temperature: 50.0C
dev.cpu.3.temperature: 36.0C
dev.cpu.2.temperature: 38.0C
dev.cpu.1.temperature: 64.0C
dev.cpu.0.temperature: 64.0C

In BIOS both PL1 and PL2 are set to 55W. Set TCC Limit to 85 °C, and Power Limit Time Window to 28 sec. Powersupply has plenty room for delivering the needed power.

Already tried
What I already tried is this, but all didn't allow me to reach the turbo frequencies:

Used both pfSense 2.7 and pfSense+ 23.5.01 (default installation)
Removed all tunables hw.acpi.cpu.cx_lowest, performance_cx_lowest, economy_cx_lowest, and dev.hwpstate_intel.[0..11].epp
Used higher C-state values C1, C2, C3 for hw.acpi.cpu.cx_lowest, performance_cx_lowest and economy_cx_lowest
Used values ranging from 10, 20, 35, 50, 95, 100 for dev.hwpstate_intel.[0..11].epp
Disabled SpeedShift and enabled SpeedStep
While using SpeedStep enabled PowerD and set AC Power, Battery Power, Unknown Power to Maximum and Hiadaptive
Load test on different cores 0..3, and combinations
Countless enabling, disabling, changing values, etc. back and forth in BIOS

Performance benchmarking in Ubuntu 23.04 allowed me to reach the 4.8 GHz speeds. So I know it's possible.

Questions

How can I instruct pfSense/FreeBSD to burst to maximum speeds when needed?
Why I am seeing only C-state C1? Where are the others?

fireodo · Jul 15, 2023, 10:43 AM

This post is deleted!

Conjurer · Jul 16, 2023, 2:07 PM

After more fiddling with options in the BIOS I have somewhat of a solution/explanation. Solution is: choosing only one P-core instead of all (two) in BIOS. Now I see the frequencies I expected to see.

sysctl -a dev.cpu | grep 'freq:'
dev.cpu.9.freq: 3583
dev.cpu.8.freq: 3583
dev.cpu.7.freq: 3583
dev.cpu.6.freq: 3583
dev.cpu.5.freq: 3583
dev.cpu.4.freq: 3583
dev.cpu.3.freq: 3583
dev.cpu.2.freq: 3583
dev.cpu.1.freq: 4776
dev.cpu.0.freq: 4776

The tradeoff when using one P-core is obviously having only one instead of two P-cores. The benefit is that all cores (P-core and E-cores) can reach their specified turbo frequencies! But I don't really know if the benefit vs. missing one P-core is worth it.

I think FreeBSD (v14.0-RELEASE) is lacking full support for heterogeneous cores, such as this one (Alderlake). In means that the scheduler doesn't know when to assign threads to specific types of cores.

Conjurer · Jul 19, 2023, 7:45 PM

I finally got it! I was able to alter the BIOS using UEFI editor so that I could enable the C states option. Created a new BIOS and reflashed.

Now in pfSense I see C states C1, C2 and C3, which I didn't see before:

sysctl -a dev.cpu | grep cx
dev.cpu.11.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc
dev.cpu.11.cx_usage_counters: 323 4007 17963
dev.cpu.11.cx_usage: 1.44% 17.97% 80.57% last 28898us
dev.cpu.11.cx_lowest: C8
dev.cpu.11.cx_supported: C1/1/1 C2/2/127 C3/3/1048
dev.cpu.10.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc
dev.cpu.10.cx_usage_counters: 356 3761 18160
dev.cpu.10.cx_usage: 1.59% 16.88% 81.51% last 18845us
dev.cpu.10.cx_lowest: C8
dev.cpu.10.cx_supported: C1/1/1 C2/2/127 C3/3/1048
...

And with dev.hwpstate_intel.[0..11].epp set to 0 I get turbo frequencies on all cores, which I could never get before (in pfSense that is):

sysctl -a dev.cpu | grep 'freq:'
dev.cpu.11.freq: 3583
dev.cpu.10.freq: 3580
dev.cpu.9.freq: 3583
dev.cpu.8.freq: 3583
dev.cpu.7.freq: 3583
dev.cpu.6.freq: 3583
dev.cpu.5.freq: 3583
dev.cpu.4.freq: 3583
dev.cpu.3.freq: 4776
dev.cpu.2.freq: 4776
dev.cpu.1.freq: 4776
dev.cpu.0.freq: 4776

stephenw10 · Aug 2, 2023, 6:35 AM

Hmm, enabling lower C-states is the only change you made?

It sounds like it's hitting a limit trying to drive all the cores at max turbo. If it's not thermal it could be a power limitation in the board. Enabling lower C-states may reduce instantaneous power use in some cores allowing it. Though I would expect all cores to be at C0 when running in turbo.

Steve

Conjurer · Aug 2, 2023, 6:35 AM

@stephenw10
Thanks for your reply. Yes, enabling C states and before that I changed the following parameters in BIOS to allow the CPU to fully boost:

TDC Current Limit: 640
AC Loadline: 180
Tcc Activation Offset: 15 (limit 85 °C)
Tcc Offset Time Window: 3 secconds
Power limit 1: 55W
Power limit 2: 55W
Power Limit 1 Time Window: 28 seconds

The factory BIOS was very crippled. The Speed Shift option was not available, but manufacturer tells me it's enabled by default. Well yes, but it's pointing to the wrong offset in NVRAM (how?!). So with some analysis I got the correct offset and modified the UEFI NVRAM and switched it to 01. Also changed the menu-structure a bit, so that I could both have access to the limited factory options (with the C states option) as well as the advanced options (above mentioned settings and more).

However I'm not planning to run dev.hwpstate_intel.[0..11].epp to 0 at all times, that would be a waste of energy. I will make a simple script and cron it to switch between for example 35, 50, 75 and 95 depending on time of day. For example, I will be running on 95 at night, limiting power usage. By the way I could not find any documentation about thresholds related to the dev.hwpstate_intel.[0..11].epp setting. I know 0 is max. performance and 100 is max. energy efficiency, but how do all the values in between relate to CPU frequency?

stephenw10 · Aug 2, 2023, 12:02 PM

Good question. The man page doesn't show anything further.
Probably buried in the Intel docs somewhere.
The only hardware I have that supports it I have set at 80. That gives a pretty good result.

Conjurer · Sep 6, 2023, 5:56 AM

Oh and I had to set loader tunable machdep.hwpstate_pkg_ctrl to 0 (default is 1) in a newly created file /boot/loader.conf.local. When using the default value 1 for machdep.hwpstate_pkg_ctrl and setting epp for all cores to 0, the P-cores would stay at ~2780 MHz.

stephenw10 · Sep 6, 2023, 12:17 PM

Yes we have that as the recommended setting in the new gui options that will be in 23.09.

It's hard to measure it though since simply running sysctl causes the CPU(cores) to ramp up.

Sep 9, 2023, 7:03 PM

@Conjurer I'm in setting up a new CWWK i7-1265u with pfSense 23.09 using fta's 6/8/2023 BIOS . I have got speed shift appearing to work with one quirk that I can't find an answer to:

CPU Type 	12th Gen Intel(R) Core(TM) i7-1265U
                Current: 2806 MHz, Max: 2688 MHz
                12 CPUs
                AES-NI CPU Crypto: Yes (active)
                IPsec-MB Crypto: Yes (active)
                QAT Crypto: No

That screen capture was with speed Shift active and a Core Level control power preference setting of 30. The 2688 Mhz response looks suspicious, see dev.cpu freq_levels below.

A "sysctl -a dev.cpu | grep cx" for cpu 0 shows:
dev.cpu.0.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc
dev.cpu.0.cx_usage_counters: 221121 1637025 8068774
dev.cpu.0.cx_usage: 2.22% 16.49% 81.28% last 477us
dev.cpu.0.cx_lowest: C8
dev.cpu.0.cx_supported: C1/1/1 C2/2/127 C3/3/1048

A freq_levels response shows:
dev.cpu.0.freq_levels: 2688/-1

I was expecting more freq_levels and not a -1 response for the mW value. Is there a BIOS setting I'm missing?

Edit:
with power preference setting of 0:
12th Gen Intel(R) Core(TM) i7-1265U
Current: 4776 MHz, Max: 2688 MHz

The 4776 MHz is expected. I don't understand the 2688 MHz max.

stephenw10 · Sep 9, 2023, 9:56 PM

AFAIK the frequency levels show there are for speedstep or other OS control methods. You won't see anything there unless powerd is enabled and SpeedShift replaces that.

Those values are passed from the BIOS when speedstep is enabled. -1 indicates no mW value was passed.

Conjurer · Sep 10, 2023, 10:45 AM

@rschell said in Can't reach max turbo CPU frequency:

sysctl -a dev.cpu | grep cx

Well to be honest, I don't know either where the Max: 2688 MHz is coming from. I just ignore it, and focus on the values presented as current. As for all other readings you posted, I have the same.

As @stephenw10 mentions, if you disable Speed Shift (loader tunable hint.hwpstate_intel.0.disabled), and enable SpeedStep (PowerD) you'll see various frequencies when performing sysctl -a dev.cpu | grep 'freq_levels\|freq'. But I wouldn't recommend it, Speed Shift is much more responsive and you can't fully utilize the pontential of this generation of Intel processor with SpeedStep.

Conjurer · Sep 17, 2023, 9:29 AM

@rschell Maybe of any help or just sharing my knowledge: I found that using command stress would give me best results when it comes to testing boost speeds. In combination with using cpuset for targeting specific cores (P-cores and/or E-cores).

Installing stress:

pkg install stress
rehash

Command cpuset -l 0-3 stress -c 4 would (if hyper-threading is enabled) stress both P-cores with 4 worker processes.