Can't reach max turbo CPU frequency

Conjurer

I finally got it! I was able to alter the BIOS using UEFI editor so that I could enable the C states option. Created a new BIOS and reflashed.

Now in pfSense I see C states C1, C2 and C3, which I didn't see before:

sysctl -a dev.cpu | grep cx
dev.cpu.11.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc
dev.cpu.11.cx_usage_counters: 323 4007 17963
dev.cpu.11.cx_usage: 1.44% 17.97% 80.57% last 28898us
dev.cpu.11.cx_lowest: C8
dev.cpu.11.cx_supported: C1/1/1 C2/2/127 C3/3/1048
dev.cpu.10.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc
dev.cpu.10.cx_usage_counters: 356 3761 18160
dev.cpu.10.cx_usage: 1.59% 16.88% 81.51% last 18845us
dev.cpu.10.cx_lowest: C8
dev.cpu.10.cx_supported: C1/1/1 C2/2/127 C3/3/1048
...

And with dev.hwpstate_intel.[0..11].epp set to 0 I get turbo frequencies on all cores, which I could never get before (in pfSense that is):

sysctl -a dev.cpu | grep 'freq:'
dev.cpu.11.freq: 3583
dev.cpu.10.freq: 3580
dev.cpu.9.freq: 3583
dev.cpu.8.freq: 3583
dev.cpu.7.freq: 3583
dev.cpu.6.freq: 3583
dev.cpu.5.freq: 3583
dev.cpu.4.freq: 3583
dev.cpu.3.freq: 4776
dev.cpu.2.freq: 4776
dev.cpu.1.freq: 4776
dev.cpu.0.freq: 4776

stephenw10

Hmm, enabling lower C-states is the only change you made?

It sounds like it's hitting a limit trying to drive all the cores at max turbo. If it's not thermal it could be a power limitation in the board. Enabling lower C-states may reduce instantaneous power use in some cores allowing it. Though I would expect all cores to be at C0 when running in turbo.

Steve

Conjurer

@stephenw10
Thanks for your reply. Yes, enabling C states and before that I changed the following parameters in BIOS to allow the CPU to fully boost:

TDC Current Limit: 640
AC Loadline: 180
Tcc Activation Offset: 15 (limit 85 °C)
Tcc Offset Time Window: 3 secconds
Power limit 1: 55W
Power limit 2: 55W
Power Limit 1 Time Window: 28 seconds

The factory BIOS was very crippled. The Speed Shift option was not available, but manufacturer tells me it's enabled by default. Well yes, but it's pointing to the wrong offset in NVRAM (how?!). So with some analysis I got the correct offset and modified the UEFI NVRAM and switched it to 01. Also changed the menu-structure a bit, so that I could both have access to the limited factory options (with the C states option) as well as the advanced options (above mentioned settings and more).

However I'm not planning to run dev.hwpstate_intel.[0..11].epp to 0 at all times, that would be a waste of energy. I will make a simple script and cron it to switch between for example 35, 50, 75 and 95 depending on time of day. For example, I will be running on 95 at night, limiting power usage. By the way I could not find any documentation about thresholds related to the dev.hwpstate_intel.[0..11].epp setting. I know 0 is max. performance and 100 is max. energy efficiency, but how do all the values in between relate to CPU frequency?

stephenw10

Good question. The man page doesn't show anything further.
Probably buried in the Intel docs somewhere.
The only hardware I have that supports it I have set at 80. That gives a pretty good result.

Conjurer

Oh and I had to set loader tunable machdep.hwpstate_pkg_ctrl to 0 (default is 1) in a newly created file /boot/loader.conf.local. When using the default value 1 for machdep.hwpstate_pkg_ctrl and setting epp for all cores to 0, the P-cores would stay at ~2780 MHz.

stephenw10

Yes we have that as the recommended setting in the new gui options that will be in 23.09.

It's hard to measure it though since simply running sysctl causes the CPU(cores) to ramp up.

A Former User

@Conjurer I'm in setting up a new CWWK i7-1265u with pfSense 23.09 using fta's 6/8/2023 BIOS . I have got speed shift appearing to work with one quirk that I can't find an answer to:

CPU Type 	12th Gen Intel(R) Core(TM) i7-1265U
                Current: 2806 MHz, Max: 2688 MHz
                12 CPUs
                AES-NI CPU Crypto: Yes (active)
                IPsec-MB Crypto: Yes (active)
                QAT Crypto: No

That screen capture was with speed Shift active and a Core Level control power preference setting of 30. The 2688 Mhz response looks suspicious, see dev.cpu freq_levels below.

A "sysctl -a dev.cpu | grep cx" for cpu 0 shows:
dev.cpu.0.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc
dev.cpu.0.cx_usage_counters: 221121 1637025 8068774
dev.cpu.0.cx_usage: 2.22% 16.49% 81.28% last 477us
dev.cpu.0.cx_lowest: C8
dev.cpu.0.cx_supported: C1/1/1 C2/2/127 C3/3/1048

A freq_levels response shows:
dev.cpu.0.freq_levels: 2688/-1

I was expecting more freq_levels and not a -1 response for the mW value. Is there a BIOS setting I'm missing?

Edit:
with power preference setting of 0:
12th Gen Intel(R) Core(TM) i7-1265U
Current: 4776 MHz, Max: 2688 MHz

The 4776 MHz is expected. I don't understand the 2688 MHz max.

stephenw10

AFAIK the frequency levels show there are for speedstep or other OS control methods. You won't see anything there unless powerd is enabled and SpeedShift replaces that.

Those values are passed from the BIOS when speedstep is enabled. -1 indicates no mW value was passed.

Conjurer

@rschell said in Can't reach max turbo CPU frequency:

sysctl -a dev.cpu | grep cx

Well to be honest, I don't know either where the Max: 2688 MHz is coming from. I just ignore it, and focus on the values presented as current. As for all other readings you posted, I have the same.

As @stephenw10 mentions, if you disable Speed Shift (loader tunable hint.hwpstate_intel.0.disabled), and enable SpeedStep (PowerD) you'll see various frequencies when performing sysctl -a dev.cpu | grep 'freq_levels\|freq'. But I wouldn't recommend it, Speed Shift is much more responsive and you can't fully utilize the pontential of this generation of Intel processor with SpeedStep.

Conjurer

@rschell Maybe of any help or just sharing my knowledge: I found that using command stress would give me best results when it comes to testing boost speeds. In combination with using cpuset for targeting specific cores (P-cores and/or E-cores).

Installing stress:

pkg install stress
rehash

Command cpuset -l 0-3 stress -c 4 would (if hyper-threading is enabled) stress both P-cores with 4 worker processes.