Upgrade to 2.3 CPU Running Harder

cmb

The dashboard is a bit more CPU-intensive than before, that's likely why. If you close out of the dashboard it probably drops back down to what it was before.

coxhaus

The load average is about the same as before. Should the load average show higher since the CPU is running faster? Maybe load average is based on current clock speed so it is showing the same but running faster.

Harvy66

I also noticed that mine is sitting at 800mhz instead of the fairly normal 400mhz. What's interesting the is CPU utilization is quite a bit lower. My guess is not so much that the new dashboard is more CPU intensive, but that it's more bursty and PowerD doesn't want to be constantly changing the CPU speed for short bursts, even if the average is lower.

To give reference, I used to sit around 4%-8% CPU load while monitoring System Activity, RRD, and the Dashboard. Now I'm hanging around 1%-3%. Even if I scale for the 400mhz vs 800mhz, that's still 4%-8% vs 2%-6%. Again, in my case.

Mr. White

I'm getting the same issue
Before the upgrade the cpu was idling at 499MHz but now after the upgrade it's not going lower than 1197MHz
This is checked through the console so no GUI to interfere..

chaos777b

I have a AMD Athlon 5350 APU with Radeon(tm) R3 and I'm seeing 40-50% cpu usage when pushing 60Mbps down after upgrading to 2.3 Before upgrading to 2.3 it would normally be under 10% when pushing 150-170Mbps down so this is a rather large change.

Guest

Perhaps you all should consider to try out the PowerD (hi adaptive) setting that let your cpu scale
from the lowest bottom to the highest top such as needed or pending on the entire workload.
Would this a point to go with?

Mr. White

I am running powerd which is why I saw this problem.
Before upgrade it was idling at a way lower speed (499MHz).

Some stats


[2.3-RELEASE][admin@fw01.tessier-ashpool.int]/root: uptime
 9:31AM  up 1 day, 29 mins, 2 users, load averages: 0.00, 0.00, 0.00
[2.3-RELEASE][admin@fw01.tessier-ashpool.int]/root: ./show_cpufreq.sh
dev.cpu.0.freq: 1197
[2.3-RELEASE][admin@fw01.tessier-ashpool.int]/root:

Just kind of annoying :P

Mr. White

Actually looks like it's a change in avaliable freq_levels
Doesn't even list the lower ones anymore…


[2.3-RELEASE][admin@fw01.tessier-ashpool.int]/root: sysctl -a | grep freq_levels
dev.cpu.0.freq_levels: 3193/9875 3192/9125 3059/8250 2926/7500 2793/6875 2660/6250 2527/5750 2394/5250 2261/4750 1197/2750
[2.3-RELEASE][admin@fw01.tessier-ashpool.int]/root:

provels

On my VIA C7 1GHz box, no alternate frequencies are shown anymore. Old, I know. Must be the FreeBSD ver.

coxhaus

How do I run ./show_cpufreq.sh ? I tried from command prompt under diagnostics. It shows as not a good command.

So are the lower run states missing from my CPU Xeon 5148?

coxhaus

I figured it out. They removed my lower frequencies.

sysctl -a | grep cpu.0.freq

dev.cpu.0.freq_levels: 2331/88000 1998/71619
dev.cpu.0.freq: 1998

Maybe the lower frequencies don't really save that much heat and power. I still have 2 left.

paftdunk

@Mr.:

Actually looks like it's a change in avaliable freq_levels
Doesn't even list the lower ones anymore…


[2.3-RELEASE][admin@fw01.tessier-ashpool.int]/root: sysctl -a | grep freq_levels
dev.cpu.0.freq_levels: 3193/9875 3192/9125 3059/8250 2926/7500 2793/6875 2660/6250 2527/5750 2394/5250 2261/4750 1197/2750
[2.3-RELEASE][admin@fw01.tessier-ashpool.int]/root:

Thanks for the pointer. It turns out this was a deliberate regression introduced in FreeBSD 10.2:
https://svnweb.freebsd.org/base?view=revision&revision=276986

After some hunting around and trial and error, I was able to get my system more or less back to where it was in pfSense 2.2.x/FBSD10.1.

My Intel i3-4130T system's minimum clock used to be 300MHz. pfSense 2.3/FBSD10.3 bumped this up to 800MHz, as Mr. White found above. With the change below, my minimum clock speed is actually 100MHz.

Warning: before changing this, make sure your PowerD setting in System->Advanced->Misc. is not Minimum. I had changed that after running into this problem and learned the hard way that permanently clamping the CPU to 100MHz max makes the GUI completely unusable.

Add the following line to /boot/loader.conf.local

hint.p4tcc.0.disabled="0"

Some systems may also require

hint.acpi_throttle.0.disabled="0"

After a reboot, sysctl dev.cpu confirms:

dev.cpu.0.freq_levels: 2900/35000 2800/33218 2600/30093 2500/28743 2300/25796 2200/24202 2100/22958 1900/20234 1800/19072 1600/16509 1500/15121 1400/14056 1225/12299 1200/11711 1100/10435 962/9130 900/8522 800/7328 700/6412 600/5496 500/4580 400/3664 300/2748 200/1832 100/916
dev.cpu.0.freq: 100

And of course dev.cpu.0.freq is scaling dynamically all the way up to 2900MHz as needed. I specifically built this system to use as little power as possible, so I'm glad to have found a workaround for this. I don't agree with the FBSD maintainers' decision to degrade power management, but it's up to the pfSense maintainers to determine whether to make this configurable, or to default back to the tried and true behavior we've all had for years.

robi

In the freeBsd patch quoted above they say

These CPU speed control techniques are usually unhelpful at best.

I can confirm this. On my Supermicro a1sri-2758f for example, enabling PowerD makes things worse. NAT throughput is only about 400Mbit/sec for the first 8-10 seconds - that's the time required for the CPU to scale up and allow NATting at 1Gbps. Enabling PowerD actually reduces performance, since network load appears in spikes, with CPU freq scaling all these spikes would be limited to 400 Mbit.

Guest

My Intel i3-4130T system's minimum clock used to be 300MHz. pfSense 2.3/FBSD10.3 bumped this up to 800MHz, as Mr. White found above. With the change below, my minimum clock speed is actually 100MHz.

Each new software even needs or want a bit more horse power and this is not only tended to the pfSense
or FreeBSD system. If you want to install perhaps a Windows 2012 Server on server hardware from 2008
you will also surprised about that performance then. Running at 100MHz will be saving perhaps power but
are not enough for a more modern WebGui to run flawless.

Mr. White

@coxhaus:

How do I run ./show_cpufreq.sh ? I tried from command prompt under diagnostics. It shows as not a good command.

So are the lower run states missing from my CPU Xeon 5148?

This actually a script i made for myself because I'm lazy…
Just a sysctl with a grep to get current cpu speed.

paftdunk

@robi:

In the freeBsd patch quoted above they say

These CPU speed control techniques are usually unhelpful at best.

I can confirm this. On my Supermicro a1sri-2758f for example, enabling PowerD makes things worse. NAT throughput is only about 400Mbit/sec for the first 8-10 seconds - that's the time required for the CPU to scale up

I don't blame you for turning it off. But CPU scaling happens so quickly, the OS can't even keep up. That's why Intel moved so much of this power management into the silicon. Voltage and C-state changes happen before the OS even has a chance to find out. Whatever you were seeing is completely broken, and not caused solely by CPU frequency scaling.

It sounds like the software side of power management in FreeBSD isn't great, which is no surprise. This thread is about why 2.3 is running at higher clocks. Folks should benchmark their local workloads and determine which of the available settings strike,the right balance for them.

Mr. White

Did the changes recommended by Paftdunk and can confirm that it worked for me.


[2.3-RELEASE][admin@fw01.tessier-ashpool.int]/root: ./show_cpufreq.sh 
dev.cpu.0.freq: 149


[2.3-RELEASE][admin@fw01.tessier-ashpool.int]/root: sysctl -a | grep freq_levels
dev.cpu.0.freq_levels: 3193/9875 3192/9125 3059/8250 2926/7500 2793/6875 2660/6250 2527/5750 2394/5250 2261/4750 1978/4156 1695/3562 1413/2968 1197/2750 1047/2406 897/2062 748/1718 598/1375 448/1031 299/687 149/343

paftdunk

Just a sysctl with a grep to get current cpu speed.

Fwiw, sysctl can interrogate subgroups directly, allowing you to skip the grep. E.g.,

root@pfsense:~ # sysctl dev.cpu.0.freq
dev.cpu.0.freq: 300

root@pfsense:~ # sysctl dev.cpu.0
dev.cpu.0.temperature: 33.0C
dev.cpu.0.coretemp.throttle_log: 0
dev.cpu.0.coretemp.tjmax: 85.0C
dev.cpu.0.coretemp.resolution: 1
dev.cpu.0.coretemp.delta: 52
dev.cpu.0.cx_usage: 100.00% 0.00% last 677us
dev.cpu.0.cx_lowest: C1
dev.cpu.0.cx_supported: C1/1/1 C2/2/148
dev.cpu.0.freq_levels: 2900/35000 2800/33218 2600/30093 2500/28743 2300/25796 2200/24202 2100/22958 1900/20234 1800/19072 1600/16509 1500/15121 1400/14056 1225/12299 1200/11711 1100/10435 962/9130 900/8522 800/7328 700/6412 600/5496 500/4580 400/3664 300/2748 200/1832 100/916
dev.cpu.0.freq: 300
dev.cpu.0.%parent: acpi0
dev.cpu.0.%pnpinfo: _HID=none _UID=0
dev.cpu.0.%location: handle=\_PR_.CPU0
dev.cpu.0.%driver: cpu
dev.cpu.0.%desc: ACPI CPU

provels

I added these back to loader.conf and got throttling back on my VIA C7 chip, too.

hint.p4tcc.0.disabled="0"
hint.acpi_throttle.0.disabled="0"

dev.cpu.0.freq_levels: 1007/-1 881/-1 755/-1 629/-1 503/-1 377/-1 251/-1 125/-1
dev.cpu.0.freq: 125

pLu

@robi:

In the freeBsd patch quoted above they say

These CPU speed control techniques are usually unhelpful at best.

I can confirm this. On my Supermicro a1sri-2758f for example, enabling PowerD makes things worse. NAT throughput is only about 400Mbit/sec for the first 8-10 seconds - that's the time required for the CPU to scale up and allow NATting at 1Gbps. Enabling PowerD actually reduces performance, since network load appears in spikes, with CPU freq scaling all these spikes would be limited to 400 Mbit.

How does sysctl hw.acpi.cpu.cx_lowest=C2 affect NAT performance (with PowerD off)?