Sharing on a 1Gbps Router built(+Tests on clocked at 1.6GHz and Default)

nexusN

Previously the speed test had CPU clocked at 1.6GHz with Turboboost off, let's see if there is any change if clock is left at default:
Download:

Upload:

Result:

I have run a few test and the results are consistent, which is around 30% at the speed.
Download and Upload shouldn't show much difference, but the refresh rate of CPU utilization is too slow to be accurate to confirm if there exist a few % difference.

jwelter99

@nexusN:

Previously the speed test had CPU clocked at 1.6GHz with Turboboost off, let's see if there is any change if clock is left at default:
Download:

Upload:

Result:

I have run a few test and the results are consistent, which is around 30% at the speed.
Download and Upload shouldn't show much difference, but the refresh rate of CPU utilization is too slow to be accurate to confirm if there exist a few % difference.

Turn on PowerNow or whatever FreeBSD call's it as it has a pretty large impact on our E3 PF boxes.

nexusN

Sorry but I am not familiar with pf, what actually is PowerNow?
I heard something called PowerD, but not sure how can I turn that on nor config it.
Please kindly advise, thanks.

stephenw10

Yes powerd is what you want.
It's in System: Advanced: Miscellaneous: in the webgui.

Steve

nexusN

@stephenw10:

Yes powerd is what you want.
It's in System: Advanced: Miscellaneous: in the webgui.

Steve

Thanks a lot, I have found and checked the box.
It would be nice to have such an option to save some energy ;D

nexusN

Sadly :(, PowerD seems to be hurting performance to quite a significant degree, the below is a comparison of throughput with PowerD ON and OFF:

stephenw10

Check your logs, make sure that powerd is working correctly.
You may have to change the kernel timecounter.
Powerd has a number of tuning options the govern how quickly it ramps up the cpu speed. These can be tuned to give you better performance/power ratio. See the man page.

Steve

nexusN

@stephenw10:

Check your logs, make sure that powerd is working correctly.
You may have to change the kernel timecounter.
Powerd has a number of tuning options the govern how quickly it ramps up the cpu speed. These can be tuned to give you better performance/power ratio. See the man page.

Steve

Thanks for the idea,
I will try to see if I can tune to strike a balance.

stephenw10

You can do some testing manually. See my experiences with it here:
http://forum.pfsense.org/index.php/topic,20095.msg161139.html#msg161139

Even at the most conservative setting it should ramp up in a few seconds. How long does that test take? It looks more like it's not running correctly.

Steve

nexusN

@stephenw10:

You can do some testing manually. See my experiences with it here:
http://forum.pfsense.org/index.php/topic,20095.msg161139.html#msg161139

Even at the most conservative setting it should ramp up in a few seconds. How long does that test take? It looks more like it's not running correctly.

Steve

Maybe I can try to "warm it up" before carrying out the test.
The test takes a very short period, like 5 s for downloading and another 5s for uploading, the time maybe too short for the router to response.

nexusN

More on the PowerD tests,
as mentioned, the PowerD should be able to speed up upon the need of performance, I have tried to run a few tests continuously and found the below:

Each pair of column in the network utilization graph refers to a single test, the bench result on the left is referring to the last test(the pair at the right most).

I was wrong on my previous statement, in a pair, the first column should be for uploading and the second column is referring to downloading,
as all the tests are showing a result for a faster downloading and slower uploading.
This is important that, it means the Uploading test triggered the software to speed up the CPU thus a better result is obtained on downloading which follows closely while the uploading itself has been stuck with a lower performance.

Also, from the test it seems the degree the CPU speeds up is quite uncertain, the flat top of each columns states that the "speed up" actually stopped at a certain level during a transfer, and this level may not be at the best performance.

nexusN

Just now I tried to add the two command in loader.conf:

hint.p4tcc.0.disabled=1
hint.acpi_throttle.0.disabled=1

Then it seems with the throttling disabled the clock of CPU is fixed as I have set in BIOS:
1.6GHz

[2.0.1-RELEASE][admin@nchannel.nelsewhere.com]/root(2): sysctl dev.cpu.0.freq_levels
dev.cpu.0.freq_levels: 1600/39000

And upon a test, there is no more performance drop with the same tests with PowerD on.
Here comes another questions, if the CPU clock has been fixed in BIOS already, is there any other pros to leave PowerD on?
I have only one 2.5" harddisk in the box, it can't be turned off for idle even PowerD is on right?
Sad that I don't have a power meter to get the ans.

Thanks for your kind attention.

SlowGrind

That is a great looking rig… And the cost is about the same as an Atom board. I might just build this same setup.

nexusN

@nexusN:

Just now I tried to add the two command in loader.conf:
hint.p4tcc.0.disabled=1
hint.acpi_throttle.0.disabled=1
Then it seems with the throttling disabled the clock of CPU is fixed as I have set in BIOS:
1.6GHz
[2.0.1-RELEASE][admin@nchannel.nelsewhere.com]/root(2): sysctl dev.cpu.0.freq_levels
dev.cpu.0.freq_levels: 1600/39000
And upon a test, there is no more performance drop with the same tests with PowerD on.
Here comes another questions, if the CPU clock has been fixed in BIOS already, is there any other pros to leave PowerD on?
I have only one 2.5" harddisk in the box, it can't be turned off for idle even PowerD is on right?
Sad that I don't have a power meter to get the ans.

Thanks for your kind attention.

Apart from PowerD,
More on PowerSaving(Hopefully):
On some Googles, I have found that pfsense 2.0.1 on default doesn't enable C3 state for CPUs,
and I checked it to be true by:

[2.0.1-RELEASE][admin@nchannel.nelsewhere.com]/root(5): sysctl dev.cpu |grep cx
dev.cpu.0.cx_supported: C1/1 C2/80 C3/104
dev.cpu.0.cx_lowest: C1
dev.cpu.0.cx_usage: 100.00% 0.00% 0.00% last 500us
dev.cpu.1.cx_supported: C1/1 C2/80 C3/104
dev.cpu.1.cx_lowest: C1
dev.cpu.1.cx_usage: 100.00% 0.00% 0.00% last 500us

To get C3 state works, the little tweak is:

# sysctl dev.cpu.0.cx_lowest=C3

then you'll see:

dev.cpu.1.cx_lowest: C1 -> C3

Do this for all cores.
Verify by:

[2.0.1-RELEASE][admin@nchannel.nelsewhere.com]/root(12): sysctl dev.cpu | grep cx
dev.cpu.0.cx_supported: C1/1 C2/80 C3/104
dev.cpu.0.cx_lowest: C3
dev.cpu.0.cx_usage: 2.76% 8.00% 89.22% last 306us
dev.cpu.1.cx_supported: C1/1 C2/80 C3/104
dev.cpu.1.cx_lowest: C3
dev.cpu.1.cx_usage: 2.86% 10.01% 87.12% last 223us

Suggested setting (one with C2 state, other as deep as possible) by vermaden
To permanently set the lowest C state, add the lines in /etc/sysctl.conf.
"Update: I can't make it to run with booting even with the above lines added to sysctl.conf nor /etc/rc.conf,
is there any other ways?
Thanks."

While I am not sure if this really helps, desperately need a power meter…...for fun ;D

Thanks the below links:
http://forums.freebsd.org/showthread.php?t=172
http://wiki.freebsd.org/TuningPowerConsumption

stephenw10

powerd is only the daemon that controls the cpu frequency dependent on load. It does so via the appropriate driver, est in my case.
If your cpu doesn't have driver support then powerd can't control it usefully.
acpi throttling provides no real power saving.

Steve

Edit: Although others seem to have had good results. Check the sysctls for est or your boot logs.

nexusN

@stephenw10:

powerd is only the daemon that controls the cpu frequency dependent on load. It does so via the appropriate driver, est in my case.
If your cpu doesn't have driver support then powerd can't control it usefully.
acpi throttling provides no real power saving.

Steve

Edit: Although others seem to have had good results. Check the sysctls for est or your boot logs.

I see, I disabled throttling for it was having a impact on performance.
If PowerD is working on EST, if I have actually locked the CPU freq at 1.6GHz(which is the lowest),
would PowerD be completely useless in the case?

stephenw10

Yes if you have the cpu locked to 1 frequency then powerd can't switch it so it will be doing nothing.
When it's working correctly you should see a number of levels reported by dev.cpu.0.freq_levels. You board has a full set of bios acpi details as it's also reporting the power level (39,000mW).

Steve

nexusN

@stephenw10:

Yes if you have the cpu locked to 1 frequency then powerd can't switch it so it will be doing nothing.
When it's working correctly you should see a number of levels reported by dev.cpu.0.freq_levels. You board has a full set of bios acpi details as it's also reporting the power level (39,000mW).

Steve

Thanks for the detailed explanation, so I now set it with PowerD off, freq locked at 1.6GHz which is the lowest EST state for this CPU(there won't be issues as it is enough and the task running time is decided by time needed for data transfer), and both core set to have lowest C state as C3.
Hope the above will save me a few watts ::)

dreamslacker

@nexusN:

I see, I disabled throttling for it was having a impact on performance.
If PowerD is working on EST, if I have actually locked the CPU freq at 1.6GHz(which is the lowest),
would PowerD be completely useless in the case?

Yes. PowerD working on EST uses speedstep for power savings. Since EIST basically does multiplier reduction, locking the multiplier or having the lowest multiplier basically means that EST cannot do anything.
The main power savings doesn't come from EIST per se, it's actually something called Idle states (sometimes known as idle64) where the CPU is designed to reduce voltage when the processor is clocked down (by EIST).

You should be able to allow the processor to run at normal speeds and simply enable EIST. PowerD with EST will basically downclock the processor when at very low loads. Similarly, when under stress, the processor should be ramped up to max speed.

nexusN

@dreamslacker:

@nexusN:

I see, I disabled throttling for it was having a impact on performance.
If PowerD is working on EST, if I have actually locked the CPU freq at 1.6GHz(which is the lowest),
would PowerD be completely useless in the case?

Yes. PowerD working on EST uses speedstep for power savings. Since EIST basically does multiplier reduction, locking the multiplier or having the lowest multiplier basically means that EST cannot do anything.
The main power savings doesn't come from EIST per se, it's actually something called Idle states (sometimes known as idle64) where the CPU is designed to reduce voltage when the processor is clocked down (by EIST).

You should be able to allow the processor to run at normal speeds and simply enable EIST. PowerD with EST will basically downclock the processor when at very low loads. Similarly, when under stress, the processor should be ramped up to max speed.

Thanks for pointing out and I was thinking for the same,
would PowerD + EIST actually be able to lower the power consumption to a greater extent than if I lock the multiplier to the smallest value I can in BIOS? I do need to verify this.
And for the idle state, would it be something similar with C state? Pfsense has a default setting at C1 only, I tried to set it to C3, not sure if this is actually saving power though.