Sharing on a 1Gbps Router built(+Tests on clocked at 1.6GHz and Default)

nexusN

Hi,

I started a thread some weeks ago consulting the building of a 1Gbps router, http://forum.pfsense.org/index.php/topic,43536.0.html.
Now it is what I have finally got, just a simple sharing of the assembly and a test to check if the performance is desired, also the respective CPU utilization for your reference.

Wanted to build one for long, but not until the installation of BB1000 I find it critical.
Here is the config:

CPU : Intel Celeron G530
MB : Intel DH61WW
RAM : Kingston DDR3 1333 2GB x2
Storage : WD 2.5" 320GB
PSU : Antec ECO 400C
Chassis : SilverStone SG-02F
1st NIC : Intel 82579V
2nd NIC : Intel 82574L
Switch : PCi FXG-08EP

OK, kick off the assembly:

CPU is on its pad:

Then MB is on its land:

Install the CPU cooler, a stock one from E3-1235, it is better that G530's given. Sure it will suffice.

Throw the PSU inside:

Take a look at those cords, you will know why a modular PSU costs:

Almost there:

Some pics of the inside:

It's ready then.

Finally the OS is pfsense, ClearOS has problem installing due to no support on 82579V.

With CPU clock restricted at 1.6GHz, Turbo Boost disabled,
A test is conducted using Iperf provided by HKBN, the following are the CPU utilization(almost constant) during the test(assume single direction):

The final speed test resulted:

Thanks for watching.

bmironb

Intersting project !
Will not have any wifi cards ?

nexusN

@bmironb:

Intersting project !
Will not have any wifi cards ?

This is the next step,
originally I thought it is hard to integrate, but later I found in the OS and community there are compatible hardware.
Maybe I will get one when I can confirm a good adapter.

bmironb

Yes there are some compatible wifi hardware, i've made some mistake and buy for example an wifi usb from d-link and it was not compatible, it's more easy to have an wifi ap on lan interface :D this my next step :Dalso on pc based pfsense 2.0.1 , intersting that is showing your cpu utilization also had celeron cpu but it's always updated in 10 seconds :))

stephenw10

So approximately 55% cpu usage at 860Mbps?
Is that in one direction or bidirectional?
Any idea what the power consumption is in the end?
Great looking box you have built anyway and it seems to be easily up to the job. :)

Steve

nexusN

@stephenw10:

So approximately 55% cpu usage at 860Mbps?
Is that in one direction or bidirectional?
Any idea what the power consumption is in the end?
Great looking box you have built anyway and it seems to be easily up to the job. :)

Steve

I believe it is a single direction usage, as the utilization is then consistent with that 20% at 320Mbps downloading.
Maybe in an extreme case, 1Gbps Up and Down will max out the CPU finally.
For power consumption, sadly I don't have the tools, but you may take reference to the following:

System consumption at Idle:

System consumption at Full Load:

Finally, yes the built is simple enough, thank Intel for such a competent Celeron with good offer and pfsense's sophisticated router OS ;D

nexusN

Previously the speed test had CPU clocked at 1.6GHz with Turboboost off, let's see if there is any change if clock is left at default:
Download:

Upload:

Result:

I have run a few test and the results are consistent, which is around 30% at the speed.
Download and Upload shouldn't show much difference, but the refresh rate of CPU utilization is too slow to be accurate to confirm if there exist a few % difference.

jwelter99

@nexusN:

Previously the speed test had CPU clocked at 1.6GHz with Turboboost off, let's see if there is any change if clock is left at default:
Download:

Upload:

Result:

I have run a few test and the results are consistent, which is around 30% at the speed.
Download and Upload shouldn't show much difference, but the refresh rate of CPU utilization is too slow to be accurate to confirm if there exist a few % difference.

Turn on PowerNow or whatever FreeBSD call's it as it has a pretty large impact on our E3 PF boxes.

nexusN

Sorry but I am not familiar with pf, what actually is PowerNow?
I heard something called PowerD, but not sure how can I turn that on nor config it.
Please kindly advise, thanks.

stephenw10

Yes powerd is what you want.
It's in System: Advanced: Miscellaneous: in the webgui.

Steve

nexusN

@stephenw10:

Yes powerd is what you want.
It's in System: Advanced: Miscellaneous: in the webgui.

Steve

Thanks a lot, I have found and checked the box.
It would be nice to have such an option to save some energy ;D

nexusN

Sadly :(, PowerD seems to be hurting performance to quite a significant degree, the below is a comparison of throughput with PowerD ON and OFF:

stephenw10

Check your logs, make sure that powerd is working correctly.
You may have to change the kernel timecounter.
Powerd has a number of tuning options the govern how quickly it ramps up the cpu speed. These can be tuned to give you better performance/power ratio. See the man page.

Steve

nexusN

@stephenw10:

Check your logs, make sure that powerd is working correctly.
You may have to change the kernel timecounter.
Powerd has a number of tuning options the govern how quickly it ramps up the cpu speed. These can be tuned to give you better performance/power ratio. See the man page.

Steve

Thanks for the idea,
I will try to see if I can tune to strike a balance.

stephenw10

You can do some testing manually. See my experiences with it here:
http://forum.pfsense.org/index.php/topic,20095.msg161139.html#msg161139

Even at the most conservative setting it should ramp up in a few seconds. How long does that test take? It looks more like it's not running correctly.

Steve

nexusN

@stephenw10:

You can do some testing manually. See my experiences with it here:
http://forum.pfsense.org/index.php/topic,20095.msg161139.html#msg161139

Even at the most conservative setting it should ramp up in a few seconds. How long does that test take? It looks more like it's not running correctly.

Steve

Maybe I can try to "warm it up" before carrying out the test.
The test takes a very short period, like 5 s for downloading and another 5s for uploading, the time maybe too short for the router to response.

nexusN

More on the PowerD tests,
as mentioned, the PowerD should be able to speed up upon the need of performance, I have tried to run a few tests continuously and found the below:

Each pair of column in the network utilization graph refers to a single test, the bench result on the left is referring to the last test(the pair at the right most).

I was wrong on my previous statement, in a pair, the first column should be for uploading and the second column is referring to downloading,
as all the tests are showing a result for a faster downloading and slower uploading.
This is important that, it means the Uploading test triggered the software to speed up the CPU thus a better result is obtained on downloading which follows closely while the uploading itself has been stuck with a lower performance.

Also, from the test it seems the degree the CPU speeds up is quite uncertain, the flat top of each columns states that the "speed up" actually stopped at a certain level during a transfer, and this level may not be at the best performance.

nexusN

Just now I tried to add the two command in loader.conf:

hint.p4tcc.0.disabled=1
hint.acpi_throttle.0.disabled=1

Then it seems with the throttling disabled the clock of CPU is fixed as I have set in BIOS:
1.6GHz

[2.0.1-RELEASE][admin@nchannel.nelsewhere.com]/root(2): sysctl dev.cpu.0.freq_levels
dev.cpu.0.freq_levels: 1600/39000

And upon a test, there is no more performance drop with the same tests with PowerD on.
Here comes another questions, if the CPU clock has been fixed in BIOS already, is there any other pros to leave PowerD on?
I have only one 2.5" harddisk in the box, it can't be turned off for idle even PowerD is on right?
Sad that I don't have a power meter to get the ans.

Thanks for your kind attention.

SlowGrind

That is a great looking rig… And the cost is about the same as an Atom board. I might just build this same setup.

nexusN

@nexusN:

Just now I tried to add the two command in loader.conf:
hint.p4tcc.0.disabled=1
hint.acpi_throttle.0.disabled=1
Then it seems with the throttling disabled the clock of CPU is fixed as I have set in BIOS:
1.6GHz
[2.0.1-RELEASE][admin@nchannel.nelsewhere.com]/root(2): sysctl dev.cpu.0.freq_levels
dev.cpu.0.freq_levels: 1600/39000
And upon a test, there is no more performance drop with the same tests with PowerD on.
Here comes another questions, if the CPU clock has been fixed in BIOS already, is there any other pros to leave PowerD on?
I have only one 2.5" harddisk in the box, it can't be turned off for idle even PowerD is on right?
Sad that I don't have a power meter to get the ans.

Thanks for your kind attention.

Apart from PowerD,
More on PowerSaving(Hopefully):
On some Googles, I have found that pfsense 2.0.1 on default doesn't enable C3 state for CPUs,
and I checked it to be true by:

[2.0.1-RELEASE][admin@nchannel.nelsewhere.com]/root(5): sysctl dev.cpu |grep cx
dev.cpu.0.cx_supported: C1/1 C2/80 C3/104
dev.cpu.0.cx_lowest: C1
dev.cpu.0.cx_usage: 100.00% 0.00% 0.00% last 500us
dev.cpu.1.cx_supported: C1/1 C2/80 C3/104
dev.cpu.1.cx_lowest: C1
dev.cpu.1.cx_usage: 100.00% 0.00% 0.00% last 500us

To get C3 state works, the little tweak is:

# sysctl dev.cpu.0.cx_lowest=C3

then you'll see:

dev.cpu.1.cx_lowest: C1 -> C3

Do this for all cores.
Verify by:

[2.0.1-RELEASE][admin@nchannel.nelsewhere.com]/root(12): sysctl dev.cpu | grep cx
dev.cpu.0.cx_supported: C1/1 C2/80 C3/104
dev.cpu.0.cx_lowest: C3
dev.cpu.0.cx_usage: 2.76% 8.00% 89.22% last 306us
dev.cpu.1.cx_supported: C1/1 C2/80 C3/104
dev.cpu.1.cx_lowest: C3
dev.cpu.1.cx_usage: 2.86% 10.01% 87.12% last 223us

Suggested setting (one with C2 state, other as deep as possible) by vermaden
To permanently set the lowest C state, add the lines in /etc/sysctl.conf.
"Update: I can't make it to run with booting even with the above lines added to sysctl.conf nor /etc/rc.conf,
is there any other ways?
Thanks."

While I am not sure if this really helps, desperately need a power meter…...for fun ;D

Thanks the below links:
http://forums.freebsd.org/showthread.php?t=172
http://wiki.freebsd.org/TuningPowerConsumption