Improving power consumption on Nuc 11 pro

ph0ton

Re the version, I think I got confused by the info mentioned here:
https://docs.netgate.com/pfsense/en/latest/releases/versions.html

Does this mean that 2.7.0 will be released based on 12.3?

stephenw10

No, that page hasn't been updated since we rebased to main. 2.7 is now built on 14.

[2.7.0-DEVELOPMENT][admin@cedev.stevew.lan]/root: uname -a
FreeBSD cedev.stevew.lan 14.0-CURRENT FreeBSD 14.0-CURRENT #0 devel-main-n255801-ee41bfde702: Tue Oct 18 06:28:31 UTC 2022     root@freebsd:/var/jenkins/workspace/pfSense-CE-snapshots-master-main/obj/amd64/Bwh8z6qs/var/jenkins/workspace/pfSense-CE-snapshots-master-main/sources/FreeBSD-src-devel-main/amd64.amd64/sys/pfSense amd64

Steve

ph0ton

Thanks, Steve.

An update. I was successful in getting the NUC down to about 4.5W, no interfaces plugged in, 6ish with both i225 running - about the same level as the Debian install. I used 2.7.0, loaded the i915kms module, set up C states for the CPU and enabled C states + framebuffer compression for the gpu.

Unfortunately, according to the GPU documentation - at least for FreeBsd 13 - loading the module has to be done in rc.conf.
https://wiki.freebsd.org/Graphics

Any ideas for loading the i915kms module, now that rc.conf isn't parsed? (I did try adding kld_list="i915kms", to rc.conf(.local) to see if it would be loaded, but no dice.

heper

@ph0ton
couldn't you just create a shellscript with all the required "commands" in it?

then use the shellcmds package to execute it after boot ?

stephenw10

That's just a list of kernel modules to load. You should just be able to load it from loader.conf.local as we do for Minnowboard.
What did you do in 2.7 to load it? I assume you just used kldload at the cli after boot? Loading it from loader.conf.local would do the same thing.

Steve

keyser

@ph0ton Its pretty darn impressive you can hit 6w with two interfaces plugged on a NUC 11. Compare that with the SG-4100/SG-6100 which has a lot less “raw” CPU power but use around 14w in the same idle situation. I understand that at load things will look VERY different with your box using up to perhaps double the power of the SG-4100/SG-6100 boxes, but still - if its a home solution with idle at more than 97% of the time, then it’s a great solution.

When considering the amount of CPU power that box have, it is a very nice and flexible solution (for heavy setup with lots of packages). Missing som NIC interfaces though….

ph0ton

Hi folks, thanks for the interest in the post so far.

I have had a few days off the project, which sometimes is a good idea. I think I must have chased some wild geese here and there, because coming back to the system, I get much better power figures than before. Who knows what I did differently this time.

I have landed on these easy-to-use settings:

2.6.0:
In advanced -> system tunables set:

This will allow the CPU cores to go down in powerstate. I am now howevering around 6.8-7.5W when not much activity is happening.
System: NUC 11 pro i5-1145, 16 GB RAM, 128 GB 660p Intel SSD, wifi card disconnected.

2.7.0 snapshot:
Same as above. Gives about the same power consumption. Loading i915kms, using "kldload i915kms" would reduce power consumption about a Watt more or less to 5.5-6.5. 2.7.0 also uses the hwpstate driver for Intel Speed step, which might improve CPU power save further.

Observations on GPU driver:

I think there are 1-2W to be had if a proper video driver can be loaded. For 2.6.0, i915kms_load="yes" would freeze my system on boot. Rebooting with the same config, but unloading the module before the kernel loads would get the system back up and running.
I think the current i915kms driver in 2.6.0. is not supporting the Iris Xe graphics inside 11th NUCs. Two i915kms drivers exists prior to freebsd 13, but the old driver was removed post Freebsd13.

As I understand it, the new driver has to be loaded by rc, post kernel load. However, since pfSense has disabled parts of rc, then there isn't a way to load it this way, other than from the command line. I found a description on a freebsd forum, but alas, I forgot to bookmark it. I will update it if I find it later on.

Having to remember to run a kernel module load after each reboot was not something I was interested in, so I will stick with the power savings as shown above.

@keyser It is indeed. Routing 1 Gbit/s bumps power consumption to about 22-25W. I think this is due to the boost frequencies giving much more performance, than what's acutally needed. I have not tried to disable boost, so target stays at 2.6 GHz max, to see if that changes things.

Thanks for the help.

stephenw10

Just working through this here for the Minnowboard in 2.7. Try this, edit the file /boot/defaults/loader.conf and comment out the line:

module_blacklist="drm drm2 radeonkms i915kms amdgpu"	# Loader module blacklist

That should allow it to load at boot as expected in 2.7.

Steve

ph0ton

@stephenw10
Thanks for the update. I unfortunately had to put the NUC into use yesterday, so I could not check to see if your suggetion worked. But I thank you for the help. I will probably pick it up when 2.7.0 is released, if it still comes with the 915kms driver.

ph0ton

I forgot to add to my findings that powersavings with the i915kms was achieved with:

added to /boot/loader.conf.local:
compat.linuxkpi.i915_enable_dc=2
compat.linuxkpi.i915_enabled_fbc=1
compat.linuxkpi.i915_disable_power_well=1

ph0ton

@keyser After disabling intel boost (can't remember the name in the bios), routing full gigabit requires 12-15W. I have used the old Apple thunderbolt + tb2->tb3 adapter to add more devices. This will give you a pcie NIC, not USB! Will increase idle consumptions by about 2 W. I am guessing that NUC + 2 x TB2 NICs will idle aroung 10W, Load 18-22W.

keyser

@ph0ton That is really impressive. Thanks for detailing your config and numbers. Always good to know what options there are within a given powerbudget.

My most wanted hardware feature request would be an option to have the smaller SG-xxx devices support powerdelivery over PoE+

ph0ton

@keyser I can see that. My biggest wish is a port of Pfsense CE to run on a Raspberry PI Compute Module + df robot router shield. However, I think that would be too disruptive to Netgate's business model.

thebear

@stephenw10 said in Improving power consumption on Nuc 11 pro:

Just working through this here for the Minnowboard in 2.7. Try this, edit the file /boot/defaults/loader.conf and comment out the line:
module_blacklist="drm drm2 radeonkms i915kms amdgpu"	# Loader module blacklist
That should allow it to load at boot as expected in 2.7.

Steve

This one is commented out at my system with 23.01 at default.

https://vermaden.wordpress.com/2018/11/28/the-power-to-serve-freebsd-power-management/ >> scroll to UPDATE 1 – Graphics Card Power Saving

Do you think these new options could help reducing power consumption within 22.05 or 23.01? @ph0ton

Just added your 3 lines to my 23.01 install, lets monitor.

added to /boot/loader.conf.local:
compat.linuxkpi.i915_enable_dc=2
compat.linuxkpi.i915_enabled_fbc=1
compat.linuxkpi.i915_disable_power_well=1

ph0ton

Hi @thebear

Sorry for the late reply... yada yada work yada yada

Yeah, I think you might get some power consumption improvements out of it. As written in an earlier post, I think it gave around 1W improvement.

One hiccup I met on the way is that, unless you reboot after removing the display cable, the GPU seem to be in a high power consumption state with display connected, which invalidated some testing. This confused me and the measurements i did on the CPU in the beginning.

In the end I have concluded that setting the C states is the most effective way of reducing power consumption, rather than trying to initialize the gpu driver. The power savings are not worth it vs the effort.

Now that the power costs are down in Europe as well, I have decided to leave it until 2.7.0 stable :)

stephenw10

@ph0ton said in Improving power consumption on Nuc 11 pro:

Now that the power costs are down in Europe

Tell that to my wallet!

thebear

@stephenw10 down down? My provider still uses the 2022 prices, need to wait to April.

Yes C states is enough, thanks for posting back to me photon.

fireodo

@ph0ton said in Improving power consumption on Nuc 11 pro:

In the end I have concluded that setting the C states is the most effective way of reducing power consumption

Glad that it works for you - I had here unpredictable shutdowns with C States lower than C1 (The CPU - see my signature - allows max C-State C3)

Just my 2 cents

ph0ton

@fireodo poops. I have never come a across a machine in the past 15 years how had problems with C-states, unless it was related to instabilities in the BIOS / UEFI bios.

@thebear @stephenw10 : I feel for your wallets . I pay by the hour, based on "energy zones", which set prices at set intervals during the day. Sometimes I pay nothing (excluding taxes and utitility fees) and sometimes I pay 1.5 eur. If the wind is in the trees, then it's a good day here ;)

fireodo

@ph0ton said in Improving power consumption on Nuc 11 pro:

I have never come a across a machine in the past 15 years how had problems with C-states, unless it was related to instabilities in the BIOS / UEFI bios.

I have now reactivate (because I cannot exclude that maybe something else has caused the shutdowns - at that time I made also some changes regarding the USB-Ports and the update from 2.5.2 -> 2.6.0) the setting: hw.acpi.cpu.cx_lowest=C3 (it sets automatic the dev.cpu.0/1.cx_lowest to C3)
and I keep a eye on it and will give feedback if something happends ...

Result:
dev.cpu.0.cx_usage: 30.82% 69.17% 0.00% last 637us
dev.cpu.1.cx_usage: 59.30% 13.53% 27.15% last 267us

Interesting: dev.cpu.0 never go to C3

Edit 09.05.23: After rebooting the system dev.cpu.0 is going also to C3:
dev.cpu.0.cx_usage: 62.51% 7.06% 30.42% last 1250us

Regards,
fireodo