Improving power consumption on Nuc 11 pro

ph0ton

Hi!

Long time lurker and user of pfsense here! I have been using pfsense in a virtualized environment for a long time. For the past year, I have been using a NUC 11 which has been rock solid.
However, I would like to move to a dedicated routing setup, since running it virtualized along with perhaps less production ready stuff isn't the best recipe for good uptime

Installing pfsense bare metal was easy - of course. However, I notice that the power consumption is higher than what I have been used to - about 13W - WAN and LAN not connected. With debian, running bare metal, I can reach as low as 4.5 W.

I did some tinkering with a freebsd 13 install. First I set up the cpu config, so that the cores could idle in C3. This gave only about a watt.
Installing drm-kmod and enabling i915kms gave a big improvment, almost halving the power consumption.

I can see that i915kms is not easily available on pfsense, without enabling freebsd repos. While I could download the drm-kmod, I struggle getting it to run on pfsense, since a lot of configuration seem different from freebsd 13 - and I am a newbie in the bsd eco system.

does anyone know:

how I can download the specific i915kms module, without forcing the Freebsd repos on (it also triggers and update of pkg, which I am a little sceptical about)
how I enable said module with in Freebsd?
if I will run into problems down the line with new freebsd version reverting my changes?

Thanks!

stephenw10

It's included in 2.6, the Minnowboard devices use it:
https://docs.netgate.com/platforms/minnowboard/faq.html#my-hdmi-doesnt-always-work-is-there-a-fix-for-this
You should only need to load it. I have no idea if it will work with your hardware of course.

I assume you have enabled powerd to take advantage of EIST?
https://docs.netgate.com/pfsense/en/latest/config/advanced-misc.html#power-savings

I would also look to see what can be disabled in the BIOS.

Steve

ph0ton

Thanks for your help, Steve. Enabling powerd only returns miniscule improvements - I also don't know if another power driver takes over on tiger lake processors.

I did the modifications, however, the machine crashes when it comes to the efifb dimensioning. I can see that the i915kms.ko module is loaded.

Hmmm ... I not sure what is the next step is. Any ideas?

stephenw10

I would try loading the module after boot without any other setting and see what happens.

If it needs to be loaded with the kernel try only adding the module load line.

What does it show when it panics?

It may simply not support that hardware yet in FreeBSD 12. You might try a 2.7 snapshot, however I know a lot has changed. The i915kms driver no longer exists in the same way there.
It's not something we ever look at directly in pfSense. If you can get it working in FreeBSD first, and it proves worthwhile, I'm sure it can be made to work in pfSense.

Steve

ph0ton

thanks again -

I've tried to load the module using kldload, and I got some notifications of some I2C bus devices being available, but not much more than that. Target did not crash, though. Could not see any improvement on power consumption, but I think that was expected, since I think the module is loaded, but not in use.

When it crashes, nothing happens. I just see some EFI frame buffer information. I can hear the fan speed up, and target stops responding.

Can i915kms coexist with drm2?

Using Freebsd 13, using the i915kms.ko module from drm-kms package yielded the desired power saving. I think it was around 6.5W. Each interface eats about a watt of power.

I'll try 2.7.0, it can't hurt. If it works, then I know the improvement waits down the line.

stephenw10

Did you load the module at boot in 13?

Might just not be supported in 12 then. It would normally show a bunch of new devices when the driver attaches.

ph0ton

Hmm, did I now? I think the drm-kmod package asks to add it to rc.conf. I might have confused myself with the fact that rc.conf does nothing in pfsense (?) and that the minnowboard guide asked to load the module in the boot loader.

Running a 2.7.0 snapshot (from the 19th), I could see that it still includes the i915kms.ko module. So I used kldload - and low and behold, the console flashed, and I could see that my monitor resolution was changed. I could also see that the power consumption went from ~13 to about 10 watts. This was very different from kldload i915kms on 2.6.0, where nothing really happened.

Now, I am wondering if I can extract the i915kms.ko and place it in a 2.6.0. install. Both 2.7.0 and 2.6.0. uses freebsd 12.3, but I don't know how much kernel interfaces can change.

Question: If freebsd asks for the module to be loaded in rc.conf, how is this done for pfsense? I might have overlooked that rc.conf is broken up into many rc scripts.

stephenw10

2.7 is built on FreeBSD main (14) so you can't use a kernel module from there in 2.6. Many kernel modules will only work at all against the specific kernel they were built against anyway.

pfSense doesn't use the FreeBSD RC system as you said but that would not normally be used to load kernel modules like that.

Steve

ph0ton

Re the version, I think I got confused by the info mentioned here:
https://docs.netgate.com/pfsense/en/latest/releases/versions.html

Does this mean that 2.7.0 will be released based on 12.3?

stephenw10

No, that page hasn't been updated since we rebased to main. 2.7 is now built on 14.

[2.7.0-DEVELOPMENT][admin@cedev.stevew.lan]/root: uname -a
FreeBSD cedev.stevew.lan 14.0-CURRENT FreeBSD 14.0-CURRENT #0 devel-main-n255801-ee41bfde702: Tue Oct 18 06:28:31 UTC 2022     root@freebsd:/var/jenkins/workspace/pfSense-CE-snapshots-master-main/obj/amd64/Bwh8z6qs/var/jenkins/workspace/pfSense-CE-snapshots-master-main/sources/FreeBSD-src-devel-main/amd64.amd64/sys/pfSense amd64

Steve

ph0ton

Thanks, Steve.

An update. I was successful in getting the NUC down to about 4.5W, no interfaces plugged in, 6ish with both i225 running - about the same level as the Debian install. I used 2.7.0, loaded the i915kms module, set up C states for the CPU and enabled C states + framebuffer compression for the gpu.

Unfortunately, according to the GPU documentation - at least for FreeBsd 13 - loading the module has to be done in rc.conf.
https://wiki.freebsd.org/Graphics

Any ideas for loading the i915kms module, now that rc.conf isn't parsed? (I did try adding kld_list="i915kms", to rc.conf(.local) to see if it would be loaded, but no dice.

heper

@ph0ton
couldn't you just create a shellscript with all the required "commands" in it?

then use the shellcmds package to execute it after boot ?

stephenw10

That's just a list of kernel modules to load. You should just be able to load it from loader.conf.local as we do for Minnowboard.
What did you do in 2.7 to load it? I assume you just used kldload at the cli after boot? Loading it from loader.conf.local would do the same thing.

Steve

keyser

@ph0ton Its pretty darn impressive you can hit 6w with two interfaces plugged on a NUC 11. Compare that with the SG-4100/SG-6100 which has a lot less “raw” CPU power but use around 14w in the same idle situation. I understand that at load things will look VERY different with your box using up to perhaps double the power of the SG-4100/SG-6100 boxes, but still - if its a home solution with idle at more than 97% of the time, then it’s a great solution.

When considering the amount of CPU power that box have, it is a very nice and flexible solution (for heavy setup with lots of packages). Missing som NIC interfaces though….

ph0ton

Hi folks, thanks for the interest in the post so far.

I have had a few days off the project, which sometimes is a good idea. I think I must have chased some wild geese here and there, because coming back to the system, I get much better power figures than before. Who knows what I did differently this time.

I have landed on these easy-to-use settings:

2.6.0:
In advanced -> system tunables set:

This will allow the CPU cores to go down in powerstate. I am now howevering around 6.8-7.5W when not much activity is happening.
System: NUC 11 pro i5-1145, 16 GB RAM, 128 GB 660p Intel SSD, wifi card disconnected.

2.7.0 snapshot:
Same as above. Gives about the same power consumption. Loading i915kms, using "kldload i915kms" would reduce power consumption about a Watt more or less to 5.5-6.5. 2.7.0 also uses the hwpstate driver for Intel Speed step, which might improve CPU power save further.

Observations on GPU driver:

I think there are 1-2W to be had if a proper video driver can be loaded. For 2.6.0, i915kms_load="yes" would freeze my system on boot. Rebooting with the same config, but unloading the module before the kernel loads would get the system back up and running.
I think the current i915kms driver in 2.6.0. is not supporting the Iris Xe graphics inside 11th NUCs. Two i915kms drivers exists prior to freebsd 13, but the old driver was removed post Freebsd13.

As I understand it, the new driver has to be loaded by rc, post kernel load. However, since pfSense has disabled parts of rc, then there isn't a way to load it this way, other than from the command line. I found a description on a freebsd forum, but alas, I forgot to bookmark it. I will update it if I find it later on.

Having to remember to run a kernel module load after each reboot was not something I was interested in, so I will stick with the power savings as shown above.

@keyser It is indeed. Routing 1 Gbit/s bumps power consumption to about 22-25W. I think this is due to the boost frequencies giving much more performance, than what's acutally needed. I have not tried to disable boost, so target stays at 2.6 GHz max, to see if that changes things.

Thanks for the help.

stephenw10

Just working through this here for the Minnowboard in 2.7. Try this, edit the file /boot/defaults/loader.conf and comment out the line:

module_blacklist="drm drm2 radeonkms i915kms amdgpu"	# Loader module blacklist

That should allow it to load at boot as expected in 2.7.

Steve

ph0ton

@stephenw10
Thanks for the update. I unfortunately had to put the NUC into use yesterday, so I could not check to see if your suggetion worked. But I thank you for the help. I will probably pick it up when 2.7.0 is released, if it still comes with the 915kms driver.

ph0ton

I forgot to add to my findings that powersavings with the i915kms was achieved with:

added to /boot/loader.conf.local:
compat.linuxkpi.i915_enable_dc=2
compat.linuxkpi.i915_enabled_fbc=1
compat.linuxkpi.i915_disable_power_well=1

ph0ton

@keyser After disabling intel boost (can't remember the name in the bios), routing full gigabit requires 12-15W. I have used the old Apple thunderbolt + tb2->tb3 adapter to add more devices. This will give you a pcie NIC, not USB! Will increase idle consumptions by about 2 W. I am guessing that NUC + 2 x TB2 NICs will idle aroung 10W, Load 18-22W.

keyser

@ph0ton That is really impressive. Thanks for detailing your config and numbers. Always good to know what options there are within a given powerbudget.

My most wanted hardware feature request would be an option to have the smaller SG-xxx devices support powerdelivery over PoE+