Prometheus Node Exporter gives log errors - fix or suppress in log

ameinild

I'm running pfSense 2.7.2 on a Protectli Vault VP2420.

I've enabled Prometheus Node Exporter, but it gives the following log error each 15 seconds:

Feb 15 09:53:57 vault node_exporter[25559]: ts=2024-02-15T08:53:57.164Z caller=collector.go:169 level=error msg="collector failed" name=uname duration_seconds=1.9687e-05 err="cannot allocate memory"
Feb 15 09:53:57 vault node_exporter[25559]: ts=2024-02-15T08:53:57.164Z caller=collector.go:169 level=error msg="collector failed" name=zfs duration_seconds=1.6108e-05 err="couldn't get sysctl: no such file or directory"

This is my Node Exporter settings:

alt text

Does anyone know why this error appears, and maybe which collector generates it?

If it's not easy to fix, I would like to at least suppress the log messages, but I can't seem to get this to work either.

I've tried:

Setting --log.level=none, but this is apparently not a valid log level
Creating a 01-node-exporter.conf file to suppress the messages and put it in /var/etc/syslog.d/, but this doesn't work either. I've tried with contents as both:

if $msg contains "collector failed" then {
    stop
}

And

:msg, contains, "collector failed" ~

Can anyone help here? Thanks.

ameinild

In addition, I've found the node_exporter config file, which is located at /usr/local/etc/rc.conf.d. This file contains:

# This file is generated by the pfSense node_exporter package.
# Do not edit this file, it will be overwritten automatically.

node_exporter_enable="YES"
node_exporter_listen_address="10.10.1.1:9100"
node_exporter_args=" --collector.boottime --collector.cpu --collector.exec --collector.filesystem --collector.loadavg --collector.meminfo --collector.netdev --collector.textfile --collector.time --log.level=error"

I'm inclined to think the erroneous log messages is a bug in Node Exporter for FreeBSD (because the collectors uname and zfs doesn't even seem to be enabled), so the short term solution would be to suppress the log messages, which I still can't get to work.

ameinild

I finally found a solution. I couldn't get any syslog configuration to work, but I managed to edit the node_exporter config file like this:

# This file is generated by the pfSense node_exporter package.
# Do not edit this file, it will be overwritten automatically.

node_exporter_enable="YES"
node_exporter_listen_address="10.10.1.1:9100"
node_exporter_args=" --collector.boottime --collector.cpu --collector.exec --collector.filesystem --collector.loadavg --collector.meminfo --collector.netdev --collector.textfile --collector.time --log.level=error --no-collector.zfs --no-collector.uname"

Note that I added --no-collector.zfs and --no-collector.uname to the node_exporter_args. This prevents these collectors from running, and thus the error messages disappear.

I would still like to know how to directly suppress log messages in FreeBSD through the syslog configuration, but that'll be another time.

ameinild

Can anyone tell me if the Node Exporter package is specific to pfSense or is it the default FreeBSD package?

I wonder if one were to file a bug report for the package configuration issue, would the bug have to be reported here, or at the Node Exporter Github page?

Thanks.

nws

Adding this here rather than a new topic since this is what comes up vv google when looking for why uname and zfs are throwing errors in the pfSense system log

Following the steps above from @ameinild does stop the messages from appearing, but only until you restart or touch the config in the ui.

After digging around here github and looking for some overly complicated way to override the default conf, a definite was in order though... its so obvious once you see it, i.e. all of the selected options are all just flags in the config, and adding --no-collector.XX is just another flag, but from what I could find this wasnt called out anywhere, despite this issue being discussed in a number of forums.

The solution is simply to add --no-collector.zfs --no-collector.uname to Extra flags and save.

Which results in /usr/local/etc/rc.conf.d/node_exporter updated as follows, and the config persisting.