NUT notifications not working on pfsense+

[dennypage]

@austin-0 said in NUT notifications not working on pfsense+:

We are currently live streaming. So it's not a good time to be installing patches, but once that is done I will try out the patch.

After installing the patch, you may also need to remove the existing notification files. And if a process is hung processing notifications, either kill it or reboot pfSense.

[Austin 0]

@dennypage Okay so I have never done this before, but from what I can tell I correctly entered th commit ID, but it is telling me it is invalid. What am I doing wrong here?

[dennypage]

@austin-0 Not sure, but you can try it in URL format:

https://github.com/pfsense/pfsense/commit/c5faa351c1ef6d4555478a7f50b3a16ece7e0b2a.patch

[Austin 0]

@dennypage Thank you. I was able to install the package, and then rebooted. Notifications are still not being emailed to me when I unplugged the USB cable to the UPS. I did get the notifications for the actual reboot.

[dennypage]

@austin-0 Can you post the grep output please?

grep notify_monitor /var/log/system.log

[Austin 0]

@dennypage Well I am no longer there and someone turned off the computer I usually use to remote in. So I won't be able to do this until Friday or Saturday.

[dennypage]

@austin-0 Okay, I just tested that patch. The effect is that all non root notifications silently fail. I've reached out to @jimp in the Redmine discussion.

In the interim, it appears that the only way you will be able to make it work is to have NUT run as root. You can do this by adding the line

RUN_AS_USER root

to the "Additional configuration lines for upsmon.conf" section in Services / UPS / Settings / Advanced settings.

[pdavis]

@dennypage I am not getting email notifications from NUT either, and hope you can point me in the right direction.

I have a Cyberpower UPS plugged into my pfSense box with a UPS, and the UPS is recognized fine within the GUI. If it is on battery, I get an alert in the GUI, but do not get an email through the normal pfSense process.

I added the additional configuration to run NUT as root, and rebooted, but that did not work. See below for my upsmon.conf and nut_email.php files, and please let me know what I am missing - thank you!

[pdavis]

@pdavis @dennypage Sorry, Denny, scratch that last request - I just got the emails following the change to have NUT send as root, so all set!

Thank you

[dennypage]

@pdavis Glad you have it working. I'm hoping that the requirement to run as root is temporary.

[Austin 0]

@dennypage Well hopefully this works for me as soon as I get a chance to test it. I am curious as to what the security impacts of running the process as root would be. Would it not be better to change the access to whatever file nut needs access to?

[dennypage]

@austin-0, @pdavis I have a PR in which will restore notification functionality. You can find it here if you want to follow it, or if you just want to grab the revised version of nut_email.php.

[Gertjan]

@dennypage said in NUT notifications not working on pfsense+:

You can find it here if you ...

The GUI says :

So, the user would except that email and/or Telegram and or Pushover works.

The file is called :

/usr/local/pkg/nut/nut_email.php

Your edit makes notifications 'email' only (no more Telegram/Pushover).
Are you sure ?

@pdavis
I propose a plan B : take control of things yourself everything :

Enable notifications".

Place this in the "Additional configuration lines for upsmon.conf" :

NOTIFYFLAG ONLINE   SYSLOG+WALL+EXEC
NOTIFYFLAG ONBATT   SYSLOG+WALL+EXEC
NOTIFYFLAG LOWBATT  SYSLOG+WALL+EXEC
NOTIFYFLAG FSD      SYSLOG+WALL+EXEC
NOTIFYFLAG COMMOK   SYSLOG+WALL+EXEC
NOTIFYFLAG COMMBAD  SYSLOG+WALL+EXEC
NOTIFYFLAG SHUTDOWN SYSLOG+WALL+EXEC
NOTIFYFLAG REPLBATT SYSLOG+WALL+EXEC
NOTIFYFLAG NOCOMM   SYSLOG+WALL+EXEC
NOTIFYFLAG NOPARENT SYSLOG+WALL+EXEC
NOTIFYCMD /usr/local/sbin/upssched
RUN_AS_USER root

Note the line "NOTIFYCMD /usr/local/sbin/upssched"

Now, continue reading here NUT upssched - read the entire thread for more info.

In short : "upssched" ( /usr/local/sbin/upssched) is the NUT event scheduler.
It has a config file here /usr/local/etc/nut ( not surfaced in the GUI !) called upssched.conf.

[23.01-RELEASE][admin@pfSense.near.by]/root: grep -v '^#' /usr/local/etc/nut/upssched.conf

CMDSCRIPT /usr/local/bin/upssched-cmd

The file has to be edited, as shown in the thread I've shown above.
With this file you can handle all possible event, and have pfSense do its thing.

Like calling :

send_smtp_message($MSG, $subject);

and/or

@notify_all_remote($subject . " - " . $MSG

Just a reminder : when updating NUT, it is very possible that the file "/usr/local/bin/upssched-cmd" gets over written by a default 'do nothing" file.
You'll have to copy in place your own version again.
This can be done using a mode made "patch" or a mere straightforward 'cp' upon boot (use the Shellcmd package, and write up your own copy command, from /root/upssched-cmd to /usr/local/bin/upssched-cmd

The pro's : using NUT as it was meant to be used.
Con's : more maintenance.

[dennypage]

@gertjan Thank you for pointing that out. I have clarified the text.

I stopped using or recommending upssched years ago. The NUT documentation does not particularly recommend its use anymore either. While upssched is useful for supporting very complex situations, the level of complexity generally just gets the average user in trouble. It's an advanced / experts only tool, and not a good choice for a firewall package where the goal is simplicity and reliability.

FWIW, using upssched would not actually address the problem. The problem is that notify_all_remote() now requires root privileges to function. Yes, I could have simply addressed the issue by requiring upsmon be run as root (no need for upssched), but I am not comfortable with making that a requirement.

As it sits right now, other than smtp, there is no standard notification system available in pfSense 23.01+ that does not require root privilege to initiate. If one is added, I will move to it when it becomes available. But in the interim, I am going with the same approach that the other non root packages have gone with, which is send_smtp_message().

[pdavis]

@dennypage Thank you, Denny! I will follow the PR

[dennypage]

@pdavis You are most welcome

[Gertjan]

@dennypage

If I understand correctly, you have to use "send_smtp_message()" because :

@notify_all_remote($subject . " - " . $message);

which is nothing more then :

function notify_all_remote($msg) {
        notify_via_smtp($msg);
        notify_via_telegram($msg);
        notify_via_pushover($msg);
        notify_via_slack($msg);
}

needs root facilities for telegram, pushover and slack but not smtp ?
Humm, that's more a s pfSense as a NUT issue for me then IMHO.

But I do get this :

My 'signal system' runs as root, so I never had any issues when sending mail - as the sched script handles all this.

Btw : I don't mind your package runs as root

[dennypage]

@gertjan said in NUT notifications not working on pfsense+:

f I understand correctly, you have to use "send_smtp_message()" because @notify_all_remote($subject . " - " . $message);
needs root facilities for telegram, pushover and slack but not smtp ?
Humm, that's more a s pfSense as a NUT issue for me then IMHO.

Yes, it is a general pfSense issue. notify_all_remote() manages smtp notifications via a queue, and that queue management now requires root privilege. Detailed discussion in the Redmine issue. Unfortunately, with the change (update #15) any use of notify_all_remote to send smtp by a non root will silently fail.

In my reading of the code, the queue only exists to suppress duplicate sends for smtp messages. It suppresses notices if the new notice is exactly the same as the prior notice sent (not the prior notice queued). I don't really see a lot of value in that, so I would have probably solved the spamming problem by simply deleting the whole duplicate suppression system. But it wasn't my code to fix, and there doesn't seem to be much interest in discussing alternative solutions, so I took the approach of bypassing the queue code and sending smtp directly as others have done.

[dennypage]

@gertjan, you motivated me to make a proposal.

[Gertjan]

@dennypage

I'm aware of issue #14032 and didn't understand what was wrong / I couldn't reproduce it / was using the NUT package for years.
The why part is now clear to me : I'm handled the 'signalling part' myself by using the complicated upsched. And did the 'bad-behavior' thing that masks the issue : I run the CMD script as root.
Now, I get it.

About your note #25 :
I agree with "Processes and tasks that do not need root, should not run as root",
but I would silently add a

RUN_AS_USER root

at the top of this part, where upsmon.conf is created.

Add a

# To be removed when "/usr/local/pkg/nut/nut_email.php" doesn't need root anymore