NUT notifications not working on pfsense+

[Austin 0]

I currently am doing some IT support for the church I attend, and I have configured their SG-1100 running pfsense+ 23.01 with the nut package in order to monitor the UPS. Friday there was a power outage and I never received the notification email. I have checked the box for the notifications. I have done some additional testing, and I don't appear to be getting any notifications from NUT. The UPS battery was completely drained before power was restored. When power was restored I did get the notification that system powered back on. So it appears that NUT is the only thing not sending notifications. I checked the documentation and I don't see any mention of the notification system for NUT at all, so that was not any help. I am not sure what logs would be helpful, but I can provide any additional information requested.

[dennypage]

Have you successfully tested SMTP notifications in System / Advanced / Notifications?

[Austin 0]

@dennypage Yes the test emails went through fine. It also sent me the notification when it powered back on after the power outage.

[dennypage]

@austin-0 Please post the contents of /usr/local/etc/nut/upsmon.conf.

And the sha sum of /usr/local/pkg/nut/nut_email.php:

[23.01-RELEASE][root@fw]/: sha1sum /usr/local/pkg/nut/nut_email.php
c389b8fe838a23c4df8b078747c259f71609da72  /usr/local/pkg/nut/nut_email.php
[23.01-RELEASE][root@fw]/:

[Austin 0]

@dennypage Thank you for the assistance. Here is the contents of the config file.

#!/usr/local/bin/php-cgi -q
<?php
/*
 * nut_email.php
 *
 * part of pfSense (https://www.pfsense.org)
 * Copyright (c) 2004-2023 Rubicon Communications, LLC (Netgate)
 * Copyright (c) 2016 Denny Page
 * All rights reserved.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

require_once("notices.inc");

$subject = "UPS Notification from " . gethostname();
$message = date('r');
$message .= "\n\n";
$message .= implode(' ', array_slice($argv, 1));

@notify_all_remote($subject . " - " . $message);

Here is the checksum

7fa23e54c2f5f81125acf6ffd2f3d3df55db208a  /usr/local/pkg/nut/nut_email.php

[dennypage]

@austin-0 It was the contents of /usr/local/etc/nut/upsmon.conf that I was looking for.

Your /usr/local/pkg/nut/nut_email.php is fine.

[Austin 0]

@dennypage Looks like I misread your request. Here is the config file.

MONITOR OfficeUPS 1 local-monitor f31728551378cda34c25 master

SHUTDOWNCMD "/sbin/shutdown -p +0"

POWERDOWNFLAG /etc/killpower

NOTIFYCMD /usr/local/pkg/nut/nut_email.php

NOTIFYFLAG ONLINE   SYSLOG+WALL+EXEC

NOTIFYFLAG ONBATT   SYSLOG+WALL+EXEC

NOTIFYFLAG LOWBATT  SYSLOG+WALL+EXEC

NOTIFYFLAG FSD	  SYSLOG+WALL+EXEC

NOTIFYFLAG COMMOK   SYSLOG+WALL+EXEC

NOTIFYFLAG COMMBAD  SYSLOG+WALL+EXEC

NOTIFYFLAG SHUTDOWN SYSLOG+WALL+EXEC

NOTIFYFLAG REPLBATT SYSLOG+WALL+EXEC

NOTIFYFLAG NOCOMM   SYSLOG+WALL+EXEC

NOTIFYFLAG NOPARENT SYSLOG+WALL+EXEC

[dennypage]

@austin-0 Please try the following:

/usr/local/pkg/nut/nut_email.php test

Does this produce an email? Or an error?

[Austin 0]

@dennypage It sent me an email.

[dennypage]

@austin-0 Hmm... I see nothing currently wrong.

Is it possible that NUT wasn't actually running at the time of your event? Or that the email messages were suppressed by your mail handler? Or put into junk? Or some other form of automated disposal?

You said that you had done additional testing. What did you test?

About the only thing left to do is to test by disconnecting the UPS, or disconnecting mains. Be sure you are logged in (ssh) when you do it. When you disconnect, you should receive both a message in your terminal window and an email message. If you don't, check both the UPS status and the system log file for indications of error.

[Austin 0]

@dennypage After first installing NUT I did make sure it actually recognized the UPS. I tried testing the email notifications at that time by simply unplugging the UPS for a few seconds, and did note that they weren't working. It just kind of got put on the back burner though to be honest. There is a small chance that the nut service was running at the time, but I was accessing the firewall the night before, and the service was running then. I am running the mail through gmail, and it emails both to and from my personal email. I have checked through the "All Mail" folder to make sure it wasn't just in junk. However I do think that this is the first time that firewall has been restarted since I turned the notifications on for the NUT package. I suppose it is possible that rebooting it fixed the issue. I won't be test unplugging it again until Friday, as that is the next time I will have physical access. Does it need to hit a certain battery percentage to send the notification?

[dennypage]

@austin-0 said in NUT notifications not working on pfsense+:

Does it need to hit a certain battery percentage to send the notification?

Initial notification that the UPS is on battery should be immediate.

FWIW, you can just disconnect the USB connection (assuming that is what you are using), and you should receive an immediate notification that the UPS is unavailable. On most UPSs you can also run a self test, which should also produce an "on battery" notification.

Call me old school, but I generally prefer pulling the USB connection, doing a self test, or even popping the breaker instead of pulling the mains cord. The reason for this is that pulling the mains cord also disconnects the ground.

[FSC830]

May be a dump question: was it possible, that the outage disconnects the line to the mail server?
I am using NUT at my SG-3100 and as far as I remember I did not configure anything else than the SMTP settings.

The UPS is scheduled for a self-test every two weeks an I get an e-mail immediately when UPS is on battery.

Regards

[Austin 0]

@fsc830 Thank you. Yes I was talking to my co-worker, and I realized that I don't currently know what the ONT is plugged into. So it 100% possible that the ONT went offline, which would explain the lack of emails. Something else I will have to look into Friday when I go up there.

[Austin 0]

I have double checked and the ont was on a battery backup. I unplugged the USB cable and still there was no notification sent. I did verify that the UPS status in nut changed. It just did not send a notification. I found a interesting message in the log though. ups_status_set: seems that UPS [OfficeUPS] is in OL+DISCHRG state now. Is it calibrating or do you perhaps want to set 'onlinedischarge' option? Some UPS models (e.g. CyberPower UT series) emit OL+DISCHRG when offline.

[dennypage]

@austin-0 said in NUT notifications not working on pfsense+:

I unplugged the USB cable and still there was no notification sent. I did verify that the UPS status in nut changed. It just did not send a notification.

This is puzzling.

Were you logged in on the console (ssh) when you disconnected the USB cable? Was there a notice there?

[Austin 0]

@dennypage I just tested it, and yes it does give a notification in the ssh connection.

[dennypage]

@austin-0 said in NUT notifications not working on pfsense+:

I just tested it, and yes it does give a notification in the ssh connection.

That's good--it means NUT itself is working.

Still very puzzling. I'd like to verify the contents of the pfSense NUT package. Can you please execute the following:

ls -l `pkg info --list-files pfSense-pkg-nut`

and

sha1sum `pkg info --list-files pfSense-pkg-nut`

[don't worry about the no such file or directory error for pfSense-pkg-nut-2.8.0_2]

[Austin 0]

@dennypage

ls: pfSense-pkg-nut-2.8.0_2:: No such file or directory
-rw-r--r--  1 root  wheel   1225 Feb 17 10:43 /etc/inc/priv/nut.priv.inc
-rw-r--r--  1 root  wheel   1692 Feb 17 10:43 /usr/local/pkg/nut.xml
-rwxr-xr-x  1 root  wheel   1007 Feb 17 10:43 /usr/local/pkg/nut/nut_email.php
-rw-r--r--  1 root  wheel  12255 Feb 17 10:43 /usr/local/pkg/nut/nut.inc
-rw-r--r--  1 root  wheel  10173 Feb 17 10:43 /usr/local/share/licenses/pfSense-pkg-nut-2.8.0_2/APACHE20
-rw-r--r--  1 root  wheel    164 Feb 17 10:43 /usr/local/share/licenses/pfSense-pkg-nut-2.8.0_2/catalog.mk
-rw-r--r--  1 root  wheel     66 Feb 17 10:43 /usr/local/share/licenses/pfSense-pkg-nut-2.8.0_2/LICENSE
-rw-r--r--  1 root  wheel    357 Feb 17 10:43 /usr/local/share/pfSense-pkg-nut/info.xml
-rw-r--r--  1 root  wheel  22284 Feb 17 10:43 /usr/local/www/nut_settings.php
-rw-r--r--  1 root  wheel   4720 Feb 17 10:43 /usr/local/www/nut_status.php
-rw-r--r--  1 root  wheel    193 Feb 17 10:43 /usr/local/www/shortcuts/pkg_nut.inc
-rw-r--r--  1 root  wheel     96 Feb 17 10:43 /usr/local/www/widgets/include/nut_status.inc
-rw-r--r--  1 root  wheel   3152 Feb 17 10:43 /usr/local/www/widgets/widgets/nut_status.widget.php

sha1sum: pfSense-pkg-nut-2.8.0_2:: No such file or directory
8c24e4933087b1174c821e66f940d0318a6e9b7e  /etc/inc/priv/nut.priv.inc
fe967ea9e93cd309d4b74a61e469bef85e4c4da9  /usr/local/pkg/nut.xml
efc07713dccf6f8bd43305a897437e11df769aa4  /usr/local/pkg/nut/nut.inc
7fa23e54c2f5f81125acf6ffd2f3d3df55db208a  /usr/local/pkg/nut/nut_email.php
a6a5418b4d67d9f3a33cbf184b25ac7f9fa87d33  /usr/local/share/licenses/pfSense-pkg-nut-2.8.0_2/APACHE20
6bea4964bbc2176ec94a06684a226b5813d94c0d  /usr/local/share/licenses/pfSense-pkg-nut-2.8.0_2/LICENSE
b26b9ab861e0d5b418b5e57021d768bd97a07915  /usr/local/share/licenses/pfSense-pkg-nut-2.8.0_2/catalog.mk
1ccd0187b82cd6c552aa34a776fa6cee31d76c19  /usr/local/share/pfSense-pkg-nut/info.xml
fe4c2381a6c4a51124ba3ed3a7c68c8935d58284  /usr/local/www/nut_settings.php
9f8e451e4300a77f89695004a5e8d33c4b4aade6  /usr/local/www/nut_status.php
cd553d2cfee28ef76482153f4a11b6fa3cc34f71  /usr/local/www/shortcuts/pkg_nut.inc
1d48164e283d9e320ac8bc81704a11b8949e3448  /usr/local/www/widgets/include/nut_status.inc
d33250c0811051d2fb43a77a3c3300d1595a06ca  /usr/local/www/widgets/widgets/nut_status.widget.php

[dennypage]

@austin-0 That is all correct. I'm not seeing anything wrong from a NUT pov.

Let's try the notification again and check the system log to see if there is any indication of the issue. Run this:

php-cgi /usr/local/pkg/nut/nut_email.php this is a test

followed by this:

tail /var/log/system.log

There should be a line in the system log for the php-cgi command referring to notify_monitor. What does it say?