NUT package (2.8.0 and below)

dennypage

Two ups each ones feeds a pdu which feeds a dual power supply host and other dual power equipment on a small 12U rack.

I see. Old school cool. Given the lower cost of hardware these days, I don't see that much anymore. These days I usually see dual systems, each with their own UPS, set up as a fault tolerant or load balanced pair. But you probably have that as well.

Unfortunately, while NUT itself has support for multiple power sources, the pfSense NUT package does not support that configuration. The pfSense NUT package is single UPS only. It would take quite a bit of work to make it support multiples in the various (extensive) configurations, and there really hasn't been any interest. You are actually the first to ask about it.

Probably your best bet is to use one of your other systems to run NUT on, and have that system initiate a remote shutdown of pfSense when the time comes. Sorry.

josepr

@josepr
Well I have it working for at least a nice widget displaying all the info for the attached ups, which was my main goal so i can get an idea of their status in one page, dont really have it shutting down anything at the moment maybe will get to that later.

For anyone that is interested all I did was:

Modified '/usr/local/pkg/nut/nut.inc'
Added two functions:

function nut_conf() {
	global $config;
	$conf = &$config['installedpackages']['nut']['config'][0];
	return $conf;
}

function list_ups() {
	$upss = array();
	$config = nut_conf();
	if (isset($config['name'])) {
		$upss[] = $config['name'];
	}
	if (isset($config['ups_conf'])) {
		$conf =  parse_ini_string(base64_decode($config['ups_conf']), true);
		foreach ($conf as $name => $_conf) {
			$upss[] = $name;
		}
	}
	return $upss;
}

and modified the start of the nut_ups_status function

function nut_ups_status($name = null)
{
	global $config;
	$status = array();

	$nut_config = &$config['installedpackages']['nut']['config'][0];
	if (isset($nut_config['type'])) {
		$type = $nut_config['type'];
	} else {
		$type = 'disabled';
	}

	if ($type == 'disabled') {
		$status['_summary'] = "Monitoring is not enabled";
		return $status;
	}

	$ups = $name ?: $nut_config['name'];

Then i modified /usr/local/www/widgets/widgets/nut_status.widget.php

<?php
require_once("/usr/local/www/widgets/include/nut_status.inc");
require_once("/usr/local/pkg/nut/nut.inc");


if ($_REQUEST && $_REQUEST['ajax']) {
	print_all_table(); #Get multiple tables instead of just one.
	exit;
}


function print_row($desc, $value) {
	#didnt touch
}

function print_row_pct($desc, $value) {
	#didnt touch
}

function print_table($name) {
	$status = nut_ups_status($name);
        ##Only added the $name parameter, the rest stayed the same
}
#New Function to print a table per UPS
function print_all_table() {
	print '<table class="table table-bordered table-dark"> <tbody>';
	foreach (list_ups() as $name) {
		print '<tr class="panel-title"><td><b>' . $name . '</b></td></tr>';
		print '<tr><td><div">';
			print '<table class="table table-striped table-hover table-condensed">';
			print '<tbody>' . print_table($name) . '</tbody>';
			print '</table>';
		print '</div></td></tr>';
	}
	print '</tbody></table>';
}
?>
<div class="table-responsive" id = "nuttable">
	<?php print_all_table() ?>
</div>

<script type="text/javascript">
#Didnt touch
</script>

And then add my second ups under the ups.conf advanced settings.

dennypage

@josepr It might be a good idea to focus on correct NUT configuration before worrying about the monitoring widget. It appears that you've set up a situation in which the pfSense box will shutdown if either UPS fails, which I'm pretty sure is not what you want. For the work that you're doing, it would be simpler to abandon the pfSense NUT package and just configure NUT directly. Of course, you'll loose the pretty widget, but it would be a lot safer.

The APC UPSs are SNMP based, and also have their own web interface. Is there a reason you are focused on having status displayed in pfSense widgets? Instead of using their built-in web monitoring? Or an SNMP web based monitor such as Librenms or the like?

josepr

@dennypage
Actions are not taken for the device configured at ups.conf, this is only for the upsd service which setups communications with devices and can also provide access to this devices to other remote services such as a remote upsmon instance in another server.
upsmon is the service that actually enforces action and it has its own configuration which the package only generates an entry for the UPS configured the normal way, I did add a two liner to the Advance part so that it acts when both UPS are down.

Auto generated by the package for UPSMON

MONITOR ups-2 1 local-monitor **** master
SHUTDOWNCMD "/sbin/shutdown -p +0"
POWERDOWNFLAG /etc/killpower

With addition to the upsmon.conf at advance in gui

MONITOR ups-2 1 local-monitor **** master
SHUTDOWNCMD "/sbin/shutdown -p +0"
POWERDOWNFLAG /etc/killpower

###ADDED LINES upsmon.conf####
MONITOR ups-1 1 local-monitor2 **** master
MINSUPPLIES 1

Did also have to add an extra user in upsd.users as the package generates a random password everytime it saves for local-monitor

[local-monitor2]
password=****
upsmon master

This tells upsmon that it should monitor the 2 ups, that both UPS provide power for one PSU in the server and that it needs a minimum of one power supply to operate, so it is now fault tolerant (Will only conduct a shutdown if both UPS are down/critical power).

The reason i add them to PFSENSE is that it depends on both UPS so I wanted a way to make sure it reflected this and didnt try anything drastic when one UPS is offline.
It is my always on device and the last device to be shutdown and can it act as a ups provider to any other nut clients in other devices.

dennypage

@josepr said in NUT package:

This tells upsmon that it should monitor the 2 ups, that both UPS provide power for one PSU in the server and that it needs a minimum of one power supply to operate, so it is now fault tolerant (Will only conduct a shutdown if both UPS are down/critical power).

MINSUPPLIES was not shown in the configuration you posted previously.

josepr

@dennypage
Yes, but i also hadnt added
MONITOR ups-1 1 local-monitor2 **** master
to upsmon so it was not watching/monitoring the second ups.

Gooberpatrol66

I have a NUT setup that used to work, but stopped working at some point (I don't know when)

UPS type:
snmp

extra arguments:
pollfreq=1

upsd.conf:
LISTEN pfsense

upsd.users:
[upsremote]
password = blablabla
upsmon master

log:
Oct 5 19:24:07 php-fpm 352 /nut_settings.php: Beginning configuration backup to https://acb.netgate.com/save
Oct 5 19:24:08 upsmon 6682 Poll UPS [tripplite1] failed - Driver not connected
Oct 5 19:24:09 php-fpm 352 /nut_settings.php: End of configuration backup to https://acb.netgate.com/save (success).
Oct 5 19:24:09 php-fpm 352 /nut_settings.php: Stopping service nut
Oct 5 19:24:09 upsmon 6682 Signal 15: exiting
Oct 5 19:24:09 upsd 7295 User local-monitor@::1 logged out from UPS [tripplite1]
Oct 5 19:24:09 upsd 7295 mainloop: Interrupted system call
Oct 5 19:24:09 upsd 7295 Signal 15: exiting
Oct 5 19:24:09 php-fpm 352 /nut_settings.php: Starting service nut
Oct 5 19:24:09 upsmon 88547 Startup successful
Oct 5 19:24:10 upsd 89034 listening on pfsense port 3493
Oct 5 19:24:10 upsd 89034 listening on ::1 port 3493
Oct 5 19:24:10 upsd 89034 listening on 127.0.0.1 port 3493
Oct 5 19:24:10 upsd 89034 Can't connect to UPS [tripplite1] (snmp-ups-tripplite1): No such file or directory
Oct 5 19:24:10 upsd 89336 Startup successful
Oct 5 19:24:12 upsd 89336 User local-monitor@::1 logged into UPS [tripplite1]
Oct 5 19:24:12 upsmon 88697 Poll UPS [tripplite1] failed - Driver not connected
Oct 5 19:24:12 upsmon 88697 Communications with UPS tripplite1 lost
Oct 5 19:24:13 php 10742 nut_email.php: Message sent to logger@gentooserver.dehnel.info OK
Oct 5 19:24:17 upsmon 88697 Poll UPS [tripplite1] failed - Driver not connected
Oct 5 19:24:17 upsmon 88697 UPS tripplite1 is unavailable
Oct 5 19:24:18 php 19486 nut_email.php: Message sent to logger@gentooserver.dehnel.info OK

dennypage

@gooberpatrol66 You haven't included the pertinent portions of your config, but best guess based on the log is that you are simply not able to connect to the UPS.

Be sure that you are referring to the UPS by address. Don't depend upon host name resolution for a UPS.
Confirm that you can actually access the UPS with the SNMP credentials specified.

Unrelated, but a poll frequency of 1 is really overkill. The default of 5 works just fine. The lowest poll frequency recommended by NUT is 2.

AndreyMoiseev

@dennypage Hello(Sorry for my english)

pfeens 2.5.2 + Nut + UPS Ippon Back Basic 650 (connected Usb Driver Blazer).

Turned on nut and set it up
Ups settings.jpg
There was a problem with the definition UPS Ippon after reboot pfsens (Before restarting the UPS, it is determined).
Status Alert.jpg

kevindd992002

@dennypage

There was a blackout in my house a few days ago. I've noticed a very weird behavior with my Eaton UPS while I was actually monitoring the shutdown process. Here are the logs:

Oct 11 11:49:14 	kernel 		Origin="AuthenticAMD" Id=0x730f01 Family=0x16 Model=0x30 Stepping=1
Oct 11 11:49:14 	kernel 		CPU: AMD GX-412TC SOC (998.15-MHz K8-class CPU)
Oct 11 11:49:14 	kernel 		VT(vga): resolution 640x480
Oct 11 11:49:14 	kernel 		FreeBSD clang version 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c611aa2)
Oct 11 11:49:14 	kernel 		FreeBSD 12.2-STABLE fd0f54f44b5c(RELENG_2_5_0) pfSense amd64
Oct 11 11:49:14 	kernel 		FreeBSD is a registered trademark of The FreeBSD Foundation.
Oct 11 11:49:14 	kernel 		The Regents of the University of California. All rights reserved.
Oct 11 11:49:14 	kernel 		Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Oct 11 11:49:14 	kernel 		Copyright (c) 1992-2020 The FreeBSD Project.
Oct 11 11:49:14 	kernel 		---<<BOOT>>---
Oct 11 11:49:14 	syslogd 		kernel boot file is /boot/kernel/kernel
Oct 11 11:42:10 	kernel 		done. 
Oct 11 09:48:27 	shutdown 	45407 	power-down by root:
Oct 11 09:48:25 	upsd 	61950 	User monuser@192.168.10.10 logged into UPS [ups]
Oct 11 09:48:22 	upsmon 	18273 	Auto logout and shutdown proceeding
Oct 11 09:48:22 	upsmon 	18273 	Executing automatic power-fail shutdown
Oct 11 09:48:21 	upsd 	61950 	User monuser@192.168.10.10 logged out from UPS [ups]
Oct 11 09:47:18 	php 	27924 	nut_email.php: Message sent to kevindd992002@yahoo.com OK
Oct 11 09:47:04 	upsd 	61950 	Client local-monitor@::1 set FSD on UPS [ups]
Oct 11 09:47:04 	upsmon 	18273 	UPS ups battery is low

The weird behavior was that after the UPS gets a shutdown command from pfsense NUT, it shuts itself down, cuts power off from the load and then turns back on right away even though there is no power from the mains yet. And then it shuts itself down again and continues this cycle until the battery gets depleted. Any ideas what's causing this?

Also, at 9:48:21 I see a logout from one of the remote monitors, but why is there another login event at 9:48:25 right before the shutdown event by root at 09:48:27?

Gertjan

@kevindd992002 said in NUT package:

Any ideas what's causing this?

The BIOS of your pfSense uses the setting "Auto power on when power comes back".
So, when the power comes back, it will boot again.
This will loop until a) the power comes back and b) the battery is charged enough so NUT decides to stay 'on'.

Easiest solution : don't auto boot when the power comes back. This means that the shut down is controlled, but human presence is needed to switch it on again.
Best solution : Have a talk with your USP. Ask it to power the oullets when a and b or ok.
This best solution is often not possible with a typical SoHO UPS.

kevindd992002

@gertjan said in NUT package:

@kevindd992002 said in NUT package:

Any ideas what's causing this?

The BIOS of your pfSense uses the setting "Auto power on when power comes back".
So, when the power comes back, it will boot again.
This will loop until a) the power comes back and b) the battery is charged enough so NUT decides to stay 'on'.

Easiest solution : don't auto boot when the power comes back. This means that the shut down is controlled, but human presence is needed to switch it on again.
Best solution : Have a talk with your USP. Ask it to power the oullets when a and b or ok.
This best solution is often not possible with a typical SoHO UPS.

Perhaps I should've been more clear with my post. This cycle happens when there are no power in the mains yet. So imagine the UPS being shutdown by pfsense and then it cuts power from its outlets and then supplies power again to them (from battery). Why would the UPS do this if it was instructed to stay off until power from the mains come back?

I have another UPS (APC) connected to another pfsense box and when it shuts down, it stays shutdown until the mains come back. After the mains come back, pfsense (and all other NUT clients) power on and NUT knows not to initiate another shutdown event even though the thresholds are still being reached because it knows that the UPS is using mains power. This avoids the cycle you are describing.

dennypage

@andreymoiseev Can't tell anything without log info. Have you tried restarting the service? (first icon on the Status line)

Gertjan

@kevindd992002 said in NUT package:

Perhaps I should've been more clear with my post. This cycle happens when there are no power in the mains yet.

That's what I understood.
The UPS shuts down the outlets (probably because not enough energy in the battery to sustain a valid AC voltage.

@kevindd992002 said in NUT package:

So imagine the UPS being shutdown by pfsense and then it cuts power from its outlets and then supplies power again to them (from battery). Why would the UPS do this if it was instructed to stay off until power from the mains come back?

I might be wrong, but NUT only gets info from the UPS. It isn't sending anything to it.
So NUT can't tell the UPS "do this when that happens".
It's the UPS that sends over a status of the UPS when it is asked to do so.
The answer contain online/offline status, and battery charge status, and some more details.
Based on the UPS status, NUT does all the action work.

I've never seen an option like "if battery is lower then 10 % then shut down outlets" and "switch outlets back on when power is back and battery is more then 30 %".

Example : my UPS is called 'ups' in my pfSense.

[2.5.2-RELEASE][root@pfsense.my-network.tld]/root: /usr/local/bin/upsc ups@localhost
battery.charge: 100
battery.charge.low: 10
battery.charge.warning: 50
battery.date: 2001/09/25
battery.mfr.date: 2019/06/25
battery.runtime: 828
battery.runtime.low: 120
battery.type: PbAc
battery.voltage: 13.5
battery.voltage.nominal: 12.0
device.mfr: American Power Conversion
device.model: Back-UPS XS 700U
device.serial: 3B1926X61525
device.type: ups
driver.name: usbhid-ups
driver.parameter.pollfreq: 30
driver.parameter.pollinterval: 2
driver.parameter.port: auto
driver.parameter.synchronous: no
driver.version: 2.7.4
driver.version.data: APC HID 0.96
driver.version.internal: 0.41
input.sensitivity: medium
input.transfer.high: 300
input.transfer.low: 150
input.voltage: 232.0
input.voltage.nominal: 230
ups.beeper.status: disabled
ups.delay.shutdown: 20
ups.firmware: 924.Z5 .I
ups.firmware.aux: Z5
ups.load: 35
ups.mfr: American Power Conversion
ups.mfr.date: 2019/06/25
ups.model: Back-UPS XS 700U
ups.productid: 0002
ups.realpower.nominal: 390
ups.serial: 3B1926X61525
ups.status: OL
ups.test.result: No test initiated
ups.timer.reboot: 0
ups.timer.shutdown: -1
ups.vendorid: 051d

I can read these parameters but the thing is : can we set (some of) them ?

@kevindd992002 said in NUT package:

I have another UPS (APC) connected to another pfsense box and when it shuts down, it stays shutdown until the mains come back. After the mains come back, pfsense (and all other NUT clients) power on

That's seems the correct behaviour to me.

@kevindd992002 said in NUT package:

and NUT knows not to initiate another shutdown event even though the thresholds are still being reached because it knows that the UPS is using mains power. This avoids the cycle you are describing.

The thing is : NUT has does nothing to here.
The 'mains' is up - the 'battery is charging' as these are told to NUT when it asked the UPS for info.

Btw : As you can see, I'm using also an APC. Not saying hey are the best, they 'work' pretty well.

kevindd992002

@gertjan said in NUT package:

@kevindd992002 said in NUT package:

Perhaps I should've been more clear with my post. This cycle happens when there are no power in the mains yet.

That's what I understood.
The UPS shuts down the outlets (probably because not enough energy in the battery to sustain a valid AC voltage.

Are you saying that when a blackout occurs, only the remote and local (pfsense) monitors shut down and the UPS continues to supply power to the outlets until the battery gets drained?

I have this arguments set and I'm not sure if they make a difference in the event when all monitors are already in shutdown state:

override.battery.charge.warning = 80
override.battery.charge.low = 70
override.battery.runtime.low = 420

@kevindd992002 said in NUT package:

So imagine the UPS being shutdown by pfsense and then it cuts power from its outlets and then supplies power again to them (from battery). Why would the UPS do this if it was instructed to stay off until power from the mains come back?

I might be wrong, but NUT only gets info from the UPS. It isn't sending anything to it.
So NUT can't tell the UPS "do this when that happens".
It's the UPS that sends over a status of the UPS when it is asked to do so.
The answer contain online/offline status, and battery charge status, and some more details.
Based on the UPS status, NUT does all the action work.

I've never seen an option like "if battery is lower then 10 % then shut down outlets" and "switch outlets back on when power is back and battery is more then 30 %".

I thought it does both. @dennypage can probably confirm but I thought NUT is able to shut down the UPS outlets. What you're saying makes sense though.

Example : my UPS is called 'ups' in my pfSense.

[2.5.2-RELEASE][root@pfsense.my-network.tld]/root: /usr/local/bin/upsc ups@localhost
battery.charge: 100
battery.charge.low: 10
battery.charge.warning: 50
battery.date: 2001/09/25
battery.mfr.date: 2019/06/25
battery.runtime: 828
battery.runtime.low: 120
battery.type: PbAc
battery.voltage: 13.5
battery.voltage.nominal: 12.0
device.mfr: American Power Conversion
device.model: Back-UPS XS 700U
device.serial: 3B1926X61525
device.type: ups
driver.name: usbhid-ups
driver.parameter.pollfreq: 30
driver.parameter.pollinterval: 2
driver.parameter.port: auto
driver.parameter.synchronous: no
driver.version: 2.7.4
driver.version.data: APC HID 0.96
driver.version.internal: 0.41
input.sensitivity: medium
input.transfer.high: 300
input.transfer.low: 150
input.voltage: 232.0
input.voltage.nominal: 230
ups.beeper.status: disabled
ups.delay.shutdown: 20
ups.firmware: 924.Z5 .I
ups.firmware.aux: Z5
ups.load: 35
ups.mfr: American Power Conversion
ups.mfr.date: 2019/06/25
ups.model: Back-UPS XS 700U
ups.productid: 0002
ups.realpower.nominal: 390
ups.serial: 3B1926X61525
ups.status: OL
ups.test.result: No test initiated
ups.timer.reboot: 0
ups.timer.shutdown: -1
ups.vendorid: 051d

I can read these parameters but the thing is : can we set (some of) them ?

I know in the past @dennypage confirmed that some of the parameters can be modified.

@kevindd992002 said in NUT package:

I have another UPS (APC) connected to another pfsense box and when it shuts down, it stays shutdown until the mains come back. After the mains come back, pfsense (and all other NUT clients) power on

That's seems the correct behaviour to me.

@kevindd992002 said in NUT package:

and NUT knows not to initiate another shutdown event even though the thresholds are still being reached because it knows that the UPS is using mains power. This avoids the cycle you are describing.

The thing is : NUT has does nothing to here.
The 'mains' is up - the 'battery is charging' as these are told to NUT when it asked the UPS for info.

Btw : As you can see, I'm using also an APC. Not saying hey are the best, they 'work' pretty well.

I see. So since the UPS tells NUT it is using mains power, NUT doesn't do any of its power shutdown sequences, correct?

dennypage

@kevindd992002 said in NUT package:

The weird behavior was that after the UPS gets a shutdown command from pfsense NUT, it shuts itself down, cuts power off from the load and then turns back on right away even though there is no power from the mains yet. And then it shuts itself down again and continues this cycle until the battery gets depleted. Any ideas what's causing this?

Issues such as these seem to be common with low end Eatons...

Research the "ondelay" parameter and these variables (if your UPS has them):

battery.charge.restart
battery.runtime.restart
ups.delay.start
ups.start.battery
ups.timer.start

Also, you might remove any variables that may have been set such as battery.charge.low or battery.runtime.low.

In the end, you may not be able to fix it.

kevindd992002

@dennypage said in NUT package:

@kevindd992002 said in NUT package:

The weird behavior was that after the UPS gets a shutdown command from pfsense NUT, it shuts itself down, cuts power off from the load and then turns back on right away even though there is no power from the mains yet. And then it shuts itself down again and continues this cycle until the battery gets depleted. Any ideas what's causing this?

Issues such as these seem to be common with low end Eatons...

Research the "ondelay" parameter and these variables (if your UPS has them):

battery.charge.restart

battery.runtime.restart

ups.delay.start

ups.start.battery

ups.timer.start

Also, you might remove any variables that may have been set such as battery.charge.low or battery.runtime.low.

In the end, you may not be able to fix it.

If I remember correctly, this is an Eaton that has 1KVA+ capacity. Are all consumer UPS Eatons considered low end?

Can you confirm what NUT does based on the discussion above? Does it send a command to the UPS to shut down its outlets during the shutdown process?

dennypage

@kevindd992002 Please see advice above.

You've configured NUT to initiate shutdown when the battery gets down to 70%. On thing that comes to mind is that the Eaton may simply not support power off at such a high level.

You should research the variables I gave you. Particularly those with battery in their name.

If you aren't able to find a resolution that way, you will probably have to contact Eaton support for help.

kevindd992002

@dennypage said in NUT package:

@kevindd992002 Please see advice above.

You've configured NUT to initiate shutdown when the battery gets down to 70%. On thing that comes to mind is that the Eaton may simply not support power off at such a high level.

You should research the variables I gave you. Particularly those with battery in their name.

If you aren't able to find a resolution that way, you will probably have to contact Eaton support for help.

Yes, I did see your advice above. I just wanted to confirm the confusions we had in the discussion above where we are not sure if NUT can send commands to the UPS or just read data from it. I assume it's the former but I wantes to confirm with you. I guess my question is: when NUT initiates a shutdown, does it only send commands to the local and remote monitors or does it also send a shutdown command to the UPS?

Aha! You're probably right. I did recently change the thresholds to a higher level and the UPS probably doesn't support that. If I remove the thresholds, I should also remove "ignorelb" so that it uses the default thresholds of the UPS, correct?

dennypage

@kevindd992002 I believe the only time you should use ignorelb is if you have a ups that incorrectly asserts low battery immediately when mains fail.

Edit: This is incorrect. Using ignorelb is required in order for battery.charge.low and battery.runtime.low to be checked. Code reference here.