Nut command to shutdown pfsense after specified amount of time

Gertjan

@DannyBoy2k said in Nut command to shutdown pfsense after specified amount of time:

I found that, during an actual power outage, my UPS randomly shut off all the outlets before it actually hit the LB alert.

Shutting down outlets ... and after that, signalling devices - hooked up to these outlets - is .... pretty broken.
While you haven't change your UPS, don't wait for UPS to signal what so ever.
If "power is down" is signaled, count to 60 seconds an go for a controlled shutdown.

@DannyBoy2k said in Nut command to shutdown pfsense after specified amount of time:

I've found a few post online about using upssched to do this, but it's not clear to me that the other files I would need to setup for that would survive pfsense updates.
Thoughts?

I'm using it.
Only if you re install and wipe the partition, I have to put an 'upsched' command script file in place and add the right commands in the right section (GUI) so NUT uses it.
That never actually happened for the last x years, but, everything you make and test yourself, you ought to have backups, or copy them in the Notes package, or use the Filer package, or by default, stick a post-it on your pfSense ;)
Using the upsched stuff needs severe testing and the total understanding of "how this all works".

DannyBoy2k

@Gertjan Many thanks for the feedback. Yes, "broken" is putting is mildly.

While I realize I wouldn't be able to just copy/paste what you have, would you might showing your upssched script file here as an example for me and the relevant sections of the NUT configuration in the GUI? I've found some pretty good examples online of upssched in a non-pfsense context, but would appreciate being able to see what has worked for you under pfsense.

~Dan

Gertjan

@DannyBoy2k

Sure :

In /usr/local/etc/nut/upssched.conf :

# Network UPS Tools - upssched.conf sample file
# see /usr/local/etc/nut/upssched.conf sample file

CMDSCRIPT /usr/local/etc/nut/upssched-cmd
PIPEFN /var/db/nut/upssched.pipe
LOCKFN /var/db/nut/upssched.lock

AT ONBATT  * START-TIMER onbatt 20
AT ONLINE  * CANCEL-TIMER onbatt online
AT LOWBATT * EXECUTE lowbatt
AT COMMBAD  * START-TIMER commbad 30
AT COMMOK  * CANCEL-TIMER commbad commok
AT NOCOMM * EXECUTE commbad
AT FSD * EXECUTE powerdown
AT SHUTDOWN * EXECUTE powerdown

/usr/local/etc/nut/upssched-cmd

#! /bin/sh

UPS="ups"
STATUS=$( upsc $UPS ups.status )
CHARGE=$( upsc $UPS battery.charge )
CHMSG="[$STATUS]:$CHARGE%"

# upssched-cmd  --slave --
logger -i -t upssched-cmd Calling upssched-cmd $1

case $1 in
    online) 
    MSG="$UPS, $CHMSG - power supply has been restored."
    ;;
    onbatt)
    MSG="$UPS, $CHMSG - power failure - save your work!"
    # Send a notification mail
    echo "The UPS has been on battery for awhile" | mail -s"UPS monitor" mon@gmail.com
    ;;
    lowbatt) 
    MSG="$UPS, $CHMSG - shutdown now!"
    ;;
    upsgone)
    MSG="$UPS, $CHMSG - UPS has been gone for awhile"
    ;;
    *) 
    logger -i -t upssched-cmd "Bad arg: \"$1\", $CHMSG"
    exit 1
    ;;
esac
logger -i -t upssched-cmd $MSG

From what I recall, this :

has to be disabled as there can be only one NOTIFYCMD in the upsmon.conf file - and it has to be used to call the upssched binanry, that uses the upssched.conf shown above.

NOTIFYCMD /usr/local/sbin/upssched

Not an issue, as you can mail pretty simple from the script files.

dennypage

@DannyBoy2k said in Nut command to shutdown pfsense after specified amount of time:

I actually have this same question. After thinking I was all set with using the low battery notification to shutdown my pfsense, I found that, during an actual power outage, my UPS randomly shut off all the outlets before it actually hit the LB alert. My UPS is a CyberPower CP1500PFCLCD PFC.

Putting aside getting a better UPS, I'd now rather have my pfsense shutdown after 30 minutes on battery as the UPS lasted almost 1 hour before shutting down even though it said it had 40 minutes of power remaining.

Sounds like you need to do a calibration run.

Setting that aside, look at using these in your config:

override.battery.charge.low = XX
override.battery.runtime.low = XXX

See the doc for usp.conf.

dennypage

And to be clear, unless one has developed an expert level understanding of NUT, I recommend against the use of upssched.

DannyBoy2k

OK... after doing a thorough review of the logs and some research, it appears I'm actually dealing with a CyberPower bug in how it handles the ondelay parameter. Easiest to quote what I found online:

The nut manual says set ups.delay.start (default 30) as the interval to wait before restarting the load (seconds) [after UPS power returns]. CyberPower UPSs restart ups.delay.start seconds after the UPS shutdown commences, regardless of the wall power status. Nut sets the value of ups.delay.start using the value of "ondelay" in ups.conf. Regardless of mains status, with:

ondelay = 0, UPS powers on the load immediately mains return;
ondelay = -1, UPS never powers on the load, even when mains return;
ondelay = xx, UPS powers on the load after xx (roughly) seconds after /usr/sbin/upsdrvctl shutdown is executed.

So, what I found in the logs was that my system did in fact shutdown properly on the first low battery. However, the UPS (which was already in a low battery state) came back online only a minute after it had shutdown (because I had ondelay set to 60). The system then wanted to go up and down till the UPS completely died.

What I thought I saw was that my outlets had been powered off before the low battery was signaled because I saw my system rebooting while the UPS was on when I walked into my server room. Should have done a more comprehensive review of the logs before posting.

I haven't fully tested the new ondelay = 0 setting yet, but, assuming that does keep the UPS powered off until line power is restored, I think use @dennypage 's suggestion to use the override.battery.charge.low so that, if line power goes up and own (which would make my CyberPower UPS come back on immediately), the system will still have time to shutdown again.

~Dan

netboy

@dennypage As per docs

Optional. When you specify this, the driver ignores a low battery condition flag that is reported by the UPS (some devices will switch off almost immediately after setting this flag, or will report this as soon as the mains fails). Instead it will use either of the following conditions to determine when the battery is low:

battery.charge < battery.charge.low
battery.runtime < battery.runtime.low
The idea is to set the battery.charge.low and/or battery.runtime.low levels in ups.conf to a value that gives enough time to cleanly shutdown your system:

override.battery.charge.low = 30
override.battery.runtime.low = 180
In order for this to work, your UPS should be able to (reliably) report charge and/or runtime remaining on battery. Use with caution!

Can you give us guidance how to test this functionality?

dennypage

@netboy Unless you know you have a problem, it's better to leave NUT alone to do its job.

If you know you have a problem, I.E. your UPS declares low battery with insufficient time to achieve a shutdown and stuff crashes, then you want to take configuration steps. Configure your UPS if possible, and if that isn't possible then configure NUT to declare the low battery for it. There have been several posts discussing this over the last few years.

As noted elsewhere, unless your UPS declares a low battery immediately upon mains failure, I do recommend against use of the ignorelb directive.

netboy

@dennypage said in Nut command to shutdown pfsense after specified amount of time:

UPS declares a low battery immediately upon mains failure

Great!

I have never tested this...Can I just pull the input power to UPS and observe if NUT complains low battery? If this is the case, I am all good (meaning it will gracefully shutdown netgate)?

DannyBoy2k

@netboy yes, you can do exactly that. If you pull the plug, you should be able to see the UPS widget (if you added it to the home page) show the system is on battery and, if your UPS reports it, the estimated remaining runtime. You could also see this on the Service/UPS/Status page.

If you want to see if your system will shutdown properly on low battery, you can also ssh into your pfsense (or use the console if you are using a Netgate appliance), and issue

/usr/local/sbin/upsmon -c fsd

When I did this with mine, I had to adjust the offdelay to 60 or the UPS would kill the power too quickly.

~Dan

netboy

@DannyBoy2k said in Nut command to shutdown pfsense after specified amount of time:

/usr/local/sbin/upsmon -c fsd

I want to try the above command but before testing out...have some questions.

The above commands shutdown all primary-mode UPSes,

What are the symtoms from this command that made you adjust offdelay to 60? Meaning if I issue the command should the pfsense gracefully shutdown (normal functionality)?

DannyBoy2k

@netboy said in Nut command to shutdown pfsense after specified amount of time:

The above commands shutdown all primary-mode UPSes,

That's correct. I should have said that you should run that command on your pfsense only if it's the primary.

What are the symtoms from this command that made you adjust offdelay to 60? Meaning if I issue the command should the pfsense gracefully shutdown (normal functionality)?

In my case, without that parameter set, my UPS would kill power to my pfsense before the pfsense had fully shutdown. I was using the console on my Netgate 3100 at the time and could see that it wasn't fully shutdown yet. It's been a little while now, but I want to say the UPS killed the power in 5 seconds or less. With my offsetdelay value of 60, it will now not kill the power until ~60 seconds has elapsed. My newer Netgate 4200 shuts down very quickly, but my older 3100 took longer and needed more than 5 seconds.

If you don't have access to a console like you would for a Netgate appliance, you can also read your system logs after the next boot to see if it was fully shutdown before the power was killed.

I should have also indicated in my post that the offdelay parameter is applicable for the usbhid-ups driver. So, if you're using a different method to communicate with UPS other than USB, you'd need to find whatever parameter would be applicable there, if even needed. In my case, the default was supposed to be 20 seconds according to the nut documentation, which would have been plenty, but my UPS was much quicker than that to kill the power.

~Dan

DannyBoy2k

@netboy said in Nut command to shutdown pfsense after specified amount of time:

What are the symtoms from this command that made you adjust offdelay to 60? Meaning if I issue the command should the pfsense gracefully shutdown (normal functionality)?

I just realized I didn't answer the core of your question - yes, nut should trigger pfsense to begin it's shutdown process immediately after you issue the upsmon command. You can interrogate your system logs on the next boot to make sure it was fully shutdown or observe that on the console, if using a Netgate appliance.

~Dan

netboy

@DannyBoy2k Sorry for the follow on questions. I want to pick your brain to understand the process. I will try this sometime this week.

Thanks for the detail explanation

dennypage

Note that these are two quite different questions:

Once the UPS declares a low battery, does the UPS have enough battery remaining to cover the time required for the NUT primary, and potentially secondaries, to complete their shutdowns? This is potentially several minutes.
After being instructed to cut the power by the NUT primary, does the UPS wait long enough for the NUT primary, and potentially secondaries, to complete their shutdowns before turning off the power? This is usually 20-60 seconds.