NUT package (2.8.0 and below)

kevindd992002

@dennypage , do you have any comments regarding my USB issues?

dennypage

@kevindd992002 You didn’t post usb entries from the system log. You need both the UPS and USB entries from the system log for the times in question.

What I’m expecting to see in there is that the USB connection is lost which is what triggers NUT’s loss of connection. This would indicate that the issue isn’t associated with NUT at all. See earlier in this thread for several discussions of USB driver issues.

kevindd992002

@dennypage said in NUT package:

@kevindd992002 You didn’t post usb entries from the system log. You need both the UPS and USB entries from the system log for the times in question.

What I’m expecting to see in there is that the USB connection is lost which is what triggers NUT’s loss of connection. This would indicate that the issue isn’t associated with NUT at all. See earlier in this thread for several discussions of USB driver issues.

Sorry, I had a typo in my previous post above. Those were the last 1000 "usb" logs (not ups logs) in the system log. I just filtered the system logs by message="ups". Did you want something else?

dennypage

@kevindd992002 What I was looking for, it was all logs containing "usb" or "ups". For instance the logs above contain only items matching "usb" which means things like "upsmon" will not be present.

However in looking at the ups logs immediately above, I don't actually see any disconnects. Did I misunderstand what your problem is? I tried going back through your posts in the thread, but they go back over a year so I may be confused as to what is current.

Can you restate what your current problem is please? Thanks.

kevindd992002

@dennypage said in NUT package:

@kevindd992002 What I was looking for, it was all logs containing "usb" or "ups". For instance the logs above contain only items matching "usb" which means things like "upsmon" will not be present.

However in looking at the ups logs immediately above, I don't actually see any disconnects. Did I misunderstand what your problem is? I tried going back through your posts in the thread, but they go back over a year so I may be confused as to what is current.

Can you restate what your current problem is please? Thanks.

The current problem are the USB disconnects in these posts:

https://forum.netgate.com/post/876424 -> these logs are filtered by message="ups", so upsmon logs are there
https://forum.netgate.com/post/876681 -> these logs are filtered by message="usb"

I'm seeing these logs:

upsmon parent process died
Communications with UPS ups lost
Can't connect to UPS [ups] (usbhid-ups-ups): No such file or directory
Can't connect to UPS [ups] (usbhid-ups-ups): Connection refused

dennypage

@kevindd992002 Are these happening at start up? Or are they happening sporadically?

kevindd992002

@dennypage These happen intermittently. As an example, here are the ups and usb logs for today when there was no power interruption of any kind:

ups logs:

Nov 27 13:38:48	upsd	41381	User local-monitor@::1 logged into UPS [ups]
Nov 27 13:38:47	upsd	41381	User monuser@192.168.10.10 logged into UPS [ups]
Nov 27 13:38:47	upsd	41381	Connected to UPS [ups]: usbhid-ups-ups
Nov 27 13:38:46	upsd	41166	Can't connect to UPS [ups] (usbhid-ups-ups): Connection refused
Nov 27 13:38:43	upsd	37228	Can't connect to UPS [ups] (usbhid-ups-ups): Connection refused
Nov 27 13:38:42	upsd	57752	User local-monitor@::1 logged out from UPS [ups]
Nov 27 13:38:07	upsd	57752	User monuser@192.168.10.10 logged into UPS [ups]
Nov 27 13:38:03	upsd	57752	User local-monitor@::1 logged into UPS [ups]
Nov 27 13:38:03	upsd	57752	Connected to UPS [ups]: usbhid-ups-ups
Nov 27 13:38:01	upsd	57439	Can't connect to UPS [ups] (usbhid-ups-ups): Connection refused
Nov 27 13:37:58	upsd	53471	Can't connect to UPS [ups] (usbhid-ups-ups): No such file or directory
Nov 27 13:37:57	upsd	12981	User local-monitor@::1 logged out from UPS [ups]

usb logs:

Nov 27 13:38:47	upsd	41381	Connected to UPS [ups]: usbhid-ups-ups
Nov 27 13:38:46	upsd	41166	Can't connect to UPS [ups] (usbhid-ups-ups): Connection refused
Nov 27 13:38:43	upsd	37228	Can't connect to UPS [ups] (usbhid-ups-ups): Connection refused
Nov 27 13:38:03	upsd	57752	Connected to UPS [ups]: usbhid-ups-ups
Nov 27 13:38:01	upsd	57439	Can't connect to UPS [ups] (usbhid-ups-ups): Connection refused
Nov 27 13:37:58	upsd	53471	Can't connect to UPS [ups] (usbhid-ups-ups): No such file or directory

kevindd992002

@dennypage

I tested my new UPS again and had these results:

If you check under UPS details, my battery.charge.low is set to 60% and battery.runtime.low is set to 396s. I unplugged the UPS from the wall outlet to recreate a blackout and I was expecting NUT to issue an FSD when either charge.low or runtime.low is reached, whichever comes first. But if you check the UPS Status, the actual charge is at 47% already and the runtime at 300s, yet I'm not still seeing it sending an FSD to the slaves.

The FSD was sent until the UPS' built-in "low battery" meter is reached and my Synology was notified to shutdown. Why is this?

dennypage

@kevindd992002 said in NUT package:

@dennypage These happen intermittently. As an example, here are the ups and usb logs for today when there was no power interruption of any kind:

Can't give you much guidance without the corresponding system logs. USB bus disconnects, WAN link changes, DHCP events, etc. can all cause a restart of NUT. You need to look at the entire system state of the system at the time of the event.

dennypage

@kevindd992002 Did you have ignorelb set?

kevindd992002

@dennypage no I didn't. Is it required so that the overrides will work? I thought otherwise.

kevindd992002

@dennypage said in NUT package:

@kevindd992002 said in NUT package:

@dennypage These happen intermittently. As an example, here are the ups and usb logs for today when there was no power interruption of any kind:

Can't give you much guidance without the corresponding system logs. USB bus disconnects, WAN link changes, DHCP events, etc. can all cause a restart of NUT. You need to look at the entire system state of the system at the time of the event.

Would you want a copy of all system logs at that time frame? I'm not sure what to check specifically because I have two pfsense boxes with roughly the same settings but I only encounter it with one box that has the APC SUA750.

dennypage

@kevindd992002 said in NUT package:

@dennypage no I didn't. Is it required so that the overrides will work? I thought otherwise.

Yes. The description in the ignorelb section of the ups.conf man page tries to say this.

While the description in the documentation is somewhat vague, the code in drivers/dstate.c makes it clear:

void status_commit(void)
{
    while (ignorelb) {
        const char  *val, *low;

        val = dstate_getinfo("battery.charge");
        low = dstate_getinfo("battery.charge.low");

        if (val && low && (strtol(val, NULL, 10) < strtol(low, NULL, 10))) {
            snprintfcat(status_buf, sizeof(status_buf), " LB");
            upsdebugx(2, "%s: appending LB flag [charge '%s' below '%s']", __func__, val, low);
            break;
        }

        val = dstate_getinfo("battery.runtime");
        low = dstate_getinfo("battery.runtime.low");

        if (val && low && (strtol(val, NULL, 10) < strtol(low, NULL, 10))) {
            snprintfcat(status_buf, sizeof(status_buf), " LB");
            upsdebugx(2, "%s: appending LB flag [runtime '%s' below '%s']", __func__, val, low);
            break;
        }

        /* LB condition not detected */
        break;
    }

    if (alarm_active) {
        dstate_setinfo("ups.status", "ALARM %s", status_buf);
    } else {
        dstate_setinfo("ups.status", "%s", status_buf);
    }
}

battery.charge.low and battery.runtime.low are not used unless ignorelb is set. Probably not how I would have done it, but I don't own the code.

dennypage

@kevindd992002 Look at the system logs for the minute preceding the event. See what's going on. You should be able to identify a trigger. If you aren't having USB disconnects, then it's likely something like an interface flap or DHCP renewal.

kevindd992002

@dennypage

When NUT successfully shuts down my Synology NAS and pfsense itself, will the UPS shut down itself to conserve battery power and cut off output power to all its output terminals?
If so, when the power comes back it should automatically boot everything up as long as the devices connected to it are set to turn on when AC power comes back, right?

I did a test today:

battery.charge.low = 60 and battery.runtime.low = 396

From 100% UPS batt, unplugged input power from mains.
At around 70% UPS batt, plugged it back to mains. So far so good.
After 10 seconds, unplugged input power from mains again.
Waited for FSD to occur and it successfully shuts down both Synology and pfsense.
After pfsense shuts down, I didn't see the UPS shut itself down and the output power from the UPS was not cut (is there a delay or something?). So I decided to plug it back to mains. The Synology NAS started booting up but pfsense stayed shut down.
After 5 seconds on being on mains, the UPS just "resets" itself (output power cut off and then on again) so everything was reset also (including the Synology that was currently in booting state).

So is there something wrong with my setup here? All I have in the extra arguments field are these:

ignorelb
override.battery.charge.warning = 70
override.battery.charge.low = 60
override.battery.runtime.low = 396

I'm not sure if there's something wrong with the UPS itself or just an issue with how I have NUT configured. Please help. Thanks.

Gertjan

@kevindd992002 said in NUT package:

(5) I didn't see the UPS shut itself down and the output power from the UPS was not cut
..... The Synology NAS started booting up but pfsense stayed shut down.

Why would the Syno boot if there was no AC cut out and restore ?

the UPS just "resets" itself (output power cut off and then on again)

Interesting. I'm using several types of APC Backup Xxx myslef for many tears now, but admit I never tested these as you did. Most setups are : "shutdown - and a human will boot you again - no AC restore", but I do have an UPS protecting my pfSEnse (master) and a Syno (slave). I'll do the same test as you didi this week, and report back.

Note : my Syno doesn't return from AC. I will have to re activate it manually - or I or can "WOL" if needed using pfSense (manually or automated - by script ...).

kevindd992002

@Gertjan said in NUT package:

@kevindd992002 said in NUT package:

(5) I didn't see the UPS shut itself down and the output power from the UPS was not cut
..... The Synology NAS started booting up but pfsense stayed shut down.

Why would the Syno boot if there was no AC cut out and restore ?

That's what I don't understand too. As soon as I plugged back the mains to the UPS, thing started to act weird. This is why I was asking how much time is the delay between pfsense shutdown and UPS shutdown? I need to wait for this time before I plug the UPS back to mains.

the UPS just "resets" itself (output power cut off and then on again)

Interesting. I'm using several types of APC Backup Xxx myslef for many tears now, but admit I never tested these as you did. Most setups are : "shutdown - and a human will boot you again - no AC restore", but I do have an UPS protecting my pfSEnse (master) and a Syno (slave). I'll do the same test as you didi this week, and report back.

Note : my Syno doesn't return from AC. I will have to re activate it manually - or I or can "WOL" if needed using pfSense (manually or automated - by script ...).

Yes, please report back. Why don't you like using the "Restart automatically after a power failure" feature in your Syno?

Gertjan

@kevindd992002 said in NUT package:

Why don't you like using the "Restart automatically after a power failure" feature in your Syno?

Because when the power goes down, I like to wait with putting my devices to on. The "Power on" situation is often as bad or even worse as the "power down" moment, with my UPS in bad shape.

dennypage

@kevindd992002 Yes, there is a delay. How this works varies by UPS make/model. Some also have a restart lockout/delay. See the NUT documentation, the NUT mailing list, and your UPS documentation for information. Don’t expect to find easy/obvious answers for this as there are a lot of differences in behavior amongst the hundreds of UPS models that NUT supports.

FWIW, You are better off asking questions about general NUT And UPS behaviors on the NUT mailing list rather than in the pfSense forum.

kevindd992002

@dennypage said in NUT package:

@kevindd992002 Yes, there is a delay. How this works varies by UPS make/model. Some also have a restart lockout/delay. See the NUT documentation, the NUT mailing list, and your UPS documentation for information. Don’t expect to find easy/obvious answers for this as there are a lot of differences in behavior amongst the hundreds of UPS models that NUT supports.

FWIW, You are better off asking questions about general NUT And UPS behaviors on the NUT mailing list rather than in the pfSense forum.

Ok, thanks.

Also, I tried recreating a shutdown sequence by issuing a /usr/local/sbin/upsmon -c fsd command in pfsense (so as to preserve battery lifetime while testing all these) but the weird thing is that my Syno doesn't seem to get a hold of the FSD event. My pfsense master and UPS shuts down properly though. As soon as I issue the FSD, the /var/log/messages in my Syno says this:

2019-12-31T02:05:03+08:00 Synology synoups: [UPS] Check Boot Status.
2019-12-31T02:05:03+08:00 Synology synoups: WARNING: UPS is On-Line

But when I recreate the shutdown sequence by unplugging the mains, all goes well and the Syno picks up the OB LB FSD event properly. Does this mean that Syno needs to see that the UPS is in OB LB state to issue a shutdown even if it sees an FSD event already?