NUT package (2.8.0 and below)

q54e3w

@dennypage I'm seeing the following log entries under 2.4.5, anything to be concerned about?

Apr 16 07:09:28	upsmon	92379	Communications with UPS UPS established
Apr 16 07:09:28	upsd	33793	User local-monitor@127.0.0.1 logged into UPS [UPS]
Apr 16 07:09:23	upsmon	92379	Communications with UPS UPS lost
Apr 16 07:09:23	upsmon	92379	Poll UPS [UPS] failed - Write error: Permission denied
Apr 16 07:09:21	nginx		2020/04/16 07:09:21 [crit] 11971#100993: *62736 SSL_write() failed (13: Permission denied) while processing HTTP/2 connection, client: 192.168.20.115, server: 0.0.0.0:4433
Apr 16 07:08:43	upsmon	92379	Communications with UPS UPS established
Apr 16 07:08:43	upsd	33793	User local-monitor@127.0.0.1 logged into UPS [UPS]
Apr 16 07:08:38	upsmon	92379	Communications with UPS UPS lost
Apr 16 07:08:38	upsmon	92379	Poll UPS [UPS] failed - Write error: Permission denied

dennypage

@q54e3w Can you provide information on your NUT configuration please?

q54e3w

UPS is a network connected Eaton 9PX with Network Card 2.

UPS Type: Remote SNMP
UPS Name: UPS
Notifications: Enabled

Driver Settings
Remote IP: 192.168.10.42
Extra args:
mibs = pw
pollfreq = 15
ignorelb

Advanced Settings
FINALDELAY 180
Addition configuration lines for ups.conf = LISTEN 192.168.10.1
Additional ups.users
[remoteuser]
password = mysupersecretpassword
upsmon slave

This provides the following status

UPS Status
Name: UPS@localhost
Summary status: On line
Runtime (H:M:S): 5:17:00
UPS Load:
36.00%
Battery charge:
100.00%
Battery voltage: 52.00
Input frequency: 60.00

dennypage

@q54e3w said in NUT package:

Poll UPS [UPS] failed - Write error: Permission denied

Is there anything that precedes this entry in the log? Perhaps something similar to this?

q54e3w

@dennypage said in NUT package:

Is there anything that precedes this entry in the log? Perhaps something similar to this?

No, no other errors or messages from upsd/upsmon above or below that one. First sign of trouble is the Poll UPS failed.

dennypage

@q54e3w said in NUT package:

@dennypage said in NUT package:

Is there anything that precedes this entry in the log? Perhaps something similar to this?

No, no other errors or messages from upsd/upsmon above or below that one. First sign of trouble is the Poll UPS failed.

I wasn’t asking about other upsd/upsmon messages, rather I was asking about messages associated with links flapping, DHCP activities, etc. Similar to the logs in the link above.

q54e3w

ahhhh, gotcha. There was another issue with UPnP that I resolved and thats when I noticed these entries. I'll keep an eye on it but as I haven't seen a repeat maybe it was related. Thank you.

kevindd992002

@dennypage Why is it that in the NUT GUI I see different set of parameters shown between my two pfsense boxes (same version of pfsense, same hardware, same NUT version, different UPS but both USB):

I'm missing battery voltage and last test result on the second one. Is this normal?

dennypage

@kevindd992002 said in NUT package:

@dennypage Why is it that in the NUT GUI I see different set of parameters shown between my two pfsense boxes (same version of pfsense, same hardware, same NUT version, different UPS but both USB):

I'm missing battery voltage and last test result on the second one. Is this normal?

Not unusual. NUT only reports what what the UPS reports.

Gertjan

True.
This wouldn't be very assuring :

( image is fake of course )

kevindd992002

Ok, that makes sense.

Any idea what can causes a UPS battery low notification even though I don't see a UPS on battery message first? Here's what I get in the system logs:

I've noticed this come and go since a few days ago and I don't know why. I was receiving email notifications about a batt low condition even without an on batt notification first. These are the settings I have:

I don't see the charge.warning and charge.low that high. But what do you think?

Also, how can I set the UPS (after a shutdown condition) to wait for a few minutes before it turns everything back on? Sort of a delay?

Gertjan

@kevindd992002 said in NUT package:

causes a UPS battery low notification even though I don't see a UPS on battery message first?

If a battery was a perfect device, it couldn't be low. But they are not. They always die .... which start with the tension that starts dropping. It isn't needed to be on battery to see that happen.
It could also be the charging electronics that adapts it's loading tension, because the load current start to rise for some reason.

Also, the "UPS battery low notification" could also be a generic message from the UPS that is shown "as is" by NUT.

@kevindd992002 said in NUT package:

Also, how can I set the UPS (after a shutdown condition) to wait for a few minutes before it turns everything back on? Sort of a delay?

You should add a question to your question :
Does my UPS supports this ?
And if the answer is yes, ask another yourself another question : Is it possible with NUT and the UPS driver to communicate such a setting ?
If yes, then yes, it can be done.
How ? Dono. Up to you to search for it.

Btw : the pfSense widget is nice to see the situation.
But UPS's (batteries) should be observed over (very long) time spans. Use tools like this to see more info.

kevindd992002

@Gertjan said in NUT package:

@kevindd992002 said in NUT package:

causes a UPS battery low notification even though I don't see a UPS on battery message first?

If a battery was a perfect device, it couldn't be low. But they are not. They always die .... which start with the tension that starts dropping. It isn't needed to be on battery to see that happen.
It could also be the charging electronics that adapts it's loading tension, because the load current start to rise for some reason.

Also, the "UPS battery low notification" could also be a generic message from the UPS that is shown "as is" by NUT.

That's the answer I'm expecting. I completely know that batteries tend to die, especially in hot climates. I just thought there's an issue with NUT or something because it was fine a few days ago. There was just a blackout a few weeks ago where the battery was fully depleted in about 10 minutes so I knew it was working fine.

@kevindd992002 said in NUT package:

Also, how can I set the UPS (after a shutdown condition) to wait for a few minutes before it turns everything back on? Sort of a delay?

You should add a question to your question :
Does my UPS supports this ?
And if the answer is yes, ask another yourself another question : Is it possible with NUT and the UPS driver to communicate such a setting ?
If yes, then yes, it can be done.
How ? Dono. Up to you to search for it.

Btw : the pfSense widget is nice to see the situation.
But UPS's (batteries) should be observed over (very long) time spans. Use tools like this to see more info.

Yes, I have to search if my UPS supports it. And yes, I'm sure NUT and the USB ups driver can communicate this setting. I read about it somewhere but cannot find that specific post anymore. This is why I posted here. I guess I have to search harder.

What is that pfsense widget you've linked? Is it a package or something? I'm not seeing such a widget in the available widgets section.

Gertjan

@kevindd992002 said in NUT package:

I guess I have to search harder.

Or being lucky : someone knows what you're asking, and has the same UPS as you (APC SUA750).
I advise you to put chances on your side ;)

@kevindd992002 said in NUT package:

package or something?

Munin.
It exists as a native FreeBSD package. To install it, the traditional FreeBSD CLI commands are needed.
You also need a web server 'some where' that will be hosting the Munin RRD-to-Images.
This means a Munin-client on pfSense, and a Munin server somewhere else.
I have multiple servers, so it somewhat pays of for me to use such a graph service. It reveals a lot of info over time.

kevindd992002

@Gertjan said in NUT package:

@kevindd992002 said in NUT package:

I guess I have to search harder.

Or being lucky : someone knows what you're asking, and has the same UPS as you (APC SUA750).
I advise you to put chances on your side ;)

That's one of my UPS'es. I also have APC BR1500GI and Eaton 5E2000i.

@kevindd992002 said in NUT package:

package or something?

Munin.
It exists as a native FreeBSD package. To install it, the traditional FreeBSD CLI commands are needed.
You also need a web server 'some where' that will be hosting the Munin RRD-to-Images.
This means a Munin-client on pfSense, and a Munin server somewhere else.
I have multiple servers, so it somewhat pays of for me to use such a graph service. It reveals a lot of info over time.

I see, I'll look into that. I can spin up a docker container that will serve as a web server real quick too. Thanks for the idea.

kevindd992002

@dennypage Do you have any ideas why this would happen in a Synology NAS when it is connected to the pfsense NUT server?

So what happened there was:

12:00:24 AM -> blackout
12:02:45 AM -> FSD issued by NUT server and Syno went to safe shutdown (expected)
12:27:05 AM -> power went back on and everything started to boot up (expected)
12:27:59 AM -> Syno went to safe shutdown again (unexpected and I don't why it did that). I know that the NUT server thinks that the UPS is still in Low Battery state but I don't think it issued another FSD but I can't say for sure.
12:30:07 AM -> server back online which means it detected that AC was restored to the UPS already (unexpected because I would've thought that this message should've logged around the same time as #3 above when power really went back on)
12:31:40 AM -> RAID got reassemble which is the most disturbing part and what I'm trying to fix here.

My pfsense "ups*" logs around that time:

Apr 22 23:30:02 	upsd 	29008 	User monuser@192.168.10.10 logged into UPS [ups]
Apr 22 23:25:38 	upsd 	29008 	User monuser@192.168.10.10 logged into UPS [ups]
Apr 22 23:25:37 	upsmon 	21716 	UPS ups battery is low
Apr 22 23:25:37 	upsmon 	21716 	Communications with UPS ups established
Apr 22 23:25:34 	upsd 	29008 	Connected to UPS [ups]: usbhid-ups-ups
Apr 22 23:25:32 	upsmon 	21716 	Communications with UPS ups lost
Apr 22 23:25:32 	upsmon 	21716 	Poll UPS [ups] failed - Driver not connected
Apr 22 23:25:32 	upsd 	29008 	User local-monitor@::1 logged into UPS [ups]
Apr 22 23:25:30 	upsd 	28190 	Can't connect to UPS [ups] (usbhid-ups-ups): No such file or directory
Apr 22 23:25:28 	upsd 	12714 	User monuser@192.168.10.10 logged into UPS [ups]
Apr 22 23:25:27 	upsd 	12000 	Can't connect to UPS [ups] (usbhid-ups-ups): No such file or directory
Apr 22 23:25:25 	upsd 	78256 	User local-monitor@::1 logged out from UPS [ups]
Apr 22 23:25:24 	upsmon 	68274 	Communications with UPS ups lost
Apr 22 23:25:24 	upsmon 	68274 	Poll UPS [ups] failed - Driver not connected
Apr 22 23:25:24 	upsd 	78256 	User local-monitor@::1 logged into UPS [ups]
Apr 22 23:25:21 	upsd 	78160 	Can't connect to UPS [ups] (usbhid-ups-ups): No such file or directory
Apr 22 23:25:20 	upsmon 	22145 	Communications with UPS ups lost
Apr 22 23:25:20 	upsmon 	22145 	Poll UPS [ups] failed - Driver not connected
Apr 22 23:25:20 	upsd 	82186 	User local-monitor@::1 logged into UPS [ups]
Apr 22 23:25:18 	upsd 	80149 	Can't connect to UPS [ups] (usbhid-ups-ups): No such file or directory
Apr 22 23:25:17 	upsd 	60464 	User local-monitor@::1 logged out from UPS [ups]
Apr 22 23:25:10 	upsmon 	1149 	UPS ups battery is low
Apr 22 23:25:10 	upsmon 	1149 	Communications with UPS ups established
Apr 22 23:25:07 	upsd 	60464 	Connected to UPS [ups]: usbhid-ups-ups
Apr 22 23:25:05 	upsd 	60464 	User monuser@192.168.10.10 logged into UPS [ups]
Apr 22 23:25:05 	upsmon 	1149 	UPS ups is unavailable
Apr 22 23:25:05 	upsmon 	1149 	Poll UPS [ups] failed - Driver not connected
Apr 22 23:25:00 	upsmon 	1149 	Communications with UPS ups lost
Apr 22 23:25:00 	upsmon 	1149 	Poll UPS [ups] failed - Driver not connected
Apr 22 23:25:00 	upsd 	60464 	User local-monitor@::1 logged into UPS [ups]
Apr 22 23:24:58 	upsd 	59485 	Can't connect to UPS [ups] (usbhid-ups-ups): No such file or directory
Apr 22 23:24:55 	upsd 	23672 	Can't connect to UPS [ups] (usbhid-ups-ups): No such file or directory
Apr 22 23:18:33 	upsd 	31647 	Client local-monitor@::1 set FSD on UPS [ups]
Apr 22 23:18:33 	upsmon 	31178 	UPS ups battery is low
Apr 22 23:15:32 	upsmon 	31178 	UPS ups on battery
Apr 22 23:14:52 	upsmon 	31178 	Communications with UPS ups established
Apr 22 23:14:51 	upsd 	31647 	UPS [ups] data is no longer stale
Apr 22 23:14:47 	upsmon 	31178 	Poll UPS [ups] failed - Data stale
Apr 22 23:14:42 	upsmon 	31178 	Poll UPS [ups] failed - Data stale
Apr 22 23:14:37 	upsmon 	31178 	Poll UPS [ups] failed - Data stale
Apr 22 23:14:32 	upsmon 	31178 	Poll UPS [ups] failed - Data stale
Apr 22 23:14:27 	upsmon 	31178 	Poll UPS [ups] failed - Data stale
Apr 22 23:14:22 	upsmon 	31178 	Poll UPS [ups] failed - Data stale
Apr 22 23:14:17 	upsmon 	31178 	Poll UPS [ups] failed - Data stale
Apr 22 23:14:12 	upsmon 	31178 	Poll UPS [ups] failed - Data stale
Apr 22 23:14:07 	upsmon 	31178 	Poll UPS [ups] failed - Data stale
Apr 22 23:14:02 	upsmon 	31178 	Poll UPS [ups] failed - Data stale

I've had the same occurrence for about 2 or 3 times now. Can you help? Thanks.

dennypage

@kevindd992002 said in NUT package:

@dennypage Do you have any ideas why this would happen in a Synology NAS when it is connected to the pfsense NUT server?

It's not related to the pfSense NUT package itself. It could be a result of how Synology manages NUT, or how Synology manages boot up. Either way, the best place to seek help would be the Synology forums.

One question comes to mind to prime the pump... did the Synology actually complete its shutdown? Did it remove the POWERDOWNFLAG?

kevindd992002

@dennypage said in NUT package:

@kevindd992002 said in NUT package:

@dennypage Do you have any ideas why this would happen in a Synology NAS when it is connected to the pfsense NUT server?

It's not related to the pfSense NUT package itself. It could be a result of how Synology manages NUT, or how Synology manages boot up. Either way, the best place to seek help would be the Synology forums.

One question comes to mind to prime the pump... did the Synology actually complete its shutdown? Did it remove the POWERDOWNFLAG?

It wasn't really acting up this way before, so I'm not sure what changed.

I wouldn't really know because Synology doesn't provide logs regarding the complete shutdown but what I do know is that it will only go to safe mode with all the lights in the NAS itself lit up but with no activity (unmounted volumes, stopped services, etc.) which it did reach. If it was an improper shutdown, it would tell in the logs too. How would I confirm if it did remove the POWERDOWNFLAG?

dennypage

@kevindd992002 said in NUT package:

I wouldn't really know because Synology doesn't provide logs regarding the complete shutdown but what I do know is that it will only go to safe mode with all the lights in the NAS itself lit up but with no activity (unmounted volumes, stopped services, etc.) which it did reach. If it was an improper shutdown, it would tell in the logs too. How would I confirm if it did remove the POWERDOWNFLAG?

These questions are best explored in the Synology forums.

kevindd992002

@dennypage said in NUT package:

@kevindd992002 said in NUT package:

I wouldn't really know because Synology doesn't provide logs regarding the complete shutdown but what I do know is that it will only go to safe mode with all the lights in the NAS itself lit up but with no activity (unmounted volumes, stopped services, etc.) which it did reach. If it was an improper shutdown, it would tell in the logs too. How would I confirm if it did remove the POWERDOWNFLAG?

These questions are best explored in the Synology forums.

@dennypage

Here's the response from Synology:

After discussing with our developer, we collect the time table as below:

    12:00:24 AM Server is on battery. - AC power fail, UPS on battery
    12:02:45 AM Server going to Safe Shutdown - enter safe mode (it will not shut down the NAS immediately but keep it on safe mode if you didn't enable the option "Shutdown UPS when the system enters Safe Mode")
    12:?? AM - We suppose that UPS stops output the power to NAS so the NAS shutdown. Unfortunately, there doesn't have any log here cause by the current design.
    12:27:05 AM System started to boot up - UPS power retrieved, reboot the NAS. 
    12:27:59 AM Server going to Safe Shutdown - When the NAS is rebooting, the UPS status changed to "on battery" and low battery, so the NAS will not show "server back online" log but enter in safe mode directly after rebooting completed.
    12:28:06 AM - During the NAS enter the safe mode, the disk 4 and 5 in the expansion unit cannot be recognized in a short time then disabled by the system. Originally, the expansion unit will reset "all SATA port" then retry the link. However, it is during the safe shutdown so the expansion unit is also during the shutdown procedure, it would lead to disk 1 to disk 3 SATA link down directly as well. In conclusion, all expansion unit HDDs are disabled at the moment. [1]
    12:30:07 AM Server back online- AC retrieved UPS power retrieved, NAS starts to leave safe mode.
    12:31:19 AM System started to boot up - System force reboot after leave safe mode, it is the current policy.
    12:31:40 AM The RAID of Storage Pool 1 has been reassembled.- When the system reboot, it tried to assemble the RAID. However, all expansion unit are disabled previously so the RAID cannot be assembled normally then system try to force assemble the RAID. [2]

It looks like the disk 4 and disk 5 model ( WD100EMAZ-00WJTA0) are not stable with the NAS. After checking the HDD compatible list, it also doesn't list in it. Therefore, we cannot ensure the stability with the NAS or expansion unit

To prevent this issue, please refer to our below suggestion:

1. Replace the two disks in the expansion unit to the compatible model.

2. The log shows the UPS has on battery/power symptoms frequently. Please double-check the UPS status and AC power stability.

 

[1]

2020-04-07T00:28:03+08:00 Synology kernel: [ 110.053013] nfsd: last server has exited, flushing export cache
2020-04-07T00:28:06+08:00 Synology kernel: [ 113.429756] ata3.03: exception Emask 0x10 SAct 0x0 SErr 0x10002 action 0xf
2020-04-07T00:28:06+08:00 Synology kernel: [ 113.437457] ata3.03: SError: { RecovComm PHYRdyChg }
2020-04-07T00:28:06+08:00 Synology kernel: [ 113.443018] ata3.04: exception Emask 0x10 SAct 0x0 SErr 0x10002 action 0xf
2020-04-07T00:28:06+08:00 Synology kernel: [ 113.450728] ata3.04: SError: { RecovComm PHYRdyChg }
2020-04-07T00:28:06+08:00 Synology kernel: [ 113.456355] ata3.03: hard resetting link
2020-04-07T00:28:07+08:00 Synology kernel: [ 114.183930] ata3.03: SATA link down (SStatus 0 SControl 330)
2020-04-07T00:28:07+08:00 Synology kernel: [ 114.190281] ata3: No present pin info for SATA link down event
2020-04-07T00:28:07+08:00 Synology kernel: [ 114.196906] ata3.04: hard resetting link
2020-04-07T00:28:08+08:00 Synology kernel: [ 114.924569] ata3.04: SATA link down (SStatus 0 SControl 330)
2020-04-07T00:28:08+08:00 Synology kernel: [ 114.930895] ata3: No present pin info for SATA link down event
2020-04-07T00:28:09+08:00 Synology kernel: [ 116.198231] ata3.03: hard resetting link
2020-04-07T00:28:09+08:00 Synology kernel: [ 116.507737] ata3.03: SATA link down (SStatus 0 SControl 330)
2020-04-07T00:28:09+08:00 Synology kernel: [ 116.514072] ata3: No present pin info for SATA link down event
2020-04-07T00:28:10+08:00 Synology kernel: [ 116.938827] ata3.04: hard resetting link
2020-04-07T00:28:10+08:00 Synology kernel: [ 117.248352] ata3.04: SATA link down (SStatus 0 SControl 330)
2020-04-07T00:28:10+08:00 Synology kernel: [ 117.254756] ata3: No present pin info for SATA link down event
2020-04-07T00:28:10+08:00 Synology kernel: [ 117.261447] ata3.03: limiting SATA link speed to 1.5 Gbps
2020-04-07T00:28:10+08:00 Synology kernel: [ 117.267527] ata3.04: limiting SATA link speed to 1.5 Gbps
2020-04-07T00:28:11+08:00 Synology kernel: [ 118.522052] ata3.03: hard resetting link
2020-04-07T00:28:12+08:00 Synology kernel: [ 118.831588] ata3.03: SATA link down (SStatus 0 SControl 310)
2020-04-07T00:28:12+08:00 Synology kernel: [ 118.837924] ata3: No present pin info for SATA link down event
2020-04-07T00:28:12+08:00 Synology kernel: [ 119.262669] ata3.04: hard resetting link
2020-04-07T00:28:12+08:00 Synology kernel: [ 119.572171] ata3.04: SATA link down (SStatus 0 SControl 310)
2020-04-07T00:28:12+08:00 Synology kernel: [ 119.578506] ata3: No present pin info for SATA link down event
2020-04-07T00:28:14+08:00 Synology kernel: [ 120.845854] ata3.03: hard resetting link
2020-04-07T00:28:14+08:00 Synology kernel: [ 121.155430] ata3.03: SATA link down (SStatus 0 SControl 300)
2020-04-07T00:28:14+08:00 Synology kernel: [ 121.161759] ata3: No present pin info for SATA link down event
2020-04-07T00:28:14+08:00 Synology kernel: [ 121.586423] ata3.04: hard resetting link
2020-04-07T00:28:15+08:00 Synology kernel: [ 121.895970] ata3.04: SATA link down (SStatus 0 SControl 300)
2020-04-07T00:28:15+08:00 Synology kernel: [ 121.902307] ata3: No present pin info for SATA link down event
2020-04-07T00:28:15+08:00 Synology kernel: [ 121.908915] ata3.03: disabled
2020-04-07T00:28:15+08:00 Synology kernel: [ 121.912251] ata3.03: already disabled (class=0x2)
2020-04-07T00:28:15+08:00 Synology kernel: [ 121.917520] ata3.03: already disabled (class=0x2)
2020-04-07T00:28:15+08:00 Synology kernel: [ 121.917691] sd 2:3:0:0: rejecting I/O to offline device
2020-04-07T00:28:15+08:00 Synology kernel: [ 121.917706] Result: hostbyte=0x01 driverbyte=0x00
2020-04-07T00:28:15+08:00 Synology kernel: [ 121.917713] cdb[0]=0x35: 35 00 00 00 00 00 00 00 00 00
2020-04-07T00:28:15+08:00 Synology kernel: [ 121.917720] end_request: I/O error, dev sdjd, sector in range 9437184 + 0-2(12)
2020-04-07T00:28:15+08:00 Synology kernel: [ 121.917722] md: super_written_retry for error=-5
2020-04-07T00:28:15+08:00 Synology kernel: [ 121.917751] sd 2:3:0:0: rejecting I/O to offline device
2020-04-07T00:28:15+08:00 Synology kernel: [ 121.917753] md: super_written gets error=-5, uptodate=0
2020-04-07T00:28:15+08:00 Synology kernel: [ 121.917755] syno_md_error: sdjd3 has been removed
2020-04-07T00:28:15+08:00 Synology kernel: [ 121.917759] md/raid:md2: Disk failure on sdjd3, disabling device.
2020-04-07T00:28:15+08:00 Synology kernel: [ 121.917759] md/raid:md2: Operation continuing on 12 devices.
2020-04-07T00:28:15+08:00 Synology kernel: [ 122.001181] ata3.04: disabled
2020-04-07T00:28:15+08:00 Synology kernel: [ 122.004540] ata3.04: already disabled (class=0x2)
2020-04-07T00:28:15+08:00 Synology kernel: [ 122.009822] ata3.04: already disabled (class=0x2)
2020-04-07T00:28:15+08:00 Synology kernel: [ 122.015122] sd 2:4:0:0: rejecting I/O to offline device
2020-04-07T00:28:15+08:00 Synology kernel: [ 122.042203] Result: hostbyte=0x01 driverbyte=0x00
2020-04-07T00:28:15+08:00 Synology kernel: [ 122.051467] cdb[0]=0x35: 35 00 00 00 00 00 00 00 00 00
2020-04-07T00:28:15+08:00 Synology kernel: [ 122.057276] end_request: I/O error, dev sdje, sector 9437192
2020-04-07T00:28:15+08:00 Synology kernel: [ 122.063604] md: super_written_retry for error=-5
2020-04-07T00:28:15+08:00 Synology kernel: [ 122.068893] sd 2:4:0:0: rejecting I/O to offline device
2020-04-07T00:28:15+08:00 Synology kernel: [ 122.074737] md: super_written gets error=-5, uptodate=0
2020-04-07T00:28:15+08:00 Synology kernel: [ 122.080575] syno_md_error: sdje3 has been removed
2020-04-07T00:28:15+08:00 Synology kernel: [ 122.085824] md/raid:md2: Disk failure on sdje3, disabling device.
2020-04-07T00:28:15+08:00 Synology kernel: [ 122.085824] md/raid:md2: Operation continuing on 11 devices.

 

[2]

2020-04-07T00:31:38+08:00 Synology kernel: [ 71.434928] md: kicking non-fresh sdje3 from array!
2020-04-07T00:31:38+08:00 Synology kernel: [ 71.451639] md: kicking non-fresh sdja3 from array!
2020-04-07T00:31:38+08:00 Synology kernel: [ 71.475308] md: kicking non-fresh sdjb3 from array!
2020-04-07T00:31:38+08:00 Synology kernel: [ 71.495328] md: kicking non-fresh sdjc3 from array!
2020-04-07T00:31:38+08:00 Synology kernel: [ 71.515317] md: kicking non-fresh sdjd3 from array!
2020-04-07T00:31:38+08:00 Synology kernel: [ 71.596768] md/raid:md2: not enough operational devices (5/13 failed)
2020-04-07T00:31:38+08:00 Synology kernel: [ 71.603977] md/raid:md2: raid level 5 active with 8 out of 13 devices, algorithm 2
2020-04-07T00:31:38+08:00 Synology kernel: [ 71.648541] Buffer I/O error on device md2, logical block 0
2020-04-07T00:31:38+08:00 Synology kernel: [ 71.654784] Buffer I/O error on device md2, logical block 0
2020-04-07T00:31:38+08:00 Synology kernel: [ 71.661032] Buffer I/O error on device md2, logical block 0
2020-04-07T00:31:38+08:00 Synology kernel: [ 71.667276] Buffer I/O error on device md2, logical block 0
2020-04-07T00:31:38+08:00 Synology kernel: [ 71.678806] Buffer I/O error on device md2, logical block in range 17567100928 + 0-2(12)
2020-04-07T00:31:38+08:00 Synology kernel: [ 71.687892] Buffer I/O error on device md2, logical block 17567101616
2020-04-07T00:31:38+08:00 Synology spacetool.shared: spacetool.c:1428 Try to force assemble RAID [/dev/md2]. [0x2000 file_get_key_value.c:81]
2020-04-07T00:31:38+08:00 Synology kernel: [ 71.727347] md: md2: set sda3 to auto_remap [0]
2020-04-07T00:31:38+08:00 Synology kernel: [ 71.732457] md: md2: set sdg3 to auto_remap [0]
2020-04-07T00:31:39+08:00 Synology kernel: [ 71.737533] md: md2: set sdh3 to auto_remap [0]
2020-04-07T00:31:39+08:00 Synology kernel: [ 71.742606] md: md2: set sde3 to auto_remap [0]
2020-04-07T00:31:39+08:00 Synology kernel: [ 71.747672] md: md2: set sdf3 to auto_remap [0]
2020-04-07T00:31:39+08:00 Synology kernel: [ 71.752740] md: md2: set sdd3 to auto_remap [0]
2020-04-07T00:31:39+08:00 Synology kernel: [ 71.757806] md: md2: set sdc3 to auto_remap [0]
2020-04-07T00:31:39+08:00 Synology kernel: [ 71.762873] md: md2: set sdb3 to auto_remap [0]
2020-04-07T00:31:39+08:00 Synology spacetool.shared: spacetool.c:1156 [Info] maximum superblock.events of RAID /dev/md2 = 296085
2020-04-07T00:31:39+08:00 Synology spacetool.shared: spacetool.c:1162 [Info] add /dev/sda3 (superblock.events = 296085) for force assemble
2020-04-07T00:31:39+08:00 Synology spacetool.shared: spacetool.c:1162 [Info] add /dev/sdb3 (superblock.events = 296085) for force assemble
2020-04-07T00:31:39+08:00 Synology spacetool.shared: spacetool.c:1162 [Info] add /dev/sdc3 (superblock.events = 296085) for force assemble
2020-04-07T00:31:39+08:00 Synology spacetool.shared: spacetool.c:1162 [Info] add /dev/sdd3 (superblock.events = 296085) for force assemble
2020-04-07T00:31:39+08:00 Synology spacetool.shared: spacetool.c:1162 [Info] add /dev/sde3 (superblock.events = 296085) for force assemble
2020-04-07T00:31:39+08:00 Synology spacetool.shared: spacetool.c:1162 [Info] add /dev/sdf3 (superblock.events = 296085) for force assemble
2020-04-07T00:31:39+08:00 Synology spacetool.shared: spacetool.c:1162 [Info] add /dev/sdg3 (superblock.events = 296085) for force assemble
2020-04-07T00:31:39+08:00 Synology spacetool.shared: spacetool.c:1162 [Info] add /dev/sdh3 (superblock.events = 296085) for force assemble
2020-04-07T00:31:39+08:00 Synology spacetool.shared: spacetool.c:1162 [Info] add /dev/sdja3 (superblock.events = 296076) for force assemble
2020-04-07T00:31:39+08:00 Synology spacetool.shared: spacetool.c:1162 [Info] add /dev/sdjb3 (superblock.events = 296076) for force assemble
2020-04-07T00:31:39+08:00 Synology spacetool.shared: spacetool.c:1162 [Info] add /dev/sdjc3 (superblock.events = 296076) for force assemble
2020-04-07T00:31:39+08:00 Synology spacetool.shared: spacetool.c:1162 [Info] add /dev/sdjd3 (superblock.events = 296076) for force assemble
2020-04-07T00:31:39+08:00 Synology spacetool.shared: spacetool.c:1162 [Info] add /dev/sdje3 (superblock.events = 296076) for force assemble

I will not replace my expansion unit's disks 4 and 5 because those are brand new 10TB drives that I shucked from WD Easystore drivers and are known to be stable with Synology NAS'es, that's not the issue.

I wanted to specifically understand the 12:27:59 AM event though and I hope you can share your ideas too. If I understand correctly, they're saying that when a UPS comes back on after a power blackout and is still in charging/low battery mode, the NAS detects that low battery condition and will go to a 2nd safe shutdown mode even though there is no FSD signal sent by NUT? If so, then that really isn't good.