Troubleshooting repeated SG-3100 lockups

tuser11

We have a SG-3100 that keeps locking up. It's happening every few weeks, 4 times now in an environment where we weren't having issues more than maybe once a year. During the lockup, I can't get data via usb console. I have remote syslog data up until the actual event. I can't tell from the logs what happened. Here's what I've gathered:

Running 23.05.1-RELEASE (arm)
Avg temp 79c
Avg ram 48%
Disk 24%
Load average 6.12, 4.60, 4.37
Power adapter changed last month
There were a bunch of BLOCK events logged before it stopped logging. Maybe 1-2 every couple of seconds.
unbound was restarted almost a minute before it went offline. But based on logs, unbound seems to always be restarting. Sometimes more than once in a single minute.
No warning or error messages were logged before it went offline.

Any ideas on how to resolve this?

SteveITS

@tuser11 said in Troubleshooting repeated SG-3100 lockups:

unbound seems to always be restarting. Sometimes more than once in a single minute

If the option to register DHCP leases in DNS is checked, Unbound will restart at every lease renewal. Doesn't cause lockups though, just brief DNS slowness/outages.

"Block" as in disk, or firewall logs? You might check https://docs.netgate.com/pfsense/en/latest/troubleshooting/disk-lifetime.html.

tuser11

@SteveITS sorry, BLOCK as in IP blocks.

Can't reach gateway on any interface even with putting a static IP on my laptop. Remote syslog no longer receiving messages from firewall. USB console connection just shows a blank screen and key inputs are unresponsive. Cold reboot is only thing I was able to make it work again.

The problem started with a old SG-3100 that was in production for years. We moved to a cold backup after problems started after updating to v23.01. Current setup is running off a cold spare that has been on shelf offline for years and has only been in production since April 2023 and receiving all updates since then.

stephenw10

Yup, try disabling DHCP leases in DNS as a test. That's a high loading and that's running quite hot as a result.

rcoleman-netgate

@tuser11 Let's keep this to a single thread, please.

Locking this one down. Please use SG3100 keeps locking up after latest update