SG3100 keeps locking up after latest update
-
Potentially it could be some hardware failure common to those devices but if it is it's not something we've seen across more. And the fact it only seemed o happen after upgrading implies ether software or something in hardware the upgrade started hitting. But again I'd expect to see it far more widespread if so. It could be a combination of uncommon config and hardware.
-
@netplumbers do you have (not service) Watchdog enabled or disabled? (https://docs.netgate.com/pfsense/en/latest/config/advanced-misc.html#watchdog) It may be completely unrelated but in the past couple years we’ve had I think 3-4 incidents of client 3100 routers rebooting for no apparent reason. One was twice, and after the second time I turned off the RAM disk to better capture any logs but it hasn’t happened in the year or so after that. Several other 3100s without issue.
-
@SteveITS said in SG3100 keeps locking up after latest update:
https://docs.netgate.com/pfsense/en/latest/config/advanced-misc.html#watchdog
I don't want to hijack this thread but, yes, watchdog is enabled with a 128s timeout. When I experience this lockup, the box reboots sometimes and doesn't others. I was experiencing this with an increasingly high frequency getting to every couple of days before replacing the 3100's power supply under TAC's advise. Now I experience the issue every few months. I think I had 49 days of uptime until it hard locked a few days ago and required a power cycle to return to service. Perhaps if I waited longer the watchdog would have rebooted the box.
-
Mmm, it's an interesting question. If the watchdog is enabled then it should have rebooted itself rather than locked up requiring manual intervention. The fact it didn't implies at the watchdog process is still running. Or that the hardware suffered something so low level that even the watchdog stopped, which seems very unlikely.
-
@netplumbers, @tuser11 Just in case you were unaware, FreeBSD 15 will remove 32 bit ARM support so the 3100s will eventually need replacing anyway. Not really a "solution" to the problem at hand, but it presumably won't follow to new hardware...
-
@SteveITS said in SG3100 keeps locking up after latest update:
@netplumbers, @tuser11 Just in case you were unaware, FreeBSD 15 will remove 32 bit ARM support so the 3100s will eventually need replacing anyway. Not really a "solution" to the problem at hand, but it presumably won't follow to new hardware...
Yes, I'm waiting on the end state of the home-lab fallout to decide what/when to replace it. I was about to replace the 3100 before this announcement.
-
How do you all deal with e-waste? I've been holding off on upgrading because we have 2 units. I always buy 2 as it's cheaper to have 1 on standby to swap out immediately than to deal with troubleshooting hardware failure in production. I guess it's technically the same problem we all face with phones, laptops, server hard drives, etc. It's not totally related to this thread but it's part of my hangup when getting ready to buy new equipment. I've started evaluating/testing moving our local servers from the Dell racks to ATX boxes or rack mountable ATX so we don't have so much waste when we just need a CPU/etc upgrade.
In this case, just the SG-3100 cpu needs to be tossed but instead we'll have to toss the whole thing. That's 2 units to the garbage. I don't really want to go back to virtualization of pfsense and Netgate doesn't seem to sell any evergreen appliances where just the appropriate components can be upgraded.
-
@tuser11 said in SG3100 keeps locking up after latest update:
e-waste
AS an MSP we recover many PCs that clients replace. We have an arrangement with a local e-waste recycler...we are a drop-off location for them, and they will come to our office for a pick-up occasionally. Since they charge a fee for some items, we get a small percentage of that, and probably break even with our time.
Very few devices can just be upgraded to "current"...CPU sockets, memory sockets, drive tech, etc. all change frequently, so trying to upgrade something in a 7 year old PC/device would basically just be trying to find a replacement part from 5-7 years ago. By the time one replaces a motherboard/CPU/RAM/drive it would have been better to just start over. I suppose in that sense virtualization is a big way to eliminate e-waste since only the host hardware needs to be swapped out.
-
@SteveITS said in SG3100 keeps locking up after latest update:
we’ve had I think 3-4 incidents of client 3100 routers rebooting for no apparent reason
One happened today, I think the first time for this client. I haven't asked but I doubt anyone was there at 7:34 am when it booted:
Dec 1 07:34:20 kernel Copyright (c) 1992-2023 The FreeBSD Project. Dec 1 07:34:20 kernel KDB: current backend: ddb Dec 1 07:34:20 kernel KDB: debugger backends: ddb gdb Dec 1 07:34:20 kernel GDB: current port: uart Dec 1 07:34:20 kernel GDB: debug ports: uart Dec 1 07:34:20 kernel ---<<BOOT>>--- Dec 1 07:34:20 syslogd kernel boot file is /boot/kernel/kernel Dec 1 07:30:07 sshd 60046 banner exchange: Connection from 192.168.16.5 port 63668: invalid format Dec 1 07:30:07 sshd 60046 error: Fssh_kex_exchange_identification: client sent invalid protocol identifier " " Dec 1 06:21:43 kernel mvneta1: promiscuous mode enabled Dec 1 06:21:33 php-cgi 58330 [Suricata] The Rules update has finished.The 7:30 entry is a network probe/scan, and benign.
-
Hmm, does that scan take longer than 5 mins?
-
@stephenw10 It's a port scan to find new PCs/devices on the network. So the computer doing it will take a while to get through the subnet but it shouldn't spend very long on each IP address.
-
I mean is it possible it trips something on the 3100 a few minutes after it tries SSH against it?
-
@stephenw10 No, it scans quite a lot actually, sorry if I wasn't clear...here's the rest of the morning after the boot until I logged in:
Dec 1 11:20:34 sshd 90215 banner exchange: Connection from 192.168.16.5 port 56634: invalid format Dec 1 11:20:34 sshd 90215 error: Fssh_kex_exchange_identification: client sent invalid protocol identifier " " Dec 1 11:15:34 sshd 65234 banner exchange: Connection from 192.168.16.5 port 56405: invalid format Dec 1 11:15:34 sshd 65234 error: Fssh_kex_exchange_identification: client sent invalid protocol identifier " " Dec 1 10:05:30 sshd 51880 banner exchange: Connection from 192.168.16.5 port 53693: invalid format Dec 1 10:05:30 sshd 51880 error: Fssh_kex_exchange_identification: client sent invalid protocol identifier " " Dec 1 10:00:30 sshd 17452 banner exchange: Connection from 192.168.16.5 port 53496: invalid format Dec 1 10:00:30 sshd 17452 error: Fssh_kex_exchange_identification: client sent invalid protocol identifier " " Dec 1 08:50:27 sshd 10830 banner exchange: Connection from 192.168.16.5 port 50636: invalid format Dec 1 08:50:27 sshd 10830 error: Fssh_kex_exchange_identification: client sent invalid protocol identifier " " Dec 1 08:45:26 sshd 73472 banner exchange: Connection from 192.168.16.5 port 50342: invalid format Dec 1 08:45:26 sshd 73472 error: Fssh_kex_exchange_identification: client sent invalid protocol identifier " "The probe is part of our RMM software so many of our clients have it set up. It's to help detect rogue PCs, printers, etc. So I doubt it's related unless it can trigger this every year or so and not hourly. It just seems like a rare spontaneous boot.
This particular router is on a UPS but they didn't have a cable so apcupsd isn't on it. But the servers didn't alert a power problem.
At this point I don't know there's much to be done and the 3100s will all get replaced in the relatively near future anyway but I wanted to point out an example. This is typical in that I don't see anything in the logs (about the boot).
-
Ah, yes, that rules that out. Yup there's never been anything logged for devices hitting this AFAIK.
-
@stephenw10 said in SG3100 keeps locking up after latest update:
never been anything logged for devices hitting this
That doesn't surprise me, we see it every 3-6 months or so across multiple clients. So even if one person noticed it might happen every 12-36 months and they just assume a power outage and move on. It didn't really dawn on me to connect them all until this thread. And that's assuming they're connected and not power related etc. Partners might notice but I'd think not all partners are MSPs and closely monitor sold devices.
This one does not have a RAM disk though and I see nothing about the hardware watchdog logged there.
Upon reflection, I am not sure any have occurred during the workday. Possibly just coincidence. Haven't been tracking them.
I think I have successfully hijacked the thread, sorry.
