SG3100 keeps locking up after latest update
-
@stephenw10 just an FYI, our lockups have continued. We even swapped SG-3100 boxes (we had a cold spare). So it looks like the issue either isn't related to v23.01 (we are on 23.05.1 now) or we're just seeing a higher number of lockups. https://forum.netgate.com/topic/182065/troubleshooting-repeated-sg-3100-lockups
-
Are you still seeing the same disk errors?
-
@stephenw10 Nope, no disk errors
-
Hmm so just stops responding entirely? Nothing on the console? Can you log the console output across an outage to see if anything is shown when it happens?
-
Just curious but can you downgrade to a 22.X code. See if the problem follows
-
@stephenw10 Nope, USB console unresponsive when connected until after reboot. I'm not sure how to log console output across an outage. I don't have a computer I can leave connected to the unit. I was hoping remote syslog would give me all data I needed in this situation but was wrong.
-
@michmoor while I agree it's a fair step of elimination (owner discussed this as soon as we realized the issues are across 2 different SG-3100 units), I'm against downgrading and not having latest security patches on a device we depend on for network security.
-
@tuser11 Two devices does make it seem unlikely to be hardware. But yet specific to your setup since others haven't had the issue. Our 3100 has 32 days' uptime because that's when I installed 23.05.1.
OK new random thought: is there a chance of a ground loop? Do you have two buildings connected with a wire? Or two power feeds/grounds in a building?
-
@SteveITS funny you should mention a ground loop issue. I do not know if we have a ground loop issue. Basic equipment and APCs equipment show no ground or wiring faults. All test we've done ourselves and by a third party electrician revealed stable power, correct wiring and solid ground. We even had the main outside disconnect panel replaced beginning of this year to resolve some corrosion related problems found. Sometimes we have random issues with voltage that cause different UPSs in the building to randomly go on backup or cycle between AC/DC as if stuck in a power loop.
Yes, we do have 2 buildings physically connected and connected with a wire. Our electrical problems are....well lets just say I'm grateful for so many UPS's and APC being so kind about honoring their warranty considering the failure rate here is abnormal.
Could a power problem sneak past our UPS and damage our equipment? We haven't had any abnormal hardware failures downstream of the UPSs beyond the SG-3100. Dell PowerEdge Servers, Switches, NAS, all good.
Or better yet, are you thinking there may be a ground issue with some other equipment that the SG-3100 is sensitive to?
-
It's possible. Does the 2nd 3100 still stop responding if you run it somewhere else?
To log the console output would require something there connected to it. That could be something else that happens to be there, a server etc. Or a laptop perhaps or a RasPi maybe.
How practical that might be would depend on how often this happens I guess. -
@tuser11 said in SG3100 keeps locking up after latest update:
Yes, we do have 2 buildings physically connected and connected with a wire
If they have separate grounds, then what happens is, that wire carries the voltage difference between the grounds. We actually measured voltage on one once but it was long ago. IIRC it was burning out switch ports. Fiber is ideal to connect buildings.
That said, the first two sites I found say it's only an issue with shielded twisted pair and UTP is not a probem, which isn't my recollection at all, but it's been a long time since I've run into a situation not using fiber.
https://www.truecable.com/blogs/cable-academy/how-to-fix-a-ground-loop
https://networkencyclopedia.com/ground-loop/Often an actual Ethernet cable isn't fed through a UPS and even if it is it's likely looking for a 1000v surge not 10 volts.
-
@stephenw10 the other SG-3100 is offline in a box. I'm going to see if i can connect a USB from the SG-3100 to one of the Dell Servers and pass it through to a virtual machine that i can leave a putty session running on. As long as i have a static IP i can plug my laptop into the same switch and get access to that virtual machine even if SG-3100 goes offline. If that doesn't work I'll setup a pi or similar.
-
@SteveITS said in SG3100 keeps locking up after latest update:
https://www.truecable.com/blogs/cable-academy/how-to-fix-a-ground-loop
Good tip, i will look into this more as i just setup a Ethernet to an outdoor building near the house at home. Would hate to see these issues start creeping up at home.
-
R rcoleman-netgate referenced this topic on
-
I had some locking up issues a while back on my sg-3100 that turned out to be a bad power supply. When you swapped for your spare, did you also change PSUs?
-
@netplumbers Good tip. I don't remember. I looked through my logs and i have a log of changing hard drive and later changing the SG-3100 unit. No notes about changing the power supply. With that said, I'm going to assume I didn't and change it anyway tomorrow.
-
@netplumbers Unfortunately the power supply swap was already done and just wasn't in my notes. When I went to swap the power supply today, the supply in the box from the old system had our internal asset ID written on it from the old system. So the power supply in production is the newest supply with less than 4 months of use.
-
I have a putty session running from SG-3100 USB passed through to a Linux virtual machine. I'm logged into the pfsense box via putty so I have somewhere to look next time the system locks up in hopes that even though connecting USB after lockup results in unresponsive putty session, maybe this will still have some output on the screen on the next lockup.
@stephenw10 Do i just leave the prompt as is after login or is there a command i should run to stream something?
-
Anything shown should be pushed to the console whatever is happening at that time.
-
@stephenw10 Hello, today there was another lockup. I had the console connected to a virtual machine for many days waiting for it to lockup again. When it did, I logged into the vm to look at the console that was connected to SG-3100 and there was no output about the event. The last message on the screen in the console was a message that I had successfully logged in via VPN many hours before.
Can this box be easily locked up via a DDoS attack? How could that be identified when there are always lots of blocked IP addresses? I have logs up until the lockup.
-
It could exhaust the state table perhaps but that would not stop it responding at the console. Also you would see the states rising in the monitoring graphs after rebooting.
Were you able to try 'ctl+t' at the console?
If it was a drive error the console would be full of errors showing that.
A hard lock like that with nothing logged at all is more likely a hardware problem IMO.
