XG-1541 10G Ports Randomly Blink
-
Hmm, it's certainly not anything I've seen (or seen reported) before. However without actively looking for it I could imagine it just wasn't seen. How often do they blink?
The fact it happens even with the unit in standby makes me think it's more likely a WoL feature. The NICs remain powered to some extent to allow that. At that level it's obviously not a driver causing it. I've never seen other WoL capable hardware do that though.
Steve
-
@stephenw10 They don't blink that often, I haven't really timed it but I'd say once a minute or couple minutes. It's so random though. I noticed if you have a cable attached, any of the 4 NICs will show link, which I attribute is most likely Wake on LAN. I haven't seen any options in the BIOS to disable wake on lan, nor any jumpers for it. I think Supermicro calls it something else, PME, I'd have to dig around to see if that exists in the BIOS.
As far as the NICs blinking when no cable is attached, if this was wake on LAN, it would be weird they would do this when no cables are attached. The motherboard BIOS mentions a BMC Heartbeat which is basically a light that blinks to say the IPMI is functioning correctly and active. The LEDs for this are both on the motherboard and appear to be tied to the NIC but their diagram is awful and makes no mention of the rate or anything. It makes it appear it's tied to both one of the 10G and 1G NIC leds, but that is not the case, only the two 10G nics blink, and not at the same time either. I was just wondering if anybody else noticed this. Probably not cuz a router is usually on 24/7 and not really paying attention to it. I will also note it appears they do this when the unit is powered on too with no cable connected.
-
Does it do it after a complete power cycle? Or whilst in the BIOS setup?
WoL usually requires setting the NICs to do that at standby. But not always!
-
@stephenw10 I've observed it when the unit is powered off (but plugged into the wall, obviously) and when the unit is powered on, pfSense is booted but no ethernet cable connected/no link). I haven't paid any attention to it when in the BIOS. IPMI is set to Dedicated. I noticed it when I was recording a video and caught the NICs randomly blinking, hence I was able to generate a .gif. They don't obviously blink at that rate, it's a repeating gif. I could point a camera at it awhile and review the footage and see how often both NICs blink. My best guess right now is BMC Heartbeat, I just wish Supermicro would of documented it a lot better. Once I document the rate, I may shoot them a ticket to figure out what causes it. There is no issue, the unit functions just fine, I was genuinely just curious though and like to know the behavior of the unit.
-
Yeah, curious indeed. Let us know what you find.
-
@stephenw10 So I pointed a camera at the XG-1541 for 5 minutes each, powered on and powered off.
Powered On: pfSense is fully booted. No network cables connected to any NIC. Dedicated IPMI port is connected and has it's own link. WAN is configured as DHCP, LAN and OPT1 static IP.
Unit Powered on Config
Powered Off: pfSense is fully halted using option 6, system is fully powered off but still plugged into power. No network cables connected to any NIC. IPMI dedicated port is connected and has link. IPMI setting is set to Dedicated, Wake-on-LAN setting is unknown.
Camera ran 5 minutes for each scenario, powered on and powered off. The timestamps are when the particular 10G on-board RJ-45 NIC blinked. The paraphrases is the interval between when that particular NIC last blinked. As you can see, it's random for the most part.
The 1G igb NICs did not blink at all regardless if the unit was on or off. igb0 is configured as an opt1 NIC but still never blinked at all when pfSense was running. Expected behavior when no cable is connected.
Bottom: ix0
Top: ix1No expansion card has been installed into the unit, yet.
Unit Powered On
00:33 Bottom
00:34 Bottom (1 second)
01:40 Bottom (1 min 6 sec)
01:47 Bottom (7 seconds)
02:06 Bottom (19 seconds)
02:16 Bottom (10 seconds)
02:39 Top
03:20 Bottom (1 min 4 sec)
04:45 Top (2 min 6 sec)Unit Powered Off
00:50 Bottom
01:06 Top
01:31 Bottom (41 seconds)
02:17 Bottom (46 seconds)
03:17 Bottom (1 minute)
03:33 Top (2 min 27 sec)
03:55 Bottom (38 seconds)
04:16 Bottom (21 seconds)
5:00 Bottom (44 seconds) -
Hmm, some local source of electrical noise maybe?
With the unit halted try connecting the ports together so the lines are terminated. Do you still see that?
-
@stephenw10 said in XG-1541 10G Ports Randomly Blink:
Hmm, some local source of electrical noise maybe?
Wouldn't doubt it.. the way the power supply cables is handled in this unit is atrocious. The 2nd SATA Power cable was buried under a nest of PSU wires that were very tightly jammed under the front panel display Shame on whoever did that, I don't know if it was Netgate, Supermicro or Coastline. I guess they assume we'll use a Molex to SATA adapter for a 2nd 2.5in drive? You know the dodgy adapters that burn your house down. Not only that but once I was able to retrieve it, it wasn't long enough to follow the same route as the other cable, so it has to go over the motherboard to reach the 2nd drive. You can tell this power supply wasn't properly designed for this chassis and motherboard. It also has an 8-pin CPU cable that isn't utilized by this motherboard, so it's jammed under the nest of wires under that front panel display.
The SATA connectors are oriented where if you use a 90-degree SATA cable they would go opposite of the drives.
The SATA Power connectors are positioned where they will have the wires face up-wards towards the hood if you use the dual 2.5in caddy, you can't flip the SSDs upside down because then the screws won't line up with the caddy. Needless to say, I'm not very impressed with Supermicro and their design. Very disappointed actually.
@stephenw10 said in XG-1541 10G Ports Randomly Blink:
With the unit halted try connecting the ports together so the lines are terminated. Do you still see that?
If I connect an Ethernet cable between both the 10G NICs, the activity light blinks on both at the same time, then seconds later they both show a link. Connect a 10G NIC to a 1G NIC there is nothing, no link. I didn't notice blinking after connecting the 10G NICs together, but I didn't observe it very long. I have bigger fish to fry right now, getting the power supply back in this unit! When I dug for the 2nd SATA I need to put the other cables back down under that front display which is a real PITA so I pulled the PSU thinking it would help me get leverage, yeah right.. I don't know how anyone managed to shove all these PSU wires in there without breaking the front panel pcb. I'm so frustrated with it, I threw in the towel for the night. What a joke..
-
Well if terminating the ports stops the blinking that seems like either external interference from something, I hadn't imagined it would be from the device itself, or it's polling for a link somehow. But the random timing seems to make polling unlikely.
I can't say I've ever tried adding drives to a 1541 myself but the cable routing we use when installing them looks like:
I'm not aware of that ever giving trouble.Steve
-
@stephenw10 Ah, you use two single 2.5in caddies, I have the dual one. It's interesting you run the SATA power cable over the motherboard like that, I don't really like that. I guess it is just low voltage DC... The problem being, that 2nd SATA power cable on the base model, at least for me, was jammed under a pigs nest of very tight wires under the front panel display. I'm shocked it isn't broke yet..
As soon as I pulled them wires out, it is hell getting them back under there juggling not to break the display with all the pressure. All I wanted to do was run two 2.5in SATA SSDs in a ZFS Mirror, but at this point I'm like why did I even bother.. it's overkill for pfSense anyway.. sigh
-
@stephenw10 I got the power supply back in the unit and the cables back where they should be. It wasn't fun... I would hate for the PSU to ever fail in this unit, I'd rather just buy a whole other system.. System boots, all is fine.. I think...
I notice now my Ventoy drive will no longer boot via UEFI.. It just shows BIOS only.. I think this was after I did a BIOS update, I did also factory reset the BIOS settings. Ventoy booted before via UEFI, now it don't like it. Was gonna boot my Ventoy drive and run Memtest86 Pro against the RAM, cuz I upgraded the RAM. Looks like I'll have to write memtest to a flash drive the old school way. This is my first Supermicro server, and I'm not impressed.. I'll definitely stick with Dell PowerEdges. Of course pop my Ventoy drive into any other system, including other servers.. boots via UEFI with no issues.
The 10G NICs are still randomly blinking of course. I'm going to file a ticket with Supermicro about it. Right now I'm fixing all the kinks with the system. Seems anything I try to do is just biting me in the rear. This is supposed to be a server but acts more like a finicky DIY system.
