X11SBA-LN4F vs A1SRi-2558F

Jailer

@Engineer What are you using for a power supply?

Engineer

@Jailer…the built in 90W supply (new) for the Antec ISK-110. The Kill-a-Watt shows 11 watts pulled at the wall.

It could be power supply but since the console doesn't drop (nor did memtest or the system when I had a USB DVD burner attached), I would think the power seems OK.

I don't know if the SuperMicro logs power issues or not. IPMI page shows everything very good (very close to rated).

rexki

I have had a SuperMicro and Asus servers lock up on the IPMI access and the best help has always been to check for BIOS updates. Other firmwares on the system board from big iron True Raid controllers and the BMI firmware for the IPMI can fight the system board bios and cause this inability to access - It should not be related to pfSense as long as the cord is connected and you have a client on the same subnet with the BMI and client sharing the same gateway IP.

Also Java (Remote redirected video console app) can get really screwed up in a way I cannot decipher and reboots fix that usually.

Glad you confirmed 11 watts - that is an awesome number for a Quad core 2MB cache ITS board with plenty of horsepower - I think this N3700 has the same power as the C2558 and J1900 but with better power efficiency. I do now worry about ECC memory requirements since it not a File Server - its really an appliance.

Jailer

@Engineer:

@Jailer…the built in 90W supply (new) for the Antec ISK-110. The Kill-a-Watt shows 11 watts pulled at the wall.

It could be power supply but since the console doesn't drop (nor did memtest or the system when I had a USB DVD burner attached), I would think the power seems OK.

I don't know if the SuperMicro logs power issues or not. IPMI page shows everything very good (very close to rated).

Well my first guess would have been instability due to poor power supply but I can't find anything on the one used in that case. If it were me I'd still try to find another suitable power supply to try out and see if that helps.

Such are the pitfalls of blazing trails with new hardware…......

Jailer

@Engineer:

I don't know if the SuperMicro logs power issues or not. IPMI page shows everything very good (very close to rated).

Check the event log in the IPMI page and see if anything shows up there. It's under Server Health -> Event Log

Edit: Just noticed from your other post your mention of a usb DVD burner. No need for it, you can mount an ISO file in the KVM viewer and install from there. IPMI is awesome.

Engineer

@Jailer:

@Engineer:

I don't know if the SuperMicro logs power issues or not. IPMI page shows everything very good (very close to rated).

Check the event log in the IPMI page and see if anything shows up there. It's under Server Health -> Event Log

Edit: Just noticed from your other post your mention of a usb DVD burner. No need for it, you can mount an ISO file in the KVM viewer and install from there. IPMI is awesome.

Yea, I found that after the initial setup using the DVD burner. Having never used IPMI before, I didn't even have it connected to the network until after the first crash. I saw the virtual drive (using an image) in the viewer. I agree that IPMI is awesome so far.

I see nothing under the Event Log on the IPMI KVM viewer.

This is the first board that I've ever had that has an IPMI port….really do like this feature!!!!!

jjduru

Were you able to stabilize it by any chance? If yes, what did you do and with what pfsense version?

At this point I am debating between Supermicro X10SBA-L and Supermicro X11SBA-LN4F. I was dead set to get the first one, until I noticed that the Intel N3700 version has AES new instructions set.

So I am kind of waiting on your thread to see if it's yay or nay. Surprisingly, your thread is almost the only useful thread on the X11SBA-LN4F mainboard so you're opening new paths here.

Thanks,

Engineer

@jjduru:

Were you able to stabilize it by any chance? If yes, what did you and with what pfsense version?

At this point I am debating between Supermicro X10SBA-L and Supermicro X11SBA-LN4F. I was dead set to get the first one, until I noticed that the Intel N3700 version has AES new instructions set.

So I am kind of waiting on your thread to see if it's yay or nay. Surprisingly, your thread is almost the only useful thread on the X11SBA-LN4F mainboard so you're opening new paths here.

Thanks,

I'm tinkering with disabling hardware offload settings right now to see if it stabilizes. Takes quite a while to test because it has gone days between crashing before. Oh, and for the record, I had actually bought the X10SBA-L but cancelled the order because of the lower power and AES-NI instructions of the X11SBA-LN4F. It was $80 more but for future proofing (and a test showing the difference AES-NI adds to encrypted throughput), I chose the X11 board.

I'll keep this updated. I'm still on 2.2.4 for now.

jjduru

Is your mainboard having the latest BIOS version by any chance?

Engineer

@jjduru:

Is your mainboard having the latest BIOS version by any chance?

I went to the SuperMicro site the day I received the board. They didn't have ANY BIOS posted…for the main board nor the IPMI BIOS. I'll check again but don't think there is anything new there.

Edit: Nothing shows up (yet)....not even the first release BIOS or IPMI firmware.

jjduru

@Engineer,

I empathize with you for the troubles you're going through with that motherboard model, but in the end I guess I'll cough up the money to go for a Supermicro A1SRi-2558F. It's on long term support from Supermicro, FreeBSD 10 certified, AES-NI + QuickAssist. As you, I probably don't need headaches but reliability in the long shot.

Engineer

@jjduru:

@Engineer,

I empathize with you for the troubles you're going through with that motherboard model, but in the end I guess I'll cough up the money to go for a Supermicro A1SRi-2558F. It's on long term support from Supermicro, FreeBSD 10 certified, AES-NI + QuickAssist. As you, I probably don't need headaches but reliability in the long shot.

Well, it's been up longer than it has been so far (over two days now) and I suspect that it will get better as time goes on (and the software bugs get squashed). If the bug I posted about in another thread (quoted below) is the true culprit, it would seem that anything Intel related (at least with same driver) could be effected by this. Since FreeBSD development has recent posts about this, I would guess it would be solved fairly soon. It wouldn't be so bad to troubleshoot if the family (wife and kids) didn't come flying to me within 20 seconds of the Internet going down (even at 2:00am in the morning).

@Engineer:

Reading some more, I found this thread (https://forum.pfsense.org/index.php?topic=96325.0) which is Intel related but I'm not so sure it doesn't go beyond that. Seem FreeBSD 10.1 has an issue that needs to be corrected at the OS level that 'could' be causing this.

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=199174

States "In Progress" under status.

Not sure if this effects Realtek cards or not but reading though the freebsd.org link, the symptoms sound exactly like mine and yours too.

rexki

The previous customer build I did was on the SuperMicro A1SRi-2558F and it too would lock up in a similar manner on 2.2.4 - tried all the normal things except TRIM - it had an SSD like almost all of my pfsense builds. Also ECC memory - which could have been more of the culprit than a friend?

The iPMI would usually get locked up as well when it failed and the pfSense web GUI was a gonner until a hard power slap. The longest it ran was probably a week and it appeared the biggest culprits of crazy DNS_error_lookups / "lockups" were pfBlocker and Snort. If these were off it would go longer on its good runs.

This appliance is now in a box awaiting a customers Comcast second cable internet install - that may take months to years? It was a full SuperMicro solution hand built from component sku's to save money. SuperMicro nickel and dimes you on the component parts like a pci riser bracket to get this into their 1U case.

Eagerly awaiting stability notes on this X11SBA-LN4F - This unit would also make a great Camera IP recorder box since the Intel i210AT can be teamed and operating system support is the biggest range of any box I have ever seen - Windows XP to Windows 10 and FreeBSD 10.

Engineer

@rexki:

Eagerly awaiting stability notes on this X11SBA-LN4F - This unit would also make a great Camera IP recorder box since the Intel i210AT can be teamed and operating system support is the biggest range of any box I have ever seen - Windows XP to Windows 10 and FreeBSD 10.

I looked at trying to update the FreeBSD driver for the Intel I210AT and noticed that the driver download page for this chipset is very large (included all of the OS that you mentioned). However, it looks like the FreeBSD version is something that has to be compiled in and not simply installed over (from what little I looked at it). I have limited (and I mean limited) knowledge of FreeBSD and building an image is not a remote possibility right now.

rexki

My Self included in drivers for Linux/BSD

noted from the freebsg bug 199174 the board was a SuperMicro X7SPA-HF with 2x GigE Intel 82574L chipset - em(4) drivers?
The BMC chip is Winbond WPCM450

The boards corresponding sub components related to this post are:
1. GigE - Intel I210-AT on the Braswell and I354 on the C2558 - seems like em(4) was the driver for both and maybe the Intel 82574L as well?
2. BMC/iPMI - Aspeed AST2400

Also read this bug reports a BMC and Ethernet lockup at the same time as I have experienced??

It looks like they have it down to a TSO setting??

Background Research since I was not sure what TSO meant -:
In computer networking, large segment offload (LSO) is a technique for increasing outbound throughput of high-bandwidth network connections by reducing CPU overhead. It works by queuing up large buffers and letting the network interface card (NIC) split them into separate packets. The technique is also called TCP segmentation offload (TSO) when applied to TCP, or generic segmentation offload (GSO).

The inbound counterpart of large segment offload is large receive offload (LRO).

pfSense - System - Advanced - Networking (tab) - Network Interfaces (section):
1. Hardware TCP Segmentation offloading (TSOx) - TX mode - disabled checked
2. Hardware Large Receive Offloading (LRO) - RX mode - disabled checked

Engineer

It's interesting to note that I had both of those checked when it crashed the first two times (maybe 3rd too…don't remember). However, I have unchecked EVERYTHING in that section and it has run for nearly 3 days without a watchdog timeout on the LAN ports. I'll keep chugging along and see what happens.

On the IPMI note, it's separate and I don't think pfsense/FreeBSD has anything to do with it on my board, correct? I've had no issues getting to the console (even during the LAN crash / watchdog timeout) using the IPMI port.

rexki

"On the IPMI note, it's separate and I don't think pfsense/FreeBSD has anything to do with it on my board, correct? I've had no issues getting to the console (even during the LAN crash / watchdog timeout) using the IPMI port."

I am not sure since the bug report you mentioned as well as my experience with all SuperMicro boards (Q2 mentioned) all have done simultaneous lockups on LANS and BMC. You are correct in that I erroneously thought your BMC locked up simultaneously - my error - sorry about the rant on BMC.

How do you get Watch Dog to tell you the LAN has frozen?

I went to my "Service WatchDog" on pfSense and only have "ipferf" and that seems to be recommended only for internal trusted LAN ports?

Engineer

I don't know how to change the watchdog, it simply popped up on my console stating watchdog timeout on igb1, which is my LAN port. Also, I couldn't get to the GUI, DHCP stopped working as well as ANYTHING Internet. However, the IPMI port, separate from the 4 regular ports, was working fine so I logged into the console and noted it was completely responsive. After the watchdog message popped up, everything reset and started working again except IPV6 sites. A watchdog error for igb1 was also logged in the system log.

I suppose IPV6 would have eventually start working but I turned it off. I've since turned it back on.

Jailer

Any updates on the stability?

Engineer

@Jailer:

Any updates on the stability?

I tried to install 2.1.5 but gave up because I couldn't get past the Root Mount Error. In the process, I somehow managed to (through BIOS) disable all of my USB ports and, for some reason, not only could I not use a USB keyboard, my IPMI port using the virtual KVM would not work (could see but could not type). I had to clear BIOS to get past that.

Once cleared and setup again, I just let it run while I was searching for answers. It has now run for 3 days with no watchdog timeouts / resets and the log seems to have fewer entries (on everything). As far as I know, the pfsense config did not change (unless the aborted 2.1.5 did something but I never made to the install part).

So running solid for 3 days but that could change at any moment. Still don't know how to get past the Root Mount Error of trying 2.1.5 though! :(

My theory (based on lots and lots and lots of reading) is that there might be an issue with the 2.4.0 IGB Intel driver that's included in FreeBSD 10.1 (and 10.2 from what I can tell). There are updated drivers (2.4.3 being the latest) but I've not tried to figure out how to compile (can't compile in pfsense as that's been removed). I have seen cases exactly like mine be solved by simply putting the older FreeBSD 8.3 Intel IGB driver (not sure of version) into this pfsense and set it to load. Was hoping 2.2.5 would have an updated driver but from talking to the folks over on the opnsense board, they think that the network drivers won't be touched much until FreeBSD 11 of mid next year.