X11SBA-LN4F vs A1SRi-2558F

jjduru

I empathize with you for the troubles you're going through with that motherboard model, but in the end I guess I'll cough up the money to go for a Supermicro A1SRi-2558F. It's on long term support from Supermicro, FreeBSD 10 certified, AES-NI + QuickAssist. As you, I probably don't need headaches but reliability in the long shot.

Engineer

@jjduru:

@Engineer,

I empathize with you for the troubles you're going through with that motherboard model, but in the end I guess I'll cough up the money to go for a Supermicro A1SRi-2558F. It's on long term support from Supermicro, FreeBSD 10 certified, AES-NI + QuickAssist. As you, I probably don't need headaches but reliability in the long shot.

Well, it's been up longer than it has been so far (over two days now) and I suspect that it will get better as time goes on (and the software bugs get squashed). If the bug I posted about in another thread (quoted below) is the true culprit, it would seem that anything Intel related (at least with same driver) could be effected by this. Since FreeBSD development has recent posts about this, I would guess it would be solved fairly soon. It wouldn't be so bad to troubleshoot if the family (wife and kids) didn't come flying to me within 20 seconds of the Internet going down (even at 2:00am in the morning).

@Engineer:

Reading some more, I found this thread (https://forum.pfsense.org/index.php?topic=96325.0) which is Intel related but I'm not so sure it doesn't go beyond that. Seem FreeBSD 10.1 has an issue that needs to be corrected at the OS level that 'could' be causing this.

https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=199174

States "In Progress" under status.

Not sure if this effects Realtek cards or not but reading though the freebsd.org link, the symptoms sound exactly like mine and yours too.

rexki

The previous customer build I did was on the SuperMicro A1SRi-2558F and it too would lock up in a similar manner on 2.2.4 - tried all the normal things except TRIM - it had an SSD like almost all of my pfsense builds. Also ECC memory - which could have been more of the culprit than a friend?

The iPMI would usually get locked up as well when it failed and the pfSense web GUI was a gonner until a hard power slap. The longest it ran was probably a week and it appeared the biggest culprits of crazy DNS_error_lookups / "lockups" were pfBlocker and Snort. If these were off it would go longer on its good runs.

This appliance is now in a box awaiting a customers Comcast second cable internet install - that may take months to years? It was a full SuperMicro solution hand built from component sku's to save money. SuperMicro nickel and dimes you on the component parts like a pci riser bracket to get this into their 1U case.

Eagerly awaiting stability notes on this X11SBA-LN4F - This unit would also make a great Camera IP recorder box since the Intel i210AT can be teamed and operating system support is the biggest range of any box I have ever seen - Windows XP to Windows 10 and FreeBSD 10.

Engineer

@rexki:

Eagerly awaiting stability notes on this X11SBA-LN4F - This unit would also make a great Camera IP recorder box since the Intel i210AT can be teamed and operating system support is the biggest range of any box I have ever seen - Windows XP to Windows 10 and FreeBSD 10.

I looked at trying to update the FreeBSD driver for the Intel I210AT and noticed that the driver download page for this chipset is very large (included all of the OS that you mentioned). However, it looks like the FreeBSD version is something that has to be compiled in and not simply installed over (from what little I looked at it). I have limited (and I mean limited) knowledge of FreeBSD and building an image is not a remote possibility right now.

rexki

My Self included in drivers for Linux/BSD

noted from the freebsg bug 199174 the board was a SuperMicro X7SPA-HF with 2x GigE Intel 82574L chipset - em(4) drivers?
The BMC chip is Winbond WPCM450

The boards corresponding sub components related to this post are:
1. GigE - Intel I210-AT on the Braswell and I354 on the C2558 - seems like em(4) was the driver for both and maybe the Intel 82574L as well?
2. BMC/iPMI - Aspeed AST2400

Also read this bug reports a BMC and Ethernet lockup at the same time as I have experienced??

It looks like they have it down to a TSO setting??

Background Research since I was not sure what TSO meant -:
In computer networking, large segment offload (LSO) is a technique for increasing outbound throughput of high-bandwidth network connections by reducing CPU overhead. It works by queuing up large buffers and letting the network interface card (NIC) split them into separate packets. The technique is also called TCP segmentation offload (TSO) when applied to TCP, or generic segmentation offload (GSO).

The inbound counterpart of large segment offload is large receive offload (LRO).

pfSense - System - Advanced - Networking (tab) - Network Interfaces (section):
1. Hardware TCP Segmentation offloading (TSOx) - TX mode - disabled checked
2. Hardware Large Receive Offloading (LRO) - RX mode - disabled checked

Engineer

It's interesting to note that I had both of those checked when it crashed the first two times (maybe 3rd too…don't remember). However, I have unchecked EVERYTHING in that section and it has run for nearly 3 days without a watchdog timeout on the LAN ports. I'll keep chugging along and see what happens.

On the IPMI note, it's separate and I don't think pfsense/FreeBSD has anything to do with it on my board, correct? I've had no issues getting to the console (even during the LAN crash / watchdog timeout) using the IPMI port.

rexki

"On the IPMI note, it's separate and I don't think pfsense/FreeBSD has anything to do with it on my board, correct? I've had no issues getting to the console (even during the LAN crash / watchdog timeout) using the IPMI port."

I am not sure since the bug report you mentioned as well as my experience with all SuperMicro boards (Q2 mentioned) all have done simultaneous lockups on LANS and BMC. You are correct in that I erroneously thought your BMC locked up simultaneously - my error - sorry about the rant on BMC.

How do you get Watch Dog to tell you the LAN has frozen?

I went to my "Service WatchDog" on pfSense and only have "ipferf" and that seems to be recommended only for internal trusted LAN ports?

Engineer

I don't know how to change the watchdog, it simply popped up on my console stating watchdog timeout on igb1, which is my LAN port. Also, I couldn't get to the GUI, DHCP stopped working as well as ANYTHING Internet. However, the IPMI port, separate from the 4 regular ports, was working fine so I logged into the console and noted it was completely responsive. After the watchdog message popped up, everything reset and started working again except IPV6 sites. A watchdog error for igb1 was also logged in the system log.

I suppose IPV6 would have eventually start working but I turned it off. I've since turned it back on.

Jailer

Any updates on the stability?

Engineer

@Jailer:

Any updates on the stability?

I tried to install 2.1.5 but gave up because I couldn't get past the Root Mount Error. In the process, I somehow managed to (through BIOS) disable all of my USB ports and, for some reason, not only could I not use a USB keyboard, my IPMI port using the virtual KVM would not work (could see but could not type). I had to clear BIOS to get past that.

Once cleared and setup again, I just let it run while I was searching for answers. It has now run for 3 days with no watchdog timeouts / resets and the log seems to have fewer entries (on everything). As far as I know, the pfsense config did not change (unless the aborted 2.1.5 did something but I never made to the install part).

So running solid for 3 days but that could change at any moment. Still don't know how to get past the Root Mount Error of trying 2.1.5 though! :(

My theory (based on lots and lots and lots of reading) is that there might be an issue with the 2.4.0 IGB Intel driver that's included in FreeBSD 10.1 (and 10.2 from what I can tell). There are updated drivers (2.4.3 being the latest) but I've not tried to figure out how to compile (can't compile in pfsense as that's been removed). I have seen cases exactly like mine be solved by simply putting the older FreeBSD 8.3 Intel IGB driver (not sure of version) into this pfsense and set it to load. Was hoping 2.2.5 would have an updated driver but from talking to the folks over on the opnsense board, they think that the network drivers won't be touched much until FreeBSD 11 of mid next year.

jjduru

@Engineer,

What I don't get is that the current "Supported Devices" list on freebsd website has i210 as a supported device.

Why don't you give it a shot to install the latest stable pfsense and disable watchdog right after install?

I'm more and more inclined to get the 2558F, Thanksgiving is right around the corner.

Engineer

@jjduru:

@Engineer,

What I don't get is that the current "Supported Devices" list on freebsd website has i210 as a supported device.

Why don't you give it a shot to install the latest stable pfsense and disable watchdog right after install?

I'm more and more inclined to get the 2558F, Thanksgiving is right around the corner.

I'm on the latest pfsense (2.2.4) and not sure how to disable watchdog. The watchdog is resetting the dropped lan. Without it, it would hang indefinitely.

While it may be supported on freebsd, reading around (FreeBSD forums, FreeNAS forums, here), there seems to be a sparse pattern of the driver from FreeBSD 10.1 for the Intel IGB (2.4.0) dropping under load. I'm at 3+ days now, the longest that it's been up. Maybe there was something in BIOS that did need to be reset and resetting it fixed the issue. If it drops again, I'm going to either figure out how to install pfsense 2.1.5 or I'm going to figure out how to swap out the Intel igb driver for the one from FreeBSD 8.3 or I'll compile the latest (2.4.3) and try it.

Guest

I'm on the latest pfsense (2.2.4) and not sure how to disable watchdog. The watchdog is resetting the dropped lan. Without it, it would hang indefinitely.

dropping packets is mostly based on a small few number of issues that might be coming temporarily
again and again back, as I know it right.

A bridged port is in usage and the hardware is not f´n fast and strong enough
The entire traffic is to much for the RJ45 interface, because the NIC or the other hardware is not capable
of handling such a amount of traffic right.
Too much more packets are installed and too much services are also running or working on the same
LAN Port. IDS/IPS, DPI, or other heavy tasks that narrows down the entire CPU power.

While it may be supported on freebsd, reading around (FreeBSD forums, FreeNAS forums, here), there seems to be a sparse pattern of the driver from FreeBSD 10.1 for the Intel IGB (2.4.0) dropping under load.

In normal this can´t be, because the i211AT is the consumer LAN Port and the i210AT is the Server grade
LAN Port series from Intel!

I'm at 3+ days now, the longest that it's been up. Maybe there was something in BIOS that did need to be reset and resetting it fixed the issue.

Did you a fresh and full install? Or is this NanoBSD on the firewall?

If it drops again, I'm going to either figure out how to install pfsense 2.1.5 or I'm going to figure out how to swap out the Intel igb driver for the one from FreeBSD 8.3 or I'll compile the latest (2.4.3) and try it.

There are several methods available for you but likes often they can´t be mixed up!
pfSense 2.1.5 = FreeBSD 8.3 and only between this drivers, .ko modules should be swapped over
pfSense 2.2.4 = FreeBSD 10.1 the same game only between them…

setting up a higher mbuf size because each core is creating a queue for each CPU core,
4 Core x 5 LAN Ports = 20 queues

And in the BIOS not ever but sometimes the IPMI port is created or set up to be a fall back WAN port
and this might be also often a problem, have you tried this out before?

Engineer

It usually hangs at 4 or 5 Megabits so I doubt that traffic load is hurting this hardware or CPU. I'm only using two of the four ports right now with the other two unassigned. One port is the WAN port and the other is the LAN ports. Nothing else in the system.

As for the FreeBSD 8.3 vs 10.1, there are others that have moved the Intel ibg driver from 8.3 into 10.1, set it to load and has corrected the same issue that I have. Not sure what to say here other than it worked for them. Regardless if the I210-AT is server grade or not, a bad driver can cause issues.

Not sure what you mean about IPMI port.

The install was a fresh full install. There were the watchdog timeout / resets until 3+ days ago when I had to reset the BIOS on the motherboard because I had lost all keyboard control by turning off something to do with the USB ports (because I was trying to install 2.1.5 FULL install).

Engineer

Two more crashes this morning, one with no traffic load at 4:50 am. Will work on it later to see if I can solve this mess…sigh

Edit: Spent the better part of Saturday learning how to compile the latest Intel driver (2.4.3) for igb ports on FreeBSD (using FreeBSD 10.1 running in a VM). Finally got it installed and working but it broke Traffic Shaping (because I didn't have ALTQ support in the driver source). Regardless, it did a watchdog timeout within 6 hours of installation.

Edit #2: Put the original driver back and connected the LAN port to another switch. It ran for over a week with nothing in the long on a 'dumb switch' so it's worth a shot to see if it's the 'smart switch' or the LAN port hardware / driver. If that doesn't work, I'll switch to an un-used port on the motherboard (surely, couldn't be two ports malfunctioning if it were hardware, right? - don't answer.....:P )

Edit #3: Moved the connection on the LAN to another switch and it just went down for the count again. Will move it back to the original switch and then move the LAN from igb1 to igb2 to see if there is a hardware issue with port 1 (igb1) of the board.

Engineer

The igb2 port did a watchdog and reset also. Unless I'm mistaken and the ports run off the same chip (don't think so), this has got to be a software or configuration error. :(

Edit: Removed VLANHWTSO (ifconfig igb2 -vlanhwtso) and turned off apinger (WAN Monitoring). Cut 95% of the log out and things are running OK so far…will see if this is the magic pill that fixes this thing.

Jailer

Watching this thread with anticipation. This looks like a good board on paper and will be on my short list for a build if you can get it to run.

Engineer

@Jailer:

Watching this thread with anticipation. This looks like a good board on paper and will be on my short list for a build if you can get it to run.

I'm doing all that I can right now. Never expected in my life to learn how to compile an Intel NIC driver in FreeBSD 10.1 just to try to get pfsense to run, lol. The only thing I've not been able to get going was 2.1.5 and I'll go back to trying that if it's not stable this time.

Engineer

Update: Two+ days of uptime since I made the last changes. About 20 entries TOTAL in the GENERAL logs and 95% of those are login / logout entries.

Keeping fingers crossed that either VLAN_HWTSO removal or apinger removal fixed this. Will probably turn one or other back on if this things runs for some time (weeks) just to figure out which one was taking down the LAN - time will tell….

Update 2: Three+ days. Only thing in logs other than login / logout entries is a Dynamic DNS check (no update required).

Engineer

Update: At 4 plus days now with no watchdog. Again, only login/logout entries in the system log now.

Also, since turning off apinger, have not lost IPV6 connectivity.

Edit: Well shit….down again. Made it 4 days this time. I'm out of ideas at this point....other than 2.1.5, which I can't get to install (Root Mount Error). I will try later (2.1.5 or 2.2.5).