X11SBA-LN4F vs A1SRi-2558F

OLBaID

Bumping this thread.

Just bought a SM SuperServer E200-9B. Added Kingston 120gb ssd and 8gb of crucial 1333 SODIMM.

Currently I am getting Watchdog timeouts on the LAN (IGB1) interface. I am watching the new 2.3 thread here:

https://forum.pfsense.org/index.php?topic=110710.15

Possible root cause:
It seems like it might be specific to SMP (>1 CPU core)

i did find older post on watchdog timeouts:

https://doc.pfsense.org/index.php/Disable_ACPI

Now I have found this thread. The SuperServer E200-9B is currently using 3.2.1 PFsense. Bios revision is 1.0 (there is a 1.0b - but no changelog information i can find). IMPI firmware is 00.55 (newest).

I do have some fallback ALIX boards in use previously but I am concerned that the SM SuperServer E200-9B Pericom 608GP is the issue here. The previous post on RMA do show an EEPROM Firmware update of the NIC:

quote author=ldean link=topic=98230.msg594532#msg594532 date=1455223690]
Just wanted to update the thread. We received our box back from supermicro yesterday and will be installing it into production tomorrow. The repair report is somewhat vague about what they changed, but maybe it makes more sense to someone else:

Customer Reported Symptoms:
Watchdog timeout on ethernet ports.  Per TS, need ECO 18137

Test result notes and repair:
REPORTED PROBLEM FOUND.  WATCHDOG TIMEOUTS ON ETHERNET PORTS.  M/B HW ECO COMPLETED BY REWORK.  M/B BIOS, IPMI FW UPDATED TO CURRENT REVISION DONE.  CPU, DIMM SLOT DETECTION VERIFIED.  NIC PORT, USB PORT, IPMI CONNECTION TEST PASS.  NIC PORT LAN EEPROM FW UPDATED TO CURRENT REVISION COMPLETE.  NIC PORT PASSED OVERNIGHT PING TEST.  COM PORT CONNECTION VERIFIED.  SYSTEM HARDWARE FUNCTIONAL TEST PASS.  ECO VERIFIED.  ALL M/B SCREWS CHECKED.  TEST PASSED.

I'm not too sure what the ECO refers to. Anyone have an idea?

https://dl.dropboxusercontent.com/u/42296/SMSSE200-9B%20block.JPG

My Question is this hardware or PFsense 2.3 related (as others are experiencing the issue as well)? I have no issue to RMA this board back (although i really like it beyond this watchdog issue).

Any response is appreciated.

Engineer

The board, since repair (modification?), has been running for 120 days now. No issues.

Much, much better.

@OLBaID - the watchdog timeouts are a hardware issue. SM made an unknown modification on my board to eliminate the watchdog timeouts. Seems related to the PCIe switching chip (the first Ethernet port is attached directly to the PCIe bus of the N3700 while the other ports go through a port switching chip. Those three ports all have watchdog timeout issues).

There's quite a bit of information in this thread including the contact (Ken Huang IIRC) that has experience in this issue.

guardian

@Engineer:

The board, since repair (modification?), has been running for 120 days now. No issues.

There's quite a bit of information in this thread including the contact (Ken Huang IIRC) that has experience in this issue.

Great work Engineer… You saved me (and likely a few others) a lot of grief... I was going to buy that board, and with my current level of BSD knowledge I'd be F#@ked!

Based on your work, I've tried to contact SM to find out if this has been incorporated into new boards and how to positively identify which boards have been modified. Time will tell if I get an answer. I'll report back to the board for the benefit of all.

Based on the way things are now, would you say that this board is a good choice (assuming the mod is done)?

If yes, can you please comment on:

Parts you used (Case/Power Supply/Fan)
What operating temp is like
What version you are running
What packages
What throughput you are getting.

Thanks again for all your good work!

Engineer

@guardian:

Parts you used (Case/Power Supply/Fan)

What operating temp is like

What version you are running

What packages

What throughput you are getting.

Thanks again for all your good work!

Case was an Antec ISK-110 (no fans)
Power supply was the 90W built in Antec supply (fanless) - measures 10-11 watts at the wall.
Temperatures hang around 45-50C on the four cores (no fans though)
Still running 2.2.5
No packages
I'm getting my full speed but that isn't much. Waiting on TWC Maxx

I built this more out of curiosity and enjoyment than really needing it. I wanted as much CPU power that I could get at as low of power that I could get (within pricing reason of course). I do run an ipsec VPN. CPU load stays mostly around 3%. Based on research, it should be good for 1Gbit plus and with AES-NI, it should be pretty good on encrypted VPN stuff. I wanted a stable router that was future proof. After the headaches of hardware issues, I think I have it. I'm not sure that SM has implemented this into production yet even though they say that they have.

Perforado

Hint: Read the contents of the NIC-EEproms

My Guess: Changing of ASPM-Parameters (I still can't fathom who invented that atrocity and also who decided to enable it on servers boards :))

I have/had 3 of X11SBA-LN4F in production and they decided to fail this weekend. :o

Now testing: ASPM Disabled, MSI-X Disabled.

OLBaID

@Engineer:

The board, since repair (modification?), has been running for 120 days now. No issues.

Much, much better.

@OLBaID - the watchdog timeouts are a hardware issue. SM made an unknown modification on my board to eliminate the watchdog timeouts. Seems related to the PCIe switching chip (the first Ethernet port is attached directly to the PCIe bus of the N3700 while the other ports go through a port switching chip. Those three ports all have watchdog timeout issues).

There's quite a bit of information in this thread including the contact (Ken Huang IIRC) that has experience in this issue.

Engineer

Thanks so much i actually did reference this post and have talked to Ken and several others at SM and the board is back for RMA already. I also mentioned the eeprom updated listed in IDean's post.

I will update this thread once it is returned to me but please note this as well:

https://redmine.pfsense.org/issues/6296

as it is relevant (same issue) but with many hardware configurations.

Appreciate everyone whom added to this thread, the community is great!

Engineer

@OLBaID:

but please note this as well:

https://redmine.pfsense.org/issues/6296

as it is relevant (same issue) but with many hardware configurations.

Appreciate everyone whom added to this thread, the community is great!

That's something new with 2.3 it seems. I'm running 2.2.5 (had the watchdog issues with 2.2.4 and 2.2.5 with the board before repair). I tried as many option items as possible and even figured out how to compile Intel's latest driver for the I210 chip. It ran but with the same watchdog timeouts on ports 2,3 and 4 (1, which is directly to the N3700 PCIe lane, always worked just fine).

Keep us updated and good luck!

guardian

@Engineer:

The board, since repair (modification?), has been running for 120 days now. No issues.

Much, much better.

@OLBaID - the watchdog timeouts are a hardware issue. SM made an unknown modification on my board to eliminate the watchdog timeouts. Seems related to the PCIe switching chip (the first Ethernet port is attached directly to the PCIe bus of the N3700 while the other ports go through a port switching chip. Those three ports all have watchdog timeout issues).

There's quite a bit of information in this thread including the contact (Ken Huang IIRC) that has experience in this issue.

I posted this information on another thread, but I thought putting it here
might save somebody some time.

Engineer seems to have figured out thought a lot of hard work that an RMA
to encorporate ECO 18137 is what is required to make the
Supermicro X11SBA-LN4F-O N3700 stable.

AND

From what I understand from reading the form, it seems to be very low power, is
easy to keep cool, and has decent performance for it's class.

This motivated me to follow up with SM Tech Support, and I got
the following back:

–----------------------------------------------------------------------------------------

-------- Forwarded Message --------
Subject: RE: X11SBA-LN4F-O - Pre-Sales Enquiry [WT]
Date: Tue, 10 May 2016 00:02:16 +0000
From: Technical Support support@supermicro.comTo: –---@---.ca>, Technical Support support@supermicro.comHello

After further investigation, the ECOs has been implemented onto
PCB 1.02 for the aforementioned issues. When you place an order
with your distributor, please ensure to specify a PCB 1.02 to be
shipped to you.

Regards,

Technical Support

If I understand what I've read, the X11SBA-LN4F-O N3700 PCB 1.02
should make a decent pfSense platform – or am I missing something?

Clearly it's no A1SRi-2558F, but in Canada the difference between the
two boards is $141 CDN based on the best prices I could find today.
Unless broadband costs drop a lot, I can't see outgrowing it for 4 or 5
years (Minimum), and by that time I'll likely have a cap dry out and have
to replace whatever I buy anyway, so I might as well put the $141 toward
a case and memory, or am I missing something?/support@supermicro.com /support@supermicro.com

Engineer

@guardian:

If I understand what I've read, the X11SBA-LN4F-O N3700 PCB 1.02
should make a decent pfSense platform – or am I missing something?

Clearly it's no A1SRi-2558F, but in Canada the difference between the
two boards is $141 CDN based on the best prices I could find today.
Unless broadband costs drop a lot, I can't see outgrowing it for 4 or 5
years (Minimum), and by that time I'll likely have a cap dry out and have
to replace whatever I buy anyway, so I might as well put the $141 toward
a case and memory, or am I missing something?

First, thanks for the update on PCB 1.02. No, I don't think you're missing a thing. Thing has been rock solid since the change (running 2.2.5). Very low power and excellent IPMI setup (in my opinion). More processing power than I need right now and hope it lasts for a long time. 10-11 watts for entire system @ wall with no fans:

SM board
Antec ITX-110 with 90W supply
2 x 4GB DDRL DDR3 modules
120GB Sandisk Plus SSD

Quite happy with mine as of right now. TWC is supposed to bumping speeds soon so I'll give it a whirl once that happens and report back.

guardian

@Engineer:

2 x 4GB DDRL DDR3 modules

Thanks for the input.

Can you please give me a part number for RAM modules?

Also, what is you CPU temp like?

Guest

@guardian:

@Engineer:

2 x 4GB DDRL DDR3 modules

Thanks for the input.

Can you please give me a part number for RAM modules?

Also, what is you CPU temp like?

Supermicro is selling this board together with a case from them too! This combination is then called superserver,
and can be often bought ready assembled, only the RAM and SSD will be needed on top of this.
Supermicro SuperServer E200-9B Server SYS-E200-9B

Engineer

@guardian:

@Engineer:

2 x 4GB DDRL DDR3 modules

Thanks for the input.

Can you please give me a part number for RAM modules?

Also, what is you CPU temp like?

CPU temps run from 45C to 52C generally (the four cores). I'll dig up the ram soon. It's Samsung brand, IIRC.

OLBaID

"Supermicro is selling this board together with a case from them too! This combination is then called superserver,
and can be often bought ready assembled, only the RAM and SSD will be needed on top of this.
Supermicro SuperServer E200-9B Server SYS-E200-9B"

This is the system i purchased, on the asset label with the serial and MAC addresses, was revision 1.00. I am going to request them provide me any updates done to my RMA and or new revisions if exchanged so you can know the proper revision without the issue. FYI I just purchased 1333 Crucial 4GB modules for this setup and a Kingston 120gb SSD that worked with no issue. (This system does leverage the Pericom 608GP controller mentioned previously in this thread).

This system also has small 1 small case fan (just a heatsink for the cpu) included and hovered around 38c with pfsense.

Engineer

For anyone interested, I monitored the CPU load while downloading at my new ISP speed (59 Mbps) and it never topped 9%. That's with PowerD on. For some reason, the CPU load shows higher with PowerD turned on than off (I'll test it off later). The CPU hangs between 3% and 5% when idle so added 4% @ 59Mbps download. Temperature was at 55C for one core and 48-49 for the other three cores.

dreamslacker

@Engineer:

For some reason, the CPU load shows higher with PowerD turned on than off (I'll test it off later).

Because the cores are clocked down by PowerD when the load is low, hence, the percentage figure is higher - i.e. If the load needs 200MHz per core avg, then at 2GHz that's 10% but when the cores are clocked down to 1GHz, the load will show as 20%.

dreamslacker

Running a A1SRi-2758f passively in the Supermicro CSE-504-203B chassis without any chassis fans.

System idles at around 56'C (in ~28'C environment) and hasn't overheated till now (running SNORT on a 100M/100M line). I might consider adding a chassis fan once I get my servers and storage in - when the unit will have to be routing multi-gigabit inter-vlans.

I'm still waiting for my custom patch cord order from AMP and my new APC Netshelter (Dell forgot to order the rack). So far, I just have the bare minimum connected to run the office.

robi

There are cases at supermicro offering front-located connection ports, fitting these motherboards. 8)

dreamslacker

@robi:

There are cases at supermicro offering front-located connection ports, fitting these motherboards. 8)

Yes, I just didn't want them because it looks ugly in the rack. This would be part of a rack and stack showcase so it has to look nice. My Netshelter will be arriving in a day's time - Dell decided to expedit my order after screwing up.

On hindsight, I should have gone with the A1SRM-2758F and the SC510-203B. Would have had enough space to stick a sticker from the pfSense merchandise store on that chassis.

Derelict

Umm. I guess beauty is in the eye of the beholder.

OLBaID

Just wanted to reply to the thread, got my RMA back and updated and have not had the issue for over 24 hours. wanted to share the notes so anyone that has a supermicro has some help if needed. Thanks again for the contacts and help in this thread.

https://dl.dropboxusercontent.com/u/42296/SuperMicro%20RMA%20notes.PDF

And just to point to the solution that was most likely the issue:

https://redmine.pfsense.org/issues/6296