X11SBA-LN4F vs A1SRi-2558F

Engineer

Have just moved this back into full service (WAN) and am running off of it since it made it 8 days (4 on Ubuntu and 4 on pfsense 2.2.5/FreeBSD) without a network going down issue. I'll keep people informed and answer questions that I can. Also enabled TRIM for the first time (finally).

Thanks for the help everyone and Merry Christmas! :)

Engineer

Currently at 4+ days on pfsense 2.2.5 / FreeBSD 10.1 without the LAN going down / Watchdog. I think it would be fair to say that this issue has been solved for my board. Now whether SuperMicro has to change other boards or BIOS is anyone's guess right now as I have no feedback from them yet.

bluepr0

Thanks for all the info!

I was wondering how it performs your board when under heavy usage and adding snort, ntopng,etc. I'm currently using a Mac Mini (late 2012) with pfSense and when using all my internet (around 200/20) the CPU usage gets around 40%). My CPU is a i5-3210M, looking at cpubenchmarks website it seems it got around the double of performance of yours… so that would mean (more or less) that when getting 200mbps from the router you should see an usage of around 80%.

Do you have any numbers on this? thanks!

Engineer

bluepr0, sorry, I don't. I looked at your specs and your processor is a dual core with hyperthreading while this one is a slower quad core processor. Your CPU load number seem higher than I would have expected thinking that your processor should handle higher speeds at lower CPU load. I'm not experienced enough yet to say that though…just a guess on my part.

Are both of your cores pulling 40% at that time or is only one core pulling 40%? When simply pinging in and out, the average was 6Mbps in / out and I never saw any core above 4% load with all three other cores running 100% idle (using command top -S -H from the shell or from the command prompt). But, I have not other packages running and am running basic, slow (compared to yours) internet. Was trying to build a low power, somewhat future proof unit that could handle up to 1Gbps including encryption (AES-NI) in the future. Your numbers make me second guess my choice and think that I may have overestimated this board. sigh

Edit: After reading a bit, it seems it doesn't take much (relatively) to run a 1Gbps plain connection but adding SNORT, etc. is where much more processor power is needed to pump data through. SNORT is single threaded so it would eat up much more of a single CPU core (I did read you can run multiple copies of SNORT and load balance it out but it was much more difficult to do so).

bluepr0

I ran again the tests, using top command to see the CPU usage (it seems the CPU usage on web interface is also accurate, making an average of all the cores or am I wrong?)

Here's the screenshots

1. This is using all my download bandwidth from the internet http://d.pr/i/1h6X9/3UOn6rvy
CPU usage is around 30% here

2. This is using all my download bandwidth from internet + iperf (maxing out gigabit) http://d.pr/i/1bL1T/5PWn64bR
CPU usage is around 84% here

What do you think?

Thanks!

Jailer

Looks like Squid is what is eating up a lot of your resources @bluepr0.

Engineer, thanks for the update. I was going to ping you and see how things were going so your response was perfect timing.

bluepr0

Yep! stopping snort, ntop reduces quite a bit the CPU usage. See http://d.pr/i/1eYwd/NinxWFhR (usage is around 50%)

Engineer

@Jailer:

Looks like Squid is what is eating up a lot of your resources @bluepr0.

Engineer, thanks for the update. I was going to ping you and see how things were going so your response was perfect timing.

Right at 7 days now. Never made it above 4.5 or so previously. :)

bluepr0

great to know! I'm wondering what they change on the board? does the other boards out there will only need a BIOS update or it's a hardware problem?

Engineer

@bluepr0:

great to know! I'm wondering what they change on the board? does the other boards out there will only need a BIOS update or it's a hardware problem?

@bluepr0, I honestly don't know. SuperMicro said it was a "hardware modification" but they will not disclose what was changed and I cannot see any physical changes to my board. When I asked whether other boards are affected or only mine, I was told that if there are any other boards out there with this problem, they will fix them. Sorry guys, I wish I could give you more but SuperMicro won't let it out. I have not seen a new BIOS other than a new IPMI firmware.

reilos

So i decided to go with the A1SRi-2558F board (and ECC RAM). Connected LAN1 to WAN (ISP router in bridge mode, connected through surge-protected RJ45 ports on the UPS), connected other 3 LAN ports and IPMI to my (managed) switch.
Installing pfSense onto the SSD through IPMI mounted ISO… Sudden reboot... 2nd try... Install went fine, but reboot after a few minutes. My first thought is was something with the pfSense install. But....
Went into BIOS settings and before i could even make any changes... Reboot. So it's NOT a pfSense issue.
Disconnected all network cables except IPMI, restored default BIOS settings, did 10 cycles of memtest86 (took almost 3 days)... No errors and no reboots.
Then I connected LAN1 to my network, configured it as the WAN port and connected an AP to LAN2 (configured as LAN). Booted pfSense, did nothing for about an hour and no reboots.

So, I am now thinking it's a problem with 1 or more of the LAN ports. I want to do some testing, maybe anyone has some idea on how to do effective / efficiënt tests on the LAN ports?

jahonix

Do you use STP or UTP cables?

reilos

@jahonix:

Do you use STP or UTP cables?

I use all CAT6 U/UTP cables. For the whole network, including test setup.

jahonix

This eliminates at least a possible hum/ground loop, well known to audio guys (me) and lesser known in the network world.
Just for the record: I do not say to only use UTP cables. It just rules out an idea I had in this case.

reilos

@jahonix:

This eliminates at least a possible hum/ground loop, well known to audio guys (me) and lesser known in the network world.
Just for the record: I do not say to only use UTP cables. It just rules out an idea I had in this case.

Yeah, know what that is. In my setup i don't really need shielded cables (and i also dont want to go through the hassle of properly grounding the whole thing).
But i'm still hoping for an idea to properly test the LAN ports…

Guest

Yeah, know what that is. In my setup i don't really need shielded cables (and i also dont want to go through the hassle of properly grounding the whole thing).

I really don´t know from where you all are and what you have to pay for network cables, but here in Germany
I have to pay the following money for each;

1 meter patch cable CAT.5e UTP 1,80 €
1 meter patch cable CAT.6a S/STP (PIMF) 2,10 €

So the difference was so small that I was changing all my patch cables to CAT.6a S/STP (PIMF)
perhaps not from a premium cable vendor but better then the lazy UTP ones.

But i'm still hoping for an idea to properly test the LAN ports…

Be sure they are not on "auto" to surround a miss match
use iPerf or NetIO from one to another PC (client & server)
Use proper shielded cables and or a "LAN tester" for testing out also the cables.

bluepr0

Just got a question related with the Atom C2758 so I thought about using this thread instead of starting a new one

These Atom Rangeley are starting to get a bit "old" so I was looking for other ideas or newer hardware. Found out that the new Xeon E3-1240Lv5 has a TDP of only 25w (5w more than the C2758). Also, it seems that on benchmarks doubles in performance the Atom (of course, not networking related task but it might help to get an overall idea).

What do the experts think about going with a "normal" server board (not so expensive, 1 IPMI, 2 LAN Intel) + this E3-1240LV5?

Prices in Spain:

Supermicro A1SRi-2558F = 440€
Total = 440€
Xeon E3-1240LV5 = 330€ (It has AES, couldn't find if it has QuickAssist)
Server board (Asrock, Gigabyte) = 250€
Total = 580€

It's a bit more expensive but also more powerful with only 5w more of TDP.

I'm genuinely asking your opinion, there's probably a lot that I'm missing. Also on a normal server board you would get "normal" server grade LAN, like the i211 or i210… while on the Supermicro you get the i350 (wondering how much of a difference this makes in practice)

Thanks!

jwt

The Xeon E3-1240LVS does not have QuickAssist, but you can add it via PCIe. Example: http://store.netgate.com/ADI/QuickAssist8955.aspx

I think you want to compare apples to apples, so you'll need 4 x 1GbE on-board.

Supermicro X11SSH-CTF will run over $400, but has dual 10G on-board.
Supermicro X11SSH-LN4F appears to be around $220 online, and has 4 x i210.
I'm seeing E3-1240LV5 at between $290 and $320 online. Call it $300 to split the difference.

So that's $520, plus ram and an enclosure for 4 x i210 and your CPU (that you probably don't need).

I don't think the Rangeley is getting a bit old. I think we've only begun to explore the acceleration potential in the SoC.

The i350 has more queues (8 per port) than the i211 (up to 2) or i210 (up to 4). We don't do a lot with RSS (yet), but it's high on the list now, and when we do, you're going to want a queue per core.

ldean

Hey Engineer,
First just wanted to say thanks for the time/effort you've put into getting this system working. We are also in the process of rolling out this system as a gateway (although not using pfsense), and also experienced the same issue with watchdog timeouts occurring on the LAN ports (the ones that go through the Pericom608GP chip). We have 2 of them in production (both exhibit this behavior) and 10 on backorder right now, so we have a vested interest in finding out/resolving what's causing this issue. I have been in contact with supermicro as well, and they asked me to RMA my board. I would like to be able to reference the case you opened with them in an effort to determine what fixes were done on your board that resolved the problem for you without a giant duplication of effort, and obviously we don't want to have to send every board we purchase to SM for repair :). Do you have a case number available?

Also, what firmware version is running on the board you got back from SM? I saw they have recently released revision 1.0a (no date mentioned but it wasn't there previously the last time I checked). I wonder if that's related or not. I installed it on the box that I have in my lab, and haven't been able to replicate the issue, but this box hadn't experienced the issue previously yet (maybe because it's not actually in production), and as you found, the issue seems to be rather sporadic and could be a few days before it occurs.

Thanks again!!

Engineer

Case# SM1511127317,

Hi, I saw the firmware last night but have not updated. No change log either. Ken Huang is the guy who handled my case. Keep me up to date if you don't mind! Thanks