X11SBA-LN4F vs A1SRi-2558F

bluepr0

I ran again the tests, using top command to see the CPU usage (it seems the CPU usage on web interface is also accurate, making an average of all the cores or am I wrong?)

Here's the screenshots

1. This is using all my download bandwidth from the internet http://d.pr/i/1h6X9/3UOn6rvy
CPU usage is around 30% here

2. This is using all my download bandwidth from internet + iperf (maxing out gigabit) http://d.pr/i/1bL1T/5PWn64bR
CPU usage is around 84% here

What do you think?

Thanks!

Jailer

Looks like Squid is what is eating up a lot of your resources @bluepr0.

Engineer, thanks for the update. I was going to ping you and see how things were going so your response was perfect timing.

bluepr0

Yep! stopping snort, ntop reduces quite a bit the CPU usage. See http://d.pr/i/1eYwd/NinxWFhR (usage is around 50%)

Engineer

@Jailer:

Looks like Squid is what is eating up a lot of your resources @bluepr0.

Engineer, thanks for the update. I was going to ping you and see how things were going so your response was perfect timing.

Right at 7 days now. Never made it above 4.5 or so previously. :)

bluepr0

great to know! I'm wondering what they change on the board? does the other boards out there will only need a BIOS update or it's a hardware problem?

Engineer

@bluepr0:

great to know! I'm wondering what they change on the board? does the other boards out there will only need a BIOS update or it's a hardware problem?

@bluepr0, I honestly don't know. SuperMicro said it was a "hardware modification" but they will not disclose what was changed and I cannot see any physical changes to my board. When I asked whether other boards are affected or only mine, I was told that if there are any other boards out there with this problem, they will fix them. Sorry guys, I wish I could give you more but SuperMicro won't let it out. I have not seen a new BIOS other than a new IPMI firmware.

reilos

So i decided to go with the A1SRi-2558F board (and ECC RAM). Connected LAN1 to WAN (ISP router in bridge mode, connected through surge-protected RJ45 ports on the UPS), connected other 3 LAN ports and IPMI to my (managed) switch.
Installing pfSense onto the SSD through IPMI mounted ISO… Sudden reboot... 2nd try... Install went fine, but reboot after a few minutes. My first thought is was something with the pfSense install. But....
Went into BIOS settings and before i could even make any changes... Reboot. So it's NOT a pfSense issue.
Disconnected all network cables except IPMI, restored default BIOS settings, did 10 cycles of memtest86 (took almost 3 days)... No errors and no reboots.
Then I connected LAN1 to my network, configured it as the WAN port and connected an AP to LAN2 (configured as LAN). Booted pfSense, did nothing for about an hour and no reboots.

So, I am now thinking it's a problem with 1 or more of the LAN ports. I want to do some testing, maybe anyone has some idea on how to do effective / efficiënt tests on the LAN ports?

jahonix

Do you use STP or UTP cables?

reilos

@jahonix:

Do you use STP or UTP cables?

I use all CAT6 U/UTP cables. For the whole network, including test setup.

jahonix

This eliminates at least a possible hum/ground loop, well known to audio guys (me) and lesser known in the network world.
Just for the record: I do not say to only use UTP cables. It just rules out an idea I had in this case.

reilos

@jahonix:

This eliminates at least a possible hum/ground loop, well known to audio guys (me) and lesser known in the network world.
Just for the record: I do not say to only use UTP cables. It just rules out an idea I had in this case.

Yeah, know what that is. In my setup i don't really need shielded cables (and i also dont want to go through the hassle of properly grounding the whole thing).
But i'm still hoping for an idea to properly test the LAN ports…

Guest

Yeah, know what that is. In my setup i don't really need shielded cables (and i also dont want to go through the hassle of properly grounding the whole thing).

I really don´t know from where you all are and what you have to pay for network cables, but here in Germany
I have to pay the following money for each;

1 meter patch cable CAT.5e UTP 1,80 €
1 meter patch cable CAT.6a S/STP (PIMF) 2,10 €

So the difference was so small that I was changing all my patch cables to CAT.6a S/STP (PIMF)
perhaps not from a premium cable vendor but better then the lazy UTP ones.

But i'm still hoping for an idea to properly test the LAN ports…

Be sure they are not on "auto" to surround a miss match
use iPerf or NetIO from one to another PC (client & server)
Use proper shielded cables and or a "LAN tester" for testing out also the cables.

bluepr0

Just got a question related with the Atom C2758 so I thought about using this thread instead of starting a new one

These Atom Rangeley are starting to get a bit "old" so I was looking for other ideas or newer hardware. Found out that the new Xeon E3-1240Lv5 has a TDP of only 25w (5w more than the C2758). Also, it seems that on benchmarks doubles in performance the Atom (of course, not networking related task but it might help to get an overall idea).

What do the experts think about going with a "normal" server board (not so expensive, 1 IPMI, 2 LAN Intel) + this E3-1240LV5?

Prices in Spain:

Supermicro A1SRi-2558F = 440€
Total = 440€
Xeon E3-1240LV5 = 330€ (It has AES, couldn't find if it has QuickAssist)
Server board (Asrock, Gigabyte) = 250€
Total = 580€

It's a bit more expensive but also more powerful with only 5w more of TDP.

I'm genuinely asking your opinion, there's probably a lot that I'm missing. Also on a normal server board you would get "normal" server grade LAN, like the i211 or i210… while on the Supermicro you get the i350 (wondering how much of a difference this makes in practice)

Thanks!

jwt

The Xeon E3-1240LVS does not have QuickAssist, but you can add it via PCIe. Example: http://store.netgate.com/ADI/QuickAssist8955.aspx

I think you want to compare apples to apples, so you'll need 4 x 1GbE on-board.

Supermicro X11SSH-CTF will run over $400, but has dual 10G on-board.
Supermicro X11SSH-LN4F appears to be around $220 online, and has 4 x i210.
I'm seeing E3-1240LV5 at between $290 and $320 online. Call it $300 to split the difference.

So that's $520, plus ram and an enclosure for 4 x i210 and your CPU (that you probably don't need).

I don't think the Rangeley is getting a bit old. I think we've only begun to explore the acceleration potential in the SoC.

The i350 has more queues (8 per port) than the i211 (up to 2) or i210 (up to 4). We don't do a lot with RSS (yet), but it's high on the list now, and when we do, you're going to want a queue per core.

ldean

Hey Engineer,
First just wanted to say thanks for the time/effort you've put into getting this system working. We are also in the process of rolling out this system as a gateway (although not using pfsense), and also experienced the same issue with watchdog timeouts occurring on the LAN ports (the ones that go through the Pericom608GP chip). We have 2 of them in production (both exhibit this behavior) and 10 on backorder right now, so we have a vested interest in finding out/resolving what's causing this issue. I have been in contact with supermicro as well, and they asked me to RMA my board. I would like to be able to reference the case you opened with them in an effort to determine what fixes were done on your board that resolved the problem for you without a giant duplication of effort, and obviously we don't want to have to send every board we purchase to SM for repair :). Do you have a case number available?

Also, what firmware version is running on the board you got back from SM? I saw they have recently released revision 1.0a (no date mentioned but it wasn't there previously the last time I checked). I wonder if that's related or not. I installed it on the box that I have in my lab, and haven't been able to replicate the issue, but this box hadn't experienced the issue previously yet (maybe because it's not actually in production), and as you found, the issue seems to be rather sporadic and could be a few days before it occurs.

Thanks again!!

Engineer

Case# SM1511127317,

Hi, I saw the firmware last night but have not updated. No change log either. Ken Huang is the guy who handled my case. Keep me up to date if you don't mind! Thanks

ldean

@Engineer:

Case# SM1511127317,

Hi, I saw the firmware last night but have not updated. No change log either. Ken Huang is the guy who handled my case. Keep me up to date if you don't mind! Thanks

Thanks!! Ken is handling my case as well. I pointed him to this thread, and summarized, but he is still requesting I RMA one of the boards. I'm going through that process now, hopefully we can determine the root cause and best way to proceed going forward!

ldean

Just a small update –
our plan was to swap a new box in for the one we are experiencing the most issues with, and RMA that one. We swapped the hardware last night and the new box seems to be even worse. We saw about 4 watchdog timeouts last night, and today, the NIC stopped functioning, but did not trigger a timeout. So it just remained indefinitely in a non-operational state, and even ifconfig igb3 down/up, and service netif restart igb3 had no effect. A reboot was the only way to restore connectivity to our LAN. During this time, both the server and the switch reported an active physical link. And this server is running the 1.0a firmware, so that obviously didn't fix anything. We might have to revert to a different device until we get this straightened out with supermicro.

Engineer

@Idean,

Thanks for the update. I suspect that SM will issue a hardware revision on this board very soon. What Hardware Version is on the replacement board that you received? 1.01 (which is what my board is)?

ldean

I haven't received a replacement from SM yet, I'm just shipping out one of the bad boards this morning. The boards I have are revision 1.01 already.