X11SBA-LN4F vs A1SRi-2558F

Engineer

Thanks David and from what little I've learned, I have about figured out what you posted above (not how to do it, just the general idea). I learned about some of it by compiling the newer Intel driver using standard stock FreeBSD 10.1 - missing ALTQ for Traffic Shaping (still had a watchdog though).

I don't think I have a hardware problem. I think something is broken at the kernel or driver level. I'm just out of ideas and not experienced enough (nor do I have time) to solve it past what I've already done. If 2.2.5 doesn't fix it, I'll have to try something else or go back to previous Asus router and simply wait to see if it ever gets solved. That's always a risk when you choose new hardware. Regardless, I thank everyone for helping and those that put this whole thing together and keep it moving forward.

Edit: Updated to 2.2.5 and will see how that goes.

Jailer

Any luck with 2.2.5?

Engineer

@Jailer:

Any luck with 2.2.5?

Update was FAST and smooth (i.e. no issues). Been up for a day+ now with no issues (again, I don't run much yet as I'm waiting for stability first). It made it to 4+ days before going down last time (2.2.4), so time will tell. The driver for the NIC is, as expected, Intel 2.4.0 (FreeBSD version - won't probably change until FreeBSD 11 - but 2.4.3 compiled on FreeBSD 10.1 didn't fix it anway).

I'll keep you updated.

Engineer

Well, went down on 2.2.5 but never received a watchdog timeout (waited nearly 5 minutes). Console worked fine and ifconfig shows both ports as active. Don't know where to go from here except back to the Asus router. I don't think it's a hardware issue (unless multiple ports have bad chipsets). Either a BIOS issue or FreeBSD/driver/issue (possibly pfsense but I suspect FreeBSD/driver).

sigh

David_W

Have you tried disabling MSIX and MSI? Instructions at https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards#MSI.2FMSIX

Engineer

@David_W:

Have you tried disabling MSIX and MSI? Instructions at https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards#MSI.2FMSIX

I tried that once before but it locked the system (tried to disable both at same time). I'll try again to see if I it makes a difference. Will have to wait as I have a house full of people - all using the web continuously. I believe I would know about the Internet going down faster (with all of the crying about it) than I would know about someone breaking into my house and shooting up the place, lol. :P

From dmesg, it would appear that the ports are using MSIX. Will disable that first and see what happens.

Edit: msix disabled now (confirmed via dmesg). Will report back. If it goes down again, will try to disable both.

Engineer

Disabling MSIX didn't fix it. Just went down again.

When I get a chance, I'll try to disable both again.

:-[

Edit: Just disabled both MSIX and MSI and the LAN went down. I logged in via IPMI and the console indicated all was fine (ifconfig - lists all as active). Tried pinging from the shell and couldn't reach anything on the LAN. Could ping the cable modem on the WAN port just fine.

Something is borking the LAN…what....I have no idea?!?

Thimee

I have an X10SBA-L build:

SSD: 120 GB Crucial BX100
Case + PSU Combo: M350 and 90W PicoPSU
RAM: 2x4GB Crucial

I had a couple of problems that might be worth mentioning:

My first build had a faulty PSU/12V adaptor (wavy lines on VGA output, buzzing noise). Wouldn't stay up for more than 30 seconds, I replaced both, after which no more PSU issues. I've ordered a 12V straight cable (in theory no PicoPSU necessary) but haven't gotten around to installing it yet.
I had lots of trouble getting the exact RAM, at first I ordered some 1.35V/1.5V from Amazon, which passed Memtest for about 15 minutes until it failed. Bizarre. The replacements I got from Microcenter (didn't want to wait for shipping) work perfectly.
I added a fancy Swellder Delta TFB0412EHN PWM fan - sounds like a rocket ship on startup but afterwards the fan never turns on, system barely gets warm.

It was a bit of a dog to install 2.2.4, I had endless crashes and even after successful install it wouldn't stay up for more than a few hours before corrupting the file system. However after running the exhaustive Memtest I saw one failure, changing the RAM out with a slightly different model fixed all my issues.

Since then I had 51 days of uptime (I use pfsense on my home netrowk for firewall, VPN server, DNS server), reset this morning to upgrade to pfsense 2.2.5 which appears to be running just great.

2c

Thimee

My X10SBA-L build use since initial startup, I have a 150MB Comcast connection, about 10-15 concurrently connected devices:

Basically total overkill for my network…

Engineer

Thimee, did your entire system crash or did your interfaces go down (and console still work)?

My console still works fine…it's just the LAN (not WAN) port that goes down and resets via Watchdog timeout.

rexki

Engineer: Doing some double checking on SuperMicro web site:

1. Still no BIOS update
2. Certified 4GB ram is only one item listed: 4GB – Hynix - MEM-DR340L-HL02-SO16 - HMT451S6BFR8A-PB - H5TC4G83BFR

Specs above further detailed:
• 4GB Memory Module
• DDR3-1600MHz
• PC3L-12800
• Non-ECC
• 1.35v Low Voltage LV
• SODIMM
• 204-pin
• Cas Latency 11
• 1Rx8 (H5TC4G83BFR - 512MB chips each)

3. Certified 8GB stick has the same exact component chip as the 4GB - H5TC4G83BFR - 512MB chips each

Your spec from page one listed as follows:
2 x 4GB Samsung PC1600 DDR3L: $36 shipped from eBay

Found Samsung 4gb pc1600 sodimm items to get comparatives:
1. Samsung Part number M471B5273DH0-CK0 - 1.50v
2. Samsung Part number M471B5173DBO-YK0 - 1.35v - CL11 - ?Single sided 1Rx4?
3. Samsung Part number M471B5173BHO-YK0 - 1.35v - CL11 - ?Single sided 1Rx4?
4. Samsung MV-3T4G3D/US - 1.35v - CL11 - 2Rx4 Double sided - 8 each

My questions:
1. Do these certified specs match exactly to yours especially 1Rx8 with 512 and CAS 11?
2. Have you run with just one chip of ram at 4GB - number one cycles solo and then number two cycled solo as well to isolate behavior?
3. Have you swapped memory places of the existing Samsung 2 *4GB sticks?

Engineer

I have re-seated the ram but not swapped it or run it in single channel (one stick). Once I completed the memtest+ test, I had assumed all was good. I suppose it could be ram but I tend to think that takes the whole ship down including the console (aka - one big crash). I suppose it might be ram. The ram is DDR3L 1.35V (I did make sure of that when I purchased it).

I'll see about removing a stick later. As for the certified memory stuff, I understand the testing and stuff but there's no real reason for most of the good stuff not to run. Just hasn't been certified….YET. When I bought the board and ram, there were NO certified sticks listed on the SuperMicro site. ;)

Oh, and for now, I've turned the Asus RT-AC68U back from AP to Router while trying to sort this out. Busy lately so I'll have to put it on hold a little. Hard to even do anything as people scream faster about the Internet being down than they do if we were being robbed, lol.

rexki

This is extremely interesting to me - the LAN1 LAN2 LAN3 and LAN4 ports on the back are not the same!!

Manual from SuperMicro - page 17 block diagram - shows:

A. 1x intel I210 is a single off the SoC on PCIe(1) and
B. the other three are 2 stepped off PCIe(2) then to a Pericom 608GP “PI7C9X2G608GP” (PCIe2 6-Port/ 8-Lane Packet Switch, GreenPacketTM Family) – add 150ns latency to stream per spec.

My thoughts would be to move the LAN port if on the three port Pericom switch to PCIe(1) directly off the SoC. Guessing this is lower left corner LAN1 while looking from back side or page 26 of the manual.

rexki

Going through the BIOS option in the manual some points worth noting to the problemamtic I210:

1. ASPM "Active State Power Management" - Auto default to disable (Applicable to the PCI-Express buses - ala LAN1,2,3,4
2. PCIe Speed from Auto to Gen2 since this is the spec for the I210 on PCIe(1) and (2).
3. Onboard LAN Option ROM Type - make sure Legacy which is default.
4. Onboard LAN1 option ROM from PXE - to Disabled - ironically the rest are off by default - my guess because of the Pericom.
5. Network Stack from Enabled to Disabled - no PXE!
6. Kill the AST2400 serial port since it will not be used?
7. last resort play with turning off and on WatchDog in BIOS

Engineer

@rexki:

This is extremely interesting to me - the LAN1 LAN2 LAN3 and LAN4 ports on the back are not the same!!

Manual from SuperMicro - page 17 block diagram - shows:

A. 1x intel I210 is a single off the SoC on PCIe(1) and
B. the other three are 2 stepped off PCIe(2) then to a Pericom 608GP “PI7C9X2G608GP” (PCIe2 6-Port/ 8-Lane Packet Switch, GreenPacketTM Family) – add 150ns latency to stream per spec.

My thoughts would be to move the LAN port if on the three port Pericom switch to PCIe(1) directly off the SoC. Guessing this is lower left corner LAN1 while looking from back side or page 26 of the manual.

1st - I've never been able to get a manual as it has not been available. Will check that out.

2nd - Very interesting indeed. Maybe that's why the WAN port stays up and the LAN ports go down.

Think I'm going to switch the LAN and WAN ports for giggles and see if the LAN port goes down on igb0.

Edit: Downloaded the manual (first time that it's been available online - thanks for the heads up) and switched the LAN and WAN ports. If it stays up, then I'm not sure where to go. It could be a hardware issue (with the PCIe lane switch on ports 2, 3 and 4) or it could be a software issue (not certified on FreeBSD 10 according to SuperMicro documents on page). Could also be a BIOS issue. If it has something to do with the three (3) ports and the internal PCIe switch, I would think the WAN port would start dropping and the LAN would stay up. I'll report back as time goes on.

Engineer

Well, I swapped the ports and the WAN went down tonight. Watchdog Timeout.

I think rexki might have found something in the PCIe switch on ports 2-4 (igb1-3). Port 1, not on the PCIe internal motherboard switch, functions normally. The others drop with timeouts.

Maybe it's a driver or FreeBSD issue, but since this board isn't FreeBSD certified by SuperMicro, I'm not sure they will even take a look at it.

Edit: I've opened a technical support case with SuperMicro to see if they can come up with something on this. Would be nice to have someone else with this board and confirm (or deny) if they have the same issues. If not, could see what they have different (power supply, cooling (or lack of as I have now), etc). I might put a USB powered fan blowing on the board to see if it makes a difference. The PCIe switch chip is only rated at 1.2W max and has no heatsink or other cooling.

Edit #2: SuperMicro came back with the suggestion to install a certain version of Ubantu on it and try?!?! WTF kind of suggestion is that, lol? I told him that was a no go unless it's just to run some sort of test program. Starting to wonder if the chip is either defective or getting hot (since no cooling - passive). I have lots of small video card heatsinks and a roll of very good heatsink tape to use if necessary. And the quest goes on….

Engineer

Guys and gals, I'm giving up for now. Went down again and I don't have the time or energy right now to try to fight this. Can't get much from SuperMicro and don't know if it's a hardware issue with the board or if it's a FreeBSD issue (since it does it on both pfsense and opnsense). If anyone else has this board and gets better results, let me know so I can then say it's hardware.

Jailer

I'd love to pick this board up and give it a try but I'm holding back based on your experience with it. If something doesn't change in the near future or someone else comes along and has success I'll likely be getting the X10SBA-L for my build.

Engineer

@Jailer:

I'd love to pick this board up and give it a try but I'm holding back based on your experience with it. If something doesn't change in the near future or someone else comes along and has success I'll likely be getting the X10SBA-L for my build.

Really sorry Jailer. I've left the LAN port connected to my network (change IP and turned off DHCP) and am hammering it with ping requests right now. I don't recall it going down when I was doing a burn in test for 8 days with NO WAN connection (i.e. no data flowing through the system). Maybe just needs a system driver update in FreeBSD since this is a VERY new system. I hate to spend the time/money to RMA it only to find out it's the OS/driver and not the hardware. I do have a five year warranty though so … :)

jwt

The i210 NICs only have 4 rx/tx queues, which is fine for the 4 core SoC (http://ark.intel.com/products/87261/Intel-Pentium-Processor-N3700-2M-Cache-up-to-2_40-GHz), but you'll find that future versions of pfSense have a minimum 4 core requirement (I might make it 8, I've not decided.)

As documented here: http://www.intel.com/content/dam/www/public/us/en/documents/datasheets/i210-ethernet-controller-datasheet.pdf , there are only 4 tx and 4 rx queues on an i210.

The SoC is significantly slower than a 4 core Rangeley (1.6GHz on the N3700, 2.4Ghz on the C2558), and this will translate into real-world performance differences. Someone pointed out 6W .vs 15W, and this is why.

Rangeley also has better (i350 .vs i210) NICs. https://twitter.com/gonzopancho/statuses/643443335114424320

I also don't believe in integrated graphics on a standalone networking device.