X11SBA-LN4F vs A1SRi-2558F

rexki

"On the IPMI note, it's separate and I don't think pfsense/FreeBSD has anything to do with it on my board, correct? I've had no issues getting to the console (even during the LAN crash / watchdog timeout) using the IPMI port."

I am not sure since the bug report you mentioned as well as my experience with all SuperMicro boards (Q2 mentioned) all have done simultaneous lockups on LANS and BMC. You are correct in that I erroneously thought your BMC locked up simultaneously - my error - sorry about the rant on BMC.

How do you get Watch Dog to tell you the LAN has frozen?

I went to my "Service WatchDog" on pfSense and only have "ipferf" and that seems to be recommended only for internal trusted LAN ports?

Engineer

I don't know how to change the watchdog, it simply popped up on my console stating watchdog timeout on igb1, which is my LAN port. Also, I couldn't get to the GUI, DHCP stopped working as well as ANYTHING Internet. However, the IPMI port, separate from the 4 regular ports, was working fine so I logged into the console and noted it was completely responsive. After the watchdog message popped up, everything reset and started working again except IPV6 sites. A watchdog error for igb1 was also logged in the system log.

I suppose IPV6 would have eventually start working but I turned it off. I've since turned it back on.

Jailer

Any updates on the stability?

Engineer

@Jailer:

Any updates on the stability?

I tried to install 2.1.5 but gave up because I couldn't get past the Root Mount Error. In the process, I somehow managed to (through BIOS) disable all of my USB ports and, for some reason, not only could I not use a USB keyboard, my IPMI port using the virtual KVM would not work (could see but could not type). I had to clear BIOS to get past that.

Once cleared and setup again, I just let it run while I was searching for answers. It has now run for 3 days with no watchdog timeouts / resets and the log seems to have fewer entries (on everything). As far as I know, the pfsense config did not change (unless the aborted 2.1.5 did something but I never made to the install part).

So running solid for 3 days but that could change at any moment. Still don't know how to get past the Root Mount Error of trying 2.1.5 though! :(

My theory (based on lots and lots and lots of reading) is that there might be an issue with the 2.4.0 IGB Intel driver that's included in FreeBSD 10.1 (and 10.2 from what I can tell). There are updated drivers (2.4.3 being the latest) but I've not tried to figure out how to compile (can't compile in pfsense as that's been removed). I have seen cases exactly like mine be solved by simply putting the older FreeBSD 8.3 Intel IGB driver (not sure of version) into this pfsense and set it to load. Was hoping 2.2.5 would have an updated driver but from talking to the folks over on the opnsense board, they think that the network drivers won't be touched much until FreeBSD 11 of mid next year.

jjduru

@Engineer,

What I don't get is that the current "Supported Devices" list on freebsd website has i210 as a supported device.

Why don't you give it a shot to install the latest stable pfsense and disable watchdog right after install?

I'm more and more inclined to get the 2558F, Thanksgiving is right around the corner.

Engineer

@jjduru:

@Engineer,

What I don't get is that the current "Supported Devices" list on freebsd website has i210 as a supported device.

Why don't you give it a shot to install the latest stable pfsense and disable watchdog right after install?

I'm more and more inclined to get the 2558F, Thanksgiving is right around the corner.

I'm on the latest pfsense (2.2.4) and not sure how to disable watchdog. The watchdog is resetting the dropped lan. Without it, it would hang indefinitely.

While it may be supported on freebsd, reading around (FreeBSD forums, FreeNAS forums, here), there seems to be a sparse pattern of the driver from FreeBSD 10.1 for the Intel IGB (2.4.0) dropping under load. I'm at 3+ days now, the longest that it's been up. Maybe there was something in BIOS that did need to be reset and resetting it fixed the issue. If it drops again, I'm going to either figure out how to install pfsense 2.1.5 or I'm going to figure out how to swap out the Intel igb driver for the one from FreeBSD 8.3 or I'll compile the latest (2.4.3) and try it.

Guest

I'm on the latest pfsense (2.2.4) and not sure how to disable watchdog. The watchdog is resetting the dropped lan. Without it, it would hang indefinitely.

dropping packets is mostly based on a small few number of issues that might be coming temporarily
again and again back, as I know it right.

A bridged port is in usage and the hardware is not f´n fast and strong enough
The entire traffic is to much for the RJ45 interface, because the NIC or the other hardware is not capable
of handling such a amount of traffic right.
Too much more packets are installed and too much services are also running or working on the same
LAN Port. IDS/IPS, DPI, or other heavy tasks that narrows down the entire CPU power.

While it may be supported on freebsd, reading around (FreeBSD forums, FreeNAS forums, here), there seems to be a sparse pattern of the driver from FreeBSD 10.1 for the Intel IGB (2.4.0) dropping under load.

In normal this can´t be, because the i211AT is the consumer LAN Port and the i210AT is the Server grade
LAN Port series from Intel!

I'm at 3+ days now, the longest that it's been up. Maybe there was something in BIOS that did need to be reset and resetting it fixed the issue.

Did you a fresh and full install? Or is this NanoBSD on the firewall?

If it drops again, I'm going to either figure out how to install pfsense 2.1.5 or I'm going to figure out how to swap out the Intel igb driver for the one from FreeBSD 8.3 or I'll compile the latest (2.4.3) and try it.

There are several methods available for you but likes often they can´t be mixed up!
pfSense 2.1.5 = FreeBSD 8.3 and only between this drivers, .ko modules should be swapped over
pfSense 2.2.4 = FreeBSD 10.1 the same game only between them…

setting up a higher mbuf size because each core is creating a queue for each CPU core,
4 Core x 5 LAN Ports = 20 queues

And in the BIOS not ever but sometimes the IPMI port is created or set up to be a fall back WAN port
and this might be also often a problem, have you tried this out before?

Engineer

It usually hangs at 4 or 5 Megabits so I doubt that traffic load is hurting this hardware or CPU. I'm only using two of the four ports right now with the other two unassigned. One port is the WAN port and the other is the LAN ports. Nothing else in the system.

As for the FreeBSD 8.3 vs 10.1, there are others that have moved the Intel ibg driver from 8.3 into 10.1, set it to load and has corrected the same issue that I have. Not sure what to say here other than it worked for them. Regardless if the I210-AT is server grade or not, a bad driver can cause issues.

Not sure what you mean about IPMI port.

The install was a fresh full install. There were the watchdog timeout / resets until 3+ days ago when I had to reset the BIOS on the motherboard because I had lost all keyboard control by turning off something to do with the USB ports (because I was trying to install 2.1.5 FULL install).

Engineer

Two more crashes this morning, one with no traffic load at 4:50 am. Will work on it later to see if I can solve this mess…sigh

Edit: Spent the better part of Saturday learning how to compile the latest Intel driver (2.4.3) for igb ports on FreeBSD (using FreeBSD 10.1 running in a VM). Finally got it installed and working but it broke Traffic Shaping (because I didn't have ALTQ support in the driver source). Regardless, it did a watchdog timeout within 6 hours of installation.

Edit #2: Put the original driver back and connected the LAN port to another switch. It ran for over a week with nothing in the long on a 'dumb switch' so it's worth a shot to see if it's the 'smart switch' or the LAN port hardware / driver. If that doesn't work, I'll switch to an un-used port on the motherboard (surely, couldn't be two ports malfunctioning if it were hardware, right? - don't answer.....:P )

Edit #3: Moved the connection on the LAN to another switch and it just went down for the count again. Will move it back to the original switch and then move the LAN from igb1 to igb2 to see if there is a hardware issue with port 1 (igb1) of the board.

Engineer

The igb2 port did a watchdog and reset also. Unless I'm mistaken and the ports run off the same chip (don't think so), this has got to be a software or configuration error. :(

Edit: Removed VLANHWTSO (ifconfig igb2 -vlanhwtso) and turned off apinger (WAN Monitoring). Cut 95% of the log out and things are running OK so far…will see if this is the magic pill that fixes this thing.

Jailer

Watching this thread with anticipation. This looks like a good board on paper and will be on my short list for a build if you can get it to run.

Engineer

@Jailer:

Watching this thread with anticipation. This looks like a good board on paper and will be on my short list for a build if you can get it to run.

I'm doing all that I can right now. Never expected in my life to learn how to compile an Intel NIC driver in FreeBSD 10.1 just to try to get pfsense to run, lol. The only thing I've not been able to get going was 2.1.5 and I'll go back to trying that if it's not stable this time.

Engineer

Update: Two+ days of uptime since I made the last changes. About 20 entries TOTAL in the GENERAL logs and 95% of those are login / logout entries.

Keeping fingers crossed that either VLAN_HWTSO removal or apinger removal fixed this. Will probably turn one or other back on if this things runs for some time (weeks) just to figure out which one was taking down the LAN - time will tell….

Update 2: Three+ days. Only thing in logs other than login / logout entries is a Dynamic DNS check (no update required).

Engineer

Update: At 4 plus days now with no watchdog. Again, only login/logout entries in the system log now.

Also, since turning off apinger, have not lost IPV6 connectivity.

Edit: Well shit….down again. Made it 4 days this time. I'm out of ideas at this point....other than 2.1.5, which I can't get to install (Root Mount Error). I will try later (2.1.5 or 2.2.5).

David_W

If your IPv6 Internet connection uses DHCP6, SLAAC and/or DHCP-PD over PPPoE, 2.2.5 correctly handles link up -> link down and link down -> link up scenarios. 2.2.4 and earlier versions simply trust that IPv6 will start working again exactly as it did before when the link returns, which is not necessarily true. My ISP doesn't install a necessary route until DHCP-PD has delegated a prefix.

If this fix is relevant to you, it won't do anything to stabilise your troublesome NICs, might help things recover properly after the interface's watchdog reset.

pfSense uses a custom kernel. If you want to experiment with kernel patches (including replacing the NIC driver entirely), you really need to be using a pfSense build environment to do so in order to avoid random breakage and loss of functionality. pfSense 2.2.5-RELEASE has 111 patches applied to the stock FreeBSD 10.1 operating system, many of them affecting the kernel in some way. The kernel is also configured differently to the FreeBSD GENERIC kernel, for example by including ALTQ support. Sadly, setting up a pfSense 2.2.x build environment is not that straightforward, especially if you have no previous experience with FreeBSD and git (the version control system used by pfSense).

Backporting fixes from one FreeBSD branch to another (for example from HEAD, which is currently FreeBSD 11, to releng/10.1, which is the base OS for pfSense 2.2) is often non-trivial and may well be difficult without some experience with Subversion (the version control system used by FreeBSD). You will struggle to manage more than the most straightforward backport without experience of programming in C, diff and patch.

I suspect your ROOT MOUNT ERROR in 2.1.5 is because FreeBSD 8.3 is too old to support the controller for the device you booted from (the USB controller if you booted from a memory stick). I would give up on 2.1.5 - pfSense 2.1 is End of Life, FreeBSD 8 is End of Life and FreeBSD ports for FreeBSD 8 are End of Life. There are almost certainly unfixed security issues in both pfSense and FreeBSD lurking within 2.1. Snort for pfSense 2.1 has been pulled, as VRT rule updates for the last version of Snort that was available for FreeBSD 8 have been discontinued.

Engineer

Thanks David and from what little I've learned, I have about figured out what you posted above (not how to do it, just the general idea). I learned about some of it by compiling the newer Intel driver using standard stock FreeBSD 10.1 - missing ALTQ for Traffic Shaping (still had a watchdog though).

I don't think I have a hardware problem. I think something is broken at the kernel or driver level. I'm just out of ideas and not experienced enough (nor do I have time) to solve it past what I've already done. If 2.2.5 doesn't fix it, I'll have to try something else or go back to previous Asus router and simply wait to see if it ever gets solved. That's always a risk when you choose new hardware. Regardless, I thank everyone for helping and those that put this whole thing together and keep it moving forward.

Edit: Updated to 2.2.5 and will see how that goes.

Jailer

Any luck with 2.2.5?

Engineer

@Jailer:

Any luck with 2.2.5?

Update was FAST and smooth (i.e. no issues). Been up for a day+ now with no issues (again, I don't run much yet as I'm waiting for stability first). It made it to 4+ days before going down last time (2.2.4), so time will tell. The driver for the NIC is, as expected, Intel 2.4.0 (FreeBSD version - won't probably change until FreeBSD 11 - but 2.4.3 compiled on FreeBSD 10.1 didn't fix it anway).

I'll keep you updated.

Engineer

Well, went down on 2.2.5 but never received a watchdog timeout (waited nearly 5 minutes). Console worked fine and ifconfig shows both ports as active. Don't know where to go from here except back to the Asus router. I don't think it's a hardware issue (unless multiple ports have bad chipsets). Either a BIOS issue or FreeBSD/driver/issue (possibly pfsense but I suspect FreeBSD/driver).

sigh

David_W

Have you tried disabling MSIX and MSI? Instructions at https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards#MSI.2FMSIX