Firebox Marvel ports locking up (CORE-E SERIES)

[deanot]

Now you mention it, it does happen with high traffic, my wife watches a lot of online tv, normal to do it then.  Especially if I log into the box using the GUI, soon as I do that BOOM!.

[Engineer]

Tried the MXI / MSIX disable and it wouldn't even let me to the LAN / Internet after a reboot.  Edited those back out via console / Shell, rebooted and it ran for 5 minutes before lockup.  Just downloaded 2.1.5 and will try it as soon as I can get people off of Netflix long enough to get it started.

[deanot]

@Engineer:

Tried the MXI / MSIX disable and it wouldn't even let me to the LAN / Internet after a reboot.  Edited those back out via console / Shell, rebooted and it ran for 5 minutes before lockup.  Just downloaded 2.1.5 and will try it as soon as I can get people off of Netflix long enough to get it started.

Interesting, I did the MXI / MSIX fix on mine, it worked ok on a reboot.

[Engineer]

@corvey:

2.1.5 works.

It might if I could get past the ROOT MOUNT ERROR on the Live CD.  I guess I'll try the memstick version.

Edit:  I've given up for tonight.  Can't get past the ROOT MOUNT ERROR on the LiveCD version nor the Memstick version.  Have followed pages of advice and just can't get past it.  Might try later….sigh

Edit #2:  Never got it to work and will try later.  Did find a link that pretty much states the same thing for my card as the em card (PCI vs PCIe I think).  https://lists.freebsd.org/pipermail/freebsd-net/2014-January/037694.html . This would seem to be an issue with ANY card with TSO, no matter the driver.

[stephenw10]

Not seen that at all on any of the fireboxes I have here.
Even on the msk ports, with msi/msix disabled they are stable. I've never seen an issue with the sk interfaces.
The first thing I would do is disable hardware checksum offload if you still have that on. I did see that break dhcp relay inexplicably while everything else worked fine.

Many of those boxes have seen a lot of hours in hot conditions. If you have turned the fans down they might be running even hotter. If all the NICs are failing it could be some hardware fault. Bad caps?

Steve

[deanot]

Well, I am done with it.  I have tried everything I can do to make this stable, it went 7 days before shitting itself 4 times today so far.  I am back to my Cisco router/firewall, at least until I can get my hands on a 4 port network card.

Anyone interested in a Firebox?

[Engineer]

sorry to hear deanot! :(

My board (igb driver) ran for 4 days before crashing twice this morning, once under almost no load at 4:30am in the morning.  I've managed to compile the latest Intel driver 2.4.3 for mine and will try it later.  Damn, took hours of reading and trial and error to get it going (had to compile it in a FreeBSD 10.1 image installed in a VM).  Hope this solves the issue.

Anyway, good luck in the future.

Edit:  Running on new driver but looks like it doesn't support traffic shaping, which is beyond my ability to fix.  Oh well, I guess I will try it out still to see how well it works (or doesn't).

Edit #2:  Looks like the driver source would need to be edited to allow 'altq' additions.  Too much to worry about right now…will just test as is to see if it locks up or not.

Edit #3: Still locked up during the night.  I'm going to move the LAN over to another, unused port to see if this is a hardware issue.

[deanot]

I do hope you get it working, I feel lost without using the red brick, kinda unsecured in a way.  I am going away for a couple of weeks, being out of the country and this being so unstable is not something I can leave running.

My phone server is protected on the other side of this brick, plus all the other devices that require it for internet access.

I thought you was going to load 2.1 back on your machine?  did you do that and are you still seeing the same problems? maybe once 2.2.5 is released on nano the problem will go away?

I am going to look for a couple of cards, just to get something up and running, at least I can dump my config and put it on a new build, might be something to try.

[Engineer]

Since this was new, I never had 2.1.5 on it in the first place.  I spent the better part of a day trying to load it with the full CD install only to be greeted by the ROOT MOUNT ERROR no matter what I did.  I finally gave up on that one (for now).

Not sure what to do now other than to switch ports (or replace the switch that igb1 is attached to - Zyxel GS1900-16 smart semi managed switch - to see if the LAN port is having issues with the switch).

Edit:  Probably a long shot but I did have it connected to another switch (dumb switch) when I did the week burn-in test (with no WAN though).  So either the switch makes a difference or something about routing traffic from the WAN to the LAN is locking up the port needing a reset.

Edit #2:  Moved the connection on the LAN to another switch and it just went down for the count again.  Will move it back to the original switch and then move the LAN from igb1 to igb2 to see if there is a hardware issue with port 1 (igb1) of the board.

Edit #3:  The igb2 port did a watchdog and reset also.  Unless I'm mistaken and the ports run off the same chip (don't think so), this has got to be a software or configuration error. :(

[corvey]

Engineer, have you not tried 2.1.5 on the Firebox yet?  I've been using that for over a year without any problems 24/7.  BSD 10.1/Pfsense 2.2.4 has a bug somewhere for sure using this old hardware.    I'm building a new system now so can just retire my Firebox on 2.1.5.

[Engineer]

@corvey:

Engineer, have you not tried 2.1.5 on the Firebox yet?  I've been using that for over a year without any problems 24/7.  BSD 10.1/Pfsense 2.2.4 has a bug somewhere for sure using this old hardware.    I'm building a new system now so can just retire my Firebox on 2.1.5.

corvey,  I'm not using a firebox…just jumped in this thread because it was very similar to my issue.  I'm running new hardware (Supermicro X11SBA-LN4F  with Intel N3700 board with 4 Intel A210-AT ports).  I tried to install 2.1.5 but couldn't get around ROOT MOUNT ERROR no matter what I did.  I spent half a day trying to get it to install from both a USB stick and burned CD....no luck.

I might try again later when people aren't in the house using the Internet (they yell within 10 seconds of the watchdog triggering...sigh)

I know, I should have started my own topic (or stayed in the X11SBA-LN4F  thread right below this one....just trying to solve both my and Deanot's issue).

[deanot]

I have an HP 4 port nic coming, Dual intel chipset.  I will build a machine outside of the Firebox, see how it works out, I do hope this is not a driver issue with the build.  If it is, I will use an older version and avoid the newer builds until certain the issues are resolved.

A few things I do know.

(1) it is not device dependent.
(2) It started to happen on the 2.2.4 Nano BSD build for myself and the regular 2.2.4 build for Engineer.
(3) Pulling the network cable from the port resets the port and the issue goes away.  For a while at least.
(4) Time is not a factor, I have seen my box run anything up to 14 days without issue, then BAM.  Or it could happen within minutes, hours or days of being up.
(5) Heavy traffic or minimal traffic has not much affect, for me, I could just log into the GUI and it would trigger it.
(6) Slowing the Nic Speed down, changing settings related to Nic cards had little to no effect.

[Engineer]

I turned off VLAN_HWTCO (ifconfig igb2 -vlanhwtco) and also apinger (WAN PORT MONITORING).  Log is down to less than 5% of what it was (things were restarting and checking VERY often.  One+ days up with little in the log.  Time will tell.

[deanot]

Keep us posted, you might drill down into what the actual issue is.

[Engineer]

@deanot:

Keep us posted, you might drill down into what the actual issue is.

Very sad.  LAN port watchdog timeout again.  :(

Not sure where I'm going from here…..

[power_matz]

Hello,

I am affected too!  :(

Just a question: I have the LCDProcDev version running on my Firebox. When the lock up happens on your machines is the LCDProc still running? On my machine there is no change on the display.
So I thougt it has completely chrashed and I switched it off andf on again.

Matthias

[deanot]

I run it too, it continues to function like nothing is wrong.

[power_matz]

I have a X750e with deactivated MSK intefaces.
I will give it a try to use them instead of the sk interface I currently use….

[corvey]

2.2.5 is out, maybe they fixed this bug.

[deanot]

I don't think 2.2.5 is a nano build is it?