Firebox Marvel ports locking up (CORE-E SERIES)
-
Thanks, I will try that. The weird thing is, the network was fine, as soon as I tried to login to the server, the port crashed/locked up/died. It does this not all the time, but seems to be triggered by it. Very odd if you ask me…
-
Reading some more, I found this thread (https://forum.pfsense.org/index.php?topic=96325.0) which is Intel related but I'm not so sure it doesn't go beyond that. Seem FreeBSD 10.1 has an issue that needs to be corrected at the OS level that 'could' be causing this.
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=199174
States "In Progress" under status.
Not sure if this effects Realtek cards or not but reading though the freebsd.org link, the symptoms sound exactly like mine and yours too.
-
Seems this issue (at least with Intel NIC's) has been around for some time: https://forums.freebsd.org/threads/workaround-freebsd-10-1-sudden-network-down.49264/
No clue if FreeBSD 10.2 fixes it or not. However, pfsense 2.1.5 has an older FreeBSD and I'll try that if it's not fixed soon.
-
What ever it is, as said before, was exacerbated with the 2.2.x upgrade in my opinion. I don't recall seeing it in past releases, maybe some have seen it, but personally I have not been using the great software near as long as some have.
I hope there is a fix for it soon, I hate to move to something else, but it is becoming too unreliable for myself to continue use.
-
What ever it is, as said before, was exacerbated with the 2.2.x upgrade in my opinion. I don't recall seeing it in past releases, maybe some have seen it, but personally I have not been using the great software near as long as some have.
I hope there is a fix for it soon, I hate to move to something else, but it is becoming too unreliable for myself to continue use.
2.2.x was the start of FreeBSD 10.1 with pfsense…as far as I can tell.
-
Interesting… I am hoping that one of the devs may chime in, maybe have some insight into this.
-
Had first LAN lockup with watchdog timeout in three days. Just turned off all hardware and checksum offloading, saved and rebooted. Now to start over monitoring.
If this doesn't work, I will probably go to 2.1.5 until (if that works) until it's resolved.
Edit: After reading around more, I'm more inclined to believe that the error is in FreeBSD and that a commit to fix the same error for the older Intel cards (em) has already been committed. https://reviews.freebsd.org/D3192
As to when it appears, not sure. Seems to have been sent to the Intel Networking for review too so time will tell.
As for the OP's issue…if this is a FreeBSD issue (seems to appear on lots of different brands of hardware), hopefully, the FreeBSD patches will fix it. Don't know enough about that to guess at this point though.
-
2.1.5 works.
-
Not sure if it matters or not but I just noticed that even though I checked to disable TSO under Advanced, Networking, the system tunable net.inet.tcp.tso was at 1. Changed it to 0, saved and then added net.inet.tcp.tso=0 to /boot/loader.conf.local just to make sure.
I guess I'll reboot now instead of waiting a few days because I had the disable TSO option checked before and it made no difference.
-
Keep us informed on how it works out, I have been up for a couple of days without issue so far. I did make the changes you mentioned earlier in the topic, it could be more stable from the changes, but this thing is temperamental and will flake out when it feels like it.
-
I changed the topic title to reflect more detail on the product.
-
Didn't take long this time. Just went down and a watchdog timeout reset it. Going to try 2.1.5 to see if that takes care of it until it's fixed. The folks at Opnsense seemed to think it wouldn't be until FreeBSD 11 before the commits were pushed into production. :(
Edit: As before, once the lockup occurs, the watchdog will reset it and restore IPV4 Internet access but IPV6 will remain down until turned off/on or the pfsense system is rebooted. Right now, my browsing is very slow as it tries to use IPV6 first and then falls back to IPV4 (haven't rebooted yet). It was a Netflix stream and possibly a TWC TV stream at the same time (high traffic load) that triggered this latest lockup.
-
Now you mention it, it does happen with high traffic, my wife watches a lot of online tv, normal to do it then. Especially if I log into the box using the GUI, soon as I do that BOOM!.
-
Tried the MXI / MSIX disable and it wouldn't even let me to the LAN / Internet after a reboot. Edited those back out via console / Shell, rebooted and it ran for 5 minutes before lockup. Just downloaded 2.1.5 and will try it as soon as I can get people off of Netflix long enough to get it started.
-
Tried the MXI / MSIX disable and it wouldn't even let me to the LAN / Internet after a reboot. Edited those back out via console / Shell, rebooted and it ran for 5 minutes before lockup. Just downloaded 2.1.5 and will try it as soon as I can get people off of Netflix long enough to get it started.
Interesting, I did the MXI / MSIX fix on mine, it worked ok on a reboot.
-
2.1.5 works.
It might if I could get past the ROOT MOUNT ERROR on the Live CD. I guess I'll try the memstick version.
Edit: I've given up for tonight. Can't get past the ROOT MOUNT ERROR on the LiveCD version nor the Memstick version. Have followed pages of advice and just can't get past it. Might try later….sigh
Edit #2: Never got it to work and will try later. Did find a link that pretty much states the same thing for my card as the em card (PCI vs PCIe I think). https://lists.freebsd.org/pipermail/freebsd-net/2014-January/037694.html . This would seem to be an issue with ANY card with TSO, no matter the driver.
-
Not seen that at all on any of the fireboxes I have here.
Even on the msk ports, with msi/msix disabled they are stable. I've never seen an issue with the sk interfaces.
The first thing I would do is disable hardware checksum offload if you still have that on. I did see that break dhcp relay inexplicably while everything else worked fine.Many of those boxes have seen a lot of hours in hot conditions. If you have turned the fans down they might be running even hotter. If all the NICs are failing it could be some hardware fault. Bad caps?
Steve
-
Well, I am done with it. I have tried everything I can do to make this stable, it went 7 days before shitting itself 4 times today so far. I am back to my Cisco router/firewall, at least until I can get my hands on a 4 port network card.
Anyone interested in a Firebox?
-
sorry to hear deanot! :(
My board (igb driver) ran for 4 days before crashing twice this morning, once under almost no load at 4:30am in the morning. I've managed to compile the latest Intel driver 2.4.3 for mine and will try it later. Damn, took hours of reading and trial and error to get it going (had to compile it in a FreeBSD 10.1 image installed in a VM). Hope this solves the issue.
Anyway, good luck in the future.
Edit: Running on new driver but looks like it doesn't support traffic shaping, which is beyond my ability to fix. Oh well, I guess I will try it out still to see how well it works (or doesn't).
Edit #2: Looks like the driver source would need to be edited to allow 'altq' additions. Too much to worry about right now…will just test as is to see if it locks up or not.
Edit #3: Still locked up during the night. I'm going to move the LAN over to another, unused port to see if this is a hardware issue.
-
I do hope you get it working, I feel lost without using the red brick, kinda unsecured in a way. I am going away for a couple of weeks, being out of the country and this being so unstable is not something I can leave running.
My phone server is protected on the other side of this brick, plus all the other devices that require it for internet access.
I thought you was going to load 2.1 back on your machine? did you do that and are you still seeing the same problems? maybe once 2.2.5 is released on nano the problem will go away?
I am going to look for a couple of cards, just to get something up and running, at least I can dump my config and put it on a new build, might be something to try.
