Firebox Marvel ports locking up (CORE-E SERIES)
-
Mine locked up around 30 minutes ago. I went to the console and it was working. Before I could go downstairs to unplug and replug the LAN, it came up with this message (added attachment). After that, Internet would partially work. Had to go in and disable IPV6 to get it all working. Note, the first time it locked up (last week), IPV6 was disabled so I'm not sure it had anything to do with IPV6. The watchdog seems to be the key here. Any thoughts?
-
I am going to quote this again, i know we are not talking about realtec cards, but the symptoms are almost identical.
Known Issues
The Realtek NICs in this box are known to suffer a lock-up condition under certain circumstances. Despite repeated efforts it has not been possible to either cure the problem or ascertain exactly what triggers it. When the problem is triggered the system log will show watchdog timeout and refer to the interface causing it. Fortunately this doesn't affect all users and even then only under some circumstances.
It would seem to be related to packet fragmentation and hardware off loading. Some users have reportedly solved the problem by disabling all hardware offloading and/or using a better switch that can reassemble packets correctly. -
My box is still doing it, had to reset twice yesterday. Might slow the ports down to 10Mbps and see what happens.
-
My box is still doing it, had to reset twice yesterday. Might slow the ports down to 10Mbps and see what happens.
Sorry if you've tried this but if not, could you try the MSI/MSIX tweak(s) posted here and see if it helps:
https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards
Seems to help some…others, not so much.
Also seems that this is something released in the 2.2.x series as people go back to 2.1.x and problem goes away for the most part.
-
Thanks, I will try that. The weird thing is, the network was fine, as soon as I tried to login to the server, the port crashed/locked up/died. It does this not all the time, but seems to be triggered by it. Very odd if you ask me…
-
Reading some more, I found this thread (https://forum.pfsense.org/index.php?topic=96325.0) which is Intel related but I'm not so sure it doesn't go beyond that. Seem FreeBSD 10.1 has an issue that needs to be corrected at the OS level that 'could' be causing this.
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=199174
States "In Progress" under status.
Not sure if this effects Realtek cards or not but reading though the freebsd.org link, the symptoms sound exactly like mine and yours too.
-
Seems this issue (at least with Intel NIC's) has been around for some time: https://forums.freebsd.org/threads/workaround-freebsd-10-1-sudden-network-down.49264/
No clue if FreeBSD 10.2 fixes it or not. However, pfsense 2.1.5 has an older FreeBSD and I'll try that if it's not fixed soon.
-
What ever it is, as said before, was exacerbated with the 2.2.x upgrade in my opinion. I don't recall seeing it in past releases, maybe some have seen it, but personally I have not been using the great software near as long as some have.
I hope there is a fix for it soon, I hate to move to something else, but it is becoming too unreliable for myself to continue use.
-
What ever it is, as said before, was exacerbated with the 2.2.x upgrade in my opinion. I don't recall seeing it in past releases, maybe some have seen it, but personally I have not been using the great software near as long as some have.
I hope there is a fix for it soon, I hate to move to something else, but it is becoming too unreliable for myself to continue use.
2.2.x was the start of FreeBSD 10.1 with pfsense…as far as I can tell.
-
Interesting… I am hoping that one of the devs may chime in, maybe have some insight into this.
-
Had first LAN lockup with watchdog timeout in three days. Just turned off all hardware and checksum offloading, saved and rebooted. Now to start over monitoring.
If this doesn't work, I will probably go to 2.1.5 until (if that works) until it's resolved.
Edit: After reading around more, I'm more inclined to believe that the error is in FreeBSD and that a commit to fix the same error for the older Intel cards (em) has already been committed. https://reviews.freebsd.org/D3192
As to when it appears, not sure. Seems to have been sent to the Intel Networking for review too so time will tell.
As for the OP's issue…if this is a FreeBSD issue (seems to appear on lots of different brands of hardware), hopefully, the FreeBSD patches will fix it. Don't know enough about that to guess at this point though.
-
2.1.5 works.
-
Not sure if it matters or not but I just noticed that even though I checked to disable TSO under Advanced, Networking, the system tunable net.inet.tcp.tso was at 1. Changed it to 0, saved and then added net.inet.tcp.tso=0 to /boot/loader.conf.local just to make sure.
I guess I'll reboot now instead of waiting a few days because I had the disable TSO option checked before and it made no difference.
-
Keep us informed on how it works out, I have been up for a couple of days without issue so far. I did make the changes you mentioned earlier in the topic, it could be more stable from the changes, but this thing is temperamental and will flake out when it feels like it.
-
I changed the topic title to reflect more detail on the product.
-
Didn't take long this time. Just went down and a watchdog timeout reset it. Going to try 2.1.5 to see if that takes care of it until it's fixed. The folks at Opnsense seemed to think it wouldn't be until FreeBSD 11 before the commits were pushed into production. :(
Edit: As before, once the lockup occurs, the watchdog will reset it and restore IPV4 Internet access but IPV6 will remain down until turned off/on or the pfsense system is rebooted. Right now, my browsing is very slow as it tries to use IPV6 first and then falls back to IPV4 (haven't rebooted yet). It was a Netflix stream and possibly a TWC TV stream at the same time (high traffic load) that triggered this latest lockup.
-
Now you mention it, it does happen with high traffic, my wife watches a lot of online tv, normal to do it then. Especially if I log into the box using the GUI, soon as I do that BOOM!.
-
Tried the MXI / MSIX disable and it wouldn't even let me to the LAN / Internet after a reboot. Edited those back out via console / Shell, rebooted and it ran for 5 minutes before lockup. Just downloaded 2.1.5 and will try it as soon as I can get people off of Netflix long enough to get it started.
-
Tried the MXI / MSIX disable and it wouldn't even let me to the LAN / Internet after a reboot. Edited those back out via console / Shell, rebooted and it ran for 5 minutes before lockup. Just downloaded 2.1.5 and will try it as soon as I can get people off of Netflix long enough to get it started.
Interesting, I did the MXI / MSIX fix on mine, it worked ok on a reboot.
-
2.1.5 works.
It might if I could get past the ROOT MOUNT ERROR on the Live CD. I guess I'll try the memstick version.
Edit: I've given up for tonight. Can't get past the ROOT MOUNT ERROR on the LiveCD version nor the Memstick version. Have followed pages of advice and just can't get past it. Might try later….sigh
Edit #2: Never got it to work and will try later. Did find a link that pretty much states the same thing for my card as the em card (PCI vs PCIe I think). https://lists.freebsd.org/pipermail/freebsd-net/2014-January/037694.html . This would seem to be an issue with ANY card with TSO, no matter the driver.