Kernel Panic

vito · Jan 28, 2011, 4:53 PM

it is a hard lock…can't type anything in.
On reboot, still do not see the crash files that jimp added.

jimp · Jan 28, 2011, 5:22 PM

it is a hard lock…can't type anything in.
On reboot, still do not see the crash files that jimp added.

As I said before, those won't be created for hard locks, only for panics/crashes.

Slaygon · Jan 28, 2011, 7:55 PM

Just seen this come out: "Fri Jan 28 13:06:21 EST 2011"…
Is this the one where the carp sync problems are addressed?

If so, it will take me a number of hours to get this applied and verified. 1-4 hours of active monitoring before the bug usually appears, however, I will not be able to start testing all too soon. Expecting about 2 to 8 hours before I can start monitoring.

If anyone could start testing sooner, that would be great!

(for those that don't know, I run two DL-type HP servers with quad Intel gbit NICs in them)

And again, great work pf team! Really appreciated!

geewhz01 · Jan 28, 2011, 8:36 PM

If the carp patches were also pushed to the AMD version 2.0-BETA5 (amd64)
built on Fri Jan 28 13:06:21 EST 2011 then it's not fixed for me. As soon as I make any change and it syncs the backup firewall locks or panics. It's not a hard lock though, it does bring me to a prompt but not seeing any of crash files in the /var/crash.

Andy

Slaygon · Jan 28, 2011, 9:08 PM

@geewhz01:

If the carp patches were also pushed to the AMD version 2.0-BETA5 (amd64)
built on Fri Jan 28 13:06:21 EST 2011 then it's not fixed for me. As soon as I make any change and it syncs the backup firewall locks or panics. It's not a hard lock though, it does bring me to a prompt but not seeing any of crash files in the /var/crash.

Andy

It's the amd64 version I am running too.
Not too good news there, Andy. Sorry to hear.
And usually, with the carp errors, there seems to be a freeze, rather than a crash, which will render no trace dumps.
Sometimes, very unusual though, I get a debug prompt at which I can get a backtrace. Mostly the box just hangs.

I guess the devs would appreciate any input they can get hold of, so if you do get to a prompt and can type something in, give "bt" (as in backtrace) a go and see if you could provide any more info.

Cheerio.

jimp · Jan 28, 2011, 9:19 PM

I just tried on my amd64 vm and when I force a manual panic (sysctl debug.kdb.panic=1) I get a textdump, and no db> prompt. I'm not sure why someone on a current snapshot update would still be getting left at a db> prompt when it crashes, they should be gathering the info and rebooting on their own.

Doesn't help with the hangs, though, but the hangs are a different problem (and a different thread :-)

geewhz01 · Jan 28, 2011, 10:04 PM

Here is what I get when carp tries to sync. This is running snapshot version amd64-20110128-0938.

Thanks,

Andy

jimp · Jan 28, 2011, 10:07 PM

That snap was before the last fixes went in. Try the newest one.

geewhz01 · Jan 28, 2011, 10:10 PM

@jimp:

That snap was before the last fixes went in. Try the newest one.

pfSense-Full-Update-2.0-BETA5-amd64-20110128-0938.tgz 28-Jan-2011 13:09 99M

That's the latest I see on the snapshots server. Am I looking in the wrong spot? http://snapshots.pfsense.org/FreeBSD_RELENG_8_1/amd64/pfSense_HEAD/updates/?C=M;O=D

jimp · Jan 28, 2011, 10:15 PM

It's probably still building right now. Give it some time…

vito · Jan 29, 2011, 3:56 PM

still problems on latest snap…
2.0-BETA5 (i386)
built on Sat Jan 29 01:09:59 EST 2011

this time, i could type bt (no hard lock)

Here is the out put (sorry had to be a cell pic)

jimp · Jan 29, 2011, 4:05 PM

One more patch went in after that build, it's going right now and will be ready in a few hours.

vito · Jan 29, 2011, 4:21 PM

sounds good thanks!

jeebsion · Jan 30, 2011, 6:00 AM

has anyone been running the latest snapshot? does it still panic and all?

geewhz01 · Jan 30, 2011, 12:30 PM

@jeebsion:

has anyone been running the latest snapshot? does it still panic and all?

I have been since late last night and I've tried everything I was doing before to make it panic but have had no success. Looks like the guys have done a good job of finding the problem.

Andy

jeebsion · Jan 30, 2011, 4:58 PM

@geewhz01:

@jeebsion:

has anyone been running the latest snapshot? does it still panic and all?

I have been since late last night and I've tried everything I was doing before to make it panic but have had no success. Looks like the guys have done a good job of finding the problem.

Andy

Oh .. that's good .. because the last update it hung whilst rebooting the moment its initializing the WAN interfaces ..

vito · Jan 30, 2011, 5:03 PM

i am still testing but so far on
snap
2.0-BETA5 (i386)
built on Sat Jan 29 23:42:13 EST 2011

Have not had a lock or panic!
:)
pushed a good amount of data with open vpn.

eri-- · Jan 30, 2011, 7:22 PM

Please monitor the output of netstat -m and see if those counters go crazy otherwise thank you for help testing.

geewhz01 · Jan 30, 2011, 11:33 PM

Had another panic just now after 21 hrs uptime. Attached is info, running build AMD64-20110129-1502.

Thanks,

Andy

eri-- · Jan 31, 2011, 8:13 AM

You need to upgrade at one snapshot after it.
This issue was reported before in this thread and it should be fixed on snapshot later than what you have installed