Kernel Panic

fasteddy

Here goes:

# sysctl debug.ddb
debug.ddb.capture.data:
debug.ddb.capture.bufsize: 49152
debug.ddb.capture.inprogress: 0
debug.ddb.capture.maxbufsize: 5242880
debug.ddb.capture.bufoff: 0
debug.ddb.scripting.unscript:
debug.ddb.scripting.scripts: lockinfo=show locks; show alllocks; show lockedvnods
kdb.enter.panic=textdump set; capture on; run lockinfo; show pcpu; bt; ps; alltrace; capture off; call doadump; reset
kdb.enter.witness=run lockinfo

debug.ddb.textdump.do_version: 1
debug.ddb.textdump.do_panic: 1
debug.ddb.textdump.do_msgbuf: 1
debug.ddb.textdump.do_ddb: 1
debug.ddb.textdump.do_config: 1
debug.ddb.textdump.pending: 0

jimp

That all looks normal, like it should be doing the textdumps…

What about:

cat /boot/kernel/pfsense_kernel.txt

fasteddy

Doesn't have a whole lot to say:

# cat /boot/kernel/pfsense_kernel.txt
UP
#

jimp

It's enough :-)

That's the uniprocessor kernel. I'm running the SMP kernels. Perhaps that is the difference. Can you switch to an SMP kernel and see if the crashes leave you at the debug prompt still? There are instructions earlier in this thread for switching kernels (also you can edit that text file and change it to SMP instead of UP and the next update will use the SMP kernel instead)

pwnell

Installed the latest one available

2.0-BETA5 (i386)
built on Sun Feb 6 23:54:00 EST 2011

I have a 1 hour 15 minute uptime …. Holding thumbs.

LostInIgnorance

JimP,

Did you need me to do anything while I am here working on other projects? Anything I can do to help debug this?

jimp

No, Ermal is going to be the one having to look into the actual panics.

I was just trying to figure out why some people aren't getting the proper crash dumps like they should.

fasteddy

I've changed to the SMP kernel and updated to the latest snapshots as of 15 minutes ago, and my machine seems to be automatically rebooting and generating proper crash reports now. Thanks jimp!

Unfortunately, I am still able to crash my system at will the same way I have been from the get go - by accessing the web gui through the VPN. I sent the (now useful!) crash report via the web gui, is there anything I can/should do beyond that at this point?

jimp

When the next snapshot comes out later this evening/tonight, update to it and try to reproduce it again.

LostInIgnorance

JimP, who exactly were you talking to? or was it just a general reference to everyone?

jimp

just in general for anyone still getting crashes… hopefully it's more stable.

LostInIgnorance

I won't get to testing that box (p4 dell gonna kick it in the parking lot) till tomorrow morning, so I'll have to get back to you on the test.

cmb

The snapshot with the fixes Jim mentioned is now up, I've upgraded all my systems to it and no issues. Granted I wasn't seeing any panics previously either, so those who can replicate issues, please test.

fasteddy

So far so good using this morning's build - tinkered around with the web gui for awhile with no crashes (couldn't even log in before…), did the same with a shell connection (which always crashed after a few minutes before) and am 1/4 of the way through transferring a 1.25GB file through the vpn to the pfSense box via SCP with no crashes so far. (It has always been communication directly with the remote pfSense box through the VPN that triggered crashes for me). Fingers crossed...

I am curious though - and I'll (dis)qualify this by saying that I am not a developer, nor do I pretend to fully understand much of the goings on in the git repository, but - I don't see anything in the list of commits since the last snapshot I tried that I would have expected to have an impact on this particular issue. Any idea as to what was causing the panics in the first place?

jimp

@fasteddy:

I am curious though - and I'll (dis)qualify this by saying that I am not a developer, nor do I pretend to fully understand much of the goings on in the git repository, but - I don't see anything in the list of commits since the last snapshot I tried that I would have expected to have an impact on this particular issue. Any idea as to what was causing the panics in the first place?

Look at the commits to the tools repo, not just the pfSense code repo. The tools repo is where things like kernel patches and builder changes happen.

fasteddy

Thanks, jimp, I had no idea! Adds a whole new level of excitement to the daily snapshots, doesn't it…

FYI, I have at this point - in addition to trying everything else that previously caused an instant panic - transferred a couple of gigs through the VPN to what I would have called my 'sick' pfsense box, and everything seems to be working and happy.

jimp

@fasteddy:

Thanks, jimp, I had no idea! Adds a whole new level of excitement to the daily snapshots, doesn't it…

Yeah, those are the main two. There are also others that are used but they change much less often, if at all (like the freesbie2 stuff).

@fasteddy:

FYI, I have at this point - in addition to trying everything else that previously caused an instant panic - transferred a couple of gigs through the VPN to what I would have called my 'sick' pfsense box, and everything seems to be working and happy.

Sounds very promising. Keep trying to kill it, see what happens. :-)

pwnell

Crashed again, after about 3 days I think…

2.0-BETA5 (i386)
built on Sun Feb 6 23:54:00 EST 2011

crash1.jpg_thumb

crash2.jpg_thumb

jimp

@pwnell:

Crashed again, after about 3 days I think…

2.0-BETA5 (i386)
built on Sun Feb 6 23:54:00 EST 2011

Update to the 9th or later, it should be stable from that point on.

pwnell

Yep did that just now, will keep an eye on it. Thanks.