Kernel Panic

fasteddy

So far so good using this morning's build - tinkered around with the web gui for awhile with no crashes (couldn't even log in before…), did the same with a shell connection (which always crashed after a few minutes before) and am 1/4 of the way through transferring a 1.25GB file through the vpn to the pfSense box via SCP with no crashes so far. (It has always been communication directly with the remote pfSense box through the VPN that triggered crashes for me). Fingers crossed...

I am curious though - and I'll (dis)qualify this by saying that I am not a developer, nor do I pretend to fully understand much of the goings on in the git repository, but - I don't see anything in the list of commits since the last snapshot I tried that I would have expected to have an impact on this particular issue. Any idea as to what was causing the panics in the first place?

jimp

@fasteddy:

I am curious though - and I'll (dis)qualify this by saying that I am not a developer, nor do I pretend to fully understand much of the goings on in the git repository, but - I don't see anything in the list of commits since the last snapshot I tried that I would have expected to have an impact on this particular issue. Any idea as to what was causing the panics in the first place?

Look at the commits to the tools repo, not just the pfSense code repo. The tools repo is where things like kernel patches and builder changes happen.

fasteddy

Thanks, jimp, I had no idea! Adds a whole new level of excitement to the daily snapshots, doesn't it…

FYI, I have at this point - in addition to trying everything else that previously caused an instant panic - transferred a couple of gigs through the VPN to what I would have called my 'sick' pfsense box, and everything seems to be working and happy.

jimp

@fasteddy:

Thanks, jimp, I had no idea! Adds a whole new level of excitement to the daily snapshots, doesn't it…

Yeah, those are the main two. There are also others that are used but they change much less often, if at all (like the freesbie2 stuff).

@fasteddy:

FYI, I have at this point - in addition to trying everything else that previously caused an instant panic - transferred a couple of gigs through the VPN to what I would have called my 'sick' pfsense box, and everything seems to be working and happy.

Sounds very promising. Keep trying to kill it, see what happens. :-)

pwnell

Crashed again, after about 3 days I think…

2.0-BETA5 (i386)
built on Sun Feb 6 23:54:00 EST 2011

crash1.jpg_thumb

crash2.jpg_thumb

jimp

@pwnell:

Crashed again, after about 3 days I think…

2.0-BETA5 (i386)
built on Sun Feb 6 23:54:00 EST 2011

Update to the 9th or later, it should be stable from that point on.

pwnell

Yep did that just now, will keep an eye on it. Thanks.

pwnell

Ran for 2 hours then everything just hung. No panic, nothing in /var/crash and no crash message after reboot. Keyboard / everything was frozen, no traffic was allowed through the device.

jimp

Any idea what you were doing at the time? FTP? PPTP?

pwnell

I had a VPN connection open (PPTP), but not really much data transfer. Not much more - I was not doing anything other than the IMAP/SMTP/SSH/HTTP traffic.

pwnell

1 hour, crashed:

2.0-BETA5 (i386)
built on Wed Feb 9 01:34:47 EST 2011

crash3.jpg_thumb

crash4.jpg_thumb

FisherKing

Just reporting my experiences -
Running 2.0-BETA5 (i386) built on Wed Feb 9 00:54:34 EST 2011 on both machines. (Did a clean install)
uptime = 7+ hours

Testing CARP & failover, FTP traffic, OpenVPN client/server setup.

No crashes so far, and they used to be pretty easy to produce.

This is only a simple test environment, so it doesn't get regular traffic, only a single client at a time for now.

geewhz01

I'm getting a panic in today's snapshot, 2.0-BETA5 (amd64)
built on Wed Feb 9 19:39:50 EST 2011 . This was caused I believe by syncing to slave. I was in the process of setting up a new interface on the master and configuring a vpn to it.

Andy

pfsense2911.png_thumb

eri--

This is master or slave?
That seems like an impossible trace but i will look at it after the information.

geewhz01

This is on the slave. I've not had any problems on the master with this, just the slave.

Thanks,

Andy

sammyfusionsleep.com

snapshot from 2/9/2011.
Dell Poweredge 860
Intel Pentium D CPU 2.80GHz
2 gigs ram.
Openvpn client export plugin is the only plugin installed.
pptp server for remote clients.
4 ipsec tunnels to other networks.

Crashes get stuck at db> prompt.
I have been unable to find a way to reliably trigger a crash but it does so every 24 hours or so.

Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address= 0x6c
fault code= supervisor read, page not present
instruction pointer= 0x20:0xc04ee165
stack pointer        = 0x28:0xc5b89c20
frame pointer        = 0x28:0xc5b89c34
code segment= base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags= interrupt enabled, resume, IOPL = 0
current process= 8 (pfpurge)
[thread]
Stopped at      pf_state_tree_ext_gwy_RB_REMOVE_COLOR+0x25:   cmpl $0x1,0x6c(%edx)
db>

Happy to provide additional information.[/thread]

jimp

@sammy@fusionsleep.com:

snapshot from 2/9/2011.
[…]
Stopped at      pf_state_tree_ext_gwy_RB_REMOVE_COLOR+0x25:   cmpl $0x1,0x6c(%edx)
db>
Happy to provide additional information.

We also need the output of "bt" at that prompt. Also, is this i386 or amd64?

sammyfusionsleep.com

Will do the bt at the prompt next crash.

snapshot from 2/9/2011.
Dell Poweredge 860
Intel Pentium D CPU 2.80GHz
2 gigs ram.
Openvpn client export plugin is the only plugin installed.
pptp server for remote clients.
4 ipsec tunnels to other networks.

jimp

None of that tells us if you used a 32-bit (i386) or 64-bit (amd64) snapshot.

sammyfusionsleep.com

i386, sorry, I didn't realize the amd64 snapshot would run on a pentium D.