Kernel Panic

LostInIgnorance · Jan 25, 2011, 1:58 AM

not that I am aware of, I do force all traffic through the OpenVPN interface (Force all client generated traffic through the tunnel)

LostInIgnorance · Jan 25, 2011, 2:30 AM

I did notice this on the console after having to uninstall and reinstall HAVP.

lock order reversal:
 1st 0xc4279df4 ufs (ufs) @ /usr/pfSensesrc/src/sys/kern/vfs_mount.c:1204
 2nd 0xc474e6a0 syncer (syncer) @ /usr/pfSensesrc/src/sys/kern/vfs_subr.c:2203
KDB: stack backtrace:
X_db_sym_numargs(c0c4e35f,d6704a3c,c092ce85,c091d9bb,c0c512cb,...) at X_db_sym_numargs+0x146
kdb_backtrace(c091d9bb,c0c512cb,c3516ee8,c3517020,d6704a98,...) at kdb_backtrace+0x29
witness_display_spinlock(c0c512cb,c474e6a0,c0c5872e,c3517020,c0c585a4,...) at witness_display_spinlock+0x75
witness_checkorder(c474e6a0,9,c0c585a4,89b,0,...) at witness_checkorder+0x839
__lockmgr_args(c474e6a0,80100,c474e6bc,0,0,...) at __lockmgr_args+0x7f5
vop_stdlock(d6704bb4,3,c0c585a4,80100,c474e648,...) at vop_stdlock+0x62
VOP_LOCK1_APV(c1032b00,d6704bb4,c08d9223,c10560a0,c474e648,...) at VOP_LOCK1_APV+0xb5
_vn_lock(c474e648,80100,c0c585a4,89b,0,...) at _vn_lock+0x5e
insmntque(d6704c58,c097342e,c474e648,0,c0c57db8,...) at insmntque+0x288
vrele(c474e648,0,c0c57db8,4f9,80,...) at vrele+0x10
dounmount(c37b2000,8080000,c3b3a780,47e,fdf65b4a,...) at dounmount+0x3ce
unmount(c3b3a780,d6704cf8,c3b3a780,d6704d2c,206,...) at unmount+0x2bf
syscall(d6704d38) at syscall+0x1da
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (22, FreeBSD ELF32, unmount), eip = 0x280dfa9f, esp = 0xbfbfe61c, ebp = 0xbfbfe6e8 ---

jimp · Jan 25, 2011, 2:44 AM

Don't worry about LORs, they're mostly harmless.

LostInIgnorance · Jan 25, 2011, 2:51 AM

Did notice this from both panics though.

Panic from old P4 computer:

Kernel page fault with the following non-sleepable locks held:
exclusive sleep mutex em0 (EM TX Lock) r = 0 (0xc2f52580) locked @ /usr/pfSensesrc/src/sys/dev/e1000/if_lem.c:1350

Panic from Soekris board:

Kernel page fault with the following non-sleepable locks held:
exclusive sleep mutex vr1 (network driver) r = 0 (0xc3640aec) locked @ /usr/pfSensesrc/src/sys/dev/vr/if_vr.c:1675

Is it just coincidence?

disa · Jan 25, 2011, 8:31 AM

Hi, I've reinstalled the secondary machine, it's now running 2.0-BETA5 (amd64) built on Mon Jan 17 22:14:04 EST 2011 (the primary has 2.0-BETA5 (amd64) built on Fri Jan 21 23:51:34 EST 2011).

I disabled the sync, created all the remaining carp vips on the primary (I've 12 of them right now), and re-enabled the sync. The secondary didn't crash.

What shall I do now? I'm a bit scared of upgrading it :-) thanks

eri-- · Jan 25, 2011, 11:38 AM

Just wait for build late build from today and it should be safe to upgrade.

disa · Jan 25, 2011, 12:03 PM

ok, and which kernel should I be running? SMP or devel? thanks

vito · Jan 25, 2011, 1:10 PM

as of snap
2.0-BETA5 (i386)
built on Mon Jan 24 18:48:13 EST 2011

fix did not work.

Maybe fix was not in this build?

jimp · Jan 25, 2011, 2:10 PM

@vito:

as of snap
2.0-BETA5 (i386)
built on Mon Jan 24 18:48:13 EST 2011

fix did not work.

Maybe fix was not in this build?

It should have been.

So it still crashed the exact same way, with the same panic in the same place?

jimp · Jan 25, 2011, 2:18 PM

@LostInIgnorance:

Did notice this from both panics though.

Panic from old P4 computer:

Kernel page fault with the following non-sleepable locks held:
exclusive sleep mutex em0 (EM TX Lock) r = 0 (0xc2f52580) locked @ /usr/pfSensesrc/src/sys/dev/e1000/if_lem.c:1350

Panic from Soekris board:

Kernel page fault with the following non-sleepable locks held:
exclusive sleep mutex vr1 (network driver) r = 0 (0xc3640aec) locked @ /usr/pfSensesrc/src/sys/dev/vr/if_vr.c:1675

Is it just coincidence?

Hard to say for sure.

I just setup my ALIX and opened a ton of browser tabs and pushed a bunch of traffic through it and I managed to crash Chrome but not the router… I wish I could reproduce this.

vito · Jan 25, 2011, 2:58 PM

@jimp:

@vito:

as of snap
2.0-BETA5 (i386)
built on Mon Jan 24 18:48:13 EST 2011

fix did not work.

Maybe fix was not in this build?

It should have been.

So it still crashed the exact same way, with the same panic in the same place?

Here is the new screen shoot of the panic.
Same symptoms, only when using openvpn

LostInIgnorance · Jan 25, 2011, 5:02 PM

The key is only when you connect to OpenVPN and then after successful connection, open a browser and the page should time out and you should get disconnected from the OpenVPN connection.

jimp · Jan 25, 2011, 5:04 PM

I had ~10 browser tabs open to all kinds of sites, streaming a youtube video, and was copying a file over SMB, all over OpenVPN from a client on the WAN side of my ALIX.

Perhaps it's a bug specific to the vr chip in the Soekris.

LostInIgnorance · Jan 25, 2011, 5:06 PM

If it is a specific issue to the vr chip, then why am I getting the same type of panic with the em (as I had posted above)?

jimp · Jan 25, 2011, 5:07 PM

Bad luck? :-)

The em chips hit by this are specific older chips that fall under the "legacy em" ("lem") portion of the driver code.

LostInIgnorance · Jan 25, 2011, 5:11 PM

LOL, seems to be me only that has the bad hardware then.

EDIT: Did you try it with squid havp, etc. loaded on it?

vito · Jan 25, 2011, 5:09 PM

@LostInIgnorance:

The key is only when you connect to OpenVPN and then after successful connection, open a browser and the page should time out and you should get disconnected from the OpenVPN connection.

Note the case for me,
Once connected i would start up outlook to sync to exchange and openvpn would crash
Same goes with doing a file/folder copy from host network to openvpn client.

LostInIgnorance · Jan 25, 2011, 5:12 PM

@vito:

@LostInIgnorance:

The key is only when you connect to OpenVPN and then after successful connection, open a browser and the page should time out and you should get disconnected from the OpenVPN connection.

Note the case for me,
Once connected i would start up outlook to sync to exchange and openvpn would crash
Same goes with doing a file/folder copy from host network to openvpn client.

Wish I could have made it as far as you vito

jimp · Jan 25, 2011, 5:13 PM

@LostInIgnorance:

EDIT: Did you try it with squid havp, etc. loaded on it?

I haven't tried it with squid, I'll load that on next. Do you have the OpenVPN server assigned as an interface and squid listening there? Otherwise, squid wouldn't be doing anything to the traffic coming from OpenVPN

LostInIgnorance · Jan 25, 2011, 5:17 PM

No I am not having squid listen to anything but the 3 different vr networks. I am just wondering if the setup may have something to do with it. Doesn't hurt to recreate it completely.