Kernel Panic

jimp

Bad luck? :-)

The em chips hit by this are specific older chips that fall under the "legacy em" ("lem") portion of the driver code.

LostInIgnorance

LOL, seems to be me only that has the bad hardware then.

EDIT: Did you try it with squid havp, etc. loaded on it?

vito

The key is only when you connect to OpenVPN and then after successful connection, open a browser and the page should time out and you should get disconnected from the OpenVPN connection.

Note the case for me,
Once connected i would start up outlook to sync to exchange and openvpn would crash
Same goes with doing a file/folder copy from host network to openvpn client.

LostInIgnorance

@vito:

@LostInIgnorance:

The key is only when you connect to OpenVPN and then after successful connection, open a browser and the page should time out and you should get disconnected from the OpenVPN connection.

Note the case for me,
Once connected i would start up outlook to sync to exchange and openvpn would crash
Same goes with doing a file/folder copy from host network to openvpn client.

Wish I could have made it as far as you vito

jimp

@LostInIgnorance:

EDIT: Did you try it with squid havp, etc. loaded on it?

I haven't tried it with squid, I'll load that on next. Do you have the OpenVPN server assigned as an interface and squid listening there? Otherwise, squid wouldn't be doing anything to the traffic coming from OpenVPN

LostInIgnorance

No I am not having squid listen to anything but the 3 different vr networks. I am just wondering if the setup may have something to do with it. Doesn't hurt to recreate it completely.

jimp

fair enough. I'll give it a go.

vito

@jimp:

@LostInIgnorance:

EDIT: Did you try it with squid havp, etc. loaded on it?

I haven't tried it with squid, I'll load that on next. Do you have the OpenVPN server assigned as an interface and squid listening there? Otherwise, squid wouldn't be doing anything to the traffic coming from OpenVPN

Jimp,
it was most likely missed in this thread, but in my original thread with this problem from 12/13
had the packages i am running. http://forum.pfsense.org/index.php/topic,31031.msg165512.html#msg165512

Packages i am running:
Squid
Squid Guard
Snort
OpenVPN Export
Light Squid
Anyterm

If need be, i can test by stopping Squid and such….

jimp

It might be worth trying without any packages. (the only one I have is OpenVPN client export) to see if it makes a difference.

One thing - for everyone seeing the panic on em(4) devices, the output from

pciconf -lvb

For the em card seen in the crash would be helpful.

LostInIgnorance

??? JimP, I am a little confused, do you want me to type that in after the crash happens and the db comes up? If not, what would you like from the crash?

clarknova

I have a net5501-70, a Pro 1000GT (pci), and a spare pppoe DSL line. Jim, I can set this up with openvpn if you want to test against it. I have no experience with openvpn though, so would need some guidance setting that up.

Also note that my DSL line has pretty poor sync rates right now, and occasional disconnects, which is why it's not in use :/

jimp

@LostInIgnorance:

??? JimP, I am a little confused, do you want me to type that in after the crash happens and the db comes up? If not, what would you like from the crash?

Just at a normal command prompt. It will show the chip id of the em card.

jimp

@clarknova:

I have a net5501-70, a Pro 1000GT (pci), and a spare pppoe DSL line. Jim, I can set this up with openvpn if you want to test against it. I have no experience with openvpn though, so would need some guidance setting that up.

Also note that my DSL line has pretty poor sync rates right now, and occasional disconnects, which is why it's not in use :/

I'll keep that in mind, though I was hoping to do it locally since that makes it a lot easier to track down…

vito

@jimp:

It might be worth trying without any packages. (the only one I have is OpenVPN client export) to see if it makes a difference.

One thing - for everyone seeing the panic on em(4) devices, the output from
pciconf -lvb
For the em card seen in the crash would be helpful.

Will try later tonight. :)

from my system
em1@pci0:3:0:0: class=0x020000 card=0x060a15d9 chip=0x10d38086 rev=0x00 hdr=0x00
class = network
subclass = ethernet
bar [10] = type Memory, range 32, base 0xfeae0000, size 131072, enabled
bar [18] = type I/O Port, range 32, base 0xec00, size 32, enabled
bar [1c] = type Memory, range 32, base 0xfeadc000, size 16384, enabled

vito

from another system having problems. (i can not get to the console to see what card)

em0@pci0:2:0:0: class=0x020000 card=0x00008086 chip=0x10d38086 rev=0x00 hdr=0x00
class = network
subclass = ethernet
bar [10] = type Memory, range 32, base 0xfe6e0000, size 131072, enabled
bar [18] = type I/O Port, range 32, base 0x9c00, size 32, enabled
bar [1c] = type Memory, range 32, base 0xfe6dc000, size 16384, enabled
em1@pci0:3:0:0: class=0x020000 card=0x00008086 chip=0x10d38086 rev=0x00 hdr=0x00
class = network
subclass = ethernet
bar [10] = type Memory, range 32, base 0xfe7e0000, size 131072, enabled
bar [18] = type I/O Port, range 32, base 0xac00, size 32, enabled
bar [1c] = type Memory, range 32, base 0xfe7dc000, size 16384, enabled
em2@pci0:4:0:0: class=0x020000 card=0x00008086 chip=0x10d38086 rev=0x00 hdr=0x00
class = network
subclass = ethernet
bar [10] = type Memory, range 32, base 0xfe8e0000, size 131072, enabled
bar [18] = type I/O Port, range 32, base 0xbc00, size 32, enabled
bar [1c] = type Memory, range 32, base 0xfe8dc000, size 16384, enabled
em3@pci0:5:0:0: class=0x020000 card=0x00008086 chip=0x10d38086 rev=0x00 hdr=0x00
class = network
subclass = ethernet
bar [10] = type Memory, range 32, base 0xfe9e0000, size 131072, enabled
bar [18] = type I/O Port, range 32, base 0xcc00, size 32, enabled
bar [1c] = type Memory, range 32, base 0xfe9dc000, size 16384, enabled
em4@pci0:6:0:0: class=0x020000 card=0x00008086 chip=0x10d38086 rev=0x00 hdr=0x00
class = network
subclass = ethernet
bar [10] = type Memory, range 32, base 0xfeae0000, size 131072, enabled
bar [18] = type I/O Port, range 32, base 0xdc00, size 32, enabled
bar [1c] = type Memory, range 32, base 0xfeadc000, size 16384, enabled
em5@pci0:7:0:0: class=0x020000 card=0x00008086 chip=0x10d38086 rev=0x00 hdr=0x00
class = network
subclass = ethernet
bar [10] = type Memory, range 32, base 0xfebe0000, size 131072, enabled
bar [18] = type I/O Port, range 32, base 0xec00, size 32, enabled
bar [1c] = type Memory, range 32, base 0xfebdc000, size 16384, enabled

jimp

Chip ID is the same on both of those, and on another one I'm talking to that crashes… Mine that run fine are a different chip.

clarknova

em0@pci0:2:0:0: class=0x020000 card=0x060a15d9 chip=0x10d38086 rev=0x00 hdr=0x00
class = network
subclass = ethernet
bar [10] = type Memory, range 32, base 0xfeae0000, size 131072, enabled
bar [18] = type I/O Port, range 32, base 0xdc00, size 32, enabled
bar [1c] = type Memory, range 32, base 0xfeadc000, size 16384, enabled
em1@pci0:3:0:0: class=0x020000 card=0x060a15d9 chip=0x10d38086 rev=0x00 hdr=0x00
class = network
subclass = ethernet
bar [10] = type Memory, range 32, base 0xfebe0000, size 131072, enabled
bar [18] = type I/O Port, range 32, base 0xec00, size 32, enabled
bar [1c] = type Memory, range 32, base 0xfebdc000, size 16384, enabled

Interesting, my SM board has the same chip ID. My panics come around at 6-8 day intervals with no vpn use, no known factor other than time elapsed.

jimp

LostInIgnorance - one more test for you since you seem to hit it faster than anyone else - can you try switching your OpenVPN instance to TCP instead of UDP and see if it makes any difference? Just curious.

jnorell

Just another "me too" .. panics in devd on the carp backup host. Just got them while adding firewall aliases on the primary (no recent changes to VIP addrs). I'll report if tomorrow's snapshot doesn't stop them.

jimp

vito and others with em failing, can you look at the output of this:

sysctl -a | grep 'dev.em.*fail'

And report any non-zero values. Specifically with dev.em.0.mbuf_alloc_fail and dev.em.0.tx_dma_fail