Pfsense 2.0.2 crashes daily

rclaus

i am running pfsense 2.02 on a vm hosted by ubuntu 12.04 server. almost daily the firewall will crash and reboot for reasons unknown to me. the vm is a amd64 type architecture and has 512k of memory with single processor. i have 4 intel em(4) type nics and i saw the pfsense advisary about these nics in amd64 machine having issues and i modified the /boot/loader.conf.local file with the setting but has not helped.
was curious if anyone had any ideas about where i may look for the issue? normally in the crash log it references issue with one of the nics as seen below in the log snippet

Fatal trap 9: general protection fault while in kernel mode
cpuid = 0; apic id = 00
instruction pointer = 0x20:0xffffffff807cad35
stack pointer = 0x28:0xffffff800010b4c0
frame pointer = 0x28:0xffffff800010b510
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 0 (em3 taskq)

rclaus

i also uploaded several crash reports and my public ip is 173.170.78.143
thanks

cmb

There are no crash reports from that IP nor anything close to it. Something upstream of you changing your IP? Are you dual stacked and crash reporting via IPv6?

rclaus

Crash report begins. Anonymous machine information:

amd64
8.1-RELEASE-p13
FreeBSD 8.1-RELEASE-p13 #1: Fri Dec 7 23:07:32 EST 2012 root@snapshots-8_1-amd64.builders.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_SMP.8

Crash report details:

Filename: /var/crash/bounds
1

Filename: /var/crash/info.0
Dump header from device /dev/ad0s1b
Architecture: amd64
Architecture Version: 1
Dump Length: 66560B (0 MB)
Blocksize: 512
Dumptime: Wed Mar 13 13:15:28 2013
Hostname: fw01data.prod.tracsoftware.com
Magic: FreeBSD Text Dump
Version String: FreeBSD 8.1-RELEASE-p13 #1: Fri Dec 7 23:07:32 EST 2012
root@snapshots-8_1-amd64.builders.pfsense.org:/usr/obj./usr/pfSensesrc/src/sys/pfSense_SMP.8
Panic String:
Dump Parity: 43261262
Bounds: 0
Dump Status: good

Filename: /var/crash/textdump.tar.0
ddb.txt��0600��0��0��140000��12120132060� 7055� ��ustar��root��wheel��db:0:kdb.enter.default> run lockinfo
db:1:lockinfo> show locks
No such command
db:1:locks> show alllocks
No such command
db:1:alllocks> show lockedvnods
Locked vnodes
db:0:kdb.enter.default> show pcpu
cpuid = 0
dynamic pcpu = 0x2fc080
curthread = 0xffffff0002514000: pid 0 "em1 taskq"
curpcb = 0xffffff8000142d40
fpcurthread = none
idlethread = 0xffffff00022a77c0: pid 11 "idle: cpu0"
curpmap = 0
tssp = 0xffffffff811ddd80
commontssp = 0xffffffff811ddd80
rsp0 = 0xffffff8000142d40
gs32p = 0xffffffff811dcbb8
ldt = 0xffffffff811dcbf8
tss = 0xffffffff811dcbe8
db:0:kdb.enter.default> bt
Tracing pid 0 tid 64034 td 0xffffff0002514000
rn_match() at rn_match+0x25
pfr_match_addr() at pfr_match_addr+0xcb
pf_test_tcp() at pf_test_tcp+0xaa1
pf_test() at pf_test+0x20fe
pf_check_in() at pf_check_in+0x39
pfil_run_hooks() at pfil_run_hooks+0xa2
ip_input() at ip_input+0x34e
netisr_dispatch_src() at netisr_dispatch_src+0x7b
ether_demux() at ether_demux+0x169
ether_input() at ether_input+0x174
lem_rxeof() at lem_rxeof+0x24d
lem_handle_rxtx() at lem_handle_rxtx+0x51
taskqueue_run() at taskqueue_run+0x93
taskqueue_thread_loop() at taskqueue_thread_loop+0x46
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
–- trap 0, rip = 0, rsp = 0xffffff8000142d30, rbp = 0 ---
db:0:kdb.enter.default> ps
pid ppid pgrp uid state wmesg wchan cmd
15717 15131 30020 0 S piperd 0xffffff0009d81888 cut
15496 15131 30020 0 S piperd 0xffffff00026935b0 grep
15477 15131 30020 0 S piperd 0xffffff0009fcc5b0 awk
15184 15131 30020 0 S select 0xffffff00096a60c0 ping
15131 58577 30020 0 S wait 0xffffff0009fdf460 sh
58577 30347 30020 0 S piperd 0xffffff0002694000 sh
53194 1 53194 0 Ss (threaded) filterdns
64140 S nanslp 0xffffffff8115cc28 filterdns
64139 S nanslp 0xffffffff8115cc28 filterdns
64138 S nanslp 0xffffffff8115cc28 filterdns
64137 S nanslp 0xffffffff8115cc28 filterdns
64124 S nanslp 0xffffffff8115cc28 filterdns
64123 S nanslp 0xffffffff8115cc28 filterdns
64122 S nanslp 0xffffffff8115cc28 filterdns
64121 S nanslp 0xffffffff8115cc28 filterdns
64120 S nanslp 0xffffffff8115cc28 filterdns
64119 S nanslp 0xffffffff8115cc28 filterdns
64118 S nanslp 0xffffffff8115cc28 filterdns
64117 S nanslp 0xffffffff8115cc28 filterdns
64116 S nanslp 0xffffffff8115cc28 filterdns
64090 S nanslp 0xffffffff8115cc28 filterdns
64089 S nanslp 0xffffffff8115cc28 filterdns
64088 S nanslp 0xffffffff8115cc28 filterdns
64087 S nanslp 0xffffffff8115cc28 filterdns
64086 S nanslp 0xffffffff8115cc28 filterdns
64085 S nanslp 0xffffffff8115cc28 filterdns
64084 S nanslp 0xffffffff8115cc28 filterdns
64083 S nanslp 0xffffffff8115cc28 filterdns
64082 S nanslp 0xffffffff8115cc28 filterdns
64081 S nanslp 0xffffffff8115cc28 filterdns
64080 S nanslp 0xffffffff8115cc28 filterdns
64133 S uwait 0xffffff00096d5d00 filterdns
39339 53340 24 0 S nanslp 0xffffffff8115cc28 sleep
16856 21207 20993 0 S accept 0xffffff0009cf6b06 initial thread
37887 52950 52950 0 S piperd 0xffffff00099b9b60 rrdtool
51997 51805 42649 2010 S select 0xffffff00096d5140 tinydns
51805 42840 42649 0 S select 0xffffff000997e440 supervise
47218 43044 47218 0 S+ ttyin 0xffffff00025968a8 sh
45695 43254 45695 0 S+ ttyin 0xffffff00025964a8 sh
43254 42580 43254 0 S+ wait 0xffffff0009962460 sh
43044 42817 43044 0 S+ wait 0xffffff00099f9460 sh
42931 43260 42931 0 Ss (threaded) sshlockout_pf
64115 S nanslp 0xffffffff8115cc28 sshlockout_pf
64108 S piperd 0xffffff00026932d8 initial thread
42817 1 42817 0 Ss+ wait 0xffffff0009961460 login
42580 1 42580 0 Ss+ wait 0xffffff00099fa460 login
42215 1 42215 0 Ss select 0xffffff0009ef6c40 ntpd
39037 43291 42649 2011 S piperd 0xffffff00099b92d8 multilog
33258 33203 33203 0 S nanslp 0xffffffff8115cc28 svscan
33203 1 33203 0 Ss wait 0xffffff000272b460 sh
31727 31094 31094 0 S nanslp 0xffffffff8115cc28 minicron
31094 1 31094 0 Ss wait 0xffffff0009a188c0 minicron
30869 30465 30465 0 S nanslp 0xffffffff8115cc28 minicron
30465 1 30465 0 Ss wait 0xffffff00099c8460 minicron
30347 30020 30020 0 S wait 0xffffff0009c2a460 minicron
30020 1 30020 0 Ss wait 0xffffff00099fb000 minicron
54657 1 54657 0 Ss nanslp 0xffffffff8115cc28 cron
53340 1 24 0 S+ wait 0xffffff00099c7000 sh
43291 42840 42649 0 S select 0xffffff00096d4dc0 supervise
43116 42649 42649 0 S piperd 0xffffff00099b9000 multilog
42840 42649 42649 0 S nanslp 0xffffffff8115cc28 svscan
42649 1 42649 0 Ss wait 0xffffff00099ca8c0 sh
39612 1 39612 0 Ss select 0xffffff000979b9c0 racoon
36104 1 36104 0 Ss (threaded) filterdns
64184 S uwrlck 0xffffff0009c68680 filterdns
64129 S uwait 0xffffff00096d5a80 filterdns
33235 1 32915 65534 S select 0xffffff0009c87bc0 dnsmasq
32174 1 32174 1002 Ss select 0xffffff00096d55c0 dhcpd
21623 21207 20993 0 S accept 0xffffff000994485e initial thread
21207 1 20993 0 S kqread 0xffffff0009bf4100 lighttpd
14440 1 14440 0 Ss (threaded) filterdns
64125 S uwrlck 0xffffff00096d4900 filterdns
64094 S uwait 0xffffff000979a800 filterdns
52950 1 52950 0 Ss select 0xffffff00096a6840 apinger
48071 1 48071 0 Ss (threaded) filterdns
64091 S uwrlck 0xffffff000979a680 filterdns
64059 S uwait 0xffffff0002616700 filterdns
45736 1 45736 0 Ss select 0xffffff00026177c0 inetd
45274 1 45274 0 Ss select 0xffffff00096a63c0 openvpn
43711 1 24 0 S+ piperd 0xffffff0002693000 logger
43661 1 24 0 S+ bpf 0xffffff0009724600 tcpdump
43260 1 43260 0 Ss select 0xffffff00096a6240 syslogd
8634 1 8634 0 Ss select 0xffffff0007f5f840 sshd
259 1 259 0 Ss select 0xffffff0002617240 devd
245 243 243 0 S kqread 0xffffff0002662000 check_reload_status
243 1 243 0 Ss kqread 0xffffff000265e600 check_reload_status
39 0 0 0 SL mdwait 0xffffff000264a800 [md0]
23 0 0 0 SL sdflush 0xffffffff811a3738 [softdepflush]
22 0 0 0 SL vlruwt 0xffffff000261a000 [vnlru]
21 0 0 0 SL syncer 0xffffffff81180e40 [syncer]
20 0 0 0 SL psleep 0xffffffff81180968 [bufdaemon]
19 0 0 0 SL pollid 0xffffffff8115ba48 [idlepoll]
18 0 0 0 SL pgzero 0xffffffff811a51cc [pagezero]
17 0 0 0 SL psleep 0xffffffff811a4568 [vmdaemon]
16 0 0 0 SL psleep 0xffffffff811a452c [pagedaemon]
9 0 0 0 SL ccb_scan 0xffffffff811229e0 [xpt_thrd]
8 0 0 0 SL pftm 0xffffffff80204dd0 [pfpurge]
7 0 0 0 SL waiting_ 0xffffffff8118ce00 [sctp_iterator]
15 0 0 0 SL (threaded) usb
64032 D - 0xffffff8000234ef0 [usbus0]
64031 D - 0xffffff8000234e98 [usbus0]
64030 D - 0xffffff8000234e40 [usbus0]
64029 D - 0xffffff8000234de8 [usbus0]
14 0 0 0 SL - 0xffffffff8115c904 [yarrow]
6 0 0 0 SL crypto_r 0xffffffff811a2660 [crypto returns]
5 0 0 0 SL crypto_w 0xffffffff811a2620 [crypto]
4 0 0 0 SL - 0xffffffff811586e8 [g_down]
3 0 0 0 SL - 0xffffffff811586e0 [g_up]
2 0 0 0 SL - 0xffffffff811586d0 [g_event]
13 0 0 0 SL sleep 0xffffffff810d19d0 [ng_queue0]
12 0 0 0 RL (threaded) intr
64040 I [swi0: uart]
64039 I [irq7: ppc0]
64038 I [irq12: psm0]
64037 I [irq1: atkbd0]
64028 RunQ [irq11: em0 em1+]
64027 I [irq15: ata1]
64026 I [irq14: ata0]
64025 I [irq9: acpi0]
64023 I [swi5: +]
64021 I [swi2: cambio]
64017 I [swi6: task queue]
64016 I [swi6: Giant taskq]
64007 I [swi3: vm]
64006 RunQ [swi4: clock]
64005 RunQ [swi1: netisr 0]
11 0 0 0 RL [idle: cpu0]
1 0 1 0 SLs wait 0xffffff00022a48c0 [init]
10 0 0 0 SL audit_wo 0xffffffff811a2b90 [audit]
0 0 0 0 RLs (threaded) kernel
64036 D - 0xffffff000254ba00 [em3 taskq]
64035 D - 0xffffff0002546280 [em2 taskq]
64034 Run CPU 0 [em1 taskq]
64033 RunQ [em0 taskq]
64024 D - 0xffffff0002444180

rclaus

firewall public ip is 199.255.156.227

charlien

@rclaus:

I have 4 intel em(4) type nics and i saw the pfsense advisary about these nics in amd64 machine having issues and i modified the /boot/loader.conf.local file with the setting but has not helped.

I was planning on using intel nics in my new setup. Where can I get more information about this?

ptt

@charlien:

I was planning on using intel nics in my new setup. Where can I get more information about this?

Just check the Docs ;D

http://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards

chpalmer

512MB of RAM total on that machine? Or allocated by the VM?

As jimp said in the other post- Id put 2.0.3 prerelease on it and see if things change.

I have several boxes with both 2.0.3 and 2.1 running and none of them ever crash. But Im not running under a VM so can't speak for that.

Supermule

I use 32 bit VM's with 2.0.2 and no issues regarding crashes. Only Snort is pulling my leg right now…

rclaus

the vm has 768mb of ram. i would like to try 2.0.3 but cannot find where i can download it from. could someone tell me where i can obtain 2.0.3?
thanks so much for the help

chpalmer

http://forum.pfsense.org/index.php/topic,60151.msg323778.html#msg323778

Are you running anything else on the physical machine? The RAM seems very anemic to me. Though I still have XP sp1 desktops (non internet) running 256MB so Im probably wrong.

tim.mcmanus

I have a somewhat detailed set of instructions for doing a manual firmware upgrade to the 2.0.3 release. See the middle of this chunky post:

http://forum.pfsense.org/index.php/topic,58933.msg316734.html#msg316734

Start reading at the paragraph that begins with "Now, updating to a more current version requires a little bit of searching but it's a handy thing to know. "

rclaus

the host server for the vm has 12 other instances of vm's so i had thought 768mb good size but can bump it up some.
thanks for all the information.

rclaus

i upgraded to 2.0.3 this morning and i just had a kernel panic on the fw and has rebooted from the crash.
i uploaded the crash dump and the fw ip should be 199.255.156.227 although the carp virtual ip for the wan is 199.255.156.226
was hoping someone could look at the dumps to see what is going on with my setup.
thx

rclaus

this issue is resolved now. i installed 2.1 beta version which has newer release of FreeBSD and updated Intel nic drivers and i am having no more daily crashes.
i assume the older version of FreeBSD that the 2.0.x pfSense uses had bad em(4) drivers :) the 2.1 beta version is performing well for me.
thanks
Richard