PfSense Crashed

markuhde · Apr 26, 2013, 2:20 PM

Here's the crash report, got a call today after a year of stable operation (ish) that the network I have a 2.1 beta pfSense on was down. Here's the crash report, any ideas? Thanks!

Crash report begins.  Anonymous machine information:

i386
8.3-RELEASE-p7
FreeBSD 8.3-RELEASE-p7 #1: Thu Apr 25 21:20:25 EDT 2013     root@snapshots-8_3-i386.builders.pfsense.org:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.8

Crash report details:

Filename: /var/crash/bounds
1

Filename: /var/crash/info.0
Dump header from device /dev/ad4s1b
  Architecture: i386
  Architecture Version: 1
  Dump Length: 71680B (0 MB)
  Blocksize: 512
  Dumptime: Fri Apr 26 08:06:13 2013
  Hostname: glacierfire.glaciercamp
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 8.3-RELEASE-p7 #1: Thu Apr 25 21:20:25 EDT 2013
    root@snapshots-8_3-i386.builders.pfsense.org:/usr/obj.pfSense/usr/pfSensesrc/src/sys/pfSense_SMP.8
  Panic String: sbdrop
  Dump Parity: 2828757601
  Bounds: 0
  Dump Status: good

Filename: /var/crash/minfree
2048

Filename: /var/crash/textdump.tar.0
ddb.txt���������������������������������������������������������������������������������������������0600����0�������0�������140000������12136504725�  7077� �����������������������������������������������������������������������������������������������������ustar���root����������������������������wheel������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������db:0:kdb.enter.default>  run lockinfo
db:1:lockinfo> show locks
No such command
db:1:locks>  show alllocks
No such command
db:1:alllocks>  show lockedvnods
Locked vnodes
db:0:kdb.enter.default>  show pcpu
cpuid        = 0
dynamic pcpu = 0x312b80
curthread    = 0xc70178a0: pid 0 "em1 taskq"
curpcb       = 0xed99fd80
fpcurthread  = none
idlethread   = 0xc6d94000: tid 100006 "idle: cpu0"
APIC ID      = 0
currentldt   = 0x50
db:0:kdb.enter.default>  bt
Tracing pid 0 tid 100061 td 0xc70178a0
kdb_enter(c0f8d1f0,c0f8d1f0,c0f93959,ed99f99c,0,...) at kdb_enter+0x3b
panic(c0f93959,c78eda00,0,0,0,...) at panic+0x102
sbdrop_internal(c7b33284,185,313,1,ed99f9e8,...) at sbdrop_internal+0x27e
tcp_do_segment(c80e5d20,34,35,0,2,...) at tcp_do_segment+0x19a7
tcp_input(c7f8e700,14,c702c800,1,0,...) at tcp_input+0xd17
ip_input(c7f8e700,c7f8e700,10,c7f73e00,100,...) at ip_input+0x13a
netisr_dispatch_src(1,0,c7f8e700,ed99fbcc,c0b5cc6f,...) at netisr_dispatch_src+0x71
netisr_dispatch(1,c7f8e700,0,ed99fbdc) at netisr_dispatch+0x20
ether_demux(c702c800,c7f8e700,3,0,3,...) at ether_demux+0x19f
ether_input(c702c800,c7f8e700,c776f100,0,0,...) at ether_input+0x174
lem_rxeof(0,c7065100,ed99fca0,c7065100,ed99fc90,...) at lem_rxeof+0x20d
lem_handle_rxtx(c7072000,1,ed99fccc,c0aa32a5,c7065100,...) at lem_handle_rxtx+0x4f
taskqueue_run_locked(c7065100,c7065118,c0f826cf,0,ed99fcf0,...) at taskqueue_run_locked+0x6d
taskqueue_thread_loop(c70765fc,ed99fd28,30646870,0,c70765fc,...) at taskqueue_thread_loop+0x44
fork_exit(c0ad7d00,c70765fc,ed99fd28) at fork_exit+0x87
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xed99fd60, ebp = 0 ---

eri-- · Apr 26, 2013, 2:20 PM

Please try a snapshot coming today later on.

markuhde · Apr 26, 2013, 2:34 PM

@ermal:

Please try a snapshot coming today later on.

Okay thanks because not only has it apparently crashed, pfSense is also totally non-functional up there. I can get in from my end fine, but no luck for them!

podilarius · Apr 26, 2013, 3:16 PM

yeah .. mine is in a crash loop also … I will have to rebuild when I get home. Running a basic config off an old CD.

Update: Meant to add that it seems to be only the i386 machines that are affected.

eri-- · Apr 26, 2013, 3:19 PM

You running CP by any chance?

podilarius · Apr 26, 2013, 3:48 PM

I am not running CP. I am running squid, rrd summary, git and pfblocker. I am also running traffic shaping and powerd. I just completed an upgrade from Last friday to this morning's latest.

markuhde · Apr 26, 2013, 4:14 PM

I'm not running CP either. I got the system up and running by rolling back to a snap from a few days ago. When it wasn't crashed but not working I had a large number of "No Traffic" states.

WhiteSpy · Apr 26, 2013, 7:46 PM

got the same problem here with that snapshot (i386) and not running anything special - just basic config + OpenVPN.

razzfazz · Apr 26, 2013, 8:16 PM

I'm seeing crashed on an AMD64 machine as well.

jcyr · Apr 26, 2013, 9:41 PM

Yep. crashed here too after upgrading at 5PM EST.

monkfish · Apr 26, 2013, 10:59 PM

Mee too!!

from 2.1-BETA1 (i386) built on Thu Apr 25 09:08:19 EDT 2013
to 2.1-BETA1 (i386) built on Thu Apr 25 20:52:41 EDT 2013

stays up for about 4 minutes then crash/automatic reset.

Crash dump submitted from gui, hope it got there.

Thanks for pfsense!

mikepell · Apr 26, 2013, 11:42 PM

same here.

2.1-BETA1 (i386)
built on Thu Apr 25 20:52:41 EDT 2013
FreeBSD 8.3-RELEASE-p7

reboots about every 5-10 minutes.

I happened to catch this message right before reboot in case it matters:

panic: hfsc_dequeue

I also submitted a crash report.

Clear-Pixel · Apr 27, 2013, 2:38 AM

I noticed some stability issues few weeks back and made a post but every one was saying there was no problem…. don't know if it was related to the OP issues or the problems others are having here.

Different symptoms, but couple of weeks back it locked up 3 or 4 times over the course of 2 weeks or so. Knock on Wood it has since been stable other than stuck wireless beacon and errors which at some point I'll try to figure out what the deal is with that. The wireless is functioning with the errors but when the beacon gets stuck it requires a reboot.

wernerj · Apr 27, 2013, 2:33 AM

I'm seeing very similar crashes (and one complete freeze with only "em0: discard frame without header" on the console) with the Thu Apr 25 21:20:47 EDT 2013 build, which at the moment is the latest one. I just updated from a version from Tuesday where I did not see that problem. Crashes uploaded from GUI.

/wj

wernerj · Apr 27, 2013, 2:50 AM

Another failure mode is the text "ipsec_filter: m_pullup failed" on the console and then a hard hang, no crash or reboot. There seems to be something seriously wrong with this kernel.

/wj

reslip · Apr 27, 2013, 6:56 AM

I'm getting the same failure as well, but there is not auto reboot. The system just hangs after a few hours and I have to manually reset it using the hw reset switch

markuhde · Apr 27, 2013, 7:13 AM

I'm guessing at this point we can be pretty confident something just went awfully, horribly wrong :) Thanks pfSense team for all your hard work, it's times like these that we appreciate how much goes into making a project like this happen!

mdima · Apr 27, 2013, 9:24 AM

Same here… from an Alix board, looking at the console, the system is locked in boot-looping, restarting at some ramdom points... :S

mdima · Apr 27, 2013, 10:37 AM

Ok, I post just to help someone in the case. The easiest way I found to recover pfSense to a working state was restoring a full backup from an image I made some time ago.
Just connect to the console, control+break the system while it is initializing, and run:

/etc/rc.restore_full_backup /root/pfSense-full-backup-20130219-2240.tgz
(just change pfSense-full-backup-20130219-2240.tgz with the filename of the full backup).

Then, just wait for the process to complete and when it's done run: reboot

Finally, you can restore the latest configuration from the webadmin interface.

Ciao,
Michele

xbipin · Apr 27, 2013, 11:58 AM

crashes here also and i guess no new snaps also being generated as builder seems down