[SOLVED] pfSense 2.2(.1) crashing regularly
-
Updated my pfSense when the 2.2 update came out, never had issues with 2.1.5 or previous. Now the system crashes like once a day, sometimes multiple times.
Haven't changed anything on the system. When it first started crash I thought it way maybe because of a bad upgrade from 2.1.5 so I did a clean install, which didn't help.
Here is the crash dump I receive, I have no idea how to interpret this:
http://pastebin.com/TbwYV25D
UDPATE: I have solved this issue by rolling back to the 2.1.5 release: https://forum.pfsense.org/index.php?topic=90981.msg513475#msg513475
-
is that the complete crash dump? could you attach it as a TXT to the post?
-
is that the complete crash dump? could you attach it as a TXT to the post?
Thanks, I only saw now that it hadn't pasted the entire log (even though the preview looked fine).
I have included a pastebin link.
-
whats the "LG electronics" device on usb ?
ugen3.2: <lg electronics="" inc.=""> at usbus3 ugen3.2: <lg electronics="" inc.=""> at usbus3 (disconnected) Fatal trap 9: general protection fault while in kernel mode</lg></lg>
seen it twice right before the panic and once where it doesnt show up … could be nothing, but wouldn't hurt to unplug whatever it is.
-
whats the "LG electronics" device on usb ?
ugen3.2: <lg electronics="" inc.=""> at usbus3 ugen3.2: <lg electronics="" inc.=""> at usbus3 (disconnected) Fatal trap 9: general protection fault while in kernel mode</lg></lg>
seen it twice right before the panic and once where it doesnt show up … could be nothing, but wouldn't hurt to unplug whatever it is.
I sometimes plug my phone in to the box to charge, it's hardly ever plugged in though and never crashed while plugged in or directly after unplugging.
It's probably just easier to roll back to 2.1.5 though, I'm just really lazy :p
-
Still crashing all the time :(
Such a mission to downgrade and set everything up again.
-
Hi drzoidberg33
"never had issues before…" I use to ear a lot theses words.
The mother fact is that : ...before the problem... there was no problem...
That's THE definition of the breakdown. :-X-- Before the BREAKDOWN, it was working ... --
AFAIK, based on years of hardware experience, the hardware malfunctions strongly tend to reveal at the UPGRADE time, or other "changes" time.
(there is a physical reason to that... But that's of topic)
But what is ON Topic and I think usefull, is to remember that the hardware go "bad" far more quickly that the code...
It is it nature !Now, before to dowgrade or reinstall or hurt in the lazyness your sweetly refer in reply « on: March 22, 2015, 01:35:05 am » , may I suggest to you to CHECK the hardware ?
by EX.
RAM with MEMTEST86+
HDD with HDSCAN or SPINRITE or any else of your choiceThe crash happened always with the IP @ the SAME RAM address .... and, you had DISK crashes too...
(IP = Instruction Pointer, it is a part of the microprocessor that keep the MEMORY ADDRESS where to run the current operation/code)So, before to "invest" time in code analysis, it is IMPORTANT to be sure that the code is not unwillingly "mangled" by the hardware.
Need more help, let me know.
Cheers.
-
Any reason you're running a snapshot and not 2.2.1-release? Admittedly your snapshot is from the day before release and they were pretty much stable at that point anyway. ;)
The crashes appear to be in pfctl, do you have some bizarre rule-set? Assuming the upgrade is the issue you may have some rules that can't load in 2.2.X though that should show an error every time you load them so it seem to be something that only occasionally applies.Steve
-
@Lo:
Hi drzoidberg33
"never had issues before…" I use to ear a lot theses words.
The mother fact is that : ...before the problem... there was no problem...
That's THE definition of the breakdown. :-X-- Before the BREAKDOWN, it was working ... --
AFAIK, based on years of hardware experience, the hardware malfunctions strongly tend to reveal at the UPGRADE time, or other "changes" time.
(there is a physical reason to that... But that's of topic)
But what is ON Topic and I think usefull, is to remember that the hardware go "bad" far more quickly that the code...
It is it nature !Now, before to dowgrade or reinstall or hurt in the lazyness your sweetly refer in reply « on: March 22, 2015, 01:35:05 am » , may I suggest to you to CHECK the hardware ?
by EX.
RAM with MEMTEST86+
HDD with HDSCAN or SPINRITE or any else of your choiceThe crash happened always with the IP @ the SAME RAM address .... and, you had DISK crashes too...
(IP = Instruction Pointer, it is a part of the microprocessor that keep the MEMORY ADDRESS where to run the current operation/code)So, before to "invest" time in code analysis, it is IMPORTANT to be sure that the code is not unwillingly "mangled" by the hardware.
Need more help, let me know.
Cheers.
Thanks for taking the time to respond.
I'll run a memtest and replace the drive while I'm at it. I agree that it could be a hardware issue, so I'll check that.
I cannot make sense of these crash logs and was hoping somebody could maybe point out a specific area to check without me having to waste a bunch of time testing everything.
Cheers.
-
Any reason you're running a snapshot and not 2.2.1-release? Admittedly your snapshot is from the day before release and they were pretty much stable at that point anyway. ;)
The crashes appear to be in pfctl, do you have some bizarre rule-set? Assuming the upgrade is the issue you may have some rules that can't load in 2.2.X though that should show an error every time you load them so it seem to be something that only occasionally applies.Steve
Yes, I had updated to the latest snapshot (at the time) hoping it would help (coming from 2.2). I updated to the release version the other night but the issue remains still.
I am using pfblocker, although I tested also with only the bare essentials (vanilla install) to see if it was something like pfblocker and it would still crash, so I just enabled it again.
I'm going to try and test my hardware as suggested above, if all else fails I'll revert to 2.1.5.
-
It could be an issue with some device in your hardware that wasn't supported at all under FreeBSD 8.X but now is to some extent in 10.1. There have been a few with acpi devices due to that.
You might also try a 2.2.2 snapshot.Steve
-
It could be an issue with some device in your hardware that wasn't supported at all under FreeBSD 8.X but now is to some extent in 10.1. There have been a few with acpi devices due to that.
You might also try a 2.2.2 snapshot.Steve
Thanks! I'll try all the above suggestions, I'm really getting annoyed with these random crashes now.
-
Well I've ruled out a memory issue, ran full memtest last night and all okay.
Going to install 2.2.2 to a new drive and see how that goes, then will install 2.1.5 back to the old drive.
-
I've given up on 2.2.x.
I tried everything. Memory checks out, replaced the disk, replaced the SATA cable, changed every conceivable BIOS setting.
Things just got worse, after installing 2.2.2 once the system crashed it wouldn't be able to boot again. It would start booting but then it would take like 2 seconds for each character to appear on the screen (very weird).
Reinstalled 2.1.5 tried to replicate the same by doing a dirty power off and it booted just fine.
Happy staying with 2.1.5 for now, there is obviously something I have that 2.2.2 doesn't like.
Here is a video I took of the weird boot issue:
https://youtu.be/MPO5nAkVm7o
-
The super slow boot is a known issue. https://doc.pfsense.org/index.php/UpgradeGuide#pfSense_2.2.2_Upgrade_Notes
Disable your serial port if you're not using it.Steve
-
The super slow boot is a known issue. https://doc.pfsense.org/index.php/UpgradeGuide#pfSense_2.2.2_Upgrade_Notes
Disable your serial port if you're not using it.Steve
Thanks! The original crashing problem still exists in 2.2.2 for me though, so for now staying with 2.1.5.
-
Haven't had one crash since rolling back to 2.1.5. So it is most definitely something with my setup that 2.2.x doesn't like.
I'm going to mark this topic solved.
-
What's your configuration like? If you wouldn't mind sharing it in its entirety, that would likely be helpful. I'd like to track down what it is that's causing the crash.
-
@cmb:
What's your configuration like? If you wouldn't mind sharing it in its entirety, that would likely be helpful. I'd like to track down what it is that's causing the crash.
I'm not at home now, so can't remember the exact model of the motherboard (it's a Zotac ITX board) but here is what is being reported by the system for the rest of the hardware (the nfe0 interface is an onboard 10/100 NIC and re0 is a PCI-e x1 Realtek card):
CPU: Intel(R) Atom(TM) CPU 330 @ 1.60GHz (1600.01-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x106c2 Family = 6 Model = 1c Stepping = 2
Features=0xbfe9fbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe>Features2=0x40e31d <sse3,dtes64,mon,ds_cpl,tm2,ssse3,cx16,xtpr,pdcm,movbe>AMD Features=0x20100800 <syscall,nx,lm>AMD Features2=0x1 <lahf>TSC: P-state invariant
real memory = 4294967296 (4096 MB)
avail memory = 3572338688 (3406 MB)
ACPI APIC Table: <121009 APIC1510>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 HTT threads
cpu0 (BSP): APIC ID: 0
cpu1 (AP/HT): APIC ID: 1
cpu2 (AP): APIC ID: 2
cpu3 (AP/HT): APIC ID: 3
ioapic0: Changing APIC ID to 4
ioapic0 <version 1.1="">irqs 0-23 on motherboard
wlan: mac acl policy registered
ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff804abaf0, 0) error 1
ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff804abb90, 0) error 1
ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff804abc30, 0) error 1
kbd1 at kbdmux0
cryptosoft0: <software crypto="">on motherboard
padlock0: No ACE support.
acpi0: <121009 RSDT1510> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of fefe1000, 1000 (3) failed
acpi0: reservation of fee01000, ff000 (3) failed
acpi0: reservation of fec00000, 1000 (3) failed
acpi0: reservation of fee00000, 1000 (3) failed
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, dff00000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0
cpu0: <acpi cpu="">on acpi0
cpu1: <acpi cpu="">on acpi0
cpu2: <acpi cpu="">on acpi0
cpu3: <acpi cpu="">on acpi0
pcib0: <acpi host-pci="" bridge="">port 0xcf8-0xcff on acpi0
pci0: <acpi pci="" bus="">on pcib0
pci0: <memory, ram="">at device 0.1 (no driver attached)
isab0: <pci-isa bridge="">port 0x4f00-0x4fff at device 3.0 on pci0
isa0: <isa bus="">on isab0
pci0: <memory, ram="">at device 3.1 (no driver attached)
pci0: <serial bus,="" smbus="">at device 3.2 (no driver attached)
pci0: <memory, ram="">at device 3.3 (no driver attached)
pci0: <processor>at device 3.5 (no driver attached)
ohci0: <nvidia nforce="" mcp79="" usb="" controller="">mem 0xfae7f000-0xfae7ffff irq 22 at device 4.0 on pci0
ohci0: [ITHREAD]
usbus0: <nvidia nforce="" mcp79="" usb="" controller="">on ohci0
ehci0: <nvidia nforce="" mcp79="" usb="" 2.0="" controller="">mem 0xfae7ec00-0xfae7ecff irq 23 at device 4.1 on pci0
ehci0: [ITHREAD]
usbus1: EHCI version 1.0
usbus1: <nvidia nforce="" mcp79="" usb="" 2.0="" controller="">on ehci0
ohci1: <nvidia nforce="" mcp79="" usb="" controller="">mem 0xfae7d000-0xfae7dfff irq 20 at device 6.0 on pci0
ohci1: [ITHREAD]
usbus2: <nvidia nforce="" mcp79="" usb="" controller="">on ohci1
ehci1: <nvidia nforce="" mcp79="" usb="" 2.0="" controller="">mem 0xfae7e800-0xfae7e8ff irq 21 at device 6.1 on pci0
ehci1: [ITHREAD]
usbus3: EHCI version 1.0
usbus3: <nvidia nforce="" mcp79="" usb="" 2.0="" controller="">on ehci1
pcib1: <acpi pci-pci="" bridge="">at device 9.0 on pci0
pci1: <acpi pci="" bus="">on pcib1
nfe0: <nvidia nforce="" mcp79="" networking="" adapter="">port 0xc080-0xc087 mem 0xfae7c000-0xfae7cfff,0xfae7e400-0xfae7e4ff,0xfae7e000-0xfae7e00f irq 22 at device 10.0 on pci0</nvidia>
miibus0: <mii bus="">on nfe0
rgephy0: <rtl8169s 8110s="" 8211b="" media="" interface="">PHY 3 on miibus0
rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
nfe0: [FILTER]
atapci0: <nvidia nforce="" mcp79="" sata300="" controller="">port 0xc000-0xc007,0xbc00-0xbc03,0xb880-0xb887,0xb800-0xb803,0xb480-0xb48f mem 0xfae7a000-0xfae7bfff irq 23 at device 11.0 on pci0
atapci0: [ITHREAD]
atapci0: AHCI v1.20 controller with 6 3Gbps ports, PM supported
ata2: <ata channel="">at channel 0 on atapci0
ata2: [ITHREAD]
ata3: <ata channel="">at channel 1 on atapci0
ata3: [ITHREAD]
ata4: <ata channel="">at channel 2 on atapci0
ata4: [ITHREAD]
ata5: <ata channel="">at channel 3 on atapci0
ata5: [ITHREAD]
ata6: <ata channel="">at channel 4 on atapci0
ata6: [ITHREAD]
ata7: <ata channel="">at channel 5 on atapci0
ata7: [ITHREAD]
pcib2: <acpi pci-pci="" bridge="">irq 20 at device 12.0 on pci0
pci2: <acpi pci="" bus="">on pcib2
pcib3: <acpi pci-pci="" bridge="">at device 16.0 on pci0
pci3: <acpi pci="" bus="">on pcib3
vgapci0: <vga-compatible display="">port 0xdc00-0xdc7f mem 0xfb000000-0xfbffffff,0xe0000000-0xefffffff,0xf6000000-0xf7ffffff irq 21 at device 0.0 on pci3
pcib4: <acpi pci-pci="" bridge="">irq 21 at device 21.0 on pci0
pci4: <acpi pci="" bus="">on pcib4
pcib5: <acpi pci-pci="" bridge="">irq 22 at device 22.0 on pci0
pci5: <acpi pci="" bus="">on pcib5
re0: <realtek 8111="" 8168="" b="" c="" cp="" d="" dp="" e="" f="" pcie="" gigabit="" ethernet="">port 0xe800-0xe8ff mem 0xfebff000-0xfebfffff,0xf9ffc000-0xf9ffffff irq 16 at device 0.0 on pci5</realtek> re0: Using 1 MSI-X message
re0: Chip rev. 0x2c000000
re0: MAC rev. 0x00000000
miibus1: <mii bus="">on re0
rgephy1: <rtl8169s 8110s="" 8211b="" media="" interface="">PHY 1 on miibus1
rgephy1: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re0: [ITHREAD]
pcib6: <acpi pci-pci="" bridge="">irq 23 at device 23.0 on pci0
pci6: <acpi pci="" bus="">on pcib6
pcib7: <acpi pci-pci="" bridge="">irq 20 at device 24.0 on pci0
pci7: <acpi pci="" bus="">on pcib7
acpi_button0: <power button="">on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
acpi_hpet0: <high precision="" event="" timer="">iomem 0xfed00000-0xfed00fff irq 2,8 on acpi0
Timecounter "HPET" frequency 25000000 Hz quality 900
atrtc0: <at realtime="" clock="">port 0x70-0x71 on acpi0
sc0: <system console="">at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <generic isa="" vga="">at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
atkbdc0: <keyboard controller="" (i8042)="">at port 0x60,0x64 on isa0
atkbd0: <at keyboard="">irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
ppc0: cannot reserve I/O port range
p4tcc0: <cpu frequency="" thermal="" control="">on cpu0
p4tcc1: <cpu frequency="" thermal="" control="">on cpu1
p4tcc2: <cpu frequency="" thermal="" control="">on cpu2
p4tcc3: <cpu frequency="" thermal="" control="">on cpu3
Timecounters tick every 1.000 msec
IPsec: Initialized Security Association Processing.
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 480Mbps High Speed USB v2.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
ad4: 152587MB <samsung he160hj="" jf800-24="">at ata2-master UDMA100 SATA 3Gb/s</samsung>
ugen0.1: <nvidia>at usbus0
uhub0: <nvidia 1="" 9="" ohci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus0
ugen1.1: <nvidia>at usbus1
uhub1: <nvidia 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr="">on usbus1
ugen2.1: <nvidia>at usbus2
uhub2: <nvidia 1="" 9="" ohci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr="">on usbus2
ugen3.1: <nvidia>at usbus3
uhub3: <nvidia 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr="">on usbus3
uhub0: 6 ports with 6 removable, self powered
uhub2: 6 ports with 6 removable, self powered
uhub1: 6 ports with 6 removable, self powered
uhub3: 6 ports with 6 removable, self powered
SMP: AP CPU #3 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
Trying to mount root from ufs:/dev/ad4s1a
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS WARNING: Recommended minimum kmem_size is 512MB; expect unstable behavior.
Consider tuning vm.kmem_size and vm.kmem_size_max
in /boot/loader.conf.
ZFS filesystem version 5
ZFS storage pool version 28
coretemp0: <cpu on-die="" thermal="" sensors="">on cpu0
coretemp1: <cpu on-die="" thermal="" sensors="">on cpu1
coretemp2: <cpu on-die="" thermal="" sensors="">on cpu2
coretemp3: <cpu on-die="" thermal="" sensors="">on cpu3</cpu></cpu></cpu></cpu></nvidia></nvidia></nvidia></nvidia></nvidia></nvidia></nvidia></nvidia></cpu></cpu></cpu></cpu></at></keyboard></generic></system></at></high></power></acpi></acpi></acpi></acpi></rtl8169s></mii></acpi></acpi></acpi></acpi></vga-compatible></acpi></acpi></acpi></acpi></ata></ata></ata></ata></ata></ata></nvidia></rtl8169s></mii></acpi></acpi></nvidia></nvidia></nvidia></nvidia></nvidia></nvidia></nvidia></nvidia></processor></memory,></serial></memory,></isa></pci-isa></memory,></acpi></acpi></acpi></acpi></acpi></acpi></software></version></lahf></syscall,nx,lm></sse3,dtes64,mon,ds_cpl,tm2,ssse3,cx16,xtpr,pdcm,movbe></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,htt,tm,pbe> -
Thanks, that might be helpful, though from the crash report I'm thinking it's not hardware-specific. I was more curious what software features you're using.