APU2E2 crashing
-
Have been experiencing crashes lately. APU2E2, 2gb ram, 120gb ssd, coreboot V: v4.11.0.6, pfSense v2.5.1. Some have been hard locks requiring a hard reset. I have 5 sets of files - how do I send these to the devs for a quick scan to see what's going on?
-
Check the backtraces at the start of the ddb file headed
> btIf they all look the same or very similar it probably a software problem. If they are all different it's probably hardware.
Just as an example:
db:0:kdb.enter.default> bt Tracing pid 12 tid 100194 td 0xfffff8000baf0620 ip_output() at ip_output+0x1418/frame 0xfffffe0857f3d1e0 ipsec_process_done() at ipsec_process_done+0x1c8/frame 0xfffffe0857f3d230 esp_output_cb() at esp_output_cb+0xeb/frame 0xfffffe0857f3d290 aesni_process() at aesni_process+0x151/frame 0xfffffe0857f3d340 crypto_dispatch() at crypto_dispatch+0x140/frame 0xfffffe0857f3d380 esp_output() at esp_output+0x5cc/frame 0xfffffe0857f3d420 ipsec4_perform_request() at ipsec4_perform_request+0x37f/frame 0xfffffe0857f3d4c0 ipsec4_forward() at ipsec4_forward+0x5a/frame 0xfffffe0857f3d4f0 ip_forward() at ip_forward+0x221/frame 0xfffffe0857f3d590 ip_input() at ip_input+0x72a/frame 0xfffffe0857f3d5f0 netisr_dispatch_src() at netisr_dispatch_src+0xa8/frame 0xfffffe0857f3d640 ether_demux() at ether_demux+0x173/frame 0xfffffe0857f3d670 ether_nh_input() at ether_nh_input+0x32b/frame 0xfffffe0857f3d6d0 netisr_dispatch_src() at netisr_dispatch_src+0xa8/frame 0xfffffe0857f3d720 ether_input() at ether_input+0x26/frame 0xfffffe0857f3d740 vlan_input() at vlan_input+0x215/frame 0xfffffe0857f3d7f0 ether_demux() at ether_demux+0x15c/frame 0xfffffe0857f3d820 ether_nh_input() at ether_nh_input+0x32b/frame 0xfffffe0857f3d880 netisr_dispatch_src() at netisr_dispatch_src+0xa8/frame 0xfffffe0857f3d8d0 ether_input() at ether_input+0x26/frame 0xfffffe0857f3d8f0 t4_eth_rx() at t4_eth_rx+0xa6/frame 0xfffffe0857f3d910 service_iq() at service_iq+0x39c/frame 0xfffffe0857f3d9c0 t4_intr() at t4_intr+0x30/frame 0xfffffe0857f3d9e0 intr_event_execute_handlers() at intr_event_execute_handlers+0xe9/frame 0xfffffe0857f3da20 ithread_loop() at ithread_loop+0xe7/frame 0xfffffe0857f3da70 fork_exit() at fork_exit+0x83/frame 0xfffffe0857f3dab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0857f3dab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 ---You can post that here as there's nothing distinguishing there.
Steve
-
@stephenw10
.0:
db:0:kdb.enter.default> bt
Tracing pid 40534 tid 100184 td 0xfffff8005f999740
kdb_enter() at kdb_enter+0x37/frame 0xfffffe001cc540c0
vpanic() at vpanic+0x197/frame 0xfffffe001cc54110
panic() at panic+0x43/frame 0xfffffe001cc54170
trap_fatal() at trap_fatal+0x391/frame 0xfffffe001cc541d0
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe001cc54220
trap() at trap+0x286/frame 0xfffffe001cc54330
calltrap() at calltrap+0x8/frame 0xfffffe001cc54330
--- trap 0xc, rip = 0xffffffff80e54d9b, rsp = 0xfffffe001cc54400, rbp = 0xfffffe001cc544b0 ---
cache_lookup() at cache_lookup+0x29b/frame 0xfffffe001cc544b0
vfs_cache_lookup() at vfs_cache_lookup+0x81/frame 0xfffffe001cc54500
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x75/frame 0xfffffe001cc54530
lookup() at lookup+0x781/frame 0xfffffe001cc545e0
namei() at namei+0x544/frame 0xfffffe001cc546e0
kern_execve() at kern_execve+0x1b9/frame 0xfffffe001cc54a40
sys_execve() at sys_execve+0x51/frame 0xfffffe001cc54ac0
amd64_syscall() at amd64_syscall+0x387/frame 0xfffffe001cc54bf0
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe001cc54bf0
--- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x80039feaa, rsp = 0x7fffffffe3e8, rbp = 0x7fffffffe4f0 ---.1:
db:0:kdb.enter.default> bt
Tracing pid 45812 tid 100248 td 0xfffff8002326f000
kdb_enter() at kdb_enter+0x37/frame 0xfffffe001cd640c0
vpanic() at vpanic+0x197/frame 0xfffffe001cd64110
panic() at panic+0x43/frame 0xfffffe001cd64170
trap_fatal() at trap_fatal+0x391/frame 0xfffffe001cd641d0
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe001cd64220
trap() at trap+0x286/frame 0xfffffe001cd64330
calltrap() at calltrap+0x8/frame 0xfffffe001cd64330
--- trap 0xc, rip = 0xffffffff8154ba50, rsp = 0xfffffe001cd64400, rbp = 0xfffffe001cd64420 ---
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x20/frame 0xfffffe001cd64420
_vn_lock() at _vn_lock+0x44/frame 0xfffffe001cd64480
vget() at vget+0x4a/frame 0xfffffe001cd644d0
cache_lookup() at cache_lookup+0x4c7/frame 0xfffffe001cd64590
vfs_cache_lookup() at vfs_cache_lookup+0x81/frame 0xfffffe001cd645e0
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x75/frame 0xfffffe001cd64610
lookup() at lookup+0x781/frame 0xfffffe001cd646c0
namei() at namei+0x544/frame 0xfffffe001cd647c0
kern_statat() at kern_statat+0x97/frame 0xfffffe001cd649c0
sys_fstatat() at sys_fstatat+0x2f/frame 0xfffffe001cd64ac0
amd64_syscall() at amd64_syscall+0x387/frame 0xfffffe001cd64bf0
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe001cd64bf0
--- syscall (552, FreeBSD ELF64, sys_fstatat), rip = 0x80168f69a, rsp = 0x7fffdc7e1c98, rbp = 0x7fffdc7e1d40 ---.2:
db:0:kdb.enter.default> bt
Tracing pid 72444 tid 100277 td 0xfffff800610d5740
kdb_enter() at kdb_enter+0x37/frame 0xfffffe001cde4db0
vpanic() at vpanic+0x197/frame 0xfffffe001cde4e00
panic() at panic+0x43/frame 0xfffffe001cde4e60
trap_fatal() at trap_fatal+0x391/frame 0xfffffe001cde4ec0
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe001cde4f10
trap() at trap+0x286/frame 0xfffffe001cde5020
calltrap() at calltrap+0x8/frame 0xfffffe001cde5020
--- trap 0xc, rip = 0xffffffff80d74a74, rsp = 0xfffffe001cde50f0, rbp = 0xfffffe001cde5170 ---
__lockmgr_args() at __lockmgr_args+0x134/frame 0xfffffe001cde5170
ffs_lock() at ffs_lock+0x7c/frame 0xfffffe001cde51c0
VOP_LOCK1_APV() at VOP_LOCK1_APV+0x7b/frame 0xfffffe001cde51f0
_vn_lock() at _vn_lock+0x44/frame 0xfffffe001cde5250
vget() at vget+0x4a/frame 0xfffffe001cde52a0
cache_lookup() at cache_lookup+0x4c7/frame 0xfffffe001cde5360
vfs_cache_lookup() at vfs_cache_lookup+0x81/frame 0xfffffe001cde53b0
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x75/frame 0xfffffe001cde53e0
lookup() at lookup+0x781/frame 0xfffffe001cde5490
namei() at namei+0x544/frame 0xfffffe001cde5590
elf64_load_file() at elf64_load_file+0xc1/frame 0xfffffe001cde55f0
exec_elf64_imgact() at exec_elf64_imgact+0x109f/frame 0xfffffe001cde56e0
kern_execve() at kern_execve+0x728/frame 0xfffffe001cde5a40
sys_execve() at sys_execve+0x51/frame 0xfffffe001cde5ac0
amd64_syscall() at amd64_syscall+0x387/frame 0xfffffe001cde5bf0
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe001cde5bf0
--- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x80039feaa, rsp = 0x7fffffffe3e8, rbp = 0x7fffffffe4f0 ---.3:
db:0:kdb.enter.default> bt
Tracing pid 52395 tid 100277 td 0xfffff8004f45c740
kdb_enter() at kdb_enter+0x37/frame 0xfffffe001cdf4250
vpanic() at vpanic+0x197/frame 0xfffffe001cdf42a0
panic() at panic+0x43/frame 0xfffffe001cdf4300
trap_fatal() at trap_fatal+0x391/frame 0xfffffe001cdf4360
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe001cdf43b0
trap() at trap+0x286/frame 0xfffffe001cdf44c0
calltrap() at calltrap+0x8/frame 0xfffffe001cdf44c0
--- trap 0xc, rip = 0xffffffff8120850c, rsp = 0xfffffe001cdf4590, rbp = 0xfffffe001cdf4590 ---
vm_radix_lookup() at vm_radix_lookup+0x2c/frame 0xfffffe001cdf4590
vm_page_grab() at vm_page_grab+0x4a/frame 0xfffffe001cdf4610
exec_map_first_page() at exec_map_first_page+0x9b/frame 0xfffffe001cdf46e0
kern_execve() at kern_execve+0x273/frame 0xfffffe001cdf4a40
sys_execve() at sys_execve+0x51/frame 0xfffffe001cdf4ac0
amd64_syscall() at amd64_syscall+0x387/frame 0xfffffe001cdf4bf0
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe001cdf4bf0
--- syscall (59, FreeBSD ELF64, sys_execve), rip = 0x80036eeaa, rsp = 0x7fffffffea18, rbp = 0x7fffffffeb20 ---.4:
db:0:kdb.enter.default> bt
Tracing pid 28070 tid 100289 td 0xfffff800542fb740
kdb_enter() at kdb_enter+0x37/frame 0xfffffe001ce28540
vpanic() at vpanic+0x197/frame 0xfffffe001ce28590
panic() at panic+0x43/frame 0xfffffe001ce285f0
trap_fatal() at trap_fatal+0x391/frame 0xfffffe001ce28650
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe001ce286a0
trap() at trap+0x286/frame 0xfffffe001ce287b0
calltrap() at calltrap+0x8/frame 0xfffffe001ce287b0
--- trap 0xc, rip = 0xffffffff80c21a08, rsp = 0xfffffe001ce28880, rbp = 0xfffffe001ce288d0 ---
devfs_populate_loop() at devfs_populate_loop+0x198/frame 0xfffffe001ce288d0
devfs_cleanup() at devfs_cleanup+0x1d/frame 0xfffffe001ce288f0
devfs_unmount() at devfs_unmount+0x74/frame 0xfffffe001ce28930
dounmount() at dounmount+0x555/frame 0xfffffe001ce289a0
vfs_unmountall() at vfs_unmountall+0xc1/frame 0xfffffe001ce289d0
bufshutdown() at bufshutdown+0x2fc/frame 0xfffffe001ce28a20
kern_reboot() at kern_reboot+0x20d/frame 0xfffffe001ce28a70
sys_reboot() at sys_reboot+0x3e5/frame 0xfffffe001ce28ac0
amd64_syscall() at amd64_syscall+0x387/frame 0xfffffe001ce28bf0
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe001ce28bf0
--- syscall (55, FreeBSD ELF64, sys_reboot), rip = 0x8002d528a, rsp = 0x7fffffffe888, rbp = 0x7fffffffea00 ---.5:
db:0:kdb.enter.default> bt
Tracing pid 7714 tid 100146 td 0xfffff80021b2f740
kdb_enter() at kdb_enter+0x37/frame 0xfffffe00005af1d0
vpanic() at vpanic+0x197/frame 0xfffffe00005af220
panic() at panic+0x43/frame 0xfffffe00005af280
trap_fatal() at trap_fatal+0x391/frame 0xfffffe00005af2e0
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00005af330
trap() at trap+0x286/frame 0xfffffe00005af440
calltrap() at calltrap+0x8/frame 0xfffffe00005af440
--- trap 0xc, rip = 0xffffffff80e54e6b, rsp = 0xfffffe00005af510, rbp = 0xfffffe00005af5c0 ---
cache_lookup() at cache_lookup+0x2bb/frame 0xfffffe00005af5c0
vfs_cache_lookup() at vfs_cache_lookup+0x81/frame 0xfffffe00005af610
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x75/frame 0xfffffe00005af640
lookup() at lookup+0x781/frame 0xfffffe00005af6f0
namei() at namei+0x544/frame 0xfffffe00005af7f0
vn_open_cred() at vn_open_cred+0x26c/frame 0xfffffe00005af940
kern_openat() at kern_openat+0x213/frame 0xfffffe00005afac0
amd64_syscall() at amd64_syscall+0x387/frame 0xfffffe00005afbf0
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00005afbf0
--- syscall (499, FreeBSD ELF64, sys_openat), rip = 0x8003f427a, rsp = 0x7fffdfdfc968, rbp = 0x7fffdfdfc9e0 --- -
Hmm, well those all look like filesystem issues of various types.
The first thing I would do there is re-install and restore the config and see and that corrects it.
If it doesn't then consider swapping out the drive. It's possible to run from SD card with some care.
Steve
-
@stephenw10 Thanks much. I went direct purchase with their "preconfigured" system 6 months ago; was hoping for no issues on that front - but will buy one to swap and see how that goes. Thanks!!!
-
Who's preconfigured system was that? I assume not from PCEngines directly.
You didn't have to install originally? That is UFS on mSATA?
Steve
-
@stephenw10 From PCEngines directly. I had them put the m2sata, memory and case on prior to shipping - because I can do it, but I figured if there are issues - would make things simpler?
I uninstalled the ntopng package - use that sporadically to check out network utilization/trends. Crashing has not occurred in the last 4 days.
-
Hmm, interesting. That crash is not obviously that at all. Maybe triggering a filesystem issue I guess.
-
@stephenw10 Right. I'll do a fresh install here in a few days; just haven't had a chance to take stuff down long enough to do it.
-
@stephenw10 Had 2 crashes in a row within 30 minutes this morning; installed 2.5.1 on an old PC and am using that for now. Re-installed fresh 2.5.1 on the apu2; 3 hours of run time and just now heard the "boot sequence" beeps out of the apu2 which means it crashed again. I'll have to take it apart, reseat stuff and then run through a mem test. Ran the extended smart diags against the drive and no errors showed, smart status is good....
-
Was it still running the default config? Pretty much has to be hardware if so.
