APU1C Kernel Panic
-
I was using 2.1.4 on my APU1C boxes and thought it was time to upgrade a few weeks ago. I wound up having some stability issues with 2.2.4 so I've updated my first box to 2.2.5 and am still having the same problems. Luckily, I managed to get my laptop hooked in with a null-modem cable and was able to see the output. I think it was a kernel panic as I was constantly seeing:
mi_switch() at mi_switch+0xe1/frame 0xfffffe001a3b3af0
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe001a3b3b20
msleep_spin_sbt() at msleep_spin_sbt+0x1a3/frame 0xfffffe001a3b3b80
taskqueue_thread_loop() at taskqueue_thread_loop+0xfd/frame 0xfffffe001a3b3bb0
fork_exit() at fork_exit+0x9a/frame 0xfffffe001a3b3bf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001a3b3bf0
–- trap 0, rip = 0, rsp = 0xfffffe001a3b3cb0, rbp = 0 ---Tracing command kernel pid 0 tid 100018 td 0xfffff80003224000
sched_switch() at sched_switch+0x2b3/frame 0xfffffe001a3aeab0
mi_switch() at mi_switch+0xe1/frame 0xfffffe001a3aeaf0
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe001a3aeb20
msleep_spin_sbt() at msleep_spin_sbt+0x1a3/frame 0xfffffe001a3aeb80
taskqueue_thread_loop() at taskqueue_thread_loop+0xfd/frame 0xfffffe001a3aebb0
fork_exit() at fork_exit+0x9a/frame 0xfffffe001a3aebf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001a3aebf0
--- trap 0, rip = 0, rsp = 0xfffffe001a3aecb0, rbp = 0 ---Tracing command kernel pid 0 tid 100014 td 0xfffff80003225490
sched_switch() at sched_switch+0x2b3/frame 0xfffffe001a39aa90
mi_switch() at mi_switch+0xe1/frame 0xfffffe001a39aad0
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe001a39ab00
_sleep() at _sleep+0x287/frame 0xfffffe001a39ab80
taskqueue_thread_loop() at taskqueue_thread_loop+0xd5/frame 0xfffffe001a39abb0
fork_exit() at fork_exit+0x9a/frame 0xfffffe001a39abf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001a39abf0
--- trap 0, rip = 0, rsp = 0xfffffe001a39acb0, rbp = 0 ---scrolling across my screen. Since /var is entirely in RAM, by the time it reboots all of my log files are gone. I'm assuming that since both of the APU1C boxes that I updated to 2.2.4 / 2.2.5 are having stability issues, it might be a FreeBSD 10 compatibility thing. I've searched and haven't found any mention of this anywhere else so I'm hoping someone could help out. Any idea where to start?
Thanks so much in advance!
-
The only APU issue I'm aware of in 2.2.x is SNMP with an empty SD card slot causing a crash, but we fixed that in 2.2.2 or 2.2.3 by just not loading the affected MIB, so that wouldn't happen on 2.2.4 nor 2.2.5.
What is your config like?
-
Well, from the APU standpoint it's a bit non-standard. We load squid w/ havp, squidguard, and snort on it, but don't hold onto the logs. We do this by copying the database files from the sd card into their spots in /var and restart the services each time it loads. We then update the files on the sd card once a month. This minimizes writes to the sd card but keeps the definitions current. In addition we run powerd, snmp, and mailreport. We also run a custom script that blinks the front LED lights to show the cables are plugged into the WAN and LAN ports and that there is some traffic. I think it polls and updates every 5 seconds.
All of this ran stable on 2.1.4. These are an upgrade to 2.2.4 and now 2.2.5. Would I be better served doing a clean install and start from scratch instead of an upgrade? Did what I post give you any indication of what was going on or is it just gibberish without context? I'm not sure what else you would want. Do you want my config file?
Thanks for the help.
-
Won't be any difference in upgrade vs. clean install.
Of what you listed, I'd guess the most likely culprit is the custom script that blinks the LED. If you remove that, does it stop crashing?
-
It's the only thing that's custom and not some form of stock, but they are just netstat commands from what I recall but I'll post more tomorrow once I have a chance to check the script.
-
It'd probably be the changing of the LEDs rather than anything to do with obtaining the NIC status (as anything you could do there would be widely touched by our code, so unlikely there is anything there that's touchy), though I guess either/or is possible.
-
it might be a FreeBSD 10 compatibility thing.
Did you install the latest BIOS file on the APU1C?
There are 4 different versions available!I've searched and haven't found any mention of this anywhere else
It might be sounding a little bit strange to you, but it is not meant in this way, if
the APU1C has only a so short live time and was be set then to the status of EoL (End of Live)
please be sure there was "anything" that was not matching well.so I'm hoping someone could help out.
What is about a full install on a mSATA without using the SD card slot or putting a empty 4 GB
SD card inside of the APU1C.Any idea where to start?
I would prefer to do a fresh install and start from scratch.
- BIOS update
- full install on mSATA
- SD card slot empty or with a empty 4 GB model
-
@cmb:
It'd probably be the changing of the LEDs rather than anything to do with obtaining the NIC status (as anything you could do there would be widely touched by our code, so unlikely there is anything there that's touchy), though I guess either/or is possible.
So here's the code. Let me know if you see anything that would cause a problem.
#Set device IDs set wan=re1 set lan=re2 set led="/dev/gpioapu" #Set starting conditions set wanStart=`netstat -I ${wan} -nWb -f link | tail -1 | awk '{print $8}'` set lanStart=`netstat -I ${lan} -nWb -f link | tail -1 | awk '{print $8}'` set wanCurrent=0 set lanCurrent=0 #Initialize LED status set powerStatus=1 set wanStatus=1 set lanStatus=1 #start loop while ( 1 ) sleep 5 #poll and test WAN if ( `ifconfig ${wan} | grep status | awk '{print $2}'` == "no") then set wanStatus=0 else set wanCurrent=`netstat -I ${wan} -nWb -f link | tail -1 | awk '{print $8}'` if ( $wanCurrent != $wanStart ) then set wanStatus=0 #echo "WAN Update: $wanStatus" echo "$powerStatus$wanStatus$lanStatus" > "$led" sleep .2 set wanStatus=1 endif echo "$powerStatus$wanStatus$lanStatus" > "$led" set wanStart=${wanCurrent} endif #poll and test LAN if ( `ifconfig ${lan} | grep status | awk '{print $2}'` == "no") then set lanStatus=0 else set lanCurrent=`netstat -I ${lan} -nWb -f link | tail -1 | awk '{print $8}'` if ( $lanCurrent != $lanStart ) then set lanStatus=0 #echo "LAN Update: $lanStatus" echo "$powerStatus$wanStatus$lanStatus" > "$led" sleep .2 set lanStatus=1 endif echo "$powerStatus$wanStatus$lanStatus" > "$led" set lanStart=${lanCurrent} endif end -
@BlueKobold:
Did you install the latest BIOS file on the APU1C?
There are 4 different versions available!I have not. I can if I need to and I've been communicating with the vendor. The BIOS I have is current, but not the latest.
@BlueKobold:
It might be sounding a little bit strange to you, but it is not meant in this way, if
the APU1C has only a so short live time and was be set then to the status of EoL (End of Live)
please be sure there was "anything" that was not matching well.The APU1C was replaced by the APU1D which is essentially the same thing. If you look at the 2 boards, you can't even tell a difference. They even use the same BIOS.
@BlueKobold:
What is about a full install on a mSATA without using the SD card slot or putting a empty 4 GB
SD card inside of the APU1C.We are looking to upgrade a bunch of devices. They are designed around the SD card so I'd like to keep it that way.
@BlueKobold:
I would prefer to do a fresh install and start from scratch.
- BIOS update
- full install on mSATA
- SD card slot empty or with a empty 4 GB model
All viable, but I was hoping to know what the error was so I could avoid the problem. I'm willing to do a reinstall but since this appears to be a panic, I was hoping to know what was crashing the boxes.
-
Why don't use just STOP messing with the LEDs since that's already been suggested - and see how it goes? I mean, it absolutely is NOT needed for anything and takes about half minute to set up.
-
Why don't use just STOP messing with the LEDs since that's already been suggested - and see how it goes? I mean, it absolutely is NOT needed for anything and takes about half minute to set up.
Thanks so much for the insight! I have disabled that script and I'll follow up if I find out more, but if running netstat or outputting to the LEDs can randomly cause the OS to Panic under FreeBSD10 when it didn't under FreeBSD8 then wouldn't that be a useful thing to know? I've put up the script so that others can look and maybe find the cause. If not, maybe this crops up later for someone else under different circumstances and it helps them diagnose their problem. CMB suggested maybe it was my script, a valid suggestion, so I said I would post the script for review. I believe that's a valid response. We are, after all, a community with some give and take.
Let me know if you come up with anything else. Really appreciate the help.
-
Yes of course it would be useful to fix that if that's the case; but – to test that, you still need to stop filling with those and see if it makes the issue go away. If it doesn't, we've been chasing ghosts there and can move to something else.
-
It'll be useful to know regardless of the end result as there's clearly a FreeBSD bug there somewhere, just need you to narrow down where it is. Removing that script completely and seeing what happens is the best place to start. Then if that doesn't crash, put the script back, but remove the part that changes the LEDs.
-
@cmb:
It'll be useful to know regardless of the end result as there's clearly a FreeBSD bug there somewhere, just need you to narrow down where it is. Removing that script completely and seeing what happens is the best place to start. Then if that doesn't crash, put the script back, but remove the part that changes the LEDs.
Since the problem seems so intermittent, there may be long gaps in my responses here but I'll let you know when I have more info.
-
It happened again without the blinkLED script running. I had the serial cable in when I issued a reboot. Here's more of the code. It only seemed to flow for a couple of minutes so I'm not sure if it's a kernel panic or not or exactly what it is. It normally doesn't happen but it just seems off…
100190 S uwait 0xfffff8000b024200 c-icap 100189 S uwait 0xfffff8000b01fd00 c-icap 100188 S uwait 0xfffff8000a487100 c-icap 100187 S uwait 0xfffff8000b137000 c-icap 100186 S uwait 0xfffff8000a380900 c-icap 100184 S uwait 0xfffff8000a380b80 c-icap 100182 S uwait 0xfffff8000a380500 c-icap 100178 S uwait 0xfffff8000a380800 c-icap 100084 S select 0xfffff8000a316340 c-icap 54725 54229 54229 0 RL (threaded) c-icap 100185 RunQ c-icap 100183 S uwait 0xfffff8000b012e00 c-icap 100181 S uwait 0xfffff8000b012880 c-icap 100180 S uwait 0xfffff8000b012680 c-icap 100179 S uwait 0xfffff8000b134e80 c-icap 100177 S uwait 0xfffff8000a380b00 c-icap 100176 S uwait 0xfffff8000b135a80 c-icap 100175 S uwait 0xfffff8000b135980 c-icap 100174 S uwait 0xfffff8000b135880 c-icap 100173 S uwait 0xfffff8000b135780 c-icap 100172 S uwait 0xfffff8000b135680 c-icap 100122 S select 0xfffff8000a315140 c-icap 54557 54229 54229 0 S (threaded) c-icap 100203 S uwait 0xfffff8000b012c80 c-icap 100202 S uwait 0xfffff8000b012b80 c-icap 100201 S uwait 0xfffff8000a37fc00 c-icap 100200 S uwait 0xfffff8000a42bf00 c-icap 100199 S uwait 0xfffff8000b135080 c-icap 100198 S uwait 0xfffff8000b013500 c-icap 100197 S uwait 0xfffff8000b134f00 c-icap 100196 S uwait 0xfffff8000a382180 c-icap 100195 S uwait 0xfffff8000b134c00 c-icap 100194 S uwait 0xfffff8000b134980 c-icap 100109 S select 0xfffff8000a4871c0 c-icap 54229 1 54229 0 Ss nanslp 0xffffffff81f6e070 c-icap 47926 1 47926 0 REs CPU 0 ntpd 34603 34295 277 0 LL *vm objec 0xfffff8000a3bed80 bandwidthd 34398 34295 277 0 RL bandwidthd 34361 34295 277 0 RL bandwidthd 34295 1 277 0 RL CPU 1 bandwidthd 33901 33508 277 0 RL bandwidthd 33885 33508 277 0 RL bandwidthd 33555 33508 277 0 RL bandwidthd 33508 1 277 0 RL bandwidthd 19003 1 19003 0 REs syslogd 4779 1 4779 0 REs powerd 97873 70713 70404 62 RE pinger 75510 70713 70404 62 RE squidGuard 75080 70713 70404 62 RE squidGuard 74600 70713 70404 62 RE squidGuard 74520 70713 70404 62 RE squidGuard 74068 70713 70404 62 RE squidGuard 73166 70713 70404 62 LE *vm objec 0xfffff8000a3bed80 squidGuard 72471 70713 70404 62 RE squidGuard 70713 70404 70404 62 RL squid 70404 1 70404 62 REs squid 61673 1 61673 0 RLs bsnmpd 56265 1 56265 1002 REs dhcpd 49671 1 49671 59 RLs (threaded) unbound 100123 S kqread 0xfffff8000a378000 unbound 100102 RunQ unbound 47535 1 47535 0 REs cron 46912 1 46885 0 RL lighttpd 40113 40092 40092 0 RE rrdtool 40092 1 40092 0 Rs apinger 37353 1 37353 0 REs inetd 35562 1 35562 0 REs filterlog 34993 1 34857 0 D ufs 0xfffff8000a422418 radvd 24012 1 24012 65 REs dhclient 17973 1 17973 0 REs dhclient 16751 1 16751 0 LEs *vm page 0xfffff8000b94cb40 sshlockout_pf 16395 1 16395 0 RLs sshd 294 292 292 0 S kqread 0xfffff80024fe7900 check_reload_status 292 1 292 0 Ss kqread 0xfffff8000a36ce00 check_reload_status 277 1 277 0 Ss kqread 0xfffff8000a378b00 php-fpm 52 0 0 0 DL mdwait 0xfffff8000a2fe000 [md1] 47 0 0 0 DL mdwait 0xfffff8000a2fe800 [md0] 20 0 0 0 DL syncer 0xffffffff81fbf008 [syncer] 19 0 0 0 DL vlruwt 0xfffff8000a30b980 [vnlru] 18 0 0 0 DL psleep 0xffffffff81fbe204 [bufdaemon] 17 0 0 0 DL pgzero 0xffffffff82110f8c [pagezero] 9 0 0 0 DL pollid 0xffffffff81f6c9f0 [idlepoll] 8 0 0 0 DL psleep 0xffffffff821106c0 [vmdaemon] 7 0 0 0 DL psleep 0xffffffff8219c504 [pagedaemon] 6 0 0 0 DL waiting_ 0xffffffff8218cf70 [sctp_iterator] 5 0 0 0 DL pftm 0xffffffff80d0ab90 [pf purge] 16 0 0 0 DL (threaded) [usb] 100061 D - 0xfffffe000097ae18 [usbus6] 100060 D - 0xfffffe000097adc0 [usbus6] 100059 D - 0xfffffe000097ad68 [usbus6] 100058 D - 0xfffffe000097ad10 [usbus6] 100057 D - 0xfffffe0000985460 [usbus5] 100056 D - 0xfffffe0000985408 [usbus5] 100055 D - 0xfffffe00009853b0 [usbus5] 100054 D - 0xfffffe0000985358 [usbus5] 100053 D - 0xfffffe0000971460 [usbus4] 100052 D - 0xfffffe0000971408 [usbus4] 100051 D - 0xfffffe00009713b0 [usbus4] 100050 D - 0xfffffe0000971358 [usbus4] 100049 D - 0xfffffe0000966e18 [usbus3] 100048 D - 0xfffffe0000966dc0 [usbus3] 100047 D - 0xfffffe0000966d68 [usbus3] 100046 D - 0xfffffe0000966d10 [usbus3] 100045 D - 0xfffffe000095d460 [usbus2] 100044 D - 0xfffffe000095d408 [usbus2] 100043 D - 0xfffffe000095d3b0 [usbus2] 100042 D - 0xfffffe000095d358 [usbus2] 100041 D - 0xfffffe000092ce18 [usbus1] 100040 D - 0xfffffe000092cdc0 [usbus1] 100039 D - 0xfffffe000092cd68 [usbus1] 100038 D - 0xfffffe000092cd10 [usbus1] 100036 D - 0xfffffe000091b460 [usbus0] 100035 D - 0xfffffe000091b408 [usbus0] 100034 D - 0xfffffe000091b3b0 [usbus0] 100033 D - 0xfffffe000091b358 [usbus0] 4 0 0 0 DL (threaded) [cam] 100069 D - 0xffffffff81ea6bc0 [scanner] 100027 D - 0xffffffff81ea6d80 [doneq0] 3 0 0 0 DL crypto_r 0xffffffff8210eb90 [crypto returns] 2 0 0 0 DL crypto_w 0xffffffff8210ea38 [crypto] 15 0 0 0 DL - 0xffffffff81ec4280 [rand_harvestq] 14 0 0 0 DL (threaded) [geom] 100013 D - 0xffffffff82181670 [g_down] 100012 D - 0xffffffff82181668 [g_up] 100011 D - 0xffffffff82181660 [g_event] 13 0 0 0 DL (threaded) [ng_queue] 100010 D sleep 0xffffffff81e650b8 [ng_queue1] 100009 D sleep 0xffffffff81e650b8 [ng_queue0] 12 0 0 0 WL (threaded) [intr] 100077 I [swi1: netisr 1] 100067 I [swi1: pfsync] 100065 I [swi1: pf send] 100062 I [swi0: uart uart] 100037 I [irq17: ehci0 ehci1+] 100032 I [irq18: ohci0 ohci1*] 100031 I [irq19: ahci0] 100030 I [irq261: re2] 100029 I [irq260: re1] 100028 I [irq259: re0] 100025 I [swi5: fast taskq] 100023 I [swi6: Giant taskq] 100021 I [swi6: task queue] 100008 I [swi3: vm] 100007 I [swi4: clock] 100006 I [swi4: clock] 100005 I [swi1: netisr 0] 11 0 0 0 RL (threaded) [idle] 100004 CanRun [idle: cpu1] 100003 CanRun [idle: cpu0] 1 0 1 0 RLs [init] 10 0 0 0 DL audit_wo 0xffffffff82193af0 [audit] 0 0 0 0 DLs (threaded) [kernel] 100113 D - 0xfffff8000a2ee700 [dummynet] 100068 D - 0xfffff800032b3000 [CAM taskq] 100063 D - 0xfffff8000a050e00 [mca taskq] 100026 D - 0xfffff800032b3200 [kqueue taskq] 100024 D - 0xfffff800032b3700 [thread] 100022 D - 0xfffff800032b3c00 [ffs_trim taskq] 100020 D - 0xfffff800032b4400 [acpi_task_2] 100019 D - 0xfffff800032b4400 [acpi_task_1] 100018 D - 0xfffff800032b4400 [acpi_task_0] 100014 D - 0xfffff800031fa500 [firmware taskq] 100000 D swapin 0xffffffff82181768 [swapper] 95668 1 1 0 Z getty 313 1 313 0 Z devd 75263 70713 70404 62 Z squidGuard 1664 1 21 0 Z+ sh 60024 1 21 0 Z sleep 26064 33508 277 0 Z bandwidthd 26121 34295 277 0 Z bandwidthd db:0:kdb.enter.default> alltrace Tracing command intr pid 12 tid 100005 td 0xfffff8000320c490 sched_switch() at sched_switch+0x2b3/frame 0xfffffe001a36db20 mi_switch() at mi_switch+0xe1/frame 0xfffffe001a36db60 ithread_loop() at ithread_loop+0x190/frame 0xfffffe001a36dbb0 fork_exit() at fork_exit+0x9a/frame 0xfffffe001a36dbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001a36dbf0 --- trap 0, rip = 0, rsp = 0xfffffe001a36dcb0, rbp = 0 --- Tracing command idle pid 11 tid 100004 td 0xfffff8000320c920 sched_switch() at sched_switch+0x2b3/frame 0xfffffe001a368ac0 mi_switch() at mi_switch+0xe1/frame 0xfffffe001a368b00 sched_idletd() at sched_idletd+0xc9/frame 0xfffffe001a368bb0 fork_exit() at fork_exit+0x9a/frame 0xfffffe001a368bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001a368bf0 --- trap 0, rip = 0, rsp = 0xfffffe001a368cb0, rbp = 0 --- Tracing command idle pid 11 tid 100003 td 0xfffff8000320d000 sched_switch() at sched_switch+0x2b3/frame 0xfffffe001a363ac0 mi_switch() at mi_switch+0xe1/frame 0xfffffe001a363b00 sched_idletd() at sched_idletd+0x3a7/frame 0xfffffe001a363bb0 fork_exit() at fork_exit+0x9a/frame 0xfffffe001a363bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001a363bf0 --- trap 0, rip = 0, rsp = 0xfffffe001a363cb0, rbp = 0 --- Tracing command init pid 1 tid 100002 td 0xfffff8000320d490 sched_switch() at sched_switch+0x2b3/frame 0xfffffe001abfb6d0 mi_switch() at mi_switch+0xe1/frame 0xfffffe001abfb710 sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe001abfb790 sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe001abfb7c0 _sleep() at _sleep+0x27d/frame 0xfffffe001abfb840 kern_wait6() at kern_wait6+0x5f4/frame 0xfffffe001abfb8f0 sys_wait4() at sys_wait4+0x72/frame 0xfffffe001abfbae0 amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe001abfbbf0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe001abfbbf0 --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x41397a, rsp = 0x7fffffffe748, rbp = 0x7fffffffe7d0 --- Tracing command audit pid 10 tid 100001 td 0xfffff8000320d920 sched_switch() at sched_switch+0x2b3/frame 0xfffffe001abf6a80 mi_switch() at mi_switch+0xe1/frame 0xfffffe001abf6ac0 sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe001abf6af0 _cv_wait() at _cv_wait+0x16d/frame 0xfffffe001abf6b40 audit_worker() at audit_worker+0xa3/frame 0xfffffe001abf6bb0 fork_exit() at fork_exit+0x9a/frame 0xfffffe001abf6bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001abf6bf0 --- trap 0, rip = 0, rsp = 0xfffffe001abf6cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100110 td 0xfffff8000aab6920 sched_switch() at sched_switch+0x2b3/frame 0xfffffe003609aab0 mi_switch() at mi_switch+0xe1/frame 0xfffffe003609aaf0 sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe003609ab20 msleep_spin_sbt() at msleep_spin_sbt+0x1a3/frame 0xfffffe003609ab80 taskqueue_thread_loop() at taskqueue_thread_loop+0xfd/frame 0xfffffe003609abb0 fork_exit() at fork_exit+0x9a/frame 0xfffffe003609abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003609abf0 --- trap 0, rip = 0, rsp = 0xfffffe003609acb0, rbp = 0 --- Tracing command kernel pid 0 tid 100068 td 0xfffff8000a295490 sched_switch() at sched_switch+0x2b3/frame 0xfffffe0035fc7a90 mi_switch() at mi_switch+0xe1/frame 0xfffffe0035fc7ad0 sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe0035fc7b00 _sleep() at _sleep+0x287/frame 0xfffffe0035fc7b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xd5/frame 0xfffffe0035fc7bb0 fork_exit() at fork_exit+0x9a/frame 0xfffffe0035fc7bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0035fc7bf0 --- trap 0, rip = 0, rsp = 0xfffffe0035fc7cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100063 td 0xfffff8000a037000 sched_switch() at sched_switch+0x2b3/frame 0xfffffe0035fadab0 mi_switch() at mi_switch+0xe1/frame 0xfffffe0035fadaf0 sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe0035fadb20 msleep_spin_sbt() at msleep_spin_sbt+0x1a3/frame 0xfffffe0035fadb80 taskqueue_thread_loop() at taskqueue_thread_loop+0xfd/frame 0xfffffe0035fadbb0 fork_exit() at fork_exit+0x9a/frame 0xfffffe0035fadbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0035fadbf0 --- trap 0, rip = 0, rsp = 0xfffffe0035fadcb0, rbp = 0 --- Tracing command kernel pid 0 tid 100026 td 0xfffff800033de490 sched_switch() at sched_switch+0x2b3/frame 0xfffffe001a3d6a90 mi_switch() at mi_switch+0xe1/frame 0xfffffe001a3d6ad0 sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe001a3d6b00 _sleep() at _sleep+0x287/frame 0xfffffe001a3d6b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xd5/frame 0xfffffe001a3d6bb0 fork_exit() at fork_exit+0x9a/frame 0xfffffe001a3d6bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001a3d6bf0 --- trap 0, rip = 0, rsp = 0xfffffe001a3d6cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100024 td 0xfffff800033df000 sched_switch() at sched_switch+0x2b3/frame 0xfffffe001a3cca90 mi_switch() at mi_switch+0xe1/frame 0xfffffe001a3ccad0 sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe001a3ccb00 _sleep() at _sleep+0x287/frame 0xfffffe001a3ccb80 taskqueue_thread_loop() at taskqueue_thread_loop+0xd5/frame 0xfffffe001a3ccbb0 fork_exit() at fork_exit+0x9a/frame 0xfffffe001a3ccbf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001a3ccbf0 --- trap 0, rip = 0, rsp = 0xfffffe001a3cccb0, rbp = 0 --- Tracing command kernel pid 0 tid 100022 td 0xfffff800033df920 sched_switch() at sched_switch+0x2b3/frame 0xfffffe001a3c2a90 mi_switch() at mi_switch+0xe1/frame 0xfffffe001a3c2ad0 sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe001a3c2b00 _sleep() at _sleep+0x287/frame 0xfffffe001a3c2b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xd5/frame 0xfffffe001a3c2bb0 fork_exit() at fork_exit+0x9a/frame 0xfffffe001a3c2bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001a3c2bf0 --- trap 0, rip = 0, rsp = 0xfffffe001a3c2cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100020 td 0xfffff800033e0490 sched_switch() at sched_switch+0x2b3/frame 0xfffffe001a3b8ab0 mi_switch() at mi_switch+0xe1/frame 0xfffffe001a3b8af0 sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe001a3b8b20 msleep_spin_sbt() at msleep_spin_sbt+0x1a3/frame 0xfffffe001a3b8b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xfd/frame 0xfffffe001a3b8bb0 fork_exit() at fork_exit+0x9a/frame 0xfffffe001a3b8bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001a3b8bf0 --- trap 0, rip = 0, rsp = 0xfffffe001a3b8cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100019 td 0xfffff800033e0920 sched_switch() at sched_switch+0x2b3/frame 0xfffffe001a3b3ab0 mi_switch() at mi_switch+0xe1/frame 0xfffffe001a3b3af0 sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe001a3b3b20 msleep_spin_sbt() at msleep_spin_sbt+0x1a3/frame 0xfffffe001a3b3b80 taskqueue_thread_loop() at taskqueue_thread_loop+0xfd/frame 0xfffffe001a3b3bb0 fork_exit() at fork_exit+0x9a/frame 0xfffffe001a3b3bf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001a3b3bf0 --- trap 0, rip = 0, rsp = 0xfffffe001a3b3cb0, rbp = 0 --- Tracing command kernel pid 0 tid 100018 td 0xfffff80003224000 sched_switch() at sched_switch+0x2b3/frame 0xfffffe001a3aeab0 mi_switch() at mi_switch+0xe1/frame 0xfffffe001a3aeaf0 sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe001a3aeb20 msleep_spin_sbt() at msleep_spin_sbt+0x1a3/frame 0xfffffe001a3aeb80 taskqueue_thread_loop() at taskqueue_thread_loop+0xfd/frame 0xfffffe001a3aebb0 fork_exit() at fork_exit+0x9a/frame 0xfffffe001a3aebf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001a3aebf0 --- trap 0, rip = 0, rsp = 0xfffffe001a3aecb0, rbp = 0 --- Tracing command kernel pid 0 tid 100014 td 0xfffff80003225490 sched_switch() at sched_switch+0x2b3/frame 0xfffffe001a39aa90 mi_switch() at mi_switch+0xe1/frame 0xfffffe001a39aad0 sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe001a39ab00 _sleep() at _sleep+0x287/frame 0xfffffe001a39ab80 taskqueue_thread_loop() at taskqueue_thread_loop+0xd5/frame 0xfffffe001a39abb0 fork_exit() at fork_exit+0x9a/frame 0xfffffe001a39abf0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001a39abf0 --- trap 0, rip = 0, rsp = 0xfffffe001a39acb0, rbp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xffffffff82181c30 sched_switch() at sched_switch+0x2b3/frame 0xffffffff824deb70 mi_switch() at mi_switch+0xe1/frame 0xffffffff824debb0 sleepq_timedwait() at sleepq_timedwait+0x3a/frame 0xffffffff824debe0 _sleep() at _sleep+0x26e/frame 0xffffffff824dec60 swapper() at swapper+0x28f/frame 0xffffffff824decb0 btext() at btext+0x2c db:0:kdb.enter.default> capture off db:0:kdb.enter.default> call doadump = 0x6 db:0:kdb.enter.default> reset PC Engines APU BIOS build date: Apr 5 2014 Reading data from file [bootorder] SeaBIOS (version ?-20140405_120742-frink) SeaBIOS (version ?-20140405_120742-frink) Found coreboot cbmem console @ 7e150400 Found mainboard PC Engines APU Relocating init from 0x000e8e71 to 0x7e1065e0 (size 39259) Found CBFS header at 0xfffffb90 found file "bootorder" in cbmem CPU Mhz=1000 Found 27 PCI devices (max PCI bus is 05) Copying PIR from 0x7e160400 to 0x000f27a0 Copying MPTABLE from 0x7e161400/7e161410 to 0x000f25b0 with length 1ec Copying ACPI RSDP from 0x7e162400 to 0x000f2590 Copying SMBIOS entry point from 0x7e16d800 to 0x000f2570 Using pmtimer, ioport 0x808 Scan for VGA option rom EHCI init on dev 00:12.2 (regs=0xf7f04420) Found 1 lpt ports Found 2 serial ports AHCI controller at 11.0, iobase f7f04000, irq 11 EHCI init on dev 00:13.2 (regs=0xf7f04520) EHCI init on dev 00:16.2 (regs=0xf7f04620) Searching bootorder for: /rom@img/setup Searching bootorder for: /rom@img/memtest OHCI init on dev 00:12.0 (regs=0xf7f00000) OHCI init on dev 00:13.0 (regs=0xf7f01000) OHCI init on dev 00:14.5 (regs=0xf7f02000) OHCI init on dev 00:16.0 (regs=0xf7f03000) Searching bootorder for: /pci@i0cf8/usb@16,2/storage@1/*@0/*@0,0 Searching bootorder for: /pci@i0cf8/usb@16,2/usb-*@1 USB MSC vendor='Multiple' product='Card Reader' rev='1.00' type=0 removable=1 USB MSC blksize=512 sectors=15523840 All threads complete. Scan for option roms Running option rom at c000:0003 iPXE (http://ipxe.org) 00:00.0 C000 PCI2.10 PnP PMMpmm call arg1=1 pmm call arg1=0 +7E0DA5C0pmm call arg1=1 pmm call arg1=0 +7E03A5C0 C000 Searching bootorder for: /rom@genroms/pxeboot.rom Build date: Apr 5 2014 System memory size: 2017 MB Press F12 for boot menu. Searching bootorder for: HALT drive 0x000f2520: PCHS=0/0/0 translation=lba LCHS=966/255/63 s=15523840 Space available for UMB: c1000-ee800, f0000-f2520 Returned 245760 bytes of ZoneHigh e820 map has 6 items: 0: 0000000000000000 - 000000000009fc00 = 1 RAM 1: 000000000009fc00 - 00000000000a0000 = 2 RESERVED 2: 00000000000f0000 - 0000000000100000 = 2 RESERVED 3: 0000000000100000 - 000000007e14c000 = 1 RAM 4: 000000007e14c000 - 000000007f000000 = 2 RESERVED 5: 00000000f8000000 - 00000000f9000000 = 2 RESERVED enter handle_19: NULL Booting from Hard Disk... Booting from 0000:7c00 1 pfSense 2 pfSense F6 PXE Boot: 1 /boot/config: -S115200 -hConsoles: serial port BIOS drive C: is disk0 BIOS 639kB/2064688kB available memory FreeBSD/x86 bootstrap loader, Revision 1.1 (root@pfs22-amd64-builder, Wed Nov 4 16:13:22 CST 2015) Loading /boot/defaults/loader.conf For reference, this is what I normally see: [code]*** FINAL System shutdown message from root@GateKeeper.localdomain *** System going down IMMEDIATELY pfSense is now shutting down ... Nov 20 00:30:26 lighttpd[45365]: (server.c.1567) server stopped by UID = 0 PID = 1 Waiting (max 60 seconds) for system process `vnlru' to stop...done Waiting (max 60 seconds) for system process `bufdaemon' to stop...done Waiting (max 60 seconds) for system process `syncer' to stop... Syncing disks, vnodes remaining...0 0 done All buffers synced. Uptime: 5m27s re0: link state changed to DOWN re1: link state changed to DOWN re1: link state changed to UP re2: link state changed to DOWN re2: link state changed to UP PC Engines APU BIOS build date: Apr 5 2014 Reading data from file [bootorder] SeaBIOS (version ?-20140405_120742-frink) SeaBIOS (version ?-20140405_120742-frink) Found coreboot cbmem console @ 7e150400 Found mainboard PC Engines APU Relocating init from 0x000e8e71 to 0x7e1065e0 (size 39259) Found CBFS header at 0xfffffb90 found file "bootorder" in cbmem CPU Mhz=1001 Found 27 PCI devices (max PCI bus is 05) Copying PIR from 0x7e160400 to 0x000f27a0 Copying MPTABLE from 0x7e161400/7e161410 to 0x000f25b0 with length 1ec Copying ACPI RSDP from 0x7e162400 to 0x000f2590 Copying SMBIOS entry point from 0x7e16d800 to 0x000f2570 Using pmtimer, ioport 0x808 Scan for VGA option rom EHCI init on dev 00:12.2 (regs=0xf7f04420) Found 1 lpt ports Found 2 serial ports AHCI controller at 11.0, iobase f7f04000, irq 11 EHCI init on dev 00:13.2 (regs=0xf7f04520) EHCI init on dev 00:16.2 (regs=0xf7f04620) Searching bootorder for: /rom@img/setup Searching bootorder for: /rom@img/memtest OHCI init on dev 00:12.0 (regs=0xf7f00000) OHCI init on dev 00:13.0 (regs=0xf7f01000) OHCI init on dev 00:14.5 (regs=0xf7f02000) OHCI init on dev 00:16.0 (regs=0xf7f03000) Searching bootorder for: /pci@i0cf8/usb@16,2/storage@1/*@0/*@0,0 Searching bootorder for: /pci@i0cf8/usb@16,2/usb-*@1 USB MSC vendor='Multiple' product='Card Reader' rev='1.00' type=0 removable=1 USB MSC blksize=512 sectors=15523840 All threads complete. Scan for option roms Running option rom at c000:0003 iPXE (http://ipxe.org) 00:00.0 C000 PCI2.10 PnP PMMpmm call arg1=1 pmm call arg1=0 +7E0DA5C0pmm call arg1=1 pmm call arg1=0 +7E03A5C0 C000 Searching bootorder for: /rom@genroms/pxeboot.rom Build date: Apr 5 2014 System memory size: 2017 MB Press F12 for boot menu. Searching bootorder for: HALT drive 0x000f2520: PCHS=0/0/0 translation=lba LCHS=966/255/63 s=15523840 Space available for UMB: c1000-ee800, f0000-f2520 Returned 245760 bytes of ZoneHigh e820 map has 6 items: 0: 0000000000000000 - 000000000009fc00 = 1 RAM 1: 000000000009fc00 - 00000000000a0000 = 2 RESERVED 2: 00000000000f0000 - 0000000000100000 = 2 RESERVED 3: 0000000000100000 - 000000007e14c000 = 1 RAM 4: 000000007e14c000 - 000000007f000000 = 2 RESERVED 5: 00000000f8000000 - 00000000f9000000 = 2 RESERVED enter handle_19: NULL Booting from Hard Disk... Booting from 0000:7c00 1 pfSense 2 pfSense [/code] I also see this in the startup: [code]Trying to mount root from ufs:/dev/ufs/pfsense0 [ro,sync,noatime]... mountroot: waiting for device /dev/ufs/pfsense0 ... da0 at umass-sim0 bus 0 scbus6 target 0 lun 0 da0: <multiple card ="" reader="" 1.00=""> Removable Direct Access SCSI-4 device da0: Serial Number 058F63666485 da0: 40.000MB/s transfers da0: 7580MB (15523840 512 byte sectors: 255H 63S/T 966C) da0: quirks=0x2 <no_6_byte>WARNING: / was not properly dismounted Configuring crash dumps... /dev/ufs/pfsense0: 30974 files, 2013156 used, 1766872 free (2632 frags, 220530 blocks, 0.1% fragmentation) /dev/ufs/cf: FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED) /dev/ufs/cf: 30 files, 3971 used, 97084 free (28 frags, 12132 blocks, 0.0% fragmentation) Filesystems are clean, continuing... Mounting filesystems... Setting up memory disks... done. [/code] Any idea where to see the crash dumps?[/thread]</no_6_byte></multiple> -
Hey guys
I have a similar problem on FreeBSD/FreeNAS and I just wanted to ask if you ever found a solution?
The error message from above:
Tracing command kernel pid 0 tid 100018 td 0xfffff80003224000
sched_switch() at sched_switch+0x2b3/frame 0xfffffe001a3aeab0
mi_switch() at mi_switch+0xe1/frame 0xfffffe001a3aeaf0
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe001a3aeb20
msleep_spin_sbt() at msleep_spin_sbt+0x1a3/frame 0xfffffe001a3aeb80
taskqueue_thread_loop() at taskqueue_thread_loop+0xfd/frame 0xfffffe001a3aebb0
fork_exit() at fork_exit+0x9a/frame 0xfffffe001a3aebf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001a3aebf0
–- trap 0, rip = 0, rsp = 0xfffffe001a3aecb0, rbp = 0 ---Tracing command kernel pid 0 tid 100014 td 0xfffff80003225490
sched_switch() at sched_switch+0x2b3/frame 0xfffffe001a39aa90
mi_switch() at mi_switch+0xe1/frame 0xfffffe001a39aad0
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe001a39ab00
_sleep() at _sleep+0x287/frame 0xfffffe001a39ab80
taskqueue_thread_loop() at taskqueue_thread_loop+0xd5/frame 0xfffffe001a39abb0
fork_exit() at fork_exit+0x9a/frame 0xfffffe001a39abf0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001a39abf0
--- trap 0, rip = 0, rsp = 0xfffffe001a39acb0, rbp = 0 ---I do see something like that - here's the https://bugs.freenas.org/issues/15089 –> second.jpg ...
I'd appreciate any help!
Thanks and best regards