Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    APU1C Kernel Panic

    Scheduled Pinned Locked Moved Hardware
    16 Posts 5 Posters 5.2k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C Offline
      cmb
      last edited by

      Won't be any difference in upgrade vs. clean install.

      Of what you listed, I'd guess the most likely culprit is the custom script that blinks the LED. If you remove that, does it stop crashing?

      1 Reply Last reply Reply Quote 0
      • S Offline
        Stewart
        last edited by

        It's the only thing that's custom and not some form of stock, but they are just netstat commands from what I recall but I'll post more tomorrow once I have a chance to check the script.

        1 Reply Last reply Reply Quote 0
        • C Offline
          cmb
          last edited by

          It'd probably be the changing of the LEDs rather than anything to do with obtaining the NIC status (as anything you could do there would be widely touched by our code, so unlikely there is anything there that's touchy), though I guess either/or is possible.

          1 Reply Last reply Reply Quote 0
          • ? This user is from outside of this forum
            Guest
            last edited by

            it might be a FreeBSD 10 compatibility thing.

            Did you install the latest BIOS file on the APU1C?
            There are 4 different versions available!

            I've searched and haven't found any mention of this anywhere else

            It might be sounding a little bit strange to you, but it is not meant in this way, if
            the APU1C has only a so short live time and was be set then to the status of EoL (End of Live)
            please be sure there was "anything" that was not matching well.

            so I'm hoping someone could help out.

            What is about a full install on a mSATA without using the SD card slot or putting a empty 4 GB
            SD card inside of the APU1C.

            Any idea where to start?

            I would prefer to do a fresh install and start from scratch.

            • BIOS update
            • full install on mSATA
            • SD card slot empty or with a empty 4 GB model
            1 Reply Last reply Reply Quote 0
            • S Offline
              Stewart
              last edited by

              @cmb:

              It'd probably be the changing of the LEDs rather than anything to do with obtaining the NIC status (as anything you could do there would be widely touched by our code, so unlikely there is anything there that's touchy), though I guess either/or is possible.

              So here's the code.  Let me know if you see anything that would cause a problem.

              #Set device IDs
              set wan=re1
              set lan=re2
              set led="/dev/gpioapu"
              
              #Set starting conditions
              set wanStart=`netstat -I ${wan} -nWb -f link | tail -1 | awk '{print $8}'`
              set lanStart=`netstat -I ${lan} -nWb -f link | tail -1 | awk '{print $8}'`
              set wanCurrent=0
              set lanCurrent=0
              
              #Initialize LED status
              set powerStatus=1
              set wanStatus=1
              set lanStatus=1
              
              #start loop
              while ( 1 )
              sleep 5
                      #poll and test WAN
                      if ( `ifconfig ${wan} | grep status | awk '{print $2}'` == "no") then
                              set wanStatus=0
                      else
                              set wanCurrent=`netstat -I ${wan} -nWb -f link | tail -1 | awk '{print $8}'`
                              if ( $wanCurrent != $wanStart ) then
                                      set wanStatus=0
                                      #echo "WAN Update: $wanStatus"
                                      echo "$powerStatus$wanStatus$lanStatus" > "$led"
                                      sleep .2
                                      set wanStatus=1
                              endif
                              echo "$powerStatus$wanStatus$lanStatus" > "$led"
                              set wanStart=${wanCurrent}
                      endif
              
                      #poll and test LAN
                      if ( `ifconfig ${lan} | grep status | awk '{print $2}'` == "no") then
                              set lanStatus=0
                      else
                              set lanCurrent=`netstat -I ${lan} -nWb -f link | tail -1 | awk '{print $8}'`
                              if ( $lanCurrent != $lanStart ) then
                                      set lanStatus=0
                                      #echo "LAN Update: $lanStatus"
                                      echo "$powerStatus$wanStatus$lanStatus" > "$led"
                                      sleep .2
                                      set lanStatus=1
                              endif
                              echo "$powerStatus$wanStatus$lanStatus" > "$led"
                              set lanStart=${lanCurrent}
                      endif
              end
              
              
              1 Reply Last reply Reply Quote 0
              • S Offline
                Stewart
                last edited by

                @BlueKobold:

                Did you install the latest BIOS file on the APU1C?
                There are 4 different versions available!

                I have not.  I can if I need to and I've been communicating with the vendor.  The BIOS I have is current, but not the latest.

                @BlueKobold:

                It might be sounding a little bit strange to you, but it is not meant in this way, if
                the APU1C has only a so short live time and was be set then to the status of EoL (End of Live)
                please be sure there was "anything" that was not matching well.

                The APU1C was replaced by the APU1D which is essentially the same thing.  If you look at the 2 boards, you can't even tell a difference.  They even use the same BIOS.

                @BlueKobold:

                What is about a full install on a mSATA without using the SD card slot or putting a empty 4 GB
                SD card inside of the APU1C.

                We are looking to upgrade a bunch of devices.  They are designed around the SD card so I'd like to keep it that way.

                @BlueKobold:

                I would prefer to do a fresh install and start from scratch.

                • BIOS update
                • full install on mSATA
                • SD card slot empty or with a empty 4 GB model

                All viable, but I was hoping to know what the error was so I could avoid the problem.  I'm willing to do a reinstall but since this appears to be a panic, I was hoping to know what was crashing the boxes.

                1 Reply Last reply Reply Quote 0
                • D Offline
                  doktornotor Banned
                  last edited by

                  Why don't use just STOP messing with the LEDs since that's already been suggested - and see how it goes? I mean, it absolutely is NOT needed for anything and takes about half minute to set up.

                  1 Reply Last reply Reply Quote 0
                  • S Offline
                    Stewart
                    last edited by

                    @doktornotor:

                    Why don't use just STOP messing with the LEDs since that's already been suggested - and see how it goes? I mean, it absolutely is NOT needed for anything and takes about half minute to set up.

                    Thanks so much for the insight!  I have disabled that script and I'll follow up if I find out more, but if running netstat or outputting to the LEDs can randomly cause the OS to Panic under FreeBSD10 when it didn't under FreeBSD8 then wouldn't that be a useful thing to know?  I've put up the script so that others can look and maybe find the cause.  If not, maybe this crops up later for someone else under different circumstances and it helps them diagnose their problem.  CMB suggested maybe it was my script, a valid suggestion, so I said I would post the script for review.  I believe that's a valid response.  We are, after all, a community with some give and take.

                    Let me know if you come up with anything else.  Really appreciate the help.

                    1 Reply Last reply Reply Quote 0
                    • D Offline
                      doktornotor Banned
                      last edited by

                      Yes of course it would be useful to fix that if that's the case; but – to test that, you still need to stop filling with those and see if it makes the issue go away. If it doesn't, we've been chasing ghosts there and can move to something else.

                      1 Reply Last reply Reply Quote 0
                      • C Offline
                        cmb
                        last edited by

                        It'll be useful to know regardless of the end result as there's clearly a FreeBSD bug there somewhere, just need you to narrow down where it is. Removing that script completely and seeing what happens is the best place to start. Then if that doesn't crash, put the script back, but remove the part that changes the LEDs.

                        1 Reply Last reply Reply Quote 0
                        • S Offline
                          Stewart
                          last edited by

                          @cmb:

                          It'll be useful to know regardless of the end result as there's clearly a FreeBSD bug there somewhere, just need you to narrow down where it is. Removing that script completely and seeing what happens is the best place to start. Then if that doesn't crash, put the script back, but remove the part that changes the LEDs.

                          Since the problem seems so intermittent, there may be long gaps in my responses here but I'll let you know when I have more info.

                          1 Reply Last reply Reply Quote 0
                          • S Offline
                            Stewart
                            last edited by

                            It happened again without the blinkLED script running.  I had the serial cable in when I issued a reboot.  Here's more of the code.  It only seemed to flow for a couple of minutes so I'm not sure if it's a kernel panic or not or exactly what it is.  It normally doesn't happen but it just seems off…

                            
                            100190                   S       uwait    0xfffff8000b024200 c-icap
                            100189                   S       uwait    0xfffff8000b01fd00 c-icap
                            100188                   S       uwait    0xfffff8000a487100 c-icap
                            100187                   S       uwait    0xfffff8000b137000 c-icap
                            100186                   S       uwait    0xfffff8000a380900 c-icap
                            100184                   S       uwait    0xfffff8000a380b80 c-icap
                            100182                   S       uwait    0xfffff8000a380500 c-icap
                            100178                   S       uwait    0xfffff8000a380800 c-icap
                            100084                   S       select   0xfffff8000a316340 c-icap
                            54725 54229 54229     0  RL      (threaded)                  c-icap
                            100185                   RunQ                                c-icap
                            100183                   S       uwait    0xfffff8000b012e00 c-icap
                            100181                   S       uwait    0xfffff8000b012880 c-icap
                            100180                   S       uwait    0xfffff8000b012680 c-icap
                            100179                   S       uwait    0xfffff8000b134e80 c-icap
                            100177                   S       uwait    0xfffff8000a380b00 c-icap
                            100176                   S       uwait    0xfffff8000b135a80 c-icap
                            100175                   S       uwait    0xfffff8000b135980 c-icap
                            100174                   S       uwait    0xfffff8000b135880 c-icap
                            100173                   S       uwait    0xfffff8000b135780 c-icap
                            100172                   S       uwait    0xfffff8000b135680 c-icap
                            100122                   S       select   0xfffff8000a315140 c-icap
                            54557 54229 54229     0  S       (threaded)                  c-icap
                            100203                   S       uwait    0xfffff8000b012c80 c-icap
                            100202                   S       uwait    0xfffff8000b012b80 c-icap
                            100201                   S       uwait    0xfffff8000a37fc00 c-icap
                            100200                   S       uwait    0xfffff8000a42bf00 c-icap
                            100199                   S       uwait    0xfffff8000b135080 c-icap
                            100198                   S       uwait    0xfffff8000b013500 c-icap
                            100197                   S       uwait    0xfffff8000b134f00 c-icap
                            100196                   S       uwait    0xfffff8000a382180 c-icap
                            100195                   S       uwait    0xfffff8000b134c00 c-icap
                            100194                   S       uwait    0xfffff8000b134980 c-icap
                            100109                   S       select   0xfffff8000a4871c0 c-icap
                            54229     1 54229     0  Ss      nanslp   0xffffffff81f6e070 c-icap
                            47926     1 47926     0  REs     CPU 0                       ntpd
                            34603 34295   277     0  LL     *vm objec 0xfffff8000a3bed80 bandwidthd
                            34398 34295   277     0  RL                                  bandwidthd
                            34361 34295   277     0  RL                                  bandwidthd
                            34295     1   277     0  RL      CPU 1                       bandwidthd
                            33901 33508   277     0  RL                                  bandwidthd
                            33885 33508   277     0  RL                                  bandwidthd
                            33555 33508   277     0  RL                                  bandwidthd
                            33508     1   277     0  RL                                  bandwidthd
                            19003     1 19003     0  REs                                 syslogd
                             4779     1  4779     0  REs                                 powerd
                            97873 70713 70404    62  RE                                  pinger
                            75510 70713 70404    62  RE                                  squidGuard
                            75080 70713 70404    62  RE                                  squidGuard
                            74600 70713 70404    62  RE                                  squidGuard
                            74520 70713 70404    62  RE                                  squidGuard
                            74068 70713 70404    62  RE                                  squidGuard
                            73166 70713 70404    62  LE     *vm objec 0xfffff8000a3bed80 squidGuard
                            72471 70713 70404    62  RE                                  squidGuard
                            70713 70404 70404    62  RL                                  squid
                            70404     1 70404    62  REs                                 squid
                            61673     1 61673     0  RLs                                 bsnmpd
                            56265     1 56265  1002  REs                                 dhcpd
                            49671     1 49671    59  RLs     (threaded)                  unbound
                            100123                   S       kqread   0xfffff8000a378000 unbound
                            100102                   RunQ                                unbound
                            47535     1 47535     0  REs                                 cron
                            46912     1 46885     0  RL                                  lighttpd
                            40113 40092 40092     0  RE                                  rrdtool
                            40092     1 40092     0  Rs                                  apinger
                            37353     1 37353     0  REs                                 inetd
                            35562     1 35562     0  REs                                 filterlog
                            34993     1 34857     0  D       ufs      0xfffff8000a422418 radvd
                            24012     1 24012    65  REs                                 dhclient
                            17973     1 17973     0  REs                                 dhclient
                            16751     1 16751     0  LEs    *vm page  0xfffff8000b94cb40 sshlockout_pf
                            16395     1 16395     0  RLs                                 sshd
                              294   292   292     0  S       kqread   0xfffff80024fe7900 check_reload_status
                              292     1   292     0  Ss      kqread   0xfffff8000a36ce00 check_reload_status
                              277     1   277     0  Ss      kqread   0xfffff8000a378b00 php-fpm
                               52     0     0     0  DL      mdwait   0xfffff8000a2fe000 [md1]
                               47     0     0     0  DL      mdwait   0xfffff8000a2fe800 [md0]
                               20     0     0     0  DL      syncer   0xffffffff81fbf008 [syncer]
                               19     0     0     0  DL      vlruwt   0xfffff8000a30b980 [vnlru]
                               18     0     0     0  DL      psleep   0xffffffff81fbe204 [bufdaemon]
                               17     0     0     0  DL      pgzero   0xffffffff82110f8c [pagezero]
                                9     0     0     0  DL      pollid   0xffffffff81f6c9f0 [idlepoll]
                                8     0     0     0  DL      psleep   0xffffffff821106c0 [vmdaemon]
                                7     0     0     0  DL      psleep   0xffffffff8219c504 [pagedaemon]
                                6     0     0     0  DL      waiting_ 0xffffffff8218cf70 [sctp_iterator]
                                5     0     0     0  DL      pftm     0xffffffff80d0ab90 [pf purge]
                               16     0     0     0  DL      (threaded)                  [usb]
                            100061                   D       -        0xfffffe000097ae18 [usbus6]
                            100060                   D       -        0xfffffe000097adc0 [usbus6]
                            100059                   D       -        0xfffffe000097ad68 [usbus6]
                            100058                   D       -        0xfffffe000097ad10 [usbus6]
                            100057                   D       -        0xfffffe0000985460 [usbus5]
                            100056                   D       -        0xfffffe0000985408 [usbus5]
                            100055                   D       -        0xfffffe00009853b0 [usbus5]
                            100054                   D       -        0xfffffe0000985358 [usbus5]
                            100053                   D       -        0xfffffe0000971460 [usbus4]
                            100052                   D       -        0xfffffe0000971408 [usbus4]
                            100051                   D       -        0xfffffe00009713b0 [usbus4]
                            100050                   D       -        0xfffffe0000971358 [usbus4]
                            100049                   D       -        0xfffffe0000966e18 [usbus3]
                            100048                   D       -        0xfffffe0000966dc0 [usbus3]
                            100047                   D       -        0xfffffe0000966d68 [usbus3]
                            100046                   D       -        0xfffffe0000966d10 [usbus3]
                            100045                   D       -        0xfffffe000095d460 [usbus2]
                            100044                   D       -        0xfffffe000095d408 [usbus2]
                            100043                   D       -        0xfffffe000095d3b0 [usbus2]
                            100042                   D       -        0xfffffe000095d358 [usbus2]
                            100041                   D       -        0xfffffe000092ce18 [usbus1]
                            100040                   D       -        0xfffffe000092cdc0 [usbus1]
                            100039                   D       -        0xfffffe000092cd68 [usbus1]
                            100038                   D       -        0xfffffe000092cd10 [usbus1]
                            100036                   D       -        0xfffffe000091b460 [usbus0]
                            100035                   D       -        0xfffffe000091b408 [usbus0]
                            100034                   D       -        0xfffffe000091b3b0 [usbus0]
                            100033                   D       -        0xfffffe000091b358 [usbus0]
                                4     0     0     0  DL      (threaded)                  [cam]
                            100069                   D       -        0xffffffff81ea6bc0 [scanner]
                            100027                   D       -        0xffffffff81ea6d80 [doneq0]
                                3     0     0     0  DL      crypto_r 0xffffffff8210eb90 [crypto returns]
                                2     0     0     0  DL      crypto_w 0xffffffff8210ea38 [crypto]
                               15     0     0     0  DL      -        0xffffffff81ec4280 [rand_harvestq]
                               14     0     0     0  DL      (threaded)                  [geom]
                            100013                   D       -        0xffffffff82181670 [g_down]
                            100012                   D       -        0xffffffff82181668 [g_up]
                            100011                   D       -        0xffffffff82181660 [g_event]
                               13     0     0     0  DL      (threaded)                  [ng_queue]
                            100010                   D       sleep    0xffffffff81e650b8 [ng_queue1]
                            100009                   D       sleep    0xffffffff81e650b8 [ng_queue0]
                               12     0     0     0  WL      (threaded)                  [intr]
                            100077                   I                                   [swi1: netisr 1]
                            100067                   I                                   [swi1: pfsync]
                            100065                   I                                   [swi1: pf send]
                            100062                   I                                   [swi0: uart uart]
                            100037                   I                                   [irq17: ehci0 ehci1+]
                            100032                   I                                   [irq18: ohci0 ohci1*]
                            100031                   I                                   [irq19: ahci0]
                            100030                   I                                   [irq261: re2]
                            100029                   I                                   [irq260: re1]
                            100028                   I                                   [irq259: re0]
                            100025                   I                                   [swi5: fast taskq]
                            100023                   I                                   [swi6: Giant taskq]
                            100021                   I                                   [swi6: task queue]
                            100008                   I                                   [swi3: vm]
                            100007                   I                                   [swi4: clock]
                            100006                   I                                   [swi4: clock]
                            100005                   I                                   [swi1: netisr 0]
                               11     0     0     0  RL      (threaded)                  [idle]
                            100004                   CanRun                              [idle: cpu1]
                            100003                   CanRun                              [idle: cpu0]
                                1     0     1     0  RLs                                 [init]
                               10     0     0     0  DL      audit_wo 0xffffffff82193af0 [audit]
                                0     0     0     0  DLs     (threaded)                  [kernel]
                            100113                   D       -        0xfffff8000a2ee700 [dummynet]
                            100068                   D       -        0xfffff800032b3000 [CAM taskq]
                            100063                   D       -        0xfffff8000a050e00 [mca taskq]
                            100026                   D       -        0xfffff800032b3200 [kqueue taskq]
                            100024                   D       -        0xfffff800032b3700 [thread]
                            100022                   D       -        0xfffff800032b3c00 [ffs_trim taskq]
                            100020                   D       -        0xfffff800032b4400 [acpi_task_2]
                            100019                   D       -        0xfffff800032b4400 [acpi_task_1]
                            100018                   D       -        0xfffff800032b4400 [acpi_task_0]
                            100014                   D       -        0xfffff800031fa500 [firmware taskq]
                            100000                   D       swapin   0xffffffff82181768 [swapper]
                            95668     1     1     0  Z                                   getty
                              313     1   313     0  Z                                   devd
                            75263 70713 70404    62  Z                                   squidGuard
                             1664     1    21     0  Z+                                  sh
                            60024     1    21     0  Z                                   sleep
                            26064 33508   277     0  Z                                   bandwidthd
                            26121 34295   277     0  Z                                   bandwidthd
                            db:0:kdb.enter.default>  alltrace
                            
                            Tracing command intr pid 12 tid 100005 td 0xfffff8000320c490
                            sched_switch() at sched_switch+0x2b3/frame 0xfffffe001a36db20
                            mi_switch() at mi_switch+0xe1/frame 0xfffffe001a36db60
                            ithread_loop() at ithread_loop+0x190/frame 0xfffffe001a36dbb0
                            fork_exit() at fork_exit+0x9a/frame 0xfffffe001a36dbf0
                            fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001a36dbf0
                            --- trap 0, rip = 0, rsp = 0xfffffe001a36dcb0, rbp = 0 ---
                            
                            Tracing command idle pid 11 tid 100004 td 0xfffff8000320c920
                            sched_switch() at sched_switch+0x2b3/frame 0xfffffe001a368ac0
                            mi_switch() at mi_switch+0xe1/frame 0xfffffe001a368b00
                            sched_idletd() at sched_idletd+0xc9/frame 0xfffffe001a368bb0
                            fork_exit() at fork_exit+0x9a/frame 0xfffffe001a368bf0
                            fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001a368bf0
                            --- trap 0, rip = 0, rsp = 0xfffffe001a368cb0, rbp = 0 ---
                            
                            Tracing command idle pid 11 tid 100003 td 0xfffff8000320d000
                            sched_switch() at sched_switch+0x2b3/frame 0xfffffe001a363ac0
                            mi_switch() at mi_switch+0xe1/frame 0xfffffe001a363b00
                            sched_idletd() at sched_idletd+0x3a7/frame 0xfffffe001a363bb0
                            fork_exit() at fork_exit+0x9a/frame 0xfffffe001a363bf0
                            fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001a363bf0
                            --- trap 0, rip = 0, rsp = 0xfffffe001a363cb0, rbp = 0 ---
                            
                            Tracing command init pid 1 tid 100002 td 0xfffff8000320d490
                            sched_switch() at sched_switch+0x2b3/frame 0xfffffe001abfb6d0
                            mi_switch() at mi_switch+0xe1/frame 0xfffffe001abfb710
                            sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe001abfb790
                            sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe001abfb7c0
                            _sleep() at _sleep+0x27d/frame 0xfffffe001abfb840
                            kern_wait6() at kern_wait6+0x5f4/frame 0xfffffe001abfb8f0
                            sys_wait4() at sys_wait4+0x72/frame 0xfffffe001abfbae0
                            amd64_syscall() at amd64_syscall+0x351/frame 0xfffffe001abfbbf0
                            Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe001abfbbf0
                            --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x41397a, rsp = 0x7fffffffe748, rbp = 0x7fffffffe7d0 ---
                            
                            Tracing command audit pid 10 tid 100001 td 0xfffff8000320d920
                            sched_switch() at sched_switch+0x2b3/frame 0xfffffe001abf6a80
                            mi_switch() at mi_switch+0xe1/frame 0xfffffe001abf6ac0
                            sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe001abf6af0
                            _cv_wait() at _cv_wait+0x16d/frame 0xfffffe001abf6b40
                            audit_worker() at audit_worker+0xa3/frame 0xfffffe001abf6bb0
                            fork_exit() at fork_exit+0x9a/frame 0xfffffe001abf6bf0
                            fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001abf6bf0
                            --- trap 0, rip = 0, rsp = 0xfffffe001abf6cb0, rbp = 0 ---
                            
                            Tracing command kernel pid 0 tid 100110 td 0xfffff8000aab6920
                            sched_switch() at sched_switch+0x2b3/frame 0xfffffe003609aab0
                            mi_switch() at mi_switch+0xe1/frame 0xfffffe003609aaf0
                            sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe003609ab20
                            msleep_spin_sbt() at msleep_spin_sbt+0x1a3/frame 0xfffffe003609ab80
                            taskqueue_thread_loop() at taskqueue_thread_loop+0xfd/frame 0xfffffe003609abb0
                            fork_exit() at fork_exit+0x9a/frame 0xfffffe003609abf0
                            fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe003609abf0
                            --- trap 0, rip = 0, rsp = 0xfffffe003609acb0, rbp = 0 ---
                            
                            Tracing command kernel pid 0 tid 100068 td 0xfffff8000a295490
                            sched_switch() at sched_switch+0x2b3/frame 0xfffffe0035fc7a90
                            mi_switch() at mi_switch+0xe1/frame 0xfffffe0035fc7ad0
                            sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe0035fc7b00
                            _sleep() at _sleep+0x287/frame 0xfffffe0035fc7b80
                            taskqueue_thread_loop() at taskqueue_thread_loop+0xd5/frame 0xfffffe0035fc7bb0
                            fork_exit() at fork_exit+0x9a/frame 0xfffffe0035fc7bf0
                            fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0035fc7bf0
                            --- trap 0, rip = 0, rsp = 0xfffffe0035fc7cb0, rbp = 0 ---
                            
                            Tracing command kernel pid 0 tid 100063 td 0xfffff8000a037000
                            sched_switch() at sched_switch+0x2b3/frame 0xfffffe0035fadab0
                            mi_switch() at mi_switch+0xe1/frame 0xfffffe0035fadaf0
                            sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe0035fadb20
                            msleep_spin_sbt() at msleep_spin_sbt+0x1a3/frame 0xfffffe0035fadb80
                            taskqueue_thread_loop() at taskqueue_thread_loop+0xfd/frame 0xfffffe0035fadbb0
                            fork_exit() at fork_exit+0x9a/frame 0xfffffe0035fadbf0
                            fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0035fadbf0
                            --- trap 0, rip = 0, rsp = 0xfffffe0035fadcb0, rbp = 0 ---
                            
                            Tracing command kernel pid 0 tid 100026 td 0xfffff800033de490
                            sched_switch() at sched_switch+0x2b3/frame 0xfffffe001a3d6a90
                            mi_switch() at mi_switch+0xe1/frame 0xfffffe001a3d6ad0
                            sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe001a3d6b00
                            _sleep() at _sleep+0x287/frame 0xfffffe001a3d6b80
                            taskqueue_thread_loop() at taskqueue_thread_loop+0xd5/frame 0xfffffe001a3d6bb0
                            fork_exit() at fork_exit+0x9a/frame 0xfffffe001a3d6bf0
                            fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001a3d6bf0
                            --- trap 0, rip = 0, rsp = 0xfffffe001a3d6cb0, rbp = 0 ---
                            
                            Tracing command kernel pid 0 tid 100024 td 0xfffff800033df000
                            sched_switch() at sched_switch+0x2b3/frame 0xfffffe001a3cca90
                            mi_switch() at mi_switch+0xe1/frame 0xfffffe001a3ccad0
                            sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe001a3ccb00
                            _sleep() at _sleep+0x287/frame 0xfffffe001a3ccb80
                            taskqueue_thread_loop() at taskqueue_thread_loop+0xd5/frame 0xfffffe001a3ccbb0
                            fork_exit() at fork_exit+0x9a/frame 0xfffffe001a3ccbf0
                            fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001a3ccbf0
                            --- trap 0, rip = 0, rsp = 0xfffffe001a3cccb0, rbp = 0 ---
                            
                            Tracing command kernel pid 0 tid 100022 td 0xfffff800033df920
                            sched_switch() at sched_switch+0x2b3/frame 0xfffffe001a3c2a90
                            mi_switch() at mi_switch+0xe1/frame 0xfffffe001a3c2ad0
                            sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe001a3c2b00
                            _sleep() at _sleep+0x287/frame 0xfffffe001a3c2b80
                            taskqueue_thread_loop() at taskqueue_thread_loop+0xd5/frame 0xfffffe001a3c2bb0
                            fork_exit() at fork_exit+0x9a/frame 0xfffffe001a3c2bf0
                            fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001a3c2bf0
                            --- trap 0, rip = 0, rsp = 0xfffffe001a3c2cb0, rbp = 0 ---
                            
                            Tracing command kernel pid 0 tid 100020 td 0xfffff800033e0490
                            sched_switch() at sched_switch+0x2b3/frame 0xfffffe001a3b8ab0
                            mi_switch() at mi_switch+0xe1/frame 0xfffffe001a3b8af0
                            sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe001a3b8b20
                            msleep_spin_sbt() at msleep_spin_sbt+0x1a3/frame 0xfffffe001a3b8b80
                            taskqueue_thread_loop() at taskqueue_thread_loop+0xfd/frame 0xfffffe001a3b8bb0
                            fork_exit() at fork_exit+0x9a/frame 0xfffffe001a3b8bf0
                            fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001a3b8bf0
                            --- trap 0, rip = 0, rsp = 0xfffffe001a3b8cb0, rbp = 0 ---
                            
                            Tracing command kernel pid 0 tid 100019 td 0xfffff800033e0920
                            sched_switch() at sched_switch+0x2b3/frame 0xfffffe001a3b3ab0
                            mi_switch() at mi_switch+0xe1/frame 0xfffffe001a3b3af0
                            sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe001a3b3b20
                            msleep_spin_sbt() at msleep_spin_sbt+0x1a3/frame 0xfffffe001a3b3b80
                            taskqueue_thread_loop() at taskqueue_thread_loop+0xfd/frame 0xfffffe001a3b3bb0
                            fork_exit() at fork_exit+0x9a/frame 0xfffffe001a3b3bf0
                            fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001a3b3bf0
                            --- trap 0, rip = 0, rsp = 0xfffffe001a3b3cb0, rbp = 0 ---
                            
                            Tracing command kernel pid 0 tid 100018 td 0xfffff80003224000
                            sched_switch() at sched_switch+0x2b3/frame 0xfffffe001a3aeab0
                            mi_switch() at mi_switch+0xe1/frame 0xfffffe001a3aeaf0
                            sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe001a3aeb20
                            msleep_spin_sbt() at msleep_spin_sbt+0x1a3/frame 0xfffffe001a3aeb80
                            taskqueue_thread_loop() at taskqueue_thread_loop+0xfd/frame 0xfffffe001a3aebb0
                            fork_exit() at fork_exit+0x9a/frame 0xfffffe001a3aebf0
                            fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001a3aebf0
                            --- trap 0, rip = 0, rsp = 0xfffffe001a3aecb0, rbp = 0 ---
                            
                            Tracing command kernel pid 0 tid 100014 td 0xfffff80003225490
                            sched_switch() at sched_switch+0x2b3/frame 0xfffffe001a39aa90
                            mi_switch() at mi_switch+0xe1/frame 0xfffffe001a39aad0
                            sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe001a39ab00
                            _sleep() at _sleep+0x287/frame 0xfffffe001a39ab80
                            taskqueue_thread_loop() at taskqueue_thread_loop+0xd5/frame 0xfffffe001a39abb0
                            fork_exit() at fork_exit+0x9a/frame 0xfffffe001a39abf0
                            fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001a39abf0
                            --- trap 0, rip = 0, rsp = 0xfffffe001a39acb0, rbp = 0 ---
                            
                            Tracing command kernel pid 0 tid 100000 td 0xffffffff82181c30
                            sched_switch() at sched_switch+0x2b3/frame 0xffffffff824deb70
                            mi_switch() at mi_switch+0xe1/frame 0xffffffff824debb0
                            sleepq_timedwait() at sleepq_timedwait+0x3a/frame 0xffffffff824debe0
                            _sleep() at _sleep+0x26e/frame 0xffffffff824dec60
                            swapper() at swapper+0x28f/frame 0xffffffff824decb0
                            btext() at btext+0x2c
                            db:0:kdb.enter.default>  capture off
                            db:0:kdb.enter.default>  call doadump
                            = 0x6
                            db:0:kdb.enter.default>  reset
                            PC Engines APU BIOS build date: Apr  5 2014
                            Reading data from file [bootorder]
                            SeaBIOS (version ?-20140405_120742-frink)
                            SeaBIOS (version ?-20140405_120742-frink)
                            Found coreboot cbmem console @ 7e150400
                            Found mainboard PC Engines APU
                            Relocating init from 0x000e8e71 to 0x7e1065e0 (size 39259)
                            Found CBFS header at 0xfffffb90
                            found file "bootorder" in cbmem
                            CPU Mhz=1000
                            Found 27 PCI devices (max PCI bus is 05)
                            Copying PIR from 0x7e160400 to 0x000f27a0
                            Copying MPTABLE from 0x7e161400/7e161410 to 0x000f25b0 with length 1ec
                            Copying ACPI RSDP from 0x7e162400 to 0x000f2590
                            Copying SMBIOS entry point from 0x7e16d800 to 0x000f2570
                            Using pmtimer, ioport 0x808
                            Scan for VGA option rom
                            EHCI init on dev 00:12.2 (regs=0xf7f04420)
                            Found 1 lpt ports
                            Found 2 serial ports
                            AHCI controller at 11.0, iobase f7f04000, irq 11
                            EHCI init on dev 00:13.2 (regs=0xf7f04520)
                            EHCI init on dev 00:16.2 (regs=0xf7f04620)
                            Searching bootorder for: /rom@img/setup
                            Searching bootorder for: /rom@img/memtest
                            OHCI init on dev 00:12.0 (regs=0xf7f00000)
                            OHCI init on dev 00:13.0 (regs=0xf7f01000)
                            OHCI init on dev 00:14.5 (regs=0xf7f02000)
                            OHCI init on dev 00:16.0 (regs=0xf7f03000)
                            Searching bootorder for: /pci@i0cf8/usb@16,2/storage@1/*@0/*@0,0
                            Searching bootorder for: /pci@i0cf8/usb@16,2/usb-*@1
                            USB MSC vendor='Multiple' product='Card  Reader' rev='1.00' type=0 removable=1
                            USB MSC blksize=512 sectors=15523840
                            All threads complete.
                            Scan for option roms
                            Running option rom at c000:0003
                            
                            iPXE (http://ipxe.org) 00:00.0 C000 PCI2.10 PnP PMMpmm call arg1=1
                            pmm call arg1=0
                            +7E0DA5C0pmm call arg1=1
                            pmm call arg1=0
                            +7E03A5C0 C000
                            
                            Searching bootorder for: /rom@genroms/pxeboot.rom
                            
                            Build date: Apr  5 2014
                            System memory size: 2017 MB
                            
                            Press F12 for boot menu.
                            
                            Searching bootorder for: HALT
                            drive 0x000f2520: PCHS=0/0/0 translation=lba LCHS=966/255/63 s=15523840
                            Space available for UMB: c1000-ee800, f0000-f2520
                            Returned 245760 bytes of ZoneHigh
                            e820 map has 6 items:
                              0: 0000000000000000 - 000000000009fc00 = 1 RAM
                              1: 000000000009fc00 - 00000000000a0000 = 2 RESERVED
                              2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
                              3: 0000000000100000 - 000000007e14c000 = 1 RAM
                              4: 000000007e14c000 - 000000007f000000 = 2 RESERVED
                              5: 00000000f8000000 - 00000000f9000000 = 2 RESERVED
                            enter handle_19:
                              NULL
                            Booting from Hard Disk...
                            Booting from 0000:7c00
                            
                            1  pfSense
                            2  pfSense
                            
                            F6 PXE
                            Boot:  1
                            /boot/config: -S115200 -hConsoles: serial port
                            BIOS drive C: is disk0
                            BIOS 639kB/2064688kB available memory
                            
                            FreeBSD/x86 bootstrap loader, Revision 1.1
                            (root@pfs22-amd64-builder, Wed Nov  4 16:13:22 CST 2015)
                            Loading /boot/defaults/loader.conf
                            
                            For reference, this is what I normally see:
                            [code]*** FINAL System shutdown message from root@GateKeeper.localdomain ***
                            
                            System going down IMMEDIATELY
                            
                            pfSense is now shutting down ...
                            
                            Nov 20 00:30:26 lighttpd[45365]: (server.c.1567) server stopped by UID = 0 PID = 1
                            Waiting (max 60 seconds) for system process `vnlru' to stop...done
                            Waiting (max 60 seconds) for system process `bufdaemon' to stop...done
                            Waiting (max 60 seconds) for system process `syncer' to stop...
                            Syncing disks, vnodes remaining...0 0 done
                            All buffers synced.
                            Uptime: 5m27s
                            re0: link state changed to DOWN
                            re1: link state changed to DOWN
                            re1: link state changed to UP
                            re2: link state changed to DOWN
                            re2: link state changed to UP
                            PC Engines APU BIOS build date: Apr  5 2014
                            Reading data from file [bootorder]
                            SeaBIOS (version ?-20140405_120742-frink)
                            SeaBIOS (version ?-20140405_120742-frink)
                            Found coreboot cbmem console @ 7e150400
                            Found mainboard PC Engines APU
                            Relocating init from 0x000e8e71 to 0x7e1065e0 (size 39259)
                            Found CBFS header at 0xfffffb90
                            found file "bootorder" in cbmem
                            CPU Mhz=1001
                            Found 27 PCI devices (max PCI bus is 05)
                            Copying PIR from 0x7e160400 to 0x000f27a0
                            Copying MPTABLE from 0x7e161400/7e161410 to 0x000f25b0 with length 1ec
                            Copying ACPI RSDP from 0x7e162400 to 0x000f2590
                            Copying SMBIOS entry point from 0x7e16d800 to 0x000f2570
                            Using pmtimer, ioport 0x808
                            Scan for VGA option rom
                            EHCI init on dev 00:12.2 (regs=0xf7f04420)
                            Found 1 lpt ports
                            Found 2 serial ports
                            AHCI controller at 11.0, iobase f7f04000, irq 11
                            EHCI init on dev 00:13.2 (regs=0xf7f04520)
                            EHCI init on dev 00:16.2 (regs=0xf7f04620)
                            Searching bootorder for: /rom@img/setup
                            Searching bootorder for: /rom@img/memtest
                            OHCI init on dev 00:12.0 (regs=0xf7f00000)
                            OHCI init on dev 00:13.0 (regs=0xf7f01000)
                            OHCI init on dev 00:14.5 (regs=0xf7f02000)
                            OHCI init on dev 00:16.0 (regs=0xf7f03000)
                            Searching bootorder for: /pci@i0cf8/usb@16,2/storage@1/*@0/*@0,0
                            Searching bootorder for: /pci@i0cf8/usb@16,2/usb-*@1
                            USB MSC vendor='Multiple' product='Card  Reader' rev='1.00' type=0 removable=1
                            USB MSC blksize=512 sectors=15523840
                            All threads complete.
                            Scan for option roms
                            Running option rom at c000:0003
                            
                            iPXE (http://ipxe.org) 00:00.0 C000 PCI2.10 PnP PMMpmm call arg1=1
                            pmm call arg1=0
                            +7E0DA5C0pmm call arg1=1
                            pmm call arg1=0
                            +7E03A5C0 C000
                            
                            Searching bootorder for: /rom@genroms/pxeboot.rom
                            
                            Build date: Apr  5 2014
                            System memory size: 2017 MB
                            
                            Press F12 for boot menu.
                            
                            Searching bootorder for: HALT
                            drive 0x000f2520: PCHS=0/0/0 translation=lba LCHS=966/255/63 s=15523840
                            Space available for UMB: c1000-ee800, f0000-f2520
                            Returned 245760 bytes of ZoneHigh
                            e820 map has 6 items:
                              0: 0000000000000000 - 000000000009fc00 = 1 RAM
                              1: 000000000009fc00 - 00000000000a0000 = 2 RESERVED
                              2: 00000000000f0000 - 0000000000100000 = 2 RESERVED
                              3: 0000000000100000 - 000000007e14c000 = 1 RAM
                              4: 000000007e14c000 - 000000007f000000 = 2 RESERVED
                              5: 00000000f8000000 - 00000000f9000000 = 2 RESERVED
                            enter handle_19:
                              NULL
                            Booting from Hard Disk...
                            Booting from 0000:7c00
                            
                            1  pfSense
                            2  pfSense
                            [/code]
                            
                            I also see this in the startup:
                            [code]Trying to mount root from ufs:/dev/ufs/pfsense0 [ro,sync,noatime]...
                            mountroot: waiting for device /dev/ufs/pfsense0 ...
                            da0 at umass-sim0 bus 0 scbus6 target 0 lun 0
                            da0: <multiple card ="" reader="" 1.00=""> Removable Direct Access SCSI-4 device
                            da0: Serial Number 058F63666485
                            da0: 40.000MB/s transfers
                            da0: 7580MB (15523840 512 byte sectors: 255H 63S/T 966C)
                            da0: quirks=0x2 <no_6_byte>WARNING: / was not properly dismounted
                            Configuring crash dumps...
                            /dev/ufs/pfsense0: 30974 files, 2013156 used, 1766872 free (2632 frags, 220530 blocks, 0.1% fragmentation)
                            /dev/ufs/cf: FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED)
                            /dev/ufs/cf: 30 files, 3971 used, 97084 free (28 frags, 12132 blocks, 0.0% fragmentation)
                            Filesystems are clean, continuing...
                            Mounting filesystems...
                            Setting up memory disks... done.
                            [/code]
                            
                            Any idea where to see the crash dumps?[/thread]</no_6_byte></multiple>
                            
                            1 Reply Last reply Reply Quote 0
                            • S Offline
                              Sylas
                              last edited by

                              Hey guys

                              I have a similar problem on FreeBSD/FreeNAS and I just wanted to ask if you ever found a solution?

                              The error message from above:

                              Tracing command kernel pid 0 tid 100018 td 0xfffff80003224000
                              sched_switch() at sched_switch+0x2b3/frame 0xfffffe001a3aeab0
                              mi_switch() at mi_switch+0xe1/frame 0xfffffe001a3aeaf0
                              sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe001a3aeb20
                              msleep_spin_sbt() at msleep_spin_sbt+0x1a3/frame 0xfffffe001a3aeb80
                              taskqueue_thread_loop() at taskqueue_thread_loop+0xfd/frame 0xfffffe001a3aebb0
                              fork_exit() at fork_exit+0x9a/frame 0xfffffe001a3aebf0
                              fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001a3aebf0
                              –- trap 0, rip = 0, rsp = 0xfffffe001a3aecb0, rbp = 0 ---

                              Tracing command kernel pid 0 tid 100014 td 0xfffff80003225490
                              sched_switch() at sched_switch+0x2b3/frame 0xfffffe001a39aa90
                              mi_switch() at mi_switch+0xe1/frame 0xfffffe001a39aad0
                              sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe001a39ab00
                              _sleep() at _sleep+0x287/frame 0xfffffe001a39ab80
                              taskqueue_thread_loop() at taskqueue_thread_loop+0xd5/frame 0xfffffe001a39abb0
                              fork_exit() at fork_exit+0x9a/frame 0xfffffe001a39abf0
                              fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe001a39abf0
                              --- trap 0, rip = 0, rsp = 0xfffffe001a39acb0, rbp = 0 ---

                              I do see something like that - here's the https://bugs.freenas.org/issues/15089 –> second.jpg ...

                              I'd appreciate any help!

                              Thanks and best regards

                              1 Reply Last reply Reply Quote 0
                              • First post
                                Last post
                              Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.