Could you help me to diagnose this crash dump?



  • My pfSense crash almost everyday for about a month.

    Here, crash report from pfSense.

    
    					Crash report begins.  Anonymous machine information:
    
    amd64
    10.3-RELEASE-p5
    FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016     root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense
    
    Crash report details:
    
    Filename: /var/crash/bounds
    3
    
    Filename: /var/crash/info.0
    Dump header from device /dev/label/swap0
      Architecture: amd64
      Architecture Version: 1
      Dump Length: 157184B (0 MB)
      Blocksize: 512
      Dumptime: Wed Aug 17 10:19:00 2016
      Hostname: proxy.nssc.ac.th
      Magic: FreeBSD Text Dump
      Version String: FreeBSD 10.3-RELEASE-p3 #2 1988fec(RELENG_2_3_1): Wed May 25 14:14:46 CDT 2016
        root@ce23-amd64-builder:/builder/pfsense-231/tmp/obj/builder/pfsense-231/tmp/FreeBSD-src/sys/pfSense
      Panic String: sbdrop
      Dump Parity: 4091060750
      Bounds: 0
      Dump Status: good
    
    Filename: /var/crash/info.1
    Dump header from device /dev/label/swap0
      Architecture: amd64
      Architecture Version: 1
      Dump Length: 157184B (0 MB)
      Blocksize: 512
      Dumptime: Wed Aug 17 12:57:34 2016
      Hostname: proxy.nssc.ac.th
      Magic: FreeBSD Text Dump
      Version String: FreeBSD 10.3-RELEASE-p3 #2 1988fec(RELENG_2_3_1): Wed May 25 14:14:46 CDT 2016
        root@ce23-amd64-builder:/builder/pfsense-231/tmp/obj/builder/pfsense-231/tmp/FreeBSD-src/sys/pfSense
      Panic String: sbdrop
      Dump Parity: 2582880782
      Bounds: 1
      Dump Status: good
    
    Filename: /var/crash/info.2
    Dump header from device /dev/label/swap0
      Architecture: amd64
      Architecture Version: 1
      Dump Length: 157184B (0 MB)
      Blocksize: 512
      Dumptime: Thu Aug 18 12:14:34 2016
      Hostname: proxy.nssc.ac.th
      Magic: FreeBSD Text Dump
      Version String: FreeBSD 10.3-RELEASE-p3 #2 1988fec(RELENG_2_3_1): Wed May 25 14:14:46 CDT 2016
        root@ce23-amd64-builder:/builder/pfsense-231/tmp/obj/builder/pfsense-231/tmp/FreeBSD-src/sys/pfSense
      Panic String: sbsndptr: sockbuf 0xfffff802b2d049b0 and mbuf 0xfffff8034df66a00 clashing
      Dump Parity: 6350396
      Bounds: 2
      Dump Status: good
    
    Filename: /var/crash/info.last
    Dump header from device /dev/label/swap0
      Architecture: amd64
      Architecture Version: 1
      Dump Length: 157184B (0 MB)
      Blocksize: 512
      Dumptime: Thu Aug 18 12:14:34 2016
      Hostname: proxy.nssc.ac.th
      Magic: FreeBSD Text Dump
      Version String: FreeBSD 10.3-RELEASE-p3 #2 1988fec(RELENG_2_3_1): Wed May 25 14:14:46 CDT 2016
        root@ce23-amd64-builder:/builder/pfsense-231/tmp/obj/builder/pfsense-231/tmp/FreeBSD-src/sys/pfSense
      Panic String: sbsndptr: sockbuf 0xfffff802b2d049b0 and mbuf 0xfffff8034df66a00 clashing
      Dump Parity: 6350396
      Bounds: 2
      Dump Status: good
    
    Filename: /var/crash/minfree
    2048
    
    Filename: /var/crash/textdump.tar.0
    ddb.txt06000014000012754753644  7113 ustarrootwheeldb:0:kdb.enter.default>  run lockinfo
    db:1:lockinfo> show locks
    No such command
    db:1:locks>  show alllocks
    No such command
    db:1:alllocks>  show lockedvnods
    Locked vnodes
    db:0:kdb.enter.default>  show pcpu
    cpuid        = 1
    dynamic pcpu = 0xfffffe04586c0500
    curthread    = 0xfffff80171cb24b0: pid 45041 "squid"
    curpcb       = 0xfffffe045ca7cb80
    fpcurthread  = 0xfffff80171cb24b0: pid 45041 "squid"
    idlethread   = 0xfffff80006386960: tid 100004 "idle: cpu1"
    curpmap      = 0xfffff800111c5bb8
    tssp         = 0xffffffff821134f8
    commontssp   = 0xffffffff821134f8
    rsp0         = 0xfffffe045ca7cb80
    gs32p        = 0xffffffff82114f50
    ldt          = 0xffffffff82114f90
    tss          = 0xffffffff82114f80
    db:0:kdb.enter.default>  bt
    Tracing pid 45041 tid 100182 td 0xfffff80171cb24b0
    kdb_enter() at kdb_enter+0x3e/frame 0xfffffe045ca7c700
    vpanic() at vpanic+0x146/frame 0xfffffe045ca7c740
    panic() at panic+0x43/frame 0xfffffe045ca7c7a0
    sbcut_internal() at sbcut_internal+0x273/frame 0xfffffe045ca7c7b0
    sbflush_internal() at sbflush_internal+0x18/frame 0xfffffe045ca7c7d0
    sbflush() at sbflush+0x46/frame 0xfffffe045ca7c800
    tcp_disconnect() at tcp_disconnect+0x52/frame 0xfffffe045ca7c830
    tcp_usr_disconnect() at tcp_usr_disconnect+0x84/frame 0xfffffe045ca7c860
    soclose() at soclose+0x3c/frame 0xfffffe045ca7c8a0
    _fdrop() at _fdrop+0x29/frame 0xfffffe045ca7c8c0
    closef() at closef+0x21e/frame 0xfffffe045ca7c950
    closefp() at closefp+0x98/frame 0xfffffe045ca7c990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045ca7cab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045ca7cab0
    --- syscall (6, FreeBSD ELF64, sys_close), rip = 0x802e036aa, rsp = 0x7fffffffe838, rbp = 0x7fffffffe850 ---
    db:0:kdb.enter.default>  ps
      pid  ppid  pgrp   uid   state   wmesg         wchan        cmd
    70396 69113   271     0  S       nanslp   0xffffffff82001171 sleep
    67718 65567    24     0  S       nanslp   0xffffffff82001172 sleep
    50630   271   271     0  S       accept   0xfffff8001144f066 php-fpm
    48339 45041 44750   100  S       sbwait   0xfffff80249a413fc squidGuard
    48118 45041 44750   100  S       sbwait   0xfffff80240937c24 squidGuard
    47892 45041 44750   100  S       sbwait   0xfffff80249a136b4 squidGuard
    47884 45041 44750   100  S       sbwait   0xfffff801eed0496c squidGuard
    28250 45041 44750   100  S       sbwait   0xfffff802403786b4 squidGuard
    27989 45041 44750   100  S       sbwait   0xfffff801eed186b4 squidGuard
    27740 45041 44750   100  S       sbwait   0xfffff801eed0596c squidGuard
    27617 45041 44750   100  S       sbwait   0xfffff80171ca7144 squidGuard
    93188 93086 93188     0  S+      ttyin    0xfffff8000f02c0a8 sh
    93086 92869 93086     0  S+      wait     0xfffff801713a2000 sh
    92958 15848 92958     0  Ss      (threaded)                  sshlockout_pf
    100204                   S       nanslp   0xffffffff82001173 sshlockout_pf
    100145                   S       piperd   0xfffff8001121eba0 sshlockout_pf
    92869     1 92869     0  Ss+     wait     0xfffff800111b89e0 login
    79892     1 79618     0  S       kqread   0xfffff801eec53b00 lighttpd_ls
    79071 78712 78712     0  S       nanslp   0xffffffff82001170 minicron
    78712     1 78712     0  Ss      wait     0xfffff801eedf64f0 minicron
    78446 78303 78303     0  S       nanslp   0xffffffff82001173 minicron
    78303     1 78303     0  Ss      wait     0xfffff80171bd79e0 minicron
    78297 77792 77792     0  S       nanslp   0xffffffff82001172 minicron
    77792     1 77792     0  Ss      wait     0xfffff8001100a9e0 minicron
    77776     1 77776    62  Ss      kqread   0xfffff801eec2ca00 ftp-proxy
    71989 45041 44750   100  S       select   0xfffff801ee3895c0 pinger
    71694 45041 44750   100  S       sbwait   0xfffff801717d36b4 squidGuard
    71422 45041 44750   100  S       sbwait   0xfffff801717153fc squidGuard
    71205 45041 44750   100  S       sbwait   0xfffff801713ed3fc squidGuard
    71062 45041 44750   100  S       sbwait   0xfffff801711f9c24 squidGuard
    70736 45041 44750   100  S       sbwait   0xfffff801713ee3fc squidGuard
    70450 45041 44750   100  S       sbwait   0xfffff801717d33fc squidGuard
    70158 45041 44750   100  S       sbwait   0xfffff8017170f3fc squidGuard
    70050 45041 44750   100  S       sbwait   0xfffff8017170fc24 squidGuard
    69113     1   271     0  S       wait     0xfffff800112ad000 sh
    50158 45041 44750   100  S       select   0xfffff801718fa540 pinger
    50100 45041 44750   100  S       piperd   0xfffff8001121e2e8 unlinkd
    45041 44750 44750   100  R       CPU 1                       squid
    44750     1 44750   100  Ss      wait     0xfffff800112524f0 squid
    15848     1 15848     0  Ss      select   0xfffff8001103b6c0 syslogd
    65567     1    24     0  S+      wait     0xfffff801713a34f0 sh
    62391     1 62391     0  Ss      select   0xfffff801712405c0 bsnmpd
    48539     1 48539  1002  Ss      select   0xfffff80171407640 dhcpd
    42314     1 42314     0  Ss      (threaded)                  ntpd
    100104                   S       select   0xfffff801712362c0 ntpd
    41886     1 41886    59  Ss      (threaded)                  unbound
    100168                   S       kqread   0xfffff8001116da00 unbound
    100167                   S       kqread   0xfffff800116cc900 unbound
    100166                   S       kqread   0xfffff800116cca00 unbound
    100106                   S       kqread   0xfffff80011b35e00 unbound
    40907     1 40907     0  Ss      nanslp   0xffffffff82001172 cron
    40326 39912 39912     0  S       kqread   0xfffff801711e1b00 nginx
    40022 39912 39912     0  S       kqread   0xfffff8000fffeb00 nginx
    39912     1 39912     0  Ss      pause    0xfffff800116020a8 nginx
    39707     1 39707     0  Ss      (threaded)                  dpinger
    100163                   S       accept   0xfffff80011450b46 dpinger
    100162                   S       nanslp   0xffffffff82001172 dpinger
    100161                   S       nanslp   0xffffffff82001172 dpinger
    100160                   S       sbwait   0xfffff80011482144 dpinger
    100159                   S       uwait    0xfffff8001106a600 dpinger
    39569     1 39569     0  Ss      (threaded)                  dpinger
    100158                   S       accept   0xfffff800114825d6 dpinger
    100157                   S       nanslp   0xffffffff82001173 dpinger
    100156                   S       nanslp   0xffffffff82001173 dpinger
    100155                   S       sbwait   0xfffff8001148296c dpinger
    100154                   S       uwait    0xfffff80011237c00 dpinger
    39016     1 39016     0  Ss      (threaded)                  dpinger
    100153                   S       accept   0xfffff80011483066 dpinger
    100152                   S       nanslp   0xffffffff82001173 dpinger
    100151                   S       nanslp   0xffffffff82001173 dpinger
    100150                   S       sbwait   0xfffff8001193a3fc dpinger
    100112                   S       uwait    0xfffff8001106b200 dpinger
    38741     1 38741     0  Ss      (threaded)                  dpinger
    100149                   S       accept   0xfffff8001179a066 dpinger
    100148                   S       nanslp   0xffffffff82001172 dpinger
    100147                   S       nanslp   0xffffffff82001173 dpinger
    100146                   S       sbwait   0xfffff8001148496c dpinger
    100131                   S       uwait    0xfffff80011237e00 dpinger
    26765     1 26765     0  Ss      select   0xfffff800116fee40 xinetd
    24799     1 24799     0  Ss      bpf      0xfffff8001100de00 filterlog
    20518     1 20518    65  Ss      select   0xfffff80011713140 dhclient
    16505     1 16505     0  Ss      select   0xfffff800117135c0 dhclient
    12263     1 12263    65  Ss      select   0xfffff80011282640 dhclient
     7755     1  7755     0  Ss      select   0xfffff8001103c840 dhclient
      324     1   324     0  Ss      select   0xfffff80011236140 devd
      310   309   309     0  S       kqread   0xfffff80011169200 check_reload_status
      309     1   309     0  Ss      kqread   0xfffff8000ffffd00 check_reload_status
      271     1   271     0  Ss      kqread   0xfffff8001116ae00 php-fpm
       58     0     0     0  DL      mdwait   0xfffff8001122d800 [md0]
       23     0     0     0  DL      vlruwt   0xfffff8001100b4f0 [vnlru]
       22     0     0     0  DL      syncer   0xffffffff82052108 [syncer]
       21     0     0     0  DL      (threaded)                  [bufdaemon]
    100082                   D       sdflush  0xfffff8001100eae8 [/ worker]
    100071                   D       psleep   0xffffffff82051304 [bufdaemon]
       20     0     0     0  DL      pollid   0xffffffff81fffa90 [idlepoll]
       19     0     0     0  DL      pgzero   0xffffffff8206243c [pagezero]
        9     0     0     0  DL      psleep   0xffffffff820617c0 [vmdaemon]
        8     0     0     0  DL      (threaded)                  [pagedaemon]
    100074                   D       umarcl   0xffffffff82061140 [uma]
    100067                   D       psleep   0xffffffff82112804 [pagedaemon]
        7     0     0     0  DL      idle     0xfffff8000654b9e0 [enc_daemon0]
        6     0     0     0  DL      waiting_ 0xffffffff821032c0 [sctp_iterator]
        5     0     0     0  DL      pftm     0xffffffff80d5d630 [pf purge]
       18     0     0     0  DL      cooling  0xfffff8000f00a358 [acpi_cooling1]
       17     0     0     0  DL      tzpoll   0xffffffff81f3dd20 [acpi_thermal]
       16     0     0     0  DL      (threaded)                  [usb]
    100053                   D       -        0xfffffe0000b90e70 [usbus2]
    100052                   D       -        0xfffffe0000b90e18 [usbus2]
    100051                   D       -        0xfffffe0000b90dc0 [usbus2]
    100050                   D       -        0xfffffe0000b90d68 [usbus2]
    100049                   D       -        0xfffffe0000b90d10 [usbus2]
    100047                   D       -        0xfffffe00009ece70 [usbus1]
    100046                   D       -        0xfffffe00009ece18 [usbus1]
    100045                   D       -        0xfffffe00009ecdc0 [usbus1]
    100044                   D       -        0xfffffe00009ecd68 [usbus1]
    100043                   D       -        0xfffffe00009ecd10 [usbus1]
    100041                   D       -        0xfffffe0000b886c0 [usbus0]
    100040                   D       -        0xfffffe0000b88668 [usbus0]
    100039                   D       -        0xfffffe0000b88610 [usbus0]
    100038                   D       -        0xfffffe0000b885b8 [usbus0]
    100037                   D       -        0xfffffe0000b88560 [usbus0]
        4     0     0     0  DL      (threaded)                  [cam]
    100065                   D       -        0xffffffff81f35cc8 [scanner]
    100025                   D       -        0xffffffff81f35e80 [doneq0]
       15     0     0     0  DL      -        0xffffffff81f575c0 [rand_harvestq]
        3     0     0     0  DL      crypto_r 0xffffffff8205fc98 [crypto returns]
        2     0     0     0  DL      crypto_w 0xffffffff8205fb40 [crypto]
       14     0     0     0  DL      (threaded)                  [geom]
    100019                   D       -        0xffffffff820f79f8 [g_down]
    100018                   D       -        0xffffffff820f79f0 [g_up]
    100017                   D       -        0xffffffff820f79e8 [g_event]
       13     0     0     0  DL      (threaded)                  [ng_queue]
    100016                   D       sleep    0xffffffff81ef4318 [ng_queue3]
    100015                   D       sleep    0xffffffff81ef4318 [ng_queue2]
    100014                   D       sleep    0xffffffff81ef4318 [ng_queue1]
    100013                   D       sleep    0xffffffff81ef4318 [ng_queue0]
       12     0     0     0  WL      (threaded)                  [intr]
    100077                   I                                   [swi1: netisr 3]
    100076                   I                                   [swi1: netisr 2]
    100075                   I                                   [swi1: netisr 1]
    100063                   I                                   [swi1: pfsync]
    100061                   I                                   [swi1: pf send]
    100058                   I                                   [irq1: atkbd0]
    100057                   I                                   [swi0: uart uart]
    100054                   I                                   [irq265: ahci0]
    100048                   I                                   [irq23: ehci1]
    100042                   I                                   [irq16: ehci0]
    100036                   I                                   [irq264: xhci0]
    100035                   I                                   [irq18: bge1 bge3]
    100034                   I                                   [irq17: bge0 bge2]
    100033                   I                                   [swi5: fast taskq]
    100031                   I                                   [swi6: Giant taskq]
    100029                   I                                   [swi6: task queue]
    100012                   I                                   [swi1: netisr 0]
    100011                   I                                   [swi3: vm]
    100010                   I                                   [swi4: clock]
    100009                   I                                   [swi4: clock]
    100008                   I                                   [swi4: clock]
    100007                   I                                   [swi4: clock]
       11     0     0     0  RL      (threaded)                  [idle]
    100006                   Run     CPU 3                       [idle: cpu3]
    100005                   Run     CPU 2                       [idle: cpu2]
    100004                   CanRun                              [idle: cpu1]
    100003                   Run     CPU 0                       [idle: cpu0]
        1     0     1     0  SLs     wait     0xfffff800063844f0 [init]
       10     0     0     0  DL      audit_wo 0xffffffff82109d60 [audit]
        0     0     0     0  DLs     (threaded)                  [kernel]
    100126                   D       -        0xfffff800116cf300 [dummynet]
    100064                   D       -        0xfffff800063aa600 [CAM taskq]
    100059                   D       -        0xfffff800067ff400 [mca taskq]
    100032                   D       -        0xfffff800063a9500 [thread]
    100030                   D       -        0xfffff800063a9a00 [ffs_trim taskq]
    100028                   D       -        0xfffff800063aa200 [acpi_task_2]
    100027                   D       -        0xfffff800063aa200 [acpi_task_1]
    100026                   D       -        0xfffff800063aa200 [acpi_task_0]
    100024                   D       -        0xfffff800063aa800 [kqueue taskq]
    100020                   D       -        0xfffff80006376100 [firmware taskq]
    100000                   D       swapin   0xffffffff820f7af8 [swapper]
    db:0:kdb.enter.default>  alltrace
    
    Tracing command sleep pid 70396 tid 100225 td 0xfffff80171c56000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045cb4e780
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045cb4e7c0
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045cb4e840
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe045cb4e870
    _sleep() at _sleep+0x238/frame 0xfffffe045cb4e8f0
    kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe045cb4e950
    sys_nanosleep() at sys_nanosleep+0x51/frame 0xfffffe045cb4e990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045cb4eab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045cb4eab0
    --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x8008f1f3a, rsp = 0x7fffffffec98, rbp = 0x7fffffffecd0 ---
    
    Tracing command sleep pid 67718 tid 100176 td 0xfffff80171258000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045ca5e780
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045ca5e7c0
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045ca5e840
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe045ca5e870
    _sleep() at _sleep+0x238/frame 0xfffffe045ca5e8f0
    kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe045ca5e950
    sys_nanosleep() at sys_nanosleep+0x51/frame 0xfffffe045ca5e990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045ca5eab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045ca5eab0
    --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x8008f1f3a, rsp = 0x7fffffffec68, rbp = 0x7fffffffeca0 ---
    
    Tracing command php-fpm pid 50630 tid 100184 td 0xfffff80011d444b0
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045ca86710
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045ca86750
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045ca867d0
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe045ca86800
    _sleep() at _sleep+0x27d/frame 0xfffffe045ca86880
    kern_accept4() at kern_accept4+0x1b1/frame 0xfffffe045ca86930
    accept1() at accept1+0x63/frame 0xfffffe045ca86990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045ca86ab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045ca86ab0
    --- syscall (30, FreeBSD ELF64, sys_accept), rip = 0x801b44fda, rsp = 0x7ffffffec6e8, rbp = 0x7ffffffec720 ---
    
    Tracing command squidGuard pid 48339 tid 100083 td 0xfffff800111c24b0
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045c903640
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045c903680
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045c903700
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe045c903730
    _sleep() at _sleep+0x27d/frame 0xfffffe045c9037b0
    sbwait() at sbwait+0x4f/frame 0xfffffe045c9037d0
    soreceive_generic() at soreceive_generic+0x26c/frame 0xfffffe045c9038a0
    dofileread() at dofileread+0x95/frame 0xfffffe045c9038f0
    kern_readv() at kern_readv+0x68/frame 0xfffffe045c903940
    sys_read() at sys_read+0x63/frame 0xfffffe045c903990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045c903ab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045c903ab0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x80116b68a, rsp = 0x7ffffffdab38, rbp = 0x7ffffffdab50 ---
    
    Tracing command squidGuard pid 48118 tid 100232 td 0xfffff801eedf74b0
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045cb71640
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045cb71680
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045cb71700
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe045cb71730
    _sleep() at _sleep+0x27d/frame 0xfffffe045cb717b0
    sbwait() at sbwait+0x4f/frame 0xfffffe045cb717d0
    soreceive_generic() at soreceive_generic+0x26c/frame 0xfffffe045cb718a0
    dofileread() at dofileread+0x95/frame 0xfffffe045cb718f0
    kern_readv() at kern_readv+0x68/frame 0xfffffe045cb71940
    sys_read() at sys_read+0x63/frame 0xfffffe045cb71990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045cb71ab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045cb71ab0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x80116b68a, rsp = 0x7ffffffdab38, rbp = 0x7ffffffdab50 ---
    
    Tracing command squidGuard pid 47892 tid 100194 td 0xfffff801ee3f54b0
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045cab8640
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045cab8680
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045cab8700
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe045cab8730
    _sleep() at _sleep+0x27d/frame 0xfffffe045cab87b0
    sbwait() at sbwait+0x4f/frame 0xfffffe045cab87d0
    soreceive_generic() at soreceive_generic+0x26c/frame 0xfffffe045cab88a0
    dofileread() at dofileread+0x95/frame 0xfffffe045cab88f0
    kern_readv() at kern_readv+0x68/frame 0xfffffe045cab8940
    sys_read() at sys_read+0x63/frame 0xfffffe045cab8990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045cab8ab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045cab8ab0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x80116b68a, rsp = 0x7ffffffdab38, rbp = 0x7ffffffdab50 ---
    
    Tracing command squidGuard pid 47884 tid 100195 td 0xfffff801ee3f5000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045cabd640
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045cabd680
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045cabd700
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe045cabd730
    _sleep() at _sleep+0x27d/frame 0xfffffe045cabd7b0
    sbwait() at sbwait+0x4f/frame 0xfffffe045cabd7d0
    soreceive_generic() at soreceive_generic+0x26c/frame 0xfffffe045cabd8a0
    dofileread() at dofileread+0x95/frame 0xfffffe045cabd8f0
    kern_readv() at kern_readv+0x68/frame 0xfffffe045cabd940
    sys_read() at sys_read+0x63/frame 0xfffffe045cabd990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045cabdab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045cabdab0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x80116b68a, rsp = 0x7ffffffdab38, rbp = 0x7ffffffdab50 ---
    
    Tracing command squidGuard pid 28250 tid 100210 td 0xfffff80171768000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045cb08640
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045cb08680
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045cb08700
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe045cb08730
    _sleep() at _sleep+0x27d/frame 0xfffffe045cb087b0
    sbwait() at sbwait+0x4f/frame 0xfffffe045cb087d0
    soreceive_generic() at soreceive_generic+0x26c/frame 0xfffffe045cb088a0
    dofileread() at dofileread+0x95/frame 0xfffffe045cb088f0
    kern_readv() at kern_readv+0x68/frame 0xfffffe045cb08940
    sys_read() at sys_read+0x63/frame 0xfffffe045cb08990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045cb08ab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045cb08ab0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x80116b68a, rsp = 0x7ffffffdab38, rbp = 0x7ffffffdab50 ---
    
    Tracing command squidGuard pid 27989 tid 100101 td 0xfffff800111b94b0
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045c95d640
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045c95d680
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045c95d700
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe045c95d730
    _sleep() at _sleep+0x27d/frame 0xfffffe045c95d7b0
    sbwait() at sbwait+0x4f/frame 0xfffffe045c95d7d0
    soreceive_generic() at soreceive_generic+0x26c/frame 0xfffffe045c95d8a0
    dofileread() at dofileread+0x95/frame 0xfffffe045c95d8f0
    kern_readv() at kern_readv+0x68/frame 0xfffffe045c95d940
    sys_read() at sys_read+0x63/frame 0xfffffe045c95d990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045c95dab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045c95dab0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x80116b68a, rsp = 0x7ffffffdab38, rbp = 0x7ffffffdab50 ---
    
    Tracing command squidGuard pid 27740 tid 100226 td 0xfffff80171ccd960
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045cb53640
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045cb53680
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045cb53700
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe045cb53730
    _sleep() at _sleep+0x27d/frame 0xfffffe045cb537b0
    sbwait() at sbwait+0x4f/frame 0xfffffe045cb537d0
    soreceive_generic() at soreceive_generic+0x26c/frame 0xfffffe045cb538a0
    dofileread() at dofileread+0x95/frame 0xfffffe045cb538f0
    kern_readv() at kern_readv+0x68/frame 0xfffffe045cb53940
    sys_read() at sys_read+0x63/frame 0xfffffe045cb53990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045cb53ab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045cb53ab0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x80116b68a, rsp = 0x7ffffffdab38, rbp = 0x7ffffffdab50 ---
    
    Tracing command squidGuard pid 27617 tid 100208 td 0xfffff80171768960
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045cafe640
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045cafe680
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045cafe700
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe045cafe730
    _sleep() at _sleep+0x27d/frame 0xfffffe045cafe7b0
    sbwait() at sbwait+0x4f/frame 0xfffffe045cafe7d0
    soreceive_generic() at soreceive_generic+0x26c/frame 0xfffffe045cafe8a0
    dofileread() at dofileread+0x95/frame 0xfffffe045cafe8f0
    kern_readv() at kern_readv+0x68/frame 0xfffffe045cafe940
    sys_read() at sys_read+0x63/frame 0xfffffe045cafe990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045cafeab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045cafeab0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x80116b68a, rsp = 0x7ffffffdab38, rbp = 0x7ffffffdab50 ---
    
    Tracing command sh pid 93188 tid 100199 td 0xfffff801eede9960
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045cad1650
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045cad1690
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045cad1710
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe045cad1740
    _cv_wait_sig() at _cv_wait_sig+0x17a/frame 0xfffffe045cad1790
    tty_wait() at tty_wait+0x1c/frame 0xfffffe045cad17b0
    ttydisc_read() at ttydisc_read+0xf4/frame 0xfffffe045cad1810
    ttydev_read() at ttydev_read+0x86/frame 0xfffffe045cad1840
    devfs_read_f() at devfs_read_f+0xd8/frame 0xfffffe045cad18a0
    dofileread() at dofileread+0x95/frame 0xfffffe045cad18f0
    kern_readv() at kern_readv+0x68/frame 0xfffffe045cad1940
    sys_read() at sys_read+0x63/frame 0xfffffe045cad1990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045cad1ab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045cad1ab0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800dee68a, rsp = 0x7fffffffe768, rbp = 0x7fffffffe860 ---
    
    Tracing command sh pid 93086 tid 100203 td 0xfffff80011f5e000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045cae5560
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045cae55a0
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045cae5620
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe045cae5650
    _sleep() at _sleep+0x27d/frame 0xfffffe045cae56d0
    kern_wait6() at kern_wait6+0x5de/frame 0xfffffe045cae57a0
    sys_wait4() at sys_wait4+0x72/frame 0xfffffe045cae5990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045cae5ab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045cae5ab0
    --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d8bd7a, rsp = 0x7fffffffe818, rbp = 0x7fffffffe8c0 ---
    
    Tracing command sshlockout_pf pid 92958 tid 100204 td 0xfffff80171c564b0
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045caea780
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045caea7c0
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045caea840
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe045caea870
    _sleep() at _sleep+0x238/frame 0xfffffe045caea8f0
    kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe045caea950
    sys_nanosleep() at sys_nanosleep+0x51/frame 0xfffffe045caea990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045caeaab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045caeaab0
    --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b18f3a, rsp = 0x7fffdfffdf48, rbp = 0x7fffdfffdf70 ---
    
    Tracing command sshlockout_pf pid 92958 tid 100145 td 0xfffff800112b04b0
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045ca136d0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045ca13710
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045ca13790
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe045ca137c0
    _sleep() at _sleep+0x27d/frame 0xfffffe045ca13840
    pipe_read() at pipe_read+0x2aa/frame 0xfffffe045ca138a0
    dofileread() at dofileread+0x95/frame 0xfffffe045ca138f0
    kern_readv() at kern_readv+0x68/frame 0xfffffe045ca13940
    sys_read() at sys_read+0x63/frame 0xfffffe045ca13990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045ca13ab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045ca13ab0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b7b68a, rsp = 0x7fffffffe7d8, rbp = 0x7fffffffe7f0 ---
    
    Tracing command login pid 92869 tid 100078 td 0xfffff800111c2960
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045c8ea560
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045c8ea5a0
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045c8ea620
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe045c8ea650
    _sleep() at _sleep+0x27d/frame 0xfffffe045c8ea6d0
    kern_wait6() at kern_wait6+0x5de/frame 0xfffffe045c8ea7a0
    sys_wait4() at sys_wait4+0x72/frame 0xfffffe045c8ea990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045c8eaab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045c8eaab0
    --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d15d7a, rsp = 0x7fffffffebc8, rbp = 0x7fffffffed40 ---
    
    Tracing command lighttpd_ls pid 79892 tid 100094 td 0xfffff8001136e960
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045c93a530
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045c93a570
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045c93a5f0
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe045c93a620
    _sleep() at _sleep+0x238/frame 0xfffffe045c93a6a0
    kern_kevent_fp() at kern_kevent_fp+0x399/frame 0xfffffe045c93a880
    kern_kevent() at kern_kevent+0x9f/frame 0xfffffe045c93a8e0
    sys_kevent() at sys_kevent+0x12a/frame 0xfffffe045c93a990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045c93aab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045c93aab0
    --- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x8012038fa, rsp = 0x7fffffffeac8, rbp = 0x7fffffffeb10 ---
    
    Tracing command minicron pid 79071 tid 100125 td 0xfffff80011606000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045c9d5780
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045c9d57c0
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045c9d5840
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe045c9d5870
    _sleep() at _sleep+0x238/frame 0xfffffe045c9d58f0
    kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe045c9d5950
    sys_nanosleep() at sys_nanosleep+0x51/frame 0xfffffe045c9d5990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045c9d5ab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045c9d5ab0
    --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x8008f2f3a, rsp = 0x7fffffffeb88, rbp = 0x7fffffffebe0 ---
    
    Tracing command minicron pid 78712 tid 100200 td 0xfffff801eede94b0
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045cad6560
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045cad65a0
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045cad6620
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe045cad6650
    _sleep() at _sleep+0x27d/frame 0xfffffe045cad66d0
    kern_wait6() at kern_wait6+0x5de/frame 0xfffffe045cad67a0
    sys_wait4() at sys_wait4+0x72/frame 0xfffffe045cad6990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045cad6ab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045cad6ab0
    --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x8008f2d7a, rsp = 0x7fffffffebe8, rbp = 0x7fffffffec30 ---
    
    Tracing command minicron pid 78446 tid 100175 td 0xfffff801711f7000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045ca59780
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045ca597c0
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045ca59840
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe045ca59870
    _sleep() at _sleep+0x238/frame 0xfffffe045ca598f0
    kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe045ca59950
    sys_nanosleep() at sys_nanosleep+0x51/frame 0xfffffe045ca59990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045ca59ab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045ca59ab0
    --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x8008f2f3a, rsp = 0x7fffffffeb98, rbp = 0x7fffffffebf0 ---
    
    Tracing command minicron pid 78303 tid 100174 td 0xfffff801711f74b0
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045ca54560
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045ca545a0
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045ca54620
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe045ca54650
    _sleep() at _sleep+0x27d/frame 0xfffffe045ca546d0
    kern_wait6() at kern_wait6+0x5de/frame 0xfffffe045ca547a0
    sys_wait4() at sys_wait4+0x72/frame 0xfffffe045ca54990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045ca54ab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045ca54ab0
    --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x8008f2d7a, rsp = 0x7fffffffebf8, rbp = 0x7fffffffec40 ---
    
    Tracing command minicron pid 78297 tid 100201 td 0xfffff801eede9000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045cadb780
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045cadb7c0
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045cadb840
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe045cadb870
    _sleep() at _sleep+0x238/frame 0xfffffe045cadb8f0
    kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe045cadb950
    sys_nanosleep() at sys_nanosleep+0x51/frame 0xfffffe045cadb990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045cadbab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045cadbab0
    --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x8008f2f3a, rsp = 0x7fffffffebb8, rbp = 0x7fffffffec10 ---
    
    Tracing command minicron pid 77792 tid 100080 td 0xfffff80011016000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045c8f4560
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045c8f45a0
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045c8f4620
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe045c8f4650
    _sleep() at _sleep+0x27d/frame 0xfffffe045c8f46d0
    kern_wait6() at kern_wait6+0x5de/frame 0xfffffe045c8f47a0
    sys_wait4() at sys_wait4+0x72/frame 0xfffffe045c8f4990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045c8f4ab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045c8f4ab0
    --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x8008f2d7a, rsp = 0x7fffffffec18, rbp = 0x7fffffffec60 ---
    
    Tracing command ftp-proxy pid 77776 tid 100197 td 0xfffff80171cb2000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045cac7530
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045cac7570
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045cac75f0
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe045cac7620
    _sleep() at _sleep+0x238/frame 0xfffffe045cac76a0
    kern_kevent_fp() at kern_kevent_fp+0x399/frame 0xfffffe045cac7880
    kern_kevent() at kern_kevent+0x9f/frame 0xfffffe045cac78e0
    sys_kevent() at sys_kevent+0x12a/frame 0xfffffe045cac7990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045cac7ab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045cac7ab0
    --- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x8009088fa, rsp = 0x7fffffffe8e8, rbp = 0x7fffffffe930 ---
    
    Tracing command pinger pid 71989 tid 100196 td 0xfffff80011f5e4b0
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045cac25c0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045cac2600
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045cac2680
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe045cac26b0
    _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x19e/frame 0xfffffe045cac2710
    seltdwait() at seltdwait+0xa4/frame 0xfffffe045cac2750
    kern_select() at kern_select+0x8fa/frame 0xfffffe045cac2950
    sys_select() at sys_select+0x54/frame 0xfffffe045cac2990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045cac2ab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045cac2ab0
    --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x8019c966a, rsp = 0x7fffffffdbe8, rbp = 0x7fffffffdc20 ---
    
    Tracing command squidGuard pid 71694 tid 100103 td 0xfffff80011606960
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045c967640
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045c967680
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045c967700
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe045c967730
    _sleep() at _sleep+0x27d/frame 0xfffffe045c9677b0
    sbwait() at sbwait+0x4f/frame 0xfffffe045c9677d0
    soreceive_generic() at soreceive_generic+0x26c/frame 0xfffffe045c9678a0
    dofileread() at dofileread+0x95/frame 0xfffffe045c9678f0
    kern_readv() at kern_readv+0x68/frame 0xfffffe045c967940
    sys_read() at sys_read+0x63/frame 0xfffffe045c967990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045c967ab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045c967ab0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x80116b68a, rsp = 0x7ffffffdab38, rbp = 0x7ffffffdab50 ---
    
    Tracing command squidGuard pid 71422 tid 100107 td 0xfffff8001136c4b0
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045c97b640
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045c97b680
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045c97b700
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe045c97b730
    _sleep() at _sleep+0x27d/frame 0xfffffe045c97b7b0
    sbwait() at sbwait+0x4f/frame 0xfffffe045c97b7d0
    soreceive_generic() at soreceive_generic+0x26c/frame 0xfffffe045c97b8a0
    dofileread() at dofileread+0x95/frame 0xfffffe045c97b8f0
    kern_readv() at kern_readv+0x68/frame 0xfffffe045c97b940
    sys_read() at sys_read+0x63/frame 0xfffffe045c97b990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045c97bab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045c97bab0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x80116b68a, rsp = 0x7ffffffdab38, rbp = 0x7ffffffdab50 ---
    
    Tracing command squidGuard pid 71205 tid 100108 td 0xfffff8001136c000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045c980640
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045c980680
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045c980700
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe045c980730
    _sleep() at _sleep+0x27d/frame 0xfffffe045c9807b0
    sbwait() at sbwait+0x4f/frame 0xfffffe045c9807d0
    soreceive_generic() at soreceive_generic+0x26c/frame 0xfffffe045c9808a0
    dofileread() at dofileread+0x95/frame 0xfffffe045c9808f0
    kern_readv() at kern_readv+0x68/frame 0xfffffe045c980940
    sys_read() at sys_read+0x63/frame 0xfffffe045c980990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045c980ab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045c980ab0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x80116b68a, rsp = 0x7ffffffdab38, rbp = 0x7ffffffdab50 ---
    
    Tracing command squidGuard pid 71062 tid 100097 td 0xfffff800111c1000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045c949640
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045c949680
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045c949700
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe045c949730
    _sleep() at _sleep+0x27d/frame 0xfffffe045c9497b0
    sbwait() at sbwait+0x4f/frame 0xfffffe045c9497d0
    soreceive_generic() at soreceive_generic+0x26c/frame 0xfffffe045c9498a0
    dofileread() at dofileread+0x95/frame 0xfffffe045c9498f0
    kern_readv() at kern_readv+0x68/frame 0xfffffe045c949940
    sys_read() at sys_read+0x63/frame 0xfffffe045c949990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045c949ab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045c949ab0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x80116b68a, rsp = 0x7ffffffdab38, rbp = 0x7ffffffdab50 ---
    
    Tracing command squidGuard pid 70736 tid 100189 td 0xfffff801ee3f7000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045ca9f640
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045ca9f680
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045ca9f700
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe045ca9f730
    _sleep() at _sleep+0x27d/frame 0xfffffe045ca9f7b0
    sbwait() at sbwait+0x4f/frame 0xfffffe045ca9f7d0
    soreceive_generic() at soreceive_generic+0x26c/frame 0xfffffe045ca9f8a0
    dofileread() at dofileread+0x95/frame 0xfffffe045ca9f8f0
    kern_readv() at kern_readv+0x68/frame 0xfffffe045ca9f940
    sys_read() at sys_read+0x63/frame 0xfffffe045ca9f990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045ca9fab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045ca9fab0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x80116b68a, rsp = 0x7ffffffdab38, rbp = 0x7ffffffdab50 ---
    
    Tracing command squidGuard pid 70450 tid 100114 td 0xfffff800112b0960
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045c99e640
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045c99e680
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045c99e700
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe045c99e730
    _sleep() at _sleep+0x27d/frame 0xfffffe045c99e7b0
    sbwait() at sbwait+0x4f/frame 0xfffffe045c99e7d0
    soreceive_generic() at soreceive_generic+0x26c/frame 0xfffffe045c99e8a0
    dofileread() at dofileread+0x95/frame 0xfffffe045c99e8f0
    kern_readv() at kern_readv+0x68/frame 0xfffffe045c99e940
    sys_read() at sys_read+0x63/frame 0xfffffe045c99e990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045c99eab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045c99eab0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x80116b68a, rsp = 0x7ffffffdab38, rbp = 0x7ffffffdab50 ---
    
    Tracing command squidGuard pid 70158 tid 100190 td 0xfffff801ee3f6960
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045caa4640
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045caa4680
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045caa4700
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe045caa4730
    _sleep() at _sleep+0x27d/frame 0xfffffe045caa47b0
    sbwait() at sbwait+0x4f/frame 0xfffffe045caa47d0
    soreceive_generic() at soreceive_generic+0x26c/frame 0xfffffe045caa48a0
    dofileread() at dofileread+0x95/frame 0xfffffe045caa48f0
    kern_readv() at kern_readv+0x68/frame 0xfffffe045caa4940
    sys_read() at sys_read+0x63/frame 0xfffffe045caa4990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045caa4ab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045caa4ab0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x80116b68a, rsp = 0x7ffffffdab38, rbp = 0x7ffffffdab50 ---
    
    Tracing command squidGuard pid 70050 tid 100178 td 0xfffff80011d36960
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045ca68640
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045ca68680
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045ca68700
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe045ca68730
    _sleep() at _sleep+0x27d/frame 0xfffffe045ca687b0
    sbwait() at sbwait+0x4f/frame 0xfffffe045ca687d0
    soreceive_generic() at soreceive_generic+0x26c/frame 0xfffffe045ca688a0
    dofileread() at dofileread+0x95/frame 0xfffffe045ca688f0
    kern_readv() at kern_readv+0x68/frame 0xfffffe045ca68940
    sys_read() at sys_read+0x63/frame 0xfffffe045ca68990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045ca68ab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045ca68ab0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x80116b68a, rsp = 0x7ffffffdab38, rbp = 0x7ffffffdab50 ---
    
    Tracing command sh pid 69113 tid 100183 td 0xfffff80011d44000
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045ca81560
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045ca815a0
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045ca81620
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe045ca81650
    _sleep() at _sleep+0x27d/frame 0xfffffe045ca816d0
    kern_wait6() at kern_wait6+0x5de/frame 0xfffffe045ca817a0
    sys_wait4() at sys_wait4+0x72/frame 0xfffffe045ca81990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045ca81ab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045ca81ab0
    --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d8bd7a, rsp = 0x7fffffffe888, rbp = 0x7fffffffe930 ---
    
    Tracing command pinger pid 50158 tid 100171 td 0xfffff800116044b0
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045ca455c0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045ca45600
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045ca45680
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe045ca456b0
    _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x19e/frame 0xfffffe045ca45710
    seltdwait() at seltdwait+0xa4/frame 0xfffffe045ca45750
    kern_select() at kern_select+0x8fa/frame 0xfffffe045ca45950
    sys_select() at sys_select+0x54/frame 0xfffffe045ca45990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045ca45ab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045ca45ab0
    --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x8019c966a, rsp = 0x7fffffffdbe8, rbp = 0x7fffffffdc20 ---
    
    Tracing command unlinkd pid 50100 tid 100191 td 0xfffff801ee3f64b0
    sched_switch() at sched_switch+0x2b3/frame 0xfffffe045caa96d0
    mi_switch() at mi_switch+0xe1/frame 0xfffffe045caa9710
    sleepq_catch_signals() at sleepq_catch_signals+0xab/frame 0xfffffe045caa9790
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe045caa97c0
    _sleep() at _sleep+0x27d/frame 0xfffffe045caa9840
    pipe_read() at pipe_read+0x2aa/frame 0xfffffe045caa98a0
    dofileread() at dofileread+0x95/frame 0xfffffe045caa98f0
    kern_readv() at kern_readv+0x68/frame 0xfffffe045caa9940
    sys_read() at sys_read+0x63/frame 0xfffffe045caa9990
    amd64_syscall() at amd64_syscall+0x40f/frame 0xfffffe045caa9ab0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe045caa9ab0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x8017ab68a, rsp = 0x7fffffffe7f8, rbp = 0x7fffffffe810 ---
    
    Tracing command squid pid 45041 tid 100182 td 0xfffff80171cb24b0
    kdb_enter() at kdb_enter+0x3e/frame 0xfffffe045ca7c700
    vpanic() at vpanic+0x146/frame 0xfffffe045ca7c740
    panic() at panic+0x43/frame 0xfffffe045ca7c7a0
    sbcut_internal() at sbcut_internal+0x273/frame 0xfffffe045ca7c7b0
    sbflush_internal() at sbflush_internal+0x18/frame 0xfffffe045ca7c7d0
    sbflush() at sbflush+0x46/frame 0xfffffe045ca7c800
    tcp_disconnect() at tcp_disconnect+0x52/frame 0xfffffe045ca7c830
    tcp_usr_disconnect() at tcp_usr_disconnect+0x84/frame 0xfffffe045ca7c860
    soclose() at soclose+0x3c/frame 0xfffffe045ca7c8a0
    _fdrop() at _fdrop+0x29/f[/thread]
    

  • Rebel Alliance Developer Netgate

    Looks almost – but not quite -- like a bug that was fixed a while back.

    Do you have any errors in your system log from the NICs? Or maybe anything you've customized in /boot/loader.conf or /boot/loader.conf.local or under System > Advanced on the Tunables tab?



  • Sorry to bump this thread but I have the same symptoms (v2.3.2 crashing everyday, multiple times): https://forum.pfsense.org/index.php?topic=120513