Crash when enabling traffic shaper on more than 1 port



  • I upgraded my firewall to pfSense 2.4-RC.

    Everything works smoothly except for the traffic shaper.
    When enabling the shaper on a single port it works flawlessly.
    As soon as I enable a shaper on a 2nd port pfSense will crash with a message regarding the wrr scheduler.
    On 2.3.4 it works without problems

    Crash report begins.  Anonymous machine information:

    amd64
    11.0-RELEASE-p12
    FreeBSD 11.0-RELEASE-p12 #44 eed6fd412cc(RELENG_2_4_0): Fri Sep 15 16:11:43 CDT 2017    root@buildbot2.netgate.com:/builder/ce-240/tmp/obj/builder/ce-240/tmp/FreeBSD-src/sys/pfSense

    Crash report details:

    No PHP errors found.

    Filename: /var/crash/bounds
    1

    Filename: /var/crash/info.0
    Dump header from device: /dev/label/swap0
      Architecture: amd64
      Architecture Version: 1
      Dump Length: 157184
      Blocksize: 512
      Dumptime: Sun Sep 17 13:00:43 2017
      Hostname: firewall
      Magic: FreeBSD Text Dump
      Version String: FreeBSD 11.0-RELEASE-p12 #44 eed6fd412cc(RELENG_2_4_0): Fri Sep 15 16:11:43 CDT 2017
        root@buildbot2.netgate.com:/builder/ce-240/tmp/obj/builder/ce-240/tmp/FreeBSD-src/sys/pfSense
      Panic String: _rmc_wrr_dequeue_next
      Dump Parity: 3076715824
      Bounds: 0
      Dump Status: good

    Filename: /var/crash/info.last
    Dump header from device: /dev/label/swap0
      Architecture: amd64
      Architecture Version: 1
      Dump Length: 157184
      Blocksize: 512
      Dumptime: Sun Sep 17 13:00:43 2017
      Hostname: firewall
      Magic: FreeBSD Text Dump
      Version String: FreeBSD 11.0-RELEASE-p12 #44 eed6fd412cc(RELENG_2_4_0): Fri Sep 15 16:11:43 CDT 2017
        root@buildbot2.netgate.com:/builder/ce-240/tmp/obj/builder/ce-240/tmp/FreeBSD-src/sys/pfSense
      Panic String: _rmc_wrr_dequeue_next
      Dump Parity: 3076715824
      Bounds: 0
      Dump Status: good



  • What hardware are you using?



  • Two different devices.

    1: Lanner FW7541D, has Intel em interfaces.
    2: Qotom i5, has intel igb interfaces.

    Both show the same issue.



  • traffic shaper crashing happens to me too on sg-1000 as well as apu2 but only when i saturate the line above 50mbps

    https://forum.pfsense.org/index.php?topic=135450.0



  • I read that thread as well when looking if it was a known issue.
    But I’m quite possitive the origin lies with the shaper.

    I run several days now with the shaper enabled on the wan link.
    As soon as I enable it on a 2nd interface, no matter which one, (lan, dmz, lab), pfSense crashes.


  • Rebel Alliance Developer Netgate

    Please post the entire crash dump, the summary is not enough information to go on. At least the backtrace and last section of the message buffer are required, but the more info we have, the better.

    Can you post the details of your shaper configuration? Preferably also including the shaper queue/rules from /tmp/rules.debug.

    Does this only happen with HFSC or does it also happen with PRIQ?

    On the system with igb NICs, try setting hw.igb.num_queues=1 in /boot/loader.conf.local and then reboot, see if it stabilizes.



  • Current configuration is 3 shapers, WAN, LAN and DMZ. (See attached picture)

    In general, the shapers are similar to the configurations from the wizard.

    WAN - HFSC, 160Mbps
        qInternet      - Prio 3, 160 Mbps
              - qAck    - prio 6, 25% borrow
              - qVoip    - prio 6, 256kbps
              - qGold    - prio 5, 15%, RED, ECN, borrow
              - qSilver  - prio 4, 44%, RED, borrow, default queue
            - qBronze  - prio 3, 15%, RED, borrow

    Lan and DMZ have similar configuration except for the VoIP queue

    LAN - HFSC, 1Gbps
        qSilver          - prio 4, 60%, RED, borrow, default queue
        qOther          - prio 4, 40%
              - qAck      - prio 6, 50%, borrow
              - qVoip    - prio 6, 256kbps
              - qGold    - prio 5, 25%, RED, ECN, borrow
              - qBronze  - prio 3, 24%, RED, borrow

    The first thing I did was reading this post FreeBSD Network Performance Tuning @ Calomel.org and changed hw.igb.num_queues=1 in /boot/loader.conf.local
    It now looks like this:

    Accept intel license

    legal.intel_iwi.license_ack=1

    kern.cam.boot_delay=10000
    kern.ipc.nmbclusters="1000000"

    ipmi_load="YES"
    hw.igb.enable_msix=1

    hw.igb.buf_ring_size="32768"
    hw.igb.num_queues="1"
    hw.igb.rx_process_limit="4096"
    hw.igb.tx_process_limit="4096"
    hw.igb.rxd="4096"
    hw.igb.txd="4096"
    net.link.ifqmaxlen="40960"
    net.pf.states_hashsize="16777216"

    Disable flow control

    dev.igb.0.fc=0
    dev.igb.1.fc=0
    dev.igb.2.fc=0
    dev.igb.3.fc=0

    That did not solve the problen
    After some crashes I changed the configuration from HFSC to CBQ but that didn't help either.

    Unfortunately I didn't save the crashdump so I cannot post one now. But I uploaded two.
    They can be identified by the name "firewall.familievink.nl"

    If required I will enable the LAN shaper as well to create a crash dump, but that requires some planning.

    ![Schermafbeelding 2017-09-19 om 18.19.27.png](/public/imported_attachments/1/Schermafbeelding 2017-09-19 om 18.19.27.png)
    ![Schermafbeelding 2017-09-19 om 18.19.27.png_thumb](/public/imported_attachments/1/Schermafbeelding 2017-09-19 om 18.19.27.png_thumb)


  • Rebel Alliance Developer Netgate

    Can you remove all of the extra tuning and just use num_queues?

    And can you try PRIQ to see if it also crashes?



  • First removed all extra configurations except for bandwidth and prio. Also no borrow from other queues.

    Enabled, both Wan and Lan
    The good thing is that the firewall is not crashing immediately.
    Let it run for 10 minutes, no problems so far, which is better than what I had.

    Then hhanged the WAN and LAN queues to PRIQ

    qAck, prio 7
    qVoip, prio 6
    qGold, prio 5
    qSilver, prio 4, default
    qBronze, prio 3

    This runs fine as well.

    So, next thing is to find out where it goes wrong. My first guess will be the borrow option.



  • Unfortunately, I was too optimistic.
    The CBQ configuration crashed after the traffic increased a bit. Not even that much. at 12Mbps… poooff

    The crashdump was uploaded for inspection, here are the first 350 lines

    Crash report begins.  Anonymous machine information:

    amd64
    11.0-RELEASE-p12
    FreeBSD 11.0-RELEASE-p12 #44 eed6fd412cc(RELENG_2_4_0): Fri Sep 15 16:11:43 CDT 2017    root@buildbot2.netgate.com:/builder/ce-240/tmp/obj/builder/ce-240/tmp/FreeBSD-src/sys/pfSense

    Crash report details:

    No PHP errors found.

    Filename: /var/crash/bounds
    2

    Filename: /var/crash/info.0
    Dump header from device: /dev/label/swap0
      Architecture: amd64
      Architecture Version: 1
      Dump Length: 87552
      Blocksize: 512
      Dumptime: Tue Sep 19 19:59:49 2017
      Hostname: firewall.familievink.nl
      Magic: FreeBSD Text Dump
      Version String: FreeBSD 11.0-RELEASE-p12 #44 eed6fd412cc(RELENG_2_4_0): Fri Sep 15 16:11:43 CDT 2017
        root@buildbot2.netgate.com:/builder/ce-240/tmp/obj/builder/ce-240/tmp/FreeBSD-src/sys/pfSense
      Panic String: _rmc_wrr_dequeue_next
      Dump Parity: 2036107568
      Bounds: 0
      Dump Status: good

    Filename: /var/crash/info.1
    Dump header from device: /dev/label/swap0
      Architecture: amd64
      Architecture Version: 1
      Dump Length: 101376
      Blocksize: 512
      Dumptime: Tue Sep 19 20:03:10 2017
      Hostname: firewall.familievink.nl
      Magic: FreeBSD Text Dump
      Version String: FreeBSD 11.0-RELEASE-p12 #44 eed6fd412cc(RELENG_2_4_0): Fri Sep 15 16:11:43 CDT 2017
        root@buildbot2.netgate.com:/builder/ce-240/tmp/obj/builder/ce-240/tmp/FreeBSD-src/sys/pfSense
      Panic String: _rmc_wrr_dequeue_next
      Dump Parity: 2995161392
      Bounds: 1
      Dump Status: good

    Filename: /var/crash/info.last
    Dump header from device: /dev/label/swap0
      Architecture: amd64
      Architecture Version: 1
      Dump Length: 101376
      Blocksize: 512
      Dumptime: Tue Sep 19 20:03:10 2017
      Hostname: firewall.familievink.nl
      Magic: FreeBSD Text Dump
      Version String: FreeBSD 11.0-RELEASE-p12 #44 eed6fd412cc(RELENG_2_4_0): Fri Sep 15 16:11:43 CDT 2017
        root@buildbot2.netgate.com:/builder/ce-240/tmp/obj/builder/ce-240/tmp/FreeBSD-src/sys/pfSense
      Panic String: _rmc_wrr_dequeue_next
      Dump Parity: 2995161392
      Bounds: 1
      Dump Status: good

    Filename: /var/crash/textdump.tar.0
    ddb.txt���������������������������������������������������������������������������������������������0600����0�������0�������140000������13160255425�  7075� �����������������������������������������������������������������������������������������������������ustar���root����������������������������wheel������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������db:0:kdb.enter.default>  run lockinfo
    db:1:lockinfo> show locks
    No such command
    db:1:locks>  show alllocks
    No such command
    db:1:alllocks>  show lockedvnods
    Locked vnodes
    db:0:kdb.enter.default>  show pcpu
    cpuid        = 2
    dynamic pcpu = 0xfffffe02a6ca5200
    curthread    = 0xfffff80006535a00: pid 12 "irq268: igb1:que 0"
    curpcb      = 0xfffffe02293deb80
    fpcurthread  = none
    idlethread  = 0xfffff80006219500: tid 100005 "idle: cpu2"
    curpmap      = 0xffffffff829e6300
    tssp        = 0xffffffff82a1ebe0
    commontssp  = 0xffffffff82a1ebe0
    rsp0        = 0xfffffe02293deb80
    gs32p        = 0xffffffff82a25438
    ldt          = 0xffffffff82a25478
    tss          = 0xffffffff82a25468
    db:0:kdb.enter.default>  bt
    Tracing pid 12 tid 100043 td 0xfffff80006535a00
    kdb_enter() at kdb_enter+0x3b/frame 0xfffffe02293ddfe0
    vpanic() at vpanic+0x19f/frame 0xfffffe02293de060
    panic() at panic+0x43/frame 0xfffffe02293de0c0
    rmc_dequeue_next() at rmc_dequeue_next+0x4b2/frame 0xfffffe02293de110
    cbq_dequeue() at cbq_dequeue+0x26/frame 0xfffffe02293de150
    tbr_dequeue() at tbr_dequeue+0xdf/frame 0xfffffe02293de1a0
    vlan_start() at vlan_start+0x12c/frame 0xfffffe02293de200
    if_transmit() at if_transmit+0x1a5/frame 0xfffffe02293de250
    ether_output() at ether_output+0x728/frame 0xfffffe02293de2e0
    ip_output() at ip_output+0x1746/frame 0xfffffe02293de430
    phyint_send() at phyint_send+0xb8/frame 0xfffffe02293de4a0
    ip_mdq() at ip_mdq+0x306/frame 0xfffffe02293de510
    X_ip_mforward() at X_ip_mforward+0x356/frame 0xfffffe02293de590
    ip_input() at ip_input+0x6b0/frame 0xfffffe02293de5f0
    netisr_dispatch_src() at netisr_dispatch_src+0xa5/frame 0xfffffe02293de650
    ether_demux() at ether_demux+0x15c/frame 0xfffffe02293de680
    ether_nh_input() at ether_nh_input+0x317/frame 0xfffffe02293de6e0
    netisr_dispatch_src() at netisr_dispatch_src+0xa5/frame 0xfffffe02293de740
    ether_input() at ether_input+0x26/frame 0xfffffe02293de760
    vlan_input() at vlan_input+0x1cb/frame 0xfffffe02293de7e0
    ether_demux() at ether_demux+0xa9/frame 0xfffffe02293de810
    ether_nh_input() at ether_nh_input+0x317/frame 0xfffffe02293de870
    netisr_dispatch_src() at netisr_dispatch_src+0xa5/frame 0xfffffe02293de8d0
    ether_input() at ether_input+0x26/frame 0xfffffe02293de8f0
    igb_rxeof() at igb_rxeof+0x81c/frame 0xfffffe02293de990
    igb_msix_que() at igb_msix_que+0x152/frame 0xfffffe02293de9e0
    intr_event_execute_handlers() at intr_event_execute_handlers+0x20f/frame 0xfffffe02293dea20
    ithread_loop() at ithread_loop+0xc6/frame 0xfffffe02293dea70
    fork_exit() at fork_exit+0x85/frame 0xfffffe02293deab0
    fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe02293deab0
    –- trap 0, rip = 0, rsp = 0, rbp = 0 ---
    db:0:kdb.enter.default>  ps
      pid  ppid  pgrp  uid  state  wmesg        wchan        cmd
    33038  298  298    0  S      accept  0xfffff8009007a06c php-fpm
    29694 35607  298    0  S      nanslp  0xffffffff82866b33 sleep
    6612    1  6612    0  Ss      accept  0xfffff801b2ff506c qstats
    35607    1  298    0  S      wait    0xfffff801b4018528 sh
    93608 92629 93608    0  S+      ttyin    0xfffff800069e60a8 tcsh
    92629 91887 92629    0  Ss+    wait    0xfffff801b4b18000 sh
    92350 62701 92350    0  Ss      (threaded)                  sshlockout_pf
    100245                  S      piperd  0xfffff8018dbb58e8 sshlockout_pf
    100757                  S      nanslp  0xffffffff82866b31 sshlockout_pf
    91887  7656 91887    0  Ss      select  0xfffff8015f7c1140 sshd
    22994    1  298    0  S      nanslp  0xffffffff82866b33 php
    16380    1 16380    0  Ss      (threaded)                  radiusd
    100299                  S      kqread  0xfffff801b9520600 radiusd
    100447                  S      usem    0xfffff801fbf0b380 radiusd
    100448                  S      usem    0xfffff80009dc5c80 radiusd
    100449                  S      usem    0xfffff801fbf0b680 radiusd
    100450                  S      usem    0xfffff8015f1a9600 radiusd
    100451                  S      usem    0xfffff80061904000 radiusd
    94578    1 94578    0  Ss      (threaded)                  snort
    101223                  S      bpf      0xfffff801a75b7000 snort
    100462                  S      nanslp  0xffffffff82866b32 snort
    100463                  S      sbwait  0xfffff8012bb5a144 snort
    23943    1 23943    0  Ss      (threaded)                  dpinger
    100197                  S      uwait    0xfffff8012bce0900 dpinger
    101214                  S      sbwait  0xfffff801bc3d64a4 dpinger
    101215                  S      nanslp  0xffffffff82866b31 dpinger
    101216                  S      nanslp  0xffffffff82866b33 dpinger
    101221                  S      accept  0xfffff800cb7bda8c dpinger
    23331    1 23331    0  Ss      (threaded)                  dpinger
    100321                  S      uwait    0xfffff80090385200 dpinger
    101181                  S      sbwait  0xfffff8012be6c4a4 dpinger
    101211                  S      nanslp  0xffffffff82866b31 dpinger
    101212                  S      nanslp  0xffffffff82866b32 dpinger
    101213                  S      accept  0xfffff8015faf63cc dpinger
    23238    1 23238    0  Ss      (threaded)                  dpinger
    100466                  S      uwait    0xfffff80009c1c280 dpinger
    100347                  S      sbwait  0xfffff80090392b64 dpinger
    100348                  S      nanslp  0xffffffff82866b32 dpinger
    100349                  S      nanslp  0xffffffff82866b32 dpinger
    101180                  S      accept  0xfffff8015f1053cc dpinger
    23049    1 23049    0  Ss      (threaded)                  dpinger
    100130                  S      uwait    0xfffff80090387c00 dpinger
    100334                  S      sbwait  0xfffff801b2fedb64 dpinger
    100335                  S      nanslp  0xffffffff82866b33 dpinger
    100336                  S      nanslp  0xffffffff82866b33 dpinger
    100337                  S      accept  0xfffff800c9bae3cc dpinger
    18455    1 11660    0  S      select  0xfffff8015fa81140 dhcp6c
    10995 96200 96200    0  S      piperd  0xfffff8006b1ce8e8 cron
    90119    1 90119    0  Ss      kqread  0xfffff800bc831200 dhcpleases6
    89812    1 89812  1002  Ss      select  0xfffff8015f588140 dhcpd
    88853    1 88853  1002  Ss      select  0xfffff8015f1a3b40 dhcpd
    79029    1 79029    59  Ss      (threaded)                  unbound
    100319                  S      kqread  0xfffff801811ce600 unbound
    100430                  S      kqread  0xfffff8019614d500 unbound
    100434                  S      kqread  0xfffff801a142ea00 unbound
    100455                  S      kqread  0xfffff801cd747800 unbound
    86160    1  298    0  S      select  0xfffff801fa40dcc0 igmpproxy
    59524    1 59524    0  Ss      (threaded)                  filterdns
    100324                  S      uwait    0xfffff8015f1a4180 chats.vpn.mrv.com
    100325                  S      uwait    0xfffff8015f588000 bos.vpn.mrv.com
    100326                  S      uwait    0xfffff800900f8d80 194.90.138.234
    100327                  S      uwait    0xfffff80090138100 olpair.com
    100328                  S      uwait    0xfffff8015fa76880 plex.tv
    100329                  S      uwait    0xfffff8015fa76c80 plex.tv
    100330                  S      uwait    0xfffff8015fa76e00 www.plex.tv
    100331                  S      uwait    0xfffff8015fa76d00 SIP.xs4all.nl
    100332                  S      uwait    0xfffff8012bbac200 signal-thread
    1275  894  1275    0  S+      ttyin    0xfffff80009b334a8 sh
    1067  695  1067    0  S+      ttyin    0xfffff800069e24a8 sh
      894 99064  894    0  SW+    wait    0xfffff80009c99528 sh
      695  202  695    0  SW+    wait    0xfffff801b4111000 sh
      202    1  202    0  SWs+    wait    0xfffff80009deca50 login
      154    1  154    0  Ss+    ttyin    0xfffff80009b8a0a8 getty
      153    1  153    0  Ss+    ttyin    0xfffff80009b8a4a8 getty
    99798    1 99798    0  Ss+    ttyin    0xfffff80009b8a8a8 getty
    99556    1 99556    0  Ss+    ttyin    0xfffff800097c94a8 getty
    99553    1 99553    0  Ss+    ttyin    0xfffff800097c98a8 getty
    99312    1 99312    0  Ss+    ttyin    0xfffff800097c9ca8 getty
    99239    1 99239    0  Ss+    ttyin    0xfffff80009b310a8 getty
    99064    1 99064    0  SWs+    wait    0xfffff80009cb1a50 login
    77672    1 77531    0  S      kqread  0xfffff8015fabdc00 lighttpd_pfb
    77073 76871 76871    0  S      nanslp  0xffffffff82866b33 minicron
    76871    1 76871    0  SWs    wait    0xfffff801b4110000 minicron
    76832 76765 76765    0  S      nanslp  0xffffffff82866b30 minicron
    76765    1 76765    0  SWs    wait    0xfffff80009c99a50 minicron
    76121 76090 76090    0  S      nanslp  0xffffffff82866b31 minicron
    76090    1 76090    0  SWs    wait    0xfffff80009cb0000 minicron
    62701    1 62701    0  Ss      select  0xfffff8015f9a3440 syslogd
    20127    1 20127    0  Ss      select  0xfffff8015fa777c0 powerd
    8847  8587  8847    0  Ss      (threaded)                  charon
    100133                  S      sigwait  0xfffff8015fa5b000 charon
    100218                  S      select  0xfffff8015fd2b740 charon
    100219                  S      uwait    0xfffff80009c1c480 charon
    100220                  S      select  0xfffff8015fa627c0 charon
    100221                  S      uwait    0xfffff80009c13680 charon
    100222                  S      uwait    0xfffff80009c1c000 charon
    100223                  S      uwait    0xfffff80009c13580 charon
    100224                  S      uwait    0xfffff801178a5580 charon
    100225                  S      uwait    0xfffff8015fd2b800 charon
    100226                  S      uwait    0xfffff8015fd2b900 charon
    100227                  S      uwait    0xfffff8015fd2ba00 charon
    100228                  S      uwait    0xfffff8015fd2bb00 charon
    100229                  S      uwait    0xfffff8015fd2bc00 charon
    100230                  S      uwait    0xfffff8015fd2bd00 charon
    100231                  S      uwait    0xfffff8015fd2be00 charon
    100232                  S      uwait    0xfffff8015fd2c380 charon
    100233                  S      uwait    0xfffff8015fd2c180 charon
    8587    1  8587    0  Ss      select  0xfffff8015fa628c0 starter
    4647    1  4545    0  S      select  0xfffff8015ef423c0 radvd
    96807    1 96807    0  Ss      (threaded)                  ntpd
    100114                  S      select  0xfffff8015f1a55c0 ntpd
    96200    1 96200    0  Ss      nanslp  0xffffffff82866b30 cron
    95969 95454 95454    0  S      kqread  0xfffff8015f63cd00 nginx
    95711 95454 95454    0  S      kqread  0xfffff8015ed92200 nginx
    95454    1 95454    0  SWs    pause    0xfffff80009ec95d0 nginx
    64781    1 64781    0  Ss      select  0xfffff80149e54240 openvpn
    61617    1 61617    0  Ss      select  0xfffff80009c12540 openvpn
    60126    1 60126    0  Ss      select  0xfffff80149e55a40 openvpn
    57281    1 57281    0  Ss      select  0xfffff80009bd1540 openvpn
    55041    1 55041    0  Ss      select  0xfffff80009e6eb40 openvpn
    46288    1 46288    65  Ss      select  0xfffff8012bc58f40 dhclient
    42347    1 42347    0  Ss      select  0xfffff80090385740 dhclient
    22284    1 22284    0  Ss      bpf      0xfffff8012baf0400 filterlog
    16036    1 16036    0  Ss      (threaded)                  mpd5
    100135                  S      select  0xfffff80009c1ce40 mpd5
    7930    1  7930    0  Ss      (threaded)                  sshlockout_pf
    100127                  S      uwait    0xfffff80009c1ed80 sshlockout_pf
    100131                  S      nanslp  0xffffffff82866b32 sshlockout_pf
    7656    1  7656    0  Ss      select  0xfffff80009c1e5c0 sshd
      352    1  352    0  Ss      select  0xfffff80009e6d740 devd
      338  336  336    0  SW      kqread  0xfffff80009c7e700 check_reload_status
      336    1  336    0  Ss      kqread  0xfffff80009b7e500 check_reload_status
      298    1  298    0  Ss      kqread  0xfffff80009ba8300 php-fpm
      58    0    0    0  DL      mdwait  0xfffff80009bd9800 [md0]
      26    0    0    0  DL      vlruwt  0xfffff80009b44528 [vnlru]
      25    0    0    0  DL      syncer  0xffffffff82960680 [syncer]
      24    0    0    0  DL      -        0xffffffff8295fbb4 [bufspacedaemon]
      23    0    0    0  DL      (threaded)                  [bufdaemon]
    100081                  D      psleep  0xffffffff8295ef04 [bufdaemon]
    100100                  D      sdflush  0xfffff80009bd6ce8 [/ worker]
      22    0    0    0  DL      pgzero  0xffffffff829742e4 [pagezero]
      21    0    0    0  DL      psleep  0xffffffff8297078c [vmdaemon]
      20    0    0    0  DL      (threaded)                  [pagedaemon]
    100078                  D      psleep  0xffffffff82a1df05 [pagedaemon]
    100085                  D      umarcl  0xffffffff829700b8 [uma]
      19    0    0    0  DL      -        0xffffffff8295e714 [soaiod4]
      18    0    0    0  DL      -        0xffffffff8295e714 [soaiod3]
        9    0    0    0  DL      -        0xffffffff8295e714 [soaiod2]
        8    0    0    0  DL      -        0xffffffff8295e714 [soaiod1]
        7    0    0    0  DL      -        0xffffffff8273b2f0 [rand_harvestq]
      17    0    0    0  DL      cooling  0xfffff800068b9d58 [acpi_cooling1]
      16    0    0    0  DL      tzpoll  0xffffffff82621f38 [acpi_thermal]
        6    0    0    0  DL      pftm    0xffffffff80f4c4f0 [pf purge]
        5    0    0    0  DL      waiting_ 0xffffffff82a0d540 [sctp_iterator]
      15    0    0    0  DL      (threaded)                  [usb]
    100034                  D      -        0xfffffe0001121460 [usbus0]
    100035                  D      -        0xfffffe00011214b8 [usbus0]
    100036                  D      -        0xfffffe0001121510 [usbus0]
    100037                  D      -        0xfffffe0001121568 [usbus0]
    100038                  D      -        0xfffffe00011215c0 [usbus0]
    100056                  D      -        0xfffffe0000f85d10 [usbus1]
    100057                  D      -        0xfffffe0000f85d68 [usbus1]
    100058                  D      -        0xfffffe0000f85dc0 [usbus1]
    100059                  D      -        0xfffffe0000f85e18 [usbus1]
    100060                  D      -        0xfffffe0000f85e70 [usbus1]
        4    0    0    0  DL      (threaded)                  [cam]
    100029                  D      -        0xffffffff8260e180 [doneq0]
    100073                  D      -        0xffffffff8260dfc8 [scanner]
        3    0    0    0  DL      crypto_r 0xffffffff8296ec70 [crypto returns]
        2    0    0    0  DL      crypto_w 0xffffffff8296eb18 [crypto]
      14    0    0    0  DL      (threaded)                  [geom]
    100023                  D      -        0xffffffff829e5740 [g_event]
    100024                  D      -        0xffffffff829e5748 [g_up]
    100025                  D      -        0xffffffff829e5750 [g_down]
      13    0    0    0  DL      (threaded)                  [ng_queue]
    100019                  D      sleep    0xffffffff825cbfd0 [ng_queue0]
    100020                  D      sleep    0xffffffff825cbfd0 [ng_queue1]
    100021                  D      sleep    0xffffffff825cbfd0 [ng_queue2]
    100022                  D      sleep    0xffffffff825cbfd0 [ng_queue3]
      12    0    0    0  RL      (threaded)                  [intr]
    100007                  I                                  [swi1: netisr 0]
    100008                  I                                  [swi3: vm]
    100009                  I                                  [swi4: clock (0)]
    100010                  I                                  [swi4: clock (1)]
    100011                  I                                  [swi4: clock (2)]
    100012                  I                                  [swi4: clock (3)]
    100015                  I                                  [swi5: fast taskq]
    100017                  I                                  [swi6: task queue]
    100018                  I                                  [swi6: Giant taskq]
    100033                  I                                  [irq264: xhci0]
    100039                  I                                  [irq265: hdac0]
    100040                  I                                  [irq266: igb0:que 0]
    100042                  I                                  [irq267: igb0:link]
    100043                  Run    CPU 2                      [irq268: igb1:que 0]
    100045                  I                                  [irq269: igb1:link]
    100046                  I                                  [irq270: igb2:que 0]
    100048                  I                                  [irq271: igb2:link]
    100051                  I                                  [irq272: iwn0]
    100052                  I                                  [irq273: igb3:que 0]
    100054                  I                                  [irq274: igb3:link]
    100055                  I                                  [irq23: ehci0]
    100061                  I                                  [irq275: ahci0]
    100062                  I                                  [swi0: uart]
    100063                  I                                  [irq1: atkbd0]
    100067                  I                                  [swi1: pf send]
    100068                  I                                  [swi1: pfsync]
    100086                  I                                  [swi1: netisr 1]
    100087                  I                                  [swi1: netisr 2]
    100088                  Run    CPU 0                      [swi1: netisr 3]
      11    0    0    0  RL      (threaded)                  [idle]
    100003                  CanRun                              [idle: cpu0]
    100004                  Run    CPU 1                      [idle: cpu1]
    100005                  CanRun                              [idle: cpu2]
    100006                  Run    CPU 3                      [idle: cpu3]
        1    0    1    0  SLs    wait    0xfffff80006217528 [init]
      10    0    0    0  DL      audit_wo 0xffffffff82a146e0 [audit]
        0    0    0    0  DLs    (threaded)                  [kernel]
    100000                  D      swapin  0xffffffff829e5778 [swapper]
    100013                  D      -        0xfffff8000623f100 [aiod_kick taskq]
    100014                  D      -        0xfffff8000623ee00



  • in my case when i use num_queues it instantly crashes on HFSC, without it im able to use till i dont go over 50mbps



  • I would test on 2.4.1.

    The igb driver on FreeBSD 11.0 (pfSense 2,4) is missing 'hw.igb.buf_ring_size' tunable so setting it on 2.4 probably has no affect, and igb had some bugs which are fixed in FreeBSD 11.1 (pfSense 2.4.1.). The sysctl 'hw.igb.buf_ring_size' is also reinstated.



  • I started getting one or two crash reboot a day on 2.4RC. I have submitted every one of them. Here is the latest:

    					Crash report begins.  Anonymous machine information:
    
    amd64
    11.0-RELEASE-p12
    FreeBSD 11.0-RELEASE-p12 #58 daf8d77a4c1(RELENG_2_4_0): Fri Sep 22 20:48:22 CDT 2017     root@buildbot2.netgate.com:/builder/ce-240/tmp/obj/builder/ce-240/tmp/FreeBSD-src/sys/pfSense
    
    Crash report details:
    
    No PHP errors found.
    
    Filename: /var/crash/bounds
    1
    
    Filename: /var/crash/info.0
    Dump header from device: /dev/gptid/09925e84-90d4-11e7-84bc-80ee73c2ffc4
      Architecture: amd64
      Architecture Version: 1
      Dump Length: 79360
      Blocksize: 512
      Dumptime: Sat Sep 23 12:45:13 2017
      Hostname: alix1.nowayout.ca
      Magic: FreeBSD Text Dump
      Version String: FreeBSD 11.0-RELEASE-p12 #58 daf8d77a4c1(RELENG_2_4_0): Fri Sep 22 20:48:22 CDT 2017
        root@buildbot2.netgate.com:/builder/ce-240/tmp/obj/builder/ce-240/tmp/FreeBSD-src/sys/pfSense
      Panic String: hfsc_dequeue:
      Dump Parity: 3690452271
      Bounds: 0
      Dump Status: good
    
    Filename: /var/crash/info.last
    Dump header from device: /dev/gptid/09925e84-90d4-11e7-84bc-80ee73c2ffc4
      Architecture: amd64
      Architecture Version: 1
      Dump Length: 79360
      Blocksize: 512
      Dumptime: Sat Sep 23 12:45:13 2017
      Hostname: alix1.nowayout.ca
      Magic: FreeBSD Text Dump
      Version String: FreeBSD 11.0-RELEASE-p12 #58 daf8d77a4c1(RELENG_2_4_0): Fri Sep 22 20:48:22 CDT 2017
        root@buildbot2.netgate.com:/builder/ce-240/tmp/obj/builder/ce-240/tmp/FreeBSD-src/sys/pfSense
      Panic String: hfsc_dequeue:
      Dump Parity: 3690452271
      Bounds: 0
      Dump Status: good
    
    Filename: /var/crash/minfree
    2048
    
    Filename: /var/crash/textdump.tar.0
    ddb.txt06000014000013161507631  7075 ustarrootwheeldb:0:kdb.enter.default>  run lockinfo
    db:1:lockinfo> show locks
    No such command
    db:1:locks>  show alllocks
    No such command
    db:1:alllocks>  show lockedvnods
    Locked vnodes
    db:0:kdb.enter.default>  show pcpu
    cpuid        = 1
    dynamic pcpu = 0xfffffe0170be4200
    curthread    = 0xfffff80003983000: pid 0 "em0 que"
    curpcb       = 0xfffffe0117d60c80
    fpcurthread  = none
    idlethread   = 0xfffff8000395fa00: tid 100004 "idle: cpu1"
    curpmap      = 0xffffffff829e6300
    tssp         = 0xffffffff82a1eb78
    commontssp   = 0xffffffff82a1eb78
    rsp0         = 0xfffffe0117d60c80
    gs32p        = 0xffffffff82a253d0
    ldt          = 0xffffffff82a25410
    tss          = 0xffffffff82a25400
    db:0:kdb.enter.default>  bt
    Tracing pid 0 tid 100027 td 0xfffff80003983000
    kdb_enter() at kdb_enter+0x3b/frame 0xfffffe0117d60370
    vpanic() at vpanic+0x19f/frame 0xfffffe0117d603f0
    panic() at panic+0x43/frame 0xfffffe0117d60450
    hfsc_dequeue() at hfsc_dequeue+0x633/frame 0xfffffe0117d60490
    tbr_dequeue() at tbr_dequeue+0xdf/frame 0xfffffe0117d604e0
    vlan_start() at vlan_start+0x12c/frame 0xfffffe0117d60540
    if_transmit() at if_transmit+0x1a5/frame 0xfffffe0117d60590
    ether_output() at ether_output+0x728/frame 0xfffffe0117d60620
    ip_tryforward() at ip_tryforward+0x726/frame 0xfffffe0117d606a0
    ip_input() at ip_input+0x3a4/frame 0xfffffe0117d60700
    netisr_dispatch_src() at netisr_dispatch_src+0xa5/frame 0xfffffe0117d60760
    ether_demux() at ether_demux+0x15c/frame 0xfffffe0117d60790
    ether_nh_input() at ether_nh_input+0x317/frame 0xfffffe0117d607f0
    netisr_dispatch_src() at netisr_dispatch_src+0xa5/frame 0xfffffe0117d60850
    ether_input() at ether_input+0x26/frame 0xfffffe0117d60870
    vlan_input() at vlan_input+0x1cb/frame 0xfffffe0117d608f0
    ether_demux() at ether_demux+0xa9/frame 0xfffffe0117d60920
    ether_nh_input() at ether_nh_input+0x317/frame 0xfffffe0117d60980
    netisr_dispatch_src() at netisr_dispatch_src+0xa5/frame 0xfffffe0117d609e0
    ether_input() at ether_input+0x26/frame 0xfffffe0117d60a00
    if_input() at if_input+0xa/frame 0xfffffe0117d60a10
    em_rxeof() at em_rxeof+0x352/frame 0xfffffe0117d60a90
    em_handle_que() at em_handle_que+0x43/frame 0xfffffe0117d60ae0
    taskqueue_run_locked() at taskqueue_run_locked+0x14a/frame 0xfffffe0117d60b40
    taskqueue_thread_loop() at taskqueue_thread_loop+0xe8/frame 0xfffffe0117d60b70
    fork_exit() at fork_exit+0x85/frame 0xfffffe0117d60bb0
    fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0117d60bb0
    --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
    db:0:kdb.enter.default>  ps
      pid  ppid  pgrp   uid   state   wmesg         wchan        cmd
    82872 90006   552     0  S       nanslp   0xffffffff82866b30 sleep
    80573   552   552     0  S       accept   0xfffff8000e29172c php-fpm
    67765     1 67616   558  S       select   0xfffff8000eae9f40 avahi-daemon
    67480     1   552     0  S       nanslp   0xffffffff82866b31 php
    63593     1 63593   556  Ss      select   0xfffff800616e2d40 dbus-daemon
    90006     1   552     0  S       wait     0xfffff8000e137000 sh
    89044     1 89044   136  Ss      select   0xfffff80061027dc0 dhcpd
    76084     1 76084    59  Ss      (threaded)                  unbound
    100251                   S       kqread   0xfffff8000e6f3500 unbound
    100272                   S       kqread   0xfffff8000e964700 unbound
    53080     1 53080     0  Ss      (threaded)                  dpinger
    100157                   S       uwait    0xfffff8000e2b2500 dpinger
    100268                   S       sbwait   0xfffff80085533144 dpinger
    100269                   S       nanslp   0xffffffff82866b30 dpinger
    100270                   S       nanslp   0xffffffff82866b31 dpinger
    100271                   S       accept   0xfffff8008550f72c dpinger
    52697     1 52697     0  Ss      (threaded)                  dpinger
    100218                   S       uwait    0xfffff8000ea68080 dpinger
    100264                   S       sbwait   0xfffff8007b0e7144 dpinger
    100265                   S       nanslp   0xffffffff82866b31 dpinger
    100266                   S       nanslp   0xffffffff82866b31 dpinger
    100267                   S       accept   0xfffff80061ad93cc dpinger
    51997     1 51997     0  Ss      (threaded)                  dpinger
    100142                   S       uwait    0xfffff8000e027780 dpinger
    100260                   S       sbwait   0xfffff800855084a4 dpinger
    100261                   S       nanslp   0xffffffff82866b30 dpinger
    100262                   S       nanslp   0xffffffff82866b31 dpinger
    100263                   S       accept   0xfffff8001460aa8c dpinger
     6177     1  6177     0  Rs      CPU 0                       openvpn
    57344     1 57344     0  Ss+     ttyin    0xfffff8000badc8a8 getty
    57229     1 57229     0  Ss+     ttyin    0xfffff8000badcca8 getty
    56954     1 56954     0  Ss+     ttyin    0xfffff8000badd8a8 getty
    56790     1 56790     0  Ss+     ttyin    0xfffff8000badd0a8 getty
    56606     1 56606     0  Ss+     ttyin    0xfffff8000badd4a8 getty
    56492     1 56492     0  Ss+     ttyin    0xfffff8000badf4a8 getty
    56169     1 56169     0  Ss+     ttyin    0xfffff8000badf8a8 getty
    56044     1 56044     0  Ss+     ttyin    0xfffff8000badfca8 getty
     9047  8128  8128     0  S       nanslp   0xffffffff82866b31 minicron
     8128     1  8128     0  Ss      wait     0xfffff800856a8528 minicron
     7894  7188  7188     0  S       nanslp   0xffffffff82866b30 minicron
     7188     1  7188     0  Ss      wait     0xfffff80021a3b000 minicron
     6870  6432  6432     0  S       nanslp   0xffffffff82866b31 minicron
     6432     1  6432     0  Ss      wait     0xfffff80085203528 minicron
     6187     1  6187     0  Ss      (threaded)                  filterdns
    100227                   S       uwait    0xfffff80061029900 ns1.he.net
    100228                   S       uwait    0xfffff8006107b000 ns2.he.net
    100229                   S       uwait    0xfffff8006107a400 ns3.he.net
    100230                   S       uwait    0xfffff8000ee31d00 ns4.he.net
    100231                   S       uwait    0xfffff80061734d00 ns5.he.net
    100232                   S       uwait    0xfffff80061736600 filterdns
    100233                   S       uwait    0xfffff800616ffd80 filterdns
    100234                   S       uwait    0xfffff8006171e680 filterdns
    100235                   S       uwait    0xfffff8006171ff00 filterdns
    100236                   S       uwait    0xfffff800616b1d80 filterdns
    100237                   S       uwait    0xfffff8000efb3300 signal-thread
     2653     1  2358     0  S       kqread   0xfffff8000e64d300 lighttpd_pfb
    82428     1 82428   556  Ss      select   0xfffff80061734f40 dbus-daemon
    70117     1 70117     0  Ss      select   0xfffff8000e663b40 syslogd
    42738     1 42738     0  Ss      select   0xfffff8000ee5b6c0 miniupnpd
    37886     1 37886     0  Ss      select   0xfffff8000e91b0c0 powerd
    70983     1 70983     0  Ss      (threaded)                  ntpd
    100203                   S       select   0xfffff8000ee04540 ntpd
    100205                   S       usem     0xfffff8000e663680 ntpd
    70411     1 70411     0  Ss      nanslp   0xffffffff82866b30 cron
    69785 69275 69275     0  S       kqread   0xfffff8000ec6f200 nginx
    69509 69275 69275     0  S       kqread   0xfffff8000ec8ae00 nginx
    69275     1 69275     0  Ss      pause    0xfffff8000e1260a8 nginx
    48784     1 48784     0  Ss      bpf      0xfffff80061192800 filterlog
    44364     1 44364     0  Ss      select   0xfffff800612d47c0 openvpn
    29529     1 29529    65  Ss      select   0xfffff8000e662c40 dhclient
    24129     1 24129     0  Ss      select   0xfffff8000e5e88c0 dhclient
    16032     1 16032     0  Ss      (threaded)                  sshlockout_pf
    100143                   S       uwait    0xfffff8000e027880 sshlockout_pf
    100156                   S       nanslp   0xffffffff82866b30 sshlockout_pf
    15937     1 15937     0  Ss      select   0xfffff80061663340 sshd
      580     1   580     0  Ss      select   0xfffff800616032c0 devd
      568   566   566     0  S       kqread   0xfffff800611b5200 check_reload_status
      566     1   566     0  Ss      kqread   0xfffff80061149e00 check_reload_status
      552     1   552     0  Ss      kqread   0xfffff800611b4700 php-fpm
       59     0     0     0  DL      mdwait   0xfffff80003bba800 [md0]
       27     0     0     0  DL      syncer   0xffffffff82960680 [syncer]
       26     0     0     0  DL      vlruwt   0xfffff8000bb1e528 [vnlru]
       25     0     0     0  DL      (threaded)                  [bufdaemon]
    100120                   D       psleep   0xffffffff8295ef04 [bufdaemon]
    100132                   D       sdflush  0xfffff8000bbb28e8 [/ worker]
       24     0     0     0  DL      -        0xffffffff8295fbb4 [bufspacedaemon]
       23     0     0     0  DL      pgzero   0xffffffff829742e4 [pagezero]
       22     0     0     0  DL      psleep   0xffffffff8297078c [vmdaemon]
       21     0     0     0  DL      (threaded)                  [pagedaemon]
    100116                   D       psleep   0xffffffff82a1df05 [pagedaemon]
    100126                   D       umarcl   0xffffffff829700b8 [uma]
       20     0     0     0  DL      idle     0xfffff8000bb20528 [enc_daemon0]
       19     0     0     0  DL      -        0xffffffff8295e714 [soaiod4]
       18     0     0     0  DL      -        0xffffffff8295e714 [soaiod3]
       17     0     0     0  DL      -        0xffffffff8295e714 [soaiod2]
        9     0     0     0  DL      -        0xffffffff8295e714 [soaiod1]
       16     0     0     0  DL      tzpoll   0xffffffff82621f38 [acpi_thermal]
        8     0     0     0  DL      -        0xffffffff8273b2f0 [rand_harvestq]
        7     0     0     0  DL      pftm     0xffffffff80f4c510 [pf purge]
        6     0     0     0  DL      waiting_ 0xffffffff82a0d540 [sctp_iterator]
        5     0     0     0  SL      -        0xfffffe0000fde000 [fw0_probe]
       15     0     0     0  DL      (threaded)                  [usb]
    100030                   D       -        0xfffffe0000f22de8 [usbus0]
    100031                   D       -        0xfffffe0000f22e40 [usbus0]
    100032                   D       -        0xfffffe0000f22e98 [usbus0]
    100033                   D       -        0xfffffe0000f22ef0 [usbus0]
    100034                   D       -        0xfffffe0000f22f48 [usbus0]
    100036                   D       -        0xfffffe0000f3ede8 [usbus1]
    100037                   D       -        0xfffffe0000f3ee40 [usbus1]
    100038                   D       -        0xfffffe0000f3ee98 [usbus1]
    100039                   D       -        0xfffffe0000f3eef0 [usbus1]
    100040                   D       -        0xfffffe0000f3ef48 [usbus1]
    100042                   D       -        0xfffffe0000f4ede8 [usbus2]
    100043                   D       -        0xfffffe0000f4ee40 [usbus2]
    100044                   D       -        0xfffffe0000f4ee98 [usbus2]
    100045                   D       -        0xfffffe0000f4eef0 [usbus2]
    100046                   D       -        0xfffffe0000f4ef48 [usbus2]
    100047                   D       -        0xfffffe0000f5ed10 [usbus3]
    100048                   D       -        0xfffffe0000f5ed68 [usbus3]
    100049                   D       -        0xfffffe0000f5edc0 [usbus3]
    100050                   D       -        0xfffffe0000f5ee18 [usbus3]
    100051                   D       -        0xfffffe0000f5ee70 [usbus3]
    100059                   D       -        0xfffffe0000f6ede8 [usbus4]
    100060                   D       -        0xfffffe0000f6ee40 [usbus4]
    100061                   D       -        0xfffffe0000f6ee98 [usbus4]
    100062                   D       -        0xfffffe0000f6eef0 [usbus4]
    100063                   D       -        0xfffffe0000f6ef48 [usbus4]
    100064                   D       -        0xfffffe0000fb4de8 [usbus5]
    100065                   D       -        0xfffffe0000fb4e40 [usbus5]
    100066                   D       -        0xfffffe0000fb4e98 [usbus5]
    100067                   D       -        0xfffffe0000fb4ef0 [usbus5]
    100068                   D       -        0xfffffe0000fb4f48 [usbus5]
    100069                   D       -        0xfffffe0000fc4de8 [usbus6]
    100070                   D       -        0xfffffe0000fc4e40 [usbus6]
    100071                   D       -        0xfffffe0000fc4e98 [usbus6]
    100072                   D       -        0xfffffe0000fc4ef0 [usbus6]
    100073                   D       -        0xfffffe0000fc4f48 [usbus6]
    100074                   D       -        0xfffffe0000fd4d10 [usbus7]
    100075                   D       -        0xfffffe0000fd4d68 [usbus7]
    100076                   D       -        0xfffffe0000fd4dc0 [usbus7]
    100077                   D       -        0xfffffe0000fd4e18 [usbus7]
    100078                   D       -        0xfffffe0000fd4e70 [usbus7]
        4     0     0     0  DL      (threaded)                  [cam]
    100026                   D       -        0xffffffff8260e180 [doneq0]
    100110                   D       -        0xffffffff8260dfc8 [scanner]
        3     0     0     0  DL      crypto_r 0xffffffff8296ec70 [crypto returns]
        2     0     0     0  DL      crypto_w 0xffffffff8296eb18 [crypto]
       14     0     0     0  DL      (threaded)                  [geom]
    100017                   D       -        0xffffffff829e5740 [g_event]
    100018                   D       -        0xffffffff829e5748 [g_up]
    100019                   D       -        0xffffffff829e5750 [g_down]
       13     0     0     0  DL      (threaded)                  [ng_queue]
    100015                   D       sleep    0xffffffff825cbfd0 [ng_queue0]
    100016                   D       sleep    0xffffffff825cbfd0 [ng_queue1]
       12     0     0     0  WL      (threaded)                  [intr]
    100005                   I                                   [swi3: vm]
    100006                   I                                   [swi4: clock (0)]
    100007                   I                                   [swi4: clock (1)]
    100008                   I                                   [swi1: netisr 0]
    100009                   I                                   [swi6: task queue]
    100010                   I                                   [swi6: Giant taskq]
    100013                   I                                   [swi5: fast taskq]
    100029                   I                                   [irq20: hpet0 uhci0*]
    100035                   I                                   [irq21: uhci1 uhci4]
    100041                   I                                   [irq22: uhci2 ehci0+]
    100052                   I                                   [irq257: hdac0]
    100053                   I                                   [irq258: pcib1]
    100056                   I                                   [irq259: iwn0]
    100057                   I                                   [irq260: pcib2]
    100058                   I                                   [irq261: pcib3]
    100079                   I                                   [irq17: fwohci0]
    100082                   I                                   [irq18: sdhci_pci0+]
    100083                   I                                   [irq262: ahci0:ch0]
    100084                   I                                   [irq263: ahci0:ch1]
    100085                   I                                   [irq264: ahci0:ch2]
    100086                   I                                   [irq265: ahci0:ch3]
    100087                   I                                   [irq266: ahci0:ch4]
    100088                   I                                   [irq267: ahci0:ch5]
    100089                   I                                   [irq268: ahci0:6]
    100090                   I                                   [irq269: ahci0:7]
    100091                   I                                   [irq270: ahci0:8]
    100092                   I                                   [irq271: ahci0:9]
    100093                   I                                   [irq272: ahci0:10]
    100094                   I                                   [irq273: ahci0:11]
    100095                   I                                   [irq274: ahci0:12]
    100096                   I                                   [irq275: ahci0:13]
    100097                   I                                   [irq276: ahci0:14]
    100098                   I                                   [irq277: ahci0:15]
    100099                   I                                   [irq1: atkbd0]
    100100                   I                                   [irq12: psm0]
    100101                   I                                   [swi0: uart]
    100105                   I                                   [swi1: pf send]
    100106                   I                                   [swi1: pfsync]
    100123                   I                                   [swi1: netisr 1]
       11     0     0     0  RL      (threaded)                  [idle]
    100003                   CanRun                              [idle: cpu0]
    100004                   CanRun                              [idle: cpu1]
        1     0     1     0  SLs     wait     0xfffff8000395d528 [init]
       10     0     0     0  DL      audit_wo 0xffffffff82a146e0 [audit]
        0     0     0     0  RLs     (threaded)                  [kernel]
    100000                   D       swapin   0xffffffff829e5778 [swapper]
    100011                   D       -        0xfffff8000394d000 [aiod_kick taskq]
    100012                   D       -        0xfffff8000394cd00 [thread]
    100014                   D       -        0xfffff8000394c800 [kqueue_ctx taskq]
    100020                   D       -        0xfffff80003989d00 [firmware taskq]
    100023                   D       -        0xfffff80003988600 [acpi_task_0]
    100024                   D       -        0xfffff80003988600 [acpi_task_1]
    100025                   D       -        0xfffff80003988600 [acpi_task_2]
    100027                   Run     CPU 1                       [em0 que]
    100028                   D       -        0xfffff80003ba7700 [em0 txq]
    100054                   D       -        0xfffff8000516a500 [iwn0 net80211 taskq]
    100055                   D       -        0xfffff8000516a300 [iwn_taskq]
    100080                   D       -        0xfffff80003bf9600 [fw0_taskq]
    100102                   D       -        0xfffff8000b098a00 [mca taskq]
    100109                   D       -        0xfffff80003988200 [CAM taskq]
    100124                   D       -        0xfffff8000baf5d00 [if_config_tqg_0]
    100125                   D       -        0xfffff8000bb79b00 [if_io_tqg_0]
    100127                   D       -        0xfffff8000bb79900 [if_io_tqg_1]
    db:0:kdb.enter.default>  alltrace
    
    Tracing command sleep pid 82872 tid 100226 td 0xfffff8008572a500
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe011f068880
    mi_switch() at mi_switch+0xd2/frame 0xfffffe011f0688b0
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011f068930
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe011f068960
    _sleep() at _sleep+0x26f/frame 0xfffffe011f0689f0
    kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe011f068a50
    sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe011f068a90
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011f068bb0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011f068bb0
    --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x8009008fa, rsp = 0x7fffffffec78, rbp = 0x7fffffffecc0 ---
    
    Tracing command php-fpm pid 80573 tid 100225 td 0xfffff8008572aa00
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe011f063810
    mi_switch() at mi_switch+0xd2/frame 0xfffffe011f063840
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011f0638c0
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011f0638f0
    _sleep() at _sleep+0x294/frame 0xfffffe011f063980
    kern_accept4() at kern_accept4+0x1f2/frame 0xfffffe011f063a30
    accept1() at accept1+0x60/frame 0xfffffe011f063a90
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011f063bb0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011f063bb0
    --- syscall (30, FreeBSD ELF64, sys_accept), rip = 0x801d939ba, rsp = 0x7ffffffec848, rbp = 0x7ffffffec880 ---
    
    Tracing command avahi-daemon pid 67765 tid 100255 td 0xfffff80021e6fa00
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe011f0f4730
    mi_switch() at mi_switch+0xd2/frame 0xfffffe011f0f4760
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011f0f47e0
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe011f0f4810
    _cv_timedwait_sig_sbt() at _cv_timedwait_sig_sbt+0x1c4/frame 0xfffffe011f0f4880
    seltdwait() at seltdwait+0xc7/frame 0xfffffe011f0f48d0
    kern_poll() at kern_poll+0x296/frame 0xfffffe011f0f4a70
    sys_poll() at sys_poll+0x61/frame 0xfffffe011f0f4a90
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011f0f4bb0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011f0f4bb0
    --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x801814f7a, rsp = 0x7fffffffe7d8, rbp = 0x7fffffffe810 ---
    
    Tracing command php pid 67480 tid 100254 td 0xfffff80021e86000
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe011f0ef880
    mi_switch() at mi_switch+0xd2/frame 0xfffffe011f0ef8b0
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011f0ef930
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe011f0ef960
    _sleep() at _sleep+0x26f/frame 0xfffffe011f0ef9f0
    kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe011f0efa50
    sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe011f0efa90
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011f0efbb0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011f0efbb0
    --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x801d848fa, rsp = 0x7fffffffc7c8, rbp = 0x7fffffffc7f0 ---
    
    Tracing command dbus-daemon pid 63593 tid 100250 td 0xfffff80021e87500
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe011f0db750
    mi_switch() at mi_switch+0xd2/frame 0xfffffe011f0db780
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011f0db800
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011f0db830
    _cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe011f0db880
    seltdwait() at seltdwait+0x10f/frame 0xfffffe011f0db8d0
    kern_poll() at kern_poll+0x296/frame 0xfffffe011f0dba70
    sys_poll() at sys_poll+0x61/frame 0xfffffe011f0dba90
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011f0dbbb0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011f0dbbb0
    --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x8011dcf7a, rsp = 0x7fffffffe858, rbp = 0x7fffffffe890 ---
    
    Tracing command sh pid 90006 tid 100146 td 0xfffff8000e0d7000
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe011ef4d660
    mi_switch() at mi_switch+0xd2/frame 0xfffffe011ef4d690
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011ef4d710
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011ef4d740
    _sleep() at _sleep+0x294/frame 0xfffffe011ef4d7d0
    kern_wait6() at kern_wait6+0x2d3/frame 0xfffffe011ef4d8a0
    sys_wait4() at sys_wait4+0x73/frame 0xfffffe011ef4da90
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011ef4dbb0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011ef4dbb0
    --- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800b5d75a, rsp = 0x7fffffffe868, rbp = 0x7fffffffe920 ---
    
    Tracing command dhcpd pid 89044 tid 100148 td 0xfffff8000e13b500
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe011ef576e0
    mi_switch() at mi_switch+0xd2/frame 0xfffffe011ef57710
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011ef57790
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011ef577c0
    _cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe011ef57810
    seltdwait() at seltdwait+0x10f/frame 0xfffffe011ef57860
    kern_select() at kern_select+0x979/frame 0xfffffe011ef57a50
    sys_select() at sys_select+0x54/frame 0xfffffe011ef57a90
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011ef57bb0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011ef57bb0
    --- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800d641ca, rsp = 0x7fffffffea08, rbp = 0x7fffffffea30 ---
    
    Tracing command unbound pid 76084 tid 100251 td 0xfffff80021e87000
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe011f0e0620
    mi_switch() at mi_switch+0xd2/frame 0xfffffe011f0e0650
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011f0e06d0
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe011f0e0700
    _sleep() at _sleep+0x26f/frame 0xfffffe011f0e0790
    kqueue_kevent() at kqueue_kevent+0x336/frame 0xfffffe011f0e0970
    kern_kevent() at kern_kevent+0xb5/frame 0xfffffe011f0e09e0
    sys_kevent() at sys_kevent+0x11c/frame 0xfffffe011f0e0a90
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011f0e0bb0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011f0e0bb0
    --- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x801b7293a, rsp = 0x7fffffffea18, rbp = 0x7fffffffea60 ---
    
    Tracing command unbound pid 76084 tid 100272 td 0xfffff800854e0a00
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe011f103620
    mi_switch() at mi_switch+0xd2/frame 0xfffffe011f103650
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011f1036d0
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011f103700
    _sleep() at _sleep+0x294/frame 0xfffffe011f103790
    kqueue_kevent() at kqueue_kevent+0x336/frame 0xfffffe011f103970
    kern_kevent() at kern_kevent+0xb5/frame 0xfffffe011f1039e0
    sys_kevent() at sys_kevent+0x11c/frame 0xfffffe011f103a90
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011f103bb0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011f103bb0
    --- syscall (363, FreeBSD ELF64, sys_kevent), rip = 0x801b7293a, rsp = 0x7fffdfffde18, rbp = 0x7fffdfffde60 ---
    
    Tracing command dpinger pid 53080 tid 100157 td 0xfffff8007b0e5500
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe011ef847f0
    mi_switch() at mi_switch+0xd2/frame 0xfffffe011ef84820
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011ef848a0
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011ef848d0
    _sleep() at _sleep+0x294/frame 0xfffffe011ef84960
    umtxq_sleep() at umtxq_sleep+0x122/frame 0xfffffe011ef849c0
    do_wait() at do_wait+0x419/frame 0xfffffe011ef84a50
    __umtx_op_wait() at __umtx_op_wait+0x77/frame 0xfffffe011ef84a90
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011ef84bb0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011ef84bb0
    --- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x80083c67c, rsp = 0x7fffffffe8f8, rbp = 0x7fffffffe980 ---
    
    Tracing command dpinger pid 53080 tid 100268 td 0xfffff800854df500
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe011efb16f0
    mi_switch() at mi_switch+0xd2/frame 0xfffffe011efb1720
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011efb17a0
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011efb17d0
    _sleep() at _sleep+0x294/frame 0xfffffe011efb1860
    sbwait() at sbwait+0x4f/frame 0xfffffe011efb1880
    soreceive_generic() at soreceive_generic+0x1ae/frame 0xfffffe011efb1950
    kern_recvit() at kern_recvit+0x1d6/frame 0xfffffe011efb1a20
    sys_recvfrom() at sys_recvfrom+0x86/frame 0xfffffe011efb1a90
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011efb1bb0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011efb1bb0
    --- syscall (29, FreeBSD ELF64, sys_recvfrom), rip = 0x800b3cf5a, rsp = 0x7fffdfffde98, rbp = 0x7fffdfffdee0 ---
    
    Tracing command dpinger pid 53080 tid 100269 td 0xfffff800854df000
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe011efb6880
    mi_switch() at mi_switch+0xd2/frame 0xfffffe011efb68b0
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011efb6930
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe011efb6960
    _sleep() at _sleep+0x26f/frame 0xfffffe011efb69f0
    kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe011efb6a50
    sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe011efb6a90
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011efb6bb0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011efb6bb0
    --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b2b8fa, rsp = 0x7fffdfdfcf58, rbp = 0x7fffdfdfcf80 ---
    
    Tracing command dpinger pid 53080 tid 100270 td 0xfffff800c31cfa00
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe011efbb880
    mi_switch() at mi_switch+0xd2/frame 0xfffffe011efbb8b0
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011efbb930
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe011efbb960
    _sleep() at _sleep+0x26f/frame 0xfffffe011efbb9f0
    kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe011efbba50
    sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe011efbba90
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011efbbbb0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011efbbbb0
    --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b2b8fa, rsp = 0x7fffdfbfbef8, rbp = 0x7fffdfbfbf20 ---
    
    Tracing command dpinger pid 53080 tid 100271 td 0xfffff800c31cf500
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe011efc0810
    mi_switch() at mi_switch+0xd2/frame 0xfffffe011efc0840
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011efc08c0
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011efc08f0
    _sleep() at _sleep+0x294/frame 0xfffffe011efc0980
    kern_accept4() at kern_accept4+0x1f2/frame 0xfffffe011efc0a30
    accept1() at accept1+0xf6/frame 0xfffffe011efc0a90
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011efc0bb0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011efc0bb0
    --- syscall (30, FreeBSD ELF64, sys_accept), rip = 0x800b2b9ba, rsp = 0x7fffdf9fae78, rbp = 0x7fffdf9faeb0 ---
    
    Tracing command dpinger pid 52697 tid 100218 td 0xfffff8007b294000
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe011f0407f0
    mi_switch() at mi_switch+0xd2/frame 0xfffffe011f040820
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011f0408a0
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011f0408d0
    _sleep() at _sleep+0x294/frame 0xfffffe011f040960
    umtxq_sleep() at umtxq_sleep+0x122/frame 0xfffffe011f0409c0
    do_wait() at do_wait+0x419/frame 0xfffffe011f040a50
    __umtx_op_wait() at __umtx_op_wait+0x77/frame 0xfffffe011f040a90
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011f040bb0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011f040bb0
    --- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x80083c67c, rsp = 0x7fffffffe908, rbp = 0x7fffffffe990 ---
    
    Tracing command dpinger pid 52697 tid 100264 td 0xfffff8007b24aa00
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe011ef936f0
    mi_switch() at mi_switch+0xd2/frame 0xfffffe011ef93720
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011ef937a0
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011ef937d0
    _sleep() at _sleep+0x294/frame 0xfffffe011ef93860
    sbwait() at sbwait+0x4f/frame 0xfffffe011ef93880
    soreceive_generic() at soreceive_generic+0x1ae/frame 0xfffffe011ef93950
    kern_recvit() at kern_recvit+0x1d6/frame 0xfffffe011ef93a20
    sys_recvfrom() at sys_recvfrom+0x86/frame 0xfffffe011ef93a90
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011ef93bb0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011ef93bb0
    --- syscall (29, FreeBSD ELF64, sys_recvfrom), rip = 0x800b3cf5a, rsp = 0x7fffdfffde98, rbp = 0x7fffdfffdee0 ---
    
    Tracing command dpinger pid 52697 tid 100265 td 0xfffff8000bb7da00
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe011ef98880
    mi_switch() at mi_switch+0xd2/frame 0xfffffe011ef988b0
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011ef98930
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe011ef98960
    _sleep() at _sleep+0x26f/frame 0xfffffe011ef989f0
    kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe011ef98a50
    sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe011ef98a90
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011ef98bb0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011ef98bb0
    --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b2b8fa, rsp = 0x7fffdfdfcf58, rbp = 0x7fffdfdfcf80 ---
    
    Tracing command dpinger pid 52697 tid 100266 td 0xfffff8007b0e4000
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe011ef9d880
    mi_switch() at mi_switch+0xd2/frame 0xfffffe011ef9d8b0
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011ef9d930
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe011ef9d960
    _sleep() at _sleep+0x26f/frame 0xfffffe011ef9d9f0
    kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe011ef9da50
    sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe011ef9da90
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011ef9dbb0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011ef9dbb0
    --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b2b8fa, rsp = 0x7fffdfbfbef8, rbp = 0x7fffdfbfbf20 ---
    
    Tracing command dpinger pid 52697 tid 100267 td 0xfffff8007b13ba00
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe011efa2810
    mi_switch() at mi_switch+0xd2/frame 0xfffffe011efa2840
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011efa28c0
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011efa28f0
    _sleep() at _sleep+0x294/frame 0xfffffe011efa2980
    kern_accept4() at kern_accept4+0x1f2/frame 0xfffffe011efa2a30
    accept1() at accept1+0xf6/frame 0xfffffe011efa2a90
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011efa2bb0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011efa2bb0
    --- syscall (30, FreeBSD ELF64, sys_accept), rip = 0x800b2b9ba, rsp = 0x7fffdf9fae78, rbp = 0x7fffdf9faeb0 ---
    
    Tracing command dpinger pid 51997 tid 100142 td 0xfffff8000e121500
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe011ef397f0
    mi_switch() at mi_switch+0xd2/frame 0xfffffe011ef39820
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011ef398a0
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011ef398d0
    _sleep() at _sleep+0x294/frame 0xfffffe011ef39960
    umtxq_sleep() at umtxq_sleep+0x122/frame 0xfffffe011ef399c0
    do_wait() at do_wait+0x419/frame 0xfffffe011ef39a50
    __umtx_op_wait() at __umtx_op_wait+0x77/frame 0xfffffe011ef39a90
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011ef39bb0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011ef39bb0
    --- syscall (454, FreeBSD ELF64, sys__umtx_op), rip = 0x80083c67c, rsp = 0x7fffffffe908, rbp = 0x7fffffffe990 ---
    
    Tracing command dpinger pid 51997 tid 100260 td 0xfffff8007b196500
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe011efd96f0
    mi_switch() at mi_switch+0xd2/frame 0xfffffe011efd9720
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011efd97a0
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011efd97d0
    _sleep() at _sleep+0x294/frame 0xfffffe011efd9860
    sbwait() at sbwait+0x4f/frame 0xfffffe011efd9880
    soreceive_generic() at soreceive_generic+0x1ae/frame 0xfffffe011efd9950
    kern_recvit() at kern_recvit+0x1d6/frame 0xfffffe011efd9a20
    sys_recvfrom() at sys_recvfrom+0x86/frame 0xfffffe011efd9a90
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011efd9bb0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011efd9bb0
    --- syscall (29, FreeBSD ELF64, sys_recvfrom), rip = 0x800b3cf5a, rsp = 0x7fffdfffde98, rbp = 0x7fffdfffdee0 ---
    
    Tracing command dpinger pid 51997 tid 100261 td 0xfffff8007b19fa00
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe011efde880
    mi_switch() at mi_switch+0xd2/frame 0xfffffe011efde8b0
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011efde930
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe011efde960
    _sleep() at _sleep+0x26f/frame 0xfffffe011efde9f0
    kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe011efdea50
    sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe011efdea90
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011efdebb0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011efdebb0
    --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b2b8fa, rsp = 0x7fffdfdfcf58, rbp = 0x7fffdfdfcf80 ---
    
    Tracing command dpinger pid 51997 tid 100262 td 0xfffff800854e0000
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe011efe3880
    mi_switch() at mi_switch+0xd2/frame 0xfffffe011efe38b0
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011efe3930
    sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe011efe3960
    _sleep() at _sleep+0x26f/frame 0xfffffe011efe39f0
    kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe011efe3a50
    sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe011efe3a90
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011efe3bb0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011efe3bb0
    --- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b2b8fa, rsp = 0x7fffdfbfbef8, rbp = 0x7fffdfbfbf20 ---
    
    Tracing command dpinger pid 51997 tid 100263 td 0xfffff800854dfa00
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe011efe8810
    mi_switch() at mi_switch+0xd2/frame 0xfffffe011efe8840
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011efe88c0
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011efe88f0
    _sleep() at _sleep+0x294/frame 0xfffffe011efe8980
    kern_accept4() at kern_accept4+0x1f2/frame 0xfffffe011efe8a30
    accept1() at accept1+0xf6/frame 0xfffffe011efe8a90
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011efe8bb0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011efe8bb0
    --- syscall (30, FreeBSD ELF64, sys_accept), rip = 0x800b2b9ba, rsp = 0x7fffdf9fae78, rbp = 0x7fffdf9faeb0 ---
    
    Tracing command openvpn pid 6177 tid 100241 td 0xfffff80021dc1500
    cpustop_handler() at cpustop_handler+0x28/frame 0xffffffff82978530
    ipi_nmi_handler() at ipi_nmi_handler+0x4a/frame 0xffffffff82978550
    trap() at trap+0x3a/frame 0xffffffff82978760
    nmi_calltrap() at nmi_calltrap+0x8/frame 0xffffffff82978760
    --- trap 0x13, rip = 0xffffffff80cf8666, rsp = 0xfffffe011f0ae880, rbp = 0xfffffe011f0ae8d0 ---
    sopoll_generic() at sopoll_generic+0x46/frame 0xfffffe011f0ae8d0
    kern_poll() at kern_poll+0x650/frame 0xfffffe011f0aea70
    sys_poll() at sys_poll+0x61/frame 0xfffffe011f0aea90
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011f0aebb0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011f0aebb0
    --- syscall (209, FreeBSD ELF64, sys_poll), rip = 0x80155cf7a, rsp = 0x7fffffffdd78, rbp = 0x7fffffffdde0 ---
    
    Tracing command getty pid 57344 tid 100150 td 0xfffff8000e13aa00
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe011ef61760
    mi_switch() at mi_switch+0xd2/frame 0xfffffe011ef61790
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011ef61810
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011ef61840
    _cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe011ef61890
    tty_wait() at tty_wait+0x1c/frame 0xfffffe011ef618b0
    ttydisc_read() at ttydisc_read+0x30a/frame 0xfffffe011ef61910
    ttydev_read() at ttydev_read+0x34/frame 0xfffffe011ef61940
    devfs_read_f() at devfs_read_f+0xe8/frame 0xfffffe011ef619a0
    dofileread() at dofileread+0x98/frame 0xfffffe011ef619f0
    kern_readv() at kern_readv+0x68/frame 0xfffffe011ef61a40
    sys_read() at sys_read+0x84/frame 0xfffffe011ef61a90
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011ef61bb0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011ef61bb0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b811ea, rsp = 0x7fffffffec78, rbp = 0x7fffffffecc0 ---
    
    Tracing command getty pid 57229 tid 100151 td 0xfffff8000e13a500
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe011ef66760
    mi_switch() at mi_switch+0xd2/frame 0xfffffe011ef66790
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011ef66810
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011ef66840
    _cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe011ef66890
    tty_wait() at tty_wait+0x1c/frame 0xfffffe011ef668b0
    ttydisc_read() at ttydisc_read+0x30a/frame 0xfffffe011ef66910
    ttydev_read() at ttydev_read+0x34/frame 0xfffffe011ef66940
    devfs_read_f() at devfs_read_f+0xe8/frame 0xfffffe011ef669a0
    dofileread() at dofileread+0x98/frame 0xfffffe011ef669f0
    kern_readv() at kern_readv+0x68/frame 0xfffffe011ef66a40
    sys_read() at sys_read+0x84/frame 0xfffffe011ef66a90
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011ef66bb0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011ef66bb0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b811ea, rsp = 0x7fffffffec78, rbp = 0x7fffffffecc0 ---
    
    Tracing command getty pid 56954 tid 100240 td 0xfffff80021e87a00
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe011f0a9760
    mi_switch() at mi_switch+0xd2/frame 0xfffffe011f0a9790
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011f0a9810
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011f0a9840
    _cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe011f0a9890
    tty_wait() at tty_wait+0x1c/frame 0xfffffe011f0a98b0
    ttydisc_read() at ttydisc_read+0x30a/frame 0xfffffe011f0a9910
    ttydev_read() at ttydev_read+0x34/frame 0xfffffe011f0a9940
    devfs_read_f() at devfs_read_f+0xe8/frame 0xfffffe011f0a99a0
    dofileread() at dofileread+0x98/frame 0xfffffe011f0a99f0
    kern_readv() at kern_readv+0x68/frame 0xfffffe011f0a9a40
    sys_read() at sys_read+0x84/frame 0xfffffe011f0a9a90
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011f0a9bb0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011f0a9bb0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b811ea, rsp = 0x7fffffffec78, rbp = 0x7fffffffecc0 ---
    
    Tracing command getty pid 56790 tid 100135 td 0xfffff8000e040a00
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe011ef16760
    mi_switch() at mi_switch+0xd2/frame 0xfffffe011ef16790
    sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011ef16810
    sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011ef16840
    _cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe011ef16890
    tty_wait() at tty_wait+0x1c/frame 0xfffffe011ef168b0
    ttydisc_read() at ttydisc_read+0x30a/frame 0xfffffe011ef16910
    ttydev_read() at ttydev_read+0x34/frame 0xfffffe011ef16940
    devfs_read_f() at devfs_read_f+0xe8/frame 0xfffffe011ef169a0
    dofileread() at dofileread+0x98/frame 0xfffffe011ef169f0
    kern_readv() at kern_readv+0x68/frame 0xfffffe011ef16a40
    sys_read() at sys_read+0x84/frame 0xfffffe011ef16a90
    amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011ef16bb0
    Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011ef16bb0
    --- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b811ea, rsp = 0x7fffffffec78, rbp = 0x7fffffffecc0 ---
    
    Tracing command getty pid 56606 tid 100224 td 0xfffff8008572b000
    sched_switch() at sched_switch+0x6cb/frame 0xfffffe011f05e760
    mi_switch() at mi_switch+0xd2/f[/thread]
    


  • @chrcoluk:

    I would test on 2.4.1.

    The igb driver on FreeBSD 11.0 (pfSense 2,4) is missing 'hw.igb.buf_ring_size' tunable so setting it on 2.4 probably has no affect, and igb had some bugs which are fixed in FreeBSD 11.1 (pfSense 2.4.1.). The sysctl 'hw.igb.buf_ring_size' is also reinstated.

    isnt it possible to backport the igb drivers from 11.1 for now rather than ship pfsense 2.4 with broken traffic shaping?



  • Did someone test the traffic shaping on 2.4.1?
    I didn’t have the time to do it this week, and next week I don’t have the time either.



  • @avink:

    Did someone test the traffic shaping on 2.4.1?
    I didn’t have the time to do it this week, and next week I don’t have the time either.

    it needs you guys with the problem to test, no point in me testing as I dont get the crash on 2.4 anyway.

    Its far easier for you to test than for someone to backport the driver, (note the fixes are also in the kernel rather than the driver).

    Do a config backup.
    Download 2.4.1 snapshot from pfsense servers.
    Install 2.4.1
    Restore the backup.



  • but the main question is how stable is 2.4.1 coz on my box on 2.4 it doesnt crash and on this i can test but where it crashes is a client box in a remote location so if 2.4.1 isnt stable then it will be cause a lot of headache for me bringing it back to 2.4



  • I tried 2.4.1 with 'em' E1000 nics on ESXi.. for me it crashes..

    Version	2.4.1-DEVELOPMENT (amd64) 
    built on Sat Sep 23 19:46:59 CDT 2017 
    FreeBSD 11.1-RELEASE-p1 
    
    
    amd64
    11.1-RELEASE-p1
    FreeBSD 11.1-RELEASE-p1 #299 r313908+13ee0afae40(RELENG_2_4): Sat Sep 23 19:54:08 CDT 2017     root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense
    
    Crash report details:
    
    No PHP errors found.
    
    Filename: /var/crash/bounds
    1
    
    Filename: /var/crash/info.0
    Dump header from device: /dev/da0s1b
      Architecture: amd64
      Architecture Version: 1
      Dump Length: 86016
      Blocksize: 512
      Dumptime: Sun Sep 24 19:07:58 2017
      Hostname: pfsense_3.local
      Magic: FreeBSD Text Dump
      Version String: FreeBSD 11.1-RELEASE-p1 #299 r313908+13ee0afae40(RELENG_2_4): Sat Sep 23 19:54:08 CDT 2017
        root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense
      Panic String: hfsc_dequeue:
      Dump Parity: 1670814533
      Bounds: 0
      Dump Status: good
    
    Filename: /var/crash/info.last
    Dump header from device: /dev/da0s1b
      Architecture: amd64
      Architecture Version: 1
      Dump Length: 86016
      Blocksize: 512
      Dumptime: Sun Sep 24 19:07:58 2017
      Hostname: pfsense_3.local
      Magic: FreeBSD Text Dump
      Version String: FreeBSD 11.1-RELEASE-p1 #299 r313908+13ee0afae40(RELENG_2_4): Sat Sep 23 19:54:08 CDT 2017
        root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense
      Panic String: hfsc_dequeue:
      Dump Parity: 1670814533
      Bounds: 0
      Dump Status: good
    
    Filename: /var/crash/minfree
    2048
    
    Filename: /var/crash/textdump.tar.0
    ddb.txt06000014000013161763156  7103 ustarrootwheeldb:0:kdb.enter.default>  run lockinfo
    db:1:lockinfo> show locks
    No such command
    db:1:locks>  show alllocks
    No such command
    db:1:alllocks>  show lockedvnods
    Locked vnodes
    db:0:kdb.enter.default>  show pcpu
    cpuid        = 1
    dynamic pcpu = 0xfffffe00981a3200
    curthread    = 0xfffff80003215560: pid 0 "em4 taskq"
    curpcb       = 0xfffffe002b4f4b80
    fpcurthread  = none
    idlethread   = 0xfffff800031df560: tid 100004 "idle: cpu1"
    curpmap      = 0xffffffff82a3a7c0
    tssp         = 0xffffffff82a738f8
    commontssp   = 0xffffffff82a738f8
    rsp0         = 0xfffffe002b4f4b80
    gs32p        = 0xffffffff82a7a150
    ldt          = 0xffffffff82a7a190
    tss          = 0xffffffff82a7a180
    db:0:kdb.enter.default>  bt
    Tracing pid 0 tid 100036 td 0xfffff80003215560
    kdb_enter() at kdb_enter+0x3b/frame 0xfffffe002b4f3cd0
    vpanic() at vpanic+0x1a3/frame 0xfffffe002b4f3d50
    panic() at panic+0x43/frame 0xfffffe002b4f3db0
    hfsc_dequeue() at hfsc_dequeue+0x1a6/frame 0xfffffe002b4f3df0
    tbr_dequeue() at tbr_dequeue+0xea/frame 0xfffffe002b4f3e40
    vlan_start() at vlan_start+0x196/frame 0xfffffe002b4f3e90
    if_transmit() at if_transmit+0x15c/frame 0xfffffe002b4f3ed0
    ether_output() at ether_output+0x718/frame 0xfffffe002b4f3f60
    ip_output() at ip_output+0x154a/frame 0xfffffe002b4f40b0
    ip_forward() at ip_forward+0x323/frame 0xfffffe002b4f4150
    ip_input() at ip_input+0x75a/frame 0xfffffe002b4f41b0
    netisr_dispatch_src() at netisr_dispatch_src+0xa0/frame 0xfffffe002b4f4200
    ng_iface_rcvdata() at ng_iface_rcvdata+0x11a/frame 0xfffffe002b4f4230
    ng_apply_item() at ng_apply_item+0xde/frame 0xfffffe002b4f42b0
    ng_snd_item() at ng_snd_item+0x18c/frame 0xfffffe002b4f42f0
    ng_apply_item() at ng_apply_item+0xde/frame 0xfffffe002b4f4370
    ng_snd_item() at ng_snd_item+0x18c/frame 0xfffffe002b4f43b0
    ng_ppp_rcvdata() at ng_ppp_rcvdata+0xe6b/frame 0xfffffe002b4f4430
    ng_apply_item() at ng_apply_item+0xde/frame 0xfffffe002b4f44b0
    ng_snd_item() at ng_snd_item+0x18c/frame 0xfffffe002b4f44f0
    ng_apply_item() at ng_apply_item+0xde/frame 0xfffffe002b4f4570
    ng_snd_item() at ng_snd_item+0x18c/frame 0xfffffe002b4f45b0
    ng_apply_item() at ng_apply_item+0xde/frame 0xfffffe002b4f4630
    ng_snd_item() at ng_snd_item+0x18c/frame 0xfffffe002b4f4670
    ether_demux() at ether_demux+0x240/frame 0xfffffe002b4f46a0
    ether_nh_input() at ether_nh_input+0x310/frame 0xfffffe002b4f4700
    netisr_dispatch_src() at netisr_dispatch_src+0xa0/frame 0xfffffe002b4f4750
    ether_input() at ether_input+0x26/frame 0xfffffe002b4f4770
    vlan_input() at vlan_input+0x1f0/frame 0xfffffe002b4f47f0
    ether_demux() at ether_demux+0x156/frame 0xfffffe002b4f4820
    ether_nh_input() at ether_nh_input+0x310/frame 0xfffffe002b4f4880
    netisr_dispatch_src() at netisr_dispatch_src+0xa0/frame 0xfffffe002b4f48d0
    ether_input() at ether_input+0x26/frame 0xfffffe002b4f48f0
    if_input() at if_input+0xa/frame 0xfffffe002b4f4900
    lem_rxeof() at lem_rxeof+0x3df/frame 0xfffffe002b4f49a0
    lem_handle_rxtx() at lem_handle_rxtx+0x32/frame 0xfffffe002b4f49e0
    taskqueue_run_locked() at taskqueue_run_locked+0x127/frame 0xfffffe002b4f4a40
    taskqueue_thread_loop() at taskqueue_thread_loop+0xc8/frame 0xfffffe002b4f4a70
    fork_exit() at fork_exit+0x85/frame 0xfffffe002b4f4ab0
    fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe002b4f4ab0
    --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
    

  • Banned

    Afraid this is really an uphill battle and altq is pretty much dead… Not sure what's the plan in FreeBSD but this buggy mess is not maintainable by Netgate alone.



  • Yes its reliant on FreeBSD upstream, pfSense may refine the code, but you entering a dangerous situation if you start making large en masse changes from the upstream product as this would need maintaining outside of upstream and be a pain on every upstream update.

    FreeBSD made a decision to not keep up with openbsd in regards to PF and ALTQ so they made the rod on their own back with that decision, the good news is that they have started to fix FreeBSD specific PF issues now and I would hope this includes ALTQ as well.

    The problem in this case there is no specific confirmed bug, just a few guys reporting crashes at high load, it needs lots of testing to dig deeper.

    My advice was on the basis I am aware of scheduler panic fixes and igb panic fixes in FreeBSD 11.1 upstream hence what I posted.


  • Banned

    @chrcoluk:

    FreeBSD made a decision to not keep up with openbsd in regards to PF and ALTQ so they made the rod on their own back with that decision, the good news is that they have started to fix FreeBSD specific PF issues now and I would hope this includes ALTQ as well.

    ALTQ has been gone from OpenBSD for years. http://undeadly.org/cgi?action=article&sid=20140419151959



  • my crash log attached as follows, im also having same issue on client box when i enable shaping, sorry the log was too long to capture completely

    crash.txt
    crash2.txt



  • A quick update from my side.
    Both ALTQ and HFSC crash on 2.4.1 as well.
    It doesn’t have to have any load. It crashes only seconds after reboot.

    Will try to upload the crash info tonight.



  • @doktornotor:

    @chrcoluk:

    FreeBSD made a decision to not keep up with openbsd in regards to PF and ALTQ so they made the rod on their own back with that decision, the good news is that they have started to fix FreeBSD specific PF issues now and I would hope this includes ALTQ as well.

    ALTQ has been gone from OpenBSD for years. http://undeadly.org/cgi?action=article&sid=20140419151959

    It is gone now yes, but before it was removed, it was updated and improved over what the version in FreeBSD is.



  • Luiz fixed it.!. ;D

    https://redmine.pfsense.org/issues/7879#note-3

    At least after initial testing with and pushing some traffic through my queue's which seemed like a sure way to crash it before, i'm gone try and see how it holds up tomorrow :)

    2.4.0-RC (amd64) 
    built on Mon Sep 25 15:04:23 CDT 2017 
    FreeBSD 11.0-RELEASE-p12 
    

    Perhaps some of you guy's here can also give it another testrun.? 8)



  • i tried on 2 apu2 boxes and now works all good, can any1 point me to the actual commit that fixed this?



  • It's running for a few hours now.
    No crash since the reboot, so this looks promising.


Log in to reply