CARP related kernel panic

innosys

Hello,

Running into a bit of a situation with two firewalls with in a CARP'd configuration.

The master is running without a hitch, but the slave runs ok for a few days, and then panics and dies.

A reboot cures it for a few days, and then it repeats.

It's been happening consistently with every build in the last few months.

Precursor of the messages received before the slave fw panics:
12:04 AM
A communications error occured while attempting XMLRPC sync with username admin https://10.0.0.2:443.

12:04AM
An error code was received while attempting XMLRPC sync with username admin https://10.0.0.2:443 - Code 2: Invalid return payload: enable debugging to examine incoming payload

12:05AM
A communications error occured while attempting XMLRPC sync with username admin https://10.0.0.2:443.

12:09AM
A communications error occured while attempting Filter sync with username admin https://10.0.0.2:443.

02:13AM
An error code was received while attempting XMLRPC sync with username admin https://10.0.0.2:443 - Code 2: Invalid return payload: enable debugging to examine incoming payload

02:13AM
A communications error occured while attempting XMLRPC sync with username admin https://10.0.0.2:443.

02:13AM
A communications error occured while attempting Filter sync with username admin https://10.0.0.2:443.

Build / Specs for both machines:

2.0-RC1 (amd64)
built on Mon May 9 04:17:17 EDT 2011

Dell R310's with Intel(R) Core(TM) i3 CPU 540 @ 3.07GHz, 4GB mem

Debug output:


db> bt
Tracing pid 27017 tid 64147 td 0xffffff002d578000
_rw_wlock_hard() at _rw_wlock_hard+0x6a
carp_setrun() at carp_setrun+0x21b
carp_ioctl() at carp_ioctl+0xc9c
in_control() at in_control+0x1fb
ifioctl() at ifioctl+0x1427
kern_ioctl() at kern_ioctl+0xf6
ioctl() at ioctl+0xfd
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800a7f3ac, rsp = 0x7fffffffe3b8, rbp = 0x8 ---
db> set $lines 0
db> allt

Tracing command sh pid 28800 tid 64101 td 0xffffff00031f8ba0
cpustop_handler() at cpustop_handler+0x40
ipi_nmi_handler() at ipi_nmi_handler+0x30
trap() at trap+0x175
nmi_calltrap() at nmi_calltrap+0x8
--- trap 0x13, rip = 0xffffffff806c4bf1, rsp = 0xffffffff812004c0, rbp = 0xffffff810dda9b00 ---
_fdrop() at _fdrop+0x1
kern_close() at kern_close+0x14d
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (6, FreeBSD ELF64, close), rip = 0x80052854c, rsp = 0x7fffffffe608, rbp = 0 ---

Tracing command php pid 28526 tid 64183 td 0xffffff002df21000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sched_preempt() at sched_preempt+0xc3
ipi_bitmap_handler() at ipi_bitmap_handler+0x70
Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x9b
--- interrupt, rip = 0x8011f43cc, rsp = 0x7fffffff3b18, rbp = 0 ---

Tracing command ifconfig pid 27017 tid 64147 td 0xffffff002d578000
_rw_wlock_hard() at _rw_wlock_hard+0x6a
carp_setrun() at carp_setrun+0x21b
carp_ioctl() at carp_ioctl+0xc9c
in_control() at in_control+0x1fb
ifioctl() at ifioctl+0x1427
kern_ioctl() at kern_ioctl+0xf6
ioctl() at ioctl+0xfd
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800a7f3ac, rsp = 0x7fffffffe3b8, rbp = 0x8 ---

Tracing command sleep pid 1637 tid 64167 td 0xffffff002d549000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_catch_signals() at sleepq_catch_signals+0x31f
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_sleep() at _sleep+0x1a3
kern_nanosleep() at kern_nanosleep+0x118
nanosleep() at nanosleep+0x6e
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80071e5cc, rsp = 0x7fffffffed28, rbp = 0x7fffffffed80 ---

Tracing command dhcpd pid 8864 tid 64107 td 0xffffff000324d000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_catch_signals() at sleepq_catch_signals+0x31f
sleepq_wait_sig() at sleepq_wait_sig+0xc
_cv_wait_sig() at _cv_wait_sig+0x120
seltdwait() at seltdwait+0xf6
kern_select() at kern_select+0x64d
select() at select+0x5d
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x800a7538c, rsp = 0x7fffffffe728, rbp = 0x7fffffffe9f0 ---

Tracing command dnsmasq pid 53632 tid 64146 td 0xffffff002d5783e0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_catch_signals() at sleepq_catch_signals+0x31f
sleepq_wait_sig() at sleepq_wait_sig+0xc
_cv_wait_sig() at _cv_wait_sig+0x120
seltdwait() at seltdwait+0xf6
kern_select() at kern_select+0x64d
select() at select+0x5d
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x80089138c, rsp = 0x7fffffffe9a8, rbp = 0x7fffffffea60 ---

Tracing command snort pid 13651 tid 64180 td 0xffffff002d539ba0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_catch_signals() at sleepq_catch_signals+0x31f
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_sleep() at _sleep+0x1a3
bpfread() at bpfread+0x28b
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x800b7940c, rsp = 0x7fffffffea58, rbp = 0x7fffffffec10 ---

Tracing command ntpd pid 40407 tid 64100 td 0xffffff000325a000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_catch_signals() at sleepq_catch_signals+0x31f
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_cv_timedwait_sig() at _cv_timedwait_sig+0x129
seltdwait() at seltdwait+0x8d
poll() at poll+0x2f8
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (209, FreeBSD ELF64, poll), rip = 0x8006ec3bc, rsp = 0x7fffffffec08, rbp = 0x3a98 ---

Tracing command ntpd pid 36740 tid 64108 td 0xffffff00031f8000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_catch_signals() at sleepq_catch_signals+0x31f
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_cv_timedwait_sig() at _cv_timedwait_sig+0x129
seltdwait() at seltdwait+0x8d
poll() at poll+0x2f8
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (209, FreeBSD ELF64, poll), rip = 0x8006ec3bc, rsp = 0x7fffffffeb08, rbp = 0 ---

Tracing command openvpn pid 28449 tid 64104 td 0xffffff000324d7c0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_catch_signals() at sleepq_catch_signals+0x31f
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_cv_timedwait_sig() at _cv_timedwait_sig+0x129
seltdwait() at seltdwait+0x8d
poll() at poll+0x2f8
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (209, FreeBSD ELF64, poll), rip = 0x800c663bc, rsp = 0x7fffffffd0f8, rbp = 0x7fffffffd2a0 ---

Tracing command rrdtool pid 42919 tid 64111 td 0xffffff00031f73e0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_catch_signals() at sleepq_catch_signals+0x31f
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x26b
pipe_read() at pipe_read+0x4c1
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x800e7840c, rsp = 0x7fffffffeb38, rbp = 0x801004000 ---

Tracing command apinger pid 42578 tid 64172 td 0xffffff002d63eba0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_catch_signals() at sleepq_catch_signals+0x31f
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_cv_timedwait_sig() at _cv_timedwait_sig+0x129
seltdwait() at seltdwait+0x8d
poll() at poll+0x2f8
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (209, FreeBSD ELF64, poll), rip = 0x8006f03bc, rsp = 0x7fffffffcab8, rbp = 0x1 ---

Tracing command getty pid 61977 tid 64151 td 0xffffff002d539000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_catch_signals() at sleepq_catch_signals+0x31f
sleepq_wait_sig() at sleepq_wait_sig+0xc
_cv_wait_sig() at _cv_wait_sig+0x120
tty_wait() at tty_wait+0x25
ttydisc_read() at ttydisc_read+0x2b1
ttydev_read() at ttydev_read+0x10e
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80084e40c, rsp = 0x7fffffffecc8, rbp = 0 ---

Tracing command getty pid 61780 tid 64098 td 0xffffff00031f9000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_catch_signals() at sleepq_catch_signals+0x31f
sleepq_wait_sig() at sleepq_wait_sig+0xc
_cv_wait_sig() at _cv_wait_sig+0x120
tty_wait() at tty_wait+0x25
ttydisc_read() at ttydisc_read+0x2b1
ttydev_read() at ttydev_read+0x10e
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80084e40c, rsp = 0x7fffffffecc8, rbp = 0 ---

Tracing command minicron pid 32770 tid 64136 td 0xffffff002d57bba0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_catch_signals() at sleepq_catch_signals+0x31f
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_sleep() at _sleep+0x1a3
kern_nanosleep() at kern_nanosleep+0x118
nanosleep() at nanosleep+0x6e
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80071e5cc, rsp = 0x7fffffffec38, rbp = 0x15180 ---

Tracing command minicron pid 31468 tid 64162 td 0xffffff002d51d3e0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_catch_signals() at sleepq_catch_signals+0x31f
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_sleep() at _sleep+0x1a3
kern_nanosleep() at kern_nanosleep+0x118
nanosleep() at nanosleep+0x6e
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80071e5cc, rsp = 0x7fffffffec48, rbp = 0xe10 ---

Tracing command minicron pid 30618 tid 64143 td 0xffffff002d57a000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_catch_signals() at sleepq_catch_signals+0x31f
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_sleep() at _sleep+0x1a3
kern_nanosleep() at kern_nanosleep+0x118
nanosleep() at nanosleep+0x6e
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80071e5cc, rsp = 0x7fffffffec48, rbp = 0xf0 ---

Tracing command cron pid 1460 tid 64140 td 0xffffff002d57aba0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_catch_signals() at sleepq_catch_signals+0x31f
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_sleep() at _sleep+0x1a3
kern_nanosleep() at kern_nanosleep+0x118
nanosleep() at nanosleep+0x6e
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80093e5cc, rsp = 0x7fffffffeb08, rbp = 0x37 ---

Tracing command sh pid 63804 tid 64127 td 0xffffff00032f3000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_catch_signals() at sleepq_catch_signals+0x31f
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x26b
kern_wait() at kern_wait+0x77e
wait4() at wait4+0x35
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x800934afc, rsp = 0x7fffffffe8a8, rbp = 0x19 ---

Tracing command relayd pid 40525 tid 64157 td 0xffffff002d51e7c0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_catch_signals() at sleepq_catch_signals+0x31f
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_sleep() at _sleep+0x1a3
kern_kevent() at kern_kevent+0x365
kevent() at kevent+0x90
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (363, FreeBSD ELF64, kevent), rip = 0x800c405fc, rsp = 0x7fffffffe788, rbp = 0x800e76000 ---

Tracing command relayd pid 40261 tid 64156 td 0xffffff002d51eba0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_catch_signals() at sleepq_catch_signals+0x31f
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_sleep() at _sleep+0x1a3
kern_kevent() at kern_kevent+0x365
kevent() at kevent+0x90
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (363, FreeBSD ELF64, kevent), rip = 0x800c405fc, rsp = 0x7fffffffe788, rbp = 0x800eb3000 ---

Tracing command relayd pid 39784 tid 64155 td 0xffffff002d538000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_catch_signals() at sleepq_catch_signals+0x31f
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x26b
kern_kevent() at kern_kevent+0x365
kevent() at kevent+0x90
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (363, FreeBSD ELF64, kevent), rip = 0x800c405fc, rsp = 0x7fffffffe7c8, rbp = 0x800ea7000 ---

Tracing command php pid 38824 tid 64137 td 0xffffff002d57b7c0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_catch_signals() at sleepq_catch_signals+0x31f
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x26b
pipe_read() at pipe_read+0x4c1
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x8011f440c, rsp = 0x7fffffff5018, rbp = 0x807381710 ---

Tracing command php pid 38594 tid 64141 td 0xffffff002d57a7c0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
_cv_wait() at _cv_wait+0x111
fork1() at fork1+0x121f
vfork() at vfork+0x1c
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (66, FreeBSD ELF64, vfork), rip = 0x80115750d, rsp = 0x7fffffff3b20, rbp = 0 ---

Tracing command php pid 38426 tid 64134 td 0xffffff002d57c3e0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_catch_signals() at sleepq_catch_signals+0x31f
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x26b
pipe_read() at pipe_read+0x4c1
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x8011f440c, rsp = 0x7fffffff4048, rbp = 0x807388428 ---

Tracing command php pid 38279 tid 64149 td 0xffffff00032f67c0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_catch_signals() at sleepq_catch_signals+0x31f
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x26b
kern_accept() at kern_accept+0x1e2
accept() at accept+0x75
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (30, FreeBSD ELF64, accept), rip = 0x8011900ec, rsp = 0x7ffffffec818, rbp = 0 ---

Tracing command php pid 38153 tid 64148 td 0xffffff00032f6ba0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_catch_signals() at sleepq_catch_signals+0x31f
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x26b
kern_wait() at kern_wait+0x77e
wait4() at wait4+0x35
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x80116cafc, rsp = 0x7fffffffc938, rbp = 0x7fffffffec9c ---

Tracing command php pid 37164 tid 64112 td 0xffffff00031f7000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_catch_signals() at sleepq_catch_signals+0x31f
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x26b
kern_wait() at kern_wait+0x77e
wait4() at wait4+0x35
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x80116cafc, rsp = 0x7fffffffc938, rbp = 0x7fffffffec9c ---

Tracing command lighttpd pid 36719 tid 64118 td 0xffffff00032f63e0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_catch_signals() at sleepq_catch_signals+0x31f
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_sleep() at _sleep+0x1a3
kern_kevent() at kern_kevent+0x365
kevent() at kevent+0x90
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (363, FreeBSD ELF64, kevent), rip = 0x800c6f5fc, rsp = 0x7fffffffe9f8, rbp = 0x800e55400 ---

Tracing command inetd pid 5520 tid 64123 td 0xffffff00032f4000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_catch_signals() at sleepq_catch_signals+0x31f
sleepq_wait_sig() at sleepq_wait_sig+0xc
_cv_wait_sig() at _cv_wait_sig+0x120
seltdwait() at seltdwait+0xf6
kern_select() at kern_select+0x64d
select() at select+0x5d
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x800a6338c, rsp = 0x7fffffffdd58, rbp = 0 ---

Tracing command logger pid 59707 tid 64116 td 0xffffff00031f2000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_catch_signals() at sleepq_catch_signals+0x31f
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x26b
pipe_read() at pipe_read+0x4c1
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x80073b40c, rsp = 0x7fffffffe848, rbp = 0x7fffffffed60 ---

Tracing command tcpdump pid 59593 tid 64102 td 0xffffff00031f87c0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_catch_signals() at sleepq_catch_signals+0x31f
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19
_sleep() at _sleep+0x1a3
bpfread() at bpfread+0x28b
devfs_read_f() at devfs_read_f+0x86
dofileread() at dofileread+0xa1
kern_readv() at kern_readv+0x60
read() at read+0x55
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (3, FreeBSD ELF64, read), rip = 0x800b9b40c, rsp = 0x7fffffffe948, rbp = 0x800e05400 ---

Tracing command syslogd pid 58514 tid 64128 td 0xffffff00032f2ba0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_catch_signals() at sleepq_catch_signals+0x31f
sleepq_wait_sig() at sleepq_wait_sig+0xc
_cv_wait_sig() at _cv_wait_sig+0x120
seltdwait() at seltdwait+0xf6
kern_select() at kern_select+0x64d
select() at select+0x5d
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x80085138c, rsp = 0x7fffffffe098, rbp = 0x800a35110 ---

Tracing command sshd pid 14560 tid 64106 td 0xffffff000324d3e0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_catch_signals() at sleepq_catch_signals+0x31f
sleepq_wait_sig() at sleepq_wait_sig+0xc
_cv_wait_sig() at _cv_wait_sig+0x120
seltdwait() at seltdwait+0xf6
kern_select() at kern_select+0x64d
select() at select+0x5d
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x80109f38c, rsp = 0x7fffffffddb8, rbp = 0x2 ---

Tracing command devd pid 269 tid 64131 td 0xffffff00032f2000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_catch_signals() at sleepq_catch_signals+0x31f
sleepq_wait_sig() at sleepq_wait_sig+0xc
_cv_wait_sig() at _cv_wait_sig+0x120
seltdwait() at seltdwait+0xf6
kern_select() at kern_select+0x64d
select() at select+0x5d
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (93, FreeBSD ELF64, select), rip = 0x440efc, rsp = 0x7fffffffe878, rbp = 0x7fffffffe890 ---

Tracing command check_reload_status pid 251 tid 64117 td 0xffffff000301cba0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_catch_signals() at sleepq_catch_signals+0x31f
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x26b
kern_kevent() at kern_kevent+0x365
kevent() at kevent+0x90
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (363, FreeBSD ELF64, kevent), rip = 0x80083a5fc, rsp = 0x7fffffffeae8, rbp = 0x7fffffffec00 ---

Tracing command check_reload_status pid 249 tid 64124 td 0xffffff00032f3ba0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_catch_signals() at sleepq_catch_signals+0x31f
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x26b
kern_kevent() at kern_kevent+0x365
kevent() at kevent+0x90
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (363, FreeBSD ELF64, kevent), rip = 0x80083a5fc, rsp = 0x7fffffffea08, rbp = 0x800c0f000 ---

Tracing command md0 pid 40 tid 64109 td 0xffffff00031f7ba0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
_sleep() at _sleep+0x31c
md_kthread() at md_kthread+0x1e0
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff810ddd1d30, rbp = 0 ---

Tracing command flowcleaner pid 24 tid 64096 td 0xffffff0002d833e0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_timedwait() at sleepq_timedwait+0x42
_cv_timedwait() at _cv_timedwait+0x129
flowtable_cleaner() at flowtable_cleaner+0xe5
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff810ba35d30, rbp = 0 ---

Tracing command softdepflush pid 23 tid 64095 td 0xffffff0002d837c0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_timedwait() at sleepq_timedwait+0x42
_sleep() at _sleep+0x308
softdep_flush() at softdep_flush+0x34d
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff810ba30d30, rbp = 0 ---

Tracing command syncer pid 22 tid 64094 td 0xffffff0002d83ba0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_timedwait() at sleepq_timedwait+0x42
_cv_timedwait() at _cv_timedwait+0x129
sched_sync() at sched_sync+0x4e6
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff810ba2bd30, rbp = 0 ---

Tracing command vnlru pid 21 tid 64093 td 0xffffff0003019000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_timedwait() at sleepq_timedwait+0x42
_sleep() at _sleep+0x308
vnlru_proc() at vnlru_proc+0x5fd
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff810ba26d30, rbp = 0 ---

Tracing command bufdaemon pid 20 tid 64092 td 0xffffff00030193e0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_timedwait() at sleepq_timedwait+0x42
_sleep() at _sleep+0x308
buf_daemon() at buf_daemon+0x192
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff810ba21d30, rbp = 0 ---

Tracing command idlepoll pid 19 tid 64091 td 0xffffff00030197c0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_timedwait() at sleepq_timedwait+0x42
_sleep() at _sleep+0x308
poll_idle() at poll_idle+0x19f
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff810ba1cd30, rbp = 0 ---

Tracing command pagezero pid 18 tid 64090 td 0xffffff0003019ba0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_timedwait() at sleepq_timedwait+0x42
_sleep() at _sleep+0x308
vm_pagezero() at vm_pagezero+0x83
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff810ba17d30, rbp = 0 ---

Tracing command vmdaemon pid 17 tid 64089 td 0xffffff000301a000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
_sleep() at _sleep+0x31c
vm_daemon() at vm_daemon+0x58
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff810ba12d30, rbp = 0 ---

Tracing command pagedaemon pid 16 tid 64088 td 0xffffff000301a3e0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_timedwait() at sleepq_timedwait+0x42
_sleep() at _sleep+0x308
vm_pageout() at vm_pageout+0x877
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff810ba0dd30, rbp = 0 ---

Tracing command xpt_thrd pid 9 tid 64087 td 0xffffff000301a7c0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
_sleep() at _sleep+0x31c
xpt_scanner_thread() at xpt_scanner_thread+0xff
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff810b958d30, rbp = 0 ---

Tracing command pfpurge pid 8 tid 64086 td 0xffffff000301aba0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_timedwait() at sleepq_timedwait+0x42
_sleep() at _sleep+0x308
pf_purge_thread() at pf_purge_thread+0x31
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff810b953d30, rbp = 0 ---

Tracing command sctp_iterator pid 7 tid 64085 td 0xffffff000301c000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
_sleep() at _sleep+0x31c
sctp_iterator_thread() at sctp_iterator_thread+0x54
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff810b94ed30, rbp = 0 ---

Tracing command usb pid 15 tid 64080 td 0xffffff0002d807c0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
_cv_wait() at _cv_wait+0x111
usb_process() at usb_process+0x18b
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff810b8a9d30, rbp = 0 ---

Tracing command usb pid 15 tid 64079 td 0xffffff0002d80ba0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
_cv_wait() at _cv_wait+0x111
usb_process() at usb_process+0x18b
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff810b8a4d30, rbp = 0 ---

Tracing command usb pid 15 tid 64078 td 0xffffff0002d81000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
_cv_wait() at _cv_wait+0x111
usb_process() at usb_process+0x18b
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff810b89fd30, rbp = 0 ---

Tracing command usb pid 15 tid 64077 td 0xffffff0002d813e0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
_cv_wait() at _cv_wait+0x111
usb_process() at usb_process+0x18b
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff810b89ad30, rbp = 0 ---

Tracing command usb pid 15 tid 64074 td 0xffffff0002d82000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
_cv_wait() at _cv_wait+0x111
usb_process() at usb_process+0x18b
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff810b6c1d30, rbp = 0 ---

Tracing command usb pid 15 tid 64073 td 0xffffff0002d823e0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
_cv_wait() at _cv_wait+0x111
usb_process() at usb_process+0x18b
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff810b6bcd30, rbp = 0 ---

Tracing command usb pid 15 tid 64072 td 0xffffff0002d827c0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
_cv_wait() at _cv_wait+0x111
usb_process() at usb_process+0x18b
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff810b6b7d30, rbp = 0 ---

Tracing command usb pid 15 tid 64071 td 0xffffff0002d82ba0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
_cv_wait() at _cv_wait+0x111
usb_process() at usb_process+0x18b
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff810b6b2d30, rbp = 0 ---

Tracing command yarrow pid 14 tid 64018 td 0xffffff0002a03000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_timedwait() at sleepq_timedwait+0x42
_sleep() at _sleep+0x308
random_kthread() at random_kthread+0x1ed
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff800007ed30, rbp = 0 ---

Tracing command crypto returns pid 6 tid 64017 td 0xffffff0002a033e0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
_sleep() at _sleep+0x31c
crypto_ret_proc() at crypto_ret_proc+0x192
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000079d30, rbp = 0 ---

Tracing command crypto pid 5 tid 64016 td 0xffffff0002a037c0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
_sleep() at _sleep+0x31c
crypto_proc() at crypto_proc+0x187
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000074d30, rbp = 0 ---

Tracing command g_down pid 4 tid 64015 td 0xffffff0002a03ba0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
_sleep() at _sleep+0x31c
g_io_schedule_down() at g_io_schedule_down+0x21c
g_down_procbody() at g_down_procbody+0x58
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff800006fd30, rbp = 0 ---

Tracing command g_up pid 3 tid 64014 td 0xffffff0002a04000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
_sleep() at _sleep+0x31c
g_io_schedule_up() at g_io_schedule_up+0xe6
g_up_procbody() at g_up_procbody+0x58
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff800006ad30, rbp = 0 ---

Tracing command g_event pid 2 tid 64013 td 0xffffff0002a043e0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_timedwait() at sleepq_timedwait+0x42
_sleep() at _sleep+0x308
g_event_procbody() at g_event_procbody+0x8f
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000065d30, rbp = 0 ---

Tracing command ng_queue pid 13 tid 64011 td 0xffffff00029e97c0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
_sleep() at _sleep+0x31c
ngthread() at ngthread+0x52
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff800005bd30, rbp = 0 ---

Tracing command ng_queue pid 13 tid 64010 td 0xffffff00029e9ba0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
_sleep() at _sleep+0x31c
ngthread() at ngthread+0x52
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000056d30, rbp = 0 ---

Tracing command intr pid 12 tid 64084 td 0xffffff000301c3e0
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 64083 td 0xffffff000301c7c0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
ithread_loop() at ithread_loop+0x1d0
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff810b944d30, rbp = 0 ---

Tracing command intr pid 12 tid 64082 td 0xffffff0002d80000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
ithread_loop() at ithread_loop+0x1d0
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff810b8f9d30, rbp = 0 ---

Tracing command intr pid 12 tid 64081 td 0xffffff0002d803e0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
ithread_loop() at ithread_loop+0x1d0
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff810b8aed30, rbp = 0 ---

Tracing command intr pid 12 tid 64076 td 0xffffff0002d817c0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
ithread_loop() at ithread_loop+0x1d0
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff810b713d30, rbp = 0 ---

Tracing command intr pid 12 tid 64075 td 0xffffff0002d81ba0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
ithread_loop() at ithread_loop+0x1d0
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff810b6ead30, rbp = 0 ---

Tracing command intr pid 12 tid 64070 td 0xffffff0002d83000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
ithread_loop() at ithread_loop+0x1d0
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff810b6add30, rbp = 0 ---

Tracing command intr pid 12 tid 64069 td 0xffffff0002c60000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
ithread_loop() at ithread_loop+0x1d0
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff810b52ad30, rbp = 0 ---

Tracing command intr pid 12 tid 64067 td 0xffffff0002c607c0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
ithread_loop() at ithread_loop+0x1d0
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80001fbd30, rbp = 0 ---

Tracing command intr pid 12 tid 64065 td 0xffffff0002c61000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
ithread_loop() at ithread_loop+0x1d0
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80001f1d30, rbp = 0 ---

Tracing command intr pid 12 tid 64064 td 0xffffff0002c613e0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
ithread_loop() at ithread_loop+0x1d0
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80001e4d30, rbp = 0 ---

Tracing command intr pid 12 tid 64062 td 0xffffff0002c61ba0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
ithread_loop() at ithread_loop+0x1d0
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80001dad30, rbp = 0 ---

Tracing command intr pid 12 tid 64060 td 0xffffff0002c633e0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
ithread_loop() at ithread_loop+0x1d0
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80001d0d30, rbp = 0 ---

Tracing command intr pid 12 tid 64058 td 0xffffff0002c63ba0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
ithread_loop() at ithread_loop+0x1d0
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80001c6d30, rbp = 0 ---

Tracing command intr pid 12 tid 64056 td 0xffffff0002bf47c0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
ithread_loop() at ithread_loop+0x1d0
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80001bcd30, rbp = 0 ---

Tracing command intr pid 12 tid 64055 td 0xffffff0002bf4ba0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
ithread_loop() at ithread_loop+0x1d0
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000197d30, rbp = 0 ---

Tracing command intr pid 12 tid 64053 td 0xffffff0002c323e0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
ithread_loop() at ithread_loop+0x1d0
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff800018dd30, rbp = 0 ---

Tracing command intr pid 12 tid 64051 td 0xffffff0002c32ba0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
ithread_loop() at ithread_loop+0x1d0
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000183d30, rbp = 0 ---

Tracing command intr pid 12 tid 64049 td 0xffffff0002c333e0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
ithread_loop() at ithread_loop+0x1d0
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000179d30, rbp = 0 ---

Tracing command intr pid 12 tid 64047 td 0xffffff0002c33ba0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
ithread_loop() at ithread_loop+0x1d0
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff800016fd30, rbp = 0 ---

Tracing command intr pid 12 tid 64046 td 0xffffff0002ba9ba0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
ithread_loop() at ithread_loop+0x1d0
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff800014ad30, rbp = 0 ---

Tracing command intr pid 12 tid 64044 td 0xffffff0002bf13e0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
ithread_loop() at ithread_loop+0x1d0
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000140d30, rbp = 0 ---

Tracing command intr pid 12 tid 64042 td 0xffffff0002bf1ba0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
ithread_loop() at ithread_loop+0x1d0
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000136d30, rbp = 0 ---

Tracing command intr pid 12 tid 64040 td 0xffffff0002bf23e0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
ithread_loop() at ithread_loop+0x1d0
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff800012cd30, rbp = 0 ---

Tracing command intr pid 12 tid 64038 td 0xffffff0002bf2ba0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
ithread_loop() at ithread_loop+0x1d0
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000122d30, rbp = 0 ---

Tracing command intr pid 12 tid 64037 td 0xffffff0002bf4000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
ithread_loop() at ithread_loop+0x1d0
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80000fdd30, rbp = 0 ---

Tracing command intr pid 12 tid 64035 td 0xffffff0002b8cba0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
ithread_loop() at ithread_loop+0x1d0
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80000f3d30, rbp = 0 ---

Tracing command intr pid 12 tid 64033 td 0xffffff0002ba63e0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
ithread_loop() at ithread_loop+0x1d0
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80000e9d30, rbp = 0 ---

Tracing command intr pid 12 tid 64031 td 0xffffff0002ba6ba0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
ithread_loop() at ithread_loop+0x1d0
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80000dfd30, rbp = 0 ---

Tracing command intr pid 12 tid 64029 td 0xffffff0002ba93e0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
ithread_loop() at ithread_loop+0x1d0
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80000d5d30, rbp = 0 ---

Tracing command intr pid 12 tid 64028 td 0xffffff0002ba97c0
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 64027 td 0xffffff0002a047c0
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 64026 td 0xffffff0002a04ba0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
ithread_loop() at ithread_loop+0x1d0
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80000a6d30, rbp = 0 ---

Tracing command intr pid 12 tid 64025 td 0xffffff0002b8a000
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 64023 td 0xffffff0002b8a7c0
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 64009 td 0xffffff00029f9000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
ithread_loop() at ithread_loop+0x1d0
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000051d30, rbp = 0 ---

Tracing command intr pid 12 tid 64008 td 0xffffff00029f93e0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
ithread_loop() at ithread_loop+0x1d0
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff800004cd30, rbp = 0 ---

Tracing command intr pid 12 tid 64007 td 0xffffff00029f97c0
fork_trampoline() at fork_trampoline

Tracing command intr pid 12 tid 64006 td 0xffffff00029e8000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
ithread_loop() at ithread_loop+0x1d0
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000042d30, rbp = 0 ---

Tracing command idle pid 11 tid 64005 td 0xffffff00029e83e0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sched_idletd() at sched_idletd+0x27c
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff800003dd30, rbp = 0 ---

Tracing command idle pid 11 tid 64004 td 0xffffff00029e87c0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sched_idletd() at sched_idletd+0x27c
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000038d30, rbp = 0 ---

Tracing command init pid 1 tid 64003 td 0xffffff00029e8ba0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_catch_signals() at sleepq_catch_signals+0x31f
sleepq_wait_sig() at sleepq_wait_sig+0xc
_sleep() at _sleep+0x26b
kern_wait() at kern_wait+0x77e
wait4() at wait4+0x35
syscall() at syscall+0x1e7
Xfast_syscall() at Xfast_syscall+0xe1
--- syscall (7, FreeBSD ELF64, wait4), rip = 0x40c4fc, rsp = 0x7fffffffe808, rbp = 0x401d40 ---

Tracing command audit pid 10 tid 64002 td 0xffffff00029e9000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
_cv_wait() at _cv_wait+0x111
audit_worker() at audit_worker+0x359
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff800002ed30, rbp = 0 ---

Tracing command kernel pid 0 tid 64068 td 0xffffff0002c603e0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
msleep_spin() at msleep_spin+0x191
taskqueue_thread_loop() at taskqueue_thread_loop+0x67
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff810b525d30, rbp = 0 ---

Tracing command kernel pid 0 tid 64066 td 0xffffff0002c60ba0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
msleep_spin() at msleep_spin+0x191
taskqueue_thread_loop() at taskqueue_thread_loop+0x67
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80001f6d30, rbp = 0 ---

Tracing command kernel pid 0 tid 64063 td 0xffffff0002c617c0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
msleep_spin() at msleep_spin+0x191
taskqueue_thread_loop() at taskqueue_thread_loop+0x67
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80001dfd30, rbp = 0 ---

Tracing command kernel pid 0 tid 64061 td 0xffffff0002c63000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
msleep_spin() at msleep_spin+0x191
taskqueue_thread_loop() at taskqueue_thread_loop+0x67
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80001d5d30, rbp = 0 ---

Tracing command kernel pid 0 tid 64059 td 0xffffff0002c637c0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
msleep_spin() at msleep_spin+0x191
taskqueue_thread_loop() at taskqueue_thread_loop+0x67
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80001cbd30, rbp = 0 ---

Tracing command kernel pid 0 tid 64057 td 0xffffff0002bf43e0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
msleep_spin() at msleep_spin+0x191
taskqueue_thread_loop() at taskqueue_thread_loop+0x67
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80001c1d30, rbp = 0 ---

Tracing command kernel pid 0 tid 64054 td 0xffffff0002c32000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
msleep_spin() at msleep_spin+0x191
taskqueue_thread_loop() at taskqueue_thread_loop+0x67
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000192d30, rbp = 0 ---

Tracing command kernel pid 0 tid 64052 td 0xffffff0002c327c0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
msleep_spin() at msleep_spin+0x191
taskqueue_thread_loop() at taskqueue_thread_loop+0x67
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000188d30, rbp = 0 ---

Tracing command kernel pid 0 tid 64050 td 0xffffff0002c33000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
msleep_spin() at msleep_spin+0x191
taskqueue_thread_loop() at taskqueue_thread_loop+0x67
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff800017ed30, rbp = 0 ---

Tracing command kernel pid 0 tid 64048 td 0xffffff0002c337c0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
msleep_spin() at msleep_spin+0x191
taskqueue_thread_loop() at taskqueue_thread_loop+0x67
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000174d30, rbp = 0 ---

Tracing command kernel pid 0 tid 64045 td 0xffffff0002bf1000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
msleep_spin() at msleep_spin+0x191
taskqueue_thread_loop() at taskqueue_thread_loop+0x67
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000145d30, rbp = 0 ---

Tracing command kernel pid 0 tid 64043 td 0xffffff0002bf17c0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
msleep_spin() at msleep_spin+0x191
taskqueue_thread_loop() at taskqueue_thread_loop+0x67
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff800013bd30, rbp = 0 ---

Tracing command kernel pid 0 tid 64041 td 0xffffff0002bf2000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
msleep_spin() at msleep_spin+0x191
taskqueue_thread_loop() at taskqueue_thread_loop+0x67
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000131d30, rbp = 0 ---

Tracing command kernel pid 0 tid 64039 td 0xffffff0002bf27c0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
msleep_spin() at msleep_spin+0x191
taskqueue_thread_loop() at taskqueue_thread_loop+0x67
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000127d30, rbp = 0 ---

Tracing command kernel pid 0 tid 64036 td 0xffffff0002b8c7c0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
msleep_spin() at msleep_spin+0x191
taskqueue_thread_loop() at taskqueue_thread_loop+0x67
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80000f8d30, rbp = 0 ---

Tracing command kernel pid 0 tid 64034 td 0xffffff0002ba6000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
msleep_spin() at msleep_spin+0x191
taskqueue_thread_loop() at taskqueue_thread_loop+0x67
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80000eed30, rbp = 0 ---

Tracing command kernel pid 0 tid 64032 td 0xffffff0002ba67c0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
msleep_spin() at msleep_spin+0x191
taskqueue_thread_loop() at taskqueue_thread_loop+0x67
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80000e4d30, rbp = 0 ---

Tracing command kernel pid 0 tid 64030 td 0xffffff0002ba9000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
msleep_spin() at msleep_spin+0x191
taskqueue_thread_loop() at taskqueue_thread_loop+0x67
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff80000dad30, rbp = 0 ---

Tracing command kernel pid 0 tid 64024 td 0xffffff0002b8a3e0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
_sleep() at _sleep+0x31c
taskqueue_thread_loop() at taskqueue_thread_loop+0xb7
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff800009cd30, rbp = 0 ---

Tracing command kernel pid 0 tid 64022 td 0xffffff0002b8aba0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
_sleep() at _sleep+0x31c
taskqueue_thread_loop() at taskqueue_thread_loop+0xb7
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000092d30, rbp = 0 ---

Tracing command kernel pid 0 tid 64021 td 0xffffff0002b8c000
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
msleep_spin() at msleep_spin+0x191
taskqueue_thread_loop() at taskqueue_thread_loop+0x67
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff800008dd30, rbp = 0 ---

Tracing command kernel pid 0 tid 64020 td 0xffffff0002b8c3e0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
msleep_spin() at msleep_spin+0x191
taskqueue_thread_loop() at taskqueue_thread_loop+0x67
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000088d30, rbp = 0 ---

Tracing command kernel pid 0 tid 64019 td 0xffffff00029f9ba0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
msleep_spin() at msleep_spin+0x191
taskqueue_thread_loop() at taskqueue_thread_loop+0x67
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000083d30, rbp = 0 ---

Tracing command kernel pid 0 tid 64012 td 0xffffff00029e93e0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_wait() at sleepq_wait+0x42
_sleep() at _sleep+0x31c
taskqueue_thread_loop() at taskqueue_thread_loop+0xb7
fork_exit() at fork_exit+0x118
fork_trampoline() at fork_trampoline+0xe
--- trap 0, rip = 0, rsp = 0xffffff8000060d30, rbp = 0 ---

Tracing command kernel pid 0 tid 64001 td 0xffffffff811734a0
sched_switch() at sched_switch+0x100
mi_switch() at mi_switch+0x16f
sleepq_timedwait() at sleepq_timedwait+0x42
_sleep() at _sleep+0x308
scheduler() at scheduler+0x343
mi_startup() at mi_startup+0x59
btext() at btext+0x2c

db> show lockedvnods
Locked vnodes
db>

Any help would be appreciated.

jimp

Are you using the Dell's onboard Broadcom NICs or something else?

innosys

@jimp:

Are you using the Dell's onboard Broadcom NICs or something else?

Using both the onboard nics, a quadport Intel and then a dedicated Intel nic just for CARP.

I'm keeping the second firewall down incase of needing more output from bt, so pulled the dmesg output from the primary firewall (machines are identical):

Onboard for WAN and first segment:
bce0: <broadcom netxtreme="" ii="" bcm5716="" 1000base-t="" (c0)="">mem 0xd8000000-0xd9ffffff irq 16 at device 0.0 on pci2
bce1: <broadcom netxtreme="" ii="" bcm5716="" 1000base-t="" (c0)="">mem 0xda000000-0xdbffffff irq 17 at device 0.1 on pci2

Additional network segments on a Dell Intel Quad Port Gigabit PCIe NIC PN: H095P
igb0: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 2.1.7="">port 0xfcc0-0xfcdf mem 0xddfc0000-0xddfdffff,0xde000000-0xde3fffff,0xddfb8000-0xddfbbfff irq 18 at device 0.0 on pci6
igb1: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 2.1.7="">port 0xfce0-0xfcff mem 0xddfe0000-0xddffffff,0xde400000-0xde7fffff,0xddfbc000-0xddfbffff irq 19 at device 0.1 on pci6
igb2: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 2.1.7="">port 0xecc0-0xecdf mem 0xdd3c0000-0xdd3dffff,0xdd400000-0xdd7fffff,0xdd3b8000-0xdd3bbfff irq 16 at device 0.0 on pci7
igb3: <intel(r) 1000="" pro="" network="" connection="" version="" -="" 2.1.7="">port 0xece0-0xecff mem 0xdd3e0000-0xdd3fffff,0xdd800000-0xddbfffff,0xdd3bc000-0xdd3bffff irq 17 at device 0.1 on pci7

Dedicated CARP interface using a Intel 82574L Gigabit Ethernet Controller:
em0: <intel(r) 1000="" pro="" network="" connection="" 7.2.3="">port 0xdce0-0xdcff mem 0xde960000-0xde97ffff,0xde980000-0xde9fffff,0xde95c000-0xde95ffff irq 16 at device 0.0 on pci8

Thanks.</intel(r)></intel(r)></intel(r)></intel(r)></intel(r)></broadcom></broadcom>

cmb

That seems like the race condition in CARP deletion, Ermal made some related commits today that may help there.

innosys

Updated to 2.0-RC2 (amd64) built on Wed May 18 01:40:14 EDT 2011 on both firewalls.

I'll monitor the situation and give an update when I know more.

Thanks.

innosys

It has been a week without incident. I believe the changes Ermal made seems to have resolved in the issue.

I'll keep on monitoring the issue and will report if there are any changes.

Thanks.