(Solved) IPv6 Kernel Panic

chpalmer

2.4.0.b.20170804.1136Looks like a recent change to IPv6 caused a kernel panic

chpalmer

Was able to catch this. Had to disable my 6to4 tunnel on WAN.

Tracing command kernel pid 0 tid 100626 td 0xfffff80006833a00
sched_switch() at sched_switch+0x6cb/frame 0xfffffe011d1d9a50
mi_switch() at mi_switch+0xd2/frame 0xfffffe011d1d9a80
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe011d1d9ab0
_sleep() at _sleep+0x2a1/frame 0xfffffe011d1d9b40
taskqueue_thread_loop() at taskqueue_thread_loop+0x141/frame 0xfffffe011d1d9b70
fork_exit() at fork_exit+0x85/frame 0xfffffe011d1d9bb0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe011d1d9bb0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Tracing command kernel pid 0 tid 100627 td 0xfffff8000682ba00
sched_switch() at sched_switch+0x6cb/frame 0xfffffe011d1dea50
mi_switch() at mi_switch+0xd2/frame 0xfffffe011d1dea80
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe011d1deab0
_sleep() at _sleep+0x2a1/frame 0xfffffe011d1deb40
taskqueue_thread_loop() at taskqueue_thread_loop+0x141/frame 0xfffffe011d1deb70
fork_exit() at fork_exit+0x85/frame 0xfffffe011d1debb0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe011d1debb0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Tracing command kernel pid 0 tid 100628 td 0xfffff8000682b500
sched_switch() at sched_switch+0x6cb/frame 0xfffffe011d1e3a50
mi_switch() at mi_switch+0xd2/frame 0xfffffe011d1e3a80
sleepq_wait() at sleepq_wait+0x3a/frame 0xfffffe011d1e3ab0
_sleep() at _sleep+0x2a1/frame 0xfffffe011d1e3b40
taskqueue_thread_loop() at taskqueue_thread_loop+0x141/frame 0xfffffe011d1e3b70
fork_exit() at fork_exit+0x85/frame 0xfffffe011d1e3bb0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe011d1e3bb0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
db:0:kdb.enter.default>  capture off
db:0:kdb.enter.default>  textdump dump
Textdump complete.
db:0:kdb.enter.default>  reset
cpu_reset: Restarting BSP
cpu_reset_proxy: Stopped CPU 2

chpalmer

More testing today-

Disable all LAN instances of IPv6 (all were set to track interface)

Set WAN back to "6to4 Tunnel"

WAN grabs IPv6 address..

Set LAN to "Track Interface"

then-

Tracing command getty pid 50415 tid 100727 td 0xfffff800236d1a00
sched_switch() at sched_switch+0x6cb/frame 0xfffffe011e1b8760
mi_switch() at mi_switch+0xd2/frame 0xfffffe011e1b8790
sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011e1b8810
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011e1b8840
_cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe011e1b8890
tty_wait() at tty_wait+0x1c/frame 0xfffffe011e1b88b0
ttydisc_read() at ttydisc_read+0x30a/frame 0xfffffe011e1b8910
ttydev_read() at ttydev_read+0x34/frame 0xfffffe011e1b8940
devfs_read_f() at devfs_read_f+0xe8/frame 0xfffffe011e1b89a0
dofileread() at dofileread+0x98/frame 0xfffffe011e1b89f0
kern_readv() at kern_readv+0x68/frame 0xfffffe011e1b8a40
sys_read() at sys_read+0x84/frame 0xfffffe011e1b8a90
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011e1b8bb0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011e1b8bb0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b811ea, rsp = 0x7fffffffec78, rbp = 0x7fffffffecc0 ---

Tracing command getty pid 50270 tid 100726 td 0xfffff800236f1000
sched_switch() at sched_switch+0x6cb/frame 0xfffffe011e1b3760
mi_switch() at mi_switch+0xd2/frame 0xfffffe011e1b3790
sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011e1b3810
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011e1b3840
_cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe011e1b3890
tty_wait() at tty_wait+0x1c/frame 0xfffffe011e1b38b0
ttydisc_read() at ttydisc_read+0x30a/frame 0xfffffe011e1b3910
ttydev_read() at ttydev_read+0x34/frame 0xfffffe011e1b3940
devfs_read_f() at devfs_read_f+0xe8/frame 0xfffffe011e1b39a0
dofileread() at dofileread+0x98/frame 0xfffffe011e1b39f0
kern_readv() at kern_readv+0x68/frame 0xfffffe011e1b3a40
sys_read() at sys_read+0x84/frame 0xfffffe011e1b3a90
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011e1b3bb0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011e1b3bb0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b811ea, rsp = 0x7fffffffec78, rbp = 0x7fffffffecc0 ---

Tracing command getty pid 49939 tid 100647 td 0xfffff80006a44000
sched_switch() at sched_switch+0x6cb/frame 0xfffffe011d445760
mi_switch() at mi_switch+0xd2/frame 0xfffffe011d445790
sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011d445810
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011d445840
_cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe011d445890
tty_wait() at tty_wait+0x1c/frame 0xfffffe011d4458b0
ttydisc_read() at ttydisc_read+0x30a/frame 0xfffffe011d445910
ttydev_read() at ttydev_read+0x34/frame 0xfffffe011d445940
devfs_read_f() at devfs_read_f+0xe8/frame 0xfffffe011d4459a0
dofileread() at dofileread+0x98/frame 0xfffffe011d4459f0
kern_readv() at kern_readv+0x68/frame 0xfffffe011d445a40
sys_read() at sys_read+0x84/frame 0xfffffe011d445a90
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011d445bb0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011d445bb0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b811ea, rsp = 0x7fffffffec78, rbp = 0x7fffffffecc0 ---

Tracing command getty pid 49736 tid 100623 td 0xfffff80006490a00
sched_switch() at sched_switch+0x6cb/frame 0xfffffe011d1cf760
mi_switch() at mi_switch+0xd2/frame 0xfffffe011d1cf790
sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011d1cf810
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011d1cf840
_cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe011d1cf890
tty_wait() at tty_wait+0x1c/frame 0xfffffe011d1cf8b0
ttydisc_read() at ttydisc_read+0x30a/frame 0xfffffe011d1cf910
ttydev_read() at ttydev_read+0x34/frame 0xfffffe011d1cf940
devfs_read_f() at devfs_read_f+0xe8/frame 0xfffffe011d1cf9a0
dofileread() at dofileread+0x98/frame 0xfffffe011d1cf9f0
kern_readv() at kern_readv+0x68/frame 0xfffffe011d1cfa40
sys_read() at sys_read+0x84/frame 0xfffffe011d1cfa90
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011d1cfbb0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011d1cfbb0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b811ea, rsp = 0x7fffffffec78, rbp = 0x7fffffffecc0 ---

Tracing command getty pid 49473 tid 100622 td 0xfffff8000649f000
sched_switch() at sched_switch+0x6cb/frame 0xfffffe011d1ca760
mi_switch() at mi_switch+0xd2/frame 0xfffffe011d1ca790
sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011d1ca810
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011d1ca840
_cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe011d1ca890
tty_wait() at tty_wait+0x1c/frame 0xfffffe011d1ca8b0
ttydisc_read() at ttydisc_read+0x30a/frame 0xfffffe011d1ca910
ttydev_read() at ttydev_read+0x34/frame 0xfffffe011d1ca940
devfs_read_f() at devfs_read_f+0xe8/frame 0xfffffe011d1ca9a0
dofileread() at dofileread+0x98/frame 0xfffffe011d1ca9f0
kern_readv() at kern_readv+0x68/frame 0xfffffe011d1caa40
sys_read() at sys_read+0x84/frame 0xfffffe011d1caa90
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011d1cabb0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011d1cabb0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b811ea, rsp = 0x7fffffffec78, rbp = 0x7fffffffecc0 ---

Tracing command getty pid 49317 tid 100695 td 0xfffff8011c81f500
sched_switch() at sched_switch+0x6cb/frame 0xfffffe011e11d760
mi_switch() at mi_switch+0xd2/frame 0xfffffe011e11d790
sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011e11d810
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011e11d840
_cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe011e11d890
tty_wait() at tty_wait+0x1c/frame 0xfffffe011e11d8b0
ttydisc_read() at ttydisc_read+0x30a/frame 0xfffffe011e11d910
ttydev_read() at ttydev_read+0x34/frame 0xfffffe011e11d940
devfs_read_f() at devfs_read_f+0xe8/frame 0xfffffe011e11d9a0
dofileread() at dofileread+0x98/frame 0xfffffe011e11d9f0
kern_readv() at kern_readv+0x68/frame 0xfffffe011e11da40
sys_read() at sys_read+0x84/frame 0xfffffe011e11da90
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011e11dbb0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011e11dbb0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b811ea, rsp = 0x7fffffffec78, rbp = 0x7fffffffecc0 ---

Tracing command login pid 49227 tid 100680 td 0xfffff80006ed6500
sched_switch() at sched_switch+0x6cb/frame 0xfffffe011e0de660
mi_switch() at mi_switch+0xd2/frame 0xfffffe011e0de690
sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011e0de710
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011e0de740
_sleep() at _sleep+0x294/frame 0xfffffe011e0de7d0
kern_wait6() at kern_wait6+0x2d3/frame 0xfffffe011e0de8a0
sys_wait4() at sys_wait4+0x73/frame 0xfffffe011e0dea90
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011e0debb0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011e0debb0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x800d2475a, rsp = 0x7fffffffebe8, rbp = 0x7fffffffed40 ---

Tracing command minicron pid 70030 tid 100682 td 0xfffff8006b0bba00
sched_switch() at sched_switch+0x6cb/frame 0xfffffe011e0e8880
mi_switch() at mi_switch+0xd2/frame 0xfffffe011e0e88b0
sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011e0e8930
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe011e0e8960
_sleep() at _sleep+0x26f/frame 0xfffffe011e0e89f0
kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe011e0e8a50
sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe011e0e8a90
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011e0e8bb0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011e0e8bb0
--- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x8009008fa, rsp = 0x7fffffffeb88, rbp = 0x7fffffffebf0 ---

Tracing command minicron pid 69805 tid 100679 td 0xfffff80006ed6a00
sched_switch() at sched_switch+0x6cb/frame 0xfffffe011e0d9660
mi_switch() at mi_switch+0xd2/frame 0xfffffe011e0d9690
sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011e0d9710
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011e0d9740
_sleep() at _sleep+0x294/frame 0xfffffe011e0d97d0
kern_wait6() at kern_wait6+0x2d3/frame 0xfffffe011e0d98a0
sys_wait4() at sys_wait4+0x73/frame 0xfffffe011e0d9a90
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011e0d9bb0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011e0d9bb0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x80090075a, rsp = 0x7fffffffebf8, rbp = 0x7fffffffec30 ---

Tracing command minicron pid 69491 tid 100621 td 0xfffff8000649f500
sched_switch() at sched_switch+0x6cb/frame 0xfffffe011d1c5880
mi_switch() at mi_switch+0xd2/frame 0xfffffe011d1c58b0
sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011d1c5930
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe011d1c5960
_sleep() at _sleep+0x26f/frame 0xfffffe011d1c59f0
kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe011d1c5a50
sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe011d1c5a90
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011d1c5bb0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011d1c5bb0
--- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x8009008fa, rsp = 0x7fffffffeb98, rbp = 0x7fffffffec00 ---

Tracing command minicron pid 69086 tid 100721 td 0xfffff800236f2a00
sched_switch() at sched_switch+0x6cb/frame 0xfffffe011e19a660
mi_switch() at mi_switch+0xd2/frame 0xfffffe011e19a690
sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011e19a710
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011e19a740
_sleep() at _sleep+0x294/frame 0xfffffe011e19a7d0
kern_wait6() at kern_wait6+0x2d3/frame 0xfffffe011e19a8a0
sys_wait4() at sys_wait4+0x73/frame 0xfffffe011e19aa90
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011e19abb0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011e19abb0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x80090075a, rsp = 0x7fffffffec08, rbp = 0x7fffffffec40 ---

Tracing command minicron pid 68807 tid 100633 td 0xfffff80006a44a00
sched_switch() at sched_switch+0x6cb/frame 0xfffffe011d1fc880
mi_switch() at mi_switch+0xd2/frame 0xfffffe011d1fc8b0
sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011d1fc930
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe011d1fc960
_sleep() at _sleep+0x26f/frame 0xfffffe011d1fc9f0
kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe011d1fca50
sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe011d1fca90
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011d1fcbb0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011d1fcbb0
--- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x8009008fa, rsp = 0x7fffffffebb8, rbp = 0x7fffffffec20 ---

Tracing command minicron pid 68205 tid 100620 td 0xfffff80006a46a00
sched_switch() at sched_switch+0x6cb/frame 0xfffffe011d1c0660
mi_switch() at mi_switch+0xd2/frame 0xfffffe011d1c0690
sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011d1c0710
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011d1c0740
_sleep() at _sleep+0x294/frame 0xfffffe011d1c07d0
kern_wait6() at kern_wait6+0x2d3/frame 0xfffffe011d1c08a0
sys_wait4() at sys_wait4+0x73/frame 0xfffffe011d1c0a90
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011d1c0bb0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011d1c0bb0
--- syscall (7, FreeBSD ELF64, sys_wait4), rip = 0x80090075a, rsp = 0x7fffffffec28, rbp = 0x7fffffffec60 ---

Tracing command sshlockout_pf pid 25888 tid 100645 td 0xfffff80006c7c000
sched_switch() at sched_switch+0x6cb/frame 0xfffffe011d43b7d0
mi_switch() at mi_switch+0xd2/frame 0xfffffe011d43b800
sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011d43b880
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011d43b8b0
_sleep() at _sleep+0x294/frame 0xfffffe011d43b940
pipe_read() at pipe_read+0x36d/frame 0xfffffe011d43b9a0
dofileread() at dofileread+0x98/frame 0xfffffe011d43b9f0
kern_readv() at kern_readv+0x68/frame 0xfffffe011d43ba40
sys_read() at sys_read+0x84/frame 0xfffffe011d43ba90
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011d43bbb0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011d43bbb0
--- syscall (3, FreeBSD ELF64, sys_read), rip = 0x800b911ea, rsp = 0x7fffffffe7d8, rbp = 0x7fffffffe7f0 ---

Tracing command sshlockout_pf pid 25888 tid 100719 td 0xfffff8013769c500
sched_switch() at sched_switch+0x6cb/frame 0xfffffe011e190880
mi_switch() at mi_switch+0xd2/frame 0xfffffe011e1908b0
sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011e190930
sleepq_timedwait_sig() at sleepq_timedwait_sig+0x10/frame 0xfffffe011e190960
_sleep() at _sleep+0x26f/frame 0xfffffe011e1909f0
kern_nanosleep() at kern_nanosleep+0x10e/frame 0xfffffe011e190a50
sys_nanosleep() at sys_nanosleep+0x53/frame 0xfffffe011e190a90
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011e190bb0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011e190bb0
--- syscall (240, FreeBSD ELF64, sys_nanosleep), rip = 0x800b288fa, rsp = 0x7fffdfffdf48, rbp = 0x7fffdfffdf70 ---

Tracing command syslogd pid 7601 tid 100651 td 0xfffff80006ce2000
sched_switch() at sched_switch+0x6cb/frame 0xfffffe011d4596e0
mi_switch() at mi_switch+0xd2/frame 0xfffffe011d459710
sleepq_catch_signals() at sleepq_catch_signals+0xb7/frame 0xfffffe011d459790
sleepq_wait_sig() at sleepq_wait_sig+0xf/frame 0xfffffe011d4597c0
_cv_wait_sig() at _cv_wait_sig+0x1b2/frame 0xfffffe011d459810
seltdwait() at seltdwait+0x10f/frame 0xfffffe011d459860
kern_select() at kern_select+0x979/frame 0xfffffe011d459a50
sys_select() at sys_select+0x54/frame 0xfffffe011d459a90
amd64_syscall() at amd64_syscall+0x4ce/frame 0xfffffe011d459bb0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe011d459bb0
--- syscall (93, FreeBSD ELF64, sys_select), rip = 0x800b851ca, rsp = 0x7fffffffdd78, rbp = 0x7fffffffec20 ---

micropone

on any reboot or upgrade.. I have to dhcp 6 on wan to none… then apply then set back to dhcp6 apply...

Rexeros

I have same problems. Had to turn 6to4 IPV6 off :( Track interface on LAN = kernel panic, and reboot loop.

I wasnt using DHCPv6. PFsense work normally, and have IP6 on WAN. But each time u setup LAN option "Track interface" to WAN - pfsense crashes and cant boot anymore (need to restore config without ipv6)

jimp

There were some changes meant to fix 6RD a few days ago, it's possible they impacted this as well.

loos

The crash was fixed in the latest snapshot. Sorry for the breakage.

chpalmer

@loos:

The crash was fixed in the latest snapshot. Sorry for the breakage.

You know what they say.. Beta happens. :P :)

Thanks!

chpalmer

Just a heads up in case.. Didn't see it in know issues.

2.4 RC on board now.

https://redmine.pfsense.org/issues/7176 still an issue. But I am no longer seeing that log.

:)

jimp

What about on the most recent snapshot as of now?

chpalmer

@jimp:

What about on the most recent snapshot as of now?

Forgot to update this last night. As of

2.4.0-RC (amd64)
built on Wed Aug 23 05:47:05 CDT 2017
FreeBSD 11.0-RELEASE-p12

working now. :)

Thanks!