Crash report 2.5.0-RC

yon 0

2.5.0-RC (amd64)
built on Tue Feb 16 03:07:30 EST 2021
FreeBSD 12.2-STABLE

Crash report begins.  Anonymous machine information:

amd64
12.2-STABLE
FreeBSD 12.2-STABLE d48fb226319(devel-12) pfSense

Crash report details:

No PHP errors found.

Filename: /var/crash/info.0
Dump header from device: /dev/label/swap0
  Architecture: amd64
  Architecture Version: 4
  Dump Length: 157696
  Blocksize: 512
  Compression: none
  Dumptime: Tue Feb 16 12:55:15 2021
  Hostname: face.xiaoyu.net
  Magic: FreeBSD Text Dump
  Version String: FreeBSD 12.2-STABLE d48fb226319(devel-12) pfSense
  Panic String: page fault
  Dump Parity: 3488812088
  Bounds: 0
  Dump Status: good

Filename: /var/crash/textdump.tar.0
ddb.txt06000014000014012740263  7071 ustarrootwheeldb:0:kdb.enter.default>  run lockinfo
db:1:lockinfo> show locks
No such command; use "help" to list available commands
db:1:lockinfo>  show alllocks
No such command; use "help" to list available commands
db:1:lockinfo>  show lockedvnods
Locked vnodes
db:0:kdb.enter.default>  show pcpu
cpuid        = 3
dynamic pcpu = 0xfffffe0080bf2380
curthread    = 0xfffff800045b1740: pid 12 tid 100042 "swi1: netisr 3"
curpcb       = 0xfffff800045b1ce0
fpcurthread  = none
idlethread   = 0xfffff800045dd740: tid 100006 "idle: cpu3"
curpmap      = 0xffffffff8368d5a8
tssp         = 0xffffffff83717758
commontssp   = 0xffffffff83717758
rsp0         = 0xfffffe0000502c40
kcr3         = 0xffffffffffffffff
ucr3         = 0xffffffffffffffff
scr3         = 0x0
gs32p        = 0xffffffff8371df70
ldt          = 0xffffffff8371dfb0
tss          = 0xffffffff8371dfa0
tlb gen      = 5196
curvnet      = 0xfffff8000408aa80
db:0:kdb.enter.default>  bt
Tracing pid 12 tid 100042 td 0xfffff800045b1740
kdb_enter() at kdb_enter+0x37/frame 0xfffffe0000501f90
vpanic() at vpanic+0x197/frame 0xfffffe0000501fe0
panic() at panic+0x43/frame 0xfffffe0000502040
trap_fatal() at trap_fatal+0x391/frame 0xfffffe00005020a0
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00005020f0
trap() at trap+0x286/frame 0xfffffe0000502200
calltrap() at calltrap+0x8/frame 0xfffffe0000502200
--- trap 0xc, rip = 0xffffffff80d7e767, rsp = 0xfffffe00005022d0, rbp = 0xfffffe0000502350 ---
__mtx_lock_sleep() at __mtx_lock_sleep+0xd7/frame 0xfffffe0000502350

100027                   D       -       0xfffff800045cd900  [softirq_2]
100028                   D       -       0xfffff800045cd700  [softirq_3]
100029                   D       -       0xfffff800045cd500  [if_config_tqg_0]
100030                   D       -       0xfffff800045cd300  [if_io_tqg_0]
100031                   D       -       0xfffff800045cd100  [if_io_tqg_1]
100032                   D       -       0xfffff800045cee00  [if_io_tqg_2]
100033                   D       -       0xfffff800045cec00  [if_io_tqg_3]
100047                   D       -       0xfffff800045cf200  [firmware taskq]
100052                   D       -       0xfffff8000457fc00  [crypto_0]
100053                   D       -       0xfffff8000457fc00  [crypto_1]
100054                   D       -       0xfffff8000457fc00  [crypto_2]
100055                   D       -       0xfffff8000457fc00  [crypto_3]
100118                   D       -       0xfffff800050b4000  [mca taskq]
100128                   D       -       0xfffff80020f61d00  [acpi_task_0]
100129                   D       -       0xfffff80020f61d00  [acpi_task_1]
100130                   D       -       0xfffff80020f61d00  [acpi_task_2]
100131                   D       -       0xfffff80004580900  [CAM taskq]
100133                   D       -       0xfffff80020f62600  [mlx4]
db:0:kdb.enter.default>  alltrace

Tracing command nice pid 26634 tid 100345 td 0xfffff8011ef3a000 (CPU 2)
cpustop_handler() at cpustop_handler+0x27/frame 0xfffffe0003f4be00
ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe0003f4be10
trap() at trap+0x44/frame 0xfffffe0003f4bf20
nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe0003f4bf20
--- trap 0x13, rip = 0xffffffff813725b1, rsp = 0xfffffe009a2d6620, rbp = 0xfffffe009a2d6690 ---
pmap_enter_quick_locked() at pmap_enter_quick_locked+0x231/frame 0xfffffe009a2d6690
pmap_enter_object() at pmap_enter_object+0x218/frame 0xfffffe009a2d6700
vm_map_pmap_enter() at vm_map_pmap_enter+0x2c9/frame 0xfffffe009a2d6780
vm_map_insert() at vm_map_insert+0x43a/frame 0xfffffe009a2d6810
vm_map_fixed() at vm_map_fixed+0x138/frame 0xfffffe009a2d6870
vm_mmap_object() at vm_mmap_object+0x1c2/frame 0xfffffe009a2d68e0
vn_mmap() at vn_mmap+0x14f/frame 0xfffffe009a2d6960
kern_mmap_fpcheck() at kern_mmap_fpcheck+0x4b0/frame 0xfffffe009a2d6a20
sys_mmap() at sys_mmap+0x23/frame 0xfffffe009a2d6a40
amd64_syscall() at amd64_syscall+0x387/frame 0xfffffe009a2d6b70
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe009a2d6b70
--- syscall (477, FreeBSD ELF64, sys_mmap), rip = 0x800221b5a, rsp = 0x7fffffffd178, rbp = 0x7fffffffd2f0 ---

Tracing command netstat pid 26245 tid 100179 td 0xfffff80020f02740
sched_switch() at sched_switch+0x630/frame 0xfffffe00845e1af0
mi_switch() at mi_switch+0xd4/frame 0xfffffe00845e1b20
critical_exit_preempt() at critical_exit_preempt+0x55/frame 0xfffffe00845e1b40
ipi_bitmap_handler() at ipi_bitmap_handler+0x8f/frame 0xfffffe00845e1b70
Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0xb3/frame 0xfffffe00845e1b70
--- interrupt, rip = 0x8002a309e, rsp = 0x7fffffffe090, rbp = 0x7fffffffe1b0 ---

pcib1: <ACPI PCI-PCI bridge> irq 18 at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
igb0: <Intel(R) PRO/1000 PCI-Express Network Driver> mem 0xff480000-0xff4fffff,0xff504000-0xff507fff irq 18 at device 0.0 on pci1
igb0: Using 1024 TX descriptors and 1024 RX descriptors
igb0: Using 4 RX queues 4 TX queues
igb0: Using MSI-X interrupts with 5 vectors
<6>igb0: Ethernet address: a0:36:9f:83:99:40
<6>igb0: netmap queues/slots: TX 4/1024, RX 4/1024
igb1: <Intel(R) PRO/1000 PCI-Express Network Driver> mem 0xff380000-0xff3fffff,0xff500000-0xff503fff irq 19 at device 0.1 on pci1
igb1: Using 1024 TX descriptors and 1024 RX descriptors
igb1: Using 4 RX queues 4 TX queues
igb1: Using MSI-X interrupts with 5 vectors
<6>igb1: Ethernet address: a0:36:9f:83:99:41
<6>igb1: netmap queues/slots: TX 4/1024, RX 4/1024
[textdump.tar.0](/assets/uploads/files/1613480696829-textdump.tar.0) <6>wg1: link state changed to UP

<118>netstat: sysctl: net.route.0.0.dump.0: Cannot allocate memory
<118>netstat: sysctl: net.route.0.0.dump.0: Cannot allocate memory
<118>done.
<118>Setting up DNSs...
<118>Synchronizing user settings...done.
<118>Starting webConfigurator...done.
<118>Configuring CRON...done.
<118>Starting NTP Server...done.
<118>Starting DHCP service...done.
<118>Starting DHCPv6 service...done.
<6>ovpnc12: link state changed to UP
<6>ovpnc9: link state changed to UP
<6>ovpnc9: link state changed to DOWN
<118>Configuring firewall..
<6>ovpnc7: link state changed to UP
<118>....done.
<118>Configuring WireGuard Tunnels... 
<6>wg1: link state changed to DOWN


Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 13
fault virtual address	= 0x410
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff80d7e767
stack pointer	        = 0x0:0xfffffe00005022d0
frame pointer	        = 0x0:0xfffffe0000502350
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 12 (swi1: netisr 3)
trap number		= 12
panic: page fault
cpuid = 3
time = 1613480115
KDB: enter: panic
panic.txt0600001214012740263  7130 ustarrootwheelpage faultversion.txt0600006214012740263  7530 ustarrootwheelFreeBSD 12.2-STABLE d48fb226319(devel-12) pfSense

yon 0

textdump.tar.0

info.0

jimp

cpustop_handler() at cpustop_handler+0x27/frame 0xfffffe0003f4be00
ipi_nmi_handler() at ipi_nmi_handler+0x39/frame 0xfffffe0003f4be10
trap() at trap+0x44/frame 0xfffffe0003f4bf20
nmi_calltrap() at nmi_calltrap+0x8/frame 0xfffffe0003f4bf20

Usually NMI (Non-maskable interrupt) calls indicate hardware problems. It's not always 100% but that's the top suspect.

yon 0

@jimp

Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 11
fault virtual address	= 0x410
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff80d7e767
stack pointer	        = 0x0:0xfffffe00004cb090
frame pointer	        = 0x0:0xfffffe00004cb110
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 0 (if_io_tqg_1)
trap number		= 12
panic: page fault
cpuid = 1
time = 1613830597
KDB: enter: panic
panic.txt0600001214014214705  7127 ustarrootwheelpage faultversion.txt0600006214014214705  7527 ustarrootwheelFreeBSD 12.2-STABLE d48fb226319(devel-12) pfSense

yon 0

https://forum.netgate.com/topic/160491/cannot-allocate-memory/3

yon 0

Feb 21 18:16:58	kernel		matchaddr failed
Feb 21 18:16:57	kernel		matchaddr failed
Feb 21 18:16:56	kernel		matchaddr failed

MikeV7896

I don't believe those "matchaddr failed" entries have to do with any crashes. I think they're related to WireGuard though, as they appeared in my logs a lot as I was getting WireGuard set up and was frequently changing settings to get things working.

yon 0

@virgiliomi

I think multiple factors may cause the crash, I think it involves FRR and wireguard.

i have report to wireguard admin.