Crash Dump

mattk926 · May 31, 2024, 4:13 PM

First crash of a pfsense router after running pfsense for 10+ years Would like some help if anyone knows why it crashed. Attached the crashdumps.

Thanks in advance

info (1).0 textdump.tar.0

stephenw10 · May 31, 2024, 5:44 PM

Backtrace:

db:0:kdb.enter.default>  bt
Tracing pid 0 tid 983791 td 0xfffffe01033623a0
kdb_enter() at kdb_enter+0x32/frame 0xfffffe0102a9d6d0
vpanic() at vpanic+0x163/frame 0xfffffe0102a9d800
panic() at panic+0x43/frame 0xfffffe0102a9d860
trap_fatal() at trap_fatal+0x40c/frame 0xfffffe0102a9d8c0
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe0102a9d920
calltrap() at calltrap+0x8/frame 0xfffffe0102a9d920
--- trap 0xc, rip = 0xffffffff80e4dce6, rsp = 0xfffffe0102a9d9f0, rbp = 0xfffffe0102a9da20 ---
rn_match() at rn_match+0x36/frame 0xfffffe0102a9da20
rn_lookup() at rn_lookup+0x43/frame 0xfffffe0102a9da50
add_route_flags() at add_route_flags+0x49/frame 0xfffffe0102a9db10
rib_add_route_px() at rib_add_route_px+0x12f/frame 0xfffffe0102a9dbb0
rtnl_handle_newroute() at rtnl_handle_newroute+0x627/frame 0xfffffe0102a9dca0
rtnl_handle_message() at rtnl_handle_message+0x132/frame 0xfffffe0102a9dd00
nl_taskqueue_handler() at nl_taskqueue_handler+0x79b/frame 0xfffffe0102a9de40
taskqueue_run_locked() at taskqueue_run_locked+0x182/frame 0xfffffe0102a9dec0
taskqueue_thread_loop() at taskqueue_thread_loop+0xc2/frame 0xfffffe0102a9def0
fork_exit() at fork_exit+0x7f/frame 0xfffffe0102a9df30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0102a9df30
--- trap 0xc, rip = 0x8256893ea, rsp = 0x82a158ca8, rbp = 0x82a158cc0 ---

Panic:

Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 02
fault virtual address	= 0x10
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff80e4dce6
stack pointer	        = 0x28:0xfffffe0102a9d9f0
frame pointer	        = 0x28:0xfffffe0102a9da20
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 0 (netlink_socket (PID)
rdi: fffff8011be116f8 rsi: 0000000000000000 rdx: fffff800080ce480
rcx: 0000000000000000  r8: 00000000ffffffff  r9: 00000000ffffffff
rax: 0000000000000010 rbx: fffff8011be116f8 rbp: fffffe0102a9da20
r10: 0000000000000000 r11: 00000000f36d6f74 r12: fffffe0102a9dc00
r13: fffff8011be11690 r14: fffff800080ce400 r15: fffff8011b80a2e0
trap number		= 12
panic: page fault
cpuid = 1
time = 1717170123
KDB: enter: panic

So some routing function.

From what I can see in the logs I'd assume that's WireGuard adding something when it connects. Unless you made some other change manually?

Steve

mattk926 · May 31, 2024, 6:18 PM

@stephenw10 No manual changes were made at the time of the crash. Just seemed to randomly crash today.

stephenw10 · May 31, 2024, 6:21 PM

First time it's crashed?

Recently upgraded? Recently installed Wireguard? Some other change?

mattk926 · May 31, 2024, 6:43 PM

@stephenw10 Nope, been running fine for the past few months. Wireguard has been running for the past few months without issue. no recent changes in the past few weeks.

stephenw10 · May 31, 2024, 8:31 PM

Hmm, odd. And this is the first time you've seen it at all?

mattk926 · Jun 1, 2024, 12:45 PM

@stephenw10 Correct first time the router has crashed have had this specific hardware running for 6+ months now this is the first crash.

stephenw10 · Jun 1, 2024, 2:01 PM

What other packages do you have installed?

Do you have static routes configured?

mattk926 · Jun 2, 2024, 1:22 PM

@stephenw10

acme
apcupsd
Filer
iperf
nmap
ntopng
openvpn-client-export
pfBlockerNG-devel
Service_Watchdog
System_Patches
Telegraf
WireGuard

Just 2 static routes for the Site to Site VPN's to two other sites over wireguard.

stephenw10 · Jun 2, 2024, 1:50 PM

Was there anything in the system, routing or gateway logs at that time?

Really I have to think that one of your gateway came up and the routes via it was added. Only WireGuard seems likely to have done that.

I haven't seen anything else hit it though.

mattk926 · Jun 4, 2024, 12:52 PM

@stephenw10
Gateway logs around the time of the crash dump looks like there may have been issues with internet circuit around the crash time shown by the gateway logs:

2024-05-31 10:42:01.607620-05:00 dpinger 72401 S2S_P x.x.x.x: Alarm latency 36343us stddev 14736us loss 22%
2024-05-31 10:42:01.601678-05:00 dpinger 71506 S2S_Ben x.x.x.x: Alarm latency 45090us stddev 1231us loss 22%
2024-05-31 10:41:54.124443-05:00 dpinger 70655 WAN_DHCP x.x.x.x: Alarm latency 8227us stddev 5511us loss 9%

General log:
2024-05-31 10:42:03.580195-05:00 kernel - r13: fffff8011be11690 r14: fffff800080ce400 r15: fffff8011b80a2e0
2024-05-31 10:42:03.580187-05:00 kernel - r10: 0000000000000000 r11: 00000000f36d6f74 r12: fffffe0102a9dc00
2024-05-31 10:42:03.580167-05:00 kernel - rax: 0000000000000010 rbx: fffff8011be116f8 rbp: fffffe0102a9da20
2024-05-31 10:42:03.374238-05:00 kernel - rcx: 0000000000000000 r8: 00000000ffffffff r9: 00000000ffffffff
2024-05-31 10:42:03.374231-05:00 kernel - rdi: fffff8011be116f8 rsi: 0000000000000000 rdx: fffff800080ce480
2024-05-31 10:42:03.374224-05:00 kernel - current process = 0 (netlink_socket (PID)
2024-05-31 10:42:03.374201-05:00 kernel - interrupt enabled, resume, IOPL = 0
2024-05-31 10:42:03.169637-05:00 kernel - processor eflags =
2024-05-31 10:42:03.169596-05:00 kernel - = DPL 0, pres 1, long 1, def32 0, gran 1
2024-05-31 10:42:03.083834-05:00 kernel - code segment = base 0x0, limit 0xfffff, type 0x1b
2024-05-31 10:42:03.083821-05:00 kernel - frame pointer = 0x28:0xfffffe0102a9da20
2024-05-31 10:42:02.964977-05:00 kernel - stack pointer = 0x28:0xfffffe0102a9d9f0
2024-05-31 10:42:02.960547-05:00 kernel - instruction pointer = 0x20:0xffffffff80e4dce6
2024-05-31 10:42:02.960541-05:00 kernel - fault code = supervisor read data, page not present
2024-05-31 10:42:02.960536-05:00 kernel - fault virtual address = 0x10
2024-05-31 10:42:02.960530-05:00 kernel - cpuid = 1; apic id = 02
2024-05-31 10:42:02.960523-05:00 kernel - Fatal trap 12: page fault while in kernel mode
2024-05-31 10:42:02.725471-05:00 php-fpm 8851 /rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''
2024-05-31 10:42:02.720799-05:00 php-fpm 96531 /rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''
2024-05-31 10:42:01.620957-05:00 check_reload_status 446 Reloading filter
2024-05-31 10:42:01.620941-05:00 check_reload_status 446 Restarting OpenVPN tunnels/interfaces
2024-05-31 10:42:01.620909-05:00 check_reload_status 446 Restarting IPsec tunnels
2024-05-31 10:42:01.620843-05:00 check_reload_status 446 updating dyndns S2S_P
2024-05-31 10:42:01.619110-05:00 rc.gateway_alarm 19166 >>> Gateway alarm: S2S_P (Addr:x.x.x.x Alarm:1 RTT:36.343ms RTTsd:14.736ms Loss:22%)
2024-05-31 10:42:01.618167-05:00 check_reload_status 446 Reloading filter
2024-05-31 10:42:01.618120-05:00 check_reload_status 446 Restarting OpenVPN tunnels/interfaces
2024-05-31 10:42:01.618080-05:00 check_reload_status 446 Restarting IPsec tunnels
2024-05-31 10:42:01.618000-05:00 check_reload_status 446 updating dyndns S2S_Ben
2024-05-31 10:42:01.616331-05:00 rc.gateway_alarm 17657 >>> Gateway alarm: S2S_Ben (Addr:x.x.x.x Alarm:1 RTT:45.090ms RTTsd:1.231ms Loss:22%)
2024-05-31 10:40:00.067019-05:00 sshguard 13955 Now monitoring attacks.
2024-05-31 10:40:00.057723-05:00 sshguard 37501 Exiting on signal.

Nothing in the routing logs

stephenw10 · Jun 4, 2024, 1:12 PM

@mattk926 said in Crash Dump:

2024-05-31 10:42:02.960523-05:00 kernel - Fatal trap 12: page fault while in kernel mode
2024-05-31 10:42:02.725471-05:00 php-fpm 8851 /rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''
2024-05-31 10:42:02.720799-05:00 php-fpm 96531 /rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''

Hmm, seems likely is was OpenVPN trying to add or remove a route in reaction to the tunnels going down. I assume you have had other gateway incidents that didn't present like that though?

Do you have any logs covering a different gateway incident to compare?

mattk926 · Jun 4, 2024, 2:27 PM

@stephenw10 Correct circuit has had issues and loss somewhat regularly without crashing.

2024-04-30 10:49:19.924249-05:00 dpinger 59402 S2S_P x.x.x.x: Clear latency 23960us stddev 20870us loss 0%
2024-04-30 10:49:19.484859-05:00 dpinger 59039 S2S_Ben x.x.x.x: Clear latency 53499us stddev 19126us loss 0%
2024-04-30 10:49:19.433845-05:00 dpinger 58719 WAN_DHCP x.x.x.x: Clear latency 19486us stddev 30985us loss 0%
2024-04-30 10:48:27.087390-05:00 dpinger 59402 S2S_P x.x.x.x: Alarm latency 2687469us stddev 5045171us loss 5%
2024-04-30 10:48:26.958875-05:00 dpinger 58719 WAN_DHCP x.x.x.x: Alarm latency 2783072us stddev 5097642us loss 1%
2024-04-30 10:48:26.926042-05:00 dpinger 59039 S2S_Ben x.x.x.x: Alarm latency 2727521us stddev 5055855us loss 6%
2024-04-30 10:48:17.945783-05:00 dpinger 59402 S2S_P x.x.x.x: Alarm latency 2665026us stddev 5028638us loss 3%
2024-04-30 10:48:17.941593-05:00 dpinger 58719 WAN_DHCP x.x.x.x: Alarm latency 2781810us stddev 5098276us loss 1%
2024-04-30 10:48:17.827582-05:00 dpinger 59039 S2S_Ben x.x.x.x: Alarm latency 2705282us stddev 5038907us loss 3%
2024-04-30 10:48:13.924410-05:00 dpinger 59402 S2S_P x.x.x.x: Alarm latency 25751us stddev 19299us loss 21%
2024-04-30 10:48:13.791832-05:00 dpinger 59039 S2S_Ben x.x.x.x: Alarm latency 56397us stddev 20270us loss 21%
2024-04-30 10:48:05.841384-05:00 dpinger 58719 WAN_DHCP x.x.x.x: Alarm latency 17649us stddev 19535us loss 9%

2024-04-30T10:48:13.805388-05:00 pfSense.home.redacted.org rc.gateway_alarm 37988 - - >>> Gateway alarm: S2S_Ben (Addr:x.x.x.x Alarm:1 RTT:56.397ms RTTsd:20.270ms Loss:21%)
2024-04-30T10:48:13.806719-05:00 pfSense.home.redacted.org check_reload_status 446 - - updating dyndns S2S_Ben
2024-04-30T10:48:13.806771-05:00 pfSense.home.redacted.org check_reload_status 446 - - Restarting IPsec tunnels
2024-04-30T10:48:13.806787-05:00 pfSense.home.redacted.org check_reload_status 446 - - Restarting OpenVPN tunnels/interfaces
2024-04-30T10:48:13.806798-05:00 pfSense.home.redacted.org check_reload_status 446 - - Reloading filter
2024-04-30T10:48:13.937670-05:00 pfSense.home.redacted.org rc.gateway_alarm 40704 - - >>> Gateway alarm: S2S_P (Addr:x.x.x.x Alarm:1 RTT:25.751ms RTTsd:19.299ms Loss:21%)
2024-04-30T10:48:13.939015-05:00 pfSense.home.redacted.org check_reload_status 446 - - updating dyndns S2S_P
2024-04-30T10:48:13.939067-05:00 pfSense.home.redacted.org check_reload_status 446 - - Restarting IPsec tunnels
2024-04-30T10:48:13.939082-05:00 pfSense.home.redacted.org check_reload_status 446 - - Restarting OpenVPN tunnels/interfaces
2024-04-30T10:48:13.939094-05:00 pfSense.home.redacted.org check_reload_status 446 - - Reloading filter
2024-04-30T10:48:14.884072-05:00 pfSense.home.redacted.org php-fpm 407 - - /rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''
2024-04-30T10:48:14.903988-05:00 pfSense.home.redacted.org php-fpm 407 - - /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use S2S_Ben.
2024-04-30T10:48:15.004429-05:00 pfSense.home.redacted.org php-fpm 406 - - /rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''
2024-04-30T10:48:15.023361-05:00 pfSense.home.redacted.org php-fpm 406 - - /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use S2S_P.
2024-04-30T10:48:17.841415-05:00 pfSense.home.redacted.org rc.gateway_alarm 10932 - - >>> Gateway alarm: S2S_Ben (Addr:x.x.x.x Alarm:1 RTT:2705.282ms RTTsd:5038.907ms Loss:3%)
2024-04-30T10:48:17.842787-05:00 pfSense.home.redacted.org check_reload_status 446 - - updating dyndns S2S_Ben
2024-04-30T10:48:17.842844-05:00 pfSense.home.redacted.org check_reload_status 446 - - Restarting IPsec tunnels
2024-04-30T10:48:17.842859-05:00 pfSense.home.redacted.org check_reload_status 446 - - Restarting OpenVPN tunnels/interfaces
2024-04-30T10:48:17.842872-05:00 pfSense.home.redacted.org check_reload_status 446 - - Reloading filter
2024-04-30T10:48:17.959533-05:00 pfSense.home.redacted.org rc.gateway_alarm 13905 - - >>> Gateway alarm: S2S_P (Addr:x.x.x.x Alarm:1 RTT:2665.026ms RTTsd:5028.638ms Loss:3%)
2024-04-30T10:48:17.960993-05:00 pfSense.home.redacted.org check_reload_status 446 - - updating dyndns S2S_P
2024-04-30T10:48:17.961047-05:00 pfSense.home.redacted.org check_reload_status 446 - - Restarting IPsec tunnels
2024-04-30T10:48:17.961062-05:00 pfSense.home.redacted.org check_reload_status 446 - - Restarting OpenVPN tunnels/interfaces
2024-04-30T10:48:17.961074-05:00 pfSense.home.redacted.org check_reload_status 446 - - Reloading filter
2024-04-30T10:48:18.909952-05:00 pfSense.home.redacted.org php-fpm 406 - - /rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''
2024-04-30T10:48:18.931078-05:00 pfSense.home.redacted.org php-fpm 406 - - /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use S2S_Ben.
2024-04-30T10:48:19.023087-05:00 pfSense.home.redacted.org php-fpm 406 - - /rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''
2024-04-30T10:48:19.042034-05:00 pfSense.home.redacted.org php-fpm 406 - - /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use S2S_P.
2024-04-30T10:48:26.937872-05:00 pfSense.home.redacted.org rc.gateway_alarm 194 - - >>> Gateway alarm: S2S_Ben (Addr:x.x.x.x Alarm:1 RTT:2727.521ms RTTsd:5055.855ms Loss:6%)
2024-04-30T10:48:26.939149-05:00 pfSense.home.redacted.org check_reload_status 446 - - updating dyndns S2S_Ben
2024-04-30T10:48:26.939266-05:00 pfSense.home.redacted.org check_reload_status 446 - - Restarting IPsec tunnels
2024-04-30T10:48:26.939312-05:00 pfSense.home.redacted.org check_reload_status 446 - - Restarting OpenVPN tunnels/interfaces
2024-04-30T10:48:26.939350-05:00 pfSense.home.redacted.org check_reload_status 446 - - Reloading filter
2024-04-30T10:48:27.100792-05:00 pfSense.home.redacted.org rc.gateway_alarm 3958 - - >>> Gateway alarm: S2S_P (Addr:x.x.x.x Alarm:1 RTT:2687.469ms RTTsd:5045.171ms Loss:5%)
2024-04-30T10:48:27.102206-05:00 pfSense.home.redacted.org check_reload_status 446 - - updating dyndns S2S_P
2024-04-30T10:48:27.102260-05:00 pfSense.home.redacted.org check_reload_status 446 - - Restarting IPsec tunnels
2024-04-30T10:48:27.102277-05:00 pfSense.home.redacted.org check_reload_status 446 - - Restarting OpenVPN tunnels/interfaces
2024-04-30T10:48:27.102291-05:00 pfSense.home.redacted.org check_reload_status 446 - - Reloading filter
2024-04-30T10:48:28.019613-05:00 pfSense.home.redacted.org php-fpm 406 - - /rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''
2024-04-30T10:48:28.039692-05:00 pfSense.home.redacted.org php-fpm 406 - - /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use S2S_Ben.
2024-04-30T10:48:28.172114-05:00 pfSense.home.redacted.org php-fpm 406 - - /rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''
2024-04-30T10:48:28.190894-05:00 pfSense.home.redacted.org php-fpm 406 - - /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use S2S_P.
2024-04-30T10:49:19.496628-05:00 pfSense.home.redacted.org rc.gateway_alarm 2480 - - >>> Gateway alarm: S2S_Ben (Addr:x.x.x.x Alarm:0 RTT:53.499ms RTTsd:19.126ms Loss:0%)
2024-04-30T10:49:19.498033-05:00 pfSense.home.redacted.org check_reload_status 446 - - updating dyndns S2S_Ben
2024-04-30T10:49:19.498085-05:00 pfSense.home.redacted.org check_reload_status 446 - - Restarting IPsec tunnels
2024-04-30T10:49:19.498101-05:00 pfSense.home.redacted.org check_reload_status 446 - - Restarting OpenVPN tunnels/interfaces
2024-04-30T10:49:19.498113-05:00 pfSense.home.redacted.org check_reload_status 446 - - Reloading filter
2024-04-30T10:49:19.936284-05:00 pfSense.home.redacted.org rc.gateway_alarm 6022 - - >>> Gateway alarm: S2S_P (Addr:x.x.x.x Alarm:0 RTT:23.960ms RTTsd:20.870ms Loss:0%)
2024-04-30T10:49:19.937811-05:00 pfSense.home.redacted.org check_reload_status 446 - - updating dyndns S2S_P
2024-04-30T10:49:19.937875-05:00 pfSense.home.redacted.org check_reload_status 446 - - Restarting IPsec tunnels
2024-04-30T10:49:19.937891-05:00 pfSense.home.redacted.org check_reload_status 446 - - Restarting OpenVPN tunnels/interfaces
2024-04-30T10:49:19.937905-05:00 pfSense.home.redacted.org check_reload_status 446 - - Reloading filter
2024-04-30T10:49:20.596949-05:00 pfSense.home.redacted.org php-fpm 406 - - /rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''
2024-04-30T10:49:20.616603-05:00 pfSense.home.redacted.org php-fpm 406 - - /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use S2S_Ben.
2024-04-30T10:49:21.011083-05:00 pfSense.home.redacted.org php-fpm 407 - - /rc.openvpn: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''
2024-04-30T10:49:21.030293-05:00 pfSense.home.redacted.org php-fpm 407 - - /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use S2S_P.

stephenw10 · Jun 4, 2024, 3:37 PM

Hmm, nothing terribly exciting there. Sure seems like it must be OpenVPN doing something.