crash report-random reboot.

[johnpoz]

@gtr_991 your attachments didn't work.

[stephenw10]

Yup, probably change the extension.

[GTR_991]

@johnpoz please check

https://justpaste.it/brk4b

https://docs.google.com/document/d/1Iu5OMH6eQdTFfQxboOLtRm3VmQ2o79FG/edit?usp=sharing&ouid=110855623675863770019&rtpof=true&sd=true

I think it might be related to IPv6?

����db:0:kdb.enter.default>  run pfs
db:1:pfs> bt
Tracing pid 0 tid 100009 td 0xfffffe00121cfc80
kdb_enter() at kdb_enter+0x32/frame 0xfffffe00109f0de0
vpanic() at vpanic+0x182/frame 0xfffffe00109f0e30
panic() at panic+0x43/frame 0xfffffe00109f0e90
trap_fatal() at trap_fatal+0x409/frame 0xfffffe00109f0ef0
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00109f0f50
calltrap() at calltrap+0x8/frame 0xfffffe00109f0f50
--- trap 0xc, rip = 0xffffffff80eb8606, rsp = 0xfffffe00109f1020, rbp = 0xfffffe00109f1020 ---
if_inc_counter() at if_inc_counter+0x6/frame 0xfffffe00109f1020
looutput() at looutput+0x4f/frame 0xfffffe00109f1050
ip6_forward() at ip6_forward+0x888/frame 0xfffffe00109f1150
pf_refragment6() at pf_refragment6+0x164/frame 0xfffffe00109f11a0
pf_test6() at pf_test6+0x1380/frame 0xfffffe00109f1310
pf_check6_out() at pf_check6_out+0x40/frame 0xfffffe00109f1340
pfil_mbuf_out() at pfil_mbuf_out+0x35/frame 0xfffffe00109f1370
ip6_output() at ip6_output+0x1204/frame 0xfffffe00109f15b0
icmp6_reflect() at icmp6_reflect+0x2dd/frame 0xfffffe00109f1660
icmp6_error() at icmp6_error+0x37c/frame 0xfffffe00109f16d0
pf_route6() at pf_route6+0x7ff/frame 0xfffffe00109f17b0
pf_test6() at pf_test6+0xce3/frame 0xfffffe00109f1930
pf_check6_out() at pf_check6_out+0x40/frame 0xfffffe00109f1960
pfil_mbuf_out() at pfil_mbuf_out+0x35/frame 0xfffffe00109f1990
ip6_forward() at ip6_forward+0x3f4/frame 0xfffffe00109f1a90
ip6_input() at ip6_input+0x9a4/frame 0xfffffe00109f1b70
netisr_dispatch_src() at netisr_dispatch_src+0x2a6/frame 0xfffffe00109f1bc0
ether_demux() at ether_demux+0x144/frame 0xfffffe00109f1bf0
ether_nh_input() at ether_nh_input+0x353/frame 0xfffffe00109f1c50
netisr_dispatch_src() at netisr_dispatch_src+0xb9/frame 0xfffffe00109f1ca0
ether_input() at ether_input+0x69/frame 0xfffffe00109f1d00
iflib_rxeof() at iflib_rxeof+0xbdb/frame 0xfffffe00109f1e00
_task_fn_rx() at _task_fn_rx+0x72/frame 0xfffffe00109f1e40
gtaskqueue_run_locked() at gtaskqueue_run_locked+0x15d/frame 0xfffffe00109f1ec0
gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xc3/frame 0xfffffe00109f1ef0
fork_exit() at fork_exit+0x7e/frame 0xfffffe00109f1f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00109f1f30
--- trap 0x3af1f406, rip = 0xb1996f6ed281a95f, rsp = 0x63ac29febd5b9411, rbp = 0x546c3774f1457d26 ---

Tracing command kernel pid 0 tid 100010 td 0xfffffe00121cf560 (CPU 3)
kdb_enter() at kdb_enter+0x32/frame 0xfffffe00109ebde0
vpanic() at vpanic+0x182/frame 0xfffffe00109ebe30
panic() at panic+0x43/frame 0xfffffe00109ebe90
trap_fatal() at trap_fatal+0x409/frame 0xfffffe00109ebef0
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00109ebf50
calltrap() at calltrap+0x8/frame 0xfffffe00109ebf50
--- trap 0xc, rip = 0xffffffff80eb8606, rsp = 0xfffffe00109ec020, rbp = 0xfffffe00109ec020 ---
if_inc_counter() at if_inc_counter+0x6/frame 0xfffffe00109ec020
looutput() at looutput+0x4f/frame 0xfffffe00109ec050
ip6_forward() at ip6_forward+0x888/frame 0xfffffe00109ec150
pf_refragment6() at pf_refragment6+0x164/frame 0xfffffe00109ec1a0
pf_test6() at pf_test6+0x1380/frame 0xfffffe00109ec310
pf_check6_out() at pf_check6_out+0x40/frame 0xfffffe00109ec340
pfil_mbuf_out() at pfil_mbuf_out+0x35/frame 0xfffffe00109ec370
ip6_output() at ip6_output+0x1204/frame 0xfffffe00109ec5b0
icmp6_reflect() at icmp6_reflect+0x2dd/frame 0xfffffe00109ec660
icmp6_error() at icmp6_error+0x37c/frame 0xfffffe00109ec6d0
pf_route6() at pf_route6+0x7ff/frame 0xfffffe00109ec7b0
pf_test6() at pf_test6+0xce3/frame 0xfffffe00109ec930
pf_check6_out() at pf_check6_out+0x40/frame 0xfffffe00109ec960
pfil_mbuf_out() at pfil_mbuf_out+0x35/frame 0xfffffe00109ec990
ip6_forward() at ip6_forward+0x3f4/frame 0xfffffe00109eca90
ip6_input() at ip6_input+0x9a4/frame 0xfffffe00109ecb70
netisr_dispatch_src() at netisr_dispatch_src+0x2a6/frame 0xfffffe00109ecbc0
ether_demux() at ether_demux+0x144/frame 0xfffffe00109ecbf0
ether_nh_input() at ether_nh_input+0x353/frame 0xfffffe00109ecc50
netisr_dispatch_src() at netisr_dispatch_src+0xb9/frame 0xfffffe00109ecca0
ether_input() at ether_input+0x69/frame 0xfffffe00109ecd00
iflib_rxeof() at iflib_rxeof+0xbdb/frame 0xfffffe00109ece00
_task_fn_rx() at _task_fn_rx+0x72/frame 0xfffffe00109ece40
gtaskqueue_run_locked() at gtaskqueue_run_locked+0x15d/frame 0xfffffe00109ecec0
gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xc3/frame 0xfffffe00109ecef0
fork_exit() at fork_exit+0x7e/frame 0xfffffe00109ecf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00109ecf30

textdump.tar info (1).txt textdump3.tar textdump2.tar

[johnpoz]

@gtr_991 if you feel its ipv6 related - could be this?

https://redmine.pfsense.org/issues/14164

Has it happened more than once? My understanding this redmine has been addressed in 23.05 and its pretty close to be released, if your seeing this quite a bit maybe you might want to update to 23.05 a bit early?

[GTR_991]

@johnpoz yes, today it has been super unstable, already the system paniced/rebooted like 7 times since 5 hours.

anything can be done meanwhile? beside updating to snapshots..

Thanks in advance,

[johnpoz]

@gtr_991 if it was me, and its ipv6 related - I would just turn off IPv6.. then its kind of hard to cause a problem?

Do you actually have some "need" for IPv6 - are you hosting or providing services inbound on ipv6? Or just outbound - because I for the life of me can not find any sort of resource that is only available via IPv6..

[GTR_991]

@johnpoz I dont actually require IPv6 connectivity, i just use it outbound.
I disabled IPv6 for testing.
i will report back by tomorrow.

[johnpoz]

@gtr_991 sounds like a very reasonable plan.. Maybe turn it back on trmw and if you start seeing the same crashes then need to look deeper into what could be causing that.

Strange that you were working fine before? And now like 7 times in a 5 hours? wonder what is triggering it, that was happening before?

[GTR_991]

@johnpoz For sure i will do it.

Honestly im not even sure, i was running openwrt previously then i installed pfsense plus around 2 days back, the first 24 hours it was very stable then suddenly the crashes started to happen around the 2nd day of install, it kept happening more and more often where in the past 5 hours it was crashing each hour. could it be something related to pfblockerNG?
BTW I think the only thing i changes i did by the 2nd day of installation , i changed the DNS setting to forwarding mode using TLS with custom DNS. might be related?

[johnpoz]

@gtr_991 said in crash report-random reboot.:

DNS setting to forwarding mode using TLS with custom DNS. might be related?

I not a fan of dot or doh - they do nothing for me.. I just resolve.. But many users seem to love handing all their dns over to providers ;)

There has been issues with dnssec being enable when you forward, where issues with dns - but have not heard of crashes because of it, etc. And some discussion around IPv6... I would see how it works with IPv6 just off - but if your forwarding be it over dot or just normally there is no point to asking for dnssec - where you forward is either doing dnssec or it isn't - you having that set is just going to be problematic in the long run without any benefit.

[stephenw10]

It's this: https://redmine.pfsense.org/issues/14077

Fixed in 23.05/2.7

Steve

[GTR_991]

Crashes seems stopped after disabling IPv6.

is there any timeframe when the build 23.05 will be released on stable channel?

[stephenw10]

Well it's targeted for May 2023. However that's always dependent on testing, new issues etc.

[GTR_991]

Been almost 24 hours and its running stable..

[stephenw10]

Yeah, if you've removed the IPv6 traffic that was triggering it you should be fine. 23.05 is not far off now anyway.