Crash report!!!!

the other

Hardware TCP Segmentation Offloading/Hardware Large Receive Offloading

Hey there,
I admit: didn't even know what those options are / do /mean and had to read it up here:
https://docs.netgate.com/pfsense/en/latest/config/advanced-networking.html

...so: have YOU read that (especially the warnings)? ;)
Just guessing here, but both warnings state not to do so for routers / firewalls and not to do so unless said by a official support representative...

Antibiotic

@the-other Yes. I know these settings. Just want to warning about crash status!

stephenw10

Hmm, backtrace is unusually long:

db:1:pfs> bt
Tracing pid 0 tid 100014 td 0xfffff80001973000
kdb_enter() at kdb_enter+0x33/frame 0xfffffe002031ae00
panic() at panic+0x43/frame 0xfffffe002031ae60
dblfault_handler() at dblfault_handler+0x1ce/frame 0xfffffe002031af20
Xdblfault() at Xdblfault+0xd7/frame 0xfffffe002031af20
--- trap 0x17, rip = 0xffffffff8494db96, rsp = 0xfffffe00c59e5028, rbp = 0xfffffe00c59e5720 ---
skip_aad_check_init() at skip_aad_check_init+0xd/frame 0xfffffe00c59e5720
iimb_crypto_dispatch() at iimb_crypto_dispatch+0x97/frame 0xfffffe00c59e5770
ovpn_transmit_to_peer() at ovpn_transmit_to_peer+0x8ca/frame 0xfffffe00c59e57e0
ovpn_output() at ovpn_output+0x2fc/frame 0xfffffe00c59e5860
ip_output_send() at ip_output_send+0xdc/frame 0xfffffe00c59e58a0
ip_output() at ip_output+0x1295/frame 0xfffffe00c59e59a0
ovpn_encap() at ovpn_encap+0x3fa/frame 0xfffffe00c59e5a20
ovpn_encrypt_tx_cb() at ovpn_encrypt_tx_cb+0xc4/frame 0xfffffe00c59e5a90
ovpn_transmit_to_peer() at ovpn_transmit_to_peer+0x8ca/frame 0xfffffe00c59e5b00
ovpn_output() at ovpn_output+0x2fc/frame 0xfffffe00c59e5b80
ip_output_send() at ip_output_send+0xdc/frame 0xfffffe00c59e5bc0
ip_output() at ip_output+0x1295/frame 0xfffffe00c59e5cc0
ovpn_encap() at ovpn_encap+0x3fa/frame 0xfffffe00c59e5d40
ovpn_encrypt_tx_cb() at ovpn_encrypt_tx_cb+0xc4/frame 0xfffffe00c59e5db0
ovpn_transmit_to_peer() at ovpn_transmit_to_peer+0x8ca/frame 0xfffffe00c59e5e20
ovpn_output() at ovpn_output+0x2fc/frame 0xfffffe00c59e5ea0
ip_output_send() at ip_output_send+0xdc/frame 0xfffffe00c59e5ee0
ip_output() at ip_output+0x1295/frame 0xfffffe00c59e5fe0
ovpn_encap() at ovpn_encap+0x3fa/frame 0xfffffe00c59e6060
ovpn_encrypt_tx_cb() at ovpn_encrypt_tx_cb+0xc4/frame 0xfffffe00c59e60d0
ovpn_transmit_to_peer() at ovpn_transmit_to_peer+0x8ca/frame 0xfffffe00c59e6140
ovpn_output() at ovpn_output+0x2fc/frame 0xfffffe00c59e61c0
ip_output_send() at ip_output_send+0xdc/frame 0xfffffe00c59e6200
ip_output() at ip_output+0x1295/frame 0xfffffe00c59e6300
ovpn_encap() at ovpn_encap+0x3fa/frame 0xfffffe00c59e6380
ovpn_encrypt_tx_cb() at ovpn_encrypt_tx_cb+0xc4/frame 0xfffffe00c59e63f0
ovpn_transmit_to_peer() at ovpn_transmit_to_peer+0x8ca/frame 0xfffffe00c59e6460
ovpn_output() at ovpn_output+0x2fc/frame 0xfffffe00c59e64e0
ip_output_send() at ip_output_send+0xdc/frame 0xfffffe00c59e6520
ip_output() at ip_output+0x1295/frame 0xfffffe00c59e6620
ovpn_encap() at ovpn_encap+0x3fa/frame 0xfffffe00c59e66a0
ovpn_encrypt_tx_cb() at ovpn_encrypt_tx_cb+0xc4/frame 0xfffffe00c59e6710
ovpn_transmit_to_peer() at ovpn_transmit_to_peer+0x8ca/frame 0xfffffe00c59e6780
ovpn_output() at ovpn_output+0x2fc/frame 0xfffffe00c59e6800
ip_output_send() at ip_output_send+0xdc/frame 0xfffffe00c59e6840
ip_output() at ip_output+0x1295/frame 0xfffffe00c59e6940
ovpn_encap() at ovpn_encap+0x3fa/frame 0xfffffe00c59e69c0
ovpn_encrypt_tx_cb() at ovpn_encrypt_tx_cb+0xc4/frame 0xfffffe00c59e6a30
ovpn_transmit_to_peer() at ovpn_transmit_to_peer+0x8ca/frame 0xfffffe00c59e6aa0
ovpn_output() at ovpn_output+0x2fc/frame 0xfffffe00c59e6b20
ip_output_send() at ip_output_send+0xdc/frame 0xfffffe00c59e6b60
ip_output() at ip_output+0x1295/frame 0xfffffe00c59e6c60
ovpn_encap() at ovpn_encap+0x3fa/frame 0xfffffe00c59e6ce0
ovpn_encrypt_tx_cb() at ovpn_encrypt_tx_cb+0xc4/frame 0xfffffe00c59e6d50
ovpn_transmit_to_peer() at ovpn_transmit_to_peer+0x8ca/frame 0xfffffe00c59e6dc0
ovpn_output() at ovpn_output+0x2fc/frame 0xfffffe00c59e6e40
ip_output_send() at ip_output_send+0xdc/frame 0xfffffe00c59e6e80
ip_output() at ip_output+0x1295/frame 0xfffffe00c59e6f80
ovpn_encap() at ovpn_encap+0x3fa/frame 0xfffffe00c59e7000
ovpn_encrypt_tx_cb() at ovpn_encrypt_tx_cb+0xc4/frame 0xfffffe00c59e7070
ovpn_transmit_to_peer() at ovpn_transmit_to_peer+0x8ca/frame 0xfffffe00c59e70e0
ovpn_output() at ovpn_output+0x2fc/frame 0xfffffe00c59e7160
ip_output_send() at ip_output_send+0xdc/frame 0xfffffe00c59e71a0
ip_output() at ip_output+0x1295/frame 0xfffffe00c59e72a0
ovpn_encap() at ovpn_encap+0x3fa/frame 0xfffffe00c59e7320
ovpn_encrypt_tx_cb() at ovpn_encrypt_tx_cb+0xc4/frame 0xfffffe00c59e7390
ovpn_transmit_to_peer() at ovpn_transmit_to_peer+0x8ca/frame 0xfffffe00c59e7400
ovpn_output() at ovpn_output+0x2fc/frame 0xfffffe00c59e7480
ip_output_send() at ip_output_send+0xdc/frame 0xfffffe00c59e74c0
ip_output() at ip_output+0x1295/frame 0xfffffe00c59e75c0
ovpn_encap() at ovpn_encap+0x3fa/frame 0xfffffe00c59e7640
ovpn_encrypt_tx_cb() at ovpn_encrypt_tx_cb+0xc4/frame 0xfffffe00c59e76b0
ovpn_transmit_to_peer() at ovpn_transmit_to_peer+0x8ca/frame 0xfffffe00c59e7720
ovpn_output() at ovpn_output+0x2fc/frame 0xfffffe00c59e77a0
ip_output_send() at ip_output_send+0xdc/frame 0xfffffe00c59e77e0
ip_output() at ip_output+0x1295/frame 0xfffffe00c59e78e0
ovpn_encap() at ovpn_encap+0x3fa/frame 0xfffffe00c59e7960
ovpn_encrypt_tx_cb() at ovpn_encrypt_tx_cb+0xc4/frame 0xfffffe00c59e79d0
ovpn_transmit_to_peer() at ovpn_transmit_to_peer+0x8ca/frame 0xfffffe00c59e7a40
ovpn_output() at ovpn_output+0x2fc/frame 0xfffffe00c59e7ac0
ip_output_send() at ip_output_send+0xdc/frame 0xfffffe00c59e7b00
ip_output() at ip_output+0x1295/frame 0xfffffe00c59e7c00
ovpn_encap() at ovpn_encap+0x3fa/frame 0xfffffe00c59e7c80
ovpn_encrypt_tx_cb() at ovpn_encrypt_tx_cb+0xc4/frame 0xfffffe00c59e7cf0
ovpn_transmit_to_peer() at ovpn_transmit_to_peer+0x8ca/frame 0xfffffe00c59e7d60
ovpn_output() at ovpn_output+0x2fc/frame 0xfffffe00c59e7de0
ip_output_send() at ip_output_send+0xdc/frame 0xfffffe00c59e7e20
ip_output() at ip_output+0x1295/frame 0xfffffe00c59e7f20
ovpn_encap() at ovpn_encap+0x3fa/frame 0xfffffe00c59e7fa0
ovpn_encrypt_tx_cb() at ovpn_encrypt_tx_cb+0xc4/frame 0xfffffe00c59e8010
ovpn_transmit_to_peer() at ovpn_transmit_to_peer+0x8ca/frame 0xfffffe00c59e8080
ovpn_output() at ovpn_output+0x2fc/frame 0xfffffe00c59e8100
ip_output_send() at ip_output_send+0xdc/frame 0xfffffe00c59e8140
ip_output() at ip_output+0x1295/frame 0xfffffe00c59e8240
ovpn_encap() at ovpn_encap+0x3fa/frame 0xfffffe00c59e82c0
ovpn_encrypt_tx_cb() at ovpn_encrypt_tx_cb+0xc4/frame 0xfffffe00c59e8330
ovpn_transmit_to_peer() at ovpn_transmit_to_peer+0x8ca/frame 0xfffffe00c59e83a0
ovpn_output() at ovpn_output+0x2fc/frame 0xfffffe00c59e8420
ip_output_send() at ip_output_send+0xdc/frame 0xfffffe00c59e8460
ip_output() at ip_output+0x1295/frame 0xfffffe00c59e8560
ovpn_encap() at ovpn_encap+0x3fa/frame 0xfffffe00c59e85e0
ovpn_encrypt_tx_cb() at ovpn_encrypt_tx_cb+0xc4/frame 0xfffffe00c59e8650
ovpn_transmit_to_peer() at ovpn_transmit_to_peer+0x8ca/frame 0xfffffe00c59e86c0
ovpn_output() at ovpn_output+0x2fc/frame 0xfffffe00c59e8740
ip_output_send() at ip_output_send+0xdc/frame 0xfffffe00c59e8780
ip_output() at ip_output+0x1295/frame 0xfffffe00c59e8880
ovpn_encap() at ovpn_encap+0x3fa/frame 0xfffffe00c59e8900
ovpn_encrypt_tx_cb() at ovpn_encrypt_tx_cb+0xc4/frame 0xfffffe00c59e8970
ovpn_transmit_to_peer() at ovpn_transmit_to_peer+0x8ca/frame 0xfffffe00c59e89e0
ovpn_output() at ovpn_output+0x2fc/frame 0xfffffe00c59e8a60
ip_tryforward() at ip_tryforward+0x583/frame 0xfffffe00c59e8b10
ip_input() at ip_input+0x349/frame 0xfffffe00c59e8b70
netisr_dispatch_src() at netisr_dispatch_src+0x22c/frame 0xfffffe00c59e8bc0
ether_demux() at ether_demux+0x149/frame 0xfffffe00c59e8bf0
ether_nh_input() at ether_nh_input+0x36d/frame 0xfffffe00c59e8c50
netisr_dispatch_src() at netisr_dispatch_src+0xaf/frame 0xfffffe00c59e8ca0
ether_input() at ether_input+0x69/frame 0xfffffe00c59e8d00
iflib_rxeof() at iflib_rxeof+0xc46/frame 0xfffffe00c59e8e00
_task_fn_rx() at _task_fn_rx+0x72/frame 0xfffffe00c59e8e40
gtaskqueue_run_locked() at gtaskqueue_run_locked+0x14e/frame 0xfffffe00c59e8ec0
gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xc2/frame 0xfffffe00c59e8ef0
fork_exit() at fork_exit+0x7f/frame 0xfffffe00c59e8f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00c59e8f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Panic:

Fatal double fault
rip 0xffffffff84b45f78 rsp 0xfffffe00c5aadfc0 rbp 0xfffffe00c5aae680
rax 0xfffff8045b7d5c00 rdx 0xfffff8045b7d5c00 rbx 0xfffff802242ef880
rcx 0 rsi 0xfffff8045bba5460 rdi 0xfffff802242ef880
r8 0 r9 0xfffff8045bba5460 r10 0
r11 0 r12 0xfffff80009a29800 r13 0xfffff8045bba5460
r14 0xfffff8045b7d5c00 r15 0xfffff803f88b6300 rflags 0x10286
cs 0x20 ss 0x28 ds 0x3b es 0x3b fs 0x13 gs 0x1b
fsbase 0x40af2841d120 gsbase 0xffffffff84010000 kgsbase 0
cpuid = 0; apic id = 00
panic: double fault
cpuid = 0
time = 1715431646
KDB: enter: panic

2nd Panic:

Fatal double fault
rip 0xffffffff8494db96 rsp 0xfffffe00c59e5028 rbp 0xfffffe00c59e5720
rax 0x10 rdx 0xfffff80235ba11b8 rbx 0x374
rcx 0xfffff801c34ba6b4 rsi 0xfffff801c3c02080 rdi 0xfffff80235bb5d80
r8 0x8 r9 0xfffff80235ba1118 r10 0x8
r11 0 r12 0xfffff801c3c02080 r13 0xfffff801c3c02080
r14 0x18 r15 0xfffff80235ba1118 rflags 0x10202
cs 0x20 ss 0x28 ds 0x3b es 0x3b fs 0x13 gs 0x1b
fsbase 0x1c2290c03360 gsbase 0xffffffff84013000 kgsbase 0
cpuid = 3; apic id = 06
panic: double fault
cpuid = 3
time = 1715451329
KDB: enter: panic

Seems to be unrelated to the hardware offload settings.

Also happened twice, did you do anything else that may have triggered it?

The backtrace is in OpenVPN and looks to be in IIMB. I assume you have that enabled? With OpenVPN DCO?

Steve

Antibiotic

@stephenw10 said in Crash report!!!!:

The backtrace is in OpenVPN and looks to be in IIMB. I assume you have that enabled? With OpenVPN DCO?

I switched ON than OFF
Second question , yes ( openvpn with dco and iimb)

Antibiotic

@stephenw10 Again crash report, but now after changing settings in limiters!

textdump.tar (1).0 info (1).0

Idk, what is going on!

stephenw10

Backtrace:

db:1:pfs> bt
Tracing pid 0 tid 100013 td 0xfffff80001974740
kdb_enter() at kdb_enter+0x33/frame 0xfffffe00c59ed370
panic() at panic+0x43/frame 0xfffffe00c59ed3d0
vm_fault() at vm_fault+0x15cc/frame 0xfffffe00c59ed4f0
vm_fault_trap() at vm_fault_trap+0xb0/frame 0xfffffe00c59ed540
trap_pfault() at trap_pfault+0x1d6/frame 0xfffffe00c59ed5a0
calltrap() at calltrap+0x8/frame 0xfffffe00c59ed5a0
--- trap 0xc, rip = 0xffffffff84451b37, rsp = 0xfffffe00c59ed670, rbp = 0xfffffe00c59ed6f0 ---
fq_codel_enqueue() at fq_codel_enqueue+0x137/frame 0xfffffe00c59ed6f0
dummynet_io() at dummynet_io+0x26f/frame 0xfffffe00c59ed740
pf_dummynet_route() at pf_dummynet_route+0x3b9/frame 0xfffffe00c59ed830
pf_route() at pf_route+0x28e/frame 0xfffffe00c59ed910
pf_test() at pf_test+0xe28/frame 0xfffffe00c59edac0
pf_check_in() at pf_check_in+0x27/frame 0xfffffe00c59edae0
pfil_mbuf_in() at pfil_mbuf_in+0x38/frame 0xfffffe00c59edb10
ip_input() at ip_input+0x3aa/frame 0xfffffe00c59edb70
netisr_dispatch_src() at netisr_dispatch_src+0x22c/frame 0xfffffe00c59edbc0
ether_demux() at ether_demux+0x149/frame 0xfffffe00c59edbf0
ether_nh_input() at ether_nh_input+0x36d/frame 0xfffffe00c59edc50
netisr_dispatch_src() at netisr_dispatch_src+0xaf/frame 0xfffffe00c59edca0
ether_input() at ether_input+0x69/frame 0xfffffe00c59edd00
iflib_rxeof() at iflib_rxeof+0xc46/frame 0xfffffe00c59ede00
_task_fn_rx() at _task_fn_rx+0x72/frame 0xfffffe00c59ede40
gtaskqueue_run_locked() at gtaskqueue_run_locked+0x14e/frame 0xfffffe00c59edec0
gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xc2/frame 0xfffffe00c59edef0
fork_exit() at fork_exit+0x7f/frame 0xfffffe00c59edf30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00c59edf30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

config_aqm Unable to configure flowset, flowset busy!
config_aqm Unable to configure flowset, flowset busy!
config_aqm Unable to configure flowset, flowset busy!
panic: vm_fault_lookup: fault on nofault entry, addr: 0xfffffe0103234000
cpuid = 2
time = 1715514190
KDB: enter: panic

So that's a completely different panic.
What exactly did you change?

Antibiotic

@stephenw10 Limiters settings

stephenw10

Right but what did you actually change? Added the limiter entirely?

Antibiotic

@stephenw10 Changed limits and flows figures

stephenw10

Hmm, was anything logged? Any alert shown other than the crash?

Did it crash immediately when you saved those changes?

Antibiotic

@stephenw10 '

None
None
Yes

stephenw10

Hmm, it is repeatable?

We'd need to replicate it locally to dig further so steps to reproduce would help a lot.

Antibiotic

@stephenw10 Idk, will it help, that what I found i sys.log. That was on next reboot after crash

2024-05-12 19:25:14.079708+03:00 check_reload_status 643 Reloading filter
2024-05-12 19:25:15.115594+03:00 php-fpm 592 /rc.newwanip: rc.newwanip: Info: starting on igc1.
2024-05-12 19:25:15.115937+03:00 php-fpm 592 /rc.newwanip: rc.newwanip: on (IP address: 192.168.10.1) (interface: LAN[lan]) (real interface: igc1).
2024-05-12 19:25:15.401258+03:00 kernel - config_aqm Unable to configure flowset, flowset busy!
2024-05-12 19:25:15.602269+03:00 kernel - config_aqm Unable to configure flowset, flowset busy!
2024-05-12 19:25:15.602325+03:00 kernel - config_aqm Unable to configure flowset, flowset busy!
2024-05-12 19:25:16.004364+03:00 kernel - config_aqm Unable to configure flowset, flowset busy!
2024-05-12 19:25:16.004547+03:00 kernel - config_aqm Unable to configure flowset, flowset busy!
2024-05-12 19:25:16.216719+03:00 php-fpm 592 /rc.newwanip: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''
2024-05-12 19:25:17.425245+03:00 php-fpm 592 /rc.newwanip: Resyncing OpenVPN instances for interface LAN.
2024-05-12 19:25:17.471028+03:00 php-fpm 592 /rc.newwanip: Creating rrd update script
2024-05-12 19:25:19.496269+03:00 php-fpm 592 /rc.newwanip: Netgate pfSense Plus package system has detected an IP change or dynamic WAN reconnection - 192.168.10.1 -> 192.168.10.1 - Restarting packages.
2024-05-12 19:25:19.496416+03:00 check_reload_status 643 Starting packages
2024-05-12 19:25:19.500231+03:00 check_reload_status 643 Reloading filter
2024-05-12 19:25:19.500275+03:00 check_reload_status 643 Reloading filter
2024-05-12 19:25:20.534000+03:00 php-fpm 87873 /rc.start_packages: Restarting/Starting all packages.
2024-05-12 19:25:20.829266+03:00 kernel - config_aqm Unable to configure flowset, flowset busy!

I think will remove Limiters as full of bugs

Antibiotic

@stephenw10 said in Crash report!!!!:

further so steps to reproduce would help a lot

Steps , what I did
Untick all offloads (Limiters were already) , than switch off back all offloads get first crash. After reboot changed some settings in Limiters as above in my post, get second crash. Next step remove Limiters and waiting now third crash.

stephenw10

And just resaving the Limiters again did not reproduce the crash?

Antibiotic

@stephenw10 Sorry, already removed Limiters and making state resetting's

kprovost

@stephenw10 said in Crash report!!!!:

Hmm, backtrace is unusually long

It's a routing loop. The OpenVPN traffic is being routing into the OpenVPN tunnel. Sooner or later we run out of stack and crash.

It's a bug in if_ovpn that it doesn't discard this traffic, but it's also a configuration error. Once the bug is fixed the tunnel still won't work.

Antibiotic

@kprovost said in Crash report!!!!:

but it's also a configuration error.

What do you mean, in my configuration error? Or OpenVPN have error itself?

kprovost

@Antibiotic You've somehow configured your system so that the OpenVPN tunnelled traffic goes down the OpenVPN tunnel.

stephenw10

Check the routing table in Diag > Routes. Is your default route getting set to the OpenVPN gateway?

Make sure the system default gateway is set to the WAN gateway in Sys > Routing > Gateways.