Reproducible kernel panic with pfSense 2.2 and IPSEC
-
Change the time source used by the kernel seems it is not working properly.
Usually you make these modifications on loader.conf.local by putting the tunable value there. -
I'm not sure this is 100% reproducible, but I have seen this numerous times in the last couple of days. Had to downgrade some production boxes to 2.1.5 because they kept randomly crashing at least once a day.
These are also Alix 2D13 boxes. I cannot make them crash at will by any means I know about, but they do crash. Particularly in the early morning when traffic on the IPsec tunnels is reestablished after idling during the night, so I believe it is also IPsec related.
ermal, could you elaborate on the time source issue??
-
For example
kern.timecounter.hardware: TSC-low
kern.timecounter.choice: TSC-low(1000) ACPI-fast(900) i8254(0) HPET(950) dummy(-1000000) -
Alright, I grabbed one Alix box and changed:
kern.timecounter.hardware: TSC -> i8254Will report back if this box keeps coredumping.
It'd be great if sh0gun could test this on his reproducible scenario
EDIT: In case it is not clear, to temporarily set the value you can run "sysctl kern.timecounter.hardware=i8254" on the shell, or force the tunable through the advanced options. Anyway, no idea about this setting
-
I'm having the same issue, in my case it crashes when I keep IP compression enabled in the advanced IPsec options. Try disabling it if enabled, maybe it helps in your case too.
I have a reproducible kernel panic with pfSense 2.2 and IPSEC.
My setup: ALIX 2D13, IPSEC with one P1. IKEv1, PSK, aggressive. One P2 (net-to-net tunnel). pfSense must be the responder upon P2 connection. Now send a ping from any host in the remote network to the local pfSense box -> immediate crash.
Fatal double fault: eip = 0xc12c6ac8 esp = 0xc8b68ff8 ebp = 0xc8b69000 cpuid = 0; apic id = 00 panic: double fault cpuid = 0 KDB: enter: panic [ thread pid 12 tid 100015 ] Stopped at kdb_enter+0x3d: movl $0,kdb_why db:0:kdb.enter.default> textdump set textdump set db:0:kdb.enter.default> capture on db:0:kdb.enter.default> run lockinfo db:1:lockinfo> show locks No such command db:1:locks> show alllocks No such command db:1:alllocks> show lockedvnods Locked vnodes db:0:kdb.enter.default> show pcpu cpuid = 0 dynamic pcpu = 0x5e7a00 curthread = 0xc4317c40: pid 12 "swi5: fast taskq" curpcb = 0xc8b6ad60 fpcurthread = none idlethread = 0xc426d000: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 db:0:kdb.enter.default> bt Tracing pid 12 tid 100015 td 0xc4317c40 kdb_enter(c142c723,c142c723,c15ea443,c205b4c8,0,...) at kdb_enter+0x3d/frame 0xc205b480 panic(c15ea443,0,0,0,c8b69000,...) at panic+0x144/frame 0xc205b4bc dblfault_handler() at dblfault_handler+0xab/frame 0xc205b4bc --- trap 0x17, eip = 0xc12c6ac8, esp = 0xc8b68ff8, ebp = 0xc8b69000 --- cpu_fill_vdso_timehands(c8b69030,80000000,c8b69098,c0ce65b6,c8b69030,...) at cpu_fill_vdso_timehands+0x18/frame 0xc8b69000 tc_fill_vdso_timehands(c8b69030,7498,2,ed7e,fffffffe,...) at tc_fill_vdso_timehands+0x63/frame 0xc8b69010 timekeep_push_vdso(0,80000000,1dafce93,0,0,...) at timekeep_push_vdso+0x36/frame 0xc8b69098 tc_windup(1,0,c142675e,219,0,...) at tc_windup+0x321/frame 0xc8b690e8 hardclock_cnt(1,0,0,0,c4317c40,...) at hardclock_cnt+0x447/frame 0xc8b69150 handleevents(0,0,0,c8b691f0,c0d12b54,...) at handleevents+0xee/frame 0xc8b691a0 timercb(c446f640,0,0,c41b1e00,0,...) at timercb+0x3b9/frame 0xc8b691fc clkintr(c446f600,0,c4317c40,0,c1e3718c,...) at clkintr+0xfc/frame 0xc8b69220 intr_event_handle(c426bb80,c8b6928c,c4268310,6,c1e3718c,...) at intr_event_handle+0x85/frame 0xc8b69240 intr_execute_handlers(c1e3718c,c8b6928c) at intr_execute_handlers+0x42/frame 0xc8b69260 atpic_handle_intr(0,c8b6928c) at atpic_handle_intr+0x5a/frame 0xc8b6927c Xatpic_intr0() at Xatpic_intr0+0x22/frame 0xc8b6927c --- interrupt, eip = 0xc1291015, esp = 0xc8b692cc, ebp = 0xc8b69388 --- in_cksum_skip(c48fa200,14,0,0,c8b69528,...) at in_cksum_skip+0x215/frame 0xc8b69388 pf_test(2,c43c9400,c8b6962c,0,c8b696c8,...) at pf_test+0x2899/frame 0xc8b695e4 pf_check_out(0,c8b6962c,c43c9400,2,0,...) at pf_check_out+0x4b/frame 0xc8b69608 pfil_run_hooks(c20a1c14,c8b696f4,c43c9400,2,0,...) at pfil_run_hooks+0x88/frame 0xc8b69660 ip_output(c48fa200,0,0,2,0,...) at ip_output+0xaac/frame 0xc8b69718 ipsec_process_done(c48fa200,c57de480,0,5e29211a,c48fa200,...) at ipsec_process_done+0x3cf/frame 0xc8b69768 esp_output_cb(c5fe0bf4,c1e21524,c8b697f8,c0f9f5e1,c48fa200,...) at esp_output_cb+0x3cd/frame 0xc8b697c0 crypto_done(c5fe0bf4,c48fa200,8c,c,c8b699a8,...) at crypto_done+0x99/frame 0xc8b697f8 swcr_process(c4269080,c5fe0bf4,0,c8b69a60,c0ccd50d,...) at swcr_process+0x6e/frame 0xc8b69a00 crypto_invoke(0,40,0,0,0,...) at crypto_invoke+0x79/frame 0xc8b69a38 crypto_dispatch(c5fe0bf4,c145696b,375,c8b69abb,c495b028,...) at crypto_dispatch+0x64/frame 0xc8b69a60 esp_output(c48fa200,c57de480,0,14,9,...) at esp_output+0x91d/frame 0xc8b69ae0 ipsec4_process_packet(c48fa200,c57de480,0,0,0,...) at ipsec4_process_packet+0x312/frame 0xc8b69b70 ip_ipsec_output(c8b69c34,0,c8b69c30,c8b69c2c,0,...) at ip_ipsec_output+0x1c8/frame 0xc8b69ba0 ip_output(c48fa200,0,0,0,0,...) at ip_output+0xa2f/frame 0xc8b69c58 icmp_reflect(c8b69d28,10,0,c8b69d04,90000,...) at icmp_reflect+0x5b5/frame 0xc8b69cb0 icmp_input(c48fa200,14,50,1,c8b69ddc,...) at icmp_input+0x9b9/frame 0xc8b69d78 ip_input(c48fa200,c20a184c,c4317c40,0,c8b69e50,...) at ip_input+0x295/frame 0xc8b69ddc netisr_dispatch_src(1,0,c48fa200) at netisr_dispatch_src+0x8b/frame 0xc8b69e24 netisr_dispatch(1,c48fa200,1,102,0,...) at netisr_dispatch+0x20/frame 0xc8b69e38 _ipip_input(c48fa200,14,4,1,3c,...) at _ipip_input+0x650/frame 0xc8b69e80 encap4_input(c48fa200,14,0,0,1,...) at encap4_input+0x210/frame 0xc8b69ee0 ip_input(c48fa200,2,40,0,0,...) at ip_input+0x295/frame 0xc8b69f48 netisr_dispatch_src(1,ca095cc9,c48fa200,101,68,...) at netisr_dispatch_src+0x8b/frame 0xc8b69f90 ipsec4_common_input_cb(c48fa200,c4f28300,14,9,0,...) at ipsec4_common_input_cb+0x276/frame 0xc8b69fd8 esp_input_cb(c5fe0bf4,c053cd1d,c4314af8,e,c495b09e,...) at esp_input_cb+0x772/frame 0xc8b6a068 crypto_done(c5fe0bf4,c8b6a240,10,10,c8b6a240,...) at crypto_done+0x99/frame 0xc8b6a0a0 swcr_process(c4269080,c5fe0bf4,0,c8b6a308,c0ccd50d,...) at swcr_process+0x6e/frame 0xc8b6a2a8 crypto_invoke(0,c495b0ae,c57dda38,c,c57dda38,...) at crypto_invoke+0x79/frame 0xc8b6a2e0 crypto_dispatch(c5fe0bf4,c145696b,1c8,c57dda38,c58c01b8,...) at crypto_dispatch+0x64/frame 0xc8b6a308 esp_input(c48fa200,c4f28300,14,9,d0,...) at esp_input+0x771/frame 0xc8b6a370 ipsec_common_input(9,2,32,c8b6a408) at ipsec_common_input+0x4f7/frame 0xc8b6a3dc ipsec4_common_input(c48fa200,14,32) at ipsec4_common_input+0x39/frame 0xc8b6a3f4 esp4_input(c48fa200,14,c43c9400,1,0,...) at esp4_input+0x20/frame 0xc8b6a408 ip_input(c48fa200,c0cf6261,0,80000000,1dafce93,...) at ip_input+0x295/frame 0xc8b6a470 netisr_dispatch_src(1,0,c48fa200) at netisr_dispatch_src+0x8b/frame 0xc8b6a4b8 netisr_dispatch(1,c48fa200,c492482a,c8b6a540,c0f90d95,...) at netisr_dispatch+0x20/frame 0xc8b6a4cc ng_iface_rcvdata(c4793680,c4a3e000,62,10000000,0,...) at ng_iface_rcvdata+0xea/frame 0xc8b6a4f4 ng_apply_item(0,c2425c80,62,67d16f46,5e7a00,...) at ng_apply_item+0x22d/frame 0xc8b6a550 ng_snd_item(c4a3e000,0,c47b9000,0,0,...) at ng_snd_item+0x1a0/frame 0xc8b6a584 ng_tcpmss_rcvdata(c4793700,c4a3e000,c0f66a20,c498e2e0,c4317c40,...) at ng_tcpmss_rcvdata+0xac/frame 0xc8b6a5cc ng_apply_item(0,c0d12971,c498e000,0,17c83,...) at ng_apply_item+0x22d/frame 0xc8b6a628 ng_snd_item(c4a3e000,0,c4793780,0,c4a3e000,...) at ng_snd_item+0x1a0/frame 0xc8b6a65c ng_ppp_comp_recv(21,0,1,c48fa200,c4a3e000,...) at ng_ppp_comp_recv+0x158/frame 0xc8b6a688 ng_ppp_crypt_recv(21,0,c4317c40,c8b6a6c4,246,...) at ng_ppp_crypt_recv+0x70/frame 0xc8b6a6a4 ng_ppp_rcvdata(c4793880,c4a3e000,c48f2900,c4d7d1fc,c4d7d1a8,...) at ng_ppp_rcvdata+0x2e4/frame 0xc8b6a700 ng_apply_item(0,c4317c40,c48dc8dc,c48f2900,c48dc9d4,...) at ng_apply_item+0x22d/frame 0xc8b6a75c ng_snd_item(c4a3e000,0,c4793b00,0,0,...) at ng_snd_item+0x1a0/frame 0xc8b6a790 ng_tee_rcvdata(c4793c00,c4a3e000,8ce9fb06,0,d7bb17a0,...) at ng_tee_rcvdata+0x156/frame 0xc8b6a7b8 ng_apply_item(0,c495b012,6,c8b6a838,1,...) at ng_apply_item+0x22d/frame 0xc8b6a814 ng_snd_item(c4a3e000,0,c4793c80,0,210,...) at ng_snd_item+0x1a0/frame 0xc8b6a848 ng_pppoe_rcvdata_ether(c4793d00,c4a3e000,c424f400,c492481a,14,...) at ng_pppoe_rcvdata_ether+0x2a3/frame 0xc8b6a8c4 ng_apply_item(0,c47afc00,c48fa200,0,c8b6a954,...) at ng_apply_item+0x22d/frame 0xc8b6a920 ng_snd_item(c4a3e000,0,c47b9a00,0,c47afc00,...) at ng_snd_item+0x1a0/frame 0xc8b6a954 ng_ether_input_orphan(c47afc00,c48fa200,31517afb,694e9713,c8b6a990,...) at ng_ether_input_orphan+0x66/frame 0xc8b6a974 ether_demux(c47afc00,c48fa200,6,0,d7bb17a0,...) at ether_demux+0x1f9/frame 0xc8b6a9a0 ether_nh_input(c48fa200,c8b6aa14,0,0,0,...) at ether_nh_input+0x37e/frame 0xc8b6a9f0 netisr_dispatch_src(9,0,c48fa200) at netisr_dispatch_src+0x8b/frame 0xc8b6aa38 netisr_dispatch(9,c48fa200) at netisr_dispatch+0x20/frame 0xc8b6aa4c ether_input(c47afc00,c48fa200,0,c10215fd,0,...) at ether_input+0x19/frame 0xc8b6aa5c vlan_input(c424fc00,c48fa200,c10215fd,c4951808,c43d0100,...) at vlan_input+0x1a8/frame 0xc8b6aa8c ether_demux(c424fc00,c48fa200,6,7f8,c4951808,...) at ether_demux+0xaf/frame 0xc8b6aab8 ether_nh_input(c48fa200,c48f8b00,18,c8b6ab70,c0b4e3ba,...) at ether_nh_input+0x37e/frame 0xc8b6ab04 netisr_dispatch_src(9,0,c48fa200) at netisr_dispatch_src+0x8b/frame 0xc8b6ab4c netisr_dispatch(9,c48fa200) at netisr_dispatch+0x20/frame 0xc8b6ab60 ether_input(c424fc00,c48fa200,c43cc000,27,c8b6abb8,...) at ether_input+0x19/frame 0xc8b6ab70 vr_rxeof(c1292b00,0,40000000,c57c8310,0,...) at vr_rxeof+0x1f1/frame 0xc8b6abb8 vr_int_task(c43cc000,1,c4317c40,c8b6ac04,c0cef78f,...) at vr_int_task+0x123/frame 0xc8b6abe8 taskqueue_run_locked(c57c8310,2710,0,0,c4317c40,...) at taskqueue_run_locked+0xee/frame 0xc8b6ac2c taskqueue_run(c4269980) at taskqueue_run+0xa3/frame 0xc8b6ac50 taskqueue_fast_run(0,0,246,0,87f59266,...) at taskqueue_fast_run+0x11/frame 0xc8b6ac5c intr_event_execute_handlers(109,c4269900,c1428033,55a,0,...) at intr_event_execute_handlers+0xaa/frame 0xc8b6ac88 ithread_loop(c41b5e70,c8b6ad08,0,0,0,...) at ithread_loop+0x80/frame 0xc8b6acc4 fork_exit(c0cb63f0,c41b5e70,c8b6ad08) at fork_exit+0xa3/frame 0xc8b6acf4 fork_trampoline() at fork_trampoline+0x8/frame 0xc8b6acf4 --- trap 0, eip = 0, esp = 0xc8b6ad40, ebp = 0 --- db:0:kdb.enter.default> ps pid ppid pgrp uid state wmesg wchan cmd 503 66422 21 0 S nanslp 0xc1efd568 sleep 81506 71168 81506 0 Ss (threaded) sshlockout_pf 100124 S nanslp 0xc1efd568 sshlockout_pf 100087 S piperd 0xc47eb660 sshlockout_pf 81187 1 81187 0 Ss+ ttyin 0xc4313470 getty 80856 1 1 0 S nanslp 0xc1efd568 getty 79382 79108 79108 0 S nanslp 0xc1efd568 minicron 79108 1 79108 0 Ss wait 0xc4a7e5e0 minicron 78867 78573 78573 0 S nanslp 0xc1efd568 minicron 78573 1 78573 0 Ss wait 0xc4a7ebc0 minicron 78272 78070 78070 0 S nanslp 0xc1efd568 minicron 78070 1 78070 0 Ss wait 0xc479a8d0 minicron 77434 285 285 0 S accept 0xc48be536 php-fpm 76420 1 76420 0 Ss nanslp 0xc1efd568 cron 71168 1 71168 0 Ss select 0xc46af6a4 syslogd 66918 1 66918 0 Ss select 0xc46af664 igmpproxy 66422 1 21 0 S+ wait 0xc48a28d0 sh 59761 1 59761 0 Ss (threaded) ntpd 100116 S usem 0xc57df500 ntpd 100067 S select 0xc4a2bde4 ntpd 54651 1 54651 0 Ss (threaded) filterdns 100098 S uwait 0xc4ec0a00 signal-thread 100097 S uwait 0xc4ec0a80 filterdns 54535 54039 54535 0 Ss (threaded) charon 100114 S select 0xc46af8e4 charon 100113 S uwait 0xc57df200 charon 100112 S uwait 0xc4e8ea80 charon 100111 S uwait 0xc57def00 charon 100110 S uwait 0xc57de180 charon 100109 S uwait 0xc57de100 charon 100108 S uwait 0xc57de080 charon 100107 S uwait 0xc47bce80 charon 100106 S uwait 0xc47bce00 charon 100105 S uwait 0xc47bcd80 charon 100104 S uwait 0xc47bcd00 charon 100103 S uwait 0xc4ec0d00 charon 100102 S uwait 0xc57b3180 charon 100101 S select 0xc46af8a4 charon 100100 S accept 0xc4d7b886 charon 100099 S uwait 0xc4a66480 charon 100088 S sigwait 0xc58cb000 charon 54039 1 54039 0 Ss select 0xc46af864 starter 50152 1 50152 1002 Ss select 0xc4a2a2e4 dhcpd 45824 1 45590 65534 S select 0xc4a2a324 dnsmasq 42452 1 42144 0 S kqread 0xc4ebd200 lighttpd 36088 36073 36073 0 S piperd 0xc47eb4c8 rrdtool 36073 1 36073 0 Ss select 0xc4a2a5a4 apinger 32981 1 32981 0 Ss select 0xc4a2a824 inetd 32537 1 32537 0 Ss select 0xc4a2a964 openvpn 32250 1 32250 0 Ss bpf 0xc4314c00 filterlog 31116 1 31116 0 Ss select 0xc4a2a6a4 openvpn 29208 1 29208 0 Ss select 0xc4a2a764 openvpn 28380 1 28380 0 Ss select 0xc4a2b0a4 openvpn 26257 1 26257 0 Ss select 0xc4a2b624 hostapd 23378 1 23378 65 Ss select 0xc4a2bb24 dhclient 18174 1 18174 0 Ss select 0xc4a2bc64 dhclient 7709 1 7709 0 Ss (threaded) sshlockout_pf 100073 S nanslp 0xc1efd568 sshlockout_pf 100072 S uwait 0xc47bae80 sshlockout_pf 7258 1 7258 0 Ss select 0xc4a2c7e4 sshd 5614 1 5614 0 Ss (threaded) mpd5 100069 S select 0xc41d6be4 mpd5 320 1 320 0 Ss select 0xc46ad464 devd 302 300 300 0 S kqread 0xc47ba980 check_reload_status 300 1 300 0 Ss kqread 0xc4793080 check_reload_status 285 1 285 0 Ss kqread 0xc4792700 php-fpm 54 0 0 0 DL mdwait 0xc477c000 [md1] 49 0 0 0 DL mdwait 0xc477c800 [md0] 20 0 0 0 DL syncer 0xc1f1cac4 [syncer] 19 0 0 0 DL vlruwt 0xc4799000 [vnlru] 18 0 0 0 DL psleep 0xc1f1c204 [bufdaemon] 17 0 0 0 DL pollid 0xc1efbf30 [idlepoll] 9 0 0 0 DL pgzero 0xc2047d20 [pagezero] 8 0 0 0 DL psleep 0xc2047a44 [vmdaemon] 7 0 0 0 DL psleep 0xc20a7484 [pagedaemon] 6 0 0 0 DL waiting_ 0xc20a1d8c [sctp_iterator] 5 0 0 0 DL pftm 0xc0f622c0 [pf purge] 16 0 0 0 DL (threaded) [usb] 100038 D - 0xc43a8d34 [usbus1] 100037 D - 0xc43a8d04 [usbus1] 100036 D - 0xc43a8cd4 [usbus1] 100035 D - 0xc43a8ca4 [usbus1] 100034 D - 0xc443cb5c [usbus0] 100033 D - 0xc443cb2c [usbus0] 100032 D - 0xc443cafc [usbus0] 100031 D - 0xc443cacc [usbus0] 4 0 0 0 DL (threaded) [cam] 100045 D - 0xc1e3e4a8 [scanner] 100017 D - 0xc1e3e600 [doneq0] 3 0 0 0 DL crypto_r 0xc2046978 [crypto returns] 2 0 0 0 DL crypto_w 0xc20468b8 [crypto] 15 0 0 0 DL - 0xc1e58680 [rand_harvestq] 14 0 0 0 DL (threaded) [geom] 100010 D - 0xc209dde0 [g_down] 100009 D - 0xc209dddc [g_up] 100008 D - 0xc209ddd8 [g_event] 13 0 0 0 DL sleep 0xc1e14bb8 [ng_queue0] 12 0 0 0 RL (threaded) [intr] 100043 I [swi1: pfsync] 100041 I [swi1: pf send] 100039 I [swi0: uart uart] 100030 I [irq12: ohci0 ehci0] 100029 I [irq15: vr2 ata1] 100028 I [irq14: ata0] 100025 I [irq9: ath0] 100023 I [swi6: Giant taskq] 100021 I [swi6: task queue] 100015 Run CPU 0 [swi5: fast taskq] 100006 I [swi3: vm] 100005 I [swi4: clock] 100004 I [swi1: netisr 0] 11 0 0 0 RL [idle: cpu0] 1 0 1 0 SLs wait 0xc42662f0 [init] 10 0 0 0 DL audit_wo 0xc20a5d88 [audit] 0 0 0 0 DLs (threaded) [kernel] 100044 D - 0xc4269800 [CAM taskq] 100027 D - 0xc43cf700 [ath0 net80211 taskq] 100026 D - 0xc43cf780 [ath0 taskq] 100024 D - 0xc4269180 [thread] 100022 D - 0xc4269300 [ffs_trim taskq] 100020 D - 0xc4269480 [acpi_task_2] 100019 D - 0xc4269480 [acpi_task_1] 100018 D - 0xc4269480 [acpi_task_0] 100016 D - 0xc4269880 [kqueue taskq] 100011 D - 0xc426aa80 [firmware taskq] 100000 D swapin 0xc209de64 [swapper] db:0:kdb.enter.default> alltrace Full trace is attached. I can not use IPSEC unfortunately. IKEv2 is not an option because there are compatibility issues with the opposite side. And: sorry for my english... [/thread] -
My system is D2500CC mini-ITX motherboard from Intel, all embedded into it.
I have configured WAN as PPPoE and LAN as 10.0.x.x
I have also configured IPsec for mobile clients using this "guide" https://forum.pfsense.org/index.php?topic=87655.msg488294#msg488294
Virtual IP network for mobile clients is configured as 10.0.y.y
When I get the successful connection it looks like traffic is not moving at all between client and firewall. Mostly it crashes with this panicFatal double fault:
eip = 0xc0cd225f
esp = 0xe0783fe8
ebp = 0xe0784050
cpuid = 1; apic id = 01
panic: double fault
cpuid = 1
KDB: enter: panicFull version is
http://pastebin.com/mSLw8QfX
-
Currently disabled IP Random id generation (Insert a stronger id into IP header of packets passing through the filter.)
Testing… -
So far so good… no problems currently with IPSEC uptime and panics. Using Stronswan android client 12 hours already.
Previous record was 1 minute with IP Random id generation enabled -
@ermal:
For example
kern.timecounter.hardware: TSC-low
kern.timecounter.choice: TSC-low(1000) ACPI-fast(900) i8254(0) HPET(950) dummy(-1000000)On my ALIX board I have the choice between the following time sources: TSC(800) i8254(0) dummy(-1000000)
Changing the timesource with
sysctl kern.timecounter.hardware=i8254 kern.timecounter.hardware: TSC -> i8254does not have an effect on this issue. After changing the timesource IPSEC keeps crashing on first ping from remote.
Here are the latest dump with timesource i8254:
Fatal double fault: eip = 0xc0cef7e0 esp = 0xc8b69000 ebp = 0xc8b69008 cpuid = 0; apic id = 00 panic: double fault cpuid = 0 KDB: enter: panic [ thread pid 12 tid 100015 ] Stopped at kdb_enter+0x3d: movl $0,kdb_why db:0:kdb.enter.default> textdump set textdump set db:0:kdb.enter.default> capture on db:0:kdb.enter.default> run lockinfo db:1:lockinfo> show locks No such command db:1:locks> show alllocks No such command db:1:alllocks> show lockedvnods Locked vnodes db:0:kdb.enter.default> show pcpu cpuid = 0 dynamic pcpu = 0x5e7a00 curthread = 0xc4317c40: pid 12 "swi5: fast taskq" curpcb = 0xc8b6ad60 fpcurthread = none idlethread = 0xc426d000: tid 100003 "idle: cpu0" APIC ID = 0 currentldt = 0x50 db:0:kdb.enter.default> bt Tracing pid 12 tid 100015 td 0xc4317c40 kdb_enter(c142c723,c142c723,c15ea443,c205b4c8,0,...) at kdb_enter+0x3d/frame 0xc205b480 panic(c15ea443,0,0,0,c8b69008,...) at panic+0x144/frame 0xc205b4bc dblfault_handler() at dblfault_handler+0xab/frame 0xc205b4bc --- trap 0x17, eip = 0xc0cef7e0, esp = 0xc8b69000, ebp = 0xc8b69008 --- critical_exit(c1e3718c,c0cd1bf4,0,0,0,...) at critical_exit/frame 0xc8b69008 i8254_get_timecount(c446f618,c1e049c4,34,0,0,...) at i8254_get_timecount+0x141/frame 0xc8b69030 tc_windup(1,0,c142675e,219,0,...) at tc_windup+0x45/frame 0xc8b69080 hardclock_cnt(1,0,c1e3718c,0,0,...) at hardclock_cnt+0x447/frame 0xc8b690e8 handleevents(0,0,0,0,0,...) at handleevents+0xee/frame 0xc8b69138 timercb(c446f640,0,c4268310,0,0,...) at timercb+0x3b9/frame 0xc8b69198 clkintr(c446f600,0,c4317c40,0,c1e3718c,...) at clkintr+0xfc/frame 0xc8b691bc intr_event_handle(c426bb80,c8b69228,c42685f0,c4ead154,c1e3718c,...) at intr_event_handle+0x85/frame 0xc8b691dc intr_execute_handlers(c1e3718c,c8b69228) at intr_execute_handlers+0x42/frame 0xc8b691fc atpic_handle_intr(0,c8b69228) at atpic_handle_intr+0x5a/frame 0xc8b69218 Xatpic_intr0() at Xatpic_intr0+0x22/frame 0xc8b69218 --- interrupt, eip = 0xc0dd89f7, esp = 0xc8b69268, ebp = 0xc8b69290 --- rn_match(c8b692c8,c4ead100,c8b692c8,c0e22d00,c1e14bb8,...) at rn_match+0x17/frame 0xc8b69290 pfr_match_addr(c4ec0000,c4c3e832,2,c48ff600,0,...) at pfr_match_addr+0xd5/frame 0xc8b692f0 pf_normalize_ip(c8b6962c,2,c4a24800,c8b69584,c8b69528,...) at pf_normalize_ip+0x2d6/frame 0xc8b69388 pf_test(2,c424f400,c8b6962c,0,c8b696c8,...) at pf_test+0x246/frame 0xc8b695e4 pf_check_out(0,c8b6962c,c424f400,2,0,...) at pf_check_out+0x4b/frame 0xc8b69608 pfil_run_hooks(c20a1c14,c8b696f4,c424f400,2,0,...) at pfil_run_hooks+0x88/frame 0xc8b69660 ip_output(c48f7b00,0,0,2,0,...) at ip_output+0xaac/frame 0xc8b69718 ipsec_process_done(c48f7b00,c4e94180,0,f228ae0e,c48f7b00,...) at ipsec_process_done+0x3cf/frame 0xc8b69768 esp_output_cb(c60c2bf4,c1e21524,c8b697f8,c0f9f5e1,c48f7b00,...) at esp_output_cb+0x3cd/frame 0xc8b697c0 crypto_done(c60c2bf4,c48f7b00,8c,c,c8b699a8,...) at crypto_done+0x99/frame 0xc8b697f8 swcr_process(c4269080,c60c2bf4,0,c8b69a60,c0ccd50d,...) at swcr_process+0x6e/frame 0xc8b69a00 crypto_invoke(0,40,0,0,0,...) at crypto_invoke+0x79/frame 0xc8b69a38 crypto_dispatch(c60c2bf4,c145696b,375,c8b69abb,c4c3e828,...) at crypto_dispatch+0x64/frame 0xc8b69a60 esp_output(c48f7b00,c4e94180,0,14,9,...) at esp_output+0x91d/frame 0xc8b69ae0 ipsec4_process_packet(c48f7b00,c4e94180,0,0,0,...) at ipsec4_process_packet+0x312/frame 0xc8b69b70 ip_ipsec_output(c8b69c34,0,c8b69c30,c8b69c2c,0,...) at ip_ipsec_output+0x1c8/frame 0xc8b69ba0 ip_output(c48f7b00,0,0,0,0,...) at ip_output+0xa2f/frame 0xc8b69c58 icmp_reflect(c8b69d28,10,0,c8b69d04,90000,...) at icmp_reflect+0x5b5/frame 0xc8b69cb0 icmp_input(c48f7b00,14,c8b69d8c,1,c8b69ddc,...) at icmp_input+0x9b9/frame 0xc8b69d78 ip_input(c48f7b00,c20a184c,c4317c40,0,c8b69e50,...) at ip_input+0x295/frame 0xc8b69ddc netisr_dispatch_src(1,0,c48f7b00) at netisr_dispatch_src+0x8b/frame 0xc8b69e24 netisr_dispatch(1,c48f7b00,1,102,0,...) at netisr_dispatch+0x20/frame 0xc8b69e38 _ipip_input(c48f7b00,14,4,1,3c,...) at _ipip_input+0x650/frame 0xc8b69e80 encap4_input(c48f7b00,14,0,c0dd4f18,1,...) at encap4_input+0x210/frame 0xc8b69ee0 ip_input(c48f7b00,2,40,0,0,...) at ip_input+0x295/frame 0xc8b69f48 netisr_dispatch_src(1,611ea2cb,c48f7b00,101,68,...) at netisr_dispatch_src+0x8b/frame 0xc8b69f90 ipsec4_common_input_cb(c48f7b00,c67d6d00,14,9,0,...) at ipsec4_common_input_cb+0x276/frame 0xc8b69fd8 esp_input_cb(c60c2bf4,c053cd1d,c4f59cf8,e,c4c3e89e,...) at esp_input_cb+0x772/frame 0xc8b6a068 crypto_done(c60c2bf4,c8b6a240,10,10,c8b6a240,...) at crypto_done+0x99/frame 0xc8b6a0a0 swcr_process(c4269080,c60c2bf4,0,c8b6a308,c0ccd50d,...) at swcr_process+0x6e/frame 0xc8b6a2a8 crypto_invoke(0,c4c3e8ae,c60850b8,c,c60850b8,...) at crypto_invoke+0x79/frame 0xc8b6a2e0 crypto_dispatch(c60c2bf4,c145696b,1c8,c60850b8,c57dc528,...) at crypto_dispatch+0x64/frame 0xc8b6a308 esp_input(c48f7b00,c67d6d00,14,9,d0,...) at esp_input+0x771/frame 0xc8b6a370 ipsec_common_input(9,2,32,c8b6a408) at ipsec_common_input+0x4f7/frame 0xc8b6a3dc ipsec4_common_input(c48f7b00,14,32) at ipsec4_common_input+0x39/frame 0xc8b6a3f4 esp4_input(c48f7b00,14,c424f400,1,0,...) at esp4_input+0x20/frame 0xc8b6a408 ip_input(c48f7b00,801,c4902a00,c43c0f68,c0d1f5ce,...) at ip_input+0x295/frame 0xc8b6a470 netisr_dispatch_src(1,0,c48f7b00) at netisr_dispatch_src+0x8b/frame 0xc8b6a4b8 netisr_dispatch(1,c48f7b00,c4902a00,c431a000,6e,...) at netisr_dispatch+0x20/frame 0xc8b6a4cc ng_iface_rcvdata(c4793680,c4a4a000,c4317c40,c4f20600,0,...) at ng_iface_rcvdata+0xea/frame 0xc8b6a4f4 ng_apply_item(0,c4902a00,14,c4a632f4,c8b6a584,...) at ng_apply_item+0x22d/frame 0xc8b6a550 ng_snd_item(c4a4a000,0,c4793700,0,0,...) at ng_snd_item+0x1a0/frame 0xc8b6a584 ng_tcpmss_rcvdata(c4793800,c4a4a000,46507c40,2f1c645d,c8b6a68c,...) at ng_tcpmss_rcvdata+0xac/frame 0xc8b6a5cc ng_apply_item(0,34,5dc,c8b6a760,0,...) at ng_apply_item+0x22d/frame 0xc8b6a628 ng_snd_item(c4a4a000,0,c47b9000,0,c4a4a000,...) at ng_snd_item+0x1a0/frame 0xc8b6a65c ng_ppp_comp_recv(21,0,1,c48f7b00,c4a4a000,...) at ng_ppp_comp_recv+0x158/frame 0xc8b6a688 ng_ppp_crypt_recv(21,0,d80e0e6a,c8b6a820,c0ee51a9,...) at ng_ppp_crypt_recv+0x70/frame 0xc8b6a6a4 ng_ppp_rcvdata(c4793b00,c4a4a000,c4902a00,0,c4317c40,...) at ng_ppp_rcvdata+0x2e4/frame 0xc8b6a700 ng_apply_item(0,0,0,246,c209f300,...) at ng_apply_item+0x22d/frame 0xc8b6a75c ng_snd_item(c4a4a000,0,c4793880,0,362cd2a1,...) at ng_snd_item+0x1a0/frame 0xc8b6a790 ng_tee_rcvdata(c4793c00,c4a4a000,1,0,c8b6a7e4,...) at ng_tee_rcvdata+0x156/frame 0xc8b6a7b8 ng_apply_item(0,c4c3e812,6,c4a645e0,1,...) at ng_apply_item+0x22d/frame 0xc8b6a814 ng_snd_item(c4a4a000,0,c4793c80,0,c46a9900,...) at ng_snd_item+0x1a0/frame 0xc8b6a848 ng_pppoe_rcvdata_ether(c4793d00,c4a4a000,c4a645e0,34,1c5,...) at ng_pppoe_rcvdata_ether+0x2a3/frame 0xc8b6a8c4 ng_apply_item(0,c47b0000,c48f7b00,0,c8b6a954,...) at ng_apply_item+0x22d/frame 0xc8b6a920 ng_snd_item(c4a4a000,0,c47b9a00,0,c47b0000,...) at ng_snd_item+0x1a0/frame 0xc8b6a954 ng_ether_input_orphan(c47b0000,c48f7b00,0,b1ce9d7,a1d20db3,...) at ng_ether_input_orphan+0x66/frame 0xc8b6a974 ether_demux(c47b0000,c48f7b00,6,c4317c40,c4317c40,...) at ether_demux+0x1f9/frame 0xc8b6a9a0 ether_nh_input(c48f7b00,c8b6aa14,0,0,0,...) at ether_nh_input+0x37e/frame 0xc8b6a9f0 netisr_dispatch_src(9,0,c48f7b00) at netisr_dispatch_src+0x8b/frame 0xc8b6aa38 netisr_dispatch(9,c48f7b00) at netisr_dispatch+0x20/frame 0xc8b6aa4c ether_input(c47b0000,c48f7b00,0,c8b6aae0,c0cd1bf4,...) at ether_input+0x19/frame 0xc8b6aa5c vlan_input(c424fc00,c48f7b00,c4317c40,c4bfd008,c43d0100,...) at vlan_input+0x1a8/frame 0xc8b6aa8c ether_demux(c424fc00,c48f7b00,6,7f8,c4bfd008,...) at ether_demux+0xaf/frame 0xc8b6aab8 ether_nh_input(c48f7b00,c4bbd200,2f4,c8b6ab70,c0b4e3ba,...) at ether_nh_input+0x37e/frame 0xc8b6ab04 netisr_dispatch_src(9,0,c48f7b00) at netisr_dispatch_src+0x8b/frame 0xc8b6ab4c netisr_dispatch(9,c48f7b00) at netisr_dispatch+0x20/frame 0xc8b6ab60 ether_input(c424fc00,c48f7b00,c43a4000,745,c8b6abb8,...) at ether_input+0x19/frame 0xc8b6ab70 vr_rxeof(0,0,c4317c40,c8b6abd8,46,...) at vr_rxeof+0x1f1/frame 0xc8b6abb8 vr_int_task(c43cc000,1,c8b6ac18,c12bc042,c426bb80,...) at vr_int_task+0x123/frame 0xc8b6abe8 taskqueue_run_locked(c8b6ac88,c128c712,0,c8b6ac44,c4310008,...) at taskqueue_run_locked+0xee/frame 0xc8b6ac2c taskqueue_run(c4269980) at taskqueue_run+0xa3/frame 0xc8b6ac50 taskqueue_fast_run(0,0,246,0,0,...) at taskqueue_fast_run+0x11/frame 0xc8b6ac5c intr_event_execute_handlers(109,c4269900,c1428033,55a,0,...) at intr_event_execute_handlers+0xaa/frame 0xc8b6ac88 ithread_loop(c41b5e70,c8b6ad08,0,0,0,...) at ithread_loop+0x80/frame 0xc8b6acc4 fork_exit(c0cb63f0,c41b5e70,c8b6ad08) at fork_exit+0xa3/frame 0xc8b6acf4 fork_trampoline() at fork_trampoline+0x8/frame 0xc8b6acf4 --- trap 0, eip = 0, esp = 0xc8b6ad40, ebp = 0 --- db:0:kdb.enter.default> ps pid ppid pgrp uid state wmesg wchan cmd 30982 285 285 0 R php-fpm 71440 73267 21 0 S nanslp 0xc1efd568 sleep 60001 59799 60001 0 Ss (threaded) charon 100155 S uwait 0xc57e6580 charon 100154 S uwait 0xc57e6500 charon 100153 S uwait 0xc4a6f280 charon 100152 S uwait 0xc57e6400 charon 100151 S uwait 0xc57e6780 charon 100150 S uwait 0xc57e7b80 charon 100149 S uwait 0xc57e7c00 charon 100148 S uwait 0xc57e6a80 charon 100147 S uwait 0xc57e6800 charon 100146 S uwait 0xc4e94b00 charon 100145 S uwait 0xc57e6600 charon 100144 S uwait 0xc4a6f380 charon 100143 S select 0xc46af6e4 charon 100142 S select 0xc46af724 charon 100141 S accept 0xc57d61e6 charon 100140 S uwait 0xc57e9700 charon 100060 S sigwait 0xc674c000 charon 59799 1 59799 0 Ss select 0xc6703824 starter 85529 89861 89861 0 S+ ttyin 0xc4313470 sh 90198 78569 90198 0 Ss (threaded) sshlockout_pf 100124 S nanslp 0xc1efd568 sshlockout_pf 100055 S piperd 0xc47f47f8 sshlockout_pf 89861 1 89861 0 Ss+ wait 0xc48ac8d0 login 89557 1 1 0 S nanslp 0xc1efd568 getty 87839 87796 87796 0 S nanslp 0xc1efd568 minicron 87796 1 87796 0 Ss wait 0xc47985e0 minicron 87654 87281 87281 0 S nanslp 0xc1efd568 minicron 87281 1 87281 0 Ss wait 0xc48ab000 minicron 86900 86382 86382 0 S nanslp 0xc1efd568 minicron 86382 1 86382 0 Ss wait 0xc4f96000 minicron 84022 1 84022 0 Ss nanslp 0xc1efd568 cron 78569 1 78569 0 Ss select 0xc4a361e4 syslogd 73996 1 73996 0 Ss select 0xc4a38ba4 igmpproxy 73267 1 21 0 S+ wait 0xc4f978d0 sh 65451 1 65451 0 Ss (threaded) ntpd 100056 S select 0xc46ac7a4 ntpd 60741 1 60741 0 Ss (threaded) filterdns 100101 S uwait 0xc4d6b500 signal-thread 100100 S uwait 0xc57e9680 filterdns 54653 1 54653 1002 Ss select 0xc4437ba4 dhcpd 48648 1 48359 65534 S select 0xc4a362e4 dnsmasq 44941 1 44633 0 S kqread 0xc4ea7b80 lighttpd 35762 35604 35604 0 S piperd 0xc47f44c8 rrdtool 35604 1 35604 0 Ss select 0xc4a365a4 apinger 32684 1 32684 0 Ss select 0xc4a36824 inetd 32208 1 32208 0 Ss select 0xc4a36224 openvpn 32123 1 32123 0 Ss bpf 0xc4314c00 filterlog 30648 1 30648 0 Ss select 0xc4a36664 openvpn 29293 1 29293 0 Ss select 0xc4a366a4 openvpn 28370 1 28370 0 Ss select 0xc4a36ba4 openvpn 25601 1 25601 0 Ss select 0xc4a376e4 hostapd 22824 1 22824 65 Ss select 0xc4a37c64 dhclient 19122 1 19122 0 Ss select 0xc4a37ce4 dhclient 8090 1 8090 0 Ss (threaded) sshlockout_pf 100073 S nanslp 0xc1efd568 sshlockout_pf 100072 S uwait 0xc47bae80 sshlockout_pf 7521 1 7521 0 Ss select 0xc4a38924 sshd 6233 1 6233 0 Ss (threaded) mpd5 100068 S select 0xc41d6ba4 mpd5 320 1 320 0 Ss select 0xc4437aa4 devd 302 300 300 0 S kqread 0xc47baa80 check_reload_status 300 1 300 0 Ss kqread 0xc47baa00 check_reload_status 285 1 285 0 Ss kqread 0xc4792700 php-fpm 54 0 0 0 DL mdwait 0xc442b000 [md1] 49 0 0 0 DL mdwait 0xc442d800 [md0] 20 0 0 0 DL syncer 0xc1f1cac4 [syncer] 19 0 0 0 DL vlruwt 0xc4799000 [vnlru] 18 0 0 0 DL psleep 0xc1f1c204 [bufdaemon] 17 0 0 0 DL pollid 0xc1efbf30 [idlepoll] 9 0 0 0 DL pgzero 0xc2047d20 [pagezero] 8 0 0 0 DL psleep 0xc2047a44 [vmdaemon] 7 0 0 0 DL psleep 0xc20a7484 [pagedaemon] 6 0 0 0 DL waiting_ 0xc20a1d8c [sctp_iterator] 5 0 0 0 DL pftm 0xc0f622c0 [pf purge] 16 0 0 0 DL (threaded) [usb] 100038 D - 0xc43a8d34 [usbus1] 100037 D - 0xc43a8d04 [usbus1] 100036 D - 0xc43a8cd4 [usbus1] 100035 D - 0xc43a8ca4 [usbus1] 100034 D - 0xc443cb5c [usbus0] 100033 D - 0xc443cb2c [usbus0] 100032 D - 0xc443cafc [usbus0] 100031 D - 0xc443cacc [usbus0] 4 0 0 0 DL (threaded) [cam] 100045 D - 0xc1e3e4a8 [scanner] 100017 D - 0xc1e3e600 [doneq0] 3 0 0 0 DL crypto_r 0xc2046978 [crypto returns] 2 0 0 0 DL crypto_w 0xc20468b8 [crypto] 15 0 0 0 DL - 0xc1e58680 [rand_harvestq] 14 0 0 0 DL (threaded) [geom] 100010 D - 0xc209dde0 [g_down] 100009 D - 0xc209dddc [g_up] 100008 D - 0xc209ddd8 [g_event] 13 0 0 0 DL sleep 0xc1e14bb8 [ng_queue0] 12 0 0 0 RL (threaded) [intr] 100043 I [swi1: pfsync] 100041 I [swi1: pf send] 100039 I [swi0: uart uart] 100030 I [irq12: ohci0 ehci0] 100029 I [irq15: vr2 ata1] 100028 I [irq14: ata0] 100025 I [irq9: ath0] 100023 I [swi6: Giant taskq] 100021 I [swi6: task queue] 100015 Run CPU 0 [swi5: fast taskq] 100006 I [swi3: vm] 100005 I [swi4: clock] 100004 I [swi1: netisr 0] 11 0 0 0 RL [idle: cpu0] 1 0 1 0 SLs wait 0xc42662f0 [init] 10 0 0 0 DL audit_wo 0xc20a5d88 [audit] 0 0 0 0 DLs (threaded) [kernel] 100044 D - 0xc4269800 [CAM taskq] 100027 D - 0xc43cf700 [ath0 net80211 taskq] 100026 D - 0xc43cf780 [ath0 taskq] 100024 D - 0xc4269180 [thread] 100022 D - 0xc4269300 [ffs_trim taskq] 100020 D - 0xc4269480 [acpi_task_2] 100019 D - 0xc4269480 [acpi_task_1] 100018 D - 0xc4269480 [acpi_task_0] 100016 D - 0xc4269880 [kqueue taskq] 100011 D - 0xc426aa80 [firmware taskq] 100000 D swapin 0xc209de64 [swapper] db:0:kdb.enter.default>[/thread] -
And you don't have "IP Random id generation" enabled?
-
@w0w:
And you don't have "IP Random id generation" enabled?
"IP Random id generation" is off. This setting I have never changed. Otherwise, I have only MSS clamping enabled and the Unity plugin disabled. "IPsec Mobile Client Support" is also turned off. All other settings are defaults.
Apparently the kernel timesource is not the cause of the error. Could someone have another look at the dump? What information is still needed?
-
We need to use bugtracker to report the bug. Please report this issue here https://redmine.pfsense.org/
Don't link the forum page, post your dumps and I'll add mine too.
FYI I don't have changed MSS and unity settings. -
https://redmine.pfsense.org/issues/4454
-
Similar issue here. If I connect via VPN, PFSense stops and restarts - the IPSEC connection lasts between 30 seconds and 5 minutes. This issue is 100% reproducible - I've crashed my box about six times today.
Fatal double fault:
eip = 0xc12c62a8
esp = 0xecf4cff8
ebp = 0xecf4d000
cpuid = 0; apic id = 00
panic: double fault
cpuid = 0
KDB: enter: panic
panic.txt0600001412471723700 7136 ustarrootwheeldouble faultversion.txt06000025112471723700 7614 ustarrootwheelFreeBSD 10.1-RELEASE-p4 #0 36d7dec(releng/10.1)-dirty: Thu Jan 22 15:12:38 CST 2015
root@pfsense-22-i386-builder:/usr/obj.i386/usr/pfSensesrc/src/sys/pfSense_SMP.10PFSense 2.2 - upgraded from 2.1.5
Hardware:
CPU: Intel(R) Atom(TM) CPU N270 @ 1.60GHz
Mobo: KINO-945GSE
Storage: 2 GB CF Card
Dual LAN: Realtek PCIe 8111CP GbE controllerIPSEC Details:
Mobile Client
No IP Compression
Unity plugin disabled
IKE v1
Virtual IP Address Assigned to Clients
IP Random ID Generation at default value (default is 0: sequential IP IDs)Interfaces:
RE0: WAN: PPOE
RE1: LAN/OPT1/OPT2 using VLAN taggingMay try a clean install of v2.2 if you think there's any mileage in it.
-
@afasoas: are you running 32 or 64 bit version? So far I think all the reports are coming from x86 versions. Is this reproducible under x64??
-
32 bit.
Well spotted.
I will add my crash dump to the bug tracker shortly. -
Upgraded my hardware so I could run the 64-bit version. No issues to report thus far. IPSEC seems solid and stable.
-
Great solution, but this is like cutting the head and sewing back a new one, more "brainful".
It could be also driver Ethernet issue with physical low memory installed. I can only wait when somebody really smart will check our crash dumps to find out the reason of double triple crash and panic. -
@w0w:
Great solution, but this is like cutting the head and sewing back a new one, more "brainful".
It could be also driver Ethernet issue with physical low memory installed. I can only wait when somebody really smart will check our crash dumps to find out the reason of double triple crash and panic.I appreciate that this isn't the most helpful solution. I just wanted to confirm that the problem went away using the same configuration with a 64-bit version.
On Edit: I realise that you are using an Atom D2500 - you can run the 64-bit version of pfSense on it, if that helps?
-
I can, really, but what if I don't need it? My typical memory usage is less then 6% and CPU is mostly 90% in peak (300Mbit internet, three clients). I can buy some XEON based proliant G8 but why? :)
Maybe I'll move to x64 platform if we can't trust x86 anymore. But I need an answer for the question. DO we really need to move to x64 just because x86 is not supported or what?
I think i'll wait for answer before buying some needless hardware. :)
Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.