Reproducible kernel panic with pfSense 2.2 and IPSEC



  • Alright, I grabbed one Alix box and changed:

    kern.timecounter.hardware: TSC -> i8254
    

    Will report back if this box keeps coredumping.

    It'd be great if sh0gun could test this on his reproducible scenario

    EDIT: In case it is not clear, to temporarily set the value you can run "sysctl kern.timecounter.hardware=i8254" on the shell, or force the tunable through the advanced options. Anyway, no idea about this setting



  • I'm having the same issue, in my case it crashes when I keep IP compression enabled in the advanced IPsec options. Try disabling it if enabled, maybe it helps in your case too.

    @sh0gun:

    I have a reproducible kernel panic with pfSense 2.2 and IPSEC.

    My setup: ALIX 2D13, IPSEC with one P1. IKEv1, PSK, aggressive. One P2 (net-to-net tunnel). pfSense must be the responder upon P2 connection. Now send a ping from any host in the remote network to the local pfSense box -> immediate crash.

    Fatal double fault:
    eip = 0xc12c6ac8
    esp = 0xc8b68ff8
    ebp = 0xc8b69000
    cpuid = 0; apic id = 00
    panic: double fault
    cpuid = 0
    KDB: enter: panic
    [ thread pid 12 tid 100015 ]
    Stopped at      kdb_enter+0x3d: movl    $0,kdb_why
    db:0:kdb.enter.default> textdump set
    textdump set
    db:0:kdb.enter.default>  capture on
    db:0:kdb.enter.default>  run lockinfo
    db:1:lockinfo> show locks
    No such command
    db:1:locks>  show alllocks
    No such command
    db:1:alllocks>  show lockedvnods
    Locked vnodes
    db:0:kdb.enter.default>  show pcpu
    cpuid        = 0
    dynamic pcpu = 0x5e7a00
    curthread    = 0xc4317c40: pid 12 "swi5: fast taskq"
    curpcb       = 0xc8b6ad60
    fpcurthread  = none
    idlethread   = 0xc426d000: tid 100003 "idle: cpu0"
    APIC ID      = 0
    currentldt   = 0x50
    db:0:kdb.enter.default>  bt
    Tracing pid 12 tid 100015 td 0xc4317c40
    kdb_enter(c142c723,c142c723,c15ea443,c205b4c8,0,...) at kdb_enter+0x3d/frame 0xc205b480
    panic(c15ea443,0,0,0,c8b69000,...) at panic+0x144/frame 0xc205b4bc
    dblfault_handler() at dblfault_handler+0xab/frame 0xc205b4bc
    --- trap 0x17, eip = 0xc12c6ac8, esp = 0xc8b68ff8, ebp = 0xc8b69000 ---
    cpu_fill_vdso_timehands(c8b69030,80000000,c8b69098,c0ce65b6,c8b69030,...) at cpu_fill_vdso_timehands+0x18/frame 0xc8b69000
    tc_fill_vdso_timehands(c8b69030,7498,2,ed7e,fffffffe,...) at tc_fill_vdso_timehands+0x63/frame 0xc8b69010
    timekeep_push_vdso(0,80000000,1dafce93,0,0,...) at timekeep_push_vdso+0x36/frame 0xc8b69098
    tc_windup(1,0,c142675e,219,0,...) at tc_windup+0x321/frame 0xc8b690e8
    hardclock_cnt(1,0,0,0,c4317c40,...) at hardclock_cnt+0x447/frame 0xc8b69150
    handleevents(0,0,0,c8b691f0,c0d12b54,...) at handleevents+0xee/frame 0xc8b691a0
    timercb(c446f640,0,0,c41b1e00,0,...) at timercb+0x3b9/frame 0xc8b691fc
    clkintr(c446f600,0,c4317c40,0,c1e3718c,...) at clkintr+0xfc/frame 0xc8b69220
    intr_event_handle(c426bb80,c8b6928c,c4268310,6,c1e3718c,...) at intr_event_handle+0x85/frame 0xc8b69240
    intr_execute_handlers(c1e3718c,c8b6928c) at intr_execute_handlers+0x42/frame 0xc8b69260
    atpic_handle_intr(0,c8b6928c) at atpic_handle_intr+0x5a/frame 0xc8b6927c
    Xatpic_intr0() at Xatpic_intr0+0x22/frame 0xc8b6927c
    --- interrupt, eip = 0xc1291015, esp = 0xc8b692cc, ebp = 0xc8b69388 ---
    in_cksum_skip(c48fa200,14,0,0,c8b69528,...) at in_cksum_skip+0x215/frame 0xc8b69388
    pf_test(2,c43c9400,c8b6962c,0,c8b696c8,...) at pf_test+0x2899/frame 0xc8b695e4
    pf_check_out(0,c8b6962c,c43c9400,2,0,...) at pf_check_out+0x4b/frame 0xc8b69608
    pfil_run_hooks(c20a1c14,c8b696f4,c43c9400,2,0,...) at pfil_run_hooks+0x88/frame 0xc8b69660
    ip_output(c48fa200,0,0,2,0,...) at ip_output+0xaac/frame 0xc8b69718
    ipsec_process_done(c48fa200,c57de480,0,5e29211a,c48fa200,...) at ipsec_process_done+0x3cf/frame 0xc8b69768
    esp_output_cb(c5fe0bf4,c1e21524,c8b697f8,c0f9f5e1,c48fa200,...) at esp_output_cb+0x3cd/frame 0xc8b697c0
    crypto_done(c5fe0bf4,c48fa200,8c,c,c8b699a8,...) at crypto_done+0x99/frame 0xc8b697f8
    swcr_process(c4269080,c5fe0bf4,0,c8b69a60,c0ccd50d,...) at swcr_process+0x6e/frame 0xc8b69a00
    crypto_invoke(0,40,0,0,0,...) at crypto_invoke+0x79/frame 0xc8b69a38
    crypto_dispatch(c5fe0bf4,c145696b,375,c8b69abb,c495b028,...) at crypto_dispatch+0x64/frame 0xc8b69a60
    esp_output(c48fa200,c57de480,0,14,9,...) at esp_output+0x91d/frame 0xc8b69ae0
    ipsec4_process_packet(c48fa200,c57de480,0,0,0,...) at ipsec4_process_packet+0x312/frame 0xc8b69b70
    ip_ipsec_output(c8b69c34,0,c8b69c30,c8b69c2c,0,...) at ip_ipsec_output+0x1c8/frame 0xc8b69ba0
    ip_output(c48fa200,0,0,0,0,...) at ip_output+0xa2f/frame 0xc8b69c58
    icmp_reflect(c8b69d28,10,0,c8b69d04,90000,...) at icmp_reflect+0x5b5/frame 0xc8b69cb0
    icmp_input(c48fa200,14,50,1,c8b69ddc,...) at icmp_input+0x9b9/frame 0xc8b69d78
    ip_input(c48fa200,c20a184c,c4317c40,0,c8b69e50,...) at ip_input+0x295/frame 0xc8b69ddc
    netisr_dispatch_src(1,0,c48fa200) at netisr_dispatch_src+0x8b/frame 0xc8b69e24
    netisr_dispatch(1,c48fa200,1,102,0,...) at netisr_dispatch+0x20/frame 0xc8b69e38
    _ipip_input(c48fa200,14,4,1,3c,...) at _ipip_input+0x650/frame 0xc8b69e80
    encap4_input(c48fa200,14,0,0,1,...) at encap4_input+0x210/frame 0xc8b69ee0
    ip_input(c48fa200,2,40,0,0,...) at ip_input+0x295/frame 0xc8b69f48
    netisr_dispatch_src(1,ca095cc9,c48fa200,101,68,...) at netisr_dispatch_src+0x8b/frame 0xc8b69f90
    ipsec4_common_input_cb(c48fa200,c4f28300,14,9,0,...) at ipsec4_common_input_cb+0x276/frame 0xc8b69fd8
    esp_input_cb(c5fe0bf4,c053cd1d,c4314af8,e,c495b09e,...) at esp_input_cb+0x772/frame 0xc8b6a068
    crypto_done(c5fe0bf4,c8b6a240,10,10,c8b6a240,...) at crypto_done+0x99/frame 0xc8b6a0a0
    swcr_process(c4269080,c5fe0bf4,0,c8b6a308,c0ccd50d,...) at swcr_process+0x6e/frame 0xc8b6a2a8
    crypto_invoke(0,c495b0ae,c57dda38,c,c57dda38,...) at crypto_invoke+0x79/frame 0xc8b6a2e0
    crypto_dispatch(c5fe0bf4,c145696b,1c8,c57dda38,c58c01b8,...) at crypto_dispatch+0x64/frame 0xc8b6a308
    esp_input(c48fa200,c4f28300,14,9,d0,...) at esp_input+0x771/frame 0xc8b6a370
    ipsec_common_input(9,2,32,c8b6a408) at ipsec_common_input+0x4f7/frame 0xc8b6a3dc
    ipsec4_common_input(c48fa200,14,32) at ipsec4_common_input+0x39/frame 0xc8b6a3f4
    esp4_input(c48fa200,14,c43c9400,1,0,...) at esp4_input+0x20/frame 0xc8b6a408
    ip_input(c48fa200,c0cf6261,0,80000000,1dafce93,...) at ip_input+0x295/frame 0xc8b6a470
    netisr_dispatch_src(1,0,c48fa200) at netisr_dispatch_src+0x8b/frame 0xc8b6a4b8
    netisr_dispatch(1,c48fa200,c492482a,c8b6a540,c0f90d95,...) at netisr_dispatch+0x20/frame 0xc8b6a4cc
    ng_iface_rcvdata(c4793680,c4a3e000,62,10000000,0,...) at ng_iface_rcvdata+0xea/frame 0xc8b6a4f4
    ng_apply_item(0,c2425c80,62,67d16f46,5e7a00,...) at ng_apply_item+0x22d/frame 0xc8b6a550
    ng_snd_item(c4a3e000,0,c47b9000,0,0,...) at ng_snd_item+0x1a0/frame 0xc8b6a584
    ng_tcpmss_rcvdata(c4793700,c4a3e000,c0f66a20,c498e2e0,c4317c40,...) at ng_tcpmss_rcvdata+0xac/frame 0xc8b6a5cc
    ng_apply_item(0,c0d12971,c498e000,0,17c83,...) at ng_apply_item+0x22d/frame 0xc8b6a628
    ng_snd_item(c4a3e000,0,c4793780,0,c4a3e000,...) at ng_snd_item+0x1a0/frame 0xc8b6a65c
    ng_ppp_comp_recv(21,0,1,c48fa200,c4a3e000,...) at ng_ppp_comp_recv+0x158/frame 0xc8b6a688
    ng_ppp_crypt_recv(21,0,c4317c40,c8b6a6c4,246,...) at ng_ppp_crypt_recv+0x70/frame 0xc8b6a6a4
    ng_ppp_rcvdata(c4793880,c4a3e000,c48f2900,c4d7d1fc,c4d7d1a8,...) at ng_ppp_rcvdata+0x2e4/frame 0xc8b6a700
    ng_apply_item(0,c4317c40,c48dc8dc,c48f2900,c48dc9d4,...) at ng_apply_item+0x22d/frame 0xc8b6a75c
    ng_snd_item(c4a3e000,0,c4793b00,0,0,...) at ng_snd_item+0x1a0/frame 0xc8b6a790
    ng_tee_rcvdata(c4793c00,c4a3e000,8ce9fb06,0,d7bb17a0,...) at ng_tee_rcvdata+0x156/frame 0xc8b6a7b8
    ng_apply_item(0,c495b012,6,c8b6a838,1,...) at ng_apply_item+0x22d/frame 0xc8b6a814
    ng_snd_item(c4a3e000,0,c4793c80,0,210,...) at ng_snd_item+0x1a0/frame 0xc8b6a848
    ng_pppoe_rcvdata_ether(c4793d00,c4a3e000,c424f400,c492481a,14,...) at ng_pppoe_rcvdata_ether+0x2a3/frame 0xc8b6a8c4
    ng_apply_item(0,c47afc00,c48fa200,0,c8b6a954,...) at ng_apply_item+0x22d/frame 0xc8b6a920
    ng_snd_item(c4a3e000,0,c47b9a00,0,c47afc00,...) at ng_snd_item+0x1a0/frame 0xc8b6a954
    ng_ether_input_orphan(c47afc00,c48fa200,31517afb,694e9713,c8b6a990,...) at ng_ether_input_orphan+0x66/frame 0xc8b6a974
    ether_demux(c47afc00,c48fa200,6,0,d7bb17a0,...) at ether_demux+0x1f9/frame 0xc8b6a9a0
    ether_nh_input(c48fa200,c8b6aa14,0,0,0,...) at ether_nh_input+0x37e/frame 0xc8b6a9f0
    netisr_dispatch_src(9,0,c48fa200) at netisr_dispatch_src+0x8b/frame 0xc8b6aa38
    netisr_dispatch(9,c48fa200) at netisr_dispatch+0x20/frame 0xc8b6aa4c
    ether_input(c47afc00,c48fa200,0,c10215fd,0,...) at ether_input+0x19/frame 0xc8b6aa5c
    vlan_input(c424fc00,c48fa200,c10215fd,c4951808,c43d0100,...) at vlan_input+0x1a8/frame 0xc8b6aa8c
    ether_demux(c424fc00,c48fa200,6,7f8,c4951808,...) at ether_demux+0xaf/frame 0xc8b6aab8
    ether_nh_input(c48fa200,c48f8b00,18,c8b6ab70,c0b4e3ba,...) at ether_nh_input+0x37e/frame 0xc8b6ab04
    netisr_dispatch_src(9,0,c48fa200) at netisr_dispatch_src+0x8b/frame 0xc8b6ab4c
    netisr_dispatch(9,c48fa200) at netisr_dispatch+0x20/frame 0xc8b6ab60
    ether_input(c424fc00,c48fa200,c43cc000,27,c8b6abb8,...) at ether_input+0x19/frame 0xc8b6ab70
    vr_rxeof(c1292b00,0,40000000,c57c8310,0,...) at vr_rxeof+0x1f1/frame 0xc8b6abb8
    vr_int_task(c43cc000,1,c4317c40,c8b6ac04,c0cef78f,...) at vr_int_task+0x123/frame 0xc8b6abe8
    taskqueue_run_locked(c57c8310,2710,0,0,c4317c40,...) at taskqueue_run_locked+0xee/frame 0xc8b6ac2c
    taskqueue_run(c4269980) at taskqueue_run+0xa3/frame 0xc8b6ac50
    taskqueue_fast_run(0,0,246,0,87f59266,...) at taskqueue_fast_run+0x11/frame 0xc8b6ac5c
    intr_event_execute_handlers(109,c4269900,c1428033,55a,0,...) at intr_event_execute_handlers+0xaa/frame 0xc8b6ac88
    ithread_loop(c41b5e70,c8b6ad08,0,0,0,...) at ithread_loop+0x80/frame 0xc8b6acc4
    fork_exit(c0cb63f0,c41b5e70,c8b6ad08) at fork_exit+0xa3/frame 0xc8b6acf4
    fork_trampoline() at fork_trampoline+0x8/frame 0xc8b6acf4
    --- trap 0, eip = 0, esp = 0xc8b6ad40, ebp = 0 ---
    db:0:kdb.enter.default>  ps
      pid  ppid  pgrp   uid   state   wmesg     wchan    cmd
      503 66422    21     0  S       nanslp   0xc1efd568 sleep
    81506 71168 81506     0  Ss      (threaded)          sshlockout_pf
    100124                   S       nanslp   0xc1efd568 sshlockout_pf
    100087                   S       piperd   0xc47eb660 sshlockout_pf
    81187     1 81187     0  Ss+     ttyin    0xc4313470 getty
    80856     1     1     0  S       nanslp   0xc1efd568 getty
    79382 79108 79108     0  S       nanslp   0xc1efd568 minicron
    79108     1 79108     0  Ss      wait     0xc4a7e5e0 minicron
    78867 78573 78573     0  S       nanslp   0xc1efd568 minicron
    78573     1 78573     0  Ss      wait     0xc4a7ebc0 minicron
    78272 78070 78070     0  S       nanslp   0xc1efd568 minicron
    78070     1 78070     0  Ss      wait     0xc479a8d0 minicron
    77434   285   285     0  S       accept   0xc48be536 php-fpm
    76420     1 76420     0  Ss      nanslp   0xc1efd568 cron
    71168     1 71168     0  Ss      select   0xc46af6a4 syslogd
    66918     1 66918     0  Ss      select   0xc46af664 igmpproxy
    66422     1    21     0  S+      wait     0xc48a28d0 sh
    59761     1 59761     0  Ss      (threaded)          ntpd
    100116                   S       usem     0xc57df500 ntpd
    100067                   S       select   0xc4a2bde4 ntpd
    54651     1 54651     0  Ss      (threaded)          filterdns
    100098                   S       uwait    0xc4ec0a00 signal-thread
    100097                   S       uwait    0xc4ec0a80 filterdns
    54535 54039 54535     0  Ss      (threaded)          charon
    100114                   S       select   0xc46af8e4 charon
    100113                   S       uwait    0xc57df200 charon
    100112                   S       uwait    0xc4e8ea80 charon
    100111                   S       uwait    0xc57def00 charon
    100110                   S       uwait    0xc57de180 charon
    100109                   S       uwait    0xc57de100 charon
    100108                   S       uwait    0xc57de080 charon
    100107                   S       uwait    0xc47bce80 charon
    100106                   S       uwait    0xc47bce00 charon
    100105                   S       uwait    0xc47bcd80 charon
    100104                   S       uwait    0xc47bcd00 charon
    100103                   S       uwait    0xc4ec0d00 charon
    100102                   S       uwait    0xc57b3180 charon
    100101                   S       select   0xc46af8a4 charon
    100100                   S       accept   0xc4d7b886 charon
    100099                   S       uwait    0xc4a66480 charon
    100088                   S       sigwait  0xc58cb000 charon
    54039     1 54039     0  Ss      select   0xc46af864 starter
    50152     1 50152  1002  Ss      select   0xc4a2a2e4 dhcpd
    45824     1 45590 65534  S       select   0xc4a2a324 dnsmasq
    42452     1 42144     0  S       kqread   0xc4ebd200 lighttpd
    36088 36073 36073     0  S       piperd   0xc47eb4c8 rrdtool
    36073     1 36073     0  Ss      select   0xc4a2a5a4 apinger
    32981     1 32981     0  Ss      select   0xc4a2a824 inetd
    32537     1 32537     0  Ss      select   0xc4a2a964 openvpn
    32250     1 32250     0  Ss      bpf      0xc4314c00 filterlog
    31116     1 31116     0  Ss      select   0xc4a2a6a4 openvpn
    29208     1 29208     0  Ss      select   0xc4a2a764 openvpn
    28380     1 28380     0  Ss      select   0xc4a2b0a4 openvpn
    26257     1 26257     0  Ss      select   0xc4a2b624 hostapd
    23378     1 23378    65  Ss      select   0xc4a2bb24 dhclient
    18174     1 18174     0  Ss      select   0xc4a2bc64 dhclient
     7709     1  7709     0  Ss      (threaded)          sshlockout_pf
    100073                   S       nanslp   0xc1efd568 sshlockout_pf
    100072                   S       uwait    0xc47bae80 sshlockout_pf
     7258     1  7258     0  Ss      select   0xc4a2c7e4 sshd
     5614     1  5614     0  Ss      (threaded)          mpd5
    100069                   S       select   0xc41d6be4 mpd5
      320     1   320     0  Ss      select   0xc46ad464 devd
      302   300   300     0  S       kqread   0xc47ba980 check_reload_status
      300     1   300     0  Ss      kqread   0xc4793080 check_reload_status
      285     1   285     0  Ss      kqread   0xc4792700 php-fpm
       54     0     0     0  DL      mdwait   0xc477c000 [md1]
       49     0     0     0  DL      mdwait   0xc477c800 [md0]
       20     0     0     0  DL      syncer   0xc1f1cac4 [syncer]
       19     0     0     0  DL      vlruwt   0xc4799000 [vnlru]
       18     0     0     0  DL      psleep   0xc1f1c204 [bufdaemon]
       17     0     0     0  DL      pollid   0xc1efbf30 [idlepoll]
        9     0     0     0  DL      pgzero   0xc2047d20 [pagezero]
        8     0     0     0  DL      psleep   0xc2047a44 [vmdaemon]
        7     0     0     0  DL      psleep   0xc20a7484 [pagedaemon]
        6     0     0     0  DL      waiting_ 0xc20a1d8c [sctp_iterator]
        5     0     0     0  DL      pftm     0xc0f622c0 [pf purge]
       16     0     0     0  DL      (threaded)          [usb]
    100038                   D       -        0xc43a8d34 [usbus1]
    100037                   D       -        0xc43a8d04 [usbus1]
    100036                   D       -        0xc43a8cd4 [usbus1]
    100035                   D       -        0xc43a8ca4 [usbus1]
    100034                   D       -        0xc443cb5c [usbus0]
    100033                   D       -        0xc443cb2c [usbus0]
    100032                   D       -        0xc443cafc [usbus0]
    100031                   D       -        0xc443cacc [usbus0]
        4     0     0     0  DL      (threaded)          [cam]
    100045                   D       -        0xc1e3e4a8 [scanner]
    100017                   D       -        0xc1e3e600 [doneq0]
        3     0     0     0  DL      crypto_r 0xc2046978 [crypto returns]
        2     0     0     0  DL      crypto_w 0xc20468b8 [crypto]
       15     0     0     0  DL      -        0xc1e58680 [rand_harvestq]
       14     0     0     0  DL      (threaded)          [geom]
    100010                   D       -        0xc209dde0 [g_down]
    100009                   D       -        0xc209dddc [g_up]
    100008                   D       -        0xc209ddd8 [g_event]
       13     0     0     0  DL      sleep    0xc1e14bb8 [ng_queue0]
       12     0     0     0  RL      (threaded)          [intr]
    100043                   I                           [swi1: pfsync]
    100041                   I                           [swi1: pf send]
    100039                   I                           [swi0: uart uart]
    100030                   I                           [irq12: ohci0 ehci0]
    100029                   I                           [irq15: vr2 ata1]
    100028                   I                           [irq14: ata0]
    100025                   I                           [irq9: ath0]
    100023                   I                           [swi6: Giant taskq]
    100021                   I                           [swi6: task queue]
    100015                   Run     CPU 0               [swi5: fast taskq]
    100006                   I                           [swi3: vm]
    100005                   I                           [swi4: clock]
    100004                   I                           [swi1: netisr 0]
       11     0     0     0  RL                          [idle: cpu0]
        1     0     1     0  SLs     wait     0xc42662f0 [init]
       10     0     0     0  DL      audit_wo 0xc20a5d88 [audit]
        0     0     0     0  DLs     (threaded)          [kernel]
    100044                   D       -        0xc4269800 [CAM taskq]
    100027                   D       -        0xc43cf700 [ath0 net80211 taskq]
    100026                   D       -        0xc43cf780 [ath0 taskq]
    100024                   D       -        0xc4269180 [thread]
    100022                   D       -        0xc4269300 [ffs_trim taskq]
    100020                   D       -        0xc4269480 [acpi_task_2]
    100019                   D       -        0xc4269480 [acpi_task_1]
    100018                   D       -        0xc4269480 [acpi_task_0]
    100016                   D       -        0xc4269880 [kqueue taskq]
    100011                   D       -        0xc426aa80 [firmware taskq]
    100000                   D       swapin   0xc209de64 [swapper]
    db:0:kdb.enter.default>  alltrace
    
    Full trace is attached. I can not use IPSEC unfortunately. IKEv2 is not an option because there are compatibility issues with the opposite side.
    
    And: sorry for my english...
    [/thread]
    


  • My system is D2500CC mini-ITX motherboard from Intel, all embedded into it.
    I have configured WAN as PPPoE and LAN as 10.0.x.x
    I have also configured IPsec for mobile clients using this "guide" https://forum.pfsense.org/index.php?topic=87655.msg488294#msg488294
    Virtual IP network for mobile clients is configured as 10.0.y.y
    When I get the successful connection it looks like traffic is not moving at all between client and firewall. Mostly it crashes with this panic

    Fatal double fault:
    eip = 0xc0cd225f
    esp = 0xe0783fe8
    ebp = 0xe0784050
    cpuid = 1; apic id = 01
    panic: double fault
    cpuid = 1
    KDB: enter: panic

    Full version is

    http://pastebin.com/mSLw8QfX



  • Currently disabled IP Random id generation (Insert a stronger id into IP header of packets passing through the filter.)
    Testing…



  • So far so good… no problems currently with IPSEC uptime and panics. Using Stronswan android client 12 hours already.
    Previous record was 1 minute with  IP Random id generation enabled



  • @ermal:

    For example
    kern.timecounter.hardware: TSC-low
    kern.timecounter.choice: TSC-low(1000) ACPI-fast(900) i8254(0) HPET(950) dummy(-1000000)

    On my ALIX board I have the choice between the following time sources: TSC(800) i8254(0) dummy(-1000000)

    Changing the timesource with

    sysctl kern.timecounter.hardware=i8254
    kern.timecounter.hardware: TSC -> i8254
    

    does not have an effect on this issue. After changing the timesource IPSEC keeps crashing on first ping from remote.

    Here are the latest dump with timesource i8254:

    Fatal double fault:
    eip = 0xc0cef7e0
    esp = 0xc8b69000
    ebp = 0xc8b69008
    cpuid = 0; apic id = 00
    panic: double fault
    cpuid = 0
    KDB: enter: panic
    [ thread pid 12 tid 100015 ]
    Stopped at      kdb_enter+0x3d: movl    $0,kdb_why
    db:0:kdb.enter.default> textdump set
    textdump set
    db:0:kdb.enter.default>  capture on
    db:0:kdb.enter.default>  run lockinfo
    db:1:lockinfo> show locks
    No such command
    db:1:locks>  show alllocks
    No such command
    db:1:alllocks>  show lockedvnods
    Locked vnodes
    db:0:kdb.enter.default>  show pcpu
    cpuid        = 0
    dynamic pcpu = 0x5e7a00
    curthread    = 0xc4317c40: pid 12 "swi5: fast taskq"
    curpcb       = 0xc8b6ad60
    fpcurthread  = none
    idlethread   = 0xc426d000: tid 100003 "idle: cpu0"
    APIC ID      = 0
    currentldt   = 0x50
    db:0:kdb.enter.default>  bt
    Tracing pid 12 tid 100015 td 0xc4317c40
    kdb_enter(c142c723,c142c723,c15ea443,c205b4c8,0,...) at kdb_enter+0x3d/frame 0xc205b480
    panic(c15ea443,0,0,0,c8b69008,...) at panic+0x144/frame 0xc205b4bc
    dblfault_handler() at dblfault_handler+0xab/frame 0xc205b4bc
    --- trap 0x17, eip = 0xc0cef7e0, esp = 0xc8b69000, ebp = 0xc8b69008 ---
    critical_exit(c1e3718c,c0cd1bf4,0,0,0,...) at critical_exit/frame 0xc8b69008
    i8254_get_timecount(c446f618,c1e049c4,34,0,0,...) at i8254_get_timecount+0x141/frame 0xc8b69030
    tc_windup(1,0,c142675e,219,0,...) at tc_windup+0x45/frame 0xc8b69080
    hardclock_cnt(1,0,c1e3718c,0,0,...) at hardclock_cnt+0x447/frame 0xc8b690e8
    handleevents(0,0,0,0,0,...) at handleevents+0xee/frame 0xc8b69138
    timercb(c446f640,0,c4268310,0,0,...) at timercb+0x3b9/frame 0xc8b69198
    clkintr(c446f600,0,c4317c40,0,c1e3718c,...) at clkintr+0xfc/frame 0xc8b691bc
    intr_event_handle(c426bb80,c8b69228,c42685f0,c4ead154,c1e3718c,...) at intr_event_handle+0x85/frame 0xc8b691dc
    intr_execute_handlers(c1e3718c,c8b69228) at intr_execute_handlers+0x42/frame 0xc8b691fc
    atpic_handle_intr(0,c8b69228) at atpic_handle_intr+0x5a/frame 0xc8b69218
    Xatpic_intr0() at Xatpic_intr0+0x22/frame 0xc8b69218
    --- interrupt, eip = 0xc0dd89f7, esp = 0xc8b69268, ebp = 0xc8b69290 ---
    rn_match(c8b692c8,c4ead100,c8b692c8,c0e22d00,c1e14bb8,...) at rn_match+0x17/frame 0xc8b69290
    pfr_match_addr(c4ec0000,c4c3e832,2,c48ff600,0,...) at pfr_match_addr+0xd5/frame 0xc8b692f0
    pf_normalize_ip(c8b6962c,2,c4a24800,c8b69584,c8b69528,...) at pf_normalize_ip+0x2d6/frame 0xc8b69388
    pf_test(2,c424f400,c8b6962c,0,c8b696c8,...) at pf_test+0x246/frame 0xc8b695e4
    pf_check_out(0,c8b6962c,c424f400,2,0,...) at pf_check_out+0x4b/frame 0xc8b69608
    pfil_run_hooks(c20a1c14,c8b696f4,c424f400,2,0,...) at pfil_run_hooks+0x88/frame 0xc8b69660
    ip_output(c48f7b00,0,0,2,0,...) at ip_output+0xaac/frame 0xc8b69718
    ipsec_process_done(c48f7b00,c4e94180,0,f228ae0e,c48f7b00,...) at ipsec_process_done+0x3cf/frame 0xc8b69768
    esp_output_cb(c60c2bf4,c1e21524,c8b697f8,c0f9f5e1,c48f7b00,...) at esp_output_cb+0x3cd/frame 0xc8b697c0
    crypto_done(c60c2bf4,c48f7b00,8c,c,c8b699a8,...) at crypto_done+0x99/frame 0xc8b697f8
    swcr_process(c4269080,c60c2bf4,0,c8b69a60,c0ccd50d,...) at swcr_process+0x6e/frame 0xc8b69a00
    crypto_invoke(0,40,0,0,0,...) at crypto_invoke+0x79/frame 0xc8b69a38
    crypto_dispatch(c60c2bf4,c145696b,375,c8b69abb,c4c3e828,...) at crypto_dispatch+0x64/frame 0xc8b69a60
    esp_output(c48f7b00,c4e94180,0,14,9,...) at esp_output+0x91d/frame 0xc8b69ae0
    ipsec4_process_packet(c48f7b00,c4e94180,0,0,0,...) at ipsec4_process_packet+0x312/frame 0xc8b69b70
    ip_ipsec_output(c8b69c34,0,c8b69c30,c8b69c2c,0,...) at ip_ipsec_output+0x1c8/frame 0xc8b69ba0
    ip_output(c48f7b00,0,0,0,0,...) at ip_output+0xa2f/frame 0xc8b69c58
    icmp_reflect(c8b69d28,10,0,c8b69d04,90000,...) at icmp_reflect+0x5b5/frame 0xc8b69cb0
    icmp_input(c48f7b00,14,c8b69d8c,1,c8b69ddc,...) at icmp_input+0x9b9/frame 0xc8b69d78
    ip_input(c48f7b00,c20a184c,c4317c40,0,c8b69e50,...) at ip_input+0x295/frame 0xc8b69ddc
    netisr_dispatch_src(1,0,c48f7b00) at netisr_dispatch_src+0x8b/frame 0xc8b69e24
    netisr_dispatch(1,c48f7b00,1,102,0,...) at netisr_dispatch+0x20/frame 0xc8b69e38
    _ipip_input(c48f7b00,14,4,1,3c,...) at _ipip_input+0x650/frame 0xc8b69e80
    encap4_input(c48f7b00,14,0,c0dd4f18,1,...) at encap4_input+0x210/frame 0xc8b69ee0
    ip_input(c48f7b00,2,40,0,0,...) at ip_input+0x295/frame 0xc8b69f48
    netisr_dispatch_src(1,611ea2cb,c48f7b00,101,68,...) at netisr_dispatch_src+0x8b/frame 0xc8b69f90
    ipsec4_common_input_cb(c48f7b00,c67d6d00,14,9,0,...) at ipsec4_common_input_cb+0x276/frame 0xc8b69fd8
    esp_input_cb(c60c2bf4,c053cd1d,c4f59cf8,e,c4c3e89e,...) at esp_input_cb+0x772/frame 0xc8b6a068
    crypto_done(c60c2bf4,c8b6a240,10,10,c8b6a240,...) at crypto_done+0x99/frame 0xc8b6a0a0
    swcr_process(c4269080,c60c2bf4,0,c8b6a308,c0ccd50d,...) at swcr_process+0x6e/frame 0xc8b6a2a8
    crypto_invoke(0,c4c3e8ae,c60850b8,c,c60850b8,...) at crypto_invoke+0x79/frame 0xc8b6a2e0
    crypto_dispatch(c60c2bf4,c145696b,1c8,c60850b8,c57dc528,...) at crypto_dispatch+0x64/frame 0xc8b6a308
    esp_input(c48f7b00,c67d6d00,14,9,d0,...) at esp_input+0x771/frame 0xc8b6a370
    ipsec_common_input(9,2,32,c8b6a408) at ipsec_common_input+0x4f7/frame 0xc8b6a3dc
    ipsec4_common_input(c48f7b00,14,32) at ipsec4_common_input+0x39/frame 0xc8b6a3f4
    esp4_input(c48f7b00,14,c424f400,1,0,...) at esp4_input+0x20/frame 0xc8b6a408
    ip_input(c48f7b00,801,c4902a00,c43c0f68,c0d1f5ce,...) at ip_input+0x295/frame 0xc8b6a470
    netisr_dispatch_src(1,0,c48f7b00) at netisr_dispatch_src+0x8b/frame 0xc8b6a4b8
    netisr_dispatch(1,c48f7b00,c4902a00,c431a000,6e,...) at netisr_dispatch+0x20/frame 0xc8b6a4cc
    ng_iface_rcvdata(c4793680,c4a4a000,c4317c40,c4f20600,0,...) at ng_iface_rcvdata+0xea/frame 0xc8b6a4f4
    ng_apply_item(0,c4902a00,14,c4a632f4,c8b6a584,...) at ng_apply_item+0x22d/frame 0xc8b6a550
    ng_snd_item(c4a4a000,0,c4793700,0,0,...) at ng_snd_item+0x1a0/frame 0xc8b6a584
    ng_tcpmss_rcvdata(c4793800,c4a4a000,46507c40,2f1c645d,c8b6a68c,...) at ng_tcpmss_rcvdata+0xac/frame 0xc8b6a5cc
    ng_apply_item(0,34,5dc,c8b6a760,0,...) at ng_apply_item+0x22d/frame 0xc8b6a628
    ng_snd_item(c4a4a000,0,c47b9000,0,c4a4a000,...) at ng_snd_item+0x1a0/frame 0xc8b6a65c
    ng_ppp_comp_recv(21,0,1,c48f7b00,c4a4a000,...) at ng_ppp_comp_recv+0x158/frame 0xc8b6a688
    ng_ppp_crypt_recv(21,0,d80e0e6a,c8b6a820,c0ee51a9,...) at ng_ppp_crypt_recv+0x70/frame 0xc8b6a6a4
    ng_ppp_rcvdata(c4793b00,c4a4a000,c4902a00,0,c4317c40,...) at ng_ppp_rcvdata+0x2e4/frame 0xc8b6a700
    ng_apply_item(0,0,0,246,c209f300,...) at ng_apply_item+0x22d/frame 0xc8b6a75c
    ng_snd_item(c4a4a000,0,c4793880,0,362cd2a1,...) at ng_snd_item+0x1a0/frame 0xc8b6a790
    ng_tee_rcvdata(c4793c00,c4a4a000,1,0,c8b6a7e4,...) at ng_tee_rcvdata+0x156/frame 0xc8b6a7b8
    ng_apply_item(0,c4c3e812,6,c4a645e0,1,...) at ng_apply_item+0x22d/frame 0xc8b6a814
    ng_snd_item(c4a4a000,0,c4793c80,0,c46a9900,...) at ng_snd_item+0x1a0/frame 0xc8b6a848
    ng_pppoe_rcvdata_ether(c4793d00,c4a4a000,c4a645e0,34,1c5,...) at ng_pppoe_rcvdata_ether+0x2a3/frame 0xc8b6a8c4
    ng_apply_item(0,c47b0000,c48f7b00,0,c8b6a954,...) at ng_apply_item+0x22d/frame 0xc8b6a920
    ng_snd_item(c4a4a000,0,c47b9a00,0,c47b0000,...) at ng_snd_item+0x1a0/frame 0xc8b6a954
    ng_ether_input_orphan(c47b0000,c48f7b00,0,b1ce9d7,a1d20db3,...) at ng_ether_input_orphan+0x66/frame 0xc8b6a974
    ether_demux(c47b0000,c48f7b00,6,c4317c40,c4317c40,...) at ether_demux+0x1f9/frame 0xc8b6a9a0
    ether_nh_input(c48f7b00,c8b6aa14,0,0,0,...) at ether_nh_input+0x37e/frame 0xc8b6a9f0
    netisr_dispatch_src(9,0,c48f7b00) at netisr_dispatch_src+0x8b/frame 0xc8b6aa38
    netisr_dispatch(9,c48f7b00) at netisr_dispatch+0x20/frame 0xc8b6aa4c
    ether_input(c47b0000,c48f7b00,0,c8b6aae0,c0cd1bf4,...) at ether_input+0x19/frame 0xc8b6aa5c
    vlan_input(c424fc00,c48f7b00,c4317c40,c4bfd008,c43d0100,...) at vlan_input+0x1a8/frame 0xc8b6aa8c
    ether_demux(c424fc00,c48f7b00,6,7f8,c4bfd008,...) at ether_demux+0xaf/frame 0xc8b6aab8
    ether_nh_input(c48f7b00,c4bbd200,2f4,c8b6ab70,c0b4e3ba,...) at ether_nh_input+0x37e/frame 0xc8b6ab04
    netisr_dispatch_src(9,0,c48f7b00) at netisr_dispatch_src+0x8b/frame 0xc8b6ab4c
    netisr_dispatch(9,c48f7b00) at netisr_dispatch+0x20/frame 0xc8b6ab60
    ether_input(c424fc00,c48f7b00,c43a4000,745,c8b6abb8,...) at ether_input+0x19/frame 0xc8b6ab70
    vr_rxeof(0,0,c4317c40,c8b6abd8,46,...) at vr_rxeof+0x1f1/frame 0xc8b6abb8
    vr_int_task(c43cc000,1,c8b6ac18,c12bc042,c426bb80,...) at vr_int_task+0x123/frame 0xc8b6abe8
    taskqueue_run_locked(c8b6ac88,c128c712,0,c8b6ac44,c4310008,...) at taskqueue_run_locked+0xee/frame 0xc8b6ac2c
    taskqueue_run(c4269980) at taskqueue_run+0xa3/frame 0xc8b6ac50
    taskqueue_fast_run(0,0,246,0,0,...) at taskqueue_fast_run+0x11/frame 0xc8b6ac5c
    intr_event_execute_handlers(109,c4269900,c1428033,55a,0,...) at intr_event_execute_handlers+0xaa/frame 0xc8b6ac88
    ithread_loop(c41b5e70,c8b6ad08,0,0,0,...) at ithread_loop+0x80/frame 0xc8b6acc4
    fork_exit(c0cb63f0,c41b5e70,c8b6ad08) at fork_exit+0xa3/frame 0xc8b6acf4
    fork_trampoline() at fork_trampoline+0x8/frame 0xc8b6acf4
    --- trap 0, eip = 0, esp = 0xc8b6ad40, ebp = 0 ---
    db:0:kdb.enter.default>  ps
      pid  ppid  pgrp   uid   state   wmesg     wchan    cmd
    30982   285   285     0  R                           php-fpm
    71440 73267    21     0  S       nanslp   0xc1efd568 sleep
    60001 59799 60001     0  Ss      (threaded)          charon
    100155                   S       uwait    0xc57e6580 charon
    100154                   S       uwait    0xc57e6500 charon
    100153                   S       uwait    0xc4a6f280 charon
    100152                   S       uwait    0xc57e6400 charon
    100151                   S       uwait    0xc57e6780 charon
    100150                   S       uwait    0xc57e7b80 charon
    100149                   S       uwait    0xc57e7c00 charon
    100148                   S       uwait    0xc57e6a80 charon
    100147                   S       uwait    0xc57e6800 charon
    100146                   S       uwait    0xc4e94b00 charon
    100145                   S       uwait    0xc57e6600 charon
    100144                   S       uwait    0xc4a6f380 charon
    100143                   S       select   0xc46af6e4 charon
    100142                   S       select   0xc46af724 charon
    100141                   S       accept   0xc57d61e6 charon
    100140                   S       uwait    0xc57e9700 charon
    100060                   S       sigwait  0xc674c000 charon
    59799     1 59799     0  Ss      select   0xc6703824 starter
    85529 89861 89861     0  S+      ttyin    0xc4313470 sh
    90198 78569 90198     0  Ss      (threaded)          sshlockout_pf
    100124                   S       nanslp   0xc1efd568 sshlockout_pf
    100055                   S       piperd   0xc47f47f8 sshlockout_pf
    89861     1 89861     0  Ss+     wait     0xc48ac8d0 login
    89557     1     1     0  S       nanslp   0xc1efd568 getty
    87839 87796 87796     0  S       nanslp   0xc1efd568 minicron
    87796     1 87796     0  Ss      wait     0xc47985e0 minicron
    87654 87281 87281     0  S       nanslp   0xc1efd568 minicron
    87281     1 87281     0  Ss      wait     0xc48ab000 minicron
    86900 86382 86382     0  S       nanslp   0xc1efd568 minicron
    86382     1 86382     0  Ss      wait     0xc4f96000 minicron
    84022     1 84022     0  Ss      nanslp   0xc1efd568 cron
    78569     1 78569     0  Ss      select   0xc4a361e4 syslogd
    73996     1 73996     0  Ss      select   0xc4a38ba4 igmpproxy
    73267     1    21     0  S+      wait     0xc4f978d0 sh
    65451     1 65451     0  Ss      (threaded)          ntpd
    100056                   S       select   0xc46ac7a4 ntpd
    60741     1 60741     0  Ss      (threaded)          filterdns
    100101                   S       uwait    0xc4d6b500 signal-thread
    100100                   S       uwait    0xc57e9680 filterdns
    54653     1 54653  1002  Ss      select   0xc4437ba4 dhcpd
    48648     1 48359 65534  S       select   0xc4a362e4 dnsmasq
    44941     1 44633     0  S       kqread   0xc4ea7b80 lighttpd
    35762 35604 35604     0  S       piperd   0xc47f44c8 rrdtool
    35604     1 35604     0  Ss      select   0xc4a365a4 apinger
    32684     1 32684     0  Ss      select   0xc4a36824 inetd
    32208     1 32208     0  Ss      select   0xc4a36224 openvpn
    32123     1 32123     0  Ss      bpf      0xc4314c00 filterlog
    30648     1 30648     0  Ss      select   0xc4a36664 openvpn
    29293     1 29293     0  Ss      select   0xc4a366a4 openvpn
    28370     1 28370     0  Ss      select   0xc4a36ba4 openvpn
    25601     1 25601     0  Ss      select   0xc4a376e4 hostapd
    22824     1 22824    65  Ss      select   0xc4a37c64 dhclient
    19122     1 19122     0  Ss      select   0xc4a37ce4 dhclient
     8090     1  8090     0  Ss      (threaded)          sshlockout_pf
    100073                   S       nanslp   0xc1efd568 sshlockout_pf
    100072                   S       uwait    0xc47bae80 sshlockout_pf
     7521     1  7521     0  Ss      select   0xc4a38924 sshd
     6233     1  6233     0  Ss      (threaded)          mpd5
    100068                   S       select   0xc41d6ba4 mpd5
      320     1   320     0  Ss      select   0xc4437aa4 devd
      302   300   300     0  S       kqread   0xc47baa80 check_reload_status
      300     1   300     0  Ss      kqread   0xc47baa00 check_reload_status
      285     1   285     0  Ss      kqread   0xc4792700 php-fpm
       54     0     0     0  DL      mdwait   0xc442b000 [md1]
       49     0     0     0  DL      mdwait   0xc442d800 [md0]
       20     0     0     0  DL      syncer   0xc1f1cac4 [syncer]
       19     0     0     0  DL      vlruwt   0xc4799000 [vnlru]
       18     0     0     0  DL      psleep   0xc1f1c204 [bufdaemon]
       17     0     0     0  DL      pollid   0xc1efbf30 [idlepoll]
        9     0     0     0  DL      pgzero   0xc2047d20 [pagezero]
        8     0     0     0  DL      psleep   0xc2047a44 [vmdaemon]
        7     0     0     0  DL      psleep   0xc20a7484 [pagedaemon]
        6     0     0     0  DL      waiting_ 0xc20a1d8c [sctp_iterator]
        5     0     0     0  DL      pftm     0xc0f622c0 [pf purge]
       16     0     0     0  DL      (threaded)          [usb]
    100038                   D       -        0xc43a8d34 [usbus1]
    100037                   D       -        0xc43a8d04 [usbus1]
    100036                   D       -        0xc43a8cd4 [usbus1]
    100035                   D       -        0xc43a8ca4 [usbus1]
    100034                   D       -        0xc443cb5c [usbus0]
    100033                   D       -        0xc443cb2c [usbus0]
    100032                   D       -        0xc443cafc [usbus0]
    100031                   D       -        0xc443cacc [usbus0]
        4     0     0     0  DL      (threaded)          [cam]
    100045                   D       -        0xc1e3e4a8 [scanner]
    100017                   D       -        0xc1e3e600 [doneq0]
        3     0     0     0  DL      crypto_r 0xc2046978 [crypto returns]
        2     0     0     0  DL      crypto_w 0xc20468b8 [crypto]
       15     0     0     0  DL      -        0xc1e58680 [rand_harvestq]
       14     0     0     0  DL      (threaded)          [geom]
    100010                   D       -        0xc209dde0 [g_down]
    100009                   D       -        0xc209dddc [g_up]
    100008                   D       -        0xc209ddd8 [g_event]
       13     0     0     0  DL      sleep    0xc1e14bb8 [ng_queue0]
       12     0     0     0  RL      (threaded)          [intr]
    100043                   I                           [swi1: pfsync]
    100041                   I                           [swi1: pf send]
    100039                   I                           [swi0: uart uart]
    100030                   I                           [irq12: ohci0 ehci0]
    100029                   I                           [irq15: vr2 ata1]
    100028                   I                           [irq14: ata0]
    100025                   I                           [irq9: ath0]
    100023                   I                           [swi6: Giant taskq]
    100021                   I                           [swi6: task queue]
    100015                   Run     CPU 0               [swi5: fast taskq]
    100006                   I                           [swi3: vm]
    100005                   I                           [swi4: clock]
    100004                   I                           [swi1: netisr 0]
       11     0     0     0  RL                          [idle: cpu0]
        1     0     1     0  SLs     wait     0xc42662f0 [init]
       10     0     0     0  DL      audit_wo 0xc20a5d88 [audit]
        0     0     0     0  DLs     (threaded)          [kernel]
    100044                   D       -        0xc4269800 [CAM taskq]
    100027                   D       -        0xc43cf700 [ath0 net80211 taskq]
    100026                   D       -        0xc43cf780 [ath0 taskq]
    100024                   D       -        0xc4269180 [thread]
    100022                   D       -        0xc4269300 [ffs_trim taskq]
    100020                   D       -        0xc4269480 [acpi_task_2]
    100019                   D       -        0xc4269480 [acpi_task_1]
    100018                   D       -        0xc4269480 [acpi_task_0]
    100016                   D       -        0xc4269880 [kqueue taskq]
    100011                   D       -        0xc426aa80 [firmware taskq]
    100000                   D       swapin   0xc209de64 [swapper]
    db:0:kdb.enter.default>[/thread]
    


  • And you don't have "IP Random id generation" enabled?



  • @w0w:

    And you don't have "IP Random id generation" enabled?

    "IP Random id generation" is off. This setting I have never changed. Otherwise, I have only MSS clamping enabled and the Unity plugin disabled. "IPsec Mobile Client Support" is also turned off. All other settings are defaults.

    Apparently the kernel timesource is not the cause of the error. Could someone have another look at the dump? What information is still needed?



  • We need to use bugtracker to report the bug. Please report this issue here https://redmine.pfsense.org/
    Don't link the forum page, post your dumps and I'll add mine too.
    FYI I don't have changed MSS and unity settings.





  • Similar issue here. If I connect via VPN, PFSense stops and restarts - the IPSEC connection lasts between 30 seconds and 5 minutes. This issue is 100% reproducible - I've crashed my box about six times today.

    Fatal double fault:
    eip = 0xc12c62a8
    esp = 0xecf4cff8
    ebp = 0xecf4d000
    cpuid = 0; apic id = 00
    panic: double fault
    cpuid = 0
    KDB: enter: panic
    panic.txt0600001412471723700  7136 ustarrootwheeldouble faultversion.txt06000025112471723700  7614 ustarrootwheelFreeBSD 10.1-RELEASE-p4 #0 36d7dec(releng/10.1)-dirty: Thu Jan 22 15:12:38 CST 2015
        root@pfsense-22-i386-builder:/usr/obj.i386/usr/pfSensesrc/src/sys/pfSense_SMP.10

    PFSense 2.2 - upgraded from 2.1.5

    Hardware:
    CPU: Intel(R) Atom(TM) CPU N270 @ 1.60GHz
    Mobo: KINO-945GSE
    Storage: 2 GB CF Card
    Dual LAN: Realtek PCIe 8111CP GbE controller

    IPSEC Details:
    Mobile Client
    No IP Compression
    Unity plugin disabled
    IKE v1
    Virtual IP Address Assigned to Clients
    IP Random ID Generation at default value (default is 0: sequential IP IDs)

    Interfaces:
    RE0: WAN: PPOE
    RE1: LAN/OPT1/OPT2 using VLAN tagging

    May try a clean install of v2.2 if you think there's any mileage in it.

    pfsenseCrashDump.txt



  • @afasoas: are you running 32 or 64 bit version? So far I think all the reports are coming from x86 versions. Is this reproducible under x64??



  • 32 bit.
    Well spotted.
    I will add my crash dump to the bug tracker shortly.



  • Upgraded my hardware so I could run the 64-bit version. No issues to report thus far. IPSEC seems solid and stable.



  • Great solution, but this is like cutting the head and sewing back a new one, more "brainful".
    It could be also driver Ethernet issue with physical low memory installed. I can only wait when somebody really smart will check our crash dumps to find out the reason of double triple crash and panic.



  • @w0w:

    Great solution, but this is like cutting the head and sewing back a new one, more "brainful".
    It could be also driver Ethernet issue with physical low memory installed. I can only wait when somebody really smart will check our crash dumps to find out the reason of double triple crash and panic.

    I appreciate that this isn't the most helpful solution. I just wanted to confirm that the problem went away using the same configuration with a 64-bit version.

    On Edit: I realise that you are using an Atom D2500 - you can run the 64-bit version of pfSense on it, if that helps?



  • I can, really, but what if I don't need it? My typical memory usage is less then 6% and CPU is mostly 90% in peak (300Mbit internet, three clients). I can buy some XEON based proliant G8 but why? :) 
    Maybe I'll move to x64 platform if we can't trust x86 anymore. But I need an answer for the question. DO we really need to move to x64 just because x86 is not supported or what?
    I think i'll wait for answer before buying some needless hardware. :)



  • And what about the ALIX boards? As far as I know they are all 32bit.


  • Banned

    @sh0gun:

    And what about the ALIX boards? As far as I know they are all 32bit.

    Not getting any IPsec panics on Alix. (Also, make sure you did not enable some stupid features, like the infamous "Insert a stronger id into IP header of packets passing through the filter.")



  • But your hardware is already 64-bit capable, at least as far as pfSense is concerned!
    Yes Intel don't provide 64-bit video drivers but seems to be a non-issue here.

    @w0w:

    My system is D2500CC mini-ITX motherboard from Intel, all embedded into it.



  • @doktornotor:

    @sh0gun:

    And what about the ALIX boards? As far as I know they are all 32bit.

    Not getting any IPsec panics on Alix. (Also, make sure you did not enable some stupid features, like the infamous "Insert a stronger id into IP header of packets passing through the filter.")

    I don't think that hiding your client OS unique ID behind firewall is so stupid as you think about it.



  • @afasoas:

    But your hardware is already 64-bit capable, at least as far as pfSense is concerned!
    Yes Intel don't provide 64-bit video drivers but seems to be a non-issue here.

    @w0w:

    My system is D2500CC mini-ITX motherboard from Intel, all embedded into it.

    Yep. But what is the point to use 64-bit OS with 2GB of RAM? It does not fix the problem in 32-bit version also :) There is some bug, that must be fixed and this is good, maybe, that it is pointed now to 32-bit version only, but next time it could be related to 64-bit only, so migrating between platforms is useless for me, until I read something like "64-bit freebsd is more secure and stable, don't use 32-bit anymore".



  • Migrating between platforms resolves your problem, for the time being.
    If your memory usage is at 6% then I figure there should not be a problem switching over to 64-bit.



  • @w0w:

    …so migrating between platforms is useless for me, until I read something like "64-bit freebsd is more secure and stable, don't use 32-bit anymore".

    Ahh, I think you mean this:
    "[_…64 bit is more widely used, what we test the most with, and what most of our development is done using.

    32 bit is a dying breed. FreeNAS and DragonflyBSD both just put out their last releases with 32 bit support. While we'll still continue to support 32 bit in 2.2.x releases and possibly beyond that, ending 32 bit support is certainly on the road map and will happen sooner than later.

    There is no reason to use 32 bit over 64 today, if your hardware is 64 bit capable, you should only be running 64 bit._](https://forum.pfsense.org/index.php?topic=84679.msg464432#msg464432)"

    Chris Buechler, November 27th, 2014


  • Rebel Alliance Developer Netgate



  • Ok… At least I'll give it a try. Later, next week maybe :)



  • Still, do we have any clues on the issue itself? It affects Alix boards and similar hardware to at least some degree (besides some other reports, I had to downgrade some production boxes myself due to random reboots which I am sure are related to all this). Unfortunately I couldn't find a specific trigger but there seem to be several crash dumps and reproducible configs available



  • https://redmine.pfsense.org/issues/4454 Looks like something moving forward, at least for me, thanks for Chris Buechler



  • Ok, installed amd64, no more kernel panic when "Insert strong ID…" option enabled. So yes, I can confirm that x86 platform is affected and amd64 not.



  • Can you try to set net.inet.ipsec.directdispatch to 0 and see if the panic goes away?



  • Update my pfsense in lab on 2.2.1 same behavior
    then i tried  to

    set net.inet.ipsec.directdispatch to 0

    looks good so far. stable since about 30 minutes (before after about 30 seconds i get a kernel panic)



  • Can you describe your WAN interface?



  • in our lab i used 3 pfsenses

    one so to say provider with an pppoe server (Version 2.2.1 Vmware)

    then i have one pfsense which stands for my company firewall (Version 2.1.5 Vmware)

    and i have another pfsense which stand for my home pfsense  (Version 2.2.1 Alix2d3)

    both pfsenses from company and home have a pppoe wan interface which is connected to my provider pfsense

    on my home pfsense i also have a vlan tag added like i have to do it at my real home pfsense.



  • Ok there is an open issue for this scenario already.

    Thank you for the information.



  • yes my scenario is like described befor in this thread.
    but set net.inet.ipsec.directdispatch to 0 seems to "workaround" the issue
    so there is may be hope for all 32 Bit Users  ;)



  • ermal, do you need my report too? :)
    Actually I am the man who reported the issue. But I moved my box on to amd64 version…
    I ask because I have troubles to restoring old x86 backup, so new installation take time... I can do it but only if it really needed.



  • Nope the scenario is clear.



  • i can reproduce the scenario in our lab any time. The VM's are set up already just need to put back the config into the alix board.
    If you need just let me know.



  • @flix87:

    Update my pfsense in lab on 2.2.1 same behavior
    then i tried  to

    set net.inet.ipsec.directdispatch to 0

    looks good so far. stable since about 30 minutes (before after about 30 seconds i get a kernel panic)

    Hello, had the same issues with reboot`s on 8 devices.

    Hardware ALIX.2 v0.99m tinyBIOS V1.4a.
    Pf.Version 2.2.2-RELEASE (i386) built on Mon Apr 13 20:10:33 CDT 2015.

    After the set of "net.inet.ipsec.directdispatch to 0" all systems works fine. No reboots, no systempanics, stable 3 days ago yet.  :)

    What is done with this adjustment ? Will this fix embedded as standard for further versions ?

    Best regards

    eeit



  • net.inet.ipsec.directdispatch=0

    fixes the issue on ALIX 2D13 (32bit) with IPSEC for me. Thank you very much! Marked as solved.


Log in to reply