Reproducible kernel panic with pfSense 2.2 and IPSEC



  • @ermal:

    For example
    kern.timecounter.hardware: TSC-low
    kern.timecounter.choice: TSC-low(1000) ACPI-fast(900) i8254(0) HPET(950) dummy(-1000000)

    On my ALIX board I have the choice between the following time sources: TSC(800) i8254(0) dummy(-1000000)

    Changing the timesource with

    sysctl kern.timecounter.hardware=i8254
    kern.timecounter.hardware: TSC -> i8254
    

    does not have an effect on this issue. After changing the timesource IPSEC keeps crashing on first ping from remote.

    Here are the latest dump with timesource i8254:

    Fatal double fault:
    eip = 0xc0cef7e0
    esp = 0xc8b69000
    ebp = 0xc8b69008
    cpuid = 0; apic id = 00
    panic: double fault
    cpuid = 0
    KDB: enter: panic
    [ thread pid 12 tid 100015 ]
    Stopped at      kdb_enter+0x3d: movl    $0,kdb_why
    db:0:kdb.enter.default> textdump set
    textdump set
    db:0:kdb.enter.default>  capture on
    db:0:kdb.enter.default>  run lockinfo
    db:1:lockinfo> show locks
    No such command
    db:1:locks>  show alllocks
    No such command
    db:1:alllocks>  show lockedvnods
    Locked vnodes
    db:0:kdb.enter.default>  show pcpu
    cpuid        = 0
    dynamic pcpu = 0x5e7a00
    curthread    = 0xc4317c40: pid 12 "swi5: fast taskq"
    curpcb       = 0xc8b6ad60
    fpcurthread  = none
    idlethread   = 0xc426d000: tid 100003 "idle: cpu0"
    APIC ID      = 0
    currentldt   = 0x50
    db:0:kdb.enter.default>  bt
    Tracing pid 12 tid 100015 td 0xc4317c40
    kdb_enter(c142c723,c142c723,c15ea443,c205b4c8,0,...) at kdb_enter+0x3d/frame 0xc205b480
    panic(c15ea443,0,0,0,c8b69008,...) at panic+0x144/frame 0xc205b4bc
    dblfault_handler() at dblfault_handler+0xab/frame 0xc205b4bc
    --- trap 0x17, eip = 0xc0cef7e0, esp = 0xc8b69000, ebp = 0xc8b69008 ---
    critical_exit(c1e3718c,c0cd1bf4,0,0,0,...) at critical_exit/frame 0xc8b69008
    i8254_get_timecount(c446f618,c1e049c4,34,0,0,...) at i8254_get_timecount+0x141/frame 0xc8b69030
    tc_windup(1,0,c142675e,219,0,...) at tc_windup+0x45/frame 0xc8b69080
    hardclock_cnt(1,0,c1e3718c,0,0,...) at hardclock_cnt+0x447/frame 0xc8b690e8
    handleevents(0,0,0,0,0,...) at handleevents+0xee/frame 0xc8b69138
    timercb(c446f640,0,c4268310,0,0,...) at timercb+0x3b9/frame 0xc8b69198
    clkintr(c446f600,0,c4317c40,0,c1e3718c,...) at clkintr+0xfc/frame 0xc8b691bc
    intr_event_handle(c426bb80,c8b69228,c42685f0,c4ead154,c1e3718c,...) at intr_event_handle+0x85/frame 0xc8b691dc
    intr_execute_handlers(c1e3718c,c8b69228) at intr_execute_handlers+0x42/frame 0xc8b691fc
    atpic_handle_intr(0,c8b69228) at atpic_handle_intr+0x5a/frame 0xc8b69218
    Xatpic_intr0() at Xatpic_intr0+0x22/frame 0xc8b69218
    --- interrupt, eip = 0xc0dd89f7, esp = 0xc8b69268, ebp = 0xc8b69290 ---
    rn_match(c8b692c8,c4ead100,c8b692c8,c0e22d00,c1e14bb8,...) at rn_match+0x17/frame 0xc8b69290
    pfr_match_addr(c4ec0000,c4c3e832,2,c48ff600,0,...) at pfr_match_addr+0xd5/frame 0xc8b692f0
    pf_normalize_ip(c8b6962c,2,c4a24800,c8b69584,c8b69528,...) at pf_normalize_ip+0x2d6/frame 0xc8b69388
    pf_test(2,c424f400,c8b6962c,0,c8b696c8,...) at pf_test+0x246/frame 0xc8b695e4
    pf_check_out(0,c8b6962c,c424f400,2,0,...) at pf_check_out+0x4b/frame 0xc8b69608
    pfil_run_hooks(c20a1c14,c8b696f4,c424f400,2,0,...) at pfil_run_hooks+0x88/frame 0xc8b69660
    ip_output(c48f7b00,0,0,2,0,...) at ip_output+0xaac/frame 0xc8b69718
    ipsec_process_done(c48f7b00,c4e94180,0,f228ae0e,c48f7b00,...) at ipsec_process_done+0x3cf/frame 0xc8b69768
    esp_output_cb(c60c2bf4,c1e21524,c8b697f8,c0f9f5e1,c48f7b00,...) at esp_output_cb+0x3cd/frame 0xc8b697c0
    crypto_done(c60c2bf4,c48f7b00,8c,c,c8b699a8,...) at crypto_done+0x99/frame 0xc8b697f8
    swcr_process(c4269080,c60c2bf4,0,c8b69a60,c0ccd50d,...) at swcr_process+0x6e/frame 0xc8b69a00
    crypto_invoke(0,40,0,0,0,...) at crypto_invoke+0x79/frame 0xc8b69a38
    crypto_dispatch(c60c2bf4,c145696b,375,c8b69abb,c4c3e828,...) at crypto_dispatch+0x64/frame 0xc8b69a60
    esp_output(c48f7b00,c4e94180,0,14,9,...) at esp_output+0x91d/frame 0xc8b69ae0
    ipsec4_process_packet(c48f7b00,c4e94180,0,0,0,...) at ipsec4_process_packet+0x312/frame 0xc8b69b70
    ip_ipsec_output(c8b69c34,0,c8b69c30,c8b69c2c,0,...) at ip_ipsec_output+0x1c8/frame 0xc8b69ba0
    ip_output(c48f7b00,0,0,0,0,...) at ip_output+0xa2f/frame 0xc8b69c58
    icmp_reflect(c8b69d28,10,0,c8b69d04,90000,...) at icmp_reflect+0x5b5/frame 0xc8b69cb0
    icmp_input(c48f7b00,14,c8b69d8c,1,c8b69ddc,...) at icmp_input+0x9b9/frame 0xc8b69d78
    ip_input(c48f7b00,c20a184c,c4317c40,0,c8b69e50,...) at ip_input+0x295/frame 0xc8b69ddc
    netisr_dispatch_src(1,0,c48f7b00) at netisr_dispatch_src+0x8b/frame 0xc8b69e24
    netisr_dispatch(1,c48f7b00,1,102,0,...) at netisr_dispatch+0x20/frame 0xc8b69e38
    _ipip_input(c48f7b00,14,4,1,3c,...) at _ipip_input+0x650/frame 0xc8b69e80
    encap4_input(c48f7b00,14,0,c0dd4f18,1,...) at encap4_input+0x210/frame 0xc8b69ee0
    ip_input(c48f7b00,2,40,0,0,...) at ip_input+0x295/frame 0xc8b69f48
    netisr_dispatch_src(1,611ea2cb,c48f7b00,101,68,...) at netisr_dispatch_src+0x8b/frame 0xc8b69f90
    ipsec4_common_input_cb(c48f7b00,c67d6d00,14,9,0,...) at ipsec4_common_input_cb+0x276/frame 0xc8b69fd8
    esp_input_cb(c60c2bf4,c053cd1d,c4f59cf8,e,c4c3e89e,...) at esp_input_cb+0x772/frame 0xc8b6a068
    crypto_done(c60c2bf4,c8b6a240,10,10,c8b6a240,...) at crypto_done+0x99/frame 0xc8b6a0a0
    swcr_process(c4269080,c60c2bf4,0,c8b6a308,c0ccd50d,...) at swcr_process+0x6e/frame 0xc8b6a2a8
    crypto_invoke(0,c4c3e8ae,c60850b8,c,c60850b8,...) at crypto_invoke+0x79/frame 0xc8b6a2e0
    crypto_dispatch(c60c2bf4,c145696b,1c8,c60850b8,c57dc528,...) at crypto_dispatch+0x64/frame 0xc8b6a308
    esp_input(c48f7b00,c67d6d00,14,9,d0,...) at esp_input+0x771/frame 0xc8b6a370
    ipsec_common_input(9,2,32,c8b6a408) at ipsec_common_input+0x4f7/frame 0xc8b6a3dc
    ipsec4_common_input(c48f7b00,14,32) at ipsec4_common_input+0x39/frame 0xc8b6a3f4
    esp4_input(c48f7b00,14,c424f400,1,0,...) at esp4_input+0x20/frame 0xc8b6a408
    ip_input(c48f7b00,801,c4902a00,c43c0f68,c0d1f5ce,...) at ip_input+0x295/frame 0xc8b6a470
    netisr_dispatch_src(1,0,c48f7b00) at netisr_dispatch_src+0x8b/frame 0xc8b6a4b8
    netisr_dispatch(1,c48f7b00,c4902a00,c431a000,6e,...) at netisr_dispatch+0x20/frame 0xc8b6a4cc
    ng_iface_rcvdata(c4793680,c4a4a000,c4317c40,c4f20600,0,...) at ng_iface_rcvdata+0xea/frame 0xc8b6a4f4
    ng_apply_item(0,c4902a00,14,c4a632f4,c8b6a584,...) at ng_apply_item+0x22d/frame 0xc8b6a550
    ng_snd_item(c4a4a000,0,c4793700,0,0,...) at ng_snd_item+0x1a0/frame 0xc8b6a584
    ng_tcpmss_rcvdata(c4793800,c4a4a000,46507c40,2f1c645d,c8b6a68c,...) at ng_tcpmss_rcvdata+0xac/frame 0xc8b6a5cc
    ng_apply_item(0,34,5dc,c8b6a760,0,...) at ng_apply_item+0x22d/frame 0xc8b6a628
    ng_snd_item(c4a4a000,0,c47b9000,0,c4a4a000,...) at ng_snd_item+0x1a0/frame 0xc8b6a65c
    ng_ppp_comp_recv(21,0,1,c48f7b00,c4a4a000,...) at ng_ppp_comp_recv+0x158/frame 0xc8b6a688
    ng_ppp_crypt_recv(21,0,d80e0e6a,c8b6a820,c0ee51a9,...) at ng_ppp_crypt_recv+0x70/frame 0xc8b6a6a4
    ng_ppp_rcvdata(c4793b00,c4a4a000,c4902a00,0,c4317c40,...) at ng_ppp_rcvdata+0x2e4/frame 0xc8b6a700
    ng_apply_item(0,0,0,246,c209f300,...) at ng_apply_item+0x22d/frame 0xc8b6a75c
    ng_snd_item(c4a4a000,0,c4793880,0,362cd2a1,...) at ng_snd_item+0x1a0/frame 0xc8b6a790
    ng_tee_rcvdata(c4793c00,c4a4a000,1,0,c8b6a7e4,...) at ng_tee_rcvdata+0x156/frame 0xc8b6a7b8
    ng_apply_item(0,c4c3e812,6,c4a645e0,1,...) at ng_apply_item+0x22d/frame 0xc8b6a814
    ng_snd_item(c4a4a000,0,c4793c80,0,c46a9900,...) at ng_snd_item+0x1a0/frame 0xc8b6a848
    ng_pppoe_rcvdata_ether(c4793d00,c4a4a000,c4a645e0,34,1c5,...) at ng_pppoe_rcvdata_ether+0x2a3/frame 0xc8b6a8c4
    ng_apply_item(0,c47b0000,c48f7b00,0,c8b6a954,...) at ng_apply_item+0x22d/frame 0xc8b6a920
    ng_snd_item(c4a4a000,0,c47b9a00,0,c47b0000,...) at ng_snd_item+0x1a0/frame 0xc8b6a954
    ng_ether_input_orphan(c47b0000,c48f7b00,0,b1ce9d7,a1d20db3,...) at ng_ether_input_orphan+0x66/frame 0xc8b6a974
    ether_demux(c47b0000,c48f7b00,6,c4317c40,c4317c40,...) at ether_demux+0x1f9/frame 0xc8b6a9a0
    ether_nh_input(c48f7b00,c8b6aa14,0,0,0,...) at ether_nh_input+0x37e/frame 0xc8b6a9f0
    netisr_dispatch_src(9,0,c48f7b00) at netisr_dispatch_src+0x8b/frame 0xc8b6aa38
    netisr_dispatch(9,c48f7b00) at netisr_dispatch+0x20/frame 0xc8b6aa4c
    ether_input(c47b0000,c48f7b00,0,c8b6aae0,c0cd1bf4,...) at ether_input+0x19/frame 0xc8b6aa5c
    vlan_input(c424fc00,c48f7b00,c4317c40,c4bfd008,c43d0100,...) at vlan_input+0x1a8/frame 0xc8b6aa8c
    ether_demux(c424fc00,c48f7b00,6,7f8,c4bfd008,...) at ether_demux+0xaf/frame 0xc8b6aab8
    ether_nh_input(c48f7b00,c4bbd200,2f4,c8b6ab70,c0b4e3ba,...) at ether_nh_input+0x37e/frame 0xc8b6ab04
    netisr_dispatch_src(9,0,c48f7b00) at netisr_dispatch_src+0x8b/frame 0xc8b6ab4c
    netisr_dispatch(9,c48f7b00) at netisr_dispatch+0x20/frame 0xc8b6ab60
    ether_input(c424fc00,c48f7b00,c43a4000,745,c8b6abb8,...) at ether_input+0x19/frame 0xc8b6ab70
    vr_rxeof(0,0,c4317c40,c8b6abd8,46,...) at vr_rxeof+0x1f1/frame 0xc8b6abb8
    vr_int_task(c43cc000,1,c8b6ac18,c12bc042,c426bb80,...) at vr_int_task+0x123/frame 0xc8b6abe8
    taskqueue_run_locked(c8b6ac88,c128c712,0,c8b6ac44,c4310008,...) at taskqueue_run_locked+0xee/frame 0xc8b6ac2c
    taskqueue_run(c4269980) at taskqueue_run+0xa3/frame 0xc8b6ac50
    taskqueue_fast_run(0,0,246,0,0,...) at taskqueue_fast_run+0x11/frame 0xc8b6ac5c
    intr_event_execute_handlers(109,c4269900,c1428033,55a,0,...) at intr_event_execute_handlers+0xaa/frame 0xc8b6ac88
    ithread_loop(c41b5e70,c8b6ad08,0,0,0,...) at ithread_loop+0x80/frame 0xc8b6acc4
    fork_exit(c0cb63f0,c41b5e70,c8b6ad08) at fork_exit+0xa3/frame 0xc8b6acf4
    fork_trampoline() at fork_trampoline+0x8/frame 0xc8b6acf4
    --- trap 0, eip = 0, esp = 0xc8b6ad40, ebp = 0 ---
    db:0:kdb.enter.default>  ps
      pid  ppid  pgrp   uid   state   wmesg     wchan    cmd
    30982   285   285     0  R                           php-fpm
    71440 73267    21     0  S       nanslp   0xc1efd568 sleep
    60001 59799 60001     0  Ss      (threaded)          charon
    100155                   S       uwait    0xc57e6580 charon
    100154                   S       uwait    0xc57e6500 charon
    100153                   S       uwait    0xc4a6f280 charon
    100152                   S       uwait    0xc57e6400 charon
    100151                   S       uwait    0xc57e6780 charon
    100150                   S       uwait    0xc57e7b80 charon
    100149                   S       uwait    0xc57e7c00 charon
    100148                   S       uwait    0xc57e6a80 charon
    100147                   S       uwait    0xc57e6800 charon
    100146                   S       uwait    0xc4e94b00 charon
    100145                   S       uwait    0xc57e6600 charon
    100144                   S       uwait    0xc4a6f380 charon
    100143                   S       select   0xc46af6e4 charon
    100142                   S       select   0xc46af724 charon
    100141                   S       accept   0xc57d61e6 charon
    100140                   S       uwait    0xc57e9700 charon
    100060                   S       sigwait  0xc674c000 charon
    59799     1 59799     0  Ss      select   0xc6703824 starter
    85529 89861 89861     0  S+      ttyin    0xc4313470 sh
    90198 78569 90198     0  Ss      (threaded)          sshlockout_pf
    100124                   S       nanslp   0xc1efd568 sshlockout_pf
    100055                   S       piperd   0xc47f47f8 sshlockout_pf
    89861     1 89861     0  Ss+     wait     0xc48ac8d0 login
    89557     1     1     0  S       nanslp   0xc1efd568 getty
    87839 87796 87796     0  S       nanslp   0xc1efd568 minicron
    87796     1 87796     0  Ss      wait     0xc47985e0 minicron
    87654 87281 87281     0  S       nanslp   0xc1efd568 minicron
    87281     1 87281     0  Ss      wait     0xc48ab000 minicron
    86900 86382 86382     0  S       nanslp   0xc1efd568 minicron
    86382     1 86382     0  Ss      wait     0xc4f96000 minicron
    84022     1 84022     0  Ss      nanslp   0xc1efd568 cron
    78569     1 78569     0  Ss      select   0xc4a361e4 syslogd
    73996     1 73996     0  Ss      select   0xc4a38ba4 igmpproxy
    73267     1    21     0  S+      wait     0xc4f978d0 sh
    65451     1 65451     0  Ss      (threaded)          ntpd
    100056                   S       select   0xc46ac7a4 ntpd
    60741     1 60741     0  Ss      (threaded)          filterdns
    100101                   S       uwait    0xc4d6b500 signal-thread
    100100                   S       uwait    0xc57e9680 filterdns
    54653     1 54653  1002  Ss      select   0xc4437ba4 dhcpd
    48648     1 48359 65534  S       select   0xc4a362e4 dnsmasq
    44941     1 44633     0  S       kqread   0xc4ea7b80 lighttpd
    35762 35604 35604     0  S       piperd   0xc47f44c8 rrdtool
    35604     1 35604     0  Ss      select   0xc4a365a4 apinger
    32684     1 32684     0  Ss      select   0xc4a36824 inetd
    32208     1 32208     0  Ss      select   0xc4a36224 openvpn
    32123     1 32123     0  Ss      bpf      0xc4314c00 filterlog
    30648     1 30648     0  Ss      select   0xc4a36664 openvpn
    29293     1 29293     0  Ss      select   0xc4a366a4 openvpn
    28370     1 28370     0  Ss      select   0xc4a36ba4 openvpn
    25601     1 25601     0  Ss      select   0xc4a376e4 hostapd
    22824     1 22824    65  Ss      select   0xc4a37c64 dhclient
    19122     1 19122     0  Ss      select   0xc4a37ce4 dhclient
     8090     1  8090     0  Ss      (threaded)          sshlockout_pf
    100073                   S       nanslp   0xc1efd568 sshlockout_pf
    100072                   S       uwait    0xc47bae80 sshlockout_pf
     7521     1  7521     0  Ss      select   0xc4a38924 sshd
     6233     1  6233     0  Ss      (threaded)          mpd5
    100068                   S       select   0xc41d6ba4 mpd5
      320     1   320     0  Ss      select   0xc4437aa4 devd
      302   300   300     0  S       kqread   0xc47baa80 check_reload_status
      300     1   300     0  Ss      kqread   0xc47baa00 check_reload_status
      285     1   285     0  Ss      kqread   0xc4792700 php-fpm
       54     0     0     0  DL      mdwait   0xc442b000 [md1]
       49     0     0     0  DL      mdwait   0xc442d800 [md0]
       20     0     0     0  DL      syncer   0xc1f1cac4 [syncer]
       19     0     0     0  DL      vlruwt   0xc4799000 [vnlru]
       18     0     0     0  DL      psleep   0xc1f1c204 [bufdaemon]
       17     0     0     0  DL      pollid   0xc1efbf30 [idlepoll]
        9     0     0     0  DL      pgzero   0xc2047d20 [pagezero]
        8     0     0     0  DL      psleep   0xc2047a44 [vmdaemon]
        7     0     0     0  DL      psleep   0xc20a7484 [pagedaemon]
        6     0     0     0  DL      waiting_ 0xc20a1d8c [sctp_iterator]
        5     0     0     0  DL      pftm     0xc0f622c0 [pf purge]
       16     0     0     0  DL      (threaded)          [usb]
    100038                   D       -        0xc43a8d34 [usbus1]
    100037                   D       -        0xc43a8d04 [usbus1]
    100036                   D       -        0xc43a8cd4 [usbus1]
    100035                   D       -        0xc43a8ca4 [usbus1]
    100034                   D       -        0xc443cb5c [usbus0]
    100033                   D       -        0xc443cb2c [usbus0]
    100032                   D       -        0xc443cafc [usbus0]
    100031                   D       -        0xc443cacc [usbus0]
        4     0     0     0  DL      (threaded)          [cam]
    100045                   D       -        0xc1e3e4a8 [scanner]
    100017                   D       -        0xc1e3e600 [doneq0]
        3     0     0     0  DL      crypto_r 0xc2046978 [crypto returns]
        2     0     0     0  DL      crypto_w 0xc20468b8 [crypto]
       15     0     0     0  DL      -        0xc1e58680 [rand_harvestq]
       14     0     0     0  DL      (threaded)          [geom]
    100010                   D       -        0xc209dde0 [g_down]
    100009                   D       -        0xc209dddc [g_up]
    100008                   D       -        0xc209ddd8 [g_event]
       13     0     0     0  DL      sleep    0xc1e14bb8 [ng_queue0]
       12     0     0     0  RL      (threaded)          [intr]
    100043                   I                           [swi1: pfsync]
    100041                   I                           [swi1: pf send]
    100039                   I                           [swi0: uart uart]
    100030                   I                           [irq12: ohci0 ehci0]
    100029                   I                           [irq15: vr2 ata1]
    100028                   I                           [irq14: ata0]
    100025                   I                           [irq9: ath0]
    100023                   I                           [swi6: Giant taskq]
    100021                   I                           [swi6: task queue]
    100015                   Run     CPU 0               [swi5: fast taskq]
    100006                   I                           [swi3: vm]
    100005                   I                           [swi4: clock]
    100004                   I                           [swi1: netisr 0]
       11     0     0     0  RL                          [idle: cpu0]
        1     0     1     0  SLs     wait     0xc42662f0 [init]
       10     0     0     0  DL      audit_wo 0xc20a5d88 [audit]
        0     0     0     0  DLs     (threaded)          [kernel]
    100044                   D       -        0xc4269800 [CAM taskq]
    100027                   D       -        0xc43cf700 [ath0 net80211 taskq]
    100026                   D       -        0xc43cf780 [ath0 taskq]
    100024                   D       -        0xc4269180 [thread]
    100022                   D       -        0xc4269300 [ffs_trim taskq]
    100020                   D       -        0xc4269480 [acpi_task_2]
    100019                   D       -        0xc4269480 [acpi_task_1]
    100018                   D       -        0xc4269480 [acpi_task_0]
    100016                   D       -        0xc4269880 [kqueue taskq]
    100011                   D       -        0xc426aa80 [firmware taskq]
    100000                   D       swapin   0xc209de64 [swapper]
    db:0:kdb.enter.default>[/thread]
    


  • And you don't have "IP Random id generation" enabled?



  • @w0w:

    And you don't have "IP Random id generation" enabled?

    "IP Random id generation" is off. This setting I have never changed. Otherwise, I have only MSS clamping enabled and the Unity plugin disabled. "IPsec Mobile Client Support" is also turned off. All other settings are defaults.

    Apparently the kernel timesource is not the cause of the error. Could someone have another look at the dump? What information is still needed?



  • We need to use bugtracker to report the bug. Please report this issue here https://redmine.pfsense.org/
    Don't link the forum page, post your dumps and I'll add mine too.
    FYI I don't have changed MSS and unity settings.





  • Similar issue here. If I connect via VPN, PFSense stops and restarts - the IPSEC connection lasts between 30 seconds and 5 minutes. This issue is 100% reproducible - I've crashed my box about six times today.

    Fatal double fault:
    eip = 0xc12c62a8
    esp = 0xecf4cff8
    ebp = 0xecf4d000
    cpuid = 0; apic id = 00
    panic: double fault
    cpuid = 0
    KDB: enter: panic
    panic.txt0600001412471723700  7136 ustarrootwheeldouble faultversion.txt06000025112471723700  7614 ustarrootwheelFreeBSD 10.1-RELEASE-p4 #0 36d7dec(releng/10.1)-dirty: Thu Jan 22 15:12:38 CST 2015
        root@pfsense-22-i386-builder:/usr/obj.i386/usr/pfSensesrc/src/sys/pfSense_SMP.10

    PFSense 2.2 - upgraded from 2.1.5

    Hardware:
    CPU: Intel(R) Atom(TM) CPU N270 @ 1.60GHz
    Mobo: KINO-945GSE
    Storage: 2 GB CF Card
    Dual LAN: Realtek PCIe 8111CP GbE controller

    IPSEC Details:
    Mobile Client
    No IP Compression
    Unity plugin disabled
    IKE v1
    Virtual IP Address Assigned to Clients
    IP Random ID Generation at default value (default is 0: sequential IP IDs)

    Interfaces:
    RE0: WAN: PPOE
    RE1: LAN/OPT1/OPT2 using VLAN tagging

    May try a clean install of v2.2 if you think there's any mileage in it.

    pfsenseCrashDump.txt



  • @afasoas: are you running 32 or 64 bit version? So far I think all the reports are coming from x86 versions. Is this reproducible under x64??



  • 32 bit.
    Well spotted.
    I will add my crash dump to the bug tracker shortly.



  • Upgraded my hardware so I could run the 64-bit version. No issues to report thus far. IPSEC seems solid and stable.



  • Great solution, but this is like cutting the head and sewing back a new one, more "brainful".
    It could be also driver Ethernet issue with physical low memory installed. I can only wait when somebody really smart will check our crash dumps to find out the reason of double triple crash and panic.



  • @w0w:

    Great solution, but this is like cutting the head and sewing back a new one, more "brainful".
    It could be also driver Ethernet issue with physical low memory installed. I can only wait when somebody really smart will check our crash dumps to find out the reason of double triple crash and panic.

    I appreciate that this isn't the most helpful solution. I just wanted to confirm that the problem went away using the same configuration with a 64-bit version.

    On Edit: I realise that you are using an Atom D2500 - you can run the 64-bit version of pfSense on it, if that helps?



  • I can, really, but what if I don't need it? My typical memory usage is less then 6% and CPU is mostly 90% in peak (300Mbit internet, three clients). I can buy some XEON based proliant G8 but why? :) 
    Maybe I'll move to x64 platform if we can't trust x86 anymore. But I need an answer for the question. DO we really need to move to x64 just because x86 is not supported or what?
    I think i'll wait for answer before buying some needless hardware. :)



  • And what about the ALIX boards? As far as I know they are all 32bit.


  • Banned

    @sh0gun:

    And what about the ALIX boards? As far as I know they are all 32bit.

    Not getting any IPsec panics on Alix. (Also, make sure you did not enable some stupid features, like the infamous "Insert a stronger id into IP header of packets passing through the filter.")



  • But your hardware is already 64-bit capable, at least as far as pfSense is concerned!
    Yes Intel don't provide 64-bit video drivers but seems to be a non-issue here.

    @w0w:

    My system is D2500CC mini-ITX motherboard from Intel, all embedded into it.



  • @doktornotor:

    @sh0gun:

    And what about the ALIX boards? As far as I know they are all 32bit.

    Not getting any IPsec panics on Alix. (Also, make sure you did not enable some stupid features, like the infamous "Insert a stronger id into IP header of packets passing through the filter.")

    I don't think that hiding your client OS unique ID behind firewall is so stupid as you think about it.



  • @afasoas:

    But your hardware is already 64-bit capable, at least as far as pfSense is concerned!
    Yes Intel don't provide 64-bit video drivers but seems to be a non-issue here.

    @w0w:

    My system is D2500CC mini-ITX motherboard from Intel, all embedded into it.

    Yep. But what is the point to use 64-bit OS with 2GB of RAM? It does not fix the problem in 32-bit version also :) There is some bug, that must be fixed and this is good, maybe, that it is pointed now to 32-bit version only, but next time it could be related to 64-bit only, so migrating between platforms is useless for me, until I read something like "64-bit freebsd is more secure and stable, don't use 32-bit anymore".



  • Migrating between platforms resolves your problem, for the time being.
    If your memory usage is at 6% then I figure there should not be a problem switching over to 64-bit.



  • @w0w:

    …so migrating between platforms is useless for me, until I read something like "64-bit freebsd is more secure and stable, don't use 32-bit anymore".

    Ahh, I think you mean this:
    "[_…64 bit is more widely used, what we test the most with, and what most of our development is done using.

    32 bit is a dying breed. FreeNAS and DragonflyBSD both just put out their last releases with 32 bit support. While we'll still continue to support 32 bit in 2.2.x releases and possibly beyond that, ending 32 bit support is certainly on the road map and will happen sooner than later.

    There is no reason to use 32 bit over 64 today, if your hardware is 64 bit capable, you should only be running 64 bit._](https://forum.pfsense.org/index.php?topic=84679.msg464432#msg464432)"

    Chris Buechler, November 27th, 2014


  • Rebel Alliance Developer Netgate



  • Ok… At least I'll give it a try. Later, next week maybe :)



  • Still, do we have any clues on the issue itself? It affects Alix boards and similar hardware to at least some degree (besides some other reports, I had to downgrade some production boxes myself due to random reboots which I am sure are related to all this). Unfortunately I couldn't find a specific trigger but there seem to be several crash dumps and reproducible configs available



  • https://redmine.pfsense.org/issues/4454 Looks like something moving forward, at least for me, thanks for Chris Buechler



  • Ok, installed amd64, no more kernel panic when "Insert strong ID…" option enabled. So yes, I can confirm that x86 platform is affected and amd64 not.



  • Can you try to set net.inet.ipsec.directdispatch to 0 and see if the panic goes away?



  • Update my pfsense in lab on 2.2.1 same behavior
    then i tried  to

    set net.inet.ipsec.directdispatch to 0

    looks good so far. stable since about 30 minutes (before after about 30 seconds i get a kernel panic)



  • Can you describe your WAN interface?



  • in our lab i used 3 pfsenses

    one so to say provider with an pppoe server (Version 2.2.1 Vmware)

    then i have one pfsense which stands for my company firewall (Version 2.1.5 Vmware)

    and i have another pfsense which stand for my home pfsense  (Version 2.2.1 Alix2d3)

    both pfsenses from company and home have a pppoe wan interface which is connected to my provider pfsense

    on my home pfsense i also have a vlan tag added like i have to do it at my real home pfsense.



  • Ok there is an open issue for this scenario already.

    Thank you for the information.



  • yes my scenario is like described befor in this thread.
    but set net.inet.ipsec.directdispatch to 0 seems to "workaround" the issue
    so there is may be hope for all 32 Bit Users  ;)



  • ermal, do you need my report too? :)
    Actually I am the man who reported the issue. But I moved my box on to amd64 version…
    I ask because I have troubles to restoring old x86 backup, so new installation take time... I can do it but only if it really needed.



  • Nope the scenario is clear.



  • i can reproduce the scenario in our lab any time. The VM's are set up already just need to put back the config into the alix board.
    If you need just let me know.



  • @flix87:

    Update my pfsense in lab on 2.2.1 same behavior
    then i tried  to

    set net.inet.ipsec.directdispatch to 0

    looks good so far. stable since about 30 minutes (before after about 30 seconds i get a kernel panic)

    Hello, had the same issues with reboot`s on 8 devices.

    Hardware ALIX.2 v0.99m tinyBIOS V1.4a.
    Pf.Version 2.2.2-RELEASE (i386) built on Mon Apr 13 20:10:33 CDT 2015.

    After the set of "net.inet.ipsec.directdispatch to 0" all systems works fine. No reboots, no systempanics, stable 3 days ago yet.  :)

    What is done with this adjustment ? Will this fix embedded as standard for further versions ?

    Best regards

    eeit



  • net.inet.ipsec.directdispatch=0

    fixes the issue on ALIX 2D13 (32bit) with IPSEC for me. Thank you very much! Marked as solved.



  • Can you please having issues with this confirm that you are running Proxy arps?
    If yes, can you try with the fix at https://redmine.pfsense.org/issues/4685 last comment.



  • @ermal:

    Can you please having issues with this confirm that you are running Proxy arps?
    If yes, can you try with the fix at https://redmine.pfsense.org/issues/4685 last comment.

    I'm having the same issue (see https://forum.pfsense.org/index.php?topic=94140.0) and afaik i'm not using proxy arps. None of the previous tunables (including net.inet.ipsec.directdispatch=0) did fix it!



  • I'm getting this error on my x86 pfsense box when booting up.

    How can I set  set net.inet.ipsec.directdispatch to 0?

    I've connected a monitor and keyboard, booted up my system
    Selected '3. Escape to loader prompt.'
    I type:: 'set net.inet.ipsec.directdispatch=0'
    Then type: 'boot'

    I get the same kernel panic? Am I doing this wrong? Can anyone help me? :)

    Thanks,
    Brad



  • @bab5470:

    I'm getting this error on my x86 pfsense box when booting up.

    Just make sure you're running 2.2.4, that's been fixed in 2.2.3 and newer. directdispatch is automatically set accordingly for 32 bit to avoid that crash. If you're already on 2.2.4, then you have a different problem, start a new thread detailing what you're seeing.



  • How can I upgrade if I can't even boot up the box though? I assume I could re-install from scratch but I don't really want to lose all my settings.


Log in to reply