PFSENSE Fatal trap 12: page fault while in kernel mode



  • I have a pfsense virtual machine in vmware.
    It has 4 vpn/ipsec tunnel mode, 1 with amazon, two with my branch office and 1 with an ISP.

    the problem im facing is, the vpn site-to-site that i have with my ISP should only be up if my interconnection goes down.
    i created this shell script to put the vpn routes only if the interconnection goes down and after it comes up again the routes are removed from spd.

    In my test environment it worked great, but yesterday when i configure the script in my production vmware it starts to crash.

    When the interconnection goes down, the script puts the routes in ipsec tunnel, but when it backs the script is not able to remove it.

    I created a cron job that runs every 1 minute.

    if i run the script manually, when it trys to remove the routes in vpn i got a bug report and pfsense just crash.

    this is my script

    #!/bin/sh
    contador=`setkey -DP | wc -l | awk -F" " '{print $1}'`
    
    ping -c 8 "$1"
    
    if [ "$?" -eq 0 ]; then
            answer=0
            else
            answer=1
    fi
    
    if [ $answer -eq 0 ] && [ $count -eq 40 ]; then
    echo "nothing to do"
    fi
    
    if [ $answer -eq 0 ] && [ $count -ne 40 ]; then
    echo "spddelete 10.200.0.0/16 10.195.0.0/16 any -P in;" | setkey -c
    echo "spddelete 10.195.7.0/24 10.200.0.0/16 any -P out;" | setkey -c
    fi
    
    if [ $answer -ne 0 ] && [ $count -eq 40 ]; then
    echo "spdadd 10.200.0.0/16 10.195.0.0/16 any -P in ipsec esp/tunnel/189.x.x.x-189.x.x.x;" | setkey -c
    echo "spdadd 10.195.7.0/24 10.200.0.0/16 any -P out ipsec esp/tunnel/189.x.x.x-189.x.x.x;" | setkey -c
    fi
    
    if [ $answer -ne 0 ] && [ $count -ne 40 ]; then
    echo "nothing to do"
    fi
    

    the bug report has 3275 lines, but i have a peace of bug that says

    Fatal trap 12: page fault while in kernel mode
    cpuid = 1; apic id = 01
    fault virtual address = 0x378
    fault code = supervisor read data, page not present
    instruction pointer = 0x20:0xffffffff80abf3d9
    stack pointer         = 0x28:0xfffffe001a3fc430
    frame pointer         = 0x28:0xfffffe001a3fc4c0
    code segment = base 0x0, limit 0xfffff, type 0x1b
    = DPL 0, pres 1, long 1, def32 0, gran 1
    processor eflags = interrupt enabled, resume, IOPL = 0
    current process = 12 (irq256: vmx0)
    Copyright © 1992-2014 The FreeBSD Project.
    Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
    FreeBSD is a registered trademark of The FreeBSD Foundation.
    FreeBSD 10.1-RELEASE-p24 #0 f27a67c(releng/10.1)-dirty: Wed Nov  4 16:18:37 CST 2015
        root@pfs22-amd64-builder:/usr/obj.RELENG_2_2.amd64/usr/pfSensesrc/src.RELENG_2_2/sys/pfSense_SMP.10 amd64
    FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
    CPU: Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz (2297.34-MHz K8-class CPU)
      Origin = "GenuineIntel"  Id = 0x206d2  Family = 0x6  Model = 0x2d  Stepping = 2
      Features=0x1fa3fbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,dts,mmx,fxsr,sse,sse2,ss,htt>Features2=0x9e982203 <sse3,pclmulqdq,ssse3,cx16,sse4.1,sse4.2,popcnt,aesni,xsave,osxsave,avx,hv>AMD Features=0x28100800 <syscall,nx,rdtscp,lm>AMD Features2=0x1 <lahf>TSC: P-state invariant
    real memory  = 2147483648 (2048 MB)
    avail memory = 2041303040 (1946 MB)
    Event timer "LAPIC" quality 600
    ACPI APIC Table: <ptltd  apic ="">FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
    FreeBSD/SMP: 1 package(s) x 2 core(s)
    cpu0 (BSP): APIC ID:  0
    cpu1 (AP): APIC ID:  1
    MADT: Forcing active-low polarity and level trigger for SCI
    ioapic0 <version 1.1="">irqs 0-23 on motherboard
    iwi_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi/.
    iwi_bss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (iwi_bss_fw, 0xffffffff806345e0, 0) error 1
    iwi_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi/.
    iwi_ibss: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (iwi_ibss_fw, 0xffffffff80634690, 0) error 1
    iwi_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_iwi/.
    iwi_monitor: If you agree with the license, set legal.intel_iwi.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (iwi_monitor_fw, 0xffffffff80634740, 0) error 1
    wlan: mac acl policy registered
    ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
    ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff8060ce10, 0) error 1
    ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
    ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff8060cec0, 0) error 1
    ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
    ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff8060cf70, 0) error 1
    random: <software, yarrow="">initialized
    kbd1 at kbdmux0
    cryptosoft0: <software crypto="">on motherboard
    padlock0: No ACE support.
    acpi0: <intel 440bx="">on motherboard
    acpi0: Power Button (fixed)
    hpet0: <high precision="" event="" timer="">iomem 0xfed00000-0xfed003ff on acpi0
    Timecounter "HPET" frequency 14318180 Hz quality 950
    cpu0: <acpi cpu="">on acpi0
    cpu1: <acpi cpu="">on acpi0
    attimer0: <at timer="">port 0x40-0x43 irq 0 on acpi0
    Timecounter "i8254" frequency 1193182 Hz quality 0
    Event timer "i8254" frequency 1193182 Hz quality 100
    atrtc0: <at realtime="" clock="">port 0x70-0x71 irq 8 on acpi0
    Event timer "RTC" frequency 32768 Hz quality 0
    Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
    acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
    pcib0: <acpi host-pci="" bridge="">port 0xcf8-0xcff on acpi0
    pci0: <acpi pci="" bus="">on pcib0
    pcib1: <acpi pci-pci="" bridge="">at device 1.0 on pci0
    pci1: <acpi pci="" bus="">on pcib1
    isab0: <pci-isa bridge="">at device 7.0 on pci0
    isa0: <isa bus="">on isab0
    atapci0: <intel piix4="" udma33="" controller="">port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1060-0x106f at device 7.1 on pci0
    ata0: <ata channel="">at channel 0 on atapci0
    ata1: <ata channel="">at channel 1 on atapci0
    pci0: <bridge>at device 7.3 (no driver attached)
    vgapci0: <vga-compatible display="">port 0x1070-0x107f mem 0xec000000-0xefffffff,0xfe000000-0xfe7fffff irq 16 at device 15.0 on pci0
    vgapci0: Boot video device
    mpt0: <lsilogic 1030="" ultra4="" adapter="">port 0x1400-0x14ff mem 0xfeba0000-0xfebbffff,0xfebc0000-0xfebdffff irq 17 at device 16.0 on pci0
    mpt0: MPI Version=1.2.0.0

    i dunno if theres another way to do what i want, i need the vpn only be up if interconnection goes down and vpn get down after interconnection returns.

    Thanks a lot.</lsilogic></vga-compatible></bridge></ata></ata></intel></isa></pci-isa></acpi></acpi></acpi></acpi></at></at></acpi></acpi></high></intel></software></software,></version></ptltd ></lahf></syscall,nx,rdtscp,lm></sse3,pclmulqdq,ssse3,cx16,sse4.1,sse4.2,popcnt,aesni,xsave,osxsave,avx,hv></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,dts,mmx,fxsr,sse,sse2,ss,htt>