PfSense 2.5 Crash report



  • Random crash occurs at random times. System reboots and is active again. Been having these since trying 2.5.

    info.0 textdump.tar.0


  • LAYER 8

    the crash was caused by ntpd, that is strange
    is always the same process that cause the crash?
    if the process change it's probably a faulty memory ... otherwise ... idk never saw that crash



  • I'll get back to you on that, cant find the other crash reports.
    Are the old ones stored on the pfsense box somewhere?


  • LAYER 8

    nope, afaik they are deleted / overwritten


  • Rebel Alliance Developer Netgate

    db:0:kdb.enter.default>  bt
    Tracing pid 8411 tid 100736 td 0xfffff800099ea000
    rtfree() at rtfree+0x6f/frame 0xfffffe004cc4e6b0
    in_pcbladdr() at in_pcbladdr+0x411/frame 0xfffffe004cc4e750
    in_pcbconnect_setup() at in_pcbconnect_setup+0x2e5/frame 0xfffffe004cc4e810
    in_pcbconnect_mbuf() at in_pcbconnect_mbuf+0x3f/frame 0xfffffe004cc4e850
    udp_connect() at udp_connect+0x16c/frame 0xfffffe004cc4e8a0
    soconnect() at soconnect+0xad/frame 0xfffffe004cc4e8e0
    kern_connectat() at kern_connectat+0xe4/frame 0xfffffe004cc4e940
    sys_connect() at sys_connect+0x77/frame 0xfffffe004cc4e980
    amd64_syscall() at amd64_syscall+0x369/frame 0xfffffe004cc4eab0
    fast_syscall_common() at fast_syscall_common+0x101/frame 0xfffffe004cc4eab0
    

    It crashed when trying to free a route. It's possible that is hardware but it's also possible it's due to changes in 2.5.0. We enabled RADIX_MPATH for multipath routing, and also enabled multipath in FRR. If you are using anything that might be touching that code, it's possible that the crash is due to some instability there. We have not seen any crashes here but we also have not extensively tested multipath.



  • I had random crashes as well after running 2.5.0 - still don't know what caused them, but I got them to go away.

    "The important setting is "IPv6 Configuration Type" for the WAN interface. It this is set to DHCP6, as it was by default, the firewalls crash regularly. If it is set to "None", there are no crashes (or at least they are so infrequent that I haven't seen them yet). Also, as described above, DHCP6 causes a lot of log entries and blocks updates."

    See here for more info



  • Fatal trap 12: page fault while in kernel mode
    cpuid = 0; apic id = 10
    fault virtual address = 0x70
    fault code = supervisor read data, page not present
    instruction pointer = 0x20:0xffffffff80e7d451
    stack pointer = 0x28:0xfffffe0095aa03c0
    frame pointer = 0x28:0xfffffe0095aa04c0
    code segment = base 0x0, limit 0xfffff, type 0x1b
    = DPL 0, pres 1, long 1, def32 0, gran 1
    processor eflags = interrupt enabled, resume, IOPL = 0
    current process = 7007 (bgpd)
    version.txt0600007113567634117 7546 ustarrootwheelFreeBSD 12.0-RELEASE-p10 fe839027797(RELENG_2_5) pfSense



  • I've had another, looks pretty much the same kind of, with my limited knowledge.
    In the mean time between crashes I have updated to a newer version.

    info(1).0 textdump.tar(1).0



  • @mi8088 This setting is and has been set to none thru all the crashes. So does not seem related.



  • @jimp I do not know for certain that I am using this. Any way to check?
    I see rtfree again in the dump.
    I have uploaded a new crash report.



  • <118>>>> Removing unnecessary packages... done.
    <118>>>> Cleanup pkg cache... done.
    <118> Starting package OpenBGPD...done.
    <118> Starting /usr/local/etc/rc.d/bgpd.sh...done.
    <118>pfSense 2.5.0-DEVELOPMENT amd64 Tue Dec 03 01:48:44 EST 2019
    <118>Bootup complete
    <6>gif4: link state changed to DOWN
    <6>gif4: link state changed to UP
    <6>gre0: link state changed to DOWN
    <6>gre0: link state changed to UP
    <6>ovpnc9: link state changed to UP
    <6>ovpnc7: link state changed to UP
    <6>ovpnc3: link state changed to UP
    <3>rn_delete: Orphaned Mask 0xfffff8008935af40 at 0
    <3>rn_delete: Orphaned Mask 0xfffff8008a218960 at 0
    <3>rn_delete: Orphaned Mask 0xfffff8008a4f8160 at 0
    <3>rn_delete: Orphaned Mask 0xfffff80086b5d400 at 0
    <3>rn_delete: Orphaned Mask 0xfffff8009bdb9e80 at 0
    <3>rn_delete: Orphaned Mask 0xfffff8008af86e00 at 0
    <3>rn_delete: Orphaned Mask 0xfffff80522579380 at 0
    <3>rn_delete: Orphaned Mask 0xfffff800854b8420 at 0
    <3>rn_delete: Orphaned Mask 0xfffff805a6e20700 at 0
    <3>rn_delete: Orphaned Mask 0xfffff8005f47d600 at 0
    <3>rn_delete: Orphaned Mask 0xfffff8042a919f20 at 0
    <3>rn_delete: Orphaned Mask 0xfffff8009c951a00 at 0
    <3>rn_delete: Orphaned Mask 0xfffff805a692b0c0 at 0
    <3>rn_delete: Orphaned Mask 0xfffff8010454e480 at 0
    <3>rn_delete: Orphaned Mask 0xfffff8041c01dec0 at 0
    <3>rn_delete: Orphaned Mask 0xfffff80106e552c0 at 0
    <3>rn_delete: Orphaned Mask 0xfffff80086c537a0 at 0
    <3>rn_delete: Orphaned Mask 0xfffff80480d99280 at 0
    <3>rn_delete: Orphaned Mask 0xfffff801068f87e0 at 0
    <3>rn_delete: Orphaned Mask 0xfffff8008546e4e0 at 0
    <3>rn_delete: Orphaned Mask 0xfffff80088e3e780 at 0
    <3>rn_delete: Orphaned Mask 0xfffff80107cb7bc0 at 0
    <3>rn_delete: Orphaned Mask 0xfffff80087f0c540 at 0
    <3>rn_delete: Orphaned Mask 0xfffff8010adb4420 at 0
    <3>rn_delete: Orphaned Mask 0xfffff8010ad90f80 at 0
    <3>rn_delete: Orphaned Mask 0xfffff8010adbbf00 at 0
    <3>rn_delete: Orphaned Mask 0xfffff8010c6ac4c0 at 0
    <3>rn_delete: Orphaned Mask 0xfffff8010c6ac2c0 at 0
    <3>rn_delete: Orphaned Mask 0xfffff8010acdbb40 at 0
    <3>rn_delete: Orphaned Mask 0xfffff804235c2020 at 0
    <3>rn_delete: Orphaned Mask 0xfffff8011f167e00 at 0
    <3>rn_delete: Orphaned Mask 0xfffff8011f1689e0 at 0
    <3>rn_delete: Orphaned Mask 0xfffff8010ad9b820 at 0
    <3>rn_delete: Orphaned Mask 0xfffff805e29b5780 at 0
    <3>rn_delete: Orphaned Mask 0xfffff800863be2e0 at 0
    <3>rn_delete: Orphaned Mask 0xfffff8008a534e40 at 0
    <3>rn_delete: Orphaned Mask 0xfffff804f3fce800 at 0
    <3>rn_delete: Orphaned Mask 0xfffff801207a6e40 at 0
    <3>rn_delete: Orphaned Mask 0xfffff80087015860 at 0
    <3>rn_delete: Orphaned Mask 0xfffff8012134d660 at 0
    <3>rn_delete: Orphaned Mask 0xfffff80121f6a3e0 at 0
    <3>rn_delete: Orphaned Mask 0xfffff801224227c0 at 0
    <3>rn_delete: Orphaned Mask 0xfffff80087e48000 at 0
    <3>rn_delete: Orphaned Mask 0xfffff805a64b7f80 at 0
    <3>rn_delete: Orphaned Mask 0xfffff801229b2c40 at 0
    <3>rn_delete: Orphaned Mask 0xfffff80088842280 at 0
    <3>rn_delete: Orphaned Mask 0xfffff801229b6500 at 0
    <3>rn_delete: Orphaned Mask 0xfffff80122a38160 at 0
    <3>rn_delete: Orphaned Mask 0xfffff801229b2640 at 0
    <3>rn_delete: Orphaned Mask 0xfffff805482ff4a0 at 0
    <3>rn_delete: Orphaned Mask 0xfffff800880759e0 at 0
    <3>rn_delete: Orphaned Mask 0xfffff80453886c40 at 0
    <3>rn_delete: Orphaned Mask 0xfffff80480033c00 at 0
    <3>rn_delete: Orphaned Mask 0xfffff8041e6e7240 at 0
    <3>rn_delete: Orphaned Mask 0xfffff80522707a40 at 0
    <3>rn_delete: Orphaned Mask 0xfffff80122908da0 at 0
    <3>rn_delete: Orphaned Mask 0xfffff801229af700 at 0
    <3>rn_delete: Orphaned Mask 0xfffff80122908880 at 0
    <3>rn_delete: Orphaned Mask 0xfffff80418ceb7c0 at 0
    <3>rn_delete: Orphaned Mask 0xfffff8041e7420e0 at 0
    <3>rn_delete: Orphaned Mask 0xfffff80480033e40 at 0
    <3>rn_delete: Orphaned Mask 0xfffff8010ad1c980 at 0
    <3>rn_delete: Orphaned Mask 0xfffff80522707b80 at 0
    <3>rn_delete: Orphaned Mask 0xfffff800896eeb60 at 0
    <3>rn_delete: Orphaned Mask 0xfffff8008864b680 at 0
    <3>rn_delete: Orphaned Mask 0xfffff8010adba3c0 at 0
    <3>rn_delete: Orphaned Mask 0xfffff8010add04e0 at 0
    <3>rn_delete: Orphaned Mask 0xfffff801212147a0 at 0
    <3>rn_delete: Orphaned Mask 0xfffff80086c1d9a0 at 0
    <3>rn_delete: Orphaned Mask 0xfffff8025b513460 at 0
    <3>rn_delete: Orphaned Mask 0xfffff8008512b420 at 0
    <3>rn_delete: Orphaned Mask 0xfffff8010e61a320 at 0
    <3>rn_delete: Orphaned Mask 0xfffff8025bbb1900 at 0
    <3>rn_delete: couldn't find our annotation

    Fatal trap 12: page fault while in kernel mode
    cpuid = 2; apic id = 12
    fault virtual address = 0x70
    fault code = supervisor read data, page not present
    instruction pointer = 0x20:0xffffffff80e7d451
    stack pointer = 0x28:0xfffffe0095c683c0
    frame pointer = 0x28:0xfffffe0095c684c0
    code segment = base 0x0, limit 0xfffff, type 0x1b
    = DPL 0, pres 1, long 1, def32 0, gran 1
    processor eflags = interrupt enabled, resume, IOPL = 0
    current process = 52869 (bgpd)
    version.txt0600007113571735606 7546 ustarrootwheelFreeBSD 12.0-RELEASE-p10 fe839027797(RELENG_2_5) pfSense


  • Netgate Administrator

    Are you using FRR there? It looks like you still have OpenBGP running.

    You should switch to FRR at this point.



  • @stephenw10 said in PfSense 2.5 Crash report:

    Are you using FRR there? It looks like you still have OpenBGP running.

    You should switch to FRR at this point.

    i have to using openbgpd, so frr will system down.

    https://forum.netgate.com/topic/147637/frr-with-get-two-bgp-full-route-tables-from-neighbors-with-openvpn-update-source/16



  • pkg install pfSense-pkg-Quagga_OSPF
    Updating pfSense-core repository catalogue...
    pfSense-core repository is up to date.
    Updating pfSense repository catalogue...
    pfSense repository is up to date.
    All repositories are up to date.
    Checking integrity... done (0 conflicting)
    The following 2 package(s) will be affected (of 0 checked):

    New packages to be INSTALLED:
    pfSense-pkg-Quagga_OSPF: 0.6.21_4 [pfSense]
    quagga: 1.2.4_7 [pfSense]

    Number of packages to be installed: 2

    The process will require 8 MiB more space.

    Proceed with this action? [y/N]: y
    [1/2] Installing quagga-1.2.4_7...
    ===> Creating groups.
    Using existing group 'quagga'.
    ===> Creating users
    Using existing user 'quagga'.
    [1/2] Extracting quagga-1.2.4_7: 57%
    pkg: Fail to extract /usr/local/lib/libospf.a from package: Lzma library error: Corrupted input data
    [1/2] Extracting quagga-1.2.4_7: 100%



  • <6>ovpnc3: link state changed to DOWN
    <6>ovpnc9: link state changed to DOWN
    <6>ovpnc3: link state changed to UP
    <6>ovpnc9: link state changed to UP
    <3>rn_delete: Orphaned Mask 0xfffff802586875e0 at 0
    <3>rn_delete: Orphaned Mask 0xfffff80259b72d40 at 0
    <3>rn_delete: Orphaned Mask 0xfffff801db1d3240 at 0
    <3>rn_delete: Orphaned Mask 0xfffff80259a5a180 at 0
    <3>rn_delete: Orphaned Mask 0xfffff8025291f840 at 0
    <3>rn_delete: Orphaned Mask 0xfffff80252827840 at 0
    <6>gif4: link state changed to DOWN
    <6>gif4: link state changed to UP
    <3>rn_delete: Orphaned Mask 0xfffff8025a41ce00 at 0
    <3>rn_delete: Orphaned Mask 0xfffff80259bdbc00 at 0
    <3>rn_delete: Orphaned Mask 0xfffff802599090a0 at 0
    <3>rn_delete: couldn't find our annotation

    <3>rn_delete: couldn't find our annotation
    <3>rn_delete: Orphaned Mask 0xfffff8056525a020 at 0
    <3>rn_delete: Orphaned Mask 0xfffff80565bf23c0 at 0
    <3>rn_delete: couldn't find our annotation
    <3>rn_delete: Orphaned Mask 0xfffff80565e9f820 at 0
    <3>rn_delete: couldn't find our annotation
    <3>rn_delete: Orphaned Mask 0xfffff802557bfbc0 at 0
    <3>rn_delete: Orphaned Mask 0xfffff802ba13f0a0 at 0
    <3>rn_delete: couldn't find our annotation
    <3>rn_delete: couldn't find our annotation
    <3>rn_delete: Orphaned Mask 0xfffff80566387460 at 0
    <3>rn_delete: Orphaned Mask 0xfffff80568e73aa0 at 0
    <3>rn_delete: Orphaned Mask 0xfffff8025a3dd680 at 0
    <3>rn_delete: couldn't find our annotation
    <3>rn_delete: Orphaned Mask 0xfffff80565bf1100 at 0
    <3>rn_delete: Orphaned Mask 0xfffff80259ea4760 at 0
    <3>rn_delete: Orphaned Mask 0xfffff80255150260 at 0
    <3>rn_delete: Orphaned Mask 0xfffff80566e7c040 at 0
    <3>rn_delete: Orphaned Mask 0xfffff80568610820 at 0
    <3>rn_delete: Orphaned Mask 0xfffff80595b31b60 at 0
    <3>rn_delete: Orphaned Mask 0xfffff80042723be0 at 0
    <3>rn_delete: Orphaned Mask 0xfffff80568e73a40 at 0
    <3>rn_delete: couldn't find our annotation
    <3>rn_delete: Orphaned Mask 0xfffff802bade9b00 at 0
    <3>rn_delete: couldn't find our annotation

    <3>rn_delete: Orphaned Mask 0xfffff8025a3cc4e0 at 0
    <3>rn_delete: couldn't find our annotation
    <3>rn_delete: Orphaned Mask 0xfffff8021ef44420 at 0

    <3>rn_delete: Orphaned Mask 0xfffff805695f5500 at 0

    <3>rn_delete: Orphaned Mask 0xfffff8056b2d2cc0 at 0
    <3>rn_delete: couldn't find our annotation
    <3>rn_delete: couldn't find our annotation
    <6>ovpnc7: link state changed to DOWN
    <6>ovpnc7: link state changed to UP
    <3>rn_delete: Orphaned Mask 0xfffff8025199b940 at 0
    <3>rn_delete: couldn't find our annotation

    Fatal trap 12: page fault while in kernel mode
    cpuid = 0; apic id = 10
    fault virtual address = 0x70
    fault code = supervisor read data, page not present
    instruction pointer = 0x20:0xffffffff80e7d451
    stack pointer = 0x28:0xfffffe0095cd43c0
    frame pointer = 0x28:0xfffffe0095cd44c0
    code segment = base 0x0, limit 0xfffff, type 0x1b
    = DPL 0, pres 1, long 1, def32 0, gran 1
    processor eflags = interrupt enabled, resume, IOPL = 0
    current process = 55611 (bgpd)
    version.txt0600007113572410756 7543 ustarrootwheelFreeBSD 12.0-RELEASE-p10 fe839027797(RELENG_2_5) pfSense



  • <118>Starting syslog...done.
    <118>Starting Secure Shell Services...done.
    <118>Setting up interfaces microcode...done.
    <118>Removed leftover dhcp6c lock file: /tmp/dhcp6c_pppoe0_lock
    <118>Removed leftover dhcp6c lock file: /tmp/dhcp6c_pppoe2_lock
    <118>Configuring loopback interface...done.
    <118>Creating wireless clone interfaces...done.
    <118>Configuring LAGG interfaces...
    <6>lagg0: IPv6 addresses on igb0 have been removed before adding it as a member to prevent IPv6 address scope violation.
    <6>lagg0: link state changed to DOWN
    <6>lagg0: IPv6 addresses on igb1 have been removed before adding it as a member to prevent IPv6 address scope violation.
    <118>done.
    <118>route: writing to routing socket: Network is unreachable
    <118>route: route has not been found
    <118>pfSense 2.5.0-DEVELOPMENT amd64 Tue Dec 10 17:22:41 EST 2019
    <118>Bootup complete
    <6>ovpnc7: link state changed to UP
    <6>ovpnc3: link state changed to UP
    <6>ovpnc7: link state changed to DOWN
    <6>ovpnc9: link state changed to UP
    <6>ovpnc7: link state changed to UP
    <3>rn_delete: Orphaned Mask 0xfffff8055cb12080 at 0
    <3>rn_delete: Orphaned Mask 0xfffff802d00985e0 at 0
    <3>rn_delete: Orphaned Mask 0xfffff8058f3bc6a0 at 0
    <3>rn_delete: Orphaned Mask 0xfffff8049b4891a0 at 0
    <3>rn_delete: Orphaned Mask 0xfffff802d04c7f00 at 0
    <3>rn_delete: Orphaned Mask 0xfffff802d0f55860 at 0
    <3>rn_delete: couldn't find our annotation

    Fatal trap 12: page fault while in kernel mode
    cpuid = 1; apic id = 11
    fault virtual address = 0x70
    fault code = supervisor read data, page not present
    instruction pointer = 0x20:0xffffffff80e7d451
    stack pointer = 0x28:0xfffffe00915553c0
    frame pointer = 0x28:0xfffffe00915554c0
    code segment = base 0x0, limit 0xfffff, type 0x1b
    = DPL 0, pres 1, long 1, def32 0, gran 1
    processor eflags = interrupt enabled, resume, IOPL = 0
    current process = 37390 (bgpd)
    version.txt0600007113574107055 7540 ustarrootwheelFreeBSD 12.0-RELEASE-p10 fe839027797(RELENG_2_5) pfSense


  • Netgate Administrator

    This crash in bgpd seems unrelated to the ntpd issue this thread was started for. You should open a new thread for it.

    Steve


Log in to reply