Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Frequent Crashing (Page Fault) After Upgrade to 2.8.0 From Latest 2.7

    Scheduled Pinned Locked Moved General pfSense Questions
    82 Posts 7 Posters 5.2k Views 7 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • stephenw10S Offline
      stephenw10 Netgate Administrator
      last edited by

      Ok the core dump yielded useful info but not enough to solve it unfortunately.

      Are you able to load a debug kernel on this and get a core with that running?

      If so grab the pkg here. Install it. Then:
      https://docs.netgate.com/pfsense/en/latest/troubleshooting/debug-kernel.html#booting-the-debug-kernel

      R 1 Reply Last reply Reply Quote 0
      • R Offline
        rfranzke @stephenw10
        last edited by

        @stephenw10 Yes I can install this and run it but not sure the proper way to actually install the pkg file. Enable SSH, copy the file over, and then run it or is there some other method to do this via WebUI?

        K 1 Reply Last reply Reply Quote 0
        • K Offline
          kprovost @rfranzke
          last edited by

          @rfranzke pkg install -U pfSense-kernel-debug-pfSense-2.8.0.b.20250814.0928.pkg

          R 1 Reply Last reply Reply Quote 0
          • R Offline
            rfranzke @kprovost
            last edited by rfranzke

            @kprovost So I think I have this loaded. I just enabled SSH on the box, connected to it, and transferred the download debug kernel file. Then ran the installer:

            [2.8.0-RELEASE][admin@fw2.mdaemon.int]/root: pkg install -y pfSense-kernel-debug-pfSense-2.8.0.b.20250814.0928.pkg
            Updating pfSense-core repository catalogue...
            Fetching meta.conf: 0%
            Fetching data.pkg: 0%
            pfSense-core repository is up to date.
            Updating pfSense repository catalogue...
            Fetching meta.conf: 0%
            Fetching data.pkg: 0%
            pfSense repository is up to date.
            All repositories are up to date.
            Checking integrity... done (0 conflicting)
            The following 1 package(s) will be affected (of 0 checked):

            New packages to be INSTALLED:
            pfSense-kernel-debug-pfSense: 2.8.0.b.20250814.0928 [unknown-repository]

            Number of packages to be installed: 1

            The process will require 254 MiB more space.
            [1/1] Installing pfSense-kernel-debug-pfSense-2.8.0.b.20250814.0928...
            Extracting pfSense-kernel-debug-pfSense-2.8.0.b.20250814.0928: 100% 193 B 0.1kB/s 00:02
            [2.8.0-RELEASE][admin@fw2.mdaemon.int]/root: reboot

            Then added this to the /boot/loader.conf.local file I created:

            kernel="kernel.debug"

            so it boots this kernel all the time. I made the file change before the reboot. Box booted up and I can access it still so assuming its running the debug kernel now. Slick way to tell if its running that new kernel?

            Edit: Maybe this tells us what we want to know?

            [2.8.0-RELEASE][admin@fw2.mdaemon.int]/root: uname -a
            FreeBSD fw2.mdaemon.int 15.0-CURRENT FreeBSD 15.0-CURRENT #1 RELENG_2_8_0-n256082-57273ac5fb19-dirty: Thu Aug 14 09:32:59 CEST 2025 root@nut:/usr/home/kp/netgate/crossbuild-2.8.0/obj/amd64/OdB78hjz/usr/home/kp/netgate/crossbuild-2.8.0/sources/FreeBSD-src-RELENG_2_8_0/amd64.amd64/sys/pfSense-DEBUG amd64
            [2.8.0-RELEASE][admin@fw2.mdaemon.int]/root:

            K 1 Reply Last reply Reply Quote 0
            • K Offline
              kprovost @rfranzke
              last edited by

              @rfranzke said in Frequent Crashing (Page Fault) After Upgrade to 2.8.0 From Latest 2.7:

              [2.8.0-RELEASE][admin@fw2.mdaemon.int]/root: uname -a
              FreeBSD fw2.mdaemon.int 15.0-CURRENT FreeBSD 15.0-CURRENT #1 RELENG_2_8_0-n256082-57273ac5fb19-dirty: Thu Aug 14 09:32:59 CEST 2025 root@nut:/usr/home/kp/netgate/crossbuild-2.8.0/obj/amd64/OdB78hjz/usr/home/kp/netgate/crossbuild-2.8.0/sources/FreeBSD-src-RELENG_2_8_0/amd64.amd64/sys/pfSense-DEBUG amd64

              Yes, that's the kernel we want to be running now.
              It has one extra assertion on top of the default debug kernel. Hopefully that will give us more clues about why the panic happens.

              1 Reply Last reply Reply Quote 1
              • R Offline
                rfranzke
                last edited by rfranzke

                So got the box to crash again but no dump file was created at all this time:

                [2.8.0-RELEASE][admin@fw2.mdaemon.int]/root: cd /var/crash
                [2.8.0-RELEASE][admin@fw2.mdaemon.int]/var/crash: ls
                [2.8.0-RELEASE][admin@fw2.mdaemon.int]/var/crash:

                No file. Any reason this debug version wouldn't create a debug file on panic? Should I have to reconfigure the debug configs as a result of loading the new kernel?

                1 Reply Last reply Reply Quote 0
                • stephenw10S Offline
                  stephenw10 Netgate Administrator
                  last edited by

                  Hmm, does sysctl debug.ddb.scripting.scripts still show the modified script?

                  R 1 Reply Last reply Reply Quote 0
                  • R Offline
                    rfranzke @stephenw10
                    last edited by rfranzke

                    @stephenw10 I believe so. Here in case I am missing something:

                    [2.8.0-RELEASE][admin@fw2.mdaemon.int]/root: sysctl debug.ddb.scripting.scripts
                    debug.ddb.scripting.scripts: lockinfo=show locks; show alllocks; show lockedvnods
                    pfs=bt ; show registers ; show pcpu ; run lockinfo ; acttrace ; ps ; alltrace
                    kdb.enter.default=bt ; show registers ; dump ; reset
                    kdb.enter.witness=run lockinfo

                    [2.8.0-RELEASE][admin@fw2.mdaemon.int]/root:

                    I forced a manual panic to test it again and it created a dump file as one would expect, so not sure what happened here:

                    [2.8.0-RELEASE][admin@fw2.mdaemon.int]/var/crash: ls
                    bounds info.0 info.last vmcore.0 vmcore.last

                    Let me know if something is missing from above. Thanks for looking.

                    1 Reply Last reply Reply Quote 0
                    • stephenw10S Offline
                      stephenw10 Netgate Administrator
                      last edited by

                      Huh that's odd. Did it actually panic before with the arp crash? Did it reboot automatically afterwards?

                      Might need a new script line there if it's doing some special. In which case we might need wisdom from @kprovost 😉

                      K R 2 Replies Last reply Reply Quote 0
                      • K Offline
                        kprovost @stephenw10
                        last edited by

                        @stephenw10 Off the top of my head I don't know of any reason why a panic wouldn't leave a core dump.

                        What makes you say the box crashed if there's no core dump? Did it reboot? Is there anything in the logs?

                        If it did crash it really should have left a core dump. The only thing that I can think of that might (and I stress might, I've not confirmed this) cause it to not produce a core dump is if the dump area is too small. How much memory does the machine have, and how big is the swap?

                        1 Reply Last reply Reply Quote 0
                        • R Offline
                          rfranzke @stephenw10
                          last edited by rfranzke

                          @stephenw10 I heard the fans spin up like it does and checked if I could access the device on the network. I couldn't so it definitely seemed to crash. Trying to get this to crash on its own now. The box definitely restarted for some reason. I assumed it was due to the same crash that I've been chasing. I suppose its possible that it just restarted for some reason, but I would say fairly unlikely that would be the case. Plenty of swap space available:

                          [2.8.0-RELEASE][admin@fw2.mdaemon.int]/root: swapinfo -h
                          Device Size Used Avail Capacity
                          /dev/gpt/swap1 128G 0B 128G 0%
                          [2.8.0-RELEASE][admin@fw2.mdaemon.int]/root:

                          1 Reply Last reply Reply Quote 0
                          • stephenw10S Offline
                            stephenw10 Netgate Administrator
                            last edited by

                            Hmm. I guess try and trigger it again and see if you can confirm it.

                            N 1 Reply Last reply Reply Quote 0
                            • N Offline
                              netblues @stephenw10
                              last edited by

                              At that point it might prove helpful to configure a serial console output and hook up the serial port to another pc running some terminal software with a big scroll back buffer.
                              It might give more insight than fans spinning at boot.

                              p.s. Strange as it is, I do suspect power issues, its a good way to crash a system without leaving ANY traces apart from the new boot logs.

                              1 Reply Last reply Reply Quote 0
                              • First post
                                Last post
                              Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.