Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Crash report!!!!

    Scheduled Pinned Locked Moved General pfSense Questions
    42 Posts 4 Posters 3.6k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A
      Antibiotic @stephenw10
      last edited by Antibiotic

      @stephenw10 Again crash report, but now after changing settings in limiters!

      textdump.tar (1).0 info (1).0

      Idk, what is going on!

      384003d7-50f9-4092-85c8-7499af25ca88-image.png

      pfSense plus 24.11 on Topton mini PC
      CPU: Intel N100
      NIC: Intel i-226v 4 pcs
      RAM : 16 GB DDR5
      Disk: 128 GB NVMe
      Brgds, Archi

      1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by

        Backtrace:

        db:1:pfs> bt
        Tracing pid 0 tid 100013 td 0xfffff80001974740
        kdb_enter() at kdb_enter+0x33/frame 0xfffffe00c59ed370
        panic() at panic+0x43/frame 0xfffffe00c59ed3d0
        vm_fault() at vm_fault+0x15cc/frame 0xfffffe00c59ed4f0
        vm_fault_trap() at vm_fault_trap+0xb0/frame 0xfffffe00c59ed540
        trap_pfault() at trap_pfault+0x1d6/frame 0xfffffe00c59ed5a0
        calltrap() at calltrap+0x8/frame 0xfffffe00c59ed5a0
        --- trap 0xc, rip = 0xffffffff84451b37, rsp = 0xfffffe00c59ed670, rbp = 0xfffffe00c59ed6f0 ---
        fq_codel_enqueue() at fq_codel_enqueue+0x137/frame 0xfffffe00c59ed6f0
        dummynet_io() at dummynet_io+0x26f/frame 0xfffffe00c59ed740
        pf_dummynet_route() at pf_dummynet_route+0x3b9/frame 0xfffffe00c59ed830
        pf_route() at pf_route+0x28e/frame 0xfffffe00c59ed910
        pf_test() at pf_test+0xe28/frame 0xfffffe00c59edac0
        pf_check_in() at pf_check_in+0x27/frame 0xfffffe00c59edae0
        pfil_mbuf_in() at pfil_mbuf_in+0x38/frame 0xfffffe00c59edb10
        ip_input() at ip_input+0x3aa/frame 0xfffffe00c59edb70
        netisr_dispatch_src() at netisr_dispatch_src+0x22c/frame 0xfffffe00c59edbc0
        ether_demux() at ether_demux+0x149/frame 0xfffffe00c59edbf0
        ether_nh_input() at ether_nh_input+0x36d/frame 0xfffffe00c59edc50
        netisr_dispatch_src() at netisr_dispatch_src+0xaf/frame 0xfffffe00c59edca0
        ether_input() at ether_input+0x69/frame 0xfffffe00c59edd00
        iflib_rxeof() at iflib_rxeof+0xc46/frame 0xfffffe00c59ede00
        _task_fn_rx() at _task_fn_rx+0x72/frame 0xfffffe00c59ede40
        gtaskqueue_run_locked() at gtaskqueue_run_locked+0x14e/frame 0xfffffe00c59edec0
        gtaskqueue_thread_loop() at gtaskqueue_thread_loop+0xc2/frame 0xfffffe00c59edef0
        fork_exit() at fork_exit+0x7f/frame 0xfffffe00c59edf30
        fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00c59edf30
        --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
        
        config_aqm Unable to configure flowset, flowset busy!
        config_aqm Unable to configure flowset, flowset busy!
        config_aqm Unable to configure flowset, flowset busy!
        panic: vm_fault_lookup: fault on nofault entry, addr: 0xfffffe0103234000
        cpuid = 2
        time = 1715514190
        KDB: enter: panic
        

        So that's a completely different panic.
        What exactly did you change?

        A 1 Reply Last reply Reply Quote 0
        • A
          Antibiotic @stephenw10
          last edited by

          @stephenw10 Limiters settings

          b25d497a-457c-4da4-a908-3eb19c9fa2c8-image.png

          pfSense plus 24.11 on Topton mini PC
          CPU: Intel N100
          NIC: Intel i-226v 4 pcs
          RAM : 16 GB DDR5
          Disk: 128 GB NVMe
          Brgds, Archi

          1 Reply Last reply Reply Quote 0
          • stephenw10S
            stephenw10 Netgate Administrator
            last edited by

            Right but what did you actually change? Added the limiter entirely?

            A 1 Reply Last reply Reply Quote 0
            • A
              Antibiotic @stephenw10
              last edited by

              @stephenw10 Changed limits and flows figures

              pfSense plus 24.11 on Topton mini PC
              CPU: Intel N100
              NIC: Intel i-226v 4 pcs
              RAM : 16 GB DDR5
              Disk: 128 GB NVMe
              Brgds, Archi

              1 Reply Last reply Reply Quote 0
              • stephenw10S
                stephenw10 Netgate Administrator
                last edited by

                Hmm, was anything logged? Any alert shown other than the crash?

                Did it crash immediately when you saved those changes?

                A 1 Reply Last reply Reply Quote 0
                • A
                  Antibiotic @stephenw10
                  last edited by

                  @stephenw10 '

                  1. None
                  2. None
                  3. Yes

                  pfSense plus 24.11 on Topton mini PC
                  CPU: Intel N100
                  NIC: Intel i-226v 4 pcs
                  RAM : 16 GB DDR5
                  Disk: 128 GB NVMe
                  Brgds, Archi

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by

                    Hmm, it is repeatable?

                    We'd need to replicate it locally to dig further so steps to reproduce would help a lot.

                    A 2 Replies Last reply Reply Quote 0
                    • A
                      Antibiotic @stephenw10
                      last edited by Antibiotic

                      @stephenw10 Idk, will it help, that what I found i sys.log. That was on next reboot after crash

                      2024-05-12 19:25:14.079708+03:00 check_reload_status 643 Reloading filter
                      2024-05-12 19:25:15.115594+03:00 php-fpm 592 /rc.newwanip: rc.newwanip: Info: starting on igc1.
                      2024-05-12 19:25:15.115937+03:00 php-fpm 592 /rc.newwanip: rc.newwanip: on (IP address: 192.168.10.1) (interface: LAN[lan]) (real interface: igc1).
                      2024-05-12 19:25:15.401258+03:00 kernel - config_aqm Unable to configure flowset, flowset busy!
                      2024-05-12 19:25:15.602269+03:00 kernel - config_aqm Unable to configure flowset, flowset busy!
                      2024-05-12 19:25:15.602325+03:00 kernel - config_aqm Unable to configure flowset, flowset busy!
                      2024-05-12 19:25:16.004364+03:00 kernel - config_aqm Unable to configure flowset, flowset busy!
                      2024-05-12 19:25:16.004547+03:00 kernel - config_aqm Unable to configure flowset, flowset busy!
                      2024-05-12 19:25:16.216719+03:00 php-fpm 592 /rc.newwanip: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.PROTO.>'' returned exit code '1', the output was ''
                      2024-05-12 19:25:17.425245+03:00 php-fpm 592 /rc.newwanip: Resyncing OpenVPN instances for interface LAN.
                      2024-05-12 19:25:17.471028+03:00 php-fpm 592 /rc.newwanip: Creating rrd update script
                      2024-05-12 19:25:19.496269+03:00 php-fpm 592 /rc.newwanip: Netgate pfSense Plus package system has detected an IP change or dynamic WAN reconnection - 192.168.10.1 -> 192.168.10.1 - Restarting packages.
                      2024-05-12 19:25:19.496416+03:00 check_reload_status 643 Starting packages
                      2024-05-12 19:25:19.500231+03:00 check_reload_status 643 Reloading filter
                      2024-05-12 19:25:19.500275+03:00 check_reload_status 643 Reloading filter
                      2024-05-12 19:25:20.534000+03:00 php-fpm 87873 /rc.start_packages: Restarting/Starting all packages.
                      2024-05-12 19:25:20.829266+03:00 kernel - config_aqm Unable to configure flowset, flowset busy!

                      I think will remove Limiters as full of bugs

                      pfSense plus 24.11 on Topton mini PC
                      CPU: Intel N100
                      NIC: Intel i-226v 4 pcs
                      RAM : 16 GB DDR5
                      Disk: 128 GB NVMe
                      Brgds, Archi

                      1 Reply Last reply Reply Quote 0
                      • A
                        Antibiotic @stephenw10
                        last edited by

                        @stephenw10 said in Crash report!!!!:

                        further so steps to reproduce would help a lot

                        Steps , what I did
                        Untick all offloads (Limiters were already) , than switch off back all offloads get first crash. After reboot changed some settings in Limiters as above in my post, get second crash. Next step remove Limiters and waiting now third crash.

                        pfSense plus 24.11 on Topton mini PC
                        CPU: Intel N100
                        NIC: Intel i-226v 4 pcs
                        RAM : 16 GB DDR5
                        Disk: 128 GB NVMe
                        Brgds, Archi

                        1 Reply Last reply Reply Quote 0
                        • stephenw10S
                          stephenw10 Netgate Administrator
                          last edited by

                          And just resaving the Limiters again did not reproduce the crash?

                          A 1 Reply Last reply Reply Quote 0
                          • A
                            Antibiotic @stephenw10
                            last edited by

                            @stephenw10 Sorry, already removed Limiters and making state resetting's

                            pfSense plus 24.11 on Topton mini PC
                            CPU: Intel N100
                            NIC: Intel i-226v 4 pcs
                            RAM : 16 GB DDR5
                            Disk: 128 GB NVMe
                            Brgds, Archi

                            1 Reply Last reply Reply Quote 0
                            • K
                              kprovost @stephenw10
                              last edited by

                              @stephenw10 said in Crash report!!!!:

                              Hmm, backtrace is unusually long

                              It's a routing loop. The OpenVPN traffic is being routing into the OpenVPN tunnel. Sooner or later we run out of stack and crash.

                              It's a bug in if_ovpn that it doesn't discard this traffic, but it's also a configuration error. Once the bug is fixed the tunnel still won't work.

                              A 1 Reply Last reply Reply Quote 0
                              • A
                                Antibiotic @kprovost
                                last edited by Antibiotic

                                @kprovost said in Crash report!!!!:

                                but it's also a configuration error.

                                What do you mean, in my configuration error? Or OpenVPN have error itself?

                                pfSense plus 24.11 on Topton mini PC
                                CPU: Intel N100
                                NIC: Intel i-226v 4 pcs
                                RAM : 16 GB DDR5
                                Disk: 128 GB NVMe
                                Brgds, Archi

                                K 1 Reply Last reply Reply Quote 0
                                • K
                                  kprovost @Antibiotic
                                  last edited by

                                  @Antibiotic You've somehow configured your system so that the OpenVPN tunnelled traffic goes down the OpenVPN tunnel.

                                  1 Reply Last reply Reply Quote 0
                                  • stephenw10S
                                    stephenw10 Netgate Administrator
                                    last edited by

                                    Check the routing table in Diag > Routes. Is your default route getting set to the OpenVPN gateway?

                                    Make sure the system default gateway is set to the WAN gateway in Sys > Routing > Gateways.

                                    A 1 Reply Last reply Reply Quote 0
                                    • A
                                      Antibiotic @stephenw10
                                      last edited by

                                      @stephenw10 That , what I have . Is this incorrect?

                                      8ef4067b-053e-44ef-b580-bccd4598849c-image.png

                                      a02a9c8a-ca50-40a9-a2e9-348811bfea31-image.png

                                      55fb3efa-2f35-472d-96dd-a248d7dc5141-image.png

                                      830e651d-fe32-4308-ba0d-7784b0e403d8-image.png

                                      2712e506-2691-481b-aa75-757b753fe8e4-image.png

                                      pfSense plus 24.11 on Topton mini PC
                                      CPU: Intel N100
                                      NIC: Intel i-226v 4 pcs
                                      RAM : 16 GB DDR5
                                      Disk: 128 GB NVMe
                                      Brgds, Archi

                                      1 Reply Last reply Reply Quote 0
                                      • stephenw10S
                                        stephenw10 Netgate Administrator
                                        last edited by

                                        Hmm, seems like you don't have a default route there for some reason. Try resaving the system default gateway and then recheck the routing table.

                                        A 1 Reply Last reply Reply Quote 0
                                        • A
                                          Antibiotic @stephenw10
                                          last edited by

                                          @stephenw10 said in Crash report!!!!:

                                          Try resaving the system default gateway

                                          Did now!

                                          Screenshot_13-5-2024_153020_192.168.10.1.jpeg

                                          pfSense plus 24.11 on Topton mini PC
                                          CPU: Intel N100
                                          NIC: Intel i-226v 4 pcs
                                          RAM : 16 GB DDR5
                                          Disk: 128 GB NVMe
                                          Brgds, Archi

                                          1 Reply Last reply Reply Quote 0
                                          • stephenw10S
                                            stephenw10 Netgate Administrator
                                            last edited by

                                            Ok you should be good.

                                            Do you see the OpenVPN server IPs there as static routes?

                                            Those /1 routes you see there are being pushed to you by the VPN providers. I would disable those clients from accepting routes like that. You don't need that if you;re policy routing traffic across them.

                                            A 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.