• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

Crash Report after runing traceroute

Scheduled Pinned Locked Moved General pfSense Questions
22 Posts 4 Posters 702 Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • S
    studeoQ
    last edited by Dec 30, 2024, 7:59 PM

    I tried running traceroute from the pf console on a fresh install - not in production - just testing - and each of the 3 times the system hangs on the first hop - then goes into panic, crashes and reboots. I repeated to process and the results were the same.
    The machine is an older Dell - i5 with 8G ram and an SSD - running the ill-fated Realtek NICS. Seems to do everything else fairly well. But this was a bit of a surprise. Even in production there will be less than 15 nodes on the entire LAN.

    See attached.
    M
    Crash_Report_12_30_24.txt

    S 1 Reply Last reply Dec 31, 2024, 12:12 AM Reply Quote 0
    • S
      studeoQ @studeoQ
      last edited by Dec 31, 2024, 12:12 AM

      I guess I should add that this is the 2.7.2-RELEASE - in mostly the default config with only the WAN/LAN ports configured. Nothing else has been added.

      1 Reply Last reply Reply Quote 0
      • S
        stephenw10 Netgate Administrator
        last edited by Dec 31, 2024, 3:54 AM

        Backtrace:

        db:0:kdb.enter.default>  bt
        Tracing pid 0 tid 100082 td 0xfffffe008c6da1e0
        kdb_enter() at kdb_enter+0x32/frame 0xfffffe0082c90a10
        vpanic() at vpanic+0x163/frame 0xfffffe0082c90b40
        panic() at panic+0x43/frame 0xfffffe0082c90ba0
        trap_fatal() at trap_fatal+0x40c/frame 0xfffffe0082c90c00
        trap_pfault() at trap_pfault+0x4f/frame 0xfffffe0082c90c60
        calltrap() at calltrap+0x8/frame 0xfffffe0082c90c60
        --- trap 0xc, rip = 0xffffffff80e24c30, rsp = 0xfffffe0082c90d30, rbp = 0xfffffe0082c90d80 ---
        ether_input() at ether_input+0x50/frame 0xfffffe0082c90d80
        re_rxeof() at re_rxeof+0x2c0/frame 0xfffffe0082c90e00
        re_int_task_8125() at re_int_task_8125+0xba/frame 0xfffffe0082c90e40
        taskqueue_run_locked() at taskqueue_run_locked+0x182/frame 0xfffffe0082c90ec0
        taskqueue_thread_loop() at taskqueue_thread_loop+0xc2/frame 0xfffffe0082c90ef0
        fork_exit() at fork_exit+0x7f/frame 0xfffffe0082c90f30
        fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0082c90f30
        --- trap 0, rip = 0, rsp = 0, rbp = 0xfffffe0082a901e8 ---
        ??() at 0/frame 0xfffffe0082a901e8
        ??() at 0xfffffe0082a90258/frame 0xfffffe0082a90f78
        ??() at 0xfffffe0082a901e8/frame 0xfffffe0082a90f08
        ??() at 0xfffffe0082a90f78
        

        Panic:

        Fatal trap 12: page fault while in kernel mode
        cpuid = 1; apic id = 02
        fault virtual address	= 0x10007
        fault code		= supervisor read data, page not present
        instruction pointer	= 0x20:0xffffffff80e24c30
        stack pointer	        = 0x28:0xfffffe0082cb8d30
        frame pointer	        = 0x28:0xfffffe0082cb8d80
        code segment		= base 0x0, limit 0xfffff, type 0x1b
        			= DPL 0, pres 1, long 1, def32 0, gran 1
        processor eflags	= interrupt enabled, resume, IOPL = 0
        current process		= 0 (re1 taskq)
        rdi: 0000000000000000 rsi: fffffe008c6daac0 rdx: 0000000000000001
        rcx: 0000000000000001  r8: 00000000ffffff31  r9: 0000000000000080
        rax: 0000000000000000 rbx: 000000000000ffff rbp: fffffe0082cb8d80
        r10: 0000000000000001 r11: fffff80001622000 r12: 0000000000008803
        r13: 000000000000ffff r14: fffffe008c6daac0 r15: 0000000000000000
        trap number		= 12
        panic: page fault
        cpuid = 1
        time = 1735585591
        

        What NIC exactly is that?

        S 2 Replies Last reply Dec 31, 2024, 4:23 AM Reply Quote 0
        • S
          studeoQ @stephenw10
          last edited by Dec 31, 2024, 4:23 AM

          @stephenw10
          From dmesg.boot below.

          The 2.5GbE is a Trendnet TEG-25GECTX - the other is the on-board.

          pci1: <ACPI PCI bus> on pcib2
          re0: <Realtek PCIe 2.5GbE Family Controller> port 0xe000-0xe0ff mem 0xf7d00000-0xf7d0ffff,0xf7d10000-0xf7d13fff at device 0.0 on pci1
          re0: Using Memory Mapping!
          re0: Using 1 MSI-X message
          re0: ASPM disabled
          re0: version:1.98.00
          re0: Ethernet address: 78:2d:7e:1e:a3:26
          
          This product is covered by one or more of the following patents:
          US6,570,884, US6,115,776, and US6,327,625.
          re0: Ethernet address: 78:2d:7e:1e:a3:26
          pcib3: <ACPI PCI-PCI bridge> at device 28.3 on pci0
          pci2: <ACPI PCI bus> on pcib3
          re1: <Realtek PCIe GbE Family Controller> port 0xd000-0xd0ff mem 0xf7c00000-0xf7c00fff,0xf0000000-0xf0003fff at device 0.0 on pci2
          re1: Using Memory Mapping!
          re1: Using 1 MSI-X message
          re1: ASPM disabled
          re1: version:1.98.00
          re1: Ethernet address: b0:83:fe:ab:1a:90
          

          pciconf -lv below:

          re0@pci0:2:0:0: class=0x020000 rev=0x00 hdr=0x00 vendor=0x10ec device=0x8125 subvendor=0x10ec subdevice=0x0123
              vendor     = 'Realtek Semiconductor Co., Ltd.'
              device     = 'RTL8125 2.5GbE Controller'
              class      = network
              subclass   = ethernet
          re1@pci0:3:0:0: class=0x020000 rev=0x0c hdr=0x00 vendor=0x10ec device=0x8168 subvendor=0x1028 subdevice=0x0612
              vendor     = 'Realtek Semiconductor Co., Ltd.'
              device     = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
              class      = network
              subclass   = ethernet
          
          1 Reply Last reply Reply Quote 1
          • S
            studeoQ @stephenw10
            last edited by Dec 31, 2024, 4:28 AM

            @stephenw10
            I forgot to mention that the behavior was the same from the GUI. Clicked on traceroute - same thing happened. Tried it from the console - identical behavior.
            M

            1 Reply Last reply Reply Quote 0
            • S
              stephenw10 Netgate Administrator
              last edited by Dec 31, 2024, 1:19 PM

              Looks like it's re1 giving the issue. Is that assigned as WAN?

              If you traceroute to something on the LAN using the other NIC does it fail?

              If you trceroute using ICMP instead of UDP does it still panic?

              I would normally suggest trying the alternative Realtek driver but it looks like you're already doing that to get the 8125 support I assume.

              S 3 Replies Last reply Dec 31, 2024, 1:24 PM Reply Quote 0
              • S
                studeoQ @stephenw10
                last edited by Dec 31, 2024, 1:24 PM

                @stephenw10
                Yes - I updated the driver before any of this happened.
                I'll have to test this tomorrow or on the weekend while everything is closed. The system is now in production.
                I will report back - as this is a bit odd. We suspected that it might be a loop on the LAN, given that it happened while in a test environment while connected to a couple of switches instead of directly to the WAN link - but that is just conjecture at this point.
                I'll definitely post here once I can test.

                1 Reply Last reply Reply Quote 0
                • S
                  studeoQ @stephenw10
                  last edited by Dec 31, 2024, 1:28 PM

                  @stephenw10 said in Crash Report after runing traceroute:

                  Looks like it's re1 giving the issue. Is that assigned as WAN?

                  re1 is currently the LAN link - but it might have been WAN when this happened.
                  I'll test and report back.

                  1 Reply Last reply Reply Quote 1
                  • S
                    studeoQ @stephenw10
                    last edited by studeoQ Jan 1, 2025, 6:13 PM Jan 1, 2025, 5:22 PM

                    @stephenw10
                    WAN - traceroute -i re0 8.8.8.8 - crash on the first hop
                    LAN - traceroute -i re1 8.8.8.8 - crash on the first hop

                    traceroute -1 re0 192.168.1.11 - crash on the first hop
                    traceroute -1 re0 192.168.1.11 - went 6 hops without crashing - but discovered zero routes to a machine sitting on the LAN.

                    traceroute to 192.168.1.11 (192.168.1.11), 64 hops max, 40 byte packets
                     1  * * *
                     2  * * *
                     3  * * *
                     4  * * *
                     5  * * *
                     6  * *^C
                    

                    traceroute from GUI - crash - presumably on the first hop

                    The system is now inline and in production with basic WAN/LAN ip assignments.
                    M

                    K 1 Reply Last reply Jan 1, 2025, 6:36 PM Reply Quote 0
                    • K
                      Konstanti @studeoQ
                      last edited by Konstanti Jan 1, 2025, 6:38 PM Jan 1, 2025, 6:36 PM

                      @studeoQ

                      Hi
                      what kind of driver are you using?
                      The site says only about support for Windows

                      https://www.trendnet.com/support/TEG-25GECTX

                      c530a20f-07b2-4ac1-9226-3ce2fb625a7b-image.png

                      bbca8201-cf24-451f-b6d0-5c9a007b922b-image.png

                      S 1 Reply Last reply Jan 1, 2025, 6:42 PM Reply Quote 0
                      • S
                        studeoQ @Konstanti
                        last edited by Jan 1, 2025, 6:42 PM

                        @Konstanti
                        I'm using the updated drivers described here.
                        The rest of the system seems stable - VLANS working, etc...

                        K 1 Reply Last reply Jan 1, 2025, 6:45 PM Reply Quote 0
                        • K
                          Konstanti @studeoQ
                          last edited by Jan 1, 2025, 6:45 PM

                          @studeoQ

                          The re driver supports RealTek RTL8139C+, RTL8169, RTL816xS, RTL811xS, RTL8168,RTL810xE and RTL8111 based Fast Ethernet and Gigabit Ethernet adapters

                          Your adapter is not on this list (I suspect the problem is with the device driver)

                          device = 'RTL8125 2.5GbE Controller'

                          S 1 Reply Last reply Jan 1, 2025, 6:51 PM Reply Quote 0
                          • S
                            studeoQ @Konstanti
                            last edited by Jan 1, 2025, 6:51 PM

                            @Konstanti
                            The behavior is the same with both NICS - which of course happen to both be Realtek.
                            Live and learn I guess.
                            If I end up building another box - I'll keep an eye out for it in the future.
                            M

                            1 Reply Last reply Reply Quote 0
                            • S
                              stephenw10 Netgate Administrator
                              last edited by Jan 1, 2025, 9:57 PM

                              The in kernel re driver doesn't support rtl8125 but the alternative kmod driver does.

                              @studeoQ said in Crash Report after runing traceroute:

                              traceroute -1 re0 192.168.1.11 - crash on the first hop
                              traceroute -1 re0 192.168.1.11 - went 6 hops without crashing - but discovered zero routes to a machine sitting on the LAN.

                              I assume one of those is a typo? And I expect the second test there should show re1 because testing to the LAN from the LAN IP removes re0 from the route. That would confirm it's actually the re0 driver causing a problem. In which case you could try going back to the in-kernel driver but that will stop the 2.5G NIC working.

                              S 1 Reply Last reply Jan 1, 2025, 11:15 PM Reply Quote 0
                              • S
                                studeoQ @stephenw10
                                last edited by Jan 1, 2025, 11:15 PM

                                @stephenw10
                                Yes - a typo - and I couldn't edit - too much time had passed.
                                Right now - the "fix" is - don't run traceroute...
                                Everything else is working as expected - so tempted to leave it alone for now.

                                G 1 Reply Last reply Jan 2, 2025, 7:51 AM Reply Quote 0
                                • G
                                  Gertjan @studeoQ
                                  last edited by Jan 2, 2025, 7:51 AM

                                  @studeoQ said in Crash Report after runing traceroute:

                                  Everything else is working as expected - so tempted to leave it alone for now.

                                  Your initial plan was the best, as 're' will come back to bite you :

                                  @studeoQ said in Crash Report after runing traceroute:

                                  If I end up building another box

                                  No need to rebuild, go visit the BIOS and do what all 're' merits : disable them.
                                  Then slide in a dual (quad ?) intel NIC, and you'll be good.

                                  No "help me" PM's please. Use the forum, the community will thank you.
                                  Edit : and where are the logs ??

                                  K 1 Reply Last reply Jan 2, 2025, 8:23 AM Reply Quote 0
                                  • K
                                    Konstanti @Gertjan
                                    last edited by Jan 2, 2025, 8:23 AM

                                    @Gertjan

                                    Here, perhaps, I agree.
                                    It is unknown from which source code this driver is built and for which version of Freebsd. If there are already problems that lead to a system failure , then it is better to abandon this idea. With Intel network cards, everything has been functioning for years without problems

                                    S 1 Reply Last reply Jan 2, 2025, 12:25 PM Reply Quote 0
                                    • S
                                      studeoQ @Konstanti
                                      last edited by Jan 2, 2025, 12:25 PM

                                      @Konstanti
                                      It's a limitation of the box I built - SFF PC with no real ability to add very many options (single PCIe 1X slot) so - finding the right NIC might be the next and only option. If not - then a new box will be in order. I was pretty impatient when it came to building the first one. I'll be more careful in the future.

                                      1 Reply Last reply Reply Quote 0
                                      • S
                                        stephenw10 Netgate Administrator
                                        last edited by Jan 2, 2025, 12:46 PM

                                        Do you see any issues if you run a traceroute from a client behind pfSense through it?

                                        S 2 Replies Last reply Jan 2, 2025, 1:03 PM Reply Quote 0
                                        • S
                                          studeoQ @stephenw10
                                          last edited by Jan 2, 2025, 1:03 PM

                                          @stephenw10
                                          Other than being a little slower than i think is normal - no - there aren't any obvious issues.
                                          It can be run from nodes residing behind it.
                                          I seriously doubt I would be running it on the device for any reason other than testing.
                                          I just thought a system-wide failure was a little odd.

                                          If anyone knows of a PCIe X1 dual NIC that might fit the bill - I'm all ears.
                                          Speed tests are normal and I'm getting the expected symmetrical speeds - so it can't be "that broken".
                                          But - I'm not a big fan of loose ends and known broken stuff just lingering around - so - I'd feel better knowing it wasn't an issue.

                                          1 Reply Last reply Reply Quote 0
                                          20 out of 22
                                          • First post
                                            20/22
                                            Last post
                                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
                                            This community forum collects and processes your personal information.
                                            consent.not_received