Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Netgate 6100 Crash On Interface Change - Not Resolved (IPv6 + PPPoE)

    Scheduled Pinned Locked Moved Official Netgate® Hardware
    42 Posts 3 Posters 4.6k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • stephenw10S
      stephenw10 Netgate Administrator
      last edited by

      @robbiett said in Netgate 6100 Crash On Interface Change:

      db:1:pfs> bt
      [snip]

      The backtrace is usually the most telling part. It appears to have been snipped, do you have it?

      RobbieTTR 1 Reply Last reply Reply Quote 0
      • RobbieTTR
        RobbieTT @stephenw10
        last edited by

        @stephenw10
        I have it all, I just ran into a forum size limit - crude cuts were made.

        db:1:pfs> bt
        Tracing pid 4585 tid 100445 td 0xfffffe00cd4ba1e0
        kdb_enter() at kdb_enter+0x32/frame 0xfffffe00cd68c790
        vpanic() at vpanic+0x182/frame 0xfffffe00cd68c7e0
        panic() at panic+0x43/frame 0xfffffe00cd68c840
        trap_fatal() at trap_fatal+0x409/frame 0xfffffe00cd68c8a0
        trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00cd68c900
        calltrap() at calltrap+0x8/frame 0xfffffe00cd68c900
        --- trap 0xc, rip = 0xffffffff80fd9293, rsp = 0xfffffe00cd68c9d0, rbp = 0xfffffe00cd68ca20 ---
        in6_unlink_ifa() at in6_unlink_ifa+0x63/frame 0xfffffe00cd68ca20
        in6_purgeaddr() at in6_purgeaddr+0x367/frame 0xfffffe00cd68cb40
        in6_purgeifaddr() at in6_purgeifaddr+0x13/frame 0xfffffe00cd68cb60
        in6_control() at in6_control+0x532/frame 0xfffffe00cd68cbc0
        ifioctl() at ifioctl+0x7bc/frame 0xfffffe00cd68ccc0
        kern_ioctl() at kern_ioctl+0x26d/frame 0xfffffe00cd68cd30
        sys_ioctl() at sys_ioctl+0x101/frame 0xfffffe00cd68ce00
        amd64_syscall() at amd64_syscall+0x10c/frame 0xfffffe00cd68cf30
        fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00cd68cf30
        --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x18f47cd96e4a, rsp = 0x18f478021f28, rbp = 0x18f478021f70 ---
        

        Is the above what you are looking for?

        ☕️

        stephenw10S 1 Reply Last reply Reply Quote 0
        • stephenw10S
          stephenw10 Netgate Administrator @RobbieTT
          last edited by

          Yes, checking....

          1 Reply Last reply Reply Quote 0
          • stephenw10S
            stephenw10 Netgate Administrator
            last edited by

            Hmm, so you're losing link on ix1 there when the switch rebooted?

            And you have IPv6 addresses on ix1 dircetly or just on VLAN 1003 on that?

            RobbieTTR 1 Reply Last reply Reply Quote 0
            • stephenw10S
              stephenw10 Netgate Administrator
              last edited by

              Are you able to replicate that manually?

              That is a bug, it should not happen. We probably need to replicate it locally to solve though.

              RobbieTTR 1 Reply Last reply Reply Quote 0
              • RobbieTTR
                RobbieTT @stephenw10
                last edited by

                @stephenw10
                Yes, lost link on firmware update (thanks UniFi) on this occasion but it can happen when changing interface settings on pfSense.

                On ix1 I have 1 x LAN and 1 x VLAN; these are my main networks. They both have IPv4 & IPv6.

                I have one other LAN interface on igc0 that acts as a management interface but it is not normally connected.

                 2023-03-23 at 15.17.01.png

                ☕️

                1 Reply Last reply Reply Quote 0
                • RobbieTTR
                  RobbieTT @stephenw10
                  last edited by

                  @stephenw10 said in Netgate 6100 Crash On Interface Change:

                  Are you able to replicate that manually?

                  That is a bug, it should not happen. We probably need to replicate it locally to solve though.

                  It is repeatable and happens roughly 60% of the time if settings are adjusted when the interface is enabled. If the interface is disabled first, then settings changed, then re-enabled it is much less prevalent, under 20% of the time at a guess.

                  The example given for this thread is clearly just 1 in 1 event, albeit with the same undesirable result.

                  ☕️

                  1 Reply Last reply Reply Quote 0
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by

                    Ok, we replicated it here. Digging now....

                    RobbieTTR 1 Reply Last reply Reply Quote 2
                    • RobbieTTR
                      RobbieTT @stephenw10
                      last edited by

                      @stephenw10 said in Netgate 6100 Crash On Interface Change:

                      Ok, we replicated it here. Digging now....

                      Fine work, fine work indeed. 👍

                      1 Reply Last reply Reply Quote 0
                      • stephenw10S
                        stephenw10 Netgate Administrator
                        last edited by

                        A fix for this has now gone in upstream: https://redmine.pfsense.org/issues/14164

                        That's not something that can be patched at run time though.

                        Steve

                        RobbieTTR 1 Reply Last reply Reply Quote 1
                        • RobbieTTR
                          RobbieTT @stephenw10
                          last edited by

                          @stephenw10 - Thanks Steve, from your comment I guess this will percolate down for a version update window at an unknown date?

                          ☕️

                          1 Reply Last reply Reply Quote 0
                          • stephenw10S
                            stephenw10 Netgate Administrator
                            last edited by

                            Yes it should be in the next version, 23.05.

                            In fact it's in our repo now. It should be available for testing in todays snapshots:
                            https://github.com/pfsense/FreeBSD-src/commit/f5a365e51feea75d1e5ebc86c53808d8cae7b6d7

                            Steve

                            RobbieTTR 2 Replies Last reply Reply Quote 0
                            • RobbieTTR
                              RobbieTT @stephenw10
                              last edited by

                              @stephenw10
                              Thanks again and I am slightly embarrassed to find a bug so soon into my Netgate journey. I'll keep my head down for a bit!

                              ☕️

                              1 Reply Last reply Reply Quote 1
                              • stephenw10S
                                stephenw10 Netgate Administrator
                                last edited by

                                Don't be. If everyone reported bugs as soon as they found them with the details you did there would be far fewer to find!

                                Steve

                                A 1 Reply Last reply Reply Quote 0
                                • A
                                  azdeltawye @stephenw10
                                  last edited by

                                  @stephenw10

                                  Would this bug affect a 4100 also?

                                  It appears that I stumbled across this issue the other day when I was messing with traffic shaping on an interface that had ipv6 enabled. The moment I clicked on 'save', the blue LED on the 4100 stopped blinking, UI became unresponsive and all clients lost internet access.
                                  I had to cycle power on the 4100 to restore normal operation.

                                  RobbieTTR 1 Reply Last reply Reply Quote 0
                                  • RobbieTTR
                                    RobbieTT @azdeltawye
                                    last edited by

                                    @azdeltawye
                                    Sounds very similar, to say the least. Hopefully the fix will solve all.

                                    ☕️

                                    1 Reply Last reply Reply Quote 0
                                    • stephenw10S
                                      stephenw10 Netgate Administrator
                                      last edited by

                                      Yeah, in fact I don't think it's system specific, or even NIC specific. But even if it is the 4100 is similar enough I'd expect it's possible to hit it there too.

                                      Steve

                                      1 Reply Last reply Reply Quote 0
                                      • RobbieTTR
                                        RobbieTT @stephenw10
                                        last edited by RobbieTT

                                        @stephenw10 said in Netgate 6100 Crash On Interface Change:

                                        Yes it should be in the next version, 23.05.

                                        In fact it's in our repo now. It should be available for testing in todays snapshots:
                                        https://github.com/pfsense/FreeBSD-src/commit/f5a365e51feea75d1e5ebc86c53808d8cae7b6d7

                                        Steve

                                        Hi Steve,
                                        Unfortunately the issue persists in 23.05. A change in interface state can still trigger a crash.

                                        I ran 7 simple and repeatable tests today - bringing the WAN interface down and back up again (this is enough to trigger the fault) via the disconnect button on Status/Interfaces.

                                        7 tests - 4 failures with hard crashes, 3 did not trigger a crash.

                                        All 4 failures produced a full crash report, info dump and textdump.tar. All available on request.

                                        The back-trace for the 4 crashes are as follows.

                                        First:

                                        Filename: /var/crash/info.0
                                        Dump header from device: /dev/nvd0p3
                                          Architecture: amd64
                                          Architecture Version: 4
                                          Dump Length: 231424
                                          Blocksize: 512
                                          Compression: none
                                          Dumptime: 2023-05-28 14:48:50 +0100
                                          Hostname: Router-8.*******.me
                                          Magic: FreeBSD Text Dump
                                          Version String: FreeBSD 14.0-CURRENT #1 plus-RELENG_23_05-n256102-7cd3d043045: Mon May 22 15:33:52 UTC 2023
                                            root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05-main/obj/amd64/LkEyii3W/var/j
                                          Panic String: page fault
                                          Dump Parity: 4132315394
                                          Bounds: 0
                                          Dump Status: good
                                        
                                        Filename: /var/crash/textdump.tar.0
                                        ddb.txt���������������������������������������������������������������������������������������������0600����0�������0�������610534������14434655702�  7122� �����������������������������������������������������������������������������������������������������ustar���root����������������������������wheel������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������db:0:kdb.enter.default>  run pfs
                                        db:1:pfs> bt
                                        Tracing pid 93402 tid 103857 td 0xfffffe00cf7cac80
                                        kdb_enter() at kdb_enter+0x32/frame 0xfffffe00cf8a0800
                                        vpanic() at vpanic+0x183/frame 0xfffffe00cf8a0850
                                        panic() at panic+0x43/frame 0xfffffe00cf8a08b0
                                        trap_fatal() at trap_fatal+0x409/frame 0xfffffe00cf8a0910
                                        trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00cf8a0970
                                        calltrap() at calltrap+0x8/frame 0xfffffe00cf8a0970
                                        --- trap 0xc, rip = 0xffffffff80f5a036, rsp = 0xfffffe00cf8a0a40, rbp = 0xfffffe00cf8a0a70 ---
                                        in6_selecthlim() at in6_selecthlim+0x96/frame 0xfffffe00cf8a0a70
                                        tcp_default_output() at tcp_default_output+0x1ded/frame 0xfffffe00cf8a0c60
                                        tcp_output() at tcp_output+0x14/frame 0xfffffe00cf8a0c80
                                        tcp6_usr_connect() at tcp6_usr_connect+0x2f4/frame 0xfffffe00cf8a0d10
                                        soconnectat() at soconnectat+0x9e/frame 0xfffffe00cf8a0d60
                                        kern_connectat() at kern_connectat+0xc9/frame 0xfffffe00cf8a0dc0
                                        sys_connect() at sys_connect+0x75/frame 0xfffffe00cf8a0e00
                                        amd64_syscall() at amd64_syscall+0x109/frame 0xfffffe00cf8a0f30
                                        fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00cf8a0f30
                                        --- syscall (98, FreeBSD ELF64, connect), rip = 0x800fddc8a, rsp = 0x7fffdf5f8c98, rbp = 0x7fffdf5f8cd0 ---
                                        db:1:pfs> 
                                        

                                        Second:

                                        Filename: /var/crash/info.0
                                        Dump header from device: /dev/nvd0p3
                                          Architecture: amd64
                                          Architecture Version: 4
                                          Dump Length: 226304
                                          Blocksize: 512
                                          Compression: none
                                          Dumptime: 2023-05-28 14:51:49 +0100
                                          Hostname: Router-8.*******.me
                                          Magic: FreeBSD Text Dump
                                          Version String: FreeBSD 14.0-CURRENT #1 plus-RELENG_23_05-n256102-7cd3d043045: Mon May 22 15:33:52 UTC 2023
                                            root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05-main/obj/amd64/LkEyii3W/var/j
                                          Panic String: page fault
                                          Dump Parity: 1095311618
                                          Bounds: 0
                                          Dump Status: good
                                        
                                        Filename: /var/crash/textdump.tar.0
                                        ddb.txt���������������������������������������������������������������������������������������������0600����0�������0�������577521������14434656165�  7136� �����������������������������������������������������������������������������������������������������ustar���root����������������������������wheel������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������db:0:kdb.enter.default>  run pfs
                                        db:1:pfs> bt
                                        Tracing pid 68614 tid 100330 td 0xfffffe00cf325720
                                        kdb_enter() at kdb_enter+0x32/frame 0xfffffe00c7d955f0
                                        vpanic() at vpanic+0x183/frame 0xfffffe00c7d95640
                                        panic() at panic+0x43/frame 0xfffffe00c7d956a0
                                        trap_fatal() at trap_fatal+0x409/frame 0xfffffe00c7d95700
                                        trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00c7d95760
                                        calltrap() at calltrap+0x8/frame 0xfffffe00c7d95760
                                        --- trap 0xc, rip = 0xffffffff80f63aa4, rsp = 0xfffffe00c7d95830, rbp = 0xfffffe00c7d95a50 ---
                                        ip6_output() at ip6_output+0xb74/frame 0xfffffe00c7d95a50
                                        udp6_send() at udp6_send+0x78e/frame 0xfffffe00c7d95c10
                                        sosend_dgram() at sosend_dgram+0x357/frame 0xfffffe00c7d95c70
                                        sousrsend() at sousrsend+0x5f/frame 0xfffffe00c7d95cd0
                                        kern_sendit() at kern_sendit+0x132/frame 0xfffffe00c7d95d60
                                        sendit() at sendit+0xb7/frame 0xfffffe00c7d95db0
                                        sys_sendto() at sys_sendto+0x4d/frame 0xfffffe00c7d95e00
                                        amd64_syscall() at amd64_syscall+0x109/frame 0xfffffe00c7d95f30
                                        fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00c7d95f30
                                        --- syscall (133, FreeBSD ELF64, sendto), rip = 0x823f95f2a, rsp = 0x8202cea88, rbp = 0x8202cead0 ---
                                        db:1:pfs>
                                        

                                        Third:

                                        Crash report details:
                                        
                                        No PHP errors found.
                                        
                                        Filename: /var/crash/info.0
                                        Dump header from device: /dev/nvd0p3
                                          Architecture: amd64
                                          Architecture Version: 4
                                          Dump Length: 229888
                                          Blocksize: 512
                                          Compression: none
                                          Dumptime: 2023-05-28 15:11:48 +0100
                                          Hostname: Router-8.*******.me
                                          Magic: FreeBSD Text Dump
                                          Version String: FreeBSD 14.0-CURRENT #1 plus-RELENG_23_05-n256102-7cd3d043045: Mon May 22 15:33:52 UTC 2023
                                            root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05-main/obj/amd64/LkEyii3W/var/j
                                          Panic String: page fault
                                          Dump Parity: 276046082
                                          Bounds: 0
                                          Dump Status: good
                                        
                                        Filename: /var/crash/textdump.tar.0
                                        ddb.txt���������������������������������������������������������������������������������������������0600����0�������0�������605706������14434660444�  7126� �����������������������������������������������������������������������������������������������������ustar���root����������������������������wheel������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������db:0:kdb.enter.default>  run pfs
                                        db:1:pfs> bt
                                        Tracing pid 3281 tid 100913 td 0xfffffe00cfe3e3a0
                                        kdb_enter() at kdb_enter+0x32/frame 0xfffffe00cfdc4800
                                        vpanic() at vpanic+0x183/frame 0xfffffe00cfdc4850
                                        panic() at panic+0x43/frame 0xfffffe00cfdc48b0
                                        trap_fatal() at trap_fatal+0x409/frame 0xfffffe00cfdc4910
                                        trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00cfdc4970
                                        calltrap() at calltrap+0x8/frame 0xfffffe00cfdc4970
                                        --- trap 0xc, rip = 0xffffffff80f5a036, rsp = 0xfffffe00cfdc4a40, rbp = 0xfffffe00cfdc4a70 ---
                                        in6_selecthlim() at in6_selecthlim+0x96/frame 0xfffffe00cfdc4a70
                                        tcp_default_output() at tcp_default_output+0x1ded/frame 0xfffffe00cfdc4c60
                                        tcp_output() at tcp_output+0x14/frame 0xfffffe00cfdc4c80
                                        tcp6_usr_connect() at tcp6_usr_connect+0x2f4/frame 0xfffffe00cfdc4d10
                                        soconnectat() at soconnectat+0x9e/frame 0xfffffe00cfdc4d60
                                        kern_connectat() at kern_connectat+0xc9/frame 0xfffffe00cfdc4dc0
                                        sys_connect() at sys_connect+0x75/frame 0xfffffe00cfdc4e00
                                        amd64_syscall() at amd64_syscall+0x109/frame 0xfffffe00cfdc4f30
                                        fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00cfdc4f30
                                        --- syscall (98, FreeBSD ELF64, connect), rip = 0x800fddc8a, rsp = 0x7fffdfbfbc98, rbp = 0x7fffdfbfbcd0 ---
                                        db:1:pfs>
                                        

                                        Fourth:

                                        Crash report details:
                                        
                                        No PHP errors found.
                                        
                                        Filename: /var/crash/info.0
                                        Dump header from device: /dev/nvd0p3
                                          Architecture: amd64
                                          Architecture Version: 4
                                          Dump Length: 230400
                                          Blocksize: 512
                                          Compression: none
                                          Dumptime: 2023-05-28 15:17:27 +0100
                                          Hostname: Router-8.*******.me
                                          Magic: FreeBSD Text Dump
                                          Version String: FreeBSD 14.0-CURRENT #1 plus-RELENG_23_05-n256102-7cd3d043045: Mon May 22 15:33:52 UTC 2023
                                            root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05-main/obj/amd64/LkEyii3W/var/j
                                          Panic String: page fault
                                          Dump Parity: 1131880706
                                          Bounds: 0
                                          Dump Status: good
                                        
                                        Filename: /var/crash/textdump.tar.0
                                        ddb.txt���������������������������������������������������������������������������������������������0600����0�������0�������607520������14434661167�  7125� �����������������������������������������������������������������������������������������������������ustar���root����������������������������wheel������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������db:0:kdb.enter.default>  run pfs
                                        db:1:pfs> bt
                                        Tracing pid 2 tid 100041 td 0xfffffe0085264560
                                        kdb_enter() at kdb_enter+0x32/frame 0xfffffe00850ad910
                                        vpanic() at vpanic+0x183/frame 0xfffffe00850ad960
                                        panic() at panic+0x43/frame 0xfffffe00850ad9c0
                                        trap_fatal() at trap_fatal+0x409/frame 0xfffffe00850ada20
                                        trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00850ada80
                                        calltrap() at calltrap+0x8/frame 0xfffffe00850ada80
                                        --- trap 0xc, rip = 0xffffffff80f5a036, rsp = 0xfffffe00850adb50, rbp = 0xfffffe00850adb80 ---
                                        in6_selecthlim() at in6_selecthlim+0x96/frame 0xfffffe00850adb80
                                        tcp_default_output() at tcp_default_output+0x1ded/frame 0xfffffe00850add70
                                        tcp_timer_rexmt() at tcp_timer_rexmt+0x514/frame 0xfffffe00850addd0
                                        tcp_timer_enter() at tcp_timer_enter+0x102/frame 0xfffffe00850ade10
                                        softclock_call_cc() at softclock_call_cc+0x13c/frame 0xfffffe00850adec0
                                        softclock_thread() at softclock_thread+0xe9/frame 0xfffffe00850adef0
                                        fork_exit() at fork_exit+0x7d/frame 0xfffffe00850adf30
                                        fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00850adf30
                                        --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
                                        db:1:pfs>
                                        

                                        The 'solved' status on Redline may need revising:
                                        https://redmine.pfsense.org/issues/14164

                                        Sorry to be the bearer of this news. It is an awkward fault to have as even a small interrupt from my ISP can trigger the router to crash.

                                        1 Reply Last reply Reply Quote 0
                                        • stephenw10S
                                          stephenw10 Netgate Administrator
                                          last edited by

                                          Urgh, that's disappointing. That looks like two slightly different crashes though. Do you see anything other than the two backtraces shown above?

                                          I reopened it.

                                          Steve

                                          RobbieTTR 1 Reply Last reply Reply Quote 0
                                          • RobbieTTR
                                            RobbieTT @stephenw10
                                            last edited by

                                            @stephenw10
                                            There should be 4 back-traces shown above but I have full captures for all 4 events that I can send to you, if that would help.

                                            I hope you remember that it's a Bank Holiday Steve, so feel free to enjoy it instead. 👍

                                            Rob

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.