• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

Netgate 6100 Crash On Interface Change - Not Resolved (IPv6 + PPPoE)

Scheduled Pinned Locked Moved Official Netgate® Hardware
42 Posts 3 Posters 4.8k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • S
    stephenw10 Netgate Administrator
    last edited by Mar 23, 2023, 2:00 PM

    @robbiett said in Netgate 6100 Crash On Interface Change:

    db:1:pfs> bt
    [snip]

    The backtrace is usually the most telling part. It appears to have been snipped, do you have it?

    R 1 Reply Last reply Mar 23, 2023, 2:09 PM Reply Quote 0
    • R
      RobbieTT @stephenw10
      last edited by Mar 23, 2023, 2:09 PM

      @stephenw10
      I have it all, I just ran into a forum size limit - crude cuts were made.

      db:1:pfs> bt
      Tracing pid 4585 tid 100445 td 0xfffffe00cd4ba1e0
      kdb_enter() at kdb_enter+0x32/frame 0xfffffe00cd68c790
      vpanic() at vpanic+0x182/frame 0xfffffe00cd68c7e0
      panic() at panic+0x43/frame 0xfffffe00cd68c840
      trap_fatal() at trap_fatal+0x409/frame 0xfffffe00cd68c8a0
      trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00cd68c900
      calltrap() at calltrap+0x8/frame 0xfffffe00cd68c900
      --- trap 0xc, rip = 0xffffffff80fd9293, rsp = 0xfffffe00cd68c9d0, rbp = 0xfffffe00cd68ca20 ---
      in6_unlink_ifa() at in6_unlink_ifa+0x63/frame 0xfffffe00cd68ca20
      in6_purgeaddr() at in6_purgeaddr+0x367/frame 0xfffffe00cd68cb40
      in6_purgeifaddr() at in6_purgeifaddr+0x13/frame 0xfffffe00cd68cb60
      in6_control() at in6_control+0x532/frame 0xfffffe00cd68cbc0
      ifioctl() at ifioctl+0x7bc/frame 0xfffffe00cd68ccc0
      kern_ioctl() at kern_ioctl+0x26d/frame 0xfffffe00cd68cd30
      sys_ioctl() at sys_ioctl+0x101/frame 0xfffffe00cd68ce00
      amd64_syscall() at amd64_syscall+0x10c/frame 0xfffffe00cd68cf30
      fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00cd68cf30
      --- syscall (54, FreeBSD ELF64, sys_ioctl), rip = 0x18f47cd96e4a, rsp = 0x18f478021f28, rbp = 0x18f478021f70 ---
      

      Is the above what you are looking for?

      ☕️

      S 1 Reply Last reply Mar 23, 2023, 2:14 PM Reply Quote 0
      • S
        stephenw10 Netgate Administrator @RobbieTT
        last edited by Mar 23, 2023, 2:14 PM

        Yes, checking....

        1 Reply Last reply Reply Quote 0
        • S
          stephenw10 Netgate Administrator
          last edited by Mar 23, 2023, 3:06 PM

          Hmm, so you're losing link on ix1 there when the switch rebooted?

          And you have IPv6 addresses on ix1 dircetly or just on VLAN 1003 on that?

          R 1 Reply Last reply Mar 23, 2023, 3:20 PM Reply Quote 0
          • S
            stephenw10 Netgate Administrator
            last edited by Mar 23, 2023, 3:18 PM

            Are you able to replicate that manually?

            That is a bug, it should not happen. We probably need to replicate it locally to solve though.

            R 1 Reply Last reply Mar 23, 2023, 3:26 PM Reply Quote 0
            • R
              RobbieTT @stephenw10
              last edited by Mar 23, 2023, 3:20 PM

              @stephenw10
              Yes, lost link on firmware update (thanks UniFi) on this occasion but it can happen when changing interface settings on pfSense.

              On ix1 I have 1 x LAN and 1 x VLAN; these are my main networks. They both have IPv4 & IPv6.

              I have one other LAN interface on igc0 that acts as a management interface but it is not normally connected.

               2023-03-23 at 15.17.01.png

              ☕️

              1 Reply Last reply Reply Quote 0
              • R
                RobbieTT @stephenw10
                last edited by Mar 23, 2023, 3:26 PM

                @stephenw10 said in Netgate 6100 Crash On Interface Change:

                Are you able to replicate that manually?

                That is a bug, it should not happen. We probably need to replicate it locally to solve though.

                It is repeatable and happens roughly 60% of the time if settings are adjusted when the interface is enabled. If the interface is disabled first, then settings changed, then re-enabled it is much less prevalent, under 20% of the time at a guess.

                The example given for this thread is clearly just 1 in 1 event, albeit with the same undesirable result.

                ☕️

                1 Reply Last reply Reply Quote 0
                • S
                  stephenw10 Netgate Administrator
                  last edited by Mar 23, 2023, 3:46 PM

                  Ok, we replicated it here. Digging now....

                  R 1 Reply Last reply Mar 23, 2023, 3:53 PM Reply Quote 2
                  • R
                    RobbieTT @stephenw10
                    last edited by Mar 23, 2023, 3:53 PM

                    @stephenw10 said in Netgate 6100 Crash On Interface Change:

                    Ok, we replicated it here. Digging now....

                    Fine work, fine work indeed. 👍

                    1 Reply Last reply Reply Quote 0
                    • S
                      stephenw10 Netgate Administrator
                      last edited by Apr 3, 2023, 11:54 AM

                      A fix for this has now gone in upstream: https://redmine.pfsense.org/issues/14164

                      That's not something that can be patched at run time though.

                      Steve

                      R 1 Reply Last reply Apr 3, 2023, 12:34 PM Reply Quote 1
                      • R
                        RobbieTT @stephenw10
                        last edited by Apr 3, 2023, 12:34 PM

                        @stephenw10 - Thanks Steve, from your comment I guess this will percolate down for a version update window at an unknown date?

                        ☕️

                        1 Reply Last reply Reply Quote 0
                        • S
                          stephenw10 Netgate Administrator
                          last edited by Apr 3, 2023, 12:39 PM

                          Yes it should be in the next version, 23.05.

                          In fact it's in our repo now. It should be available for testing in todays snapshots:
                          https://github.com/pfsense/FreeBSD-src/commit/f5a365e51feea75d1e5ebc86c53808d8cae7b6d7

                          Steve

                          R 2 Replies Last reply Apr 3, 2023, 2:53 PM Reply Quote 0
                          • R
                            RobbieTT @stephenw10
                            last edited by Apr 3, 2023, 2:53 PM

                            @stephenw10
                            Thanks again and I am slightly embarrassed to find a bug so soon into my Netgate journey. I'll keep my head down for a bit!

                            ☕️

                            1 Reply Last reply Reply Quote 1
                            • S
                              stephenw10 Netgate Administrator
                              last edited by Apr 3, 2023, 3:09 PM

                              Don't be. If everyone reported bugs as soon as they found them with the details you did there would be far fewer to find!

                              Steve

                              A 1 Reply Last reply Apr 5, 2023, 9:02 PM Reply Quote 0
                              • A
                                azdeltawye @stephenw10
                                last edited by Apr 5, 2023, 9:02 PM

                                @stephenw10

                                Would this bug affect a 4100 also?

                                It appears that I stumbled across this issue the other day when I was messing with traffic shaping on an interface that had ipv6 enabled. The moment I clicked on 'save', the blue LED on the 4100 stopped blinking, UI became unresponsive and all clients lost internet access.
                                I had to cycle power on the 4100 to restore normal operation.

                                R 1 Reply Last reply Apr 5, 2023, 9:09 PM Reply Quote 0
                                • R
                                  RobbieTT @azdeltawye
                                  last edited by Apr 5, 2023, 9:09 PM

                                  @azdeltawye
                                  Sounds very similar, to say the least. Hopefully the fix will solve all.

                                  ☕️

                                  1 Reply Last reply Reply Quote 0
                                  • stephenw10S
                                    stephenw10 Netgate Administrator
                                    last edited by Apr 5, 2023, 9:38 PM

                                    Yeah, in fact I don't think it's system specific, or even NIC specific. But even if it is the 4100 is similar enough I'd expect it's possible to hit it there too.

                                    Steve

                                    1 Reply Last reply Reply Quote 0
                                    • RobbieTTR
                                      RobbieTT @stephenw10
                                      last edited by RobbieTT May 28, 2023, 3:00 PM May 28, 2023, 2:58 PM

                                      @stephenw10 said in Netgate 6100 Crash On Interface Change:

                                      Yes it should be in the next version, 23.05.

                                      In fact it's in our repo now. It should be available for testing in todays snapshots:
                                      https://github.com/pfsense/FreeBSD-src/commit/f5a365e51feea75d1e5ebc86c53808d8cae7b6d7

                                      Steve

                                      Hi Steve,
                                      Unfortunately the issue persists in 23.05. A change in interface state can still trigger a crash.

                                      I ran 7 simple and repeatable tests today - bringing the WAN interface down and back up again (this is enough to trigger the fault) via the disconnect button on Status/Interfaces.

                                      7 tests - 4 failures with hard crashes, 3 did not trigger a crash.

                                      All 4 failures produced a full crash report, info dump and textdump.tar. All available on request.

                                      The back-trace for the 4 crashes are as follows.

                                      First:

                                      Filename: /var/crash/info.0
                                      Dump header from device: /dev/nvd0p3
                                        Architecture: amd64
                                        Architecture Version: 4
                                        Dump Length: 231424
                                        Blocksize: 512
                                        Compression: none
                                        Dumptime: 2023-05-28 14:48:50 +0100
                                        Hostname: Router-8.*******.me
                                        Magic: FreeBSD Text Dump
                                        Version String: FreeBSD 14.0-CURRENT #1 plus-RELENG_23_05-n256102-7cd3d043045: Mon May 22 15:33:52 UTC 2023
                                          root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05-main/obj/amd64/LkEyii3W/var/j
                                        Panic String: page fault
                                        Dump Parity: 4132315394
                                        Bounds: 0
                                        Dump Status: good
                                      
                                      Filename: /var/crash/textdump.tar.0
                                      ddb.txt���������������������������������������������������������������������������������������������0600����0�������0�������610534������14434655702�  7122� �����������������������������������������������������������������������������������������������������ustar���root����������������������������wheel������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������db:0:kdb.enter.default>  run pfs
                                      db:1:pfs> bt
                                      Tracing pid 93402 tid 103857 td 0xfffffe00cf7cac80
                                      kdb_enter() at kdb_enter+0x32/frame 0xfffffe00cf8a0800
                                      vpanic() at vpanic+0x183/frame 0xfffffe00cf8a0850
                                      panic() at panic+0x43/frame 0xfffffe00cf8a08b0
                                      trap_fatal() at trap_fatal+0x409/frame 0xfffffe00cf8a0910
                                      trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00cf8a0970
                                      calltrap() at calltrap+0x8/frame 0xfffffe00cf8a0970
                                      --- trap 0xc, rip = 0xffffffff80f5a036, rsp = 0xfffffe00cf8a0a40, rbp = 0xfffffe00cf8a0a70 ---
                                      in6_selecthlim() at in6_selecthlim+0x96/frame 0xfffffe00cf8a0a70
                                      tcp_default_output() at tcp_default_output+0x1ded/frame 0xfffffe00cf8a0c60
                                      tcp_output() at tcp_output+0x14/frame 0xfffffe00cf8a0c80
                                      tcp6_usr_connect() at tcp6_usr_connect+0x2f4/frame 0xfffffe00cf8a0d10
                                      soconnectat() at soconnectat+0x9e/frame 0xfffffe00cf8a0d60
                                      kern_connectat() at kern_connectat+0xc9/frame 0xfffffe00cf8a0dc0
                                      sys_connect() at sys_connect+0x75/frame 0xfffffe00cf8a0e00
                                      amd64_syscall() at amd64_syscall+0x109/frame 0xfffffe00cf8a0f30
                                      fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00cf8a0f30
                                      --- syscall (98, FreeBSD ELF64, connect), rip = 0x800fddc8a, rsp = 0x7fffdf5f8c98, rbp = 0x7fffdf5f8cd0 ---
                                      db:1:pfs> 
                                      

                                      Second:

                                      Filename: /var/crash/info.0
                                      Dump header from device: /dev/nvd0p3
                                        Architecture: amd64
                                        Architecture Version: 4
                                        Dump Length: 226304
                                        Blocksize: 512
                                        Compression: none
                                        Dumptime: 2023-05-28 14:51:49 +0100
                                        Hostname: Router-8.*******.me
                                        Magic: FreeBSD Text Dump
                                        Version String: FreeBSD 14.0-CURRENT #1 plus-RELENG_23_05-n256102-7cd3d043045: Mon May 22 15:33:52 UTC 2023
                                          root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05-main/obj/amd64/LkEyii3W/var/j
                                        Panic String: page fault
                                        Dump Parity: 1095311618
                                        Bounds: 0
                                        Dump Status: good
                                      
                                      Filename: /var/crash/textdump.tar.0
                                      ddb.txt���������������������������������������������������������������������������������������������0600����0�������0�������577521������14434656165�  7136� �����������������������������������������������������������������������������������������������������ustar���root����������������������������wheel������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������db:0:kdb.enter.default>  run pfs
                                      db:1:pfs> bt
                                      Tracing pid 68614 tid 100330 td 0xfffffe00cf325720
                                      kdb_enter() at kdb_enter+0x32/frame 0xfffffe00c7d955f0
                                      vpanic() at vpanic+0x183/frame 0xfffffe00c7d95640
                                      panic() at panic+0x43/frame 0xfffffe00c7d956a0
                                      trap_fatal() at trap_fatal+0x409/frame 0xfffffe00c7d95700
                                      trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00c7d95760
                                      calltrap() at calltrap+0x8/frame 0xfffffe00c7d95760
                                      --- trap 0xc, rip = 0xffffffff80f63aa4, rsp = 0xfffffe00c7d95830, rbp = 0xfffffe00c7d95a50 ---
                                      ip6_output() at ip6_output+0xb74/frame 0xfffffe00c7d95a50
                                      udp6_send() at udp6_send+0x78e/frame 0xfffffe00c7d95c10
                                      sosend_dgram() at sosend_dgram+0x357/frame 0xfffffe00c7d95c70
                                      sousrsend() at sousrsend+0x5f/frame 0xfffffe00c7d95cd0
                                      kern_sendit() at kern_sendit+0x132/frame 0xfffffe00c7d95d60
                                      sendit() at sendit+0xb7/frame 0xfffffe00c7d95db0
                                      sys_sendto() at sys_sendto+0x4d/frame 0xfffffe00c7d95e00
                                      amd64_syscall() at amd64_syscall+0x109/frame 0xfffffe00c7d95f30
                                      fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00c7d95f30
                                      --- syscall (133, FreeBSD ELF64, sendto), rip = 0x823f95f2a, rsp = 0x8202cea88, rbp = 0x8202cead0 ---
                                      db:1:pfs>
                                      

                                      Third:

                                      Crash report details:
                                      
                                      No PHP errors found.
                                      
                                      Filename: /var/crash/info.0
                                      Dump header from device: /dev/nvd0p3
                                        Architecture: amd64
                                        Architecture Version: 4
                                        Dump Length: 229888
                                        Blocksize: 512
                                        Compression: none
                                        Dumptime: 2023-05-28 15:11:48 +0100
                                        Hostname: Router-8.*******.me
                                        Magic: FreeBSD Text Dump
                                        Version String: FreeBSD 14.0-CURRENT #1 plus-RELENG_23_05-n256102-7cd3d043045: Mon May 22 15:33:52 UTC 2023
                                          root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05-main/obj/amd64/LkEyii3W/var/j
                                        Panic String: page fault
                                        Dump Parity: 276046082
                                        Bounds: 0
                                        Dump Status: good
                                      
                                      Filename: /var/crash/textdump.tar.0
                                      ddb.txt���������������������������������������������������������������������������������������������0600����0�������0�������605706������14434660444�  7126� �����������������������������������������������������������������������������������������������������ustar���root����������������������������wheel������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������db:0:kdb.enter.default>  run pfs
                                      db:1:pfs> bt
                                      Tracing pid 3281 tid 100913 td 0xfffffe00cfe3e3a0
                                      kdb_enter() at kdb_enter+0x32/frame 0xfffffe00cfdc4800
                                      vpanic() at vpanic+0x183/frame 0xfffffe00cfdc4850
                                      panic() at panic+0x43/frame 0xfffffe00cfdc48b0
                                      trap_fatal() at trap_fatal+0x409/frame 0xfffffe00cfdc4910
                                      trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00cfdc4970
                                      calltrap() at calltrap+0x8/frame 0xfffffe00cfdc4970
                                      --- trap 0xc, rip = 0xffffffff80f5a036, rsp = 0xfffffe00cfdc4a40, rbp = 0xfffffe00cfdc4a70 ---
                                      in6_selecthlim() at in6_selecthlim+0x96/frame 0xfffffe00cfdc4a70
                                      tcp_default_output() at tcp_default_output+0x1ded/frame 0xfffffe00cfdc4c60
                                      tcp_output() at tcp_output+0x14/frame 0xfffffe00cfdc4c80
                                      tcp6_usr_connect() at tcp6_usr_connect+0x2f4/frame 0xfffffe00cfdc4d10
                                      soconnectat() at soconnectat+0x9e/frame 0xfffffe00cfdc4d60
                                      kern_connectat() at kern_connectat+0xc9/frame 0xfffffe00cfdc4dc0
                                      sys_connect() at sys_connect+0x75/frame 0xfffffe00cfdc4e00
                                      amd64_syscall() at amd64_syscall+0x109/frame 0xfffffe00cfdc4f30
                                      fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe00cfdc4f30
                                      --- syscall (98, FreeBSD ELF64, connect), rip = 0x800fddc8a, rsp = 0x7fffdfbfbc98, rbp = 0x7fffdfbfbcd0 ---
                                      db:1:pfs>
                                      

                                      Fourth:

                                      Crash report details:
                                      
                                      No PHP errors found.
                                      
                                      Filename: /var/crash/info.0
                                      Dump header from device: /dev/nvd0p3
                                        Architecture: amd64
                                        Architecture Version: 4
                                        Dump Length: 230400
                                        Blocksize: 512
                                        Compression: none
                                        Dumptime: 2023-05-28 15:17:27 +0100
                                        Hostname: Router-8.*******.me
                                        Magic: FreeBSD Text Dump
                                        Version String: FreeBSD 14.0-CURRENT #1 plus-RELENG_23_05-n256102-7cd3d043045: Mon May 22 15:33:52 UTC 2023
                                          root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_05-main/obj/amd64/LkEyii3W/var/j
                                        Panic String: page fault
                                        Dump Parity: 1131880706
                                        Bounds: 0
                                        Dump Status: good
                                      
                                      Filename: /var/crash/textdump.tar.0
                                      ddb.txt���������������������������������������������������������������������������������������������0600����0�������0�������607520������14434661167�  7125� �����������������������������������������������������������������������������������������������������ustar���root����������������������������wheel������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������db:0:kdb.enter.default>  run pfs
                                      db:1:pfs> bt
                                      Tracing pid 2 tid 100041 td 0xfffffe0085264560
                                      kdb_enter() at kdb_enter+0x32/frame 0xfffffe00850ad910
                                      vpanic() at vpanic+0x183/frame 0xfffffe00850ad960
                                      panic() at panic+0x43/frame 0xfffffe00850ad9c0
                                      trap_fatal() at trap_fatal+0x409/frame 0xfffffe00850ada20
                                      trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00850ada80
                                      calltrap() at calltrap+0x8/frame 0xfffffe00850ada80
                                      --- trap 0xc, rip = 0xffffffff80f5a036, rsp = 0xfffffe00850adb50, rbp = 0xfffffe00850adb80 ---
                                      in6_selecthlim() at in6_selecthlim+0x96/frame 0xfffffe00850adb80
                                      tcp_default_output() at tcp_default_output+0x1ded/frame 0xfffffe00850add70
                                      tcp_timer_rexmt() at tcp_timer_rexmt+0x514/frame 0xfffffe00850addd0
                                      tcp_timer_enter() at tcp_timer_enter+0x102/frame 0xfffffe00850ade10
                                      softclock_call_cc() at softclock_call_cc+0x13c/frame 0xfffffe00850adec0
                                      softclock_thread() at softclock_thread+0xe9/frame 0xfffffe00850adef0
                                      fork_exit() at fork_exit+0x7d/frame 0xfffffe00850adf30
                                      fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00850adf30
                                      --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
                                      db:1:pfs>
                                      

                                      The 'solved' status on Redline may need revising:
                                      https://redmine.pfsense.org/issues/14164

                                      Sorry to be the bearer of this news. It is an awkward fault to have as even a small interrupt from my ISP can trigger the router to crash.

                                      1 Reply Last reply Reply Quote 0
                                      • stephenw10S
                                        stephenw10 Netgate Administrator
                                        last edited by May 29, 2023, 12:50 AM

                                        Urgh, that's disappointing. That looks like two slightly different crashes though. Do you see anything other than the two backtraces shown above?

                                        I reopened it.

                                        Steve

                                        RobbieTTR 1 Reply Last reply May 29, 2023, 9:55 AM Reply Quote 0
                                        • RobbieTTR
                                          RobbieTT @stephenw10
                                          last edited by May 29, 2023, 9:55 AM

                                          @stephenw10
                                          There should be 4 back-traces shown above but I have full captures for all 4 events that I can send to you, if that would help.

                                          I hope you remember that it's a Bank Holiday Steve, so feel free to enjoy it instead. 👍

                                          Rob

                                          1 Reply Last reply Reply Quote 0
                                          • First post
                                            Last post
                                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
                                            This community forum collects and processes your personal information.
                                            consent.not_received