• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

Crash when switching interface OFF and ON again

Scheduled Pinned Locked Moved 2.5 Development Snapshots (Retired)
27 Posts 6 Posters 1.2k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • J
    JeGr LAYER 8 Moderator
    last edited by Jul 22, 2020, 5:58 AM

    @jimp also had 3-4 crashes in a week again. More than the whole last year with 2.4.x stable. Something is definitely off or making trouble driver-vise. And it's always something interface (maybe v6) related as it is every time my ISP does some "work" (meh) and the internet's gone.

    Don't forget to upvote πŸ‘ those who kindly offered their time and brainpower to help you!

    If you're interested, I'm available to discuss details of German-speaking paid support (for companies) if needed.

    1 Reply Last reply Reply Quote 0
    • L
      louis2
      last edited by Jul 22, 2020, 6:23 PM

      @JeGR , @jimp,

      In the past month my system was crashing like hel as soon if was even looking at the switch interface "off and on" button. Since a couple of days that seems a bit better, but it is not OK yet.

      I still can easily provoke a crash, but it is more in specific situations now, where switching an interface was almost a certain crash in the past 2 month.

      Note that:

      • I have native IPV6 so that can be related!
      • there are also multicast related issues, which are causing interface crashes

      I desperately tried to get PIMD to work, simply impossible on my system/configuration, but that also has to do with interfaces and vifs:

      • switching off and on the corresponding vip ==> crash
      • interfaces not being process correctly: "Jul 22 19:19:51 pfSense pimd[35276]: /var/etc/pimd/pimd.conf:7 - Invalid phyint address 'ix1.116'

      These issues are probably not caused by the pfSense software itself, but it definitively affect it. There are probably things in the FreeBSD-kernel not 100% OK,

      I did generate a FreeBSD-bug report (248103). I can see the wrong behavoir, however

      I do not have testing facilities, debug kernels etc, to tell what is technically exactly wrong (which module, which function call).

      So, where I am hoping for! is that Netgate having far better testing facilities and the skilled engineers, are willing to point more exact to the β€œdeeper down problem cause", so that the FreeBSD engineers can fix it!

      Quite sure that there is at least one sevire bug related to interface processing down there in the kernel (interrupt handling, iflib, multicast routing or so).

      Below a small part of a crash report I often see.

      Louis
      PS note that I amj using an amd64 system and the interfaces I am swithing off and on are all intel igb or ix based, and all using the iflib driver type.

      Fatal trap 12: page fault while in kernel mode
      cpuid = 1; apic id = 01
      fault virtual address = 0x1000
      fault code = supervisor write data, page not present
      instruction pointer = 0x20:0xffffffff80e934f5
      stack pointer = 0x28:0xfffffe00004de7f0
      frame pointer = 0x28:0xfffffe00004de7f0
      code segment = base 0x0, limit 0xfffff, type 0x1b
      = DPL 0, pres 1, long 1, def32 0, gran 1
      processor eflags = interrupt enabled, resume, IOPL = 0
      current process = 12 (swi1: netisr 2)
      trap number = 12
      panic: page fault
      cpuid = 1
      time = 1594920087
      KDB: enter: panic

      1 Reply Last reply Reply Quote 0
      • L
        louis2
        last edited by Jul 22, 2020, 8:18 PM

        This post is deleted!
        1 Reply Last reply Reply Quote 0
        • W
          w0w
          last edited by Jul 23, 2020, 6:13 AM

          I don't think it's related to IPv6, but to RADIX_MPATH option.
          https://forum.netgate.com/topic/150986/pf_test-kif-null-if_xname-on-multi-wan-and-reset-all-states-if-wan-ip-address-changes/33
          I've tested different variations and also disabled IPv6, nothing.
          My current crashes are:

          Filename: /var/crash/info.0
          Dump header from device: /dev/ada1p3
            Architecture: amd64
            Architecture Version: 4
            Dump Length: 74752
            Blocksize: 512
            Compression: none
            Dumptime: Fri Jul 10 22:17:47 2020
            Hostname: -
            Magic: FreeBSD Text Dump
            Version String: FreeBSD 12.1-STABLE df4360fdf61(devel-12) pfSense
            Panic String: sleeping thread
            Dump Parity: 4138356775
            Bounds: 0
            Dump Status: good
          
          Filename: /var/crash/info.1
          Dump header from device: /dev/ada1p3
            Architecture: amd64
            Architecture Version: 4
            Dump Length: 157696
            Blocksize: 512
            Compression: none
            Dumptime: Fri Jul 10 22:28:12 2020
            Hostname: -
            Magic: FreeBSD Text Dump
            Version String: FreeBSD 12.1-STABLE df4360fdf61(devel-12) pfSense
            Panic String: page fault
            Dump Parity: 30683997
            Bounds: 1
            Dump Status: good
          
          Filename: /var/crash/info.2
          Dump header from device: /dev/ada1p3
            Architecture: amd64
            Architecture Version: 4
            Dump Length: 157696
            Blocksize: 512
            Compression: none
            Dumptime: Tue Jul 21 13:41:55 2020
            Hostname: -
            Magic: FreeBSD Text Dump
            Version String: FreeBSD 12.1-STABLE fcbc0f88231(devel-12) pfSense
            Panic String: general protection fault
            Dump Parity: 1753744959
            Bounds: 2
            Dump Status: good
          
          Filename: /var/crash/info.3
          Dump header from device: /dev/ada1p3
            Architecture: amd64
            Architecture Version: 4
            Dump Length: 157696
            Blocksize: 512
            Compression: none
            Dumptime: Tue Jul 21 13:58:07 2020
            Hostname: -
            Magic: FreeBSD Text Dump
            Version String: FreeBSD 12.1-STABLE cf48cd75cf5(devel-12) pfSense
            Panic String: page fault
            Dump Parity: 680143114
            Bounds: 3
            Dump Status: good
          
          Filename: /var/crash/info.4
          Dump header from device: /dev/ada1p3
            Architecture: amd64
            Architecture Version: 4
            Dump Length: 75776
            Blocksize: 512
            Compression: none
            Dumptime: Wed Jul 22 12:01:14 2020
            Hostname: -
            Magic: FreeBSD Text Dump
            Version String: FreeBSD 12.1-STABLE cf48cd75cf5(devel-12) pfSense
            Panic String: general protection fault
            Dump Parity: 387144248
            Bounds: 4
            Dump Status: good
          

          I have ix card also. Played with drivers some time ago. Nothing. Since I've backup firewall I've moved backup to 2.5 and primary to 2.4. The fatal errors are still coming only on 2.5 version even on absolutely different hardware.

          1 Reply Last reply Reply Quote 0
          • J
            JeGr LAYER 8 Moderator
            last edited by Jul 23, 2020, 8:44 AM

            Then it may be another problem for you @w0w as my appliance uses igb NICs and it's crash-dumping immediatly if something is triggering the NIC like an "internet down" or "interface down" on the WAN. The last few weeks if my ISP has problems and reboots my front router (AVM box) I can wait for the "beep" of my pfSense box shortly after as it crashdumps and reboots.

            Don't forget to upvote πŸ‘ those who kindly offered their time and brainpower to help you!

            If you're interested, I'm available to discuss details of German-speaking paid support (for companies) if needed.

            W 1 Reply Last reply Jul 23, 2020, 9:15 AM Reply Quote 0
            • W
              w0w @JeGr
              last edited by Jul 23, 2020, 9:15 AM

              @JeGr
              No, my backup firewall uses igb and moreover, I have tested em card, same crashes. I do not think it is hardware related, I am pretty sure it is not.

              1 Reply Last reply Reply Quote 0
              • L
                louis2
                last edited by Jul 23, 2020, 1:44 PM

                @w0w said in Crash when switching interface OFF and ON again:

                @JeGr
                No, my backup firewall uses igb and moreover, I have tested em card, same crashes. I do not think it is hardware related, I am pretty sure it is not.

                We simply do not know what is causing the crashes. Can be one bug very deep down or multiple bugs.

                However be aware that FreeBSD is in FreeBSD12 redesigning its driver structure.

                Where each interface type used to have its own driver, they are transforming that in the direction of "a mini driver per device type" and doing the upperlayer driver things in a new library called "iflib"

                I know that on my intel64 system all three IO-ports em, igb and ix are using this principle / the "iflib".

                Louis

                1 Reply Last reply Reply Quote 0
                • W
                  w0w
                  last edited by Jul 23, 2020, 4:52 PM

                  @louis2 Yep. But those or similar crashes you have now, I have had when 2.5 was based on 12.0-RELEASE using standard driver. The situation was exactly the same, any movements like up/down, unplugging-replugging interface caused the crash, not always and sometimes it was immediately, sometimes not. After some trial and error, googling, digging and so on I think now that it is radix_mpath option and maybe this feature is not buggy by itself but it shows some system errors that never otherwise appear. The only way I see to test it is to compile the kernel without radix_mpath option enabled.

                  1 Reply Last reply Reply Quote 0
                  • L
                    louis2
                    last edited by Jul 23, 2020, 6:39 PM

                    @w0w said in Crash when switching interface OFF and ON again:

                    radix_mpath

                    radix_mpath is related to multipath rooting. I did not dig into that, but it sounds like route redundancy, something I am not using (what not necessary implies that I am not affected).

                    I also noticed that there is already a pfSense feature request, noting that there are known issues inside radix_mpath (Feature #9544).
                    However my personal feeling is that in my case it is more likely to expect the problem in some multicast module or in some generic interface handler.

                    If …. I just had an "in circuit emulator" here attached to my system / your system and all sources, we would know within hours, ..... since most issues are reproduceable ....... but of course we do not have that kind of very advanced tooling.

                    If some one would compile a pfSence version having a lot of extra debug statements .... that would probably help as well ....

                    Louis

                    1 Reply Last reply Reply Quote 0
                    • W
                      w0w
                      last edited by Jul 29, 2020, 9:08 AM

                      May be @rschell can help, at least with pfSense version compiled without radix_mpath. 😊
                      Unfortunately, my knowledge is not enough to build the kernel myself.

                      1 Reply Last reply Reply Quote 0
                      • L
                        louis2
                        last edited by Jul 29, 2020, 12:12 PM

                        @w0w, @rschell

                        I did open bug at FreeBSD Bugzilla – Bug 248243
                        https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=248243

                        It would help if you support me there by entering info about the issues you noticed (and perhaps your verdict). Of cause supported by facts as far as possible.

                        Louis

                        ? 1 Reply Last reply Jul 29, 2020, 6:55 PM Reply Quote 0
                        • ?
                          A Former User @louis2
                          last edited by Jul 29, 2020, 6:55 PM

                          @louis2 I have a 2.5 development version without the RADIX_MPATH option enabled (no additional debug statements) that I could share with you. You have a preference ISO or memstick?

                          W 1 Reply Last reply Jul 30, 2020, 10:03 AM Reply Quote 0
                          • L
                            louis2
                            last edited by Jul 29, 2020, 8:07 PM

                            @rschell, @w0w,

                            I am not sure RADIX_MPATH is casing the trouble for me, at least not if I do interpret the reason RADIX_MPATH correctly (I do not use multiple paths)

                            However to be sure I could try a memstick version, if it is based on a very recent generic 2.5 build.

                            What I really would love to have !!! , is a pfSense debug version tracking all communication between pimd and the freebsd kernel and saving that with timestamps in a file which is not lost in case of a crash.

                            Louis

                            1 Reply Last reply Reply Quote 0
                            • W
                              w0w @A Former User
                              last edited by Jul 30, 2020, 10:03 AM

                              @rschell
                              Memstick would be great. Thanks!

                              1 Reply Last reply Reply Quote 0
                              27 out of 27
                              • First post
                                27/27
                                Last post
                              Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
                                This community forum collects and processes your personal information.
                                consent.not_received