Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Intel X520-DA2, kernel: CRITICAL: ECC ERROR!! Please Reboot!!

    Scheduled Pinned Locked Moved Hardware
    60 Posts 13 Posters 29.4k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      miloman
      last edited by

      Any advice on this error i'm getting with pfsense 2.1RC2 amd64? (yes yes, stable is out, i know)

      i installed pfsense on a ibm server with an intel X520-DA2. when i connected it to a dell 10GbE switch using a DAC sfp+ cable, the link went up/down/up/down/up/down, and the console wrote out this message:

      Sep 10 16:16:06 check_reload_status: Linkup starting ix1
      Sep 10 16:16:06 kernel: ix1:
      Sep 10 16:16:06 kernel: ix1: link state changed to DOWN
      Sep 10 16:16:06 kernel:
      Sep 10 16:16:06 kernel: CRITICAL: ECC ERROR!! Please Reboot!!
      Sep 10 16:16:06 kernel: ix1:
      Sep 10 16:16:06 kernel: CRITICAL: ECC ERROR!! Please Reboot!!
      Sep 10 16:16:06 kernel:

      what to do?

      1 Reply Last reply Reply Quote 0
      • C
        cmb
        last edited by

        I've seen that too on one of our test systems. Haven't seen it on any production systems yet. I'm not sure what it means, haven't had time to look into it in much depth yet. -RELEASE wouldn't be any different, the driver didn't change since pre-RC2.

        1 Reply Last reply Reply Quote 0
        • D
          doktornotor Banned
          last edited by

          Well, this is the relevant part of the source code if it helps… :P

          
          if (reg_eicr & IXGBE_EICR_ECC) {
                         	device_printf(adapter->dev, "\nCRITICAL: ECC ERROR!! "
          		    "Please Reboot!!\n");
          		IXGBE_WRITE_REG(hw, IXGBE_EICR, IXGBE_EICR_ECC);
          	} else
          
          
          1 Reply Last reply Reply Quote 0
          • ?
            Guest
            last edited by

            have you tried rebooting?

            1 Reply Last reply Reply Quote 0
            • M
              miloman
              last edited by

              @cmb:

              I've seen that too on one of our test systems. Haven't seen it on any production systems yet. I'm not sure what it means, haven't had time to look into it in much depth yet. -RELEASE wouldn't be any different, the driver didn't change since pre-RC2.

              Even though it doesn't help me, your post gave me more information than google.  :)

              I have commercial support, so i'll ask "the guys" and see what they have to say.

              1 Reply Last reply Reply Quote 0
              • M
                miloman
                last edited by

                @gonzopancho:

                have you tried rebooting?

                Haven't we all?

                1 Reply Last reply Reply Quote 0
                • G
                  gaulfinger
                  last edited by

                  I have two pfsense test units with 10Gb-Base-LR Intel cards. Neither had this error on 2.03. Both started after the 2.1-RELEASE upgrade.

                  1 Reply Last reply Reply Quote 0
                  • jimpJ
                    jimp Rebel Alliance Developer Netgate
                    last edited by

                    For those seeing this error, do the NICs actually function when they hit this error? Or do they stop passing traffic?

                    Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                    Need help fast? Netgate Global Support!

                    Do not Chat/PM for help!

                    1 Reply Last reply Reply Quote 0
                    • W
                      wladikz
                      last edited by

                      Hi,

                      NICs are working but some problem with MBUF reuse. i have 2 firewalls with x520-DA2 and x540-T2 cards that crash every 8 hours after MBUFs hit 100%.
                      i have 512000 MBUFs defined on each system. i need to find solution for this problem ASAP.

                      1 Reply Last reply Reply Quote 0
                      • M
                        msi
                        last edited by

                        Hmm, I also get this error with a X520-DA2 on an IBM box after updating to 2.1.
                        I upgraded from a -RC0 around July, that was before the last ixgbe Update before shipping -RELEASE
                        I have the VLAN fix as per wiki documentation.

                        I'll have some chances to beat that thing a little bit and see how the MBUFs develop.
                        Otherwise I'll have to presto go back to 2.1-RC0 which was rock-solid (sorry to say I wasn't able to give -RC1/2 any beating) :-\

                        Update: I might give the .ko module from FreeBSD 8.3 or 8.4 a try - as workaround - since my RC0 at least shipped the version that was (almost) equivalent to what went into 8.4.

                        1 Reply Last reply Reply Quote 0
                        • W
                          wladikz
                          last edited by

                          @MatSim:

                          Hmm, I also get this error with a X520-DA2 on an IBM box after updating to 2.1.
                          I upgraded from a -RC0 around July, that was before the last ixgbe Update before shipping -RELEASE
                          I have the VLAN fix as per wiki documentation.

                          I'll have some chances to beat that thing a little bit and see how the MBUFs develop.
                          Otherwise I'll have to presto go back to 2.1-RC0 which was rock-solid (sorry to say I wasn't able to give -RC1/2 any beating) :-\

                          Update: I might give the .ko module from FreeBSD 8.3 or 8.4 a try - as workaround - since my RC0 at least shipped the version that was (almost) equivalent to what went into 8.4.

                          can you upload your .ko module? i'll try to see if the mbuf problem exists or not

                          1 Reply Last reply Reply Quote 0
                          • M
                            msi
                            last edited by

                            Hi all

                            While it's certainly not great to see that this issue slipped into 2.1-RELEASE I've heard from ESF crew, that they are actively working on a fix for this issue. If you want to join forces, offer them some of your support hours so they can work for this on paid basis. Anyhow, my not-so quick fiddlings on my part:

                            • The plain, but old ixgbe 2.4.5 from 8.3-RELEASE works with the X520-DA2 I have here.

                            • An experiment with 8.3's source tree + cherry-picks of 8.4's ixgbe paniced about at the level of boot comand disabling the VLAN filter (that was needed since RC0 for VLANs)

                            You can get the modules that I tried on my box here: http://id.gymkl.ch/pfsense/ixgbe
                            That said: This all is not endorsed by ESF crew or anyone else (not even myself). This may, or may not cause problems on your systems, nonetheless, if you are brave give it a try for workaround.

                            
                            # On the pfSense shell
                            cd /boot/kernel
                            fetch http://id.gymkl.ch/pfsense/ixgbe/ixgbe2.4.5-fbsd8.3-amd64.ko
                            chmod 555 ixgbe2.4.5-fbsd8.3-amd64.ko
                            
                            

                            Afterwards you add the following line to your /boot/loader.conf.local and then reboot the system:

                            
                            ixgbe2.4.5-fbsd8.3-amd64_load="YES"
                            
                            

                            You should see the ixgbe 2.4.5 version being loaded in /var/log/dmesg.boot afterwards.

                            1 Reply Last reply Reply Quote 0
                            • W
                              wladikz
                              last edited by

                              hi,

                              As i remember ixgbe version 2.4.5 has problems with VLANs. can some one check?

                              1 Reply Last reply Reply Quote 0
                              • M
                                msi
                                last edited by

                                I have a couple of VLANs on one of my ix interfaces, yes they do work without hickups here as for now. fingers crossed

                                You can easily check if VLANs are passing through when you fire up a 'tcpdump -i ix<number>_vlan<id>'. If you only see traffic on ix<number>, but not on the VLAN interface then you may want to give that VLAN hw filter a try like it is found on the wiki documentation. I currently have that set from -RC0 times.

                                Although remember this was not required until some builds that brough newer ixgbe drivers with lat BETA and RC builds. I also have followed Intels ixgbe instructions for loader.conf(.local) for the mbufs where they recommend larger nmbcluster than for the 1GE models. The num_queues options was select to 4 since this system has 4 cores / 4 threads so it can use up to 4 queues not use more (and possibly exhaust CPU capacity).

                                
                                kern.ipc.nmbclusters=262144
                                kern.ipc.nmbjumbop=262144
                                hw.ixgbe.num_queues="4"
                                ixgbe2.4.5-fbsd8.3-amd64_load="YES"
                                
                                ```</number></id></number>
                                1 Reply Last reply Reply Quote 0
                                • W
                                  wladikz
                                  last edited by

                                  hi,

                                  2.4.5 don't support one of  my 10G cards  :'( Can some one to compile 2.5.1 or 2.5.8?

                                  1 Reply Last reply Reply Quote 0
                                  • M
                                    msi
                                    last edited by

                                    Try 2.5.0 that is uploaded alongside, but as said.

                                    It made my box panic, but it may work for you, give it a try.

                                    It is line by line the same code as is in 8.4 and 8-STABLE.

                                    1 Reply Last reply Reply Quote 0
                                    • W
                                      wladikz
                                      last edited by

                                      i'll try to set my own build environment for 2.1 and i'll try to build

                                      1 Reply Last reply Reply Quote 0
                                      • rbgargaR
                                        rbgarga Developer Netgate Administrator
                                        last edited by

                                        I built a new 2.5.15 driver, with some fixes. Could you try it and let me know how it goes?

                                        You can get it at http://files.pfsense.org/garga/ixgbe_modules/2.1/

                                        Just put the ixgbe.ko at /boot/modules and add the following line to /boot/loader.conf.local

                                        ixgbe_load="YES"

                                        Best regards

                                        Renato Botelho

                                        1 Reply Last reply Reply Quote 0
                                        • W
                                          wladikz
                                          last edited by

                                          @Renato:

                                          I built a new 2.5.15 driver, with some fixes. Could you try it and let me know how it goes?

                                          You can get it at http://files.pfsense.org/garga/ixgbe_modules/2.1/

                                          Just put the ixgbe.ko at /boot/modules and add the following line to /boot/loader.conf.local

                                          ixgbe_load="YES"

                                          Best regards

                                          can you post also fixes?

                                          1 Reply Last reply Reply Quote 0
                                          • W
                                            wladikz
                                            last edited by

                                            @Renato:

                                            I built a new 2.5.15 driver, with some fixes. Could you try it and let me know how it goes?

                                            You can get it at http://files.pfsense.org/garga/ixgbe_modules/2.1/

                                            Just put the ixgbe.ko at /boot/modules and add the following line to /boot/loader.conf.local

                                            ixgbe_load="YES"

                                            Best regards

                                            system crashed  :'(

                                            crashreport.txt

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.