Navigation

    Netgate Discussion Forum
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search

    Chelsio card becoming unresponsive on PFSense 2.2.6

    Hardware
    4
    7
    1358
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • E
      ericnk last edited by

      Hi,

      We have tried Chelsio T520 series cards in our PFSense firewall.  The first was a T520-LL-CR, the second was a T520-CR.  Both cards exhibit the same problem.  The second card was the very model that PFSense offers for sale on their store.  And, from what I've read, PFSense appliances use Chelsio 10G cards without incident.  So, naturally I'm confused when the cards we installed in our hardware become unresponsive, and ultimately require us to reboot our firewall to bring it back up.  Specifically, we get the following error in the system logs….

      Jun 6 15:02:55 kernel: [zone: mbuf] kern.ipc.nmbufs limit reached

      …  Other than this message, the only thing we have noticed is an increase in interrupts in the RRD Graph right about the same time.  Research on this message indicates that some system tunables need to be modified.  But, we've tried a few of the  changes mentioned in this forum and still the problem occurs.  It happens about every 24 hours.  Once the previous message occurs, everything goes down hill and a reboot is required.  Can anyone offer any suggestions as to how to resolve this problem?  Any replies are appreciated.  Thanks.

      Eric

      1 Reply Last reply Reply Quote 0
      • ?
        Guest last edited by

        Jun 6 15:02:55   kernel: [zone: mbuf] kern.ipc.nmbufs limit reached
        

        You should high up the mbuf size to 1000000 or 2000000 in your case, but only if you have enough RAM
        inside otherwise you could be ending up in a so called booting loop due the fact of an to less amount of RAM.

        And please create a loader.conf.local file and insert this changes the inside too.
        pfSense NIC tuning guide

        1 Reply Last reply Reply Quote 0
        • E
          ericnk last edited by

          Thanks for your reply.  The nmbclusters is already at 1000000.  We can try 2000000.  The system has 16GB of RAM.  But, somehow, I don't think it is the mbuf size, as the mbuf usage on the dashboard never goes much above 36k.  The dashboard indicator usually says…  4% (36230/1000000).  Regardless of what the error message says, I believe it may be a result of the initial problem, which could have been caused by the interrupt spikes we noticed in the RRD graph at about the time the error message occurred.  If the interrupt spike caused the card to become unresponsive, then the mbuf limit would have eventually been reached.  Regardless, we'll try the 2000000 setting.  Any other suggestions besides the tuning guide would be helpful as we have tried most all of the tuning suggestions in it.  Thanks.

          Eric

          1 Reply Last reply Reply Quote 0
          • D
            dreamslacker last edited by

            Do you have any active cooling for the Chelsio cards?

            Those units can run rather hot if you do not have forced airflow over them.

            1 Reply Last reply Reply Quote 0
            • E
              ericnk last edited by

              Yes.  We do have fans keeping a good flow over the card.  It stays within the operating temperature laid out in the card specs.

              1 Reply Last reply Reply Quote 0
              • D
                dreamslacker last edited by

                Can you share what are the tunings you've tried?

                I believe they'd be mostly related to MSI/ MSIX but what else have you tried?

                1 Reply Last reply Reply Quote 0
                • C
                  cmb last edited by

                  There's an mbuf leak under some circumstance in the Chelsio driver. Almost certain that's what you're running into given the description. It's been fixed in newer, not yet released OS versions. The workaround in the mean time, put the following in /boot/loader.conf.local

                  hw.cxgbe.allow_mbufs_in_cluster=0
                  
                  1 Reply Last reply Reply Quote 0
                  • First post
                    Last post

                  Products

                  • Platform Overview
                  • TNSR
                  • pfSense
                  • Appliances

                  Services

                  • Training
                  • Professional Services

                  Support

                  • Subscription Plans
                  • Contact Support
                  • Product Lifecycle
                  • Documentation

                  News

                  • Media Coverage
                  • Press
                  • Events

                  Resources

                  • Blog
                  • FAQ
                  • Find a Partner
                  • Resource Library
                  • Security Information

                  Company

                  • About Us
                  • Careers
                  • Partners
                  • Contact Us
                  • Legal
                  Our Mission

                  We provide leading-edge network security at a fair price - regardless of organizational size or network sophistication. We believe that an open-source security model offers disruptive pricing along with the agility required to quickly address emerging threats.

                  Subscribe to our Newsletter

                  Product information, software announcements, and special offers. See our newsletter archive to sign up for future newsletters and to read past announcements.

                  © 2021 Rubicon Communications, LLC | Privacy Policy