Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Kern.ipc.nmbufs limit reached

    Scheduled Pinned Locked Moved General pfSense Questions
    9 Posts 4 Posters 6.4k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      mr_steve
      last edited by

      Hey folks,

      I've got this recurring crash that I can't figure out; about once a week or so my pfSense box just drops off the network and stops passing traffic. When I look at the console, it says:

      [zone: mbuf] kern.ipc.nmbufs limit reached
      

      Everything I could find on the forum here and around the web suggested tweaking kern.ipc.nmbclusters upwards. I've done that, maybe a couple times. I don't remember what the default was but it's currently set to 262144.

      This is on 2.2.6, it also happened on the previous release and maybe the one before that as well.

      Any ideas?

      1 Reply Last reply Reply Quote 0
      • H
        heper
        last edited by

        https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards

        1 Reply Last reply Reply Quote 0
        • M
          mr_steve
          last edited by

          @heper:

          https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards

          Alright, yeah. That's the document where I got the idea to increase kern.ipc.nmbclusters.

          I suppose I'll try the recommended Broadcom bge tweaks and see what happens. It's a right pain to troubleshoot when there's such a long interval between crashes.

          1 Reply Last reply Reply Quote 0
          • H
            heper
            last edited by

            you can use the mbuf rrd graphs to see the frequency, or find out if you tweaks seem to solve your issue

            1 Reply Last reply Reply Quote 0
            • M
              mr_steve
              last edited by

              @heper:

              you can use the mbuf rrd graphs to see the frequency, or find out if you tweaks seem to solve your issue

              That's another thing that seems kind of odd. According to my RRD graphs, my mbuf usage never changes much at all, and never gets anywhere close to the maximum.

              24 hour data, including the most recent crash; you can see at the tail end where I just tweaked nmbclusters back down to the recommended bge settings:

              1 year data, since I brought this firewall up last April:

              Also, the y-axis label says 'percent', that can't be right. UI bug?

              1 Reply Last reply Reply Quote 0
              • M
                mr_steve
                last edited by

                So, here we are exactly a month later, and I had another nmbufs crash. I was able to get a local shell, but I could really tell was that the system load average was about 2.5, with pfctl and dhcpd each using around 50% CPU.

                I checked dmesg and saw nothing but the nmbufs error. /var/log/system.log didn't have any entries at all since Feb. 21, which was when it crashed last.

                I probably should spend a bit more time learning what files and commands on a BSD system will give me better info, for the next time this happens.

                1 Reply Last reply Reply Quote 0
                • ?
                  Guest
                  last edited by

                  Hello,

                  on some 10 GbE cards it was helping out to adjust this mbuf sizes to something around 65000
                  and in other cases pending on the installed RAM or system memory it was better to go in some
                  steps upwards, likes 250000, 500000 and 1000000. So you will see it is not like something to
                  type it in and then al is going fine, it is more a try and error game to get the best results matching
                  your system hardware and according to your RAM size. Please be careful to set it too low! Then you can
                  be easily ending up in a so called booting loop.

                  Even to tune one thing and then you might be getting over a smaller time a closer look on this system
                  if it was this exactly is better then turning on 50 things and you will never find out what was it exactly
                  in your situation!

                  1 Reply Last reply Reply Quote 0
                  • D
                    divsys
                    last edited by

                    I didn't notice any mention in your previous posts,
                    What hardware is this running on and are you using Nano or a full install, 32 or 63 bit?

                    -jfp

                    1 Reply Last reply Reply Quote 0
                    • M
                      mr_steve
                      last edited by

                      This is a 32-bit Intel system running a full install. 1 gig of RAM. The really weird thing is that the RRD graphs still show a completely steady low mbuf usage, much less than the limit set by kern.ipc.nmbufs.

                      1 Reply Last reply Reply Quote 0
                      • First post
                        Last post
                      Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.