Netgate 6100 - High Mbuf usage
-
I have a Netgate 6100 running 23.09.1-RELEASE that is showing high Mbuf usage from time to time. I'm talking 80% and above. When it goes higher, I get dropped connections and unstable internet access. Eventually everything grinds to a halt and I have to reboot to recover.
Is this a known bug? What can I do to help figure out what is causing this?
Thanks
-
80% of what?
What does:
netstat -mshow?Steve
-
1556504/3571/1560075 mbufs in use (current/cache/total) 794224/3336/797560/1000000 mbuf clusters in use (current/cache/total/max) 751810/1554 mbuf+clusters out of packet secondary zone in use (current/cache) 74/1196/1270/524288 4k (page size) jumbo clusters in use (current/cache/total/max) 0/0/0/524288 9k jumbo clusters in use (current/cache/total/max) 0/0/0/41595 16k jumbo clusters in use (current/cache/total/max) 1977872K/12348K/1990221K bytes allocated to network (current/cache/total) 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) 0/0/0 requests for mbufs delayed (mbufs/clusters/mbuf+clusters) 0/0/0 requests for jumbo clusters delayed (4k/9k/16k) 0/0/0 requests for jumbo clusters denied (4k/9k/16k) 0 sendfile syscalls 0 sendfile syscalls completed without I/O request 0 requests for I/O initiated by sendfile 0 pages read by sendfile as part of a request 0 pages were valid at time of a sendfile request 0 pages were valid and substituted to bogus page 0 pages were requested for read ahead by applications 0 pages were read ahead by sendfile 0 times sendfile encountered an already busy page 0 requests for sfbufs denied 0 requests for sfbufs delayed -
Here's the picture on the Dashboard page
-
Hmm, using 800K mbufs is a lot. How much traffic are you passing?
Check the System - Mbufs Clusters graph in Status > Monitoring. Is it leaking mbufs over time?
-
Uptime is 9days, i think i rebooted on the March 13th and probably on March 4th too. Thank you so much for taking a look!
-
Hmm, not linear like a leak, just looks to be using a lot.
How much traffic is the 6100 passing?
What packages do you have installed?
-
@stephenw10
Little traffic
Here's my list of packages.
-
Hmm, nothing vert unusual there. Throughput is relatively low.
What NICs are you using?
-
Not sure where to find this. It's the default ones that come with the 6100.
-
Ok, which interfaces are in use in your setup then?
The defaults use IX for WAN and IGC for LAN so both NIC types. If you're using only one types that would be a clue. Or if you have added something for example.
-
@stephenw10
Oh I thought you meant if I'm using some different type of NIC hardware.My setup is a little bit complicated. I changed the defaults because I needed more than 1gbps on the WAN so I took advantage of one of the 2.5 Gbps ports as WAN instead. I have several WAPs that are capable of 2.5Gbps so I put those on the rest of the LAN ports and grouped them into a Bridge. Finally, I have some other networks of a hardwired switch using VLANs so those are trunked into ix2. See below.
-
Ok so you have a bridge interface which is relatively unusual. Which interfaces are in the bridge? Did you set any non-default value in the bridge config?
-
@stephenw10
Bridge interfaces are very common in most network equipment, not sure why you think it's unusual. The interfaces in the bridge are igc2,3,4. And no, there is no non-default-values in the bridge config. -
@ptchuba said in Netgate 6100 - High Mbuf usage:
Bridge interfaces are very common in most network equipment
Not so much in pfSense though. Running interfaces in a bridge rather than using a switch has a number of drawbacks. The biggest of which is that the firewall has to pass all the traffic between clients in the same segment and it uses almost as many CPU cycles as routing.
Either way most pfSense installs including most 6100s do not have bridges configured so they are unusual in that respect. I'm not aware of mbuf leaks on any other 6100 so the first thing to check is some unusual config. To be clear bridged interfaces should work fine and I'm aware of any issue with them.
Are you able to test without the bridge to rule that out?
-
@stephenw10
Thank you so much for helping with this. While the network is designed to have the ability to switch traffic within the bridge, there is actually very little of that in practice. Most of the devices are just wifi clients heading to the internet. That said, I know there is broadcast traffic and other multicast and discovery traffic that would flow automatically within the bridge but other than that there is just limited use of intra-bridge traffic for things like IoT controller-devices communications.The network is in production and I would prefer not to have to remove the bridge unless it's my last option.
Thanks again for helping.
