BSNMP causing massive memory use spikes since 26.03 update

keyser

These last few days have brought some really strange memory behaviour on my 6100 that i monitor using BSNMP from Zabbix.

Things have been running smoothly since I upgraded to 26.03 about four weeks ago, but suddenly last week memory usage went “beserk”.
Yesterday it took another massive jump in usage, and I decided to restart the BSNMP service today to see if it was the culprit - it was….

Any tips on how to diagnose whats causing those massive jumps in memory usage?

keyser

Here’s how it looks when i restart the BSNMP service in “Services”:

stephenw10

Anything else logged at that time?

There is another similar thread for this here: https://forum.netgate.com/topic/198889/ce-2.8.1-bsnmpd-memory-leak/

As far as I know we have not pinned it down to anything specific. We have not been able to replicate it locally so it may require some specific amount or traffic or devices.

keyser

@stephenw10 Yeah I noticed, but my conclusion was that it does not seem to be the same problem. My firewall shows no running memory leak from BSNMP - it stays flat until i does not, and eats 1+ GB of memory in one gulp at a time.

I do however use the same SNMP Template to my Zabbix 7.0.x install.

There is nothing suspicious or even remotely related logged on the firewall when BSNMP eats a large chunk of memory - except of couse the last time (last night) when that caused swap to go from 0 to 100% in a few seconds.

EDIT: Maybe I stand corrected - Here’s the graph from the last month - I updated to 26.03 on April 9th.

But I can’t see where the memory goes - it just goes from free to inactive. Then sudden a week ago about 3Gb of memory is eaten/allocated by BSNMP according the used memory line.

keyser

I have just checked another 6100 I have installed at a much less used location - It exibits the very same memory drop going from free to inactive, but at a slower pace than the 6100 I posted about (which is located at a location with more activity). It has not yet made the 3GB+ Used memory jump by BSNMP, but I expect it will in the not very distant future.

They both have more or less the exact same configuration (one is a copy of the other XML with IP adresses changed), apart from a few special rules on each. Same packages with the same config.

That suggests to me this is related to queries about pf-rules, states and such since the memory degradation is faster on the more active site.

EDIT: Scratch that. The memory drop goes at the same speed - I got confused by a reboot of that box 12 days ago.

Here’s the memory graph for that. It was upgraded on the 29 of march as far as I recall. But rebooted a couple of time the first week.

keyser

I can confirm it is some kind of memory leak in BSNMPD. Both firewalls shows the BSNMPD process slowly but steadily allocating more and more memory. So this is very likely the same case/issue as the previously referenced thread. Let’s use that thread going forward and stop posting here :-)