CE 2.8.1 bsnmpd Memory Leak
-
Just to add one vote to confirm memory leak. Using with zabbix standard template reading values, every week needs to restart snmpd service due to swap space full.
As a temporary workaround, added daily in cron: pfSsh.php playback svc restart bsnmpd -
@psp As Steve said, I've been unable to reproduce this.
It'd be helpful if you could narrow down the OID to (repeatedly) query to reproduce the leak.
-
-
@Averlon No telegraf here. Standard Zabbix "pfSense by SNMP" template uses BEGEMOT-PF-MIB and HOST-RESOURCES-MIB to walk interfaces and get OID values.
-
@kprovost BEGEMOT-PF-MIB and HOST-RESOURCES-MIB it is.
-
@Averlon I still can't reproduce this leak. That's why I asked if you could identify the specific OID that's causing the leak. That way I could inspect the code and recent related changes to see if I could identify the leak.
Naturally I cannot do that for the entire trees under those two nodes. -
@kprovost I can nail down the BEGEMOT-PF-MIB to these OIDs for regular walks
- 1.3.6.1.4.1.12325.1.200.1.1
- 1.3.6.1.4.1.12325.1.200.1.11
- 1.3.6.1.4.1.12325.1.200.1.2
- 1.3.6.1.4.1.12325.1.200.1.3
- 1.3.6.1.4.1.12325.1.200.1.5
- 1.3.6.1.4.1.12325.1.200.1.8
For HOST-RESOURCES-MIB to these OIDs
- 1.3.6.1.2.1.25.3.3.1
- 1.3.6.1.2.1.25.4.2.1
After 24h the bsnmp process is at approx. 500MB if the monitoring is querying the device on a 60 second base. Not sure why this doesn't occur when you try to reproduce it. It may require a certain count of firewall rules to trigger the condition which causing the leak. The firewalls I observed this, have ~ 400 to 600 rules.
-
@Averlon Do each of those leak individually or do you need to query all of those to provoke the leak?
You've listed most of the BEGEMOT-PF-MIB mib here.
My test setup has the standard CE ruleset, which is already has 100 rules in it, and there's no point at which the relevant code does different things for more rules.
All of the nodes you mentioned are populated in my test setup, and I've been polling as quickly as the test device would respond, not every 60 seconds. I'd expect that to result in an even faster leak that you described, but there's nothing.
-
@kprovost I'm at the point where I doubt the leak can be triggered by simply running queries on specific MIBs. Monitoring with SNMP is still very common and this topic hasn't got much attention, what let assume that not every configuration is affected by this. In addition your tests shown that the conditions to reproduce this issue, isn't straight forward as running snmpwalks as fast as possible.
I have three different pfSense installations which are affected by leaks of the bsnmp process. All of these have this in common:
- These are VMs
- SNMP v2 is in use (YES - Security Not My Problem)
- Bridge Interfaces for transparent filtering are in used (physical interfaces only, no bridge interface)
- IPSec with VTI Interfaces are used
- The SNMP monitoring runs queries via the VTI IFs to the LAN IP, these may flap.
- BGP from FFR Package is running and peering via VTIs
- There is a pretty high load of dropped / logged IP communication from the firewall
- The Table sizes are pretty large due pfBlockerNG rules
I'm currently on a business trip and cannot do further tests in my environment.I may find some time next weekend to do more investigation.
Meanwhile maybe @psp can share some details about his environment to find the common ground for this issue.
-
Just got home today and my wife told me about the plans for the weekend. Unfortunately it doesn't include troubleshooting sessions for this issue. I'll have to postpone contributions till next week - sorry.
-
Priorities.
