[IMPORTANT pfSense+ 25.11 - ntopng memory leak? rendering 7100 DT unresponsive

sandie

Hi guys,

There is some scenario for memory leak in 25.11 Plus - feel free to say what kind of traces should I collect for you.

In 7100 DT I am running ntopng since few years (next to haproxy and other packages), but when ntopng runs for a while it will lead to complete unresponsiveness of device. I noticed router slowness and 96% of consumed memory. I was able to reboot router, but SSH was slow and "console menu/banner" was loading line by line literally.
Moreover today in the morning device completely hung, so I had to power it off, because could not connect via SSH to it [actually I connected twice, but console menu/banner did not display after few minutes]. I tried also connecting to webUI - it was spinning for few minutes and finally resulted in "crash reporter". I was not able to manage device in any reasonable way (maybe Nexus would help, but I am not using it - willing to see some public presentation from you guys, because it seems to be good stuff :))

I made some basic testing and observations on upgraded 7100 DT, but there is something really WRONG with ntopng in this release. It eats whole RAM over time and renders device unusable. I did not face troubles when ntopng was down. Note that already consumed memory was not released when I stopped ntopng (!).

Here are some top screens I collected for you when ntopng was running:

Here are screens from dashboard - but this problem is NOT dashboard related (I remember in the past you had some memory leak/problem in dashboard, but I made observations and testing without sitting in Dashboard):

I am 99.99% sure that ntopng leads to hung/unresponsiveness of my 7100 DT.

Please suggest anything I should collect.

I disabled ntopng in 7100 DT and so far I am NOT seeing memory utilization going crazy. Will confirm tommorow. I want to avoid downgrade / BootEnv restore point.

BTW. I am using ntopng in 6100 too. Device uptime is: "Uptime 1 Day 08 Hours 01 Minute 26 Seconds" and completely no problems faced on 6100 :( (7100 DT died in a day)
Something with 7100 DT or its setup is going wild :-(
I turned off ntopng in 6100 to avoid troubles although I am not observing there memory leak I think. Maybe it has some relation to haproxy too? (I do not run haproxy on 6100)

BTW2. Please note ARC cache is growing on 7100 DT over time [pretty fast] and "wired memory" generally only increases over time. It is not recycled when I stop ntopng. ARC grows strangely.

dennypage

@sandie said in [IMPORTANT pfSense+ 25.11 - ntopng memory leak? rendering 7100 DT unresponsive:

I made some basic testing and observations on upgraded 7100 DT, but there is something really WRONG with ntopng in this release. It eats whole RAM over time and renders device unusable. I did not face troubles when ntopng was down.
...
Here are some top screens I collected for you when ntopng was running

There is no question that ntopng is a very expensive process to run, particularly cpu and io. That said, I'm curious how did you came to the conclusion that ntopng was consuming all memory?

Neither of the snapshots posted show ntopng consuming a lot of memory. 221MB of memory is actually rather small for ntopng. The suricata process in your screenshot is over three times the size of ntopng.

Note that already consumed memory was not released when I stopped ntopng (!).

Hmm... this would be a strong indication that ntopng itself is not the source of whatever memory issue you are seeing.

You're not using RAM disks are you?

sandie

@dennypage Hi Denny,
I am not using RAM disks even though 7100 DT has lots of RAM.
I could show you that when ntopng is enabled in 7100 DT "wired memory" will only be growing over time. And it grows pretty quickly - 1% for some minutes.
I do not understand why ARC mem is also growing so fast (never was), but ARC mem is released and goes down. However something bad happens and "wired" just grows continously when ntopng is working in 7100 DT. Moreover wired is not released when I stop ntopng.
~1 day of uptime and it will NOT be possible to login to 7100 DT anymore. Router will start working noticeably slow. SSH connection will estabilish, but it will become so slow that it will not print menu to console in few minutes.
I know it is strange observation and it does not happen in 6100, but this weirdness hit me.

BTW. During weekend I am copying terabytes of data between 2 NASes within 1 location (7100 DT routed). 6100 is not involved in these transfers.

PS. With ntopng disabled 7100 DT router sits with mem utilization < 10%. I will confirm tommorow, but it is quite stable. Also ARC does not grow so rapidly.

7100 DT without ntopng (stable for last 03 Hours 14 Minutes 06 Seconds:

It is easily observable on my device (7100 DT).

PS2. It could also be in theory Suricata which is not running in 6100. But I did not disable Suricata in 7100 DT, only ntopng. And I think problems are gone (memory utilization is rock solid now, even data transfers are still running).

dennypage

@sandie said in [IMPORTANT pfSense+ 25.11 - ntopng memory leak? rendering 7100 DT unresponsive:

I could show you that when ntopng is enabled in 7100 DT "wired memory" will only be growing over time. And it grows pretty quickly - 1% for some minutes.

I do not understand why ARC mem is also growing so fast (never was), but ARC mem is released and goes down. However something bad happens and "wired" just grows continously when ntopng is working in 7100 DT. Moreover wired is not released when I stop ntopng.

I suggest two google searches:

"freebsd what is wired memory"
"freebsd what is arc memory"

Short version: Wired and ARC are kernel memory, and could not be the result of a "memory leak" in ntopng itself. Running ntopng, which is a large process, may certainly help contribute to whatever issue you are experiencing, but so could any other process.

Further, given that your screenshots show 22GB and 25GB of free memory, it's unclear that the problem you are experiencing is associated with memory to begin with.

You might want to perform a broader investigation before narrowing your focus so.

sandie

@dennypage Sure, but 2 observations.

ARC never was growing this fast like when ntopng worked in 25.11.
There are cycles and ARC is reduced but wired is only ever growing over time - it will reach 96% and it will reach more rendering device pretty much unusable. This was not observed before in any pfSense+ release.

I am only mentioning what I am observing. ntopng is down, data is still being transffered (I will stop transfer about 08:00 AM CET) and memory consumption does not grow rapdily anymore.

So far I never also seen died(unresponsive) pfSense box. It was first time (box was responding to pings and was routing traffic, but slower).

I tried enabling 2x ntopng today and it always resulted it growing memory utilization (I observed over 1,5h how situation looks like and collected screens). I reported here my observations even though I am not sure what is going on. I can collect whatever you guys need. I can easily replicate this issue. Perhaps significant data transfer is required.

Last 2 days (weekend) I am transferring data between 2 NASes. Nothing fancy. DT 7100 is busy, 6100 is idling in other location. Both devices are no longer running ntopng, because I am not looking for problems on Monday (I am not going to experiment during working day).

dennypage

@sandie You are making a lot of assumptions. Some of them may be correct, some of them may be incorrect. But by narrowing your focus to ntopng so strongly, you are preventing yourself from considering broader issues. Your choice I guess.

If you aren't using an external monitoring system, I would recommend at least examining the CPU and memory graphs available in Status / Monitoring. You'll get a better view of cpu and memory over time.

sandie

@dennypage Ok, I do not like bad assumptions too :)
Will add monitoring to pfSense+ boxes in near future, but I am truly busy with other things.
I am trying to help you and myself, not willing to mislead anyone here. These guys know what they are doing, so I am sure will guide me to collect whatever is needed.

sandie

Seems I am not the only one affected - https://www.reddit.com/r/PFSENSE/s/AoqmetTxur