pfsense 2.5.2 slowly leaking memory

[SteveITS]

@murzik And just to be clear, you've patched for the pcscd issue and/or that service is stopped (Status/Services)? We do have one 2.5.2 and its graphs are literally flat today except at 6:30 this morning when free memory increased 1%. That's when Suricata updates.

Do you have pfBlocker's DNSBL active, or any other packages?

[johnpoz]

You can see exactly when I turned off pcscd a couple of days ago.. I thought I had stopped it, but it was running again..

[murzik]

Yes, I've patched pcscd. Do have pfblocker, snort, squid installed. But I had all the same packages for years.

[SteveITS]

@murzik I don't use DNSBL or Squid but you might read this recent post about pfBlocker.

You could try looking at Diagnostics/System Activity, or run "top -o res" (sort by memory) at a command line, and see what is using the memory. I get:

PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
81511 root         10  20    0   381M   337M nanslp   0 586:13   1.49% suricata
89847 root          1  52    0   135M    47M accept   0   0:29   0.00% php-fpm
93096 root          1  21    0   133M    46M accept   0   0:13   0.29% php-fpm
11372 root          1  52    0   132M    45M accept   1   0:03   0.00% php-fpm
338 root          1  52    0   133M    45M accept   0   0:11   0.00% php-fpm
337 root          1  52    0   132M    44M accept   1   0:34   0.00% php-fpm
26574 root          1  52    0    59M    38M piperd   1   0:00   0.00% php_pfb
73546 unbound       4  20    0    68M    32M kqread   1   1:17   0.00% unbound
336 root          1  20    0   102M    25M kqread   0   1:25   0.00% php-fpm
37003 root          1  20    0    29M  9648K kqread   2   0:07   0.05% nginx
36796 root          1  20    0    29M  9644K kqread   3   0:03   0.00% nginx
...

[murzik]

@steveits
The problem not what application uses the most memory, but that memory usage grow over the time. I have to reboot every other day.

Also memory usage displayed with "top -o res" does not account for all memory used memory shown in GUI.
Stopping PF blocker does not free memory.

[SteveITS]

@murzik said in pfsense 2.5.2 slowly leaking memory:

memory usage grow over the time

I was assuming there was a particular process increasing in size. If it's multiple processes that would make it a bit harder to diagnose, perhaps just watch over time. But if it isn't happening to everyone then it seems like it's not a generic FreeBSD/pfSense problem...

That said unbound on the two routers I just looked at is much smaller than 700 MB, around 30-50 MB. We don't have pfBlocker's DNSBL running though.

[Gertjan]

@murzik

pfBlockerNG, the DNSBL part : you're using the python mode ?

My unbound process is one of the biggest (but not the biggest) process in my system, but stays at around 100 Mbytes :

96339 unbound       2  20    0   101M    95M kqread   0   0:03   0.37% unbound

[stephenw10]

@murzik said in pfsense 2.5.2 slowly leaking memory:

I have to reboot every other day.

What happens if you do not reboot it?

That is something leaking quite fast if you are using 8GB in 2 days.

Steve

[murzik]

@stephenw10
If I do not reboot, pfsense will use up all the memory and swap and crash.

[murzik]

@gertja
Yes, I do use DNSBL in python mode. But I've been using it in python mode since it become available without any issues.... Also disabling DNSBL, restarting unbond, snort etc, does not free any memory....

[SteveITS]

@murzik said in pfsense 2.5.2 slowly leaking memory:

pfsense will use up all the memory and swap and crash

We can't see your process list...you'll need to tell us which processes are using up the memory.

[murzik]

@steveits Unfortunately I am not able to tell. When system crashes it is too late to check. However, I am starting to suspect that ntopng could be the problem. I remoted ntopng yesterday, so far memory usage stays around 30% . Will see in few days.

[Gertjan]

@murzik said in pfsense 2.5.2 slowly leaking memory:

ntopng

Aha !
You to forget to mention that you were using the big resource hog, the beast that should be set up carefully, and has to be managed 24/24h.
Tools like ntopng should not be left alone, as it files up files rapidly.
Most 'space' issues are not the RAM that starts too be consumed, but your entire disk space.
The result is the same : booooom.
With a full RAM and swap : the system goes down - but can restart it without issues. That is, if the file system wasn't corrupted when doing the unplanned system reboot.
A full file system : your system won't boot any more.

[murzik]

@gertjan
Apparently ntopng was not the cause. As of right now

[stephenw10]

If it doesn't show in the process list it's probably something in kernel like a driver.

Are all the affected systems running on the same hardware?

Steve