23.01 grep consuming all memory

motific

I migrated from 2.6.0CE to plus then up to 23.01 and now I'm getting issues with memory exhaustion to the point where the system becomes inoperable.

Packages installed are

acpusd
avahi
bandwidthd
darkstat
nmap
pfBlockerNG-devel

all are currently up to date.

This is the output from top showing grep eating all resources.

CPU: 70.8% user,  0.0% nice, 25.5% system,  3.8% interrupt,  0.0% idle
Mem: 5640M Active, 135M Inact, 753M Laundry, 1117M Wired, 759M Buf, 30M Free
Swap: 4096M Total, 4096M Used, K Free, 100% Inuse, 148M In

  PID USERNAME    THR PRI NICE   SIZE    RES STATE    C   TIME    WCPU COMMAND
45388 root          1 108    0   518M   422M RUN      2 234:08  78.30% grep
39739 root          1  99    0   512M   423M CPU0     0  40:15  51.22% grep
82865 root          1  96    0   518M   423M RUN      1 368:45  33.26% grep
34308 root          1 101    0   518M   465M RUN      3   7:22  27.23% grep
54032 root          1  96    0   512M   423M CPU1     1 109:08  23.92% grep
67624 root          1  93    0   512M   424M RUN      3 299:25  15.39% grep
67011 root          1  68    0   512M   424M pfault   1  17:46  14.51% grep
78875 root          1  95    0   512M   422M RUN      1 181:07  11.82% grep
83870 root          1  93    0   518M   424M RUN      1 139:15  10.87% grep
72681 root          1  96    0   512M   424M RUN      1 445:19   4.46% grep
29609 root          1  23    0   518M   393M pfault   2  24:56   4.14% grep
77378 root          1  21    0   512M    83M pfault   0  49:03   3.67% grep
40524 root          1  21    0   512M    73M pfault   0  37:07   3.58% grep
15487 root          1  23    0   512M   196M pfault   0  92:18   3.14% grep
70544 root          1  68    0   518M   416M pfault   0  37:18   2.81% grep
53575 root          1  23    0   518M   224M pfault   0  15:29   2.51% grep
96266 root          1  68    0   518M   423M pfault   0 164:41   2.47% grep
55409 root          1  23    0   512M    69M pfault   3  44:53   2.18% grep
34192 root          1  68    0   518M   423M pfault   0 126:20   1.54% grep

I can't see what's causing grep to do that much work and I'm going to start with disabling pfBlocker, but any other pointers would be valuable. Thanks!

Gertjan

@motific

If you think it might be pfBlocker, and grep is mishabaving, then you might be interested in what has been said in the pfBlocker pfSense support forum.

These was an issue, a couple of weeks ago, where grep would take ages to execute, and clog up all processor power.
However : that was solved with the latest version pfBlockerNG-devel 3.2.0_3. I guess the 'pfBlockerNG' (without de devel part) is also ok.

If it's not pfBlocker , remove the big packages and see that the issue is gone..
Now, install one of them, and observe.
If it's safe - you found nothing, add another package. As soon as 'grep' comes back, you'll know what to stop using ;)

stephenw10

You can probably see what it running those greps easily enough using ps. Like: ps -auxwwd

motific

@gertjan - I do know how to do the troubleshooting, I was just trying to avoid the 24 hours for each iteration but I'm sure it will help someone else if they're reading.

I had already updated pfBlocker but it looks like those processes were orphaned from before I updated the package. All seems to be fine now.