Memory usage climbing

cpthk

This is what I got, anything I could do to help on reducing the memory? I have 1GB of ram.
It looks like tcpdump uses the most, I do know tcpdump is the tool to capture network packets. In pfsense, which feature is it being used?

last pid: 9692; load averages: 0.00, 0.00, 0.00 up 14+22:26:54 04:48:15
117 processes: 3 running, 99 sleeping, 15 waiting

Mem: 134M Active, 15M Inact, 58M Wired, 1044K Cache, 26M Buf, 755M Free
Swap: 2048M Total, 2048M Free

PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
11 root 171 ki31 0K 32K RUN 0 355.6H 100.00% {idle: cpu0}
11 root 171 ki31 0K 32K CPU1 1 355.0H 100.00% {idle: cpu1}
12 root -68 - 0K 240K WAIT 1 75:56 0.39% {irq16: dc0 uhci0}
11958 root 76 0 102M 21408K piperd 0 0:02 0.10% php
12 root -32 - 0K 240K WAIT 1 129:17 0.00% {swi4: clock}
0 root -68 0 0K 112K - 0 33:17 0.00% {em0 taskq}
14 root -16 - 0K 16K - 1 5:01 0.00% yarrow
24300 root 44 0 14776K 2796K select 0 4:36 0.00% syslogd
21240 root 44 0 69112K 62560K bpf 1 3:05 0.00% tcpdump
21296 root 44 0 5832K 1040K piperd 1 2:04 0.00% logger
59743 root 76 20 8292K 1700K wait 1 1:17 0.00% sh
31670 root 64 20 5836K 1508K select 0 1:13 0.00% apinger
46248 nobody 44 0 10144K 3460K select 1 1:00 0.00% dnsmasq
0 root 45 0 0K 112K sched 0 0:50 0.00% {swapper}
40988 dhcpd 44 0 13052K 7276K select 1 0:47 0.00% dhcpd
25 root 44 - 0K 16K syncer 1 0:46 0.00% syncer
33972 root 44 0 25768K 5684K kqread 1 0:39 0.00% lighttpd
2 root -8 - 0K 16K - 1 0:37 0.00% g_event

stephenw10

To be honest 20% memory use doesn't seem like a problem to me. Unless it's still climbing.
My own system is usually at around 30% (of 512MB) when idle. See attached RRD graph.
For comparrsion here is my top -SH output:

last pid: 46273;  load averages:  0.28,  0.12,  0.04                                                up 37+20:42:52  13:10:27
108 processes: 2 running, 90 sleeping, 16 waiting
CPU:  0.0% user,  0.0% nice,  0.7% system,  0.0% interrupt, 99.3% idle
Mem: 61M Active, 18M Inact, 62M Wired, 680K Cache, 59M Buf, 343M Free
Swap:

  PID USERNAME PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
   10 root     171 ki31     0K     8K RUN    876.8H 98.00% idle
   11 root     -32    -     0K   128K WAIT   116:14  0.00% {swi4: clock}
   11 root     -68    -     0K   128K WAIT    52:18  0.00% {irq18: em0 ath0+}
12511 root      76   20  3656K  1580K wait    23:31  0.00% sh
    0 root     -68    0     0K    88K -       20:41  0.00% {em0 taskq}
 1807 nobody    74  r30  3368K  1484K nanslp  16:27  0.00% LCDd
   11 root     -68    -     0K   128K WAIT    16:24  0.00% {irq17: fxp2 fxp6}
63939 root      65   20 46428K 17528K nanslp  13:21  0.00% php
   11 root     -44    -     0K   128K WAIT    10:50  0.00% {swi1: netisr 0}
   13 root     -16    -     0K     8K -        6:23  0.00% yarrow
    0 root     -68    0     0K    88K -        4:49  0.00% {ath0 taskq}
29948 root      64   20  3316K  1336K select   4:01  0.00% apinger
    0 root     -68    0     0K    88K -        4:01  0.00% {em2 taskq}
    0 root     -68    0     0K    88K -        3:52  0.00% {em1 taskq}
   20 root      44    -     0K     8K syncer   3:38  0.00% syncer
   35 root      -8    -     0K     8K mdwait   3:23  0.00% md1
    2 root      -8    -     0K     8K -        2:46  0.00% g_event
   11 root     -68    -     0K   128K WAIT     2:27  0.00% {irq19: fxp0 fxp4}
 2392 dhcpd     44    0  8436K  6012K select   2:11  0.00% dhcpd
    0 root     -16    0     0K    88K sched    2:09  0.00% {swapper}
17355 root      44    0  6588K  5060K kqread   2:04  0.00% lighttpd
    4 root      -8    -     0K     8K -        1:47  0.00% g_down
   12 root     -16    -     0K     8K sleep    1:47  0.00% ng_queue
   29 root      -8    -     0K     8K mdwait   1:45  0.00% md0
27610 root      44    0  8464K  4424K select   1:43  0.00% {mpd5}
47631 root      44    0  5912K  3336K bpf      0:58  0.00% tcpdump
33556 root      44    0  3352K  1308K select   0:54  0.00% miniupnpd
    3 root      -8    -     0K     8K -        0:54  0.00% g_up
13826 root      44    0  8464K  4168K select   0:52  0.00% {mpd5}
    7 root     -16    -     0K     8K pftm     0:35  0.00% pfpurge
   11 root     -64    -     0K   128K WAIT     0:34  0.00% {irq14: ata0}
53459 nobody    44    0  5556K  3116K select   0:26  0.00% dnsmasq
 6151 root      76    0 43356K 18728K accept   0:23  0.00% php
39429 root      76    0 43356K 18468K accept   0:23  0.00% php
 9777 root      76    0 43356K 16596K accept   0:21  0.00% php
39767 root      76    0 43356K 17108K accept   0:21  0.00% php

Your tcpdump memory use is far higher than mine but that could be nothing to worry about.

Steve

Edit: Hmm, that tcpdump value is pretty high! Have you used the packet capture tool at all?

memusage.jpg_thumb

cpthk

Thanks for your reply. Today, it just climbed to 25% off 1G memory. Here is my system summary. I am not using any packet capture. Everything in my pfsense is stock, no any extra package or plugin installed. The only features I use in pfsense if NAT port forwarding and dynamic DNS. Does port forwarding require tcpdump?
Could anyone suggest on a feature that could possibly be using tcpdump that I could check if mine is enabled?

last pid: 24302; load averages: 0.00, 0.00, 0.00 up 16+21:42:07 04:03:28
117 processes: 3 running, 99 sleeping, 15 waiting

Mem: 179M Active, 15M Inact, 59M Wired, 1044K Cache, 26M Buf, 711M Free
Swap: 2048M Total, 2048M Free

PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
11 root 171 ki31 0K 32K RUN 0 402.4H 100.00% {idle: cpu0}
11 root 171 ki31 0K 32K CPU1 1 401.5H 100.00% {idle: cpu1}
12 root -68 - 0K 240K WAIT 1 98:03 1.46% {irq16: dc0 uhci0}
0 root -68 0 0K 112K - 0 42:58 0.29% {em0 taskq}
11958 root 44 0 102M 24160K piperd 0 0:02 0.10% php
12 root -32 - 0K 240K WAIT 1 146:21 0.00% {swi4: clock}
24300 root 44 0 14776K 2796K select 1 7:24 0.00% syslogd
14 root -16 - 0K 16K - 1 6:22 0.00% yarrow
21240 root 44 0 111M 104M bpf 0 6:00 0.00% tcpdump
21296 root 44 0 5832K 1040K piperd 0 3:17 0.00% logger
59743 root 76 20 8292K 1700K wait 1 1:39 0.00% sh
31670 root 64 20 5836K 1508K select 0 1:36 0.00% apinger
46248 nobody 44 0 10144K 3460K select 1 1:13 0.00% dnsmasq
40988 dhcpd 44 0 13052K 7276K select 1 0:54 0.00% dhcpd
25 root 44 - 0K 16K syncer 1 0:53 0.00% syncer
0 root 45 0 0K 112K sched 0 0:50 0.00% {swapper}
9 root 44 - 0K 16K pftm 0 0:44 0.00% pfpurge
33972 root 44 0 25768K 5684K kqread 1 0:42 0.00% lighttpd

@stephenw10:

To be honest 20% memory use doesn't seem like a problem to me. Unless it's still climbing.
My own system is usually at around 30% (of 512MB) when idle. See attached RRD graph.
For comparrsion here is my top -SH output:

last pid: 46273;  load averages:  0.28,  0.12,  0.04                                                up 37+20:42:52  13:10:27
108 processes: 2 running, 90 sleeping, 16 waiting
CPU:  0.0% user,  0.0% nice,  0.7% system,  0.0% interrupt, 99.3% idle
Mem: 61M Active, 18M Inact, 62M Wired, 680K Cache, 59M Buf, 343M Free
Swap:

  PID USERNAME PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
   10 root     171 ki31     0K     8K RUN    876.8H 98.00% idle
   11 root     -32    -     0K   128K WAIT   116:14  0.00% {swi4: clock}
   11 root     -68    -     0K   128K WAIT    52:18  0.00% {irq18: em0 ath0+}
12511 root      76   20  3656K  1580K wait    23:31  0.00% sh
    0 root     -68    0     0K    88K -       20:41  0.00% {em0 taskq}
 1807 nobody    74  r30  3368K  1484K nanslp  16:27  0.00% LCDd
   11 root     -68    -     0K   128K WAIT    16:24  0.00% {irq17: fxp2 fxp6}
63939 root      65   20 46428K 17528K nanslp  13:21  0.00% php
   11 root     -44    -     0K   128K WAIT    10:50  0.00% {swi1: netisr 0}
   13 root     -16    -     0K     8K -        6:23  0.00% yarrow
    0 root     -68    0     0K    88K -        4:49  0.00% {ath0 taskq}
29948 root      64   20  3316K  1336K select   4:01  0.00% apinger
    0 root     -68    0     0K    88K -        4:01  0.00% {em2 taskq}
    0 root     -68    0     0K    88K -        3:52  0.00% {em1 taskq}
   20 root      44    -     0K     8K syncer   3:38  0.00% syncer
   35 root      -8    -     0K     8K mdwait   3:23  0.00% md1
    2 root      -8    -     0K     8K -        2:46  0.00% g_event
   11 root     -68    -     0K   128K WAIT     2:27  0.00% {irq19: fxp0 fxp4}
 2392 dhcpd     44    0  8436K  6012K select   2:11  0.00% dhcpd
    0 root     -16    0     0K    88K sched    2:09  0.00% {swapper}
17355 root      44    0  6588K  5060K kqread   2:04  0.00% lighttpd
    4 root      -8    -     0K     8K -        1:47  0.00% g_down
   12 root     -16    -     0K     8K sleep    1:47  0.00% ng_queue
   29 root      -8    -     0K     8K mdwait   1:45  0.00% md0
27610 root      44    0  8464K  4424K select   1:43  0.00% {mpd5}
47631 root      44    0  5912K  3336K bpf      0:58  0.00% tcpdump
33556 root      44    0  3352K  1308K select   0:54  0.00% miniupnpd
    3 root      -8    -     0K     8K -        0:54  0.00% g_up
13826 root      44    0  8464K  4168K select   0:52  0.00% {mpd5}
    7 root     -16    -     0K     8K pftm     0:35  0.00% pfpurge
   11 root     -64    -     0K   128K WAIT     0:34  0.00% {irq14: ata0}
53459 nobody    44    0  5556K  3116K select   0:26  0.00% dnsmasq
 6151 root      76    0 43356K 18728K accept   0:23  0.00% php
39429 root      76    0 43356K 18468K accept   0:23  0.00% php
 9777 root      76    0 43356K 16596K accept   0:21  0.00% php
39767 root      76    0 43356K 17108K accept   0:21  0.00% php

Your tcpdump memory use is far higher than mine but that could be nothing to worry about.

Steve

Edit: Hmm, that tcpdump value is pretty high! Have you used the packet capture tool at all?

cmb

tcpdump is for the firewall logs, assuming it's running on pflog which it almost certainly is. Whether or not increasing memory usage is something to worry about depends on how it's increasing. If it steadily increases forever, that's an issue. If it stays where it is more or less after it's been up a while, which is typical, then that's fine.

cpthk

It just ate up all my memories: 94% memory, 64% SWAP

last pid: 47724; load averages: 0.24, 0.32, 0.34 up 35+00:22:37 06:43:58
114 processes: 3 running, 96 sleeping, 15 waiting

Mem: 856M Active, 4316K Inact, 73M Wired, 30M Cache, 52M Buf, 136K Free
Swap: 2048M Total, 1318M Used, 729M Free, 64% Inuse

PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
11 root 171 ki31 0K 32K CPU1 1 809.9H 98.19% {idle: cpu1}
11 root 171 ki31 0K 32K RUN 0 785.4H 90.58% {idle: cpu0}
19 root 50 - 0K 16K psleep 0 491:41 8.69% pagedaemon
11183 root 65 20 1477M 852M swread 0 59.3H 1.86% tcpdump
3 root -8 - 0K 16K - 0 9:14 0.29% g_up
12 root -68 - 0K 240K WAIT 1 262:35 0.20% {irq16: dc0 uhci0}
3344 root 70 0 104M 8488K piperd 0 0:07 0.10% php
12 root -32 - 0K 240K WAIT 1 307:06 0.00% {swi4: clock}
0 root -68 0 0K 112K - 0 119:43 0.00% {em0 taskq}
24300 root 44 0 14776K 1056K select 1 34:59 0.00% syslogd
14 root 44 - 0K 16K - 1 15:37 0.00% yarrow
11359 root 64 20 5832K 220K piperd 0 7:44 0.00% logger
59743 root 76 20 8292K 324K wait 0 5:08 0.00% sh
29925 root 64 20 5836K 452K select 0 4:14 0.00% apinger
46248 nobody 44 0 10144K 1000K select 1 3:50 0.00% dnsmasq
4 root -8 - 0K 16K - 0 2:28 0.00% g_down
12 root -64 - 0K 240K WAIT 1 2:19 0.00% {irq19: uhci2 uhc}
25 root 44 - 0K 16K syncer 1 2:15 0.00% syncer

stephenw10

Hmm, well that seems clearly wrong.

Are you experiencing any problems?

Steve

cpthk

The network is still fine, but when I login to admin dashboard, it is super slow.
This is obviously tcpdump problem. How do I tell which feature uses tcpdump exactly? I wanted to turn it off.

stephenw10

TBH I don't know. However as Chris said above it's used for firewall logging.
Have you changed the logging settings at all?
Do you have a large number of firewall rules?

Steve

cpthk

No, I only added 2 entries other than the default. Do you know how do I disable logging?

stephenw10

You can disable logging by the default block rule in:
Status: System logs: Settings:
I don't know much difference it will make since logging on other rules will still take place. It's worth a try though since it's easy to do.

Steve

jimp

Disabling logging of the default block doesn't disable that process, but it might lighten its load.

From the console or Diag > command, run: killall -9 tcpdump, then go to System > General and press Save, that should give tcpdump a kick and make it release all the memory it was holding.

I added a bit of code in 2.1 today to restart tcpdump from Status > System Logs on the Settings tab when Save is pressed. It was supposed to be restarted from System > General but it appears as though the code/test to match the tcpdump process may be incorrect ( I fixed that too, but I don't think it was matching properly on 2.0.x ).

stephenw10

Any reason it might climb like that though? I've not seen anyone else complaining about it. Some specific circumstance? :-\

Steve

jimp

I have yet to reproduce it reliably, so I don't know. I've seen it maybe a half dozen times over the years, from 1.2 to 2.x. It's rare, but it does happen.

I don't know if it's due to an especially high/sustained rate of logging or something else that puts it over the edge.