Memory consumption keeps rising until reboot

[Tillebeck]

Hello
When visition RDD grapphs in the PfSense I get some really nice paterns. It is as if memory keeps rising and rising under normal load. I do not know if it is neccesary but every time a user calls in complaining about poor performance I reboot the router and thereby release the used memory. It gives som really nice patterns. I have attached a screendump where router has been rebooted aprox. once a week.

Is this normal behavior and should I worry that memory is "used up" over a week or so under normal use. It seems like the router still performs just fine with most memory used.

BR. Anders

[wallabybob]

What version of pfSense are you running?

What interfaces are in use? (Please post the output of pfSense shell command```
/etc/rc.banner

@Tillebeck:

> every time a user calls in complaining about poor performance I reboot the router and thereby release the used memory.

That suggests you might have a memory leak - some kernel component s allocating "dynamic" memory but not releasing the memory when the component has finished using it. Network performance suffers when dynamic memory is nearly exhausted.

It would probably be a good idea to monitor mbuf usage (say at start and end of a day for a couple of weeks) to look for trends. mbuf usage will generally climb soon after startup and then level out. If it keeps climbing significantly after a few days uptime you probably have a problem. pfSense shell command```
# netstat -m
```will show mbuf usage. Here is an example from my system (256MB RAM):

> $ netstat -m
> 173/472/645 mbufs in use (current/cache/total)
> 171/219/390/8000 mbuf clusters in use (current/cache/total/max)
> 170/214 mbuf+clusters out of packet secondary zone in use (current/cache)
> 0/20/20/4000 4k (page size) jumbo clusters in use (current/cache/total/max)
> 0/0/0/2000 9k jumbo clusters in use (current/cache/total/max)
> 0/0/0/1000 16k jumbo clusters in use (current/cache/total/max)
> 385K/636K/1021K bytes allocated to network (current/cache/total)
> 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
> 0/0/0 requests for jumbo clusters denied (4k/9k/16k)
> 0/5/2256 sfbufs in use (current/peak/max)
> 0 requests for sfbufs denied
> 0 requests for sfbufs delayed
> 0 requests for I/O initiated by sendfile
> 0 calls to protocol drain routines
> $

Do the performance complaints seem correspond to any particular activity? (Maybe you need to tweak your system for a regular peak load.)

What is the nature of the performance complaints? (slow? connection attempts fail? …)

[jimp]

It would also help to see the full output of "ps uxawww" and "top -SH"

[Tillebeck]

Thanks

Complains are:
Slow speeds, timeout of DNS (that might not be a router issue), problems getting an DHCP IP from router.

$ netstat -m
405/795/1200 mbufs in use (current/cache/total)
392/646/1038/8640 mbuf clusters in use (current/cache/total/max)
391/633 mbuf+clusters out of packet secondary zone in use (current/cache)
0/20/20/4320 4k (page size) jumbo clusters in use (current/cache/total/max)
0/0/0/2160 9k jumbo clusters in use (current/cache/total/max)
0/0/0/1080 16k jumbo clusters in use (current/cache/total/max)
885K/1570K/2456K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/4/2416 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines

/etc/rc.banner
This one stopped the router from responding over the web. It is stille routing and I can access other equipment set up with NAT through the router. But the webinterface did not come up again. Right now I have been waiting aprox. 15minuttes and the webinterface is still not up. It does not timeout. Just keeps loading and loading. So I will have to come back with the results from the the other commands later.

[rplattn]

Hi!

I have the same problems with pfsense 2.0.1 64bit on intel hw.
The memory consumption is rising until reboot.
We noticed that the tcpdump process is consuming the memory over time!

/usr/sbin/tcpdump -n -e -ttt -i pflog0

Any suggestions?

[stephenw10]

@Tillebeck:

/etc/rc.banner
This one stopped the router from responding over the web. It is still routing and I can access other equipment set up with NAT through the router.

Did you run that from the CLI or via the webGUI: Diagnostics: Command Prompt: ?

The GUI Command Prompt function will fail if you try to run something that doesn't finish. It will result in a stuck php process that locks up the web gui. I have no idea how it would handle rc.banner. Like it says 'this function is unsupported. Use it on your own risk!'.  ;)
You can escape from that situation by killing php from the CLI.

Steve

[rplattn]

It could be a stuck tcpdump. Under normal conditions no tcpdump is running?

[stephenw10]

tcpdump is used for logging and should be running normally. If you search the forum there have been a few instances of tcpdump using excess memory lately though. I would guess that's your problem.

Steve

[rplattn]

When I look at Status: System logs: Firewall a tcpdump process is started for logging?

[rplattn]

Looks like a tcpdump problem?

tcpdump version 4.0.0
libpcap version 1.0.0

[stephenw10]

See this post/thread: http://forum.pfsense.org/index.php/topic,55441.msg296357.html#msg296357

Steve