Memory consumption keeps rising until reboot



  • Hello
    When visition RDD grapphs in the PfSense I get some really nice paterns. It is as if memory keeps rising and rising under normal load. I do not know if it is neccesary but every time a user calls in complaining about poor performance I reboot the router and thereby release the used memory. It gives som really nice patterns. I have attached a screendump where router has been rebooted aprox. once a week.

    Is this normal behavior and should I worry that memory is "used up" over a week or so under normal use. It seems like the router still performs just fine with most memory used.

    BR. Anders



  • What version of pfSense are you running?

    What interfaces are in use? (Please post the output of pfSense shell command```
    /etc/rc.banner

    
    @Tillebeck:
    
    > every time a user calls in complaining about poor performance I reboot the router and thereby release the used memory.
    
    That suggests you might have a memory leak - some kernel component s allocating "dynamic" memory but not releasing the memory when the component has finished using it. Network performance suffers when dynamic memory is nearly exhausted.
    
    It would probably be a good idea to monitor mbuf usage (say at start and end of a day for a couple of weeks) to look for trends. mbuf usage will generally climb soon after startup and then level out. If it keeps climbing significantly after a few days uptime you probably have a problem. pfSense shell command```
    # netstat -m
    ```will show mbuf usage. Here is an example from my system (256MB RAM):
    
    > $ netstat -m
    > 173/472/645 mbufs in use (current/cache/total)
    > 171/219/390/8000 mbuf clusters in use (current/cache/total/max)
    > 170/214 mbuf+clusters out of packet secondary zone in use (current/cache)
    > 0/20/20/4000 4k (page size) jumbo clusters in use (current/cache/total/max)
    > 0/0/0/2000 9k jumbo clusters in use (current/cache/total/max)
    > 0/0/0/1000 16k jumbo clusters in use (current/cache/total/max)
    > 385K/636K/1021K bytes allocated to network (current/cache/total)
    > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
    > 0/0/0 requests for jumbo clusters denied (4k/9k/16k)
    > 0/5/2256 sfbufs in use (current/peak/max)
    > 0 requests for sfbufs denied
    > 0 requests for sfbufs delayed
    > 0 requests for I/O initiated by sendfile
    > 0 calls to protocol drain routines
    > $
    
    Do the performance complaints seem correspond to any particular activity? (Maybe you need to tweak your system for a regular peak load.)
    
    What is the nature of the performance complaints? (slow? connection attempts fail? …)

  • Rebel Alliance Developer Netgate

    It would also help to see the full output of "ps uxawww" and "top -SH"



  • Thanks

    Complains are:
    Slow speeds, timeout of DNS (that might not be a router issue), problems getting an DHCP IP from router.

    $ netstat -m
    405/795/1200 mbufs in use (current/cache/total)
    392/646/1038/8640 mbuf clusters in use (current/cache/total/max)
    391/633 mbuf+clusters out of packet secondary zone in use (current/cache)
    0/20/20/4320 4k (page size) jumbo clusters in use (current/cache/total/max)
    0/0/0/2160 9k jumbo clusters in use (current/cache/total/max)
    0/0/0/1080 16k jumbo clusters in use (current/cache/total/max)
    885K/1570K/2456K bytes allocated to network (current/cache/total)
    0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
    0/0/0 requests for jumbo clusters denied (4k/9k/16k)
    0/4/2416 sfbufs in use (current/peak/max)
    0 requests for sfbufs denied
    0 requests for sfbufs delayed
    0 requests for I/O initiated by sendfile
    0 calls to protocol drain routines

    /etc/rc.banner
    This one stopped the router from responding over the web. It is stille routing and I can access other equipment set up with NAT through the router. But the webinterface did not come up again. Right now I have been waiting aprox. 15minuttes and the webinterface is still not up. It does not timeout. Just keeps loading and loading. So I will have to come back with the results from the the other commands later.



  • Hi!

    I have the same problems with pfsense 2.0.1 64bit on intel hw.
    The memory consumption is rising until reboot.
    We noticed that the tcpdump process is consuming the memory over time!

    /usr/sbin/tcpdump -n -e -ttt -i pflog0

    Any suggestions?


  • Netgate Administrator

    @Tillebeck:

    /etc/rc.banner
    This one stopped the router from responding over the web. It is still routing and I can access other equipment set up with NAT through the router.

    Did you run that from the CLI or via the webGUI: Diagnostics: Command Prompt: ?

    The GUI Command Prompt function will fail if you try to run something that doesn't finish. It will result in a stuck php process that locks up the web gui. I have no idea how it would handle rc.banner. Like it says 'this function is unsupported. Use it on your own risk!'.  ;)
    You can escape from that situation by killing php from the CLI.

    Steve



  • It could be a stuck tcpdump. Under normal conditions no tcpdump is running?


  • Netgate Administrator

    tcpdump is used for logging and should be running normally. If you search the forum there have been a few instances of tcpdump using excess memory lately though. I would guess that's your problem.

    Steve



  • When I look at Status: System logs: Firewall a tcpdump process is started for logging?



  • Looks like a tcpdump problem?

    tcpdump version 4.0.0
    libpcap version 1.0.0


  • Netgate Administrator


Locked