pfSense memory usage



  • We do use a pfSense stack with 2 members, running both on 2.4.4-RELEASE-p3, virtual, running on ESX 6.something. This setup uses CARP to replicate/provide failover. Both are running fine but the backup/secondary node shows much higher memory usage than the primary member. We do not have any issues in performance or availability at this moment but I am wondering what causes this usage. If I restart the node, memory usage will be lower but slowly growing up to about 60% and higher after about 20-25 days.

    dashboard
    stats
    system_activity

    Both nodes are running the same release, have been upgraded from 2.2.something and have the same services/packages and hardware specs.

    2.4.4-RELEASE-p3 (amd64)
    built on Wed May 15 18:53:44 EDT 2019
    FreeBSD 11.2-RELEASE-p10

    List of installed packages:

    • cron
    • haproxy
    • nmap
    • open-vm-tools
    • openvpn-client-export
    • sudo

    Current states:
    State Table Total Rate
    current entries 797
    searches 260031939 163.6/s

    Aliases: about 30 (groups)
    Firewall rules: 5 zones, about 80 rules in total

    Found a lot of related info but none with a solution...
    https://forum.netgate.com/topic/130622/is-high-memory-usage-normal
    https://forum.netgate.com/topic/50032/high-memory-usage/9
    https://forum.netgate.com/topic/61420/memory-leak
    https://forum.netgate.com/topic/4667/possible-memory-leak/5
    https://www.reddit.com/r/PFSENSE/comments/bg1ogg/wired_memory_slowly_creeping_up/
    https://redmine.pfsense.org/issues/8249
    https://forum.netgate.com/topic/47513/memory-usage-climbing/9
    https://redmine.pfsense.org/issues/2819
    https://forum.netgate.com/topic/130396/wire-memory-slowly-increasing/10

    Can someone help me how to find the cause, we do use snmp monitoring, what brought me to this question. The memload does not get lower when I restart the (running) services, FPM-service or webconfigurator.


  • Netgate Administrator

    Does it just continue to grow if you don't restart the node?

    And the primary does not show that?

    Try running top at the command line instead of using Diag > System Activity. Then sort by size instead of cpu usage.
    Compare that output on both nodes.

    Steve



  • Hi, thanks for your reply. Current status on dashboard is "62% of 2000 MiB" so it is slightly growing. Primary node doesn't.
    as requested, top -aSH , sorted by size:

    ===Backup appliance

    last pid:  2448;  load averages:  0.32,  0.28,  0.26                                                                                                                          up 19+10:23:42  15:39:42
    204 processes: 3 running, 154 sleeping, 47 waiting
    CPU:  0.2% user,  0.0% nice,  0.0% system,  0.0% interrupt, 99.8% idle
    Mem: 27M Active, 433M Inact, 1170M Wired, 198M Buf, 317M Free
    Swap: 4096M Total, 4096M Free
    
      PID USERNAME      PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
    34541 root           20    0 99820K 39164K accept  1   0:21   0.00% php-fpm: pool nginx (php-fpm){php-fpm}
    34864 root           52    0 99820K 39128K accept  0   0:21   0.00% php-fpm: pool nginx (php-fpm){php-fpm}
    47659 root           20    0 97772K 38560K accept  0   0:21   0.00% php-fpm: pool nginx (php-fpm){php-fpm}
    74679 root           52    0 97772K 38288K accept  0   0:00   0.00% php-fpm: pool nginx (php-fpm){php-fpm}
    30427 root           20    0 88384K 28712K kqread  1   0:03   0.00% php-fpm: master process (/usr/local/lib/php-fpm.conf) (php-fpm)
    37340 root           20    0 67592K 61624K select  1   4:05   0.00% /usr/sbin/bsnmpd -c /var/etc/snmpd.conf -p /var/run/snmpd.pid
    52219 root           20    0 48060K 43496K select  0  14:14   0.04% /usr/local/bin/vmtoolsd -c /usr/local/share/vmware-tools/tools.conf -p /usr/local/lib/open-vm-tools/plugins/vmsvc
    67851 unbound        20    0 38172K 18752K kqread  1   0:00   0.00% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}
    67851 unbound        20    0 38172K 18752K kqread  1   0:00   0.00% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}
    43029 root           20    0 37904K 18280K select  0   0:01   0.00% /usr/local/libexec/ipsec/charon --use-syslog{charon}
    43029 root           20    0 37904K 18280K uwait   0   0:00   0.00% /usr/local/libexec/ipsec/charon --use-syslog{charon}
    43029 root           20    0 37904K 18280K uwait   1   0:00   0.00% /usr/local/libexec/ipsec/charon --use-syslog{charon}
    43029 root           20    0 37904K 18280K uwait   1   0:00   0.00% /usr/local/libexec/ipsec/charon --use-syslog{charon}
    43029 root           20    0 37904K 18280K uwait   0   0:00   0.00% /usr/local/libexec/ipsec/charon --use-syslog{charon}
    43029 root           20    0 37904K 18280K uwait   0   0:00   0.00% /usr/local/libexec/ipsec/charon --use-syslog{charon}
    43029 root           20    0 37904K 18280K uwait   0   0:00   0.00% /usr/local/libexec/ipsec/charon --use-syslog{charon}
    43029 root           20    0 37904K 18280K uwait   1   0:00   0.00% /usr/local/libexec/ipsec/charon --use-syslog{charon}
    43029 root           20    0 37904K 18280K uwait   1   0:00   0.00% /usr/local/libexec/ipsec/charon --use-syslog{charon}
    43029 root           20    0 37904K 18280K uwait   1   0:00   0.00% /usr/local/libexec/ipsec/charon --use-syslog{charon}
    43029 root           20    0 37904K 18280K uwait   0   0:00   0.00% /usr/local/libexec/ipsec/charon --use-syslog{charon}
    43029 root           20    0 37904K 18280K uwait   0   0:00   0.00% /usr/local/libexec/ipsec/charon --use-syslog{charon}
    43029 root           20    0 37904K 18280K uwait   1   0:00   0.00% /usr/local/libexec/ipsec/charon --use-syslog{charon}
    43029 root           31    0 37904K 18280K sigwai  0   0:00   0.00% /usr/local/libexec/ipsec/charon --use-syslog{charon}
    43029 root           31    0 37904K 18280K uwait   1   0:00   0.00% /usr/local/libexec/ipsec/charon --use-syslog{charon}
    43029 root           31    0 37904K 18280K select  1   0:00   0.00% /usr/local/libexec/ipsec/charon --use-syslog{charon}
    43029 root           31    0 37904K 18280K uwait   0   0:00   0.00% /usr/local/libexec/ipsec/charon --use-syslog{charon}
    94154 root           20    0 23596K 10304K kqread  1   0:10   0.00% nginx: worker process (nginx)
    93939 root           20    0 23596K  9620K kqread  0   0:06   0.00% nginx: worker process (nginx)
    93731 root           52    0 21548K  7752K pause   1   0:00   0.00% nginx: master process /usr/local/sbin/nginx -c /var/etc/nginx-webConfigurator.conf (nginx)
    27209 root           20    0 12908K  9488K select  0   0:00   0.02% sshd: admin@pts/2 (sshd)
    14001 root           20    0 12616K  8824K select  1   0:00   0.00% /usr/sbin/sshd
    17482 root           20    0 12400K 12504K select  0   0:07   0.01% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid{ntpd}
    86276 root           20    0 11912K  7772K piperd  0   0:00   0.00% /usr/local/libexec/sshg-parser
    26928 root           20    0 10216K  6516K select  1   0:10   0.00% /usr/local/sbin/openvpn --config /var/et
    

    ===

    ===Master appliance

    last pid: 69934;  load averages:  0.10,  0.14,  0.15                                                                                                                          up 19+10:13:34  15:40:24
    210 processes: 3 running, 160 sleeping, 47 waiting
    CPU:  0.2% user,  0.0% nice,  0.2% system,  0.2% interrupt, 99.4% idle
    Mem: 72M Active, 420M Inact, 373M Wired, 198M Buf, 1082M Free
    Swap: 4096M Total, 4096M Free
    
      PID USERNAME      PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
    13320 root           52    0 99820K 40504K accept  0   0:09   0.00% php-fpm: pool nginx (php-fpm){php-fpm}
    72977 root           52    0 99820K 40488K accept  0   0:10   0.00% php-fpm: pool nginx (php-fpm){php-fpm}
     5805 root           52    0 99820K 40368K accept  0   0:04   0.00% php-fpm: pool nginx (php-fpm){php-fpm}
    72353 root           20    0 97772K 39788K accept  0   0:07   0.00% php-fpm: pool nginx (php-fpm){php-fpm}
    63653 root           24    0 97772K 39604K accept  0   0:06   0.00% php-fpm: pool nginx (php-fpm){php-fpm}
      709 root           20    0 88384K 26292K kqread  0   0:41   0.00% php-fpm: master process (/usr/local/lib/php-fpm.conf) (php-fpm)
    44910 root           20    0 67592K 62272K select  1  33:51   0.01% /usr/sbin/bsnmpd -c /var/etc/snmpd.conf -p /var/run/snmpd.pid
    63513 root           20    0 48060K 43420K select  0  14:29   0.04% /usr/local/bin/vmtoolsd -c /usr/local/share/vmware-tools/tools.conf -p /usr/local/lib/open-vm-tools/plugins/vmsvc
     4081 unbound        20    0 38172K 18780K kqread  1   0:01   0.00% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}
     4081 unbound        20    0 38172K 18780K kqread  1   0:01   0.00% /usr/local/sbin/unbound -c /var/unbound/unbound.conf{unbound}
    42282 root           20    0 37904K 21536K uwait   1   0:07   0.01% /usr/local/libexec/ipsec/charon --use-syslog{charon}
    42282 root           20    0 37904K 21536K select  1   0:10   0.00% /usr/local/libexec/ipsec/charon --use-syslog{charon}
    42282 root           20    0 37904K 21536K uwait   0   0:07   0.00% /usr/local/libexec/ipsec/charon --use-syslog{charon}
    42282 root           20    0 37904K 21536K uwait   1   0:23   0.00% /usr/local/libexec/ipsec/charon --use-syslog{charon}
    42282 root           20    0 37904K 21536K uwait   1   0:07   0.00% /usr/local/libexec/ipsec/charon --use-syslog{charon}
    42282 root           20    0 37904K 21536K uwait   1   0:07   0.00% /usr/local/libexec/ipsec/charon --use-syslog{charon}
    42282 root           20    0 37904K 21536K uwait   1   0:07   0.00% /usr/local/libexec/ipsec/charon --use-syslog{charon}
    42282 root           20    0 37904K 21536K uwait   0   0:07   0.00% /usr/local/libexec/ipsec/charon --use-syslog{charon}
    42282 root           20    0 37904K 21536K uwait   1   0:07   0.00% /usr/local/libexec/ipsec/charon --use-syslog{charon}
    42282 root           20    0 37904K 21536K uwait   0   0:07   0.00% /usr/local/libexec/ipsec/charon --use-syslog{charon}
    42282 root           20    0 37904K 21536K uwait   1   0:07   0.00% /usr/local/libexec/ipsec/charon --use-syslog{charon}
    42282 root           20    0 37904K 21536K uwait   1   0:07   0.00% /usr/local/libexec/ipsec/charon --use-syslog{charon}
    42282 root           20    0 37904K 21536K uwait   1   0:07   0.00% /usr/local/libexec/ipsec/charon --use-syslog{charon}
    42282 root           20    0 37904K 21536K uwait   0   0:07   0.00% /usr/local/libexec/ipsec/charon --use-syslog{charon}
    42282 root           20    0 37904K 21536K uwait   0   0:07   0.00% /usr/local/libexec/ipsec/charon --use-syslog{charon}
    42282 root           47    0 37904K 21536K sigwai  1   0:02   0.00% /usr/local/libexec/ipsec/charon --use-syslog{charon}
    42282 root           20    0 37904K 21536K select  1   0:01   0.00% /usr/local/libexec/ipsec/charon --use-syslog{charon}
    45159 www            20    0 30436K 21912K kqread  1  78:33   0.61% /usr/local/sbin/haproxy -f /var/etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -D -sf 15930
     3197 root           20    0 23596K  9456K kqread  0   0:11   0.00% nginx: worker process (nginx)
     2977 root           20    0 23596K  9428K kqread  0   0:04   0.00% nginx: worker process (nginx)
     2816 root           52    0 21548K  7692K pause   1   0:00   0.00% nginx: master process /usr/local/sbin/nginx -c /var/etc/nginx-webConfigurator.conf (nginx)
     9925 root           20    0 12908K  9456K select  1   0:00   0.01% sshd: admin@pts/0 (sshd)
    14363 root           20    0 12616K  8804K select  0   0:14   0.00% /usr/sbin/sshd
     7964 root           20    0 12400K 12504K select  0   1:47   0.00% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid{ntpd}
    98280 root           20    0 11912K  7772K piperd  0   0:04   0.00% /usr/local/libexec/sshg-parser
    

    ===


  • Netgate Administrator

    Hmm, well it's all wired usage by the looks of it. Nothing looks dramatically incorrect there really.
    I would want to see if that continues to climb if you don't reboot it.

    Steve



  • Memory usage is still growing, today's update:

    ===
    Uptime 28 Days 05 Hours 55 Minutes 19 Seconds
    Memory usage 81% of 2000 MiB

    203 processes: 3 running, 153 sleeping, 47 waiting
    
    Mem: 12M Active, 307M Inact, 1558M Wired, 198M Buf, 71M Free
    Swap: 4096M Total, 4096M Free
    

    ===

    Still no issues found besides our monitoring check that complains about memory usage :-)



  • Another week has passed, host is using 98% of 2000MiB ram now, still alive and nothing useful is being logged, it has started to use swap as well, currently using 6%. Console (ssh) responds a bit slow but still operates.

    I am a bit desperate for the next step, wait for release 2.5, another 2.4.4 patch, reinstall this host, reboot it once a month...

    Primary node is using only 22% of its ram, same uptime.



  • System ran out of swap last night and killed all processes:

    pid 52219 (vmtoolsd), uid 0, was killed: out of swap space
    pid 37340 (bsnmpd), uid 0, was killed: out of swap space
    pid 99704 (netstat), uid 0, was killed: out of swap space
    pid 16857 (php-fpm), uid 0, was killed: out of swap space
    pid 93238 (php-fpm), uid 0, was killed: out of swap space
    pid 55684 (php-fpm), uid 0, was killed: out of swap space
    pid 67420 (php-fpm), uid 0, was killed: out of swap space
    pid 14021 (php-fpm), uid 0, was killed: out of swap space
    pid 4367 (php-fpm), uid 0, was killed: out of swap space
    pid 57915 (php-fpm), uid 0, was killed: out of swap space
    pid 82469 (php-cgi), uid 0, was killed: out of swap space

    WebGui died, unable to start it again, console / SSH was available but very slow on response.
    I had to reboot the VM to get it online again. Memory usage @14% after reboot.



  • @marcvw said in pfSense memory usage:

    /usr/local/libexec/ipsec/charon

    I compared your (visible) processes with mine.
    True, something is eating memory. Up to you to find out what process is doing so.

    I'm not using this : /usr/local/libexec/ipsec/charon : can you 'kill' ipsec for some time ?

    I do not use haproxy.

    Btw : I'm using Munin because memory that should be tracked of time - among many other things.



  • Thanks, I have stopped the ipsec service (which killed the processes), we need this because we use ipsec/vpn but to keep track on this issue I can stop it for a while. Despite it has been stopped for some time now the usage of wired mem still grows but I don't see a big difference in- or decrease in usage. I will re-check the values on Monday.



  • Rechecked the memory usage growth, same as before, enabled the ipsec service again. I will continue monitor and troubleshoot this appliance...


  • Netgate Administrator

    So disabling IPSec had no significant effect on the memory usage growth?


Log in to reply