running out of memory and swap



  • Hi, i have a fairly new pfsense build that is running out of RAM and swap. The web console becomes unresponsive, as do ssh sessions.

    I've noticed on some reboots I get the "out of swap" message during boot, but usually it takes a short while to show up. I've disabled Squid and squidguard to no avail. I've also turned off snort on the WAN port, again without fix. Here's the output of my top:

    last pid: 59890;  load averages:  0.35,  0.63,  0.68                                                                                                                                   up 0+01:04:33  11:27:57
    98 processes:  2 running, 86 sleeping, 10 waiting
    CPU:  0.0% user,  0.0% nice,  9.6% system,  0.4% interrupt, 90.0% idle
    Mem: 6048M Active, 24K Inact, 767M Wired, 615M Buf, 38M Free
    Swap: 410M Total, 410M Used, K Free, 100% Inuse, 152K In
    
      PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
    60448 root          1  20    0 12400K 12504K select  0   0:01   0.01% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid
    80422 root          2  20    0   105M 12212K bpf     0   0:07   0.30% /usr/local/bin/snort -R 30980 -D -q --suppress-config-log -l /var/log/snort/snort_em130980 --pid-path /var/run --nolock-pidfile
    88211 root         12  20    0 52836K  9556K pfault  0   0:10   0.36% /usr/local/bin/telegraf -config=/usr/local/etc/telegraf.conf
      311 root          1  20    0 94296K  6716K kqread  0   0:00   0.01% php-fpm: master process (/usr/local/lib/php-fpm.conf) (php-fpm)
    28884 root          1  20    0 51312K  6148K piperd  1   0:00   0.00% /usr/local/bin/php -f /usr/local/pkg/pfblockerng/pfblockerng.inc dnsbl
    26609 root          1  20    0 51376K  5808K piperd  0   0:00   0.00% /usr/local/bin/php_pfb -f /usr/local/pkg/pfblockerng/pfblockerng.inc filterlog
    25396 root          1  20    0 51376K  5800K piperd  1   0:00   0.00% /usr/local/bin/php_pfb -f /usr/local/pkg/pfblockerng/pfblockerng.inc filterlog
    22984 root          1  20    0 12908K  5004K select  0   0:01   0.02% sshd: root@pts/0 (sshd)
    28596 root          1  20    0 10440K  4912K kqread  1   0:00   0.01% /usr/local/sbin/lighttpd_pfb -f /var/unbound/pfb_dnsbl_lighty.conf
    28946 root          1  52    0 51312K  4644K piperd  0   0:00   0.00% /usr/local/bin/php -f /usr/local/pkg/pfblockerng/pfblockerng.inc index
    96242 root          1  20    0  9468K  4296K select  0   0:01   0.00% /usr/local/sbin/miniupnpd -f /var/etc/miniupnpd.conf -P /var/run/miniupnpd.pid
     7464 root          1  20    0 12616K  3652K select  0   0:00   0.00% /usr/sbin/sshd
    30698 root          1  20    0  7816K  2876K CPU1    1   0:02   0.06% top -aS
    61506 root          1  20    0  7816K  2600K select  1   0:02   0.05% top -aS
    66174 dhcpd         1  20    0 12580K  2228K select  0   0:01   0.02% /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid em1
    58033 root          1  20    0  6748K  2084K pfault  1   0:00   0.00% /usr/bin/netstat -ibdnW (<netstat>)
    66520 root          1  20    0  6268K  2000K select  0   0:01   0.00% /usr/local/sbin/radvd -p /var/run/radvd.pid -C /var/etc/radvd.conf -m syslog
    26352 root          1  20    0  4644K  2000K select  0   0:04   0.11% /usr/local/sbin/clog_pfb -f /var/log/filter.log
    25266 root          1  20    0  4644K  2000K select  0   0:04   0.09% /usr/local/sbin/clog_pfb -f /var/log/filter.log
    16152 root          1  20    0  6396K  1952K select  1   0:00   0.00% /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /etc/syslog.conf
     9276 root          2  26    0  6528K  1932K piperd  0   0:00   0.00% /usr/local/libexec/sshg-blocker
     9224 root          1  20    0 11912K  1928K piperd  1   0:00   0.00% /usr/local/libexec/sshg-parser
    17076 root          1  20    0  6604K  1904K bpf     0   0:01   0.01% /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid
    
    

    Any help appreciated.


  • Netgate Administrator

    Something there is out of control. That's running in 8GB? It should never be using swap.

    It's probably Snort but could be pfBlocker if you have a very large number of lists loaded.

    Steve



  • Thanks for replying- I don't think I have enabled very many lists at all. Is there a way for me to dump out the list to show that it is/isn't large?

    Between one of the "crashes" I disabled snort on the WAN interface and it didn't make a difference. I'm the only person home, so there also isn't much inbound or outbound network activity. Was there a reason that telegraph was listed so high on the top output that would be cause for concern?



  • it hasn't crashed yet this time... i've disabled telegraph this time, but it's already used a bit of swap:

    last pid: 46024;  load averages:  0.29,  0.24,  0.20                                                                                                                                   up 0+00:36:15  12:52:16
    90 processes:  2 running, 87 sleeping, 1 waiting
    CPU:  0.0% user,  0.0% nice,  0.2% system,  0.2% interrupt, 99.7% idle
    Mem: 2332M Active, 3474M Inact, 472M Laundry, 391M Wired, 234M Buf, 183M Free
    Swap: 410M Total, 11M Used, 399M Free, 2% Inuse
    
      PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
    54046 unbound       2  20    0 11154M   289M kqread  1   1:38   0.00% /usr/local/sbin/unbound -c /var/unbound/unbound.conf
    50190 root          2  20    0   105M 61396K bpf     0   0:01   0.02% /usr/local/bin/snort -R 21545 -D -q --suppress-config-log -l /var/log/snort/snort_em021545 --pid-path /var/run --nolock-pidfile
    49646 root          2  20    0   105M 61212K bpf     1   0:01   0.03% /usr/local/bin/snort -R 30980 -D -q --suppress-config-log -l /var/log/snort/snort_em130980 --pid-path /var/run --nolock-pidfile
    91204 root          1  20    0 51312K 28148K piperd  0   0:00   0.00% /usr/local/bin/php -f /usr/local/pkg/pfblockerng/pfblockerng.inc dnsbl
    89983 root          1  20    0 51376K 27028K piperd  1   0:00   0.00% /usr/local/bin/php_pfb -f /usr/local/pkg/pfblockerng/pfblockerng.inc filterlog
    88685 root          1  20    0 51376K 27028K piperd  0   0:00   0.00% /usr/local/bin/php_pfb -f /usr/local/pkg/pfblockerng/pfblockerng.inc filterlog
    91401 root          1  52    0 51312K 26948K piperd  0   0:00   0.00% /usr/local/bin/php -f /usr/local/pkg/pfblockerng/pfblockerng.inc index
      337 root          1  20    0 94296K 18328K kqread  0   0:00   0.01% php-fpm: master process (/usr/local/lib/php-fpm.conf) (php-fpm)
    58764 root          1  20    0 12400K 12504K select  1   0:00   0.01% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid
    68100 root          1  20    0 12908K  7248K select  1   0:00   0.00% sshd: root@pts/0 (sshd)
    66197 dhcpd         1  20    0 12580K  6816K select  0   0:00   0.02% /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid em1
    91190 root          1  20    0 10440K  6748K kqread  1   0:00   0.00% /usr/local/sbin/lighttpd_pfb -f /var/unbound/pfb_dnsbl_lighty.conf
     7577 root          1  20    0 12616K  6020K select  1   0:00   0.00% /usr/sbin/sshd
    95474 root          1  20    0  9468K  5312K select  1   0:00   0.04% /usr/local/sbin/miniupnpd -f /var/etc/miniupnpd.conf -P /var/run/miniupnpd.pid
      415 root          1  20    0  9188K  4440K select  0   0:00   0.00% /sbin/devd -q -f /etc/pfSense-devd.conf
    58090 root          1  20    0 23596K  4004K kqread  0   0:00   0.00% nginx: worker process (nginx)
    26429 root          1  20    0  7816K  3512K CPU0    0   0:01   0.02% top -aS
    91662 root          1  20    0 51312K  3312K nanslp  0   0:00   0.00% /usr/local/bin/php -f /usr/local/pkg/pfblockerng/pfblockerng.inc queries
     7893 root          1  20    0 11912K  2712K piperd  0   0:00   0.00% /usr/local/libexec/sshg-parser
     8953 root          1  52    0  6968K  2592K ttyin   0   0:00   0.00% /bin/sh /etc/rc.initial
    87005 root          1  52   20  6968K  2532K wait    0   0:00   0.00% /bin/sh /var/db/rrd/updaterrd.sh
    16125 root          1  20    0  6396K  2524K select  1   0:00   0.01% /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /etc/syslog.conf
     8458 root          1  33    0  6968K  2500K piperd  0   0:00   0.00% /bin/sh /usr/local/libexec/sshg-fw-pf
     7906 root          2  24    0  6528K  2468K piperd  0   0:00   0.00% /usr/local/libexec/sshg-blocker
    88541 root          1  20    0  4644K  2452K select  1   0:02   0.11% /usr/local/sbin/clog_pfb -f /var/log/filter.log
    
    

  • Netgate Administrator

    I would just disable all the packages as a test. Snort, pfBlocker and Squid can all use a lot of RAM but 8GB should be sufficient to prevent it swapping with reasonable list sizes.
    Telegraf really has no reason to use a lot of RAM so if it is it's a problem.

    Steve


Log in to reply