running out of memory and swap



  • Hi, i have a fairly new pfsense build that is running out of RAM and swap. The web console becomes unresponsive, as do ssh sessions.

    I've noticed on some reboots I get the "out of swap" message during boot, but usually it takes a short while to show up. I've disabled Squid and squidguard to no avail. I've also turned off snort on the WAN port, again without fix. Here's the output of my top:

    last pid: 59890;  load averages:  0.35,  0.63,  0.68                                                                                                                                   up 0+01:04:33  11:27:57
    98 processes:  2 running, 86 sleeping, 10 waiting
    CPU:  0.0% user,  0.0% nice,  9.6% system,  0.4% interrupt, 90.0% idle
    Mem: 6048M Active, 24K Inact, 767M Wired, 615M Buf, 38M Free
    Swap: 410M Total, 410M Used, K Free, 100% Inuse, 152K In
    
      PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
    60448 root          1  20    0 12400K 12504K select  0   0:01   0.01% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid
    80422 root          2  20    0   105M 12212K bpf     0   0:07   0.30% /usr/local/bin/snort -R 30980 -D -q --suppress-config-log -l /var/log/snort/snort_em130980 --pid-path /var/run --nolock-pidfile
    88211 root         12  20    0 52836K  9556K pfault  0   0:10   0.36% /usr/local/bin/telegraf -config=/usr/local/etc/telegraf.conf
      311 root          1  20    0 94296K  6716K kqread  0   0:00   0.01% php-fpm: master process (/usr/local/lib/php-fpm.conf) (php-fpm)
    28884 root          1  20    0 51312K  6148K piperd  1   0:00   0.00% /usr/local/bin/php -f /usr/local/pkg/pfblockerng/pfblockerng.inc dnsbl
    26609 root          1  20    0 51376K  5808K piperd  0   0:00   0.00% /usr/local/bin/php_pfb -f /usr/local/pkg/pfblockerng/pfblockerng.inc filterlog
    25396 root          1  20    0 51376K  5800K piperd  1   0:00   0.00% /usr/local/bin/php_pfb -f /usr/local/pkg/pfblockerng/pfblockerng.inc filterlog
    22984 root          1  20    0 12908K  5004K select  0   0:01   0.02% sshd: root@pts/0 (sshd)
    28596 root          1  20    0 10440K  4912K kqread  1   0:00   0.01% /usr/local/sbin/lighttpd_pfb -f /var/unbound/pfb_dnsbl_lighty.conf
    28946 root          1  52    0 51312K  4644K piperd  0   0:00   0.00% /usr/local/bin/php -f /usr/local/pkg/pfblockerng/pfblockerng.inc index
    96242 root          1  20    0  9468K  4296K select  0   0:01   0.00% /usr/local/sbin/miniupnpd -f /var/etc/miniupnpd.conf -P /var/run/miniupnpd.pid
     7464 root          1  20    0 12616K  3652K select  0   0:00   0.00% /usr/sbin/sshd
    30698 root          1  20    0  7816K  2876K CPU1    1   0:02   0.06% top -aS
    61506 root          1  20    0  7816K  2600K select  1   0:02   0.05% top -aS
    66174 dhcpd         1  20    0 12580K  2228K select  0   0:01   0.02% /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid em1
    58033 root          1  20    0  6748K  2084K pfault  1   0:00   0.00% /usr/bin/netstat -ibdnW (<netstat>)
    66520 root          1  20    0  6268K  2000K select  0   0:01   0.00% /usr/local/sbin/radvd -p /var/run/radvd.pid -C /var/etc/radvd.conf -m syslog
    26352 root          1  20    0  4644K  2000K select  0   0:04   0.11% /usr/local/sbin/clog_pfb -f /var/log/filter.log
    25266 root          1  20    0  4644K  2000K select  0   0:04   0.09% /usr/local/sbin/clog_pfb -f /var/log/filter.log
    16152 root          1  20    0  6396K  1952K select  1   0:00   0.00% /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /etc/syslog.conf
     9276 root          2  26    0  6528K  1932K piperd  0   0:00   0.00% /usr/local/libexec/sshg-blocker
     9224 root          1  20    0 11912K  1928K piperd  1   0:00   0.00% /usr/local/libexec/sshg-parser
    17076 root          1  20    0  6604K  1904K bpf     0   0:01   0.01% /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid
    
    

    Any help appreciated.


  • Netgate Administrator

    Something there is out of control. That's running in 8GB? It should never be using swap.

    It's probably Snort but could be pfBlocker if you have a very large number of lists loaded.

    Steve



  • Thanks for replying- I don't think I have enabled very many lists at all. Is there a way for me to dump out the list to show that it is/isn't large?

    Between one of the "crashes" I disabled snort on the WAN interface and it didn't make a difference. I'm the only person home, so there also isn't much inbound or outbound network activity. Was there a reason that telegraph was listed so high on the top output that would be cause for concern?



  • it hasn't crashed yet this time... i've disabled telegraph this time, but it's already used a bit of swap:

    last pid: 46024;  load averages:  0.29,  0.24,  0.20                                                                                                                                   up 0+00:36:15  12:52:16
    90 processes:  2 running, 87 sleeping, 1 waiting
    CPU:  0.0% user,  0.0% nice,  0.2% system,  0.2% interrupt, 99.7% idle
    Mem: 2332M Active, 3474M Inact, 472M Laundry, 391M Wired, 234M Buf, 183M Free
    Swap: 410M Total, 11M Used, 399M Free, 2% Inuse
    
      PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
    54046 unbound       2  20    0 11154M   289M kqread  1   1:38   0.00% /usr/local/sbin/unbound -c /var/unbound/unbound.conf
    50190 root          2  20    0   105M 61396K bpf     0   0:01   0.02% /usr/local/bin/snort -R 21545 -D -q --suppress-config-log -l /var/log/snort/snort_em021545 --pid-path /var/run --nolock-pidfile
    49646 root          2  20    0   105M 61212K bpf     1   0:01   0.03% /usr/local/bin/snort -R 30980 -D -q --suppress-config-log -l /var/log/snort/snort_em130980 --pid-path /var/run --nolock-pidfile
    91204 root          1  20    0 51312K 28148K piperd  0   0:00   0.00% /usr/local/bin/php -f /usr/local/pkg/pfblockerng/pfblockerng.inc dnsbl
    89983 root          1  20    0 51376K 27028K piperd  1   0:00   0.00% /usr/local/bin/php_pfb -f /usr/local/pkg/pfblockerng/pfblockerng.inc filterlog
    88685 root          1  20    0 51376K 27028K piperd  0   0:00   0.00% /usr/local/bin/php_pfb -f /usr/local/pkg/pfblockerng/pfblockerng.inc filterlog
    91401 root          1  52    0 51312K 26948K piperd  0   0:00   0.00% /usr/local/bin/php -f /usr/local/pkg/pfblockerng/pfblockerng.inc index
      337 root          1  20    0 94296K 18328K kqread  0   0:00   0.01% php-fpm: master process (/usr/local/lib/php-fpm.conf) (php-fpm)
    58764 root          1  20    0 12400K 12504K select  1   0:00   0.01% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid
    68100 root          1  20    0 12908K  7248K select  1   0:00   0.00% sshd: root@pts/0 (sshd)
    66197 dhcpd         1  20    0 12580K  6816K select  0   0:00   0.02% /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid em1
    91190 root          1  20    0 10440K  6748K kqread  1   0:00   0.00% /usr/local/sbin/lighttpd_pfb -f /var/unbound/pfb_dnsbl_lighty.conf
     7577 root          1  20    0 12616K  6020K select  1   0:00   0.00% /usr/sbin/sshd
    95474 root          1  20    0  9468K  5312K select  1   0:00   0.04% /usr/local/sbin/miniupnpd -f /var/etc/miniupnpd.conf -P /var/run/miniupnpd.pid
      415 root          1  20    0  9188K  4440K select  0   0:00   0.00% /sbin/devd -q -f /etc/pfSense-devd.conf
    58090 root          1  20    0 23596K  4004K kqread  0   0:00   0.00% nginx: worker process (nginx)
    26429 root          1  20    0  7816K  3512K CPU0    0   0:01   0.02% top -aS
    91662 root          1  20    0 51312K  3312K nanslp  0   0:00   0.00% /usr/local/bin/php -f /usr/local/pkg/pfblockerng/pfblockerng.inc queries
     7893 root          1  20    0 11912K  2712K piperd  0   0:00   0.00% /usr/local/libexec/sshg-parser
     8953 root          1  52    0  6968K  2592K ttyin   0   0:00   0.00% /bin/sh /etc/rc.initial
    87005 root          1  52   20  6968K  2532K wait    0   0:00   0.00% /bin/sh /var/db/rrd/updaterrd.sh
    16125 root          1  20    0  6396K  2524K select  1   0:00   0.01% /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /etc/syslog.conf
     8458 root          1  33    0  6968K  2500K piperd  0   0:00   0.00% /bin/sh /usr/local/libexec/sshg-fw-pf
     7906 root          2  24    0  6528K  2468K piperd  0   0:00   0.00% /usr/local/libexec/sshg-blocker
    88541 root          1  20    0  4644K  2452K select  1   0:02   0.11% /usr/local/sbin/clog_pfb -f /var/log/filter.log
    
    

  • Netgate Administrator

    I would just disable all the packages as a test. Snort, pfBlocker and Squid can all use a lot of RAM but 8GB should be sufficient to prevent it swapping with reasonable list sizes.
    Telegraf really has no reason to use a lot of RAM so if it is it's a problem.

    Steve



  • @stephenw10
    Hi, i've moved away from using proxmox and done a complete bare metal reinstall.

    I have the same issue again... I haven't installed squid, snort or suricata... so I can be fairly certain that pfblockerNG is the cause. This is an i3 with 8GB RAM and what I would consider to be a modest amount of enabled feeds. Do you think with my spec I should be able to handle this without 100% swap usage?

    pfB_PRI1_v4	19,501	
    DNSBL_EasyList	3,402	
    DNSBL_ADs	71,763	
    DNSBL_Malicious	165,171	
    DNSBL_hpHosts	717,337	
    DNSBL_BBcan177	15,732	
    DNSBL_BBC	863,588	
    

    here's top

    last pid: 30150;  load averages:  0.66,  0.54,  0.44                                                                                                               up 2+18:14:27  16:34:30
    67 processes:  1 running, 64 sleeping, 2 waiting
    CPU:  0.0% user,  0.0% nice,  6.9% system,  0.0% interrupt, 93.1% idle
    Mem: 6917M Active, 52K Inact, 861M Wired, 617M Buf, 37M Free
    Swap: 4096M Total, 4096M Used, K Free, 100% Inuse
    
      PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
    28544 root          1  20    0  7812K  3112K CPU2    2   0:00   0.04% top
    44668 root          1  20    0  4644K  1880K select  3   0:01   0.04% clog_pfb
    46479 nobody        1  20    0 10868K  5124K select  2   0:19   0.03% darkstat
    54428 root          5  52    0  6900K  2016K uwait   3   0:12   0.02% dpinger
    20699 root          1  20    0 12904K  5888K select  3   0:00   0.01% sshd
    45573 dhcpd         1  20    0 12580K  2196K select  0   0:00   0.01% dhcpd
    43012 root          1  20    0 12908K 13012K select  3   0:09   0.01% ntpd
    96605 root          1  20    0 10436K  5324K kqread  1   0:00   0.00% lighttpd_pfb
      339 root          1  20    0 94292K  8504K kqread  3   0:06   0.00% php-fpm
    15832 root          1  20    0  6600K  1856K bpf     3   0:11   0.00% filterlog
    34948 unbound       4  22    0 11300M     0K pfault  2   0:25   0.00% <unbound>
     1287 root          1  22    0  8841M     0K pfault  2   0:20   0.00% <unbound-checkconf>
      340 root          1  52    0 98844K 14372K piperd  3   0:08   0.00% php-fpm
    18959 root          1  52   20  6968K     0K wait    2   0:07   0.00% <sh>
     6254 root          1  20    0  6400K  2008K select  0   0:05   0.00% syslogd
      341 root          1  52    0 94488K     0K accept  2   0:02   0.00% <php-fpm>
    37594 root          1  52    0 94488K     0K accept  2   0:01   0.00% <php-fpm>
    97162 root          1  23    0 51308K  1164K piperd  3   0:01   0.00% php
    29657 root          1  41   20 25424K     0K pfault  1   0:00   0.00% <unbound-control>
    42631 root          1  20    0  6368K     0K WAIT    0   0:00   0.00% <cron>
    96743 root          1  20    0 51308K  6244K piperd  1   0:00   0.00% php
    42079 root          1  20    0 23592K     0K kqread  2   0:00   0.00% <nginx>
    65015 root          2  34    0  6528K  1912K piperd  3   0:00   0.00% sshg-blocker
    44931 root          1  20    0 51372K  5156K piperd  1   0:00   0.00% php_pfb
    96827 root          1  52    0 51308K  4844K piperd  3   0:00   0.00% php
      417 root          1  20    0  9184K   312K select  3   0:00   0.00% devd
    


  • @meem Something is terribly wrong with your system at no fault of yours...time to install a fresh copy and restore configuration from backup, it seems.


  • Netgate Administrator

    Hmm yeah, probably feeds in DNS-BL. Look at the size of Unbound trying to load it, 11.3GB.

    For comparison I have just the Easylist feed giving ~20K entries and Unbound runs ~200MB.

    I would open a thread in the pfBlocker sub to get more detailed info there.

    Steve



  • @NollipfSense This is from a week-old clean install with manual reconfiguration (ie not even restoring from backup).

    Given i've had issues on multiple installs (some involving restoration of a backup from my old ESXi guest), I really can't see that it's related to a bad or corrupt install.

    The pfblockerng feeds that I am using come with a recommendation of >2Gb RAM, so my 8Gb really should be comfortable. I'll post over there now it seems to be narrowed down, thanks



  • unbound-checkconf is grabbing a big chunk of memory. It should exit before starting unbound.

    When did you reboot last ? What's the size of unbound.conf.
    Did you inspect System and Resolver log ? PfblockerNG.log ?

    If you stop unbound, is the unbound-checkconf process still present.


Log in to reply