running out of memory and swap
-
Hi, i have a fairly new pfsense build that is running out of RAM and swap. The web console becomes unresponsive, as do ssh sessions.
I've noticed on some reboots I get the "out of swap" message during boot, but usually it takes a short while to show up. I've disabled Squid and squidguard to no avail. I've also turned off snort on the WAN port, again without fix. Here's the output of my top:
last pid: 59890; load averages: 0.35, 0.63, 0.68 up 0+01:04:33 11:27:57 98 processes: 2 running, 86 sleeping, 10 waiting CPU: 0.0% user, 0.0% nice, 9.6% system, 0.4% interrupt, 90.0% idle Mem: 6048M Active, 24K Inact, 767M Wired, 615M Buf, 38M Free Swap: 410M Total, 410M Used, K Free, 100% Inuse, 152K In PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 60448 root 1 20 0 12400K 12504K select 0 0:01 0.01% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid 80422 root 2 20 0 105M 12212K bpf 0 0:07 0.30% /usr/local/bin/snort -R 30980 -D -q --suppress-config-log -l /var/log/snort/snort_em130980 --pid-path /var/run --nolock-pidfile 88211 root 12 20 0 52836K 9556K pfault 0 0:10 0.36% /usr/local/bin/telegraf -config=/usr/local/etc/telegraf.conf 311 root 1 20 0 94296K 6716K kqread 0 0:00 0.01% php-fpm: master process (/usr/local/lib/php-fpm.conf) (php-fpm) 28884 root 1 20 0 51312K 6148K piperd 1 0:00 0.00% /usr/local/bin/php -f /usr/local/pkg/pfblockerng/pfblockerng.inc dnsbl 26609 root 1 20 0 51376K 5808K piperd 0 0:00 0.00% /usr/local/bin/php_pfb -f /usr/local/pkg/pfblockerng/pfblockerng.inc filterlog 25396 root 1 20 0 51376K 5800K piperd 1 0:00 0.00% /usr/local/bin/php_pfb -f /usr/local/pkg/pfblockerng/pfblockerng.inc filterlog 22984 root 1 20 0 12908K 5004K select 0 0:01 0.02% sshd: root@pts/0 (sshd) 28596 root 1 20 0 10440K 4912K kqread 1 0:00 0.01% /usr/local/sbin/lighttpd_pfb -f /var/unbound/pfb_dnsbl_lighty.conf 28946 root 1 52 0 51312K 4644K piperd 0 0:00 0.00% /usr/local/bin/php -f /usr/local/pkg/pfblockerng/pfblockerng.inc index 96242 root 1 20 0 9468K 4296K select 0 0:01 0.00% /usr/local/sbin/miniupnpd -f /var/etc/miniupnpd.conf -P /var/run/miniupnpd.pid 7464 root 1 20 0 12616K 3652K select 0 0:00 0.00% /usr/sbin/sshd 30698 root 1 20 0 7816K 2876K CPU1 1 0:02 0.06% top -aS 61506 root 1 20 0 7816K 2600K select 1 0:02 0.05% top -aS 66174 dhcpd 1 20 0 12580K 2228K select 0 0:01 0.02% /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid em1 58033 root 1 20 0 6748K 2084K pfault 1 0:00 0.00% /usr/bin/netstat -ibdnW (<netstat>) 66520 root 1 20 0 6268K 2000K select 0 0:01 0.00% /usr/local/sbin/radvd -p /var/run/radvd.pid -C /var/etc/radvd.conf -m syslog 26352 root 1 20 0 4644K 2000K select 0 0:04 0.11% /usr/local/sbin/clog_pfb -f /var/log/filter.log 25266 root 1 20 0 4644K 2000K select 0 0:04 0.09% /usr/local/sbin/clog_pfb -f /var/log/filter.log 16152 root 1 20 0 6396K 1952K select 1 0:00 0.00% /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /etc/syslog.conf 9276 root 2 26 0 6528K 1932K piperd 0 0:00 0.00% /usr/local/libexec/sshg-blocker 9224 root 1 20 0 11912K 1928K piperd 1 0:00 0.00% /usr/local/libexec/sshg-parser 17076 root 1 20 0 6604K 1904K bpf 0 0:01 0.01% /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pidAny help appreciated.
-
Something there is out of control. That's running in 8GB? It should never be using swap.
It's probably Snort but could be pfBlocker if you have a very large number of lists loaded.
Steve
-
Thanks for replying- I don't think I have enabled very many lists at all. Is there a way for me to dump out the list to show that it is/isn't large?
Between one of the "crashes" I disabled snort on the WAN interface and it didn't make a difference. I'm the only person home, so there also isn't much inbound or outbound network activity. Was there a reason that telegraph was listed so high on the top output that would be cause for concern?
-
it hasn't crashed yet this time... i've disabled telegraph this time, but it's already used a bit of swap:
last pid: 46024; load averages: 0.29, 0.24, 0.20 up 0+00:36:15 12:52:16 90 processes: 2 running, 87 sleeping, 1 waiting CPU: 0.0% user, 0.0% nice, 0.2% system, 0.2% interrupt, 99.7% idle Mem: 2332M Active, 3474M Inact, 472M Laundry, 391M Wired, 234M Buf, 183M Free Swap: 410M Total, 11M Used, 399M Free, 2% Inuse PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 54046 unbound 2 20 0 11154M 289M kqread 1 1:38 0.00% /usr/local/sbin/unbound -c /var/unbound/unbound.conf 50190 root 2 20 0 105M 61396K bpf 0 0:01 0.02% /usr/local/bin/snort -R 21545 -D -q --suppress-config-log -l /var/log/snort/snort_em021545 --pid-path /var/run --nolock-pidfile 49646 root 2 20 0 105M 61212K bpf 1 0:01 0.03% /usr/local/bin/snort -R 30980 -D -q --suppress-config-log -l /var/log/snort/snort_em130980 --pid-path /var/run --nolock-pidfile 91204 root 1 20 0 51312K 28148K piperd 0 0:00 0.00% /usr/local/bin/php -f /usr/local/pkg/pfblockerng/pfblockerng.inc dnsbl 89983 root 1 20 0 51376K 27028K piperd 1 0:00 0.00% /usr/local/bin/php_pfb -f /usr/local/pkg/pfblockerng/pfblockerng.inc filterlog 88685 root 1 20 0 51376K 27028K piperd 0 0:00 0.00% /usr/local/bin/php_pfb -f /usr/local/pkg/pfblockerng/pfblockerng.inc filterlog 91401 root 1 52 0 51312K 26948K piperd 0 0:00 0.00% /usr/local/bin/php -f /usr/local/pkg/pfblockerng/pfblockerng.inc index 337 root 1 20 0 94296K 18328K kqread 0 0:00 0.01% php-fpm: master process (/usr/local/lib/php-fpm.conf) (php-fpm) 58764 root 1 20 0 12400K 12504K select 1 0:00 0.01% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid 68100 root 1 20 0 12908K 7248K select 1 0:00 0.00% sshd: root@pts/0 (sshd) 66197 dhcpd 1 20 0 12580K 6816K select 0 0:00 0.02% /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid em1 91190 root 1 20 0 10440K 6748K kqread 1 0:00 0.00% /usr/local/sbin/lighttpd_pfb -f /var/unbound/pfb_dnsbl_lighty.conf 7577 root 1 20 0 12616K 6020K select 1 0:00 0.00% /usr/sbin/sshd 95474 root 1 20 0 9468K 5312K select 1 0:00 0.04% /usr/local/sbin/miniupnpd -f /var/etc/miniupnpd.conf -P /var/run/miniupnpd.pid 415 root 1 20 0 9188K 4440K select 0 0:00 0.00% /sbin/devd -q -f /etc/pfSense-devd.conf 58090 root 1 20 0 23596K 4004K kqread 0 0:00 0.00% nginx: worker process (nginx) 26429 root 1 20 0 7816K 3512K CPU0 0 0:01 0.02% top -aS 91662 root 1 20 0 51312K 3312K nanslp 0 0:00 0.00% /usr/local/bin/php -f /usr/local/pkg/pfblockerng/pfblockerng.inc queries 7893 root 1 20 0 11912K 2712K piperd 0 0:00 0.00% /usr/local/libexec/sshg-parser 8953 root 1 52 0 6968K 2592K ttyin 0 0:00 0.00% /bin/sh /etc/rc.initial 87005 root 1 52 20 6968K 2532K wait 0 0:00 0.00% /bin/sh /var/db/rrd/updaterrd.sh 16125 root 1 20 0 6396K 2524K select 1 0:00 0.01% /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /etc/syslog.conf 8458 root 1 33 0 6968K 2500K piperd 0 0:00 0.00% /bin/sh /usr/local/libexec/sshg-fw-pf 7906 root 2 24 0 6528K 2468K piperd 0 0:00 0.00% /usr/local/libexec/sshg-blocker 88541 root 1 20 0 4644K 2452K select 1 0:02 0.11% /usr/local/sbin/clog_pfb -f /var/log/filter.log -
I would just disable all the packages as a test. Snort, pfBlocker and Squid can all use a lot of RAM but 8GB should be sufficient to prevent it swapping with reasonable list sizes.
Telegraf really has no reason to use a lot of RAM so if it is it's a problem.Steve
-
@stephenw10
Hi, i've moved away from using proxmox and done a complete bare metal reinstall.I have the same issue again... I haven't installed squid, snort or suricata... so I can be fairly certain that pfblockerNG is the cause. This is an i3 with 8GB RAM and what I would consider to be a modest amount of enabled feeds. Do you think with my spec I should be able to handle this without 100% swap usage?
pfB_PRI1_v4 19,501 DNSBL_EasyList 3,402 DNSBL_ADs 71,763 DNSBL_Malicious 165,171 DNSBL_hpHosts 717,337 DNSBL_BBcan177 15,732 DNSBL_BBC 863,588here's top
last pid: 30150; load averages: 0.66, 0.54, 0.44 up 2+18:14:27 16:34:30 67 processes: 1 running, 64 sleeping, 2 waiting CPU: 0.0% user, 0.0% nice, 6.9% system, 0.0% interrupt, 93.1% idle Mem: 6917M Active, 52K Inact, 861M Wired, 617M Buf, 37M Free Swap: 4096M Total, 4096M Used, K Free, 100% Inuse PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 28544 root 1 20 0 7812K 3112K CPU2 2 0:00 0.04% top 44668 root 1 20 0 4644K 1880K select 3 0:01 0.04% clog_pfb 46479 nobody 1 20 0 10868K 5124K select 2 0:19 0.03% darkstat 54428 root 5 52 0 6900K 2016K uwait 3 0:12 0.02% dpinger 20699 root 1 20 0 12904K 5888K select 3 0:00 0.01% sshd 45573 dhcpd 1 20 0 12580K 2196K select 0 0:00 0.01% dhcpd 43012 root 1 20 0 12908K 13012K select 3 0:09 0.01% ntpd 96605 root 1 20 0 10436K 5324K kqread 1 0:00 0.00% lighttpd_pfb 339 root 1 20 0 94292K 8504K kqread 3 0:06 0.00% php-fpm 15832 root 1 20 0 6600K 1856K bpf 3 0:11 0.00% filterlog 34948 unbound 4 22 0 11300M 0K pfault 2 0:25 0.00% <unbound> 1287 root 1 22 0 8841M 0K pfault 2 0:20 0.00% <unbound-checkconf> 340 root 1 52 0 98844K 14372K piperd 3 0:08 0.00% php-fpm 18959 root 1 52 20 6968K 0K wait 2 0:07 0.00% <sh> 6254 root 1 20 0 6400K 2008K select 0 0:05 0.00% syslogd 341 root 1 52 0 94488K 0K accept 2 0:02 0.00% <php-fpm> 37594 root 1 52 0 94488K 0K accept 2 0:01 0.00% <php-fpm> 97162 root 1 23 0 51308K 1164K piperd 3 0:01 0.00% php 29657 root 1 41 20 25424K 0K pfault 1 0:00 0.00% <unbound-control> 42631 root 1 20 0 6368K 0K WAIT 0 0:00 0.00% <cron> 96743 root 1 20 0 51308K 6244K piperd 1 0:00 0.00% php 42079 root 1 20 0 23592K 0K kqread 2 0:00 0.00% <nginx> 65015 root 2 34 0 6528K 1912K piperd 3 0:00 0.00% sshg-blocker 44931 root 1 20 0 51372K 5156K piperd 1 0:00 0.00% php_pfb 96827 root 1 52 0 51308K 4844K piperd 3 0:00 0.00% php 417 root 1 20 0 9184K 312K select 3 0:00 0.00% devd -
@meem Something is terribly wrong with your system at no fault of yours...time to install a fresh copy and restore configuration from backup, it seems.
-
Hmm yeah, probably feeds in DNS-BL. Look at the size of Unbound trying to load it, 11.3GB.
For comparison I have just the Easylist feed giving ~20K entries and Unbound runs ~200MB.
I would open a thread in the pfBlocker sub to get more detailed info there.
Steve
-
@NollipfSense This is from a week-old clean install with manual reconfiguration (ie not even restoring from backup).
Given i've had issues on multiple installs (some involving restoration of a backup from my old ESXi guest), I really can't see that it's related to a bad or corrupt install.
The pfblockerng feeds that I am using come with a recommendation of >2Gb RAM, so my 8Gb really should be comfortable. I'll post over there now it seems to be narrowed down, thanks
-
unbound-checkconf is grabbing a big chunk of memory. It should exit before starting unbound.
When did you reboot last ? What's the size of unbound.conf.
Did you inspect System and Resolver log ? PfblockerNG.log ?If you stop unbound, is the unbound-checkconf process still present.
