Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    running out of memory and swap

    Scheduled Pinned Locked Moved
    General pfSense Questions
    4
    10
    810
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      meem
      last edited by

      Hi, i have a fairly new pfsense build that is running out of RAM and swap. The web console becomes unresponsive, as do ssh sessions.

      I've noticed on some reboots I get the "out of swap" message during boot, but usually it takes a short while to show up. I've disabled Squid and squidguard to no avail. I've also turned off snort on the WAN port, again without fix. Here's the output of my top:

      last pid: 59890;  load averages:  0.35,  0.63,  0.68                                                                                                                                   up 0+01:04:33  11:27:57
98 processes:  2 running, 86 sleeping, 10 waiting
CPU:  0.0% user,  0.0% nice,  9.6% system,  0.4% interrupt, 90.0% idle
Mem: 6048M Active, 24K Inact, 767M Wired, 615M Buf, 38M Free
Swap: 410M Total, 410M Used, K Free, 100% Inuse, 152K In

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
60448 root          1  20    0 12400K 12504K select  0   0:01   0.01% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid
80422 root          2  20    0   105M 12212K bpf     0   0:07   0.30% /usr/local/bin/snort -R 30980 -D -q --suppress-config-log -l /var/log/snort/snort_em130980 --pid-path /var/run --nolock-pidfile
88211 root         12  20    0 52836K  9556K pfault  0   0:10   0.36% /usr/local/bin/telegraf -config=/usr/local/etc/telegraf.conf
  311 root          1  20    0 94296K  6716K kqread  0   0:00   0.01% php-fpm: master process (/usr/local/lib/php-fpm.conf) (php-fpm)
28884 root          1  20    0 51312K  6148K piperd  1   0:00   0.00% /usr/local/bin/php -f /usr/local/pkg/pfblockerng/pfblockerng.inc dnsbl
26609 root          1  20    0 51376K  5808K piperd  0   0:00   0.00% /usr/local/bin/php_pfb -f /usr/local/pkg/pfblockerng/pfblockerng.inc filterlog
25396 root          1  20    0 51376K  5800K piperd  1   0:00   0.00% /usr/local/bin/php_pfb -f /usr/local/pkg/pfblockerng/pfblockerng.inc filterlog
22984 root          1  20    0 12908K  5004K select  0   0:01   0.02% sshd: root@pts/0 (sshd)
28596 root          1  20    0 10440K  4912K kqread  1   0:00   0.01% /usr/local/sbin/lighttpd_pfb -f /var/unbound/pfb_dnsbl_lighty.conf
28946 root          1  52    0 51312K  4644K piperd  0   0:00   0.00% /usr/local/bin/php -f /usr/local/pkg/pfblockerng/pfblockerng.inc index
96242 root          1  20    0  9468K  4296K select  0   0:01   0.00% /usr/local/sbin/miniupnpd -f /var/etc/miniupnpd.conf -P /var/run/miniupnpd.pid
 7464 root          1  20    0 12616K  3652K select  0   0:00   0.00% /usr/sbin/sshd
30698 root          1  20    0  7816K  2876K CPU1    1   0:02   0.06% top -aS
61506 root          1  20    0  7816K  2600K select  1   0:02   0.05% top -aS
66174 dhcpd         1  20    0 12580K  2228K select  0   0:01   0.02% /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid em1
58033 root          1  20    0  6748K  2084K pfault  1   0:00   0.00% /usr/bin/netstat -ibdnW (<netstat>)
66520 root          1  20    0  6268K  2000K select  0   0:01   0.00% /usr/local/sbin/radvd -p /var/run/radvd.pid -C /var/etc/radvd.conf -m syslog
26352 root          1  20    0  4644K  2000K select  0   0:04   0.11% /usr/local/sbin/clog_pfb -f /var/log/filter.log
25266 root          1  20    0  4644K  2000K select  0   0:04   0.09% /usr/local/sbin/clog_pfb -f /var/log/filter.log
16152 root          1  20    0  6396K  1952K select  1   0:00   0.00% /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /etc/syslog.conf
 9276 root          2  26    0  6528K  1932K piperd  0   0:00   0.00% /usr/local/libexec/sshg-blocker
 9224 root          1  20    0 11912K  1928K piperd  1   0:00   0.00% /usr/local/libexec/sshg-parser
17076 root          1  20    0  6604K  1904K bpf     0   0:01   0.01% /usr/local/sbin/filterlog -i pflog0 -p /var/run/filterlog.pid

      Any help appreciated.

      1 Reply Last reply Reply Quote 0
      • stephenw10S
        stephenw10 Netgate Administrator
        last edited by

        Something there is out of control. That's running in 8GB? It should never be using swap.

        It's probably Snort but could be pfBlocker if you have a very large number of lists loaded.

        Steve

        M 1 Reply Last reply Reply Quote 0
        • M
          meem @stephenw10
          last edited by

          Thanks for replying- I don't think I have enabled very many lists at all. Is there a way for me to dump out the list to show that it is/isn't large?

          Between one of the "crashes" I disabled snort on the WAN interface and it didn't make a difference. I'm the only person home, so there also isn't much inbound or outbound network activity. Was there a reason that telegraph was listed so high on the top output that would be cause for concern?

          1 Reply Last reply Reply Quote 0
          • M
            meem
            last edited by

            it hasn't crashed yet this time... i've disabled telegraph this time, but it's already used a bit of swap:

            last pid: 46024;  load averages:  0.29,  0.24,  0.20                                                                                                                                   up 0+00:36:15  12:52:16
90 processes:  2 running, 87 sleeping, 1 waiting
CPU:  0.0% user,  0.0% nice,  0.2% system,  0.2% interrupt, 99.7% idle
Mem: 2332M Active, 3474M Inact, 472M Laundry, 391M Wired, 234M Buf, 183M Free
Swap: 410M Total, 11M Used, 399M Free, 2% Inuse

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
54046 unbound       2  20    0 11154M   289M kqread  1   1:38   0.00% /usr/local/sbin/unbound -c /var/unbound/unbound.conf
50190 root          2  20    0   105M 61396K bpf     0   0:01   0.02% /usr/local/bin/snort -R 21545 -D -q --suppress-config-log -l /var/log/snort/snort_em021545 --pid-path /var/run --nolock-pidfile
49646 root          2  20    0   105M 61212K bpf     1   0:01   0.03% /usr/local/bin/snort -R 30980 -D -q --suppress-config-log -l /var/log/snort/snort_em130980 --pid-path /var/run --nolock-pidfile
91204 root          1  20    0 51312K 28148K piperd  0   0:00   0.00% /usr/local/bin/php -f /usr/local/pkg/pfblockerng/pfblockerng.inc dnsbl
89983 root          1  20    0 51376K 27028K piperd  1   0:00   0.00% /usr/local/bin/php_pfb -f /usr/local/pkg/pfblockerng/pfblockerng.inc filterlog
88685 root          1  20    0 51376K 27028K piperd  0   0:00   0.00% /usr/local/bin/php_pfb -f /usr/local/pkg/pfblockerng/pfblockerng.inc filterlog
91401 root          1  52    0 51312K 26948K piperd  0   0:00   0.00% /usr/local/bin/php -f /usr/local/pkg/pfblockerng/pfblockerng.inc index
  337 root          1  20    0 94296K 18328K kqread  0   0:00   0.01% php-fpm: master process (/usr/local/lib/php-fpm.conf) (php-fpm)
58764 root          1  20    0 12400K 12504K select  1   0:00   0.01% /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid
68100 root          1  20    0 12908K  7248K select  1   0:00   0.00% sshd: root@pts/0 (sshd)
66197 dhcpd         1  20    0 12580K  6816K select  0   0:00   0.02% /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid em1
91190 root          1  20    0 10440K  6748K kqread  1   0:00   0.00% /usr/local/sbin/lighttpd_pfb -f /var/unbound/pfb_dnsbl_lighty.conf
 7577 root          1  20    0 12616K  6020K select  1   0:00   0.00% /usr/sbin/sshd
95474 root          1  20    0  9468K  5312K select  1   0:00   0.04% /usr/local/sbin/miniupnpd -f /var/etc/miniupnpd.conf -P /var/run/miniupnpd.pid
  415 root          1  20    0  9188K  4440K select  0   0:00   0.00% /sbin/devd -q -f /etc/pfSense-devd.conf
58090 root          1  20    0 23596K  4004K kqread  0   0:00   0.00% nginx: worker process (nginx)
26429 root          1  20    0  7816K  3512K CPU0    0   0:01   0.02% top -aS
91662 root          1  20    0 51312K  3312K nanslp  0   0:00   0.00% /usr/local/bin/php -f /usr/local/pkg/pfblockerng/pfblockerng.inc queries
 7893 root          1  20    0 11912K  2712K piperd  0   0:00   0.00% /usr/local/libexec/sshg-parser
 8953 root          1  52    0  6968K  2592K ttyin   0   0:00   0.00% /bin/sh /etc/rc.initial
87005 root          1  52   20  6968K  2532K wait    0   0:00   0.00% /bin/sh /var/db/rrd/updaterrd.sh
16125 root          1  20    0  6396K  2524K select  1   0:00   0.01% /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /etc/syslog.conf
 8458 root          1  33    0  6968K  2500K piperd  0   0:00   0.00% /bin/sh /usr/local/libexec/sshg-fw-pf
 7906 root          2  24    0  6528K  2468K piperd  0   0:00   0.00% /usr/local/libexec/sshg-blocker
88541 root          1  20    0  4644K  2452K select  1   0:02   0.11% /usr/local/sbin/clog_pfb -f /var/log/filter.log
            1 Reply Last reply Reply Quote 0
            • stephenw10S
              stephenw10 Netgate Administrator
              last edited by

              I would just disable all the packages as a test. Snort, pfBlocker and Squid can all use a lot of RAM but 8GB should be sufficient to prevent it swapping with reasonable list sizes.
              Telegraf really has no reason to use a lot of RAM so if it is it's a problem.

              Steve

              M 1 Reply Last reply Reply Quote 1
              • M
                meem @stephenw10
                last edited by

                @stephenw10
                Hi, i've moved away from using proxmox and done a complete bare metal reinstall.

                I have the same issue again... I haven't installed squid, snort or suricata... so I can be fairly certain that pfblockerNG is the cause. This is an i3 with 8GB RAM and what I would consider to be a modest amount of enabled feeds. Do you think with my spec I should be able to handle this without 100% swap usage?

                pfB_PRI1_v4	19,501	
DNSBL_EasyList	3,402	
DNSBL_ADs	71,763	
DNSBL_Malicious	165,171	
DNSBL_hpHosts	717,337	
DNSBL_BBcan177	15,732	
DNSBL_BBC	863,588

                here's top

                last pid: 30150;  load averages:  0.66,  0.54,  0.44                                                                                                               up 2+18:14:27  16:34:30
67 processes:  1 running, 64 sleeping, 2 waiting
CPU:  0.0% user,  0.0% nice,  6.9% system,  0.0% interrupt, 93.1% idle
Mem: 6917M Active, 52K Inact, 861M Wired, 617M Buf, 37M Free
Swap: 4096M Total, 4096M Used, K Free, 100% Inuse

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
28544 root          1  20    0  7812K  3112K CPU2    2   0:00   0.04% top
44668 root          1  20    0  4644K  1880K select  3   0:01   0.04% clog_pfb
46479 nobody        1  20    0 10868K  5124K select  2   0:19   0.03% darkstat
54428 root          5  52    0  6900K  2016K uwait   3   0:12   0.02% dpinger
20699 root          1  20    0 12904K  5888K select  3   0:00   0.01% sshd
45573 dhcpd         1  20    0 12580K  2196K select  0   0:00   0.01% dhcpd
43012 root          1  20    0 12908K 13012K select  3   0:09   0.01% ntpd
96605 root          1  20    0 10436K  5324K kqread  1   0:00   0.00% lighttpd_pfb
  339 root          1  20    0 94292K  8504K kqread  3   0:06   0.00% php-fpm
15832 root          1  20    0  6600K  1856K bpf     3   0:11   0.00% filterlog
34948 unbound       4  22    0 11300M     0K pfault  2   0:25   0.00% <unbound>
 1287 root          1  22    0  8841M     0K pfault  2   0:20   0.00% <unbound-checkconf>
  340 root          1  52    0 98844K 14372K piperd  3   0:08   0.00% php-fpm
18959 root          1  52   20  6968K     0K wait    2   0:07   0.00% <sh>
 6254 root          1  20    0  6400K  2008K select  0   0:05   0.00% syslogd
  341 root          1  52    0 94488K     0K accept  2   0:02   0.00% <php-fpm>
37594 root          1  52    0 94488K     0K accept  2   0:01   0.00% <php-fpm>
97162 root          1  23    0 51308K  1164K piperd  3   0:01   0.00% php
29657 root          1  41   20 25424K     0K pfault  1   0:00   0.00% <unbound-control>
42631 root          1  20    0  6368K     0K WAIT    0   0:00   0.00% <cron>
96743 root          1  20    0 51308K  6244K piperd  1   0:00   0.00% php
42079 root          1  20    0 23592K     0K kqread  2   0:00   0.00% <nginx>
65015 root          2  34    0  6528K  1912K piperd  3   0:00   0.00% sshg-blocker
44931 root          1  20    0 51372K  5156K piperd  1   0:00   0.00% php_pfb
96827 root          1  52    0 51308K  4844K piperd  3   0:00   0.00% php
  417 root          1  20    0  9184K   312K select  3   0:00   0.00% devd
                NollipfSenseN 1 Reply Last reply Reply Quote 0
                • NollipfSenseN
                  NollipfSense @meem
                  last edited by

                  @meem Something is terribly wrong with your system at no fault of yours...time to install a fresh copy and restore configuration from backup, it seems.

                  pfSense+ 23.09 Lenovo Thinkcentre M93P SFF Quadcore i7 dual Raid-ZFS 128GB-SSD 32GB-RAM PCI-Intel i350-t4 NIC, -Intel QAT 8950.
                  pfSense+ 23.09 VM-Proxmox, Dell Precision Xeon-W2155 Nvme 500GB-ZFS 128GB-RAM PCIe-Intel i350-t4, Intel QAT-8950, P-cloud.

                  M 1 Reply Last reply Reply Quote 0
                  • stephenw10S
                    stephenw10 Netgate Administrator
                    last edited by

                    Hmm yeah, probably feeds in DNS-BL. Look at the size of Unbound trying to load it, 11.3GB.

                    For comparison I have just the Easylist feed giving ~20K entries and Unbound runs ~200MB.

                    I would open a thread in the pfBlocker sub to get more detailed info there.

                    Steve

                    1 Reply Last reply Reply Quote 0
                    • M
                      meem @NollipfSense
                      last edited by

                      @NollipfSense This is from a week-old clean install with manual reconfiguration (ie not even restoring from backup).

                      Given i've had issues on multiple installs (some involving restoration of a backup from my old ESXi guest), I really can't see that it's related to a bad or corrupt install.

                      The pfblockerng feeds that I am using come with a recommendation of >2Gb RAM, so my 8Gb really should be comfortable. I'll post over there now it seems to be narrowed down, thanks

                      1 Reply Last reply Reply Quote 0
                      • RonpfSR
                        RonpfS
                        last edited by RonpfS

                        unbound-checkconf is grabbing a big chunk of memory. It should exit before starting unbound.

                        When did you reboot last ? What's the size of unbound.conf.
                        Did you inspect System and Resolver log ? PfblockerNG.log ?

                        If you stop unbound, is the unbound-checkconf process still present.

                        2.4.5-RELEASE-p1 (amd64)
                        Intel Core2 Quad CPU Q8400 @ 2.66GHz 8GB
                        Backup 0.5_5, Bandwidthd 0.7.4_4, Cron 0.3.7_5, pfBlockerNG-devel 3.0.0_16, Status_Traffic_Totals 2.3.1_1, System_Patches 1.2_5

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post
                        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.