pfSense crashes after wire memory increase
-
Hi all,
I am getting some critical issues with wire memory. This memory is constantly increasing up to reach the memory limit. At this time the firewall is crashed. I attach the memory graphs.
I read that sometimes the ARC of ZFS is the issue, but I don't have it in my system. I have the next modules installed in PfSense:
- sysutils
- net-mgmt
- security
- sysutils
Also my top output is this:
`[root@user path]# last pid: 28440; load averages: 0.70, 0.73, 0.74 up 1+08:53:20 16:42:16 78 processes: 1 running, 77 sleeping CPU: 24.8% user, 0.0% nice, 0.0% system, 5.7% interrupt, 69.5% idle Mem: 36M Active, 155M Inact, 461M Wired, 25M Buf, 1300M Free Swap: 4096M Total, 4096M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 33810 root 17 52 0 249M 51816K sigwai 1 788:58 80.06% charon 28440 root 1 20 0 20068K 3492K CPU3 3 0:00 0.65% top 64989 root 1 20 0 78872K 8184K select 1 0:02 0.26% sshd 16721 root 1 20 0 10488K 2576K select 1 2:15 0.00% syslogd 8083 root 1 20 0 12736K 2380K bpf 0 1:28 0.00% filterlog 316 root 1 52 0 268M 45144K accept 3 0:43 0.00% php-fpm 317 root 1 52 0 274M 45612K accept 2 0:42 0.00% php-fpm 10244 root 5 52 0 10988K 2320K uwait 1 0:23 0.00% dpinger 10467 root 5 52 0 10988K 2320K uwait 2 0:22 0.00% dpinger 37180 root 1 52 20 13096K 2596K wait 0 0:21 0.00% sh 26615 root 1 20 0 24656K 12480K select 0 0:09 0.00% ntpd 315 root 1 20 0 259M 26400K kqread 0 0:07 0.00% php-fpm 387 root 1 20 0 9176K 4908K select 2 0:05 0.00% devd 64666 nagios 1 20 0 17444K 5528K select 3 0:02 0.00% nrpe2 25277 root 1 20 0 37704K 8416K kqread 2 0:02 0.00% nginx 25479 root 1 20 0 37704K 8888K kqread 1 0:02 0.00% nginx 14101 root 1 20 0 13808K 3872K wait 0 0:01 0.00% bash 73374 root 12 20 0 72108K 9420K nanslp 2 0:00 0.00% collectd 19795 unbound 4 20 0 81156K 22764K kqread 3 0:00 0.00% unbound 17438 nagios 1 20 0 16800K 2420K pause 2 0:00 0.00% iostat 68628 root 1 20 0 13400K 3884K pause 0 0:00 0.00% tcsh 20498 root 1 20 0 53524K 6808K select 1 0:00 0.00% sshd 26027 root 1 20 0 12504K 2360K nanslp 2 0:00 0.00% cron 90996 root 2 20 0 10588K 2320K piperd 3 0:00 0.00% sshlockout_pf 17577 nagios 1 20 0 14740K 2532K sbwait 1 0:00 0.00% egrep 73618 nagios 1 20 0 14656K 3848K wait 2 0:00 0.00% ksh93 65700 root 1 20 0 8232K 2028K nanslp 0 0:00 0.00% minicron 43681 nagios 1 20 0 16800K 2424K pause 3 0:00 0.00% iostat 32225 nagios 1 20 0 16800K 2424K pause 1 0:00 0.00% iostat 10834 nagios 1 20 0 16800K 2424K pause 2 0:00 0.00% iostat 68189 nagios 1 20 0 16800K 2424K pause 3 0:00 0.00% iostat 34550 nagios 1 20 0 16800K 2424K pause 2 0:00 0.00% iostat 10868 nagios 1 20 0 14740K 2532K sbwait 1 0:00 0.00% egrep 21794 nagios 1 20 0 16800K 2424K pause 3 0:00 0.00% iostat 354 root 1 40 20 19456K 4384K kqread 1 0:00 0.00% check_reload_status 68413 nagios 1 20 0 14740K 2532K sbwait 2 0:00 0.00% egrep 43807 nagios 1 20 0 14740K 2532K sbwait 1 0:00 0.00% egrep 34770 nagios 1 20 0 14740K 2532K sbwait 3 0:00 0.00% egrep 32292 nagios 1 20 0 14740K 2532K sbwait 1 0:00 0.00% egrep 18168 nagios 1 20 0 10668K 2504K sbwait 1 0:00 0.00% awk 3236 nagios 1 20 0 16800K 2424K pause 1 0:00 0.00% iostat 22001 nagios 1 20 0 14740K 2524K sbwait 1 0:00 0.00% egrep 17367 root 1 52 0 39440K 2844K wait 2 0:00 0.00% login 17824 nagios 1 20 0 14740K 2536K sbwait 2 0:00 0.00% egrep 66516 root 1 20 0 8232K 2028K nanslp 1 0:00 0.00% minicron 65397 root 1 52 0 13096K 2812K wait 3 0:00 0.00% sh 3435 nagios 1 20 0 14740K 2512K sbwait 3 0:00 0.00% egrep 19010 root 1 52 0 13096K 2932K wait 3 0:00 0.00% sh 33681 root 1 52 0 26468K 3772K select 0 0:00 0.00% starter 35215 nagios 1 20 0 10668K 2480K sbwait 3 0:00 0.00% awk 32392 nagios 1 20 0 10668K 2480K sbwait 3 0:00 0.00% awk 44430 nagios 1 20 0 10668K 2480K sbwait 0 0:00 0.00% awk 68615 nagios 1 20 0 10668K 2480K sbwait 3 0:00 0.00% awk 11120 nagios 1 20 0 10668K 2480K sbwait 3 0:00 0.00% awk 19134 root 1 52 0 13096K 2812K ttyin 0 0:00 0.00% sh 68449 nagios 1 20 0 14740K 2536K sbwait 1 0:00 0.00% egrep 10883 nagios 1 20 0 14740K 2536K sbwait 0 0:00 0.00% egrep 34986 nagios 1 20 0 14740K 2536K sbwait 0 0:00 0.00% egrep 44144 nagios 1 20 0 14740K 2536K sbwait 3 0:00 0.00% egrep 32305 nagios 1 20 0 14740K 2536K sbwait 1 0:00 0.00% egrep 18672 root 1 52 0 10396K 2140K ttyin 3 0:00 0.00% getty 22167 nagios 1 20 0 14740K 2520K sbwait 3 0:00 0.00% egrep 18241 root 1 52 0 10396K 2140K ttyin 3 0:00 0.00% getty 17659 root 1 52 0 10396K 2140K ttyin 2 0:00 0.00% getty 3439 nagios 1 20 0 14740K 2496K sbwait 1 0:00 0.00% egrep 18531 root 1 52 0 10396K 2140K ttyin 3 0:00 0.00% getty 17743 root 1 52 0 10396K 2140K ttyin 2 0:00 0.00% getty 18704 root 1 52 0 10396K 2140K ttyin 0 0:00 0.00% getty 22381 nagios 1 20 0 10668K 2476K sbwait 0 0:00 0.00% awk 3580 nagios 1 20 0 10668K 2432K sbwait 2 0:00 0.00% awk 18065 root 1 52 0 10396K 2140K ttyin 2 0:00 0.00% getty 25143 root 1 52 0 35656K 6816K pause 2 0:00 0.00% nginx 6802 root 1 52 20 6180K 1936K nanslp 3 0:00 0.00% sleep 66020 root 1 52 0 8232K 2012K wait 2 0:00 0.00% minicron 66667 root 1 52 0 8232K 2012K wait 2 0:00 0.00% minicron 66977 root 1 20 0 8232K 2028K nanslp 0 0:00 0.00% minicron 65459 root 1 52 0 8232K 2012K wait 2 0:00 0.00% minicron 356 root 1 52 20 19456K 4200K kqread 0 0:00 0.00% check_reload_statusvmstat -m output:
Type InUse MemUse HighUse Requests Size(s) acpidev 19 2K - 19 64 hdaa 5 5K - 5 512,1024,2048 hdac 1 1K - 1 1024 hdacc 1 1K - 1 32 feeder 12 1K - 14 32,128 mixer 1 4K - 1 4096 CAM SIM 3 1K - 3 256 CAM XPT 18 2K - 34 32,64,128,512,1024,2048 CAM DEV 4 8K - 12 2048 CAM CCB 0 0K - 79012 2048 CAM path 5 1K - 42 32 CAM periph 4 1K - 39502 16,32,64,128,256 USB 18 15K - 18 16,128,512,4096 USBdev 15 2K - 15 32,64,128,256 pci_link 10 1K - 10 16,128 CAM queue 7 3K - 32 16,32,512 scsi_cd 0 0K - 9 16 CAM dev queue 3 1K - 3 64 vtbuf 24 1968K - 46 4096 vt 11 6K - 11 512 DEVFS3 111 28K - 131 256 DEVFS1 86 43K - 93 512 DEVFS 13 1K - 14 16,32,128 DEVFSP 1 1K - 57 64 NFSD V4client 1 1K - 1 256 NFSD lckfile 1 1K - 1 256 NFSD session 1 1K - 1 1024 nullfs_hash 1 512K - 1 pfs_nodes 21 6K - 21 256 GEOM 90 22K - 3857 16,32,64,128,256,512,1024,2048,8192 raid_data 0 0K - 84 32,128,256 isadev 8 1K - 8 128 cdev 6 2K - 6 256 filedesc 9 53K - 87883 32,4096,16384 sigio 3 1K - 5 64 filecaps 0 0K - 78 16,64 kdtrace 369 73K - 1232497 64,256 kenv 109 12K - 583 16,32,64,128,8192 kqueue 128 27K - 616105 64,256,512,2048 proc-args 77 6K - 507488 16,32,64,128,256 Fail Points 0 0K - 7900 1024 hhook 17 4K - 17 32,256 ithread 179 27K - 180 32,128,256 KTRACE 100 13K - 100 128 linker 294 56K - 308 16,32,64,128,256,512,1024,2048,4096,8192 lockf 11 2K - 259343 64,128 loginclass 3 1K - 3 64 devbuf 4494 4345K - 5004 16,32,64,128,256,512,1024,2048,4096,8192,16384,65536 temp 93 23K - 1904882 16,32,64,128,256,512,1024,2048,4096,8192,16384,65536 ata_pci 1 1K - 1 64 module 648 81K - 655 128 mtx_pool 2 16K - 2 8192 osd 3 1K - 9 16,32,64,128,256 pmchooks 1 1K - 1 128 pgrp 44 6K - 12835 128 session 39 5K - 568 128 proc 2 16K - 2 8192 subproc 270 510K - 616223 512,4096 cred 47 12K - 208670 256 plimit 23 6K - 7305 256 uidinfo 4 3K - 28648 128,2048 dumper 1 1K - 1 512 kbdmux 6 22K - 6 16,512,1024,2048,16384 sysctl 0 0K - 1048972 16,32,64 sysctloid 4069 218K - 4188 16,32,64,128 sysctltmp 0 0K - 98236 16,32,64,128,256,1024 tidhash 1 16K - 1 16384 callout 5 2184K - 5 umtx 720 90K - 720 128 p1003.1b 1 1K - 1 16 SWAP 2 277K - 2 64 bus 1309 108K - 197165 16,32,64,128,256,1024 bus-sc 63 222K - 3927 16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536 devstat 4 9K - 4 32,4096 eventhandler 146 12K - 146 64,128 taskqueue 17 18K - 17 16,32,256,8192 kobj 400 1600K - 559 4096 Per-cpu 1 1K - 1 32 rman 155 19K - 513 16,32,128 sbuf 1 1K - 327692 16,32,64,128,256,512,1024,2048,4096,8192,16384,65536 sglist 21 22K - 21 32,1024,2048 toponodes 18 3K - 18 128 stack 0 0K - 79000 256 taskqueue 90 10K - 90 16,32,256 terminal 11 3K - 11 256 Unitno 36 3K - 867494 32,64 vmem 3 16K - 4 2048,4096,8192 ioctlops 0 0K - 1683777 256,512,1024,2048,4096 select 135 17K - 135 128 iov 0 0K - 19295839 16,64,128,256,512 msg 4 89K - 4 1024,8192,16384,65536 sem 4 106K - 4 2048,4096 shm 1 32K - 1 32768 tty 13 13K - 809 1024,2048 pts 1 1K - 7 256 mbuf_tag 5 1K - 985733517 32,64 shmfd 1 8K - 1 8192 soname 14 1K - 10328680 16,32,64,128 pcb 31 407K - 1588 16,32,64,128,1024,2048,8192 md_disk 11 3K - 11 32,2048 vfscache 4 1105K - 4 512,16384,65536 cl_savebuf 0 0K - 1901 64 vfs_hash 1 512K - 1 vnodes 1 1K - 396 32,256 mount 42 2K - 224 16,32,64,128,256 statfs 0 0K - 46838 512 md_sectors 10 40K - 10 4096 vnodemarker 0 0K - 19000 512 arc4random 1 2K - 1 2048 BPF 18 1027K - 229 16,128,256,512,4096 ifnet 15 29K - 18 128,2048 ifaddr 124 29K - 131 16,32,64,128,256,512,2048,4096 ether_multi 150 9K - 152 16,32,64 clone 16 2K - 16 128 ipsec 1 1K - 1 256 lltable 77 26K - 1250 256,512 routetbl 136 24K - 1773 32,64,128,256,512 netgraph 5 1K - 10 64 netgraph_msg 0 0K - 5 128 netgraph_node 8 2K - 67 128,256 netgraph_path 0 0K - 5 16 netgraph_sock 2 1K - 106 128 igmp 14 2K - 17 128 ipid 2 24K - 2 8192,16384 in_mfilter 4 4K - 4 1024 in_multi 5 2K - 6 256 ip_moptions 8 2K - 8 64,256 encap_export_host 4 4K - 4 1024 mroutetbl 1 1K - 1 256 sctp_a_it 0 0K - 4 16 sctp_vrf 1 1K - 1 64 sctp_ifa 13 2K - 14 128 sctp_ifn 6 1K - 6 128 sctp_iter 0 0K - 4 256 hostcache 1 32K - 1 32768 tcpfunc 1 1K - 1 32 syncache 1 68K - 1 in6_mfilter 4 4K - 4 1024 in6_multi 65 9K - 65 32,256 ip6_moptions 8 2K - 8 32,256 ip6opt 5 2K - 10 256 mld 12 2K - 12 128 ip6ndp 18 3K - 23 64,256 inpcbpolicy 43 2K - 137106 32 secasvar 822 207K - 248821 256,1024 sahead 12 4K - 6464 256,1024 ipsecpolicy 32 9K - 608090 256,1024 ipsecrequest 20 3K - 608074 128 ipsec-misc 5331 173K - 1154723 16,32,64 ipsec-saq 2 2K - 133 256,1024 ipsec-reg 2 1K - 2 32 pfsync 1 1K - 1 1024 pf_hash 3 2880K - 3 pf_ifnet 65 13K - 3856 128,256,2048 pf_rule 232 382K - 3561 128,2048 pf_osfp 1184 122K - 42624 64,128 pf_table 15 30K - 295 2048 crypto 167171 329994K - 658843 64,256,512,1024,2048,4096,8192,16384,32768,65536 xform 0 0K - 368469415 16,32,64 rpc 2 16K - 2 8192 audit_evclass 187 6K - 232 32 pagedep 1 128K - 1914 256 inodedep 4 514K - 33188 512 bmsafemap 2 9K - 26045 256,8192 newblk 6 1026K - 36787 256 indirdep 0 0K - 57 128,32768 freefrag 0 0K - 5961 128 freeblks 2 1K - 24024 256 freefile 0 0K - 21366 64 diradd 0 0K - 21461 128 mkdir 0 0K - 30 128 dirrem 0 0K - 21428 128 newdirblk 0 0K - 15 64 freework 7 1K - 29964 16,128 freedep 0 0K - 4 64 jaddref 0 0K - 21491 128 jremref 0 0K - 21454 128 jmvref 0 0K - 1 128 jnewblk 0 0K - 36786 128 jfreefrag 0 0K - 5961 128 jseg 1 1K - 5057 128 jsegdep 8 1K - 85692 64 sbdep 0 0K - 3316 64 savedino 0 0K - 161 256 jblocks 2 1K - 2 128,256 softdep 1 1K - 1 512 ufs_dirhash 57 11K - 57 16,32,64,512 ufs_quota 1 512K - 1 ufs_mount 6 25K - 6 512,4096,8192 vm_pgdata 1 1K - 1 128 UMAHash 8 39K - 19 512,1024,2048,4096,8192,16384,32768 memdesc 1 4K - 1 4096 atkbddev 2 1K - 2 64 ddb_capture 1 64K - 1 65536 spkr 0 0K - 5 512 apmdev 1 1K - 1 128 madt_table 0 0K - 1 4096 entropy 0 0K - 11053 4096 acpiintr 1 1K - 1 64 io_apic 1 2K - 1 2048 acpica 1765 194K - 483433 16,32,64,128,256,512,1024,2048,8192 MCA 10 2K - 10 128 msi 32 4K - 32 128 nexusdev 6 1K - 6 16 acpitask 1 64K - 1 65536 acpisem 26 4K - 26 128 cpuctl 1 1K - 1 32System info:
Netgate Device ID: 7dbf3a7c78e98f0075e1 BIOS Vendor: Seabios Version: 0.5.1 Release Date: Mon Jan 1 2007 Version 2.4.3-RELEASE-p1 (amd64) built on Thu May 10 15:02:52 CDT 2018 FreeBSD 11.1-RELEASE-p10 Version 2.4.4 is available. Version information updated at Thu Oct 25 16:40:36 CEST 2018 CPU Type QEMU Virtual CPU version (cpu64-rhel6) 4 CPUs: 2 package(s) x 2 core(s) AES-NI CPU Crypto: No Kernel PTI EnabledI don't know what is the problem with wire memory. Do you have any idea about this issue? How can I debug it? Any contribution will be appreciated.
Thanks!!!
-
Are you using the DNS Resolver, perhaps? Maybe DNS over TLS? or DNSBL?
We found out there are some memory leaks in the version of unbound shipped with 2.4.4. They were recently fixed upstream in Unbound, and we'll have them in 2.4.4-p1 soon.
That's the only known memory leaks at the moment that I can think of.
Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.