pfSense crashes after wire memory increase



  • Hi all,

    I am getting some critical issues with wire memory. This memory is constantly increasing up to reach the memory limit. At this time the firewall is crashed. I attach the memory graphs.

    1_1540478358700_screenshot_1.png 0_1540478358700_screenshot_2.png

    I read that sometimes the ARC of ZFS is the issue, but I don't have it in my system. I have the next modules installed in PfSense:

    • sysutils
    • mail
    • net-mgmt
    • security
    • sysutils

    Also my top output is this:

    `[root@user path]#
    last pid: 28440;  load averages:  0.70,  0.73,  0.74                                                                                                                                                                                                                                                   up 1+08:53:20  16:42:16
    78 processes:  1 running, 77 sleeping
    CPU: 24.8% user,  0.0% nice,  0.0% system,  5.7% interrupt, 69.5% idle
    Mem: 36M Active, 155M Inact, 461M Wired, 25M Buf, 1300M Free
    Swap: 4096M Total, 4096M Free
    
      PID USERNAME  THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
    33810 root       17  52    0   249M 51816K sigwai  1 788:58  80.06% charon
    28440 root        1  20    0 20068K  3492K CPU3    3   0:00   0.65% top
    64989 root        1  20    0 78872K  8184K select  1   0:02   0.26% sshd
    16721 root        1  20    0 10488K  2576K select  1   2:15   0.00% syslogd
     8083 root        1  20    0 12736K  2380K bpf     0   1:28   0.00% filterlog
      316 root        1  52    0   268M 45144K accept  3   0:43   0.00% php-fpm
      317 root        1  52    0   274M 45612K accept  2   0:42   0.00% php-fpm
    10244 root        5  52    0 10988K  2320K uwait   1   0:23   0.00% dpinger
    10467 root        5  52    0 10988K  2320K uwait   2   0:22   0.00% dpinger
    37180 root        1  52   20 13096K  2596K wait    0   0:21   0.00% sh
    26615 root        1  20    0 24656K 12480K select  0   0:09   0.00% ntpd
      315 root        1  20    0   259M 26400K kqread  0   0:07   0.00% php-fpm
      387 root        1  20    0  9176K  4908K select  2   0:05   0.00% devd
    64666 nagios      1  20    0 17444K  5528K select  3   0:02   0.00% nrpe2
    25277 root        1  20    0 37704K  8416K kqread  2   0:02   0.00% nginx
    25479 root        1  20    0 37704K  8888K kqread  1   0:02   0.00% nginx
    14101 root        1  20    0 13808K  3872K wait    0   0:01   0.00% bash
    73374 root       12  20    0 72108K  9420K nanslp  2   0:00   0.00% collectd
    19795 unbound     4  20    0 81156K 22764K kqread  3   0:00   0.00% unbound
    17438 nagios      1  20    0 16800K  2420K pause   2   0:00   0.00% iostat
    68628 root        1  20    0 13400K  3884K pause   0   0:00   0.00% tcsh
    20498 root        1  20    0 53524K  6808K select  1   0:00   0.00% sshd
    26027 root        1  20    0 12504K  2360K nanslp  2   0:00   0.00% cron
    90996 root        2  20    0 10588K  2320K piperd  3   0:00   0.00% sshlockout_pf
    17577 nagios      1  20    0 14740K  2532K sbwait  1   0:00   0.00% egrep
    73618 nagios      1  20    0 14656K  3848K wait    2   0:00   0.00% ksh93
    65700 root        1  20    0  8232K  2028K nanslp  0   0:00   0.00% minicron
    43681 nagios      1  20    0 16800K  2424K pause   3   0:00   0.00% iostat
    32225 nagios      1  20    0 16800K  2424K pause   1   0:00   0.00% iostat
    10834 nagios      1  20    0 16800K  2424K pause   2   0:00   0.00% iostat
    68189 nagios      1  20    0 16800K  2424K pause   3   0:00   0.00% iostat
    34550 nagios      1  20    0 16800K  2424K pause   2   0:00   0.00% iostat
    10868 nagios      1  20    0 14740K  2532K sbwait  1   0:00   0.00% egrep
    21794 nagios      1  20    0 16800K  2424K pause   3   0:00   0.00% iostat
      354 root        1  40   20 19456K  4384K kqread  1   0:00   0.00% check_reload_status
    68413 nagios      1  20    0 14740K  2532K sbwait  2   0:00   0.00% egrep
    43807 nagios      1  20    0 14740K  2532K sbwait  1   0:00   0.00% egrep
    34770 nagios      1  20    0 14740K  2532K sbwait  3   0:00   0.00% egrep
    32292 nagios      1  20    0 14740K  2532K sbwait  1   0:00   0.00% egrep
    18168 nagios      1  20    0 10668K  2504K sbwait  1   0:00   0.00% awk
     3236 nagios      1  20    0 16800K  2424K pause   1   0:00   0.00% iostat
    22001 nagios      1  20    0 14740K  2524K sbwait  1   0:00   0.00% egrep
    17367 root        1  52    0 39440K  2844K wait    2   0:00   0.00% login
    17824 nagios      1  20    0 14740K  2536K sbwait  2   0:00   0.00% egrep
    66516 root        1  20    0  8232K  2028K nanslp  1   0:00   0.00% minicron
    65397 root        1  52    0 13096K  2812K wait    3   0:00   0.00% sh
     3435 nagios      1  20    0 14740K  2512K sbwait  3   0:00   0.00% egrep
    19010 root        1  52    0 13096K  2932K wait    3   0:00   0.00% sh
    33681 root        1  52    0 26468K  3772K select  0   0:00   0.00% starter
    35215 nagios      1  20    0 10668K  2480K sbwait  3   0:00   0.00% awk
    32392 nagios      1  20    0 10668K  2480K sbwait  3   0:00   0.00% awk
    44430 nagios      1  20    0 10668K  2480K sbwait  0   0:00   0.00% awk
    68615 nagios      1  20    0 10668K  2480K sbwait  3   0:00   0.00% awk
    11120 nagios      1  20    0 10668K  2480K sbwait  3   0:00   0.00% awk
    19134 root        1  52    0 13096K  2812K ttyin   0   0:00   0.00% sh
    68449 nagios      1  20    0 14740K  2536K sbwait  1   0:00   0.00% egrep
    10883 nagios      1  20    0 14740K  2536K sbwait  0   0:00   0.00% egrep
    34986 nagios      1  20    0 14740K  2536K sbwait  0   0:00   0.00% egrep
    44144 nagios      1  20    0 14740K  2536K sbwait  3   0:00   0.00% egrep
    32305 nagios      1  20    0 14740K  2536K sbwait  1   0:00   0.00% egrep
    18672 root        1  52    0 10396K  2140K ttyin   3   0:00   0.00% getty
    22167 nagios      1  20    0 14740K  2520K sbwait  3   0:00   0.00% egrep
    18241 root        1  52    0 10396K  2140K ttyin   3   0:00   0.00% getty
    17659 root        1  52    0 10396K  2140K ttyin   2   0:00   0.00% getty
     3439 nagios      1  20    0 14740K  2496K sbwait  1   0:00   0.00% egrep
    18531 root        1  52    0 10396K  2140K ttyin   3   0:00   0.00% getty
    17743 root        1  52    0 10396K  2140K ttyin   2   0:00   0.00% getty
    18704 root        1  52    0 10396K  2140K ttyin   0   0:00   0.00% getty
    22381 nagios      1  20    0 10668K  2476K sbwait  0   0:00   0.00% awk
     3580 nagios      1  20    0 10668K  2432K sbwait  2   0:00   0.00% awk
    18065 root        1  52    0 10396K  2140K ttyin   2   0:00   0.00% getty
    25143 root        1  52    0 35656K  6816K pause   2   0:00   0.00% nginx
     6802 root        1  52   20  6180K  1936K nanslp  3   0:00   0.00% sleep
    66020 root        1  52    0  8232K  2012K wait    2   0:00   0.00% minicron
    66667 root        1  52    0  8232K  2012K wait    2   0:00   0.00% minicron
    66977 root        1  20    0  8232K  2028K nanslp  0   0:00   0.00% minicron
    65459 root        1  52    0  8232K  2012K wait    2   0:00   0.00% minicron
      356 root        1  52   20 19456K  4200K kqread  0   0:00   0.00% check_reload_status
    
    

    vmstat -m output:

             Type InUse MemUse HighUse Requests  Size(s)
          acpidev    19     2K       -       19  64
             hdaa     5     5K       -        5  512,1024,2048
             hdac     1     1K       -        1  1024
            hdacc     1     1K       -        1  32
           feeder    12     1K       -       14  32,128
            mixer     1     4K       -        1  4096
          CAM SIM     3     1K       -        3  256
          CAM XPT    18     2K       -       34  32,64,128,512,1024,2048
          CAM DEV     4     8K       -       12  2048
          CAM CCB     0     0K       -    79012  2048
         CAM path     5     1K       -       42  32
       CAM periph     4     1K       -    39502  16,32,64,128,256
              USB    18    15K       -       18  16,128,512,4096
           USBdev    15     2K       -       15  32,64,128,256
         pci_link    10     1K       -       10  16,128
        CAM queue     7     3K       -       32  16,32,512
          scsi_cd     0     0K       -        9  16
    CAM dev queue     3     1K       -        3  64
            vtbuf    24  1968K       -       46  4096
               vt    11     6K       -       11  512
           DEVFS3   111    28K       -      131  256
           DEVFS1    86    43K       -       93  512
            DEVFS    13     1K       -       14  16,32,128
           DEVFSP     1     1K       -       57  64
    NFSD V4client     1     1K       -        1  256
     NFSD lckfile     1     1K       -        1  256
     NFSD session     1     1K       -        1  1024
      nullfs_hash     1   512K       -        1  
        pfs_nodes    21     6K       -       21  256
             GEOM    90    22K       -     3857  16,32,64,128,256,512,1024,2048,8192
        raid_data     0     0K       -       84  32,128,256
           isadev     8     1K       -        8  128
             cdev     6     2K       -        6  256
         filedesc     9    53K       -    87883  32,4096,16384
            sigio     3     1K       -        5  64
         filecaps     0     0K       -       78  16,64
          kdtrace   369    73K       -  1232497  64,256
             kenv   109    12K       -      583  16,32,64,128,8192
           kqueue   128    27K       -   616105  64,256,512,2048
        proc-args    77     6K       -   507488  16,32,64,128,256
      Fail Points     0     0K       -     7900  1024
            hhook    17     4K       -       17  32,256
          ithread   179    27K       -      180  32,128,256
           KTRACE   100    13K       -      100  128
           linker   294    56K       -      308  16,32,64,128,256,512,1024,2048,4096,8192
            lockf    11     2K       -   259343  64,128
       loginclass     3     1K       -        3  64
           devbuf  4494  4345K       -     5004  16,32,64,128,256,512,1024,2048,4096,8192,16384,65536
             temp    93    23K       -  1904882  16,32,64,128,256,512,1024,2048,4096,8192,16384,65536
          ata_pci     1     1K       -        1  64
           module   648    81K       -      655  128
         mtx_pool     2    16K       -        2  8192
              osd     3     1K       -        9  16,32,64,128,256
         pmchooks     1     1K       -        1  128
             pgrp    44     6K       -    12835  128
          session    39     5K       -      568  128
             proc     2    16K       -        2  8192
          subproc   270   510K       -   616223  512,4096
             cred    47    12K       -   208670  256
           plimit    23     6K       -     7305  256
          uidinfo     4     3K       -    28648  128,2048
           dumper     1     1K       -        1  512
           kbdmux     6    22K       -        6  16,512,1024,2048,16384
           sysctl     0     0K       -  1048972  16,32,64
        sysctloid  4069   218K       -     4188  16,32,64,128
        sysctltmp     0     0K       -    98236  16,32,64,128,256,1024
          tidhash     1    16K       -        1  16384
          callout     5  2184K       -        5  
             umtx   720    90K       -      720  128
         p1003.1b     1     1K       -        1  16
             SWAP     2   277K       -        2  64
              bus  1309   108K       -   197165  16,32,64,128,256,1024
           bus-sc    63   222K       -     3927  16,32,64,128,256,512,1024,2048,4096,8192,16384,32768,65536
          devstat     4     9K       -        4  32,4096
     eventhandler   146    12K       -      146  64,128
        taskqueue    17    18K       -       17  16,32,256,8192
             kobj   400  1600K       -      559  4096
          Per-cpu     1     1K       -        1  32
             rman   155    19K       -      513  16,32,128
             sbuf     1     1K       -   327692  16,32,64,128,256,512,1024,2048,4096,8192,16384,65536
           sglist    21    22K       -       21  32,1024,2048
        toponodes    18     3K       -       18  128
            stack     0     0K       -    79000  256
        taskqueue    90    10K       -       90  16,32,256
         terminal    11     3K       -       11  256
           Unitno    36     3K       -   867494  32,64
             vmem     3    16K       -        4  2048,4096,8192
         ioctlops     0     0K       -  1683777  256,512,1024,2048,4096
           select   135    17K       -      135  128
              iov     0     0K       - 19295839  16,64,128,256,512
              msg     4    89K       -        4  1024,8192,16384,65536
              sem     4   106K       -        4  2048,4096
              shm     1    32K       -        1  32768
              tty    13    13K       -      809  1024,2048
              pts     1     1K       -        7  256
         mbuf_tag     5     1K       - 985733517  32,64
            shmfd     1     8K       -        1  8192
           soname    14     1K       - 10328680  16,32,64,128
              pcb    31   407K       -     1588  16,32,64,128,1024,2048,8192
          md_disk    11     3K       -       11  32,2048
         vfscache     4  1105K       -        4  512,16384,65536
       cl_savebuf     0     0K       -     1901  64
         vfs_hash     1   512K       -        1  
           vnodes     1     1K       -      396  32,256
            mount    42     2K       -      224  16,32,64,128,256
           statfs     0     0K       -    46838  512
       md_sectors    10    40K       -       10  4096
      vnodemarker     0     0K       -    19000  512
       arc4random     1     2K       -        1  2048
              BPF    18  1027K       -      229  16,128,256,512,4096
            ifnet    15    29K       -       18  128,2048
           ifaddr   124    29K       -      131  16,32,64,128,256,512,2048,4096
      ether_multi   150     9K       -      152  16,32,64
            clone    16     2K       -       16  128
            ipsec     1     1K       -        1  256
          lltable    77    26K       -     1250  256,512
         routetbl   136    24K       -     1773  32,64,128,256,512
         netgraph     5     1K       -       10  64
     netgraph_msg     0     0K       -        5  128
    netgraph_node     8     2K       -       67  128,256
    netgraph_path     0     0K       -        5  16
    netgraph_sock     2     1K       -      106  128
             igmp    14     2K       -       17  128
             ipid     2    24K       -        2  8192,16384
       in_mfilter     4     4K       -        4  1024
         in_multi     5     2K       -        6  256
      ip_moptions     8     2K       -        8  64,256
    encap_export_host     4     4K       -        4  1024
        mroutetbl     1     1K       -        1  256
        sctp_a_it     0     0K       -        4  16
         sctp_vrf     1     1K       -        1  64
         sctp_ifa    13     2K       -       14  128
         sctp_ifn     6     1K       -        6  128
        sctp_iter     0     0K       -        4  256
        hostcache     1    32K       -        1  32768
          tcpfunc     1     1K       -        1  32
         syncache     1    68K       -        1  
      in6_mfilter     4     4K       -        4  1024
        in6_multi    65     9K       -       65  32,256
     ip6_moptions     8     2K       -        8  32,256
           ip6opt     5     2K       -       10  256
              mld    12     2K       -       12  128
           ip6ndp    18     3K       -       23  64,256
      inpcbpolicy    43     2K       -   137106  32
         secasvar   822   207K       -   248821  256,1024
           sahead    12     4K       -     6464  256,1024
      ipsecpolicy    32     9K       -   608090  256,1024
     ipsecrequest    20     3K       -   608074  128
       ipsec-misc  5331   173K       -  1154723  16,32,64
        ipsec-saq     2     2K       -      133  256,1024
        ipsec-reg     2     1K       -        2  32
           pfsync     1     1K       -        1  1024
          pf_hash     3  2880K       -        3  
         pf_ifnet    65    13K       -     3856  128,256,2048
          pf_rule   232   382K       -     3561  128,2048
          pf_osfp  1184   122K       -    42624  64,128
         pf_table    15    30K       -      295  2048
           crypto 167171 329994K       -   658843  64,256,512,1024,2048,4096,8192,16384,32768,65536
            xform     0     0K       - 368469415  16,32,64
              rpc     2    16K       -        2  8192
    audit_evclass   187     6K       -      232  32
          pagedep     1   128K       -     1914  256
         inodedep     4   514K       -    33188  512
        bmsafemap     2     9K       -    26045  256,8192
           newblk     6  1026K       -    36787  256
         indirdep     0     0K       -       57  128,32768
         freefrag     0     0K       -     5961  128
         freeblks     2     1K       -    24024  256
         freefile     0     0K       -    21366  64
           diradd     0     0K       -    21461  128
            mkdir     0     0K       -       30  128
           dirrem     0     0K       -    21428  128
        newdirblk     0     0K       -       15  64
         freework     7     1K       -    29964  16,128
          freedep     0     0K       -        4  64
          jaddref     0     0K       -    21491  128
          jremref     0     0K       -    21454  128
           jmvref     0     0K       -        1  128
          jnewblk     0     0K       -    36786  128
        jfreefrag     0     0K       -     5961  128
             jseg     1     1K       -     5057  128
          jsegdep     8     1K       -    85692  64
            sbdep     0     0K       -     3316  64
         savedino     0     0K       -      161  256
          jblocks     2     1K       -        2  128,256
          softdep     1     1K       -        1  512
      ufs_dirhash    57    11K       -       57  16,32,64,512
        ufs_quota     1   512K       -        1  
        ufs_mount     6    25K       -        6  512,4096,8192
        vm_pgdata     1     1K       -        1  128
          UMAHash     8    39K       -       19  512,1024,2048,4096,8192,16384,32768
          memdesc     1     4K       -        1  4096
         atkbddev     2     1K       -        2  64
      ddb_capture     1    64K       -        1  65536
             spkr     0     0K       -        5  512
           apmdev     1     1K       -        1  128
       madt_table     0     0K       -        1  4096
          entropy     0     0K       -    11053  4096
         acpiintr     1     1K       -        1  64
          io_apic     1     2K       -        1  2048
           acpica  1765   194K       -   483433  16,32,64,128,256,512,1024,2048,8192
              MCA    10     2K       -       10  128
              msi    32     4K       -       32  128
         nexusdev     6     1K       -        6  16
         acpitask     1    64K       -        1  65536
          acpisem    26     4K       -       26  128
           cpuctl     1     1K       -        1  32
    
    

    System info:

    Netgate Device ID: 7dbf3a7c78e98f0075e1	
    BIOS 	Vendor: Seabios
    Version: 0.5.1
    Release Date: Mon Jan 1 2007
    Version 	2.4.3-RELEASE-p1 (amd64)
    built on Thu May 10 15:02:52 CDT 2018
    FreeBSD 11.1-RELEASE-p10
    
    Version 2.4.4 is available.
    Version information updated at Thu Oct 25 16:40:36 CEST 2018  
    CPU Type 	QEMU Virtual CPU version (cpu64-rhel6)
    4 CPUs: 2 package(s) x 2 core(s)
    AES-NI CPU Crypto: No
    Kernel PTI 	Enabled
    

    I don't know what is the problem with wire memory. Do you have any idea about this issue? How can I debug it? Any contribution will be appreciated.

    Thanks!!!


  • Rebel Alliance Developer Netgate

    Are you using the DNS Resolver, perhaps? Maybe DNS over TLS? or DNSBL?

    We found out there are some memory leaks in the version of unbound shipped with 2.4.4. They were recently fixed upstream in Unbound, and we'll have them in 2.4.4-p1 soon.

    That's the only known memory leaks at the moment that I can think of.