Memory Leak



  • I have a pfSense 2.1 (amd64) virtual firewall running under QEMU with 2 CPU cores & 1GB RAM.  I am experiencing a slow memory leak that will consume memory at the rate of about 1MB every 90 minutes.  This firewall has a fairly static configuration, and the memory leak seems to be triggered by some unknown event outside of any administrative changes.

    I've attached a couple of RRD graphs showing the leaks.

    Here are some notes on what we've discovered so far:

    1.  This problem occurred under both pfSense 2.0.2 and pfSense 2.1.

    2.  When the problem occurs the performance of the entire system seems to slow.  The most noticeable way this manifests itself is in voice-quality issues on VoIP connections.

    3.  The leak is preceded by log entries like this.  As time passes, less and less tftp-proxy requests go through while more and more display this error. 
    Oct  4 00:07:24 [daemon.err] pf connection lookup failed (no rdr?)
    Oct  4 00:07:24 [daemon.warning] /usr/libexec/tftp-proxy[35113]: exited, status 1

    4.  At about the same time the leak starts, the log lines above turn into lines like this.
    Oct  4 17:09:30 [daemon.err] couldn't forward tftp packet: Operation not permitted
    Oct  4 17:09:30 [daemon.warning] /usr/libexec/tftp-proxy[28967]: exited, status 1

    5.  The memory leak does not appear to be in user space.  No user processes appear to be consuming huge amounts of memory.

    6.  CPU usage is low during the problem.  Occasionally dnsmasq will consume 20-30% of the CPU, but most of the time the CPU usage is <2%.

    7.  Rebooting the firewall fixes the problem for anywhere from a couple of days to a couple of months.

    Here is the output of "ps uxawww":

    USER     PID %CPU %MEM   VSZ   RSS  TT  STAT STARTED      TIME COMMAND
    root      11 112.0  0.0     0    32  ??  RL   23Sep13 38419:04.70 [idle]
    root       0 55.0  0.0     0   208  ??  DLs  23Sep13 527:25.26 [kernel]
    root      12 12.0  0.0     0   240  ??  WL   23Sep13 474:09.17 [intr]
    nobody  7850  5.0  0.3 10100  3108  ??  S    11:49AM   1:16.75 [dnsmasq]
    root       1  0.0  0.1  3200   584  ??  ILs  23Sep13   0:00.14 /sbin/init --
    root       2  0.0  0.0     0    16  ??  DL   23Sep13   0:00.04 [g_event]
    root       3  0.0  0.0     0    16  ??  DL   23Sep13   9:52.53 [g_up]
    root       4  0.0  0.0     0    16  ??  DL   23Sep13   8:12.40 [g_down]
    root       5  0.0  0.0     0    16  ??  DL   23Sep13   0:00.00 [crypto]
    root       6  0.0  0.0     0    16  ??  DL   23Sep13   0:00.00 [crypto returns]
    root       7  0.0  0.0     0    16  ??  DL   23Sep13   0:00.00 [sctp_iterator]
    root       8  0.0  0.0     0    16  ??  DL   23Sep13   1:48.33 [pfpurge]
    root       9  0.0  0.0     0    16  ??  DL   23Sep13   0:00.00 [xpt_thrd]
    root      10  0.0  0.0     0    16  ??  DL   23Sep13   0:00.00 [audit]
    root      13  0.0  0.0     0    32  ??  DL   23Sep13   1:58.59 [ng_queue]
    root      14  0.0  0.0     0    16  ??  DL   23Sep13   4:10.67 [yarrow]
    root      15  0.0  0.0     0    64  ??  DL   23Sep13   0:22.20 [usb]
    root      16  0.0  0.0     0    16  ??  DL   23Sep13   0:04.57 [pagedaemon]
    root      17  0.0  0.0     0    16  ??  DL   23Sep13   0:00.00 [vmdaemon]
    root      18  0.0  0.0     0    16  ??  DL   23Sep13   0:00.04 [pagezero]
    root      19  0.0  0.0     0    16  ??  DL   23Sep13   0:04.88 [idlepoll]
    root      20  0.0  0.0     0    16  ??  DL   23Sep13   0:08.57 [bufdaemon]
    root      21  0.0  0.0     0    16  ??  DL   23Sep13   1:57.08 [syncer]
    root      22  0.0  0.0     0    16  ??  DL   23Sep13   0:09.88 [vnlru]
    root      23  0.0  0.0     0    16  ??  DL   23Sep13   0:10.54 [softdepflush]
    root      36  0.0  0.0     0    32  ??  DL   23Sep13   0:22.39 [zfskern]
    root      66  0.0  0.0     0    16  ??  DL   23Sep13   0:42.69 [md0]
    root     264  0.0  0.1  6908  1388  ??  INs  23Sep13   5:41.09 /usr/local/sbin/check_reload_status
    root     269  0.0  0.1  6908  1284  ??  IN   23Sep13   0:00.00 check_reload_status: Monitoring daemon of check_reload_status
    root     275  0.0  0.3  5248  3148  ??  Is   23Sep13   0:00.06 /sbin/devd
    root    6842  0.0  0.1  5780  1444  ??  Is   11:49AM   0:00.08 /usr/local/sbin/dhcpleases -l /var/dhcpd/var/db/dhcpd.leases -d example.com -p /var/run/dnsmasq.pid -h /var/etc/hosts
    dhcpd  11788  0.0  1.2 17104 12516  ??  Ss   11:49AM   0:00.45 /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid em0 em3
    root   13148  0.0  0.7 15264  7164  ??  Ss   11:49AM   0:00.77 /usr/local/bin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid
    root   15208  0.0  0.1  7036  1320  ??  Is   23Sep13   0:00.07 /usr/local/sbin/sshlockout_pf 15
    root   15439  0.0  0.2  8984  1624  ??  Is   23Sep13   0:38.31 /usr/sbin/inetd -wW -R 0 -a 127.0.0.1 /var/etc/inetd.conf
    root   15899  0.0  0.2  6956  1660  ??  Ss   27Sep13   5:44.31 /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -f /var/etc/syslog.conf
    root   16509  0.0  0.6 13796  6264  ??  S    27Sep13   0:40.78 /usr/sbin/tcpdump -s 256 -v -S -l -n -e -ttt -i pflog0
    root   16751  0.0  0.1  5780  1148  ??  S    27Sep13   0:22.68 logger -t pf -p local0.info
    root   17261  0.0  0.1  7036  1368  ??  Is   27Sep13   0:00.05 /usr/local/sbin/sshlockout_pf 15
    root   17591  0.0  0.1  5780  1472  ??  Ss   23Sep13  13:30.58 /usr/local/sbin/apinger -c /var/etc/apinger.conf
    root   17754  0.0  0.2 14384  2352  ??  I    23Sep13   0:07.53 /usr/local/bin/rrdtool -
    root   29260  0.0  0.4 15268  3752  ??  INs  11:49AM   0:00.00 /usr/sbin/sshd
    root   34568  0.0  0.4 24220  4308  ??  SN   11:49AM   0:00.49 /usr/local/sbin/lighttpd -f /var/etc/lighty-webConfigurator.conf
    root   35228  0.0  0.2  8296  1928  ??  SN   11:49AM   0:00.29 /bin/sh /var/db/rrd/updaterrd.sh
    root   35431  0.0  0.1  6812  1252  ??  Is    9:12AM   0:00.45 /usr/local/sbin/qstats -p /var/run/qstats.pid
    root   37915  0.0  0.4 23232  4556  ??  Is   23Sep13   0:06.76 /usr/local/sbin/mpd4 -b -d /var/etc/pptp-vpn -p /var/run/pptp-vpn.pid -s pptps pptps
    root   39676  0.0  0.1  2716   968  ??  SN   12:12PM   0:00.00 sleep 60
    root   41395  0.0  0.4 16776  4432  ??  Ss   23Sep13   3:58.02 /usr/local/sbin/racoon -f /var/etc/ipsec/racoon.conf
    root   41921  0.0  0.2  7168  1916  ??  Is   23Sep13   2:45.46 /usr/local/sbin/filterdns -p /var/run/filterdns-ipsec.pid -i 60 -c /var/etc/ipsec/filterdns-ipsec.hosts -d 1
    root   42087  0.0  3.7 145192 37500  ??  S    11:56AM   0:04.79 /usr/local/bin/php
    root   44922  0.0  1.1 30464 11520  ??  Ss   23Sep13   2:41.21 /usr/sbin/bsnmpd -c /var/etc/snmpd.conf -p /var/run/snmpd.pid
    root   59046  0.0  0.1  7036  1368  ??  Is   27Sep13   0:00.12 /usr/local/sbin/sshlockout_pf 15
    root   63668  0.0  0.4 26168  4272  ??  Ss   10:08AM   0:03.25 sshd: root@pts/2 (sshd)
    root   65508  0.0  0.4 26168  4272  ??  Ss    8:56AM   0:09.27 sshd: root@pts/0 (sshd)
    root   66403  0.0  2.1 139048 21968  ??  Is   10:43AM   0:00.18 /usr/local/bin/php
    root   69793  0.0  2.1 139048 21968  ??  Is   10:43AM   0:00.11 /usr/local/bin/php
    root   74436  0.0  4.2 143144 42760  ??  I    10:43AM   0:02.27 /usr/local/bin/php
    root   80776  0.0  0.2  7928  1620  ??  Ss   23Sep13   0:02.43 /usr/sbin/cron -s
    root   82170  0.0  0.1  5784  1184  ??  Is   23Sep13   0:00.00 /usr/local/bin/minicron 240 /var/run/ping_hosts.pid /usr/local/bin/ping_hosts.sh
    root   82404  0.0  0.1  5784  1232  ??  I    23Sep13   0:01.70 minicron: helper /usr/local/bin/ping_hosts.sh  (minicron)
    root   82519  0.0  0.1  5784  1184  ??  Is   23Sep13   0:00.00 /usr/local/bin/minicron 3600 /var/run/expire_accounts.pid /etc/rc.expireaccounts
    root   82802  0.0  0.1  5784  1232  ??  I    23Sep13   0:00.11 minicron: helper /etc/rc.expireaccounts  (minicron)
    root   83415  0.0  0.1  5784  1184  ??  Is   23Sep13   0:00.00 /usr/local/bin/minicron 86400 /var/run/update_alias_url_data.pid /etc/rc.update_alias_url_data
    root   83460  0.0  0.1  5784  1232  ??  I    23Sep13   0:00.01 minicron: helper /etc/rc.update_alias_url_data  (minicron)
    root   89243  0.0  0.1  7036  1364  ??  Is   23Sep13   0:00.09 /usr/local/sbin/sshlockout_pf 15
    root   91238  0.0  0.4 26168  4272  ??  Is    8:57AM   0:04.30 sshd: root@pts/1 (sshd)
    root   93548  0.0  0.1  7036  1368  ??  Is   27Sep13   0:00.05 /usr/local/sbin/sshlockout_pf 15
    root   98081  0.0  0.1  5780  1264  ??  Is   11:49AM   0:00.00 /usr/local/sbin/choparp em3 auto 64.198.147.131/32
    root   88803  0.0  0.2 19480  1844  v0  Is   23Sep13   0:00.01 login [pam] (login)
    root   89468  0.0  0.2  8296  1800  v0  I    23Sep13   0:00.00 -sh (sh)
    root   92299  0.0  0.2  8296  1800  v0  I+   23Sep13   0:00.01 /bin/sh /etc/rc.initial
    root   88913  0.0  0.2 19480  1844  v1  Is   23Sep13   0:00.02 login [pam] (login)
    root   89785  0.0  0.2  8296  1800  v1  I    23Sep13   0:00.00 -sh (sh)
    root   91982  0.0  0.2  8296  1800  v1  I+   23Sep13   0:00.00 /bin/sh /etc/rc.initial
    root   34966  0.0  0.2  9324  2372   0  S+   10:54AM   0:03.77 top
    root   87219  0.0  0.2  8296  1800   0  Is    8:56AM   0:00.01 -sh (sh)
    root   88110  0.0  0.2  8296  1804   0  I     8:56AM   0:00.01 /bin/sh /etc/rc.initial
    root   90558  0.0  0.3  8268  2864   0  I     8:56AM   0:00.16 /bin/tcsh
    root   30839  0.0  0.3  8268  2644   1  I     9:30AM   0:00.29 /bin/tcsh
    root   87723  0.0  0.1  2716   968   1  I+   12:11PM   0:00.00 sleep 60
    root   91596  0.0  0.2  8296  1848   1  Is    8:57AM   0:00.01 -sh (sh)
    root   92587  0.0  0.2  8296  1852   1  I     8:57AM   0:00.01 /bin/sh /etc/rc.initial
    root   39919  0.0  0.1  7992  1532   2  R+   12:12PM   0:00.00 ps uxawww
    root   74148  0.0  0.2  8296  1876   2  Is   10:08AM   0:00.00 -sh (sh)
    root   75219  0.0  0.2  8296  1880   2  I    10:08AM   0:00.01 /bin/sh /etc/rc.initial
    root   85289  0.0  0.3  8268  2860   2  S    10:43AM   0:00.23 /bin/tcsh
    

    I am completely stumped as to the cause of this problem.  Any help would be greatly appreciated.  Thanks!

    -Jason Crowley | Innovative Networks





  • Im seeing something too that Ive been watching…  I reboot the box and start out at 5% memory usage according to the dashboard.  Today (bottom) its sitting at 30%.

    Im running Siproxd, LCDproc dev, shellcmd, file manager, pfblocker and System patches with a widescreen mod.  It seems when I restart siproxd it goes back down... Restarting the others has no effect.

    
    last pid:  4328;  load averages:  0.28,  0.51,  0.29  up 0+00:04:22    23:16:07
    115 processes: 3 running, 97 sleeping, 15 waiting
    
    Mem: 57M Active, 15M Inact, 75M Wired, 248K Cache, 24M Buf, 1835M Free
    Swap: 2048M Total, 2048M Free
    
      PID USERNAME PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
       10 root     171 ki31     0K     8K RUN      3:10 96.97% [idle]
    51713 root      76    0 78996K 30068K piperd   0:05  1.95% /usr/local/bin/php{php}
       11 root     -68    -     0K   128K RUN      0:02  1.95% [intr{irq16: mskc0 skc}]
        0 root     -16    0     0K    64K sched    0:50  0.00% [kernel{swapper}]
      255 root      76   20  3352K  1204K kqread   0:24  0.00% /usr/local/sbin/check_reload_status
       11 root     -32    -     0K   128K WAIT     0:01  0.00% [intr{swi4: clock}]
    45472 root      76    0 76820K 24328K accept   0:01  0.00% /usr/local/bin/php
       14 root     -40    -     0K   160K -        0:00  0.00% [usb{usbus4}]
       14 root     -40    -     0K   160K -        0:00  0.00% [usb{usbus3}]
       14 root     -40    -     0K   160K -        0:00  0.00% [usb{usbus0}]
       14 root     -40    -     0K   160K -        0:00  0.00% [usb{usbus1}]
       14 root     -40    -     0K   160K -        0:00  0.00% [usb{usbus2}]
        1 root      76    0  1888K   492K wait     0:00  0.00% [init]
    44008 root      44    0  6980K  4844K kqread   0:00  0.00% /usr/local/sbin/lighttpd -f /var/etc/lighty-we
       13 root     -16    -     0K     8K -        0:00  0.00% [yarrow]
    63464 nobody    74  r30  3316K  1360K nanslp   0:00  0.00% /usr/pbi/lcdproc-i386/sbin/LCDd -c /usr/local/
        4 root      -8    -     0K     8K -        0:00  0.00% [g_down]
    94747 root      76   20  3644K  1492K wait     0:00  0.00% /bin/sh /usr/local/sbin/fanctrld.sh
    
    last pid: 25864;  load averages:  0.12,  0.05,  0.01  up 1+21:28:30    11:53:05
    116 processes: 4 running, 96 sleeping, 16 waiting
    
    Mem: 344M Active, 17M Inact, 82M Wired, 236K Cache, 30M Buf, 1539M Free
    Swap: 2048M Total, 2048M Free
    
      PID USERNAME PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
       10 root     171 ki31     0K     8K RUN     43.5H 100.00% [idle]
    51903 root      61    0 77972K 28316K piperd   0:01  0.98% /usr/local/bin/php{php}
       11 root     -32    -     0K   128K WAIT    17:24  0.00% [intr{swi4: clock}]
    65599 root      76   20   202M   152M nanslp   5:57  0.00% /usr/local/bin/php -f /usr/local/pkg/lcdproc_c
    65078 root      76   20   202M   152M nanslp   5:57  0.00% /usr/local/bin/php -f /usr/local/pkg/lcdproc_c
       11 root     -68    -     0K   128K WAIT     3:00  0.00% [intr{irq16: mskc0 skc}]
      255 root      76   20  3352K  1204K kqread   1:25  0.00% /usr/local/sbin/check_reload_status
        0 root     -16    0     0K    64K sched    1:24  0.00% [kernel{swapper}]
    65364 nobody    74  r30  3316K  1432K RUN      0:54  0.00% /usr/pbi/lcdproc-i386/sbin/LCDd -c /usr/local/
     2034 root      76   20  3644K  1492K wait     0:48  0.00% /bin/sh /usr/local/sbin/fanctrld.sh
     4340 root      76    0  3644K  1492K wait     0:47  0.00% /bin/sh /usr/local/sbin/fanctrld.sh
    13568 root      76   20  3644K  1536K wait     0:16  0.00% /bin/sh /var/db/rrd/updaterrd.sh
       13 root     -16    -     0K     8K -        0:15  0.00% [yarrow]
       11 root     -44    -     0K   128K WAIT     0:15  0.00% [intr{swi1: netisr 0}]
    33160 root      44    0  3264K  1244K select   0:12  0.00% /usr/local/sbin/apinger -c /var/etc/apinger.co
    94971 root      44    0  8808K  6068K select   0:07  0.00% /usr/pbi/siproxd-i386/sbin/siproxd -c /usr/pbi
    44839 root      44    0 76820K 29588K accept   0:06  0.00% /usr/local/bin/php
        3 root      -8    -     0K     8K -        0:05  0.00% [g_up]
    
    last pid: 45195;  load averages:  0.00,  0.01,  0.00  up 4+23:37:23    14:01:58
    117 processes: 4 running, 97 sleeping, 16 waiting
    
    Mem: 279M Active, 20M Inact, 86M Wired, 232K Cache, 34M Buf, 1597M Free
    Swap: 2048M Total, 2048M Free
    
      PID USERNAME PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
       10 root     171 ki31     0K     8K RUN    115.2H 100.00% [idle]
       11 root     -32    -     0K   128K WAIT    44:44  0.00% [intr{swi4: clock}]
       11 root     -68    -     0K   128K WAIT    10:41  0.00% [intr{irq16: mskc0 skc}]
     1749 root      76   20   277M   228M nanslp   9:55  0.00% /usr/local/bin/php -f /usr/local/pkg/lcdproc_c
      255 root      76   20  3352K  1204K kqread   2:18  0.00% /usr/local/sbin/check_reload_status
     2034 root      76   20  3644K  1492K wait     2:06  0.00% /bin/sh /usr/local/sbin/fanctrld.sh
     4340 root      76    0  3644K  1492K wait     2:05  0.00% /bin/sh /usr/local/sbin/fanctrld.sh
        0 root     -16    0     0K    64K sched    1:24  0.00% [kernel{swapper}]
     1469 nobody    74  r30  3316K  1428K RUN      1:17  0.00% /usr/pbi/lcdproc-i386/sbin/LCDd -c /usr/local/
       13 root     -16    -     0K     8K -        0:51  0.00% [yarrow]
       11 root     -68    -     0K   128K WAIT     0:44  0.00% [intr{irq19: mskc3 skc}]
    13568 root      76   20  3644K  1536K wait     0:42  0.00% /bin/sh /var/db/rrd/updaterrd.sh
       11 root     -44    -     0K   128K WAIT     0:30  0.00% [intr{swi1: netisr 0}]
    33160 root      44    0  3264K  1244K select   0:30  0.00% /usr/local/sbin/apinger -c /var/etc/apinger.co
    43794 root      44    0  8004K  5628K kqread   0:24  0.00% /usr/local/sbin/lighttpd -f /var/etc/lighty-we
    65053 root      64   20  5432K  3784K select   0:22  0.00% /usr/local/sbin/openvpn --config /var/etc/open
        3 root      -8    -     0K     8K -        0:14  0.00% [g_up]
     6671 root      64   20  6280K  6300K select   0:12  0.00% /usr/local/bin/ntpd -g -c /var/etc/ntpd.conf -
    
    


  • An identical watchguard- No siproxd or pfblocker…

    edit= which does not exhibit the behavior...

    last pid: 53797;  load averages:  0.04,  0.03,  0.00  up 13+19:32:07    19:07:20
    110 processes: 4 running, 90 sleeping, 16 waiting
    
    Mem: 108M Active, 18M Inact, 79M Wired, 1036K Cache, 61M Buf, 1784M Free
    Swap: 
    
      PID USERNAME PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
       10 root     171 ki31     0K     8K RUN    321.5H 98.97% [idle]
       11 root     -32    -     0K   128K WAIT   101:51  0.00% [intr{swi4: clock}]
    80288 root      76   20   122M 61656K RUN     37:12  0.00% /usr/local/bin/php -f /usr/local/pkg/lcdproc_c
       11 root     -68    -     0K   128K WAIT    33:22  0.00% [intr{irq16: skc0 uhci}]
       11 root     -68    -     0K   128K WAIT    21:12  0.00% [intr{irq17: skc1}]
    80188 nobody    74  r30  3316K  1424K nanslp   7:39  0.00% /usr/pbi/lcdproc-i386/sbin/LCDd -c /usr/local/
       13 root     -16    -     0K     8K -        4:38  0.00% [yarrow]
    20489 root      44    0  3264K  1232K select   3:07  0.00% /usr/local/sbin/apinger -c /var/etc/apinger.co
       11 root     -44    -     0K   128K WAIT     1:52  0.00% [intr{swi1: netisr 0}]
        0 root     -16    0     0K    56K sched    1:52  0.00% [kernel{swapper}]
    44231 root      76   20  3644K  1436K wait     1:44  0.00% /bin/sh /var/db/rrd/updaterrd.sh
      296 root      76   20  3352K  1180K kqread   0:54  0.00% /usr/local/sbin/check_reload_status
    79315 root      64   20  6280K  6300K select   0:49  0.00% /usr/local/bin/ntpd -g -c /var/etc/ntpd.conf -
    33590 dhcpd     64   20 11456K  8052K select   0:46  0.00% /usr/local/sbin/dhcpd -user dhcpd -group _dhcp
    30371 nobody    64   20  5512K  2868K select   0:39  0.00% /usr/local/sbin/dnsmasq --all-servers --rebind
       49 root      -8    -     0K     8K mdwait   0:31  0.00% [md1]
       20 root      20    -     0K     8K syncer   0:27  0.00% [syncer]
    29805 root      44    0  8004K  5484K kqread   0:23  0.00% /usr/local/sbin/lighttpd -f /var/etc/lighty-we
    
    pfSense is © 2004 - 2013 by Electric Sheep Fencing LLC. All Righ
    


  • Note that the memory usage I see increasing is "Wired" memory.  Also, it's interesting to note that kernel tasks related to NICs show up at the top of the list.  The em0 and em1 interfaces are LAN and WAN respectively.

    Here is my output from "top -SH" during the problem.

    last pid: 82067;  load averages:  2.83,  1.79,  1.22                                       up 13+19:23:15  12:14:23
    129 processes: 6 running, 108 sleeping, 15 waiting
    CPU:  0.0% user,  0.5% nice, 93.6% system,  0.2% interrupt,  5.7% idle
    Mem: 103M Active, 30M Inact, 315M Wired, 164K Cache, 71M Buf, 523M Free
    Swap: 1024M Total, 1024M Free
    
      PID USERNAME PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
        0 root     -68    0     0K   208K -       0 192:25 78.96% kernel{em0 taskq}
        0 root     -68    0     0K   208K -       1 243:36 36.96% kernel{em1 taskq}
     7850 nobody    76    0 10100K  3140K select  1   1:43 34.96% dnsmasq
       11 root     171 ki31     0K    32K RUN     1 322.6H 17.97% idle{idle: cpu1}
       11 root     171 ki31     0K    32K RUN     0 317.8H 11.96% idle{idle: cpu0}
       12 root     -44    -     0K   240K WAIT    0 193:38  8.98% intr{swi1: netisr 0}
        0 root     -68    0     0K   208K -       1  91:55  1.95% kernel{em3 taskq}
        0 root     -16    0     0K   208K sched   0 575.4H  0.00% kernel{swapper}
       12 root     -32    -     0K   240K WAIT    0 116:57  0.00% intr{swi4: clock}
       12 root     -44    -     0K   240K WAIT    1  72:41  0.00% intr{swi1: netisr 1}
       12 root     -68    -     0K   240K WAIT    0  70:08  0.00% intr{irq11: em1 em2+}
       12 root     -32    -     0K   240K WAIT    0  17:21  0.00% intr{swi4: clock}
    17591 root      44    0  5780K  1472K select  1  13:31  0.00% apinger
        3 root      -8    -     0K    16K -       0   9:53  0.00% g_up
        4 root      -8    -     0K    16K -       1   8:12  0.00% g_down
    15899 root      44    0  6956K  1660K select  1   5:45  0.00% syslogd
      264 root      76   20  6908K  1388K kqread  0   5:41  0.00% check_reload_status
       14 root     -16    -     0K    16K -       0   4:11  0.00% yarrow
    41395 root      44    0 16776K  4432K select  0   3:58  0.00% racoon
       12 root     -64    -     0K   240K WAIT    0   3:41  0.00% intr{irq14: ata0}
    41921 root      44    0  7168K  1916K ucond   0   2:46  0.00% filterdns{vpn.in-kc.com}
    44922 root      44    0 30464K 11520K select  1   2:41  0.00% bsnmpd
       21 root      20    -     0K    16K syncer  1   1:57  0.00% syncer
        8 root     -16    -     0K    16K pftm    1   1:49  0.00% pfpurge
       13 root     -16    -     0K    32K sleep   0   0:59  0.00% ng_queue{ng_queue0}
       13 root     -16    -     0K    32K sleep   0   0:59  0.00% ng_queue{ng_queue1}
        0 root     -68    0     0K   208K -       0   0:54  0.00% kernel{em2 taskq}
       66 root      -8    -     0K    16K mdwait  1   0:43  0.00% md0
    16509 root      44    0 13796K  6264K bpf     1   0:41  0.00% tcpdump
    15439 root      44    0  8984K  1624K select  0   0:38  0.00% inetd
    16751 root      44    0  5780K  1148K piperd  0   0:23  0.00% logger
       15 root     -40    -     0K    64K -       0   0:22  0.00% usb{usbus0}
       36 root      -8    -     0K    32K arc_re  1   0:12  0.00% zfskern{arc_reclaim_thre}
       23 root     -16    -     0K    16K sdflus  0   0:11  0.00% softdepflush
       36 root      -8    -     0K    32K l2arc_  1   0:10  0.00% zfskern{l2arc_feed_threa}
       22 root     -16    -     0K    16K vlruwt  1   0:10  0.00% vnlru
    65508 root      44    0 26168K  4272K select  1   0:10  0.00% sshd
       20 root     -16    -     0K    16K psleep  1   0:09  0.00% bufdaemon
    37915 root      44    0 23232K  4556K select  1   0:07  0.00% mpd4{mpd4}
    42087 root      76    0   142M 38032K accept  0   0:05  0.00% php
       19 root      76 ki-6     0K    16K pollid  0   0:05  0.00% idlepoll
       16 root     -16    -     0K    16K psleep  0   0:05  0.00% pagedaemon
    91238 root      44    0 26168K  4272K select  1   0:04  0.00% sshd
    63668 root      44    0 26168K  4272K select  0   0:03  0.00% sshd
    34966 root      44    0  9324K  2372K select  1   0:03  0.00% top
    80776 root      44    0  7928K  1620K nanslp  0   0:02  0.00% cron
    74436 root      52    0   140M 42760K accept  1   0:02  0.00% php
    82404 root      76    0  5784K  1232K nanslp  0   0:02  0.00% minicron
    17754 root      44    0 14384K  2352K piperd  0   0:01  0.00% rrdtool
    13148 root      44    0 15264K  7164K select  1   0:01  0.00% ntpd
    34568 root      64   20 24220K  4308K kqread  1   0:01  0.00% lighttpd
    11788 dhcpd     44    0 17104K 12516K select  1   0:00  0.00% dhcpd
    35431 root      44    0  6812K  1252K accept  0   0:00  0.00% qstats
    35228 root      76   20  8296K  1928K piperd  0   0:00  0.00% sh
    30839 root      45    0  8268K  2644K pause   0   0:00  0.00% tcsh
    85289 root      44    0  8268K  2860K pause   1   0:00  0.00% tcsh
    66403 root      76    0   136M 21968K wait    0   0:00  0.00% php
    90558 root      44    0  8268K  2864K pause   1   0:00  0.00% tcsh
        1 root      44    0  3200K   584K wait    1   0:00  0.00% init
    69793 root      76    0   136M 21968K wait    0   0:00  0.00% php
    82802 root      46    0  5784K  1232K nanslp  1   0:00  0.00% minicron
     6842 root      44    0  5780K  1444K kqread  0   0:00  0.00% dhcpleases
    59046 root      44    0  7036K  1368K uwait   0   0:00  0.00% sshlockout_pf{initial thread}
      275 root      44    0  5248K  3148K select  1   0:00  0.00% devd
    15208 root      44    0  7036K  1320K nanslp  0   0:00  0.00% sshlockout_pf{sshlockout_pf}
    89243 root      44    0  7036K  1364K nanslp  1   0:00  0.00% sshlockout_pf{sshlockout_pf}
    
    

  • Banned

    I dont see that either on ESXi and no siproxd here.



  • Im up to 40% this morning and restarting any service has no effect this time. Im going to have to reboot the box before I leave to make sure it stays working throughout the day.



  • Not seeing any issues here, I'm on 2.1 Release 64Bit. At work I'm using 2.1 RC0 from May 32 Bit and no issue there as well it's been running rock solid for months without any resets. Just wanted to add another data point if it helps. No proxies or anything other packages. I have a Open VPN site to site tunnel to various sites around the State. Multiple VLANs (10 or so) and Dual WANs 100/25 connections.


  • Banned

    So it seems that siproxd is the culprit so far….



  • I dont see where the OP has Siproxd running on his setup…  Am I missing it?

    Ive got several boxes running in other locations without this issue.



  • No such issues here, running squid, DG, ntop, OpenVPN, IPSec tunnel, Sarg, and pfBlocker.

    
    USER     PID %CPU %MEM   VSZ   RSS  TT  STAT STARTED      TIME COMMAND
    root      11 650.8  0.0     0   128  ??  RL   29Sep13 101590:37.68 [idle]
    root   58837 90.0  0.0 21668  3940  ??  R     4:00PM  10:05.57 /usr/pbi/sarg-amd64/bin/sarg
    root       0  0.0  0.0     0   320  ??  DLs  29Sep13 104:29.50 [kernel]
    root       1  0.0  0.0  3200   388  ??  ILs  29Sep13   0:00.23 /sbin/init --
    root       2  0.0  0.0     0    16  ??  DL   29Sep13   0:01.17 [g_event]
    root       3  0.0  0.0     0    16  ??  DL   29Sep13  22:33.14 [g_up]
    root       4  0.0  0.0     0    16  ??  DL   29Sep13  31:32.75 [g_down]
    root       5  0.0  0.0     0    16  ??  DL   29Sep13   0:00.00 [crypto]
    root       6  0.0  0.0     0    16  ??  DL   29Sep13   0:00.00 [crypto returns]
    root       7  0.0  0.0     0    16  ??  DL   29Sep13   0:00.00 [mpt_recovery0]
    root       8  0.0  0.0     0    16  ??  DL   29Sep13   0:02.85 [fdc0]
    root       9  0.0  0.0     0    16  ??  DL   29Sep13   0:00.00 [sctp_iterator]
    root      10  0.0  0.0     0    16  ??  DL   29Sep13   0:00.00 [audit]
    root      12  0.0  0.0     0   464  ??  WL   29Sep13 198:47.13 [intr]
    root      13  0.0  0.0     0   128  ??  DL   29Sep13   0:00.00 [ng_queue]
    root      14  0.0  0.0     0    16  ??  DL   29Sep13   5:32.45 [yarrow]
    root      15  0.0  0.0     0    16  ??  DL   29Sep13   1:12.62 [pfpurge]
    root      16  0.0  0.0     0    16  ??  DL   29Sep13   0:00.00 [xpt_thrd]
    root      17  0.0  0.0     0    16  ??  DL   29Sep13   0:01.26 [pagedaemon]
    root      18  0.0  0.0     0    16  ??  DL   29Sep13   0:00.00 [vmdaemon]
    root      19  0.0  0.0     0    16  ??  DL   29Sep13   0:00.01 [pagezero]
    root      20  0.0  0.0     0    16  ??  DL   29Sep13   0:00.76 [idlepoll]
    root      21  0.0  0.0     0    16  ??  DL   29Sep13   0:43.36 [bufdaemon]
    root      22  0.0  0.0     0    16  ??  DL   29Sep13  14:33.51 [syncer]
    root      23  0.0  0.0     0    16  ??  DL   29Sep13   0:39.40 [vnlru]
    root      24  0.0  0.0     0    16  ??  DL   29Sep13   0:04.24 [softdepflush]
    root      37  0.0  0.0     0    32  ??  DL   29Sep13   0:07.04 [zfskern]
    root      63  0.0  0.0     0    16  ??  DL   29Sep13   0:05.05 [md0]
    root     262  0.0  0.0  6908  1320  ??  INs  29Sep13 325:27.54 /usr/local/sbin/check_reload_status
    root     267  0.0  0.0  6908  1216  ??  IN   29Sep13   0:00.00 check_reload_status: Monitoring daemon of check_reload_status
    root     278  0.0  0.0  5248  2972  ??  Is   29Sep13   0:00.01 /sbin/devd
    root     435  0.0  0.4 143144 32720  ??  I     3:42PM   0:05.83 /usr/local/bin/php
    root     540  0.0  0.0 16776  4124  ??  Ss    1Oct13   0:15.90 /usr/local/sbin/racoon -f /var/etc/ipsec/racoon.conf
    root     658  0.0  0.0  7928  1524  ??  Ss   29Sep13   0:01.89 /usr/sbin/cron -s
    root    3365  0.0  0.1 15264  7164  ??  Ss   29Sep13   0:26.02 /usr/local/bin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid
    root    6393  0.0  0.3 28308 24808  ??  Is   Mon03PM   0:06.52 /usr/local/sbin/dhcpleases -l /var/dhcpd/var/db/dhcpd.leases -d mrpcap.com -p /var/run/dnsmasq.pid -h /var/etc/hosts
    nobody  9927  0.0  0.0 10100  2912  ??  S    Mon03PM   0:30.61 [dnsmasq]
    root    9994  0.0  0.0 15268  2580  ??  Is   29Sep13   0:00.01 /usr/sbin/sshd
    root   10260  0.0  0.0  7036  1272  ??  Is   29Sep13   0:00.02 /usr/local/sbin/sshlockout_pf 15
    dhcpd  12656  0.0  0.1 17104 11804  ??  Ss   Mon03PM   0:06.78 /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid em1
    root   20627  0.0  0.1 23744  5776  ??  Is   Mon04PM   0:00.00 /usr/pbi/squid-amd64/sbin/squid -f /usr/pbi/squid-amd64/etc/squid/squid.conf
    proxy  21192  0.0  0.8 85192 62772  ??  S    Mon04PM  24:45.99 (squid) -f /usr/pbi/squid-amd64/etc/squid/squid.conf (squid)
    proxy  21468  0.0  0.0  7056  1620  ??  I    Mon04PM   0:00.00 (unlinkd) (unlinkd)
    root   22372  0.0  0.1 13488  4440  ??  Ss   29Sep13   0:49.46 /usr/local/sbin/openvpn --config /var/etc/openvpn/server1.conf
    root   22494  0.0  0.1 13488  4216  ??  Ss   29Sep13   0:04.06 /usr/local/sbin/openvpn --config /var/etc/openvpn/server2.conf
    root   23250  0.0  0.0  8984  1500  ??  Is   29Sep13   0:00.01 /usr/sbin/inetd -wW -R 0 -a 127.0.0.1 /var/etc/inetd.conf
    root   25786  0.0  0.0  5780  1424  ??  Ss   29Sep13   1:53.46 /usr/local/sbin/apinger -c /var/etc/apinger.conf
    root   25813  0.0  0.0 14384  1728  ??  I    29Sep13   0:03.61 /usr/local/bin/rrdtool -
    root   26185  0.0  0.0  8296  1448  ??  IN   29Sep13   0:09.70 /bin/sh /usr/local/pkg/sqpmon.sh
    clamav 29306  0.0  0.4 57768 35980  ??  I     9:53AM   0:01.24 /usr/local/sbin/dansguardian
    clamav 29569  0.0  0.4 55720 35160  ??  S     9:53AM   0:01.23 /usr/local/sbin/dansguardian
    clamav 29570  0.0  0.4 57768 37220  ??  I     9:53AM   0:01.20 /usr/local/sbin/dansguardian
    clamav 29812  0.0  0.4 55720 35152  ??  I     9:53AM   0:01.20 /usr/local/sbin/dansguardian
    clamav 31805  0.0  0.1 22952  7496  ??  S     4:29PM   0:12.22 /usr/local/sbin/dansguardian
    clamav 31963  0.0  0.1 22952  7960  ??  S     4:29PM   0:06.45 /usr/local/sbin/dansguardian
    clamav 31991  0.0  0.4 57768 36196  ??  I     4:29PM   0:08.86 /usr/local/sbin/dansguardian
    clamav 32027  0.0  0.4 57768 37188  ??  I     4:29PM   0:17.31 /usr/local/sbin/dansguardian
    clamav 32399  0.0  0.4 57768 37120  ??  S     4:29PM   0:11.53 /usr/local/sbin/dansguardian
    clamav 32676  0.0  0.4 57768 36076  ??  I     4:29PM   0:07.82 /usr/local/sbin/dansguardian
    clamav 33294  0.0  0.4 57768 36232  ??  S     4:29PM   0:06.85 /usr/local/sbin/dansguardian
    clamav 33629  0.0  0.4 57768 36292  ??  I     4:29PM   0:07.69 /usr/local/sbin/dansguardian
    clamav 33711  0.0  0.4 55720 35576  ??  I     4:29PM   0:04.72 /usr/local/sbin/dansguardian
    clamav 33954  0.0  0.4 55720 35532  ??  I     4:29PM   0:02.17 /usr/local/sbin/dansguardian
    clamav 34172  0.0  0.4 55720 35660  ??  S     4:29PM   0:04.46 /usr/local/sbin/dansguardian
    clamav 34368  0.0  0.4 57768 35916  ??  I     4:35PM   0:00.72 /usr/local/sbin/dansguardian
    clamav 34554  0.0  0.4 55720 35548  ??  I     4:35PM   0:00.92 /usr/local/sbin/dansguardian
    clamav 34660  0.0  0.4 57768 35556  ??  I     4:29PM   0:05.49 /usr/local/sbin/dansguardian
    clamav 34671  0.0  0.4 57768 37120  ??  I     4:29PM   0:04.31 /usr/local/sbin/dansguardian
    clamav 34888  0.0  0.4 55720 35164  ??  I     4:35PM   0:00.70 /usr/local/sbin/dansguardian
    clamav 34979  0.0  0.4 57768 36180  ??  S     4:29PM   0:04.81 /usr/local/sbin/dansguardian
    clamav 35071  0.0  0.4 55720 35164  ??  I     4:35PM   0:00.85 /usr/local/sbin/dansguardian
    clamav 35305  0.0  0.4 55720 35276  ??  I     4:35PM   0:00.67 /usr/local/sbin/dansguardian
    clamav 35465  0.0  0.4 55720 35356  ??  S     4:35PM   0:00.58 /usr/local/sbin/dansguardian
    clamav 35664  0.0  0.4 57768 36200  ??  S     4:35PM   0:00.60 /usr/local/sbin/dansguardian
    clamav 35730  0.0  0.4 55720 35324  ??  S     4:35PM   0:00.50 /usr/local/sbin/dansguardian
    clamav 35977  0.0  0.4 55720 35312  ??  S     4:35PM   0:00.82 /usr/local/sbin/dansguardian
    clamav 36009  0.0  0.4 55720 35048  ??  S     4:35PM   0:00.45 /usr/local/sbin/dansguardian
    clamav 36208  0.0  0.4 55720 34916  ??  I     8:39AM   0:00.01 /usr/local/sbin/dansguardian
    clamav 36222  0.0  0.4 55720 34924  ??  I     8:39AM   0:00.01 /usr/local/sbin/dansguardian
    clamav 36248  0.0  0.4 55720 35408  ??  I     4:35PM   0:00.54 /usr/local/sbin/dansguardian
    clamav 36327  0.0  0.4 55720 34916  ??  I     8:39AM   0:00.01 /usr/local/sbin/dansguardian
    clamav 36594  0.0  0.4 55720 35028  ??  I     4:35PM   0:00.59 /usr/local/sbin/dansguardian
    clamav 36642  0.0  0.4 55720 34928  ??  I     8:39AM   0:00.05 /usr/local/sbin/dansguardian
    clamav 36665  0.0  0.4 55720 35392  ??  I     4:29PM   0:06.44 /usr/local/sbin/dansguardian
    clamav 36740  0.0  0.4 55720 34916  ??  I     8:39AM   0:00.02 /usr/local/sbin/dansguardian
    clamav 36799  0.0  0.4 57768 35760  ??  I     4:35PM   0:00.50 /usr/local/sbin/dansguardian
    clamav 36803  0.0  0.4 55720 34976  ??  I     4:35PM   0:00.38 /usr/local/sbin/dansguardian
    clamav 36929  0.0  0.4 55720 34916  ??  I     8:39AM   0:00.02 /usr/local/sbin/dansguardian
    clamav 36951  0.0  0.4 57768 35700  ??  I     4:35PM   0:00.45 /usr/local/sbin/dansguardian
    clamav 37008  0.0  0.4 57768 35608  ??  I     4:29PM   0:04.74 /usr/local/sbin/dansguardian
    clamav 37105  0.0  0.4 57768 36188  ??  S     4:29PM   0:04.20 /usr/local/sbin/dansguardian
    clamav 37226  0.0  0.4 55720 34916  ??  I     8:39AM   0:00.02 /usr/local/sbin/dansguardian
    clamav 37232  0.0  0.4 55720 34980  ??  I     4:35PM   0:00.38 /usr/local/sbin/dansguardian
    clamav 37302  0.0  0.4 57768 35784  ??  S     4:29PM   0:01.80 /usr/local/sbin/dansguardian
    clamav 37363  0.0  0.4 57768 36200  ??  I     4:29PM   0:05.56 /usr/local/sbin/dansguardian
    clamav 37485  0.0  0.4 55720 34972  ??  I     4:35PM   0:00.51 /usr/local/sbin/dansguardian
    clamav 37594  0.0  0.4 57768 35576  ??  I     4:29PM   0:03.61 /usr/local/sbin/dansguardian
    clamav 37625  0.0  0.4 57768 36072  ??  S     4:29PM   0:04.70 /usr/local/sbin/dansguardian
    clamav 37739  0.0  0.4 55720 35620  ??  I     4:35PM   0:00.56 /usr/local/sbin/dansguardian
    clamav 37909  0.0  0.4 55720 35416  ??  I     4:35PM   0:00.43 /usr/local/sbin/dansguardian
    clamav 38165  0.0  0.4 55720 35212  ??  I     4:35PM   0:00.53 /usr/local/sbin/dansguardian
    clamav 38358  0.0  0.4 57768 36044  ??  S     4:29PM   0:03.52 /usr/local/sbin/dansguardian
    clamav 38619  0.0  0.4 57768 35536  ??  S     4:29PM   0:03.15 /usr/local/sbin/dansguardian
    clamav 38740  0.0  0.4 57768 36816  ??  I     4:29PM   0:03.75 /usr/local/sbin/dansguardian
    clamav 38897  0.0  0.4 57768 35360  ??  I     4:29PM   0:25.72 /usr/local/sbin/dansguardian
    clamav 38899  0.0  0.4 55720 35428  ??  S     4:29PM   0:02.14 /usr/local/sbin/dansguardian
    clamav 38905  0.0  0.4 55720 35364  ??  S     4:29PM   0:03.22 /usr/local/sbin/dansguardian
    clamav 39087  0.0  0.4 57768 36248  ??  I     4:29PM   0:03.17 /usr/local/sbin/dansguardian
    clamav 39388  0.0  0.4 57768 36156  ??  I     4:29PM   0:03.02 /usr/local/sbin/dansguardian
    clamav 39572  0.0  0.4 55720 35312  ??  I     4:35PM   0:00.32 /usr/local/sbin/dansguardian
    clamav 39664  0.0  0.4 55720 35392  ??  I     4:29PM   0:03.83 /usr/local/sbin/dansguardian
    clamav 39695  0.0  0.4 55720 35288  ??  I     4:35PM   0:00.64 /usr/local/sbin/dansguardian
    clamav 39877  0.0  0.4 57768 35652  ??  I     4:35PM   0:00.39 /usr/local/sbin/dansguardian
    clamav 39963  0.0  0.4 55720 35404  ??  I     4:29PM   0:03.20 /usr/local/sbin/dansguardian
    clamav 40058  0.0  0.4 55720 35000  ??  I     4:35PM   0:00.37 /usr/local/sbin/dansguardian
    clamav 40236  0.0  0.4 57768 35584  ??  I     4:35PM   0:00.39 /usr/local/sbin/dansguardian
    clamav 40395  0.0  0.4 55720 35168  ??  I     4:35PM   0:00.32 /usr/local/sbin/dansguardian
    clamav 40434  0.0  0.4 55720 35416  ??  I     4:35PM   0:00.32 /usr/local/sbin/dansguardian
    clamav 40465  0.0  0.4 57768 35296  ??  I     4:29PM   0:03.44 /usr/local/sbin/dansguardian
    clamav 40713  0.0  0.4 57768 35660  ??  I     4:35PM   0:00.35 /usr/local/sbin/dansguardian
    clamav 40835  0.0  0.4 57768 36096  ??  S     4:29PM   0:03.06 /usr/local/sbin/dansguardian
    clamav 41012  0.0  0.4 55720 34968  ??  I     4:35PM   0:00.65 /usr/local/sbin/dansguardian
    clamav 41156  0.0  0.4 55720 35668  ??  I     4:29PM   0:03.92 /usr/local/sbin/dansguardian
    clamav 41273  0.0  0.4 55720 35196  ??  I     4:35PM   0:00.28 /usr/local/sbin/dansguardian
    clamav 41354  0.0  0.4 55720 35004  ??  I     4:35PM   0:00.25 /usr/local/sbin/dansguardian
    clamav 41471  0.0  0.4 55720 35492  ??  I     4:29PM   0:03.16 /usr/local/sbin/dansguardian
    clamav 41525  0.0  0.4 55720 34968  ??  I     4:35PM   0:00.37 /usr/local/sbin/dansguardian
    clamav 41556  0.0  0.4 55720 34960  ??  I     4:35PM   0:00.30 /usr/local/sbin/dansguardian
    clamav 41596  0.0  0.4 57768 36020  ??  S     4:29PM   0:03.44 /usr/local/sbin/dansguardian
    clamav 41813  0.0  0.4 57768 36104  ??  I     4:29PM   0:02.76 /usr/local/sbin/dansguardian
    clamav 41869  0.0  0.4 57768 35876  ??  S     4:35PM   0:00.33 /usr/local/sbin/dansguardian
    clamav 41951  0.0  0.4 55720 35112  ??  I     4:35PM   0:00.37 /usr/local/sbin/dansguardian
    clamav 42127  0.0  0.4 57768 35644  ??  I     4:29PM   0:03.53 /usr/local/sbin/dansguardian
    clamav 42209  0.0  3.1 298568 260464  ??  Ss    4:29PM   2:33.38 /usr/pbi/dansguardian-amd64/sbin/clamd
    clamav 42297  0.0  0.4 57768 36168  ??  I     4:35PM   0:00.33 /usr/local/sbin/dansguardian
    clamav 42335  0.0  0.4 57768 36272  ??  I     4:29PM   0:02.82 /usr/local/sbin/dansguardian
    clamav 42535  0.0  0.4 55720 35440  ??  S     4:29PM   0:02.75 /usr/local/sbin/dansguardian
    clamav 42547  0.0  0.4 55720 34952  ??  I     4:35PM   0:00.33 /usr/local/sbin/dansguardian
    clamav 42594  0.0  0.4 55720 34988  ??  I     4:35PM   0:00.30 /usr/local/sbin/dansguardian
    clamav 42709  0.0  0.4 57768 36152  ??  I     4:29PM   0:02.36 /usr/local/sbin/dansguardian
    clamav 42731  0.0  0.4 57768 36200  ??  S     4:29PM   0:03.19 /usr/local/sbin/dansguardian
    clamav 42776  0.0  0.4 57768 35668  ??  I     4:35PM   0:00.38 /usr/local/sbin/dansguardian
    clamav 42879  0.0  0.4 55720 35388  ??  S     4:29PM   0:02.09 /usr/local/sbin/dansguardian
    clamav 42897  0.0  0.4 55720 35336  ??  I     4:35PM   0:00.57 /usr/local/sbin/dansguardian
    clamav 43158  0.0  0.4 57768 36104  ??  I     4:29PM   0:03.36 /usr/local/sbin/dansguardian
    clamav 43218  0.0  0.4 55720 34968  ??  I     4:29PM   0:00.57 /usr/local/sbin/dansguardian
    clamav 43449  0.0  0.4 55720 35392  ??  I     4:29PM   0:02.77 /usr/local/sbin/dansguardian
    clamav 43607  0.0  0.4 57768 35748  ??  I     4:29PM   0:02.52 /usr/local/sbin/dansguardian
    clamav 43790  0.0  0.4 57768 35992  ??  I     4:29PM   0:02.76 /usr/local/sbin/dansguardian
    clamav 44115  0.0  0.4 55720 35384  ??  I     4:29PM   0:02.34 /usr/local/sbin/dansguardian
    clamav 44312  0.0  0.4 57768 36132  ??  I     4:29PM   0:02.52 /usr/local/sbin/dansguardian
    clamav 44394  0.0  0.4 55720 35312  ??  I     4:29PM   0:00.77 /usr/local/sbin/dansguardian
    clamav 44550  0.0  0.4 55720 35580  ??  I     4:29PM   0:02.11 /usr/local/sbin/dansguardian
    clamav 44802  0.0  0.4 57768 36252  ??  I     4:29PM   0:01.98 /usr/local/sbin/dansguardian
    clamav 44907  0.0  0.4 55720 35604  ??  I     4:29PM   0:02.26 /usr/local/sbin/dansguardian
    clamav 44935  0.0  0.4 55720 35288  ??  I     4:29PM   0:00.69 /usr/local/sbin/dansguardian
    clamav 45257  0.0  0.4 57768 36624  ??  I     4:29PM   0:02.64 /usr/local/sbin/dansguardian
    clamav 45589  0.0  0.4 57768 35164  ??  I     4:29PM   0:02.26 /usr/local/sbin/dansguardian
    clamav 46536  0.0  0.4 55720 35024  ??  I     4:36PM   0:00.36 /usr/local/sbin/dansguardian
    clamav 46826  0.0  0.4 55720 34952  ??  I     4:36PM   0:00.23 /usr/local/sbin/dansguardian
    clamav 46978  0.0  0.4 55720 35140  ??  I     4:36PM   0:00.25 /usr/local/sbin/dansguardian
    clamav 47073  0.0  0.4 55720 34964  ??  I     4:36PM   0:00.26 /usr/local/sbin/dansguardian
    clamav 47373  0.0  0.4 55720 35204  ??  I     4:36PM   0:00.37 /usr/local/sbin/dansguardian
    clamav 47680  0.0  0.4 55720 34976  ??  I     4:36PM   0:00.22 /usr/local/sbin/dansguardian
    clamav 47697  0.0  0.4 55720 34952  ??  I     4:36PM   0:00.27 /usr/local/sbin/dansguardian
    clamav 47918  0.0  0.4 57768 35692  ??  I     4:36PM   0:00.37 /usr/local/sbin/dansguardian
    clamav 47980  0.0  0.4 55720 34956  ??  I     4:36PM   0:00.15 /usr/local/sbin/dansguardian
    clamav 48013  0.0  0.4 55720 35144  ??  S     4:36PM   0:00.20 /usr/local/sbin/dansguardian
    clamav 48300  0.0  0.4 57768 36204  ??  I     4:29PM   0:01.86 /usr/local/sbin/dansguardian
    clamav 48304  0.0  0.4 57768 36684  ??  I     4:36PM   0:00.21 /usr/local/sbin/dansguardian
    clamav 48372  0.0  0.4 57768 35664  ??  I     4:36PM   0:00.42 /usr/local/sbin/dansguardian
    clamav 48428  0.0  0.4 57768 35980  ??  I     4:29PM   0:02.27 /usr/local/sbin/dansguardian
    clamav 48437  0.0  0.4 57768 35736  ??  I     4:29PM   0:01.84 /usr/local/sbin/dansguardian
    clamav 48537  0.0  0.4 55720 34952  ??  I     4:36PM   0:00.15 /usr/local/sbin/dansguardian
    clamav 48716  0.0  0.4 57768 37060  ??  I     4:29PM   0:02.47 /usr/local/sbin/dansguardian
    clamav 48812  0.0  0.4 55720 35000  ??  I     4:36PM   0:00.13 /usr/local/sbin/dansguardian
    clamav 48878  0.0  0.4 57768 36048  ??  S     4:29PM   0:02.80 /usr/local/sbin/dansguardian
    clamav 49151  0.0  0.4 55720 34972  ??  I     4:36PM   0:00.22 /usr/local/sbin/dansguardian
    clamav 49155  0.0  0.4 57768 36612  ??  S     4:29PM   0:01.86 /usr/local/sbin/dansguardian
    clamav 49269  0.0  0.4 55720 34964  ??  I     4:36PM   0:00.23 /usr/local/sbin/dansguardian
    clamav 49364  0.0  0.4 57768 35504  ??  I     4:29PM   0:02.03 /usr/local/sbin/dansguardian
    clamav 49492  0.0  0.4 55720 34960  ??  I     4:36PM   0:00.17 /usr/local/sbin/dansguardian
    clamav 49598  0.0  0.4 55720 34952  ??  I     4:36PM   0:00.16 /usr/local/sbin/dansguardian
    clamav 49639  0.0  0.4 57768 36120  ??  I     4:29PM   0:02.26 /usr/local/sbin/dansguardian
    clamav 49912  0.0  0.4 57768 36112  ??  I     4:36PM   0:00.28 /usr/local/sbin/dansguardian
    clamav 49933  0.0  0.4 57768 36168  ??  I     4:29PM   0:02.11 /usr/local/sbin/dansguardian
    clamav 50165  0.0  0.4 55720 35368  ??  I     4:36PM   0:00.17 /usr/local/sbin/dansguardian
    clamav 50176  0.0  0.4 57768 35380  ??  I     4:29PM   0:01.67 /usr/local/sbin/dansguardian
    clamav 50211  0.0  0.4 57768 36176  ??  I     4:29PM   0:03.94 /usr/local/sbin/dansguardian
    clamav 50254  0.0  0.4 57768 36020  ??  S     4:29PM   0:01.62 /usr/local/sbin/dansguardian
    clamav 50404  0.0  0.4 55720 35640  ??  I     4:29PM   0:01.92 /usr/local/sbin/dansguardian
    clamav 50631  0.0  0.4 57768 36076  ??  I     4:29PM   0:01.68 /usr/local/sbin/dansguardian
    clamav 50905  0.0  0.4 55720 35544  ??  I     4:29PM   0:01.58 /usr/local/sbin/dansguardian
    clamav 51233  0.0  0.4 57768 35816  ??  I     4:29PM   0:03.95 /usr/local/sbin/dansguardian
    clamav 51490  0.0  0.4 55720 35256  ??  S     4:29PM   0:01.64 /usr/local/sbin/dansguardian
    clamav 51669  0.0  0.4 55720 35308  ??  I     4:29PM   0:02.07 /usr/local/sbin/dansguardian
    clamav 51775  0.0  0.4 55720 35292  ??  S     4:29PM   0:01.79 /usr/local/sbin/dansguardian
    clamav 51920  0.0  0.4 57768 35768  ??  I     4:29PM   0:02.29 /usr/local/sbin/dansguardian
    clamav 52027  0.0  0.4 57768 36200  ??  S    10:18AM   0:01.08 /usr/local/sbin/dansguardian
    clamav 52042  0.0  0.4 57768 36304  ??  I     4:29PM   0:01.53 /usr/local/sbin/dansguardian
    clamav 52126  0.0  0.4 57768 35068  ??  S     4:29PM   0:01.54 /usr/local/sbin/dansguardian
    clamav 52487  0.0  0.4 57768 36256  ??  I     4:29PM   0:01.62 /usr/local/sbin/dansguardian
    clamav 52693  0.0  0.4 55720 35228  ??  I     4:29PM   0:01.51 /usr/local/sbin/dansguardian
    clamav 52968  0.0  0.4 55720 35456  ??  I     4:29PM   0:01.30 /usr/local/sbin/dansguardian
    clamav 53215  0.0  0.4 57768 35664  ??  I     4:29PM   0:01.52 /usr/local/sbin/dansguardian
    clamav 53364  0.0  0.4 55720 35624  ??  I     4:29PM   0:01.45 /usr/local/sbin/dansguardian
    clamav 53416  0.0  0.4 57768 35128  ??  I     4:29PM   0:01.73 /usr/local/sbin/dansguardian
    root   53585  0.0  0.0  5784  1096  ??  Is   29Sep13   0:00.00 /usr/local/bin/minicron 240 /var/run/ping_hosts.pid /usr/local/bin/ping_hosts.sh
    clamav 53725  0.0  0.4 55720 35420  ??  I     4:29PM   0:01.56 /usr/local/sbin/dansguardian
    root   53849  0.0  0.0  5784  1144  ??  I    29Sep13   0:00.50 minicron: helper /usr/local/bin/ping_hosts.sh  (minicron)
    clamav 53890  0.0  0.4 57768 35472  ??  I     4:29PM   0:01.27 /usr/local/sbin/dansguardian
    clamav 54038  0.0  0.4 57768 36748  ??  I     4:29PM   0:01.48 /usr/local/sbin/dansguardian
    root   54046  0.0  0.0  5784  1096  ??  Is   29Sep13   0:00.00 /usr/local/bin/minicron 3600 /var/run/expire_accounts.pid /etc/rc.expireaccounts
    clamav 54107  0.0  0.4 57768 35756  ??  S     4:29PM   0:01.56 /usr/local/sbin/dansguardian
    clamav 54152  0.0  0.4 55720 35632  ??  S     4:29PM   0:02.10 /usr/local/sbin/dansguardian
    root   54269  0.0  0.0  5784  1144  ??  I    29Sep13   0:00.04 minicron: helper /etc/rc.expireaccounts  (minicron)
    clamav 54508  0.0  0.4 55720 35332  ??  I     4:29PM   0:01.62 /usr/local/sbin/dansguardian
    root   54534  0.0  0.0  5784  1096  ??  Is   29Sep13   0:00.00 /usr/local/bin/minicron 86400 /var/run/update_alias_url_data.pid /etc/rc.update_alias_url_data
    root   54891  0.0  0.0  5784  1144  ??  I    29Sep13   0:00.00 minicron: helper /etc/rc.update_alias_url_data  (minicron)
    clamav 55534  0.0  0.4 55720 35560  ??  I     7:09AM   0:03.12 /usr/local/sbin/dansguardian
    clamav 55543  0.0  0.4 57768 37232  ??  I     7:09AM   0:04.92 /usr/local/sbin/dansguardian
    clamav 55554  0.0  0.4 57768 36956  ??  S     7:09AM   0:03.13 /usr/local/sbin/dansguardian
    clamav 55802  0.0  0.4 57768 36096  ??  I     7:09AM   0:02.79 /usr/local/sbin/dansguardian
    root   55834  0.0  5.9 952676 491184  ??  Ss   29Sep13 166:18.81 /usr/local/bin/ntop -i em0,em4,em1,em3 -u root -d -4 -M -x 8102 -X 8192
    root   55967  0.0  0.0  9084  1324  ??  Is   29Sep13   0:00.02 /usr/local/sbin/sshlockout_pf 15
    clamav 56138  0.0  0.4 57768 35716  ??  I     7:09AM   0:04.63 /usr/local/sbin/dansguardian
    clamav 56288  0.0  0.4 57768 35720  ??  S     7:09AM   0:02.83 /usr/local/sbin/dansguardian
    root   56290  0.0  0.0  7928  1560  ??  I     4:00PM   0:00.00 cron: running job (cron)
    clamav 56334  0.0  0.4 57768 36180  ??  S     7:09AM   0:02.53 /usr/local/sbin/dansguardian
    clamav 56510  0.0  0.4 55720 35572  ??  S     7:09AM   0:02.41 /usr/local/sbin/dansguardian
    clamav 56788  0.0  0.4 57768 36180  ??  I     7:09AM   0:03.75 /usr/local/sbin/dansguardian
    clamav 57059  0.0  0.4 55720 35612  ??  I     7:09AM   0:02.76 /usr/local/sbin/dansguardian
    clamav 57338  0.0  0.4 55720 35548  ??  I     7:09AM   0:02.66 /usr/local/sbin/dansguardian
    root   57347  0.0  0.5 155432 45808  ??  Is    4:00PM   0:00.16 /usr/local/bin/php /usr/local/www/sarg.php 0
    clamav 57376  0.0  0.4 55720 35404  ??  I     7:09AM   0:02.94 /usr/local/sbin/dansguardian
    clamav 57391  0.0  0.4 55720 35644  ??  S     7:09AM   0:02.70 /usr/local/sbin/dansguardian
    clamav 57511  0.0  0.4 57768 36068  ??  I     7:09AM   0:02.51 /usr/local/sbin/dansguardian
    clamav 57611  0.0  0.4 57768 35348  ??  S     7:09AM   0:01.65 /usr/local/sbin/dansguardian
    clamav 57910  0.0  0.4 57768 36056  ??  S     7:09AM   0:03.86 /usr/local/sbin/dansguardian
    clamav 58028  0.0  0.4 55720 35260  ??  S     7:09AM   0:00.38 /usr/local/sbin/dansguardian
    clamav 58372  0.0  0.4 55720 35276  ??  I     7:09AM   0:04.12 /usr/local/sbin/dansguardian
    clamav 58650  0.0  0.4 57768 35708  ??  I     7:09AM   0:02.02 /usr/local/sbin/dansguardian
    clamav 58899  0.0  0.4 55720 35524  ??  S     7:09AM   0:03.82 /usr/local/sbin/dansguardian
    clamav 59068  0.0  0.4 57768 36124  ??  I     7:09AM   0:01.95 /usr/local/sbin/dansguardian
    clamav 59267  0.0  0.4 55720 35320  ??  I     7:09AM   0:04.04 /usr/local/sbin/dansguardian
    clamav 59330  0.0  0.4 55720 35576  ??  I     7:09AM   0:02.07 /usr/local/sbin/dansguardian
    clamav 59511  0.0  0.4 55720 35612  ??  I     7:09AM   0:02.14 /usr/local/sbin/dansguardian
    clamav 59687  0.0  0.4 55720 34964  ??  I     7:09AM   0:00.97 /usr/local/sbin/dansguardian
    clamav 59771  0.0  0.4 55720 35640  ??  I     7:09AM   0:01.78 /usr/local/sbin/dansguardian
    clamav 60009  0.0  0.4 55720 35464  ??  I     7:09AM   0:02.08 /usr/local/sbin/dansguardian
    clamav 60339  0.0  0.4 57768 35804  ??  I     7:09AM   0:02.01 /usr/local/sbin/dansguardian
    clamav 60342  0.0  0.4 55720 35196  ??  I     4:33PM   0:01.22 /usr/local/sbin/dansguardian
    clamav 60510  0.0  0.4 57768 36084  ??  S     7:09AM   0:01.86 /usr/local/sbin/dansguardian
    clamav 60664  0.0  0.4 57768 35652  ??  I     4:33PM   0:00.96 /usr/local/sbin/dansguardian
    clamav 60704  0.0  0.4 55720 34948  ??  I     7:09AM   0:00.07 /usr/local/sbin/dansguardian
    clamav 60773  0.0  0.4 55720 34944  ??  I     7:09AM   0:00.07 /usr/local/sbin/dansguardian
    clamav 60828  0.0  0.4 55720 35464  ??  I     4:33PM   0:01.18 /usr/local/sbin/dansguardian
    clamav 60917  0.0  0.4 55720 35380  ??  I     4:33PM   0:01.16 /usr/local/sbin/dansguardian
    clamav 60946  0.0  0.4 55720 34932  ??  I     7:09AM   0:00.06 /usr/local/sbin/dansguardian
    clamav 61101  0.0  0.4 55720 34928  ??  I     7:09AM   0:00.04 /usr/local/sbin/dansguardian
    clamav 61421  0.0  0.4 55720 34920  ??  I     7:09AM   0:00.04 /usr/local/sbin/dansguardian
    clamav 61506  0.0  0.4 55720 35344  ??  S     4:33PM   0:01.01 /usr/local/sbin/dansguardian
    clamav 61748  0.0  0.4 55720 34928  ??  I     7:09AM   0:00.04 /usr/local/sbin/dansguardian
    clamav 61790  0.0  0.4 55720 34844  ??  Ss   Thu12PM   7:30.87 /usr/local/sbin/dansguardian
    clamav 61814  0.0  0.4 55720 35636  ??  I     4:33PM   0:01.36 /usr/local/sbin/dansguardian
    clamav 61867  0.0  0.4 57768 36320  ??  I     4:33PM   0:00.93 /usr/local/sbin/dansguardian
    clamav 61994  0.0  0.4 55720 34940  ??  I     7:09AM   0:00.05 /usr/local/sbin/dansguardian
    clamav 62010  0.0  0.4 55720 34936  ??  I     7:09AM   0:00.06 /usr/local/sbin/dansguardian
    clamav 62019  0.0  0.4 57768 35028  ??  I     4:33PM   0:01.44 /usr/local/sbin/dansguardian
    clamav 62083  0.0  0.4 55720 35324  ??  S     4:33PM   0:00.94 /usr/local/sbin/dansguardian
    clamav 62192  0.0  0.4 55720 34920  ??  I     7:09AM   0:00.06 /usr/local/sbin/dansguardian
    clamav 62380  0.0  0.4 55720 35232  ??  S     4:33PM   0:01.59 /usr/local/sbin/dansguardian
    clamav 62539  0.0  0.4 55720 34940  ??  I     7:09AM   0:00.04 /usr/local/sbin/dansguardian
    clamav 62671  0.0  0.4 55720 35576  ??  I     4:33PM   0:01.12 /usr/local/sbin/dansguardian
    clamav 62717  0.0  0.4 55720 34920  ??  I     7:09AM   0:00.03 /usr/local/sbin/dansguardian
    clamav 62762  0.0  0.4 55720 35336  ??  I     4:33PM   0:00.89 /usr/local/sbin/dansguardian
    clamav 62853  0.0  0.4 55720 35188  ??  I     7:09AM   0:00.05 /usr/local/sbin/dansguardian
    clamav 63081  0.0  0.4 55720 34936  ??  I     7:09AM   0:00.04 /usr/local/sbin/dansguardian
    clamav 63086  0.0  0.4 55720 34932  ??  I     7:09AM   0:00.04 /usr/local/sbin/dansguardian
    clamav 63106  0.0  0.4 55720 35464  ??  I     4:33PM   0:00.99 /usr/local/sbin/dansguardian
    clamav 63254  0.0  0.4 55720 34916  ??  I     7:09AM   0:00.04 /usr/local/sbin/dansguardian
    clamav 63407  0.0  0.4 57768 36160  ??  S     4:33PM   0:01.07 /usr/local/sbin/dansguardian
    clamav 63551  0.0  0.4 55720 34936  ??  I     7:09AM   0:00.07 /usr/local/sbin/dansguardian
    clamav 63609  0.0  0.4 55720 35456  ??  I     4:33PM   0:00.82 /usr/local/sbin/dansguardian
    clamav 63713  0.0  0.4 57768 35832  ??  I     4:33PM   0:01.16 /usr/local/sbin/dansguardian
    clamav 63794  0.0  0.4 55720 34928  ??  I     7:09AM   0:00.07 /usr/local/sbin/dansguardian
    clamav 63912  0.0  0.4 57768 35044  ??  I     4:33PM   0:01.03 /usr/local/sbin/dansguardian
    clamav 64082  0.0  0.4 55720 34920  ??  I     7:09AM   0:00.04 /usr/local/sbin/dansguardian
    clamav 64150  0.0  0.4 55720 35636  ??  I     4:33PM   0:00.87 /usr/local/sbin/dansguardian
    clamav 64317  0.0  0.4 55720 34940  ??  I     7:09AM   0:00.07 /usr/local/sbin/dansguardian
    clamav 64441  0.0  0.4 55720 34924  ??  I     7:09AM   0:00.04 /usr/local/sbin/dansguardian
    clamav 64448  0.0  0.4 57768 35980  ??  I     4:33PM   0:00.92 /usr/local/sbin/dansguardian
    clamav 64656  0.0  0.4 55720 34932  ??  I     7:09AM   0:00.29 /usr/local/sbin/dansguardian
    clamav 64660  0.0  0.4 57768 36300  ??  I     4:33PM   0:00.91 /usr/local/sbin/dansguardian
    clamav 64793  0.0  0.4 55720 34972  ??  I     4:33PM   0:00.91 /usr/local/sbin/dansguardian
    clamav 64898  0.0  0.4 55720 34932  ??  I     7:09AM   0:00.04 /usr/local/sbin/dansguardian
    clamav 65010  0.0  0.4 55720 35092  ??  I     4:33PM   0:00.89 /usr/local/sbin/dansguardian
    clamav 65157  0.0  0.4 55720 34936  ??  I     7:09AM   0:00.05 /usr/local/sbin/dansguardian
    clamav 65187  0.0  0.4 55720 35632  ??  I     4:33PM   0:00.67 /usr/local/sbin/dansguardian
    clamav 65299  0.0  0.4 55720 34920  ??  I     7:09AM   0:00.04 /usr/local/sbin/dansguardian
    clamav 65308  0.0  0.4 55720 34932  ??  I     7:09AM   0:00.04 /usr/local/sbin/dansguardian
    clamav 65317  0.0  0.4 57768 36716  ??  I     4:33PM   0:01.36 /usr/local/sbin/dansguardian
    clamav 65510  0.0  0.4 55720 35292  ??  I     4:33PM   0:00.84 /usr/local/sbin/dansguardian
    clamav 65556  0.0  0.4 57768 37116  ??  I     4:33PM   0:00.70 /usr/local/sbin/dansguardian
    clamav 65576  0.0  0.4 55720 34944  ??  I     7:09AM   0:00.06 /usr/local/sbin/dansguardian
    clamav 65625  0.0  0.4 57768 35092  ??  I     4:33PM   0:00.81 /usr/local/sbin/dansguardian
    clamav 65664  0.0  0.4 55720 35604  ??  S     4:33PM   0:00.83 /usr/local/sbin/dansguardian
    clamav 65676  0.0  0.4 55720 35116  ??  I     7:09AM   0:00.04 /usr/local/sbin/dansguardian
    clamav 65791  0.0  0.4 57768 36936  ??  I     4:33PM   0:00.94 /usr/local/sbin/dansguardian
    clamav 65962  0.0  0.4 55720 34920  ??  I     7:09AM   0:00.03 /usr/local/sbin/dansguardian
    root   66036  0.0  0.1 13796  4884  ??  S    Mon04PM   0:03.97 /usr/sbin/tcpdump -s 256 -v -S -l -n -e -ttt -i pflog0
    clamav 66059  0.0  0.4 55720 34916  ??  I     7:09AM   0:00.02 /usr/local/sbin/dansguardian
    clamav 66078  0.0  0.4 57768 35988  ??  I     4:33PM   0:00.87 /usr/local/sbin/dansguardian
    clamav 66109  0.0  0.4 55720 35572  ??  I     4:33PM   0:00.61 /usr/local/sbin/dansguardian
    clamav 66146  0.0  0.4 55720 34920  ??  I     7:09AM   0:00.03 /usr/local/sbin/dansguardian
    clamav 66237  0.0  0.4 55720 34920  ??  I     7:09AM   0:00.02 /usr/local/sbin/dansguardian
    root   66306  0.0  0.0  5780  1060  ??  S    Mon04PM   0:03.35 logger -t pf -p local0.info
    clamav 66356  0.0  0.4 55720 34952  ??  I     7:09AM   0:00.04 /usr/local/sbin/dansguardian
    clamav 66451  0.0  0.4 55720 35288  ??  I     4:33PM   0:00.72 /usr/local/sbin/dansguardian
    clamav 66629  0.0  0.4 55720 34924  ??  I     7:09AM   0:00.05 /usr/local/sbin/dansguardian
    clamav 66691  0.0  0.4 55720 35356  ??  I     4:33PM   0:01.12 /usr/local/sbin/dansguardian
    clamav 66741  0.0  0.4 57768 36184  ??  I     4:33PM   0:00.68 /usr/local/sbin/dansguardian
    clamav 66843  0.0  0.4 55720 35188  ??  I     4:33PM   0:00.68 /usr/local/sbin/dansguardian
    clamav 66867  0.0  0.4 55720 35268  ??  I     4:33PM   0:00.54 /usr/local/sbin/dansguardian
    clamav 66928  0.0  0.4 55720 34924  ??  I     7:09AM   0:00.05 /usr/local/sbin/dansguardian
    clamav 67056  0.0  0.4 55720 35388  ??  I     4:33PM   0:00.61 /usr/local/sbin/dansguardian
    clamav 67202  0.0  0.4 55720 35596  ??  I     4:33PM   0:00.60 /usr/local/sbin/dansguardian
    clamav 67215  0.0  0.4 55720 34924  ??  I     7:09AM   0:00.04 /usr/local/sbin/dansguardian
    clamav 67298  0.0  0.4 55720 34936  ??  I     7:09AM   0:00.04 /usr/local/sbin/dansguardian
    clamav 67332  0.0  0.4 55720 34916  ??  I     7:09AM   0:00.03 /usr/local/sbin/dansguardian
    clamav 67488  0.0  0.4 55720 34920  ??  I     7:09AM   0:00.04 /usr/local/sbin/dansguardian
    clamav 67812  0.0  0.4 55720 34924  ??  I     7:09AM   0:00.04 /usr/local/sbin/dansguardian
    clamav 68075  0.0  0.4 55720 34936  ??  I     7:09AM   0:00.03 /usr/local/sbin/dansguardian
    clamav 68288  0.0  0.4 55720 34916  ??  I     7:09AM   0:00.03 /usr/local/sbin/dansguardian
    clamav 68315  0.0  0.4 55720 34916  ??  I     7:09AM   0:00.02 /usr/local/sbin/dansguardian
    clamav 68348  0.0  0.4 55720 34916  ??  I     7:09AM   0:00.02 /usr/local/sbin/dansguardian
    clamav 68408  0.0  0.4 55720 34920  ??  I     7:09AM   0:00.02 /usr/local/sbin/dansguardian
    clamav 68565  0.0  0.4 55720 34916  ??  I     7:09AM   0:00.03 /usr/local/sbin/dansguardian
    clamav 68746  0.0  0.4 55720 34956  ??  I     7:09AM   0:00.04 /usr/local/sbin/dansguardian
    clamav 68970  0.0  0.4 55720 34928  ??  I     7:09AM   0:00.03 /usr/local/sbin/dansguardian
    clamav 69198  0.0  0.4 55720 34916  ??  I     7:09AM   0:00.02 /usr/local/sbin/dansguardian
    clamav 69431  0.0  0.4 55720 34916  ??  I     7:09AM   0:00.03 /usr/local/sbin/dansguardian
    clamav 69489  0.0  0.4 55720 34916  ??  I     7:09AM   0:00.02 /usr/local/sbin/dansguardian
    clamav 69492  0.0  0.4 55720 34952  ??  I     7:09AM   0:00.03 /usr/local/sbin/dansguardian
    clamav 69795  0.0  0.4 55720 35204  ??  I     7:09AM   0:00.03 /usr/local/sbin/dansguardian
    clamav 69988  0.0  0.4 55720 34932  ??  I     7:09AM   0:00.06 /usr/local/sbin/dansguardian
    clamav 70328  0.0  0.4 55720 35204  ??  I     7:09AM   0:00.03 /usr/local/sbin/dansguardian
    clamav 70565  0.0  0.4 55720 34916  ??  I     7:09AM   0:00.02 /usr/local/sbin/dansguardian
    clamav 70674  0.0  0.4 55720 34920  ??  I     7:09AM   0:00.03 /usr/local/sbin/dansguardian
    clamav 70753  0.0  0.4 55720 34916  ??  I     7:09AM   0:00.02 /usr/local/sbin/dansguardian
    clamav 70928  0.0  0.4 55720 34920  ??  I     7:09AM   0:00.02 /usr/local/sbin/dansguardian
    clamav 71077  0.0  0.4 55720 34916  ??  I     7:09AM   0:00.02 /usr/local/sbin/dansguardian
    clamav 71100  0.0  0.4 55720 34916  ??  I     7:09AM   0:00.02 /usr/local/sbin/dansguardian
    clamav 71224  0.0  0.4 55720 34920  ??  I     7:09AM   0:00.04 /usr/local/sbin/dansguardian
    clamav 71556  0.0  0.4 55720 34916  ??  I     7:09AM   0:00.03 /usr/local/sbin/dansguardian
    clamav 71767  0.0  0.4 55720 34928  ??  I     7:09AM   0:00.02 /usr/local/sbin/dansguardian
    clamav 71933  0.0  0.4 55720 34916  ??  I     7:09AM   0:00.02 /usr/local/sbin/dansguardian
    clamav 72045  0.0  0.4 55720 34920  ??  I     7:09AM   0:00.03 /usr/local/sbin/dansguardian
    clamav 72357  0.0  0.4 55720 34916  ??  I     7:09AM   0:00.02 /usr/local/sbin/dansguardian
    clamav 72596  0.0  0.4 55720 34916  ??  I     7:09AM   0:00.02 /usr/local/sbin/dansguardian
    clamav 72726  0.0  0.4 55720 34916  ??  I     7:09AM   0:00.01 /usr/local/sbin/dansguardian
    clamav 72734  0.0  0.4 55720 34920  ??  I     7:09AM   0:00.01 /usr/local/sbin/dansguardian
    clamav 73046  0.0  0.4 55720 34916  ??  I     7:09AM   0:00.02 /usr/local/sbin/dansguardian
    clamav 73277  0.0  0.4 55720 35356  ??  I     4:36PM   0:00.23 /usr/local/sbin/dansguardian
    clamav 73308  0.0  0.4 55720 35132  ??  I     4:36PM   0:00.15 /usr/local/sbin/dansguardian
    clamav 73599  0.0  0.4 55720 34960  ??  I     4:36PM   0:00.14 /usr/local/sbin/dansguardian
    clamav 73766  0.0  0.4 55720 34976  ??  I     4:36PM   0:00.14 /usr/local/sbin/dansguardian
    root   73802  0.0  0.5 147240 40232  ??  I     3:56PM   0:06.84 /usr/local/bin/php
    clamav 74055  0.0  0.4 55720 35404  ??  I     4:36PM   0:00.17 /usr/local/sbin/dansguardian
    clamav 74077  0.0  0.4 55720 34968  ??  I     4:36PM   0:00.19 /usr/local/sbin/dansguardian
    clamav 74301  0.0  0.4 55720 34944  ??  I     4:36PM   0:00.17 /usr/local/sbin/dansguardian
    clamav 74428  0.0  0.4 55720 35168  ??  I     4:36PM   0:00.19 /usr/local/sbin/dansguardian
    clamav 74539  0.0  0.4 55720 34948  ??  I     4:36PM   0:00.24 /usr/local/sbin/dansguardian
    clamav 74865  0.0  0.4 55720 34964  ??  I     4:36PM   0:00.21 /usr/local/sbin/dansguardian
    clamav 75153  0.0  0.4 55720 35116  ??  I     4:36PM   0:00.15 /usr/local/sbin/dansguardian
    clamav 75218  0.0  0.4 57768 35516  ??  I     4:36PM   0:00.15 /usr/local/sbin/dansguardian
    clamav 75265  0.0  0.4 57768 35776  ??  I     4:36PM   0:00.25 /usr/local/sbin/dansguardian
    clamav 75406  0.0  0.4 55720 34932  ??  I     4:36PM   0:00.10 /usr/local/sbin/dansguardian
    clamav 75457  0.0  0.4 55720 34944  ??  I     4:36PM   0:00.20 /usr/local/sbin/dansguardian
    clamav 75469  0.0  0.4 55720 35140  ??  I     4:36PM   0:00.23 /usr/local/sbin/dansguardian
    clamav 75606  0.0  0.4 55720 34948  ??  I     4:36PM   0:00.13 /usr/local/sbin/dansguardian
    clamav 75898  0.0  0.4 55720 35120  ??  I     4:36PM   0:00.14 /usr/local/sbin/dansguardian
    clamav 76065  0.0  0.4 55720 34952  ??  I     4:36PM   0:00.26 /usr/local/sbin/dansguardian
    clamav 76354  0.0  0.4 55720 35056  ??  I     4:36PM   0:00.11 /usr/local/sbin/dansguardian
    clamav 76672  0.0  0.4 55720 34948  ??  I     4:36PM   0:00.12 /usr/local/sbin/dansguardian
    clamav 76886  0.0  0.4 55720 34928  ??  I     4:36PM   0:00.12 /usr/local/sbin/dansguardian
    clamav 77061  0.0  0.4 55720 34932  ??  I     4:36PM   0:00.11 /usr/local/sbin/dansguardian
    clamav 77330  0.0  0.4 55720 34932  ??  I     4:36PM   0:00.13 /usr/local/sbin/dansguardian
    clamav 77594  0.0  0.4 57768 35508  ??  I     4:36PM   0:00.09 /usr/local/sbin/dansguardian
    clamav 77685  0.0  0.4 55720 35436  ??  I     4:36PM   0:00.08 /usr/local/sbin/dansguardian
    clamav 77906  0.0  0.4 55720 35152  ??  I     4:36PM   0:00.09 /usr/local/sbin/dansguardian
    clamav 78001  0.0  0.4 55720 34952  ??  I     4:36PM   0:00.11 /usr/local/sbin/dansguardian
    clamav 78334  0.0  0.4 57768 35636  ??  I     4:36PM   0:00.12 /usr/local/sbin/dansguardian
    clamav 78485  0.0  0.4 55720 34948  ??  I     4:36PM   0:00.11 /usr/local/sbin/dansguardian
    clamav 78513  0.0  0.4 55720 34932  ??  I     4:36PM   0:00.09 /usr/local/sbin/dansguardian
    clamav 78770  0.0  0.4 55720 34944  ??  I     4:36PM   0:00.31 /usr/local/sbin/dansguardian
    root   78906  0.0  0.1 32412  7372  ??  S     4:28PM   0:16.21 /usr/local/sbin/lighttpd -f /var/etc/lighty-webConfigurator.conf
    clamav 78981  0.0  0.4 55720 34956  ??  I     4:36PM   0:00.10 /usr/local/sbin/dansguardian
    clamav 79057  0.0  0.4 55720 34940  ??  I     4:36PM   0:00.06 /usr/local/sbin/dansguardian
    root   79384  0.0  0.2 139048 18336  ??  Is    4:28PM   0:00.05 /usr/local/bin/php
    root   80970  0.0  0.3 49924 25120  ??  Ss   29Sep13  18:44.36 /usr/sbin/bsnmpd -c /var/etc/snmpd.conf -p /var/run/snmpd.pid
    root   81369  0.0  0.0  2716   880  ??  IN    4:11PM   0:00.00 sleep 60
    root   81701  0.0  0.0 26168  3584  ??  Ss    4:11PM   0:00.02 sshd: root@pts/1 (sshd)
    clamav 83032  0.0  0.4 55720 34960  ??  I     1:55PM   0:00.38 /usr/local/sbin/dansguardian
    clamav 83164  0.0  0.4 55720 35168  ??  I     1:55PM   0:00.38 /usr/local/sbin/dansguardian
    clamav 83269  0.0  0.4 55720 34964  ??  I     1:55PM   0:00.38 /usr/local/sbin/dansguardian
    clamav 83413  0.0  0.4 55720 34916  ??  I     1:55PM   0:00.01 /usr/local/sbin/dansguardian
    clamav 83498  0.0  0.4 55720 34920  ??  I     1:55PM   0:00.01 /usr/local/sbin/dansguardian
    clamav 83504  0.0  0.4 55720 34904  ??  I     1:55PM   0:00.01 /usr/local/sbin/dansguardian
    clamav 83645  0.0  0.4 55720 34904  ??  I     1:55PM   0:00.01 /usr/local/sbin/dansguardian
    clamav 83666  0.0  0.4 55720 34912  ??  I     1:55PM   0:00.01 /usr/local/sbin/dansguardian
    clamav 83737  0.0  0.4 55720 34916  ??  I     1:55PM   0:00.01 /usr/local/sbin/dansguardian
    clamav 83785  0.0  0.4 55720 34912  ??  I     1:55PM   0:00.01 /usr/local/sbin/dansguardian
    clamav 83893  0.0  0.4 55720 34904  ??  I     1:55PM   0:00.01 /usr/local/sbin/dansguardian
    clamav 84016  0.0  0.4 55720 34904  ??  I     1:55PM   0:00.01 /usr/local/sbin/dansguardian
    clamav 84079  0.0  0.4 55720 34916  ??  I     1:55PM   0:00.01 /usr/local/sbin/dansguardian
    clamav 84201  0.0  0.4 55720 34904  ??  I     1:55PM   0:00.00 /usr/local/sbin/dansguardian
    root   84433  0.0  0.0  2716   880  ??  IN    4:11PM   0:00.00 sleep 55
    clamav 84436  0.0  0.4 55720 34912  ??  I     1:55PM   0:00.01 /usr/local/sbin/dansguardian
    root   91780  0.0  0.2 139048 18336  ??  Is    4:28PM   0:00.08 /usr/local/bin/php
    root   95972  0.0  0.0  6956  1552  ??  Ss   29Sep13   1:44.09 /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -f /var/etc/syslog.conf
    root   55672  0.0  0.0 19480  1696  v0  Is   29Sep13   0:00.02 login [pam] (login)
    root   56282  0.0  0.0  8296  1460  v0  I    29Sep13   0:00.00 -sh (sh)
    root   58194  0.0  0.0  8296  1460  v0  I+   29Sep13   0:00.00 /bin/sh /etc/rc.initial
    root   79823  0.0  0.0  8296  1716   0- IN    4:28PM   0:12.21 /bin/sh /var/db/rrd/updaterrd.sh
    root   84550  0.0  0.0  8296  1644   1  Is    4:11PM   0:00.00 -sh (sh)
    root   85190  0.0  0.0  8296  1648   1  I     4:11PM   0:00.00 /bin/sh /etc/rc.initial
    root   87858  0.0  0.0  8268  2436   1  S     4:11PM   0:00.01 /bin/tcsh
    root   88539  0.0  0.0  7992  1844   1  R+    4:12PM   0:00.00 ps uxawww
    
    




  • We are not running sipproxd or any other add-on package for that matter.  We have about 20 phones that use tftp to download configuration from an Internet-based SIP provider, so we use tftp-proxy a bit.  Other than that most of our traffic is SIP/RTP, and plain old Internet browsing (DNS, HTTP, HTTPS, etc.).

    It's not a very complicated setup.  We had some traffic shaping set up for the VOIP traffic, and we used the "Single Wan multi Lan" wizard to set it up.  VOIP traffic is on a different VLAN than the rest of the traffic, so that wizard seemed appropriate.  I have since disabled the traffic shaper in an attempt to narrow down the problem.



  • Strange - My memory utilization has creeped up to over 80% now, but I suspect it won't go much higher and I think its pure squid usage.  What do your states look like?



  • States almost never go above 2000.  They're pretty consistent too.  There aren't spikes in the state table size when the problem occurs.






  • It's also interesting to note that when the firewall starts having issues, neither the processor usage nor the memory usage is above 50%.



  • kejianshi,

    The dansguardian processes were in a post from timthetortoise who says he's not having issues.  I (Jason, the OP) am not running dansguardian or ClamAV, but I am having a memory leak issue.  My process lists is in the original post.

    Thanks for your help though!



  • HAHA - Yea.  I saw that after I posted and realized my mistake and deleted it…  I was hoping no one saw it.  You caught me!

    Question.  Is this system upgraded or is it a fresh clean install?

    I've been told repeatedly that that shouldn't matter, but I'm not sure how true that is.



  • The system was upgraded to 2.1–not a clean install.  The problem existed both on 2.0.2 and continues on 2.1.



  • Is a new clean install out of the question?



  • At this point, nothing is out of the question.  Of course a new install is a pain, but we have to solve the problem.  The last thing we did was turn off the traffic shaper all together.  The folks behind this firewall don't seem to consume their Internet pipe (10Mb symmetrical), so the traffic shaper isn't that big of a deal.  Now I'm just waiting for the memory leak to start again.  Sometimes it takes a few days; others it takes a month or more.

    We're in a virtualized environment, so I've considered setting up a separate gateway in a high-availability configuration.  It'd give me somewhere to move traffic when the problem occurs assuming the memory leak doesn't show up on both firewalls at the same time.  That's a few hours work and I have to make sure I have extra Internet IPs available, but it would reduce the risk of having to incur a middle-of-the-workday outage.



  • "traffic shaper" - Is that no longer running?



  • The traffic shaper is no longer running.



  • I'd wipe the drive and install clean.  Unless your rules are special in some way, I would not even do a restore of settings.  I'd redo them by hand to be sure you didn't import trouble.  Is this a 64 or 32 bit install?  Because I'd also prefer 32bit.



  • For your information, I think this problem has followed us through several iterations of pfSense from 1.2.3 (i think) on an ALIX board to 2.0.1 on a PC to 2.0.2 on a virtual server to 2.1 on the same server.  I do think the same XML config has been ported through all of those environments–both 32- and 64-bit.  I guess I'm in agreement that a scratch rebuild might be the next good thing to do.

    It's an amd64 install today.  I'm curious, why do you prefer 32-bit?  Does anyone else have thoughts on 32-bit vs 64-bit?



  • Because with less than 4GB ram you don't NEED 64bit version and I've seen more issues with 64 than 32.  Some would say its my imagination, but I don't think so.  I think the 32bit version is just working better under VMs.  I've experienced that anyway.  Seems others have also, but thats more of something I've noticed than a scientific study of the issue.



  • I have a customer on a 64 bit version and it behaves stupendously. Not one issue!  :)



  • We also have at least six other 64-bit installs that are working flawlessly.  A couple of them have fairly complicated configurations with ~10 interfaces, ~20 IPsec tunnels, 100s of firewall & NAT rules, traffic shaping, etc.  With all of that, they're still able to sustain 100Mb/s throughput easily.  I'm curious if there would be performance degradation moving from 64-bit to 32-bit.



  • The problems I personally experienced were on 64bit version on ESXi 5.

    Same install, nothing else modified and it worked fine.  Tried several times.

    I couldn't solve it, so I just stayed with 32bit version.

    Sounds like your hypervisor isn't same, so probably depends.

    Also, the problems I had were mainly either with NTP or with nothing working well with 5 or more interfaces.

    There are just too many variables in hardware  and network to say anthing for sure for everybody, but as a rule, if I don't need 64bit, I use 32bit where I can.  As for it being a point of contention for this or that person, I say do whatever floats your boat and works for you.

    There won't be any performance losses.  64bit is just able to address more RAM.  Thats all.



  • Jason: Did you ever solve this?

    I ended up just re-installing the firmware over the original install via the firmware page and that seems to have solved it for me here.

    My other watchguard "customer" had his unit do the same thing starting last Wednesday and build up to over 30% before we caught it.  Restarted Apinger and it has been normal since…




  • Since we've been through at least four firmware updates on this firewall and the problem still persists, I'm doubting a firmware reinstall will fix the problem.  I'm also pretty sure apinger is not the problem as I believe that would cause memory usage to spike in user space.  I'm seeing wired kernel memory balloon.

    My plan at this point is to build another near-identical firewall through the web gui, download the XML config from the new and old ones, and compare.  Maybe that will reveal some anomaly.  This XML has been pulled up through the 1.2 versions to 2.1.  Maybe something bad slipped through.



  • My memory does exactly the same thing.  Looks like utilization climbs and climbs and then it hits about 90%, teeters there for a while and then will drop back down to about 30% all by its self, with no reset, reboot or anything and no ill effects and then the cycle starts again.  I don't worry about it.



  • Im actually still seeing this intermittently on a couple of boxes.  Though last time it happened to me here it actually used up all memory before it reset itself.

    But since Ive reset the webgui from a console it has not occurred.  Re-booting the box never solved it. In fact it would simply do the same right after the re-boot.


  • Rebel Alliance Developer Netgate

    When the memory usage is high, have you checked the process list to see what is using the RAM?



  • I'll chuck this in this mix as its running in a VM.
    Brute force
    http://www.run-virtual.com/?p=695

    then how to reset the VM password which might be sat in a data centre and doesnt get checked often
    http://blog.geekdiary.me/?p=21

    because it might be running auto failover type of software like this:
    http://www.veeam.com/virtual-machine-backup-solution-free.html



  • @jimp:

    When the memory usage is high, have you checked the process list to see what is using the RAM?

    Just this-    https://forum.pfsense.org/index.php/topic,67703.msg370496.html#msg370496

    It hasn't happened since the webgui reset from the console so I cant look now. Im watching for it.



  • After some testing it seems the LCDProc package is causing this for me.

    When I do a webgui reset the display goes to a screen that shows Cli: 0  Scr: 0 with no backlight. I didn't notice it till days after my initial webgui reset. Once I restarted the LCD package the memory leak started again. Reset tonight and Im going to leave the package in the inop state for a few weeks.

    Running the dev package.