Memory Leak
-
I have a pfSense 2.1 (amd64) virtual firewall running under QEMU with 2 CPU cores & 1GB RAM. I am experiencing a slow memory leak that will consume memory at the rate of about 1MB every 90 minutes. This firewall has a fairly static configuration, and the memory leak seems to be triggered by some unknown event outside of any administrative changes.
I've attached a couple of RRD graphs showing the leaks.
Here are some notes on what we've discovered so far:
1. This problem occurred under both pfSense 2.0.2 and pfSense 2.1.
2. When the problem occurs the performance of the entire system seems to slow. The most noticeable way this manifests itself is in voice-quality issues on VoIP connections.
3. The leak is preceded by log entries like this. As time passes, less and less tftp-proxy requests go through while more and more display this error.
Oct 4 00:07:24 [daemon.err] pf connection lookup failed (no rdr?)
Oct 4 00:07:24 [daemon.warning] /usr/libexec/tftp-proxy[35113]: exited, status 14. At about the same time the leak starts, the log lines above turn into lines like this.
Oct 4 17:09:30 [daemon.err] couldn't forward tftp packet: Operation not permitted
Oct 4 17:09:30 [daemon.warning] /usr/libexec/tftp-proxy[28967]: exited, status 15. The memory leak does not appear to be in user space. No user processes appear to be consuming huge amounts of memory.
6. CPU usage is low during the problem. Occasionally dnsmasq will consume 20-30% of the CPU, but most of the time the CPU usage is <2%.
7. Rebooting the firewall fixes the problem for anywhere from a couple of days to a couple of months.
Here is the output of "ps uxawww":
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND root 11 112.0 0.0 0 32 ?? RL 23Sep13 38419:04.70 [idle] root 0 55.0 0.0 0 208 ?? DLs 23Sep13 527:25.26 [kernel] root 12 12.0 0.0 0 240 ?? WL 23Sep13 474:09.17 [intr] nobody 7850 5.0 0.3 10100 3108 ?? S 11:49AM 1:16.75 [dnsmasq] root 1 0.0 0.1 3200 584 ?? ILs 23Sep13 0:00.14 /sbin/init -- root 2 0.0 0.0 0 16 ?? DL 23Sep13 0:00.04 [g_event] root 3 0.0 0.0 0 16 ?? DL 23Sep13 9:52.53 [g_up] root 4 0.0 0.0 0 16 ?? DL 23Sep13 8:12.40 [g_down] root 5 0.0 0.0 0 16 ?? DL 23Sep13 0:00.00 [crypto] root 6 0.0 0.0 0 16 ?? DL 23Sep13 0:00.00 [crypto returns] root 7 0.0 0.0 0 16 ?? DL 23Sep13 0:00.00 [sctp_iterator] root 8 0.0 0.0 0 16 ?? DL 23Sep13 1:48.33 [pfpurge] root 9 0.0 0.0 0 16 ?? DL 23Sep13 0:00.00 [xpt_thrd] root 10 0.0 0.0 0 16 ?? DL 23Sep13 0:00.00 [audit] root 13 0.0 0.0 0 32 ?? DL 23Sep13 1:58.59 [ng_queue] root 14 0.0 0.0 0 16 ?? DL 23Sep13 4:10.67 [yarrow] root 15 0.0 0.0 0 64 ?? DL 23Sep13 0:22.20 [usb] root 16 0.0 0.0 0 16 ?? DL 23Sep13 0:04.57 [pagedaemon] root 17 0.0 0.0 0 16 ?? DL 23Sep13 0:00.00 [vmdaemon] root 18 0.0 0.0 0 16 ?? DL 23Sep13 0:00.04 [pagezero] root 19 0.0 0.0 0 16 ?? DL 23Sep13 0:04.88 [idlepoll] root 20 0.0 0.0 0 16 ?? DL 23Sep13 0:08.57 [bufdaemon] root 21 0.0 0.0 0 16 ?? DL 23Sep13 1:57.08 [syncer] root 22 0.0 0.0 0 16 ?? DL 23Sep13 0:09.88 [vnlru] root 23 0.0 0.0 0 16 ?? DL 23Sep13 0:10.54 [softdepflush] root 36 0.0 0.0 0 32 ?? DL 23Sep13 0:22.39 [zfskern] root 66 0.0 0.0 0 16 ?? DL 23Sep13 0:42.69 [md0] root 264 0.0 0.1 6908 1388 ?? INs 23Sep13 5:41.09 /usr/local/sbin/check_reload_status root 269 0.0 0.1 6908 1284 ?? IN 23Sep13 0:00.00 check_reload_status: Monitoring daemon of check_reload_status root 275 0.0 0.3 5248 3148 ?? Is 23Sep13 0:00.06 /sbin/devd root 6842 0.0 0.1 5780 1444 ?? Is 11:49AM 0:00.08 /usr/local/sbin/dhcpleases -l /var/dhcpd/var/db/dhcpd.leases -d example.com -p /var/run/dnsmasq.pid -h /var/etc/hosts dhcpd 11788 0.0 1.2 17104 12516 ?? Ss 11:49AM 0:00.45 /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid em0 em3 root 13148 0.0 0.7 15264 7164 ?? Ss 11:49AM 0:00.77 /usr/local/bin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid root 15208 0.0 0.1 7036 1320 ?? Is 23Sep13 0:00.07 /usr/local/sbin/sshlockout_pf 15 root 15439 0.0 0.2 8984 1624 ?? Is 23Sep13 0:38.31 /usr/sbin/inetd -wW -R 0 -a 127.0.0.1 /var/etc/inetd.conf root 15899 0.0 0.2 6956 1660 ?? Ss 27Sep13 5:44.31 /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -f /var/etc/syslog.conf root 16509 0.0 0.6 13796 6264 ?? S 27Sep13 0:40.78 /usr/sbin/tcpdump -s 256 -v -S -l -n -e -ttt -i pflog0 root 16751 0.0 0.1 5780 1148 ?? S 27Sep13 0:22.68 logger -t pf -p local0.info root 17261 0.0 0.1 7036 1368 ?? Is 27Sep13 0:00.05 /usr/local/sbin/sshlockout_pf 15 root 17591 0.0 0.1 5780 1472 ?? Ss 23Sep13 13:30.58 /usr/local/sbin/apinger -c /var/etc/apinger.conf root 17754 0.0 0.2 14384 2352 ?? I 23Sep13 0:07.53 /usr/local/bin/rrdtool - root 29260 0.0 0.4 15268 3752 ?? INs 11:49AM 0:00.00 /usr/sbin/sshd root 34568 0.0 0.4 24220 4308 ?? SN 11:49AM 0:00.49 /usr/local/sbin/lighttpd -f /var/etc/lighty-webConfigurator.conf root 35228 0.0 0.2 8296 1928 ?? SN 11:49AM 0:00.29 /bin/sh /var/db/rrd/updaterrd.sh root 35431 0.0 0.1 6812 1252 ?? Is 9:12AM 0:00.45 /usr/local/sbin/qstats -p /var/run/qstats.pid root 37915 0.0 0.4 23232 4556 ?? Is 23Sep13 0:06.76 /usr/local/sbin/mpd4 -b -d /var/etc/pptp-vpn -p /var/run/pptp-vpn.pid -s pptps pptps root 39676 0.0 0.1 2716 968 ?? SN 12:12PM 0:00.00 sleep 60 root 41395 0.0 0.4 16776 4432 ?? Ss 23Sep13 3:58.02 /usr/local/sbin/racoon -f /var/etc/ipsec/racoon.conf root 41921 0.0 0.2 7168 1916 ?? Is 23Sep13 2:45.46 /usr/local/sbin/filterdns -p /var/run/filterdns-ipsec.pid -i 60 -c /var/etc/ipsec/filterdns-ipsec.hosts -d 1 root 42087 0.0 3.7 145192 37500 ?? S 11:56AM 0:04.79 /usr/local/bin/php root 44922 0.0 1.1 30464 11520 ?? Ss 23Sep13 2:41.21 /usr/sbin/bsnmpd -c /var/etc/snmpd.conf -p /var/run/snmpd.pid root 59046 0.0 0.1 7036 1368 ?? Is 27Sep13 0:00.12 /usr/local/sbin/sshlockout_pf 15 root 63668 0.0 0.4 26168 4272 ?? Ss 10:08AM 0:03.25 sshd: root@pts/2 (sshd) root 65508 0.0 0.4 26168 4272 ?? Ss 8:56AM 0:09.27 sshd: root@pts/0 (sshd) root 66403 0.0 2.1 139048 21968 ?? Is 10:43AM 0:00.18 /usr/local/bin/php root 69793 0.0 2.1 139048 21968 ?? Is 10:43AM 0:00.11 /usr/local/bin/php root 74436 0.0 4.2 143144 42760 ?? I 10:43AM 0:02.27 /usr/local/bin/php root 80776 0.0 0.2 7928 1620 ?? Ss 23Sep13 0:02.43 /usr/sbin/cron -s root 82170 0.0 0.1 5784 1184 ?? Is 23Sep13 0:00.00 /usr/local/bin/minicron 240 /var/run/ping_hosts.pid /usr/local/bin/ping_hosts.sh root 82404 0.0 0.1 5784 1232 ?? I 23Sep13 0:01.70 minicron: helper /usr/local/bin/ping_hosts.sh (minicron) root 82519 0.0 0.1 5784 1184 ?? Is 23Sep13 0:00.00 /usr/local/bin/minicron 3600 /var/run/expire_accounts.pid /etc/rc.expireaccounts root 82802 0.0 0.1 5784 1232 ?? I 23Sep13 0:00.11 minicron: helper /etc/rc.expireaccounts (minicron) root 83415 0.0 0.1 5784 1184 ?? Is 23Sep13 0:00.00 /usr/local/bin/minicron 86400 /var/run/update_alias_url_data.pid /etc/rc.update_alias_url_data root 83460 0.0 0.1 5784 1232 ?? I 23Sep13 0:00.01 minicron: helper /etc/rc.update_alias_url_data (minicron) root 89243 0.0 0.1 7036 1364 ?? Is 23Sep13 0:00.09 /usr/local/sbin/sshlockout_pf 15 root 91238 0.0 0.4 26168 4272 ?? Is 8:57AM 0:04.30 sshd: root@pts/1 (sshd) root 93548 0.0 0.1 7036 1368 ?? Is 27Sep13 0:00.05 /usr/local/sbin/sshlockout_pf 15 root 98081 0.0 0.1 5780 1264 ?? Is 11:49AM 0:00.00 /usr/local/sbin/choparp em3 auto 64.198.147.131/32 root 88803 0.0 0.2 19480 1844 v0 Is 23Sep13 0:00.01 login [pam] (login) root 89468 0.0 0.2 8296 1800 v0 I 23Sep13 0:00.00 -sh (sh) root 92299 0.0 0.2 8296 1800 v0 I+ 23Sep13 0:00.01 /bin/sh /etc/rc.initial root 88913 0.0 0.2 19480 1844 v1 Is 23Sep13 0:00.02 login [pam] (login) root 89785 0.0 0.2 8296 1800 v1 I 23Sep13 0:00.00 -sh (sh) root 91982 0.0 0.2 8296 1800 v1 I+ 23Sep13 0:00.00 /bin/sh /etc/rc.initial root 34966 0.0 0.2 9324 2372 0 S+ 10:54AM 0:03.77 top root 87219 0.0 0.2 8296 1800 0 Is 8:56AM 0:00.01 -sh (sh) root 88110 0.0 0.2 8296 1804 0 I 8:56AM 0:00.01 /bin/sh /etc/rc.initial root 90558 0.0 0.3 8268 2864 0 I 8:56AM 0:00.16 /bin/tcsh root 30839 0.0 0.3 8268 2644 1 I 9:30AM 0:00.29 /bin/tcsh root 87723 0.0 0.1 2716 968 1 I+ 12:11PM 0:00.00 sleep 60 root 91596 0.0 0.2 8296 1848 1 Is 8:57AM 0:00.01 -sh (sh) root 92587 0.0 0.2 8296 1852 1 I 8:57AM 0:00.01 /bin/sh /etc/rc.initial root 39919 0.0 0.1 7992 1532 2 R+ 12:12PM 0:00.00 ps uxawww root 74148 0.0 0.2 8296 1876 2 Is 10:08AM 0:00.00 -sh (sh) root 75219 0.0 0.2 8296 1880 2 I 10:08AM 0:00.01 /bin/sh /etc/rc.initial root 85289 0.0 0.3 8268 2860 2 S 10:43AM 0:00.23 /bin/tcshI am completely stumped as to the cause of this problem. Any help would be greatly appreciated. Thanks!
-Jason Crowley | Innovative Networks
-
Im seeing something too that Ive been watching… I reboot the box and start out at 5% memory usage according to the dashboard. Today (bottom) its sitting at 30%.
Im running Siproxd, LCDproc dev, shellcmd, file manager, pfblocker and System patches with a widescreen mod. It seems when I restart siproxd it goes back down... Restarting the others has no effect.
last pid: 4328; load averages: 0.28, 0.51, 0.29 up 0+00:04:22 23:16:07 115 processes: 3 running, 97 sleeping, 15 waiting Mem: 57M Active, 15M Inact, 75M Wired, 248K Cache, 24M Buf, 1835M Free Swap: 2048M Total, 2048M Free PID USERNAME PRI NICE SIZE RES STATE TIME WCPU COMMAND 10 root 171 ki31 0K 8K RUN 3:10 96.97% [idle] 51713 root 76 0 78996K 30068K piperd 0:05 1.95% /usr/local/bin/php{php} 11 root -68 - 0K 128K RUN 0:02 1.95% [intr{irq16: mskc0 skc}] 0 root -16 0 0K 64K sched 0:50 0.00% [kernel{swapper}] 255 root 76 20 3352K 1204K kqread 0:24 0.00% /usr/local/sbin/check_reload_status 11 root -32 - 0K 128K WAIT 0:01 0.00% [intr{swi4: clock}] 45472 root 76 0 76820K 24328K accept 0:01 0.00% /usr/local/bin/php 14 root -40 - 0K 160K - 0:00 0.00% [usb{usbus4}] 14 root -40 - 0K 160K - 0:00 0.00% [usb{usbus3}] 14 root -40 - 0K 160K - 0:00 0.00% [usb{usbus0}] 14 root -40 - 0K 160K - 0:00 0.00% [usb{usbus1}] 14 root -40 - 0K 160K - 0:00 0.00% [usb{usbus2}] 1 root 76 0 1888K 492K wait 0:00 0.00% [init] 44008 root 44 0 6980K 4844K kqread 0:00 0.00% /usr/local/sbin/lighttpd -f /var/etc/lighty-we 13 root -16 - 0K 8K - 0:00 0.00% [yarrow] 63464 nobody 74 r30 3316K 1360K nanslp 0:00 0.00% /usr/pbi/lcdproc-i386/sbin/LCDd -c /usr/local/ 4 root -8 - 0K 8K - 0:00 0.00% [g_down] 94747 root 76 20 3644K 1492K wait 0:00 0.00% /bin/sh /usr/local/sbin/fanctrld.sh last pid: 25864; load averages: 0.12, 0.05, 0.01 up 1+21:28:30 11:53:05 116 processes: 4 running, 96 sleeping, 16 waiting Mem: 344M Active, 17M Inact, 82M Wired, 236K Cache, 30M Buf, 1539M Free Swap: 2048M Total, 2048M Free PID USERNAME PRI NICE SIZE RES STATE TIME WCPU COMMAND 10 root 171 ki31 0K 8K RUN 43.5H 100.00% [idle] 51903 root 61 0 77972K 28316K piperd 0:01 0.98% /usr/local/bin/php{php} 11 root -32 - 0K 128K WAIT 17:24 0.00% [intr{swi4: clock}] 65599 root 76 20 202M 152M nanslp 5:57 0.00% /usr/local/bin/php -f /usr/local/pkg/lcdproc_c 65078 root 76 20 202M 152M nanslp 5:57 0.00% /usr/local/bin/php -f /usr/local/pkg/lcdproc_c 11 root -68 - 0K 128K WAIT 3:00 0.00% [intr{irq16: mskc0 skc}] 255 root 76 20 3352K 1204K kqread 1:25 0.00% /usr/local/sbin/check_reload_status 0 root -16 0 0K 64K sched 1:24 0.00% [kernel{swapper}] 65364 nobody 74 r30 3316K 1432K RUN 0:54 0.00% /usr/pbi/lcdproc-i386/sbin/LCDd -c /usr/local/ 2034 root 76 20 3644K 1492K wait 0:48 0.00% /bin/sh /usr/local/sbin/fanctrld.sh 4340 root 76 0 3644K 1492K wait 0:47 0.00% /bin/sh /usr/local/sbin/fanctrld.sh 13568 root 76 20 3644K 1536K wait 0:16 0.00% /bin/sh /var/db/rrd/updaterrd.sh 13 root -16 - 0K 8K - 0:15 0.00% [yarrow] 11 root -44 - 0K 128K WAIT 0:15 0.00% [intr{swi1: netisr 0}] 33160 root 44 0 3264K 1244K select 0:12 0.00% /usr/local/sbin/apinger -c /var/etc/apinger.co 94971 root 44 0 8808K 6068K select 0:07 0.00% /usr/pbi/siproxd-i386/sbin/siproxd -c /usr/pbi 44839 root 44 0 76820K 29588K accept 0:06 0.00% /usr/local/bin/php 3 root -8 - 0K 8K - 0:05 0.00% [g_up] last pid: 45195; load averages: 0.00, 0.01, 0.00 up 4+23:37:23 14:01:58 117 processes: 4 running, 97 sleeping, 16 waiting Mem: 279M Active, 20M Inact, 86M Wired, 232K Cache, 34M Buf, 1597M Free Swap: 2048M Total, 2048M Free PID USERNAME PRI NICE SIZE RES STATE TIME WCPU COMMAND 10 root 171 ki31 0K 8K RUN 115.2H 100.00% [idle] 11 root -32 - 0K 128K WAIT 44:44 0.00% [intr{swi4: clock}] 11 root -68 - 0K 128K WAIT 10:41 0.00% [intr{irq16: mskc0 skc}] 1749 root 76 20 277M 228M nanslp 9:55 0.00% /usr/local/bin/php -f /usr/local/pkg/lcdproc_c 255 root 76 20 3352K 1204K kqread 2:18 0.00% /usr/local/sbin/check_reload_status 2034 root 76 20 3644K 1492K wait 2:06 0.00% /bin/sh /usr/local/sbin/fanctrld.sh 4340 root 76 0 3644K 1492K wait 2:05 0.00% /bin/sh /usr/local/sbin/fanctrld.sh 0 root -16 0 0K 64K sched 1:24 0.00% [kernel{swapper}] 1469 nobody 74 r30 3316K 1428K RUN 1:17 0.00% /usr/pbi/lcdproc-i386/sbin/LCDd -c /usr/local/ 13 root -16 - 0K 8K - 0:51 0.00% [yarrow] 11 root -68 - 0K 128K WAIT 0:44 0.00% [intr{irq19: mskc3 skc}] 13568 root 76 20 3644K 1536K wait 0:42 0.00% /bin/sh /var/db/rrd/updaterrd.sh 11 root -44 - 0K 128K WAIT 0:30 0.00% [intr{swi1: netisr 0}] 33160 root 44 0 3264K 1244K select 0:30 0.00% /usr/local/sbin/apinger -c /var/etc/apinger.co 43794 root 44 0 8004K 5628K kqread 0:24 0.00% /usr/local/sbin/lighttpd -f /var/etc/lighty-we 65053 root 64 20 5432K 3784K select 0:22 0.00% /usr/local/sbin/openvpn --config /var/etc/open 3 root -8 - 0K 8K - 0:14 0.00% [g_up] 6671 root 64 20 6280K 6300K select 0:12 0.00% /usr/local/bin/ntpd -g -c /var/etc/ntpd.conf -
-
An identical watchguard- No siproxd or pfblocker…
edit= which does not exhibit the behavior...
last pid: 53797; load averages: 0.04, 0.03, 0.00 up 13+19:32:07 19:07:20 110 processes: 4 running, 90 sleeping, 16 waiting Mem: 108M Active, 18M Inact, 79M Wired, 1036K Cache, 61M Buf, 1784M Free Swap: PID USERNAME PRI NICE SIZE RES STATE TIME WCPU COMMAND 10 root 171 ki31 0K 8K RUN 321.5H 98.97% [idle] 11 root -32 - 0K 128K WAIT 101:51 0.00% [intr{swi4: clock}] 80288 root 76 20 122M 61656K RUN 37:12 0.00% /usr/local/bin/php -f /usr/local/pkg/lcdproc_c 11 root -68 - 0K 128K WAIT 33:22 0.00% [intr{irq16: skc0 uhci}] 11 root -68 - 0K 128K WAIT 21:12 0.00% [intr{irq17: skc1}] 80188 nobody 74 r30 3316K 1424K nanslp 7:39 0.00% /usr/pbi/lcdproc-i386/sbin/LCDd -c /usr/local/ 13 root -16 - 0K 8K - 4:38 0.00% [yarrow] 20489 root 44 0 3264K 1232K select 3:07 0.00% /usr/local/sbin/apinger -c /var/etc/apinger.co 11 root -44 - 0K 128K WAIT 1:52 0.00% [intr{swi1: netisr 0}] 0 root -16 0 0K 56K sched 1:52 0.00% [kernel{swapper}] 44231 root 76 20 3644K 1436K wait 1:44 0.00% /bin/sh /var/db/rrd/updaterrd.sh 296 root 76 20 3352K 1180K kqread 0:54 0.00% /usr/local/sbin/check_reload_status 79315 root 64 20 6280K 6300K select 0:49 0.00% /usr/local/bin/ntpd -g -c /var/etc/ntpd.conf - 33590 dhcpd 64 20 11456K 8052K select 0:46 0.00% /usr/local/sbin/dhcpd -user dhcpd -group _dhcp 30371 nobody 64 20 5512K 2868K select 0:39 0.00% /usr/local/sbin/dnsmasq --all-servers --rebind 49 root -8 - 0K 8K mdwait 0:31 0.00% [md1] 20 root 20 - 0K 8K syncer 0:27 0.00% [syncer] 29805 root 44 0 8004K 5484K kqread 0:23 0.00% /usr/local/sbin/lighttpd -f /var/etc/lighty-we pfSense is © 2004 - 2013 by Electric Sheep Fencing LLC. All Righ
-
Note that the memory usage I see increasing is "Wired" memory. Also, it's interesting to note that kernel tasks related to NICs show up at the top of the list. The em0 and em1 interfaces are LAN and WAN respectively.
Here is my output from "top -SH" during the problem.
last pid: 82067; load averages: 2.83, 1.79, 1.22 up 13+19:23:15 12:14:23 129 processes: 6 running, 108 sleeping, 15 waiting CPU: 0.0% user, 0.5% nice, 93.6% system, 0.2% interrupt, 5.7% idle Mem: 103M Active, 30M Inact, 315M Wired, 164K Cache, 71M Buf, 523M Free Swap: 1024M Total, 1024M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 0 root -68 0 0K 208K - 0 192:25 78.96% kernel{em0 taskq} 0 root -68 0 0K 208K - 1 243:36 36.96% kernel{em1 taskq} 7850 nobody 76 0 10100K 3140K select 1 1:43 34.96% dnsmasq 11 root 171 ki31 0K 32K RUN 1 322.6H 17.97% idle{idle: cpu1} 11 root 171 ki31 0K 32K RUN 0 317.8H 11.96% idle{idle: cpu0} 12 root -44 - 0K 240K WAIT 0 193:38 8.98% intr{swi1: netisr 0} 0 root -68 0 0K 208K - 1 91:55 1.95% kernel{em3 taskq} 0 root -16 0 0K 208K sched 0 575.4H 0.00% kernel{swapper} 12 root -32 - 0K 240K WAIT 0 116:57 0.00% intr{swi4: clock} 12 root -44 - 0K 240K WAIT 1 72:41 0.00% intr{swi1: netisr 1} 12 root -68 - 0K 240K WAIT 0 70:08 0.00% intr{irq11: em1 em2+} 12 root -32 - 0K 240K WAIT 0 17:21 0.00% intr{swi4: clock} 17591 root 44 0 5780K 1472K select 1 13:31 0.00% apinger 3 root -8 - 0K 16K - 0 9:53 0.00% g_up 4 root -8 - 0K 16K - 1 8:12 0.00% g_down 15899 root 44 0 6956K 1660K select 1 5:45 0.00% syslogd 264 root 76 20 6908K 1388K kqread 0 5:41 0.00% check_reload_status 14 root -16 - 0K 16K - 0 4:11 0.00% yarrow 41395 root 44 0 16776K 4432K select 0 3:58 0.00% racoon 12 root -64 - 0K 240K WAIT 0 3:41 0.00% intr{irq14: ata0} 41921 root 44 0 7168K 1916K ucond 0 2:46 0.00% filterdns{vpn.in-kc.com} 44922 root 44 0 30464K 11520K select 1 2:41 0.00% bsnmpd 21 root 20 - 0K 16K syncer 1 1:57 0.00% syncer 8 root -16 - 0K 16K pftm 1 1:49 0.00% pfpurge 13 root -16 - 0K 32K sleep 0 0:59 0.00% ng_queue{ng_queue0} 13 root -16 - 0K 32K sleep 0 0:59 0.00% ng_queue{ng_queue1} 0 root -68 0 0K 208K - 0 0:54 0.00% kernel{em2 taskq} 66 root -8 - 0K 16K mdwait 1 0:43 0.00% md0 16509 root 44 0 13796K 6264K bpf 1 0:41 0.00% tcpdump 15439 root 44 0 8984K 1624K select 0 0:38 0.00% inetd 16751 root 44 0 5780K 1148K piperd 0 0:23 0.00% logger 15 root -40 - 0K 64K - 0 0:22 0.00% usb{usbus0} 36 root -8 - 0K 32K arc_re 1 0:12 0.00% zfskern{arc_reclaim_thre} 23 root -16 - 0K 16K sdflus 0 0:11 0.00% softdepflush 36 root -8 - 0K 32K l2arc_ 1 0:10 0.00% zfskern{l2arc_feed_threa} 22 root -16 - 0K 16K vlruwt 1 0:10 0.00% vnlru 65508 root 44 0 26168K 4272K select 1 0:10 0.00% sshd 20 root -16 - 0K 16K psleep 1 0:09 0.00% bufdaemon 37915 root 44 0 23232K 4556K select 1 0:07 0.00% mpd4{mpd4} 42087 root 76 0 142M 38032K accept 0 0:05 0.00% php 19 root 76 ki-6 0K 16K pollid 0 0:05 0.00% idlepoll 16 root -16 - 0K 16K psleep 0 0:05 0.00% pagedaemon 91238 root 44 0 26168K 4272K select 1 0:04 0.00% sshd 63668 root 44 0 26168K 4272K select 0 0:03 0.00% sshd 34966 root 44 0 9324K 2372K select 1 0:03 0.00% top 80776 root 44 0 7928K 1620K nanslp 0 0:02 0.00% cron 74436 root 52 0 140M 42760K accept 1 0:02 0.00% php 82404 root 76 0 5784K 1232K nanslp 0 0:02 0.00% minicron 17754 root 44 0 14384K 2352K piperd 0 0:01 0.00% rrdtool 13148 root 44 0 15264K 7164K select 1 0:01 0.00% ntpd 34568 root 64 20 24220K 4308K kqread 1 0:01 0.00% lighttpd 11788 dhcpd 44 0 17104K 12516K select 1 0:00 0.00% dhcpd 35431 root 44 0 6812K 1252K accept 0 0:00 0.00% qstats 35228 root 76 20 8296K 1928K piperd 0 0:00 0.00% sh 30839 root 45 0 8268K 2644K pause 0 0:00 0.00% tcsh 85289 root 44 0 8268K 2860K pause 1 0:00 0.00% tcsh 66403 root 76 0 136M 21968K wait 0 0:00 0.00% php 90558 root 44 0 8268K 2864K pause 1 0:00 0.00% tcsh 1 root 44 0 3200K 584K wait 1 0:00 0.00% init 69793 root 76 0 136M 21968K wait 0 0:00 0.00% php 82802 root 46 0 5784K 1232K nanslp 1 0:00 0.00% minicron 6842 root 44 0 5780K 1444K kqread 0 0:00 0.00% dhcpleases 59046 root 44 0 7036K 1368K uwait 0 0:00 0.00% sshlockout_pf{initial thread} 275 root 44 0 5248K 3148K select 1 0:00 0.00% devd 15208 root 44 0 7036K 1320K nanslp 0 0:00 0.00% sshlockout_pf{sshlockout_pf} 89243 root 44 0 7036K 1364K nanslp 1 0:00 0.00% sshlockout_pf{sshlockout_pf}
-
I dont see that either on ESXi and no siproxd here.
-
Im up to 40% this morning and restarting any service has no effect this time. Im going to have to reboot the box before I leave to make sure it stays working throughout the day.
-
Not seeing any issues here, I'm on 2.1 Release 64Bit. At work I'm using 2.1 RC0 from May 32 Bit and no issue there as well it's been running rock solid for months without any resets. Just wanted to add another data point if it helps. No proxies or anything other packages. I have a Open VPN site to site tunnel to various sites around the State. Multiple VLANs (10 or so) and Dual WANs 100/25 connections.
-
So it seems that siproxd is the culprit so far….
-
I dont see where the OP has Siproxd running on his setup… Am I missing it?
Ive got several boxes running in other locations without this issue.
-
No such issues here, running squid, DG, ntop, OpenVPN, IPSec tunnel, Sarg, and pfBlocker.
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND root 11 650.8 0.0 0 128 ?? RL 29Sep13 101590:37.68 [idle] root 58837 90.0 0.0 21668 3940 ?? R 4:00PM 10:05.57 /usr/pbi/sarg-amd64/bin/sarg root 0 0.0 0.0 0 320 ?? DLs 29Sep13 104:29.50 [kernel] root 1 0.0 0.0 3200 388 ?? ILs 29Sep13 0:00.23 /sbin/init -- root 2 0.0 0.0 0 16 ?? DL 29Sep13 0:01.17 [g_event] root 3 0.0 0.0 0 16 ?? DL 29Sep13 22:33.14 [g_up] root 4 0.0 0.0 0 16 ?? DL 29Sep13 31:32.75 [g_down] root 5 0.0 0.0 0 16 ?? DL 29Sep13 0:00.00 [crypto] root 6 0.0 0.0 0 16 ?? DL 29Sep13 0:00.00 [crypto returns] root 7 0.0 0.0 0 16 ?? DL 29Sep13 0:00.00 [mpt_recovery0] root 8 0.0 0.0 0 16 ?? DL 29Sep13 0:02.85 [fdc0] root 9 0.0 0.0 0 16 ?? DL 29Sep13 0:00.00 [sctp_iterator] root 10 0.0 0.0 0 16 ?? DL 29Sep13 0:00.00 [audit] root 12 0.0 0.0 0 464 ?? WL 29Sep13 198:47.13 [intr] root 13 0.0 0.0 0 128 ?? DL 29Sep13 0:00.00 [ng_queue] root 14 0.0 0.0 0 16 ?? DL 29Sep13 5:32.45 [yarrow] root 15 0.0 0.0 0 16 ?? DL 29Sep13 1:12.62 [pfpurge] root 16 0.0 0.0 0 16 ?? DL 29Sep13 0:00.00 [xpt_thrd] root 17 0.0 0.0 0 16 ?? DL 29Sep13 0:01.26 [pagedaemon] root 18 0.0 0.0 0 16 ?? DL 29Sep13 0:00.00 [vmdaemon] root 19 0.0 0.0 0 16 ?? DL 29Sep13 0:00.01 [pagezero] root 20 0.0 0.0 0 16 ?? DL 29Sep13 0:00.76 [idlepoll] root 21 0.0 0.0 0 16 ?? DL 29Sep13 0:43.36 [bufdaemon] root 22 0.0 0.0 0 16 ?? DL 29Sep13 14:33.51 [syncer] root 23 0.0 0.0 0 16 ?? DL 29Sep13 0:39.40 [vnlru] root 24 0.0 0.0 0 16 ?? DL 29Sep13 0:04.24 [softdepflush] root 37 0.0 0.0 0 32 ?? DL 29Sep13 0:07.04 [zfskern] root 63 0.0 0.0 0 16 ?? DL 29Sep13 0:05.05 [md0] root 262 0.0 0.0 6908 1320 ?? INs 29Sep13 325:27.54 /usr/local/sbin/check_reload_status root 267 0.0 0.0 6908 1216 ?? IN 29Sep13 0:00.00 check_reload_status: Monitoring daemon of check_reload_status root 278 0.0 0.0 5248 2972 ?? Is 29Sep13 0:00.01 /sbin/devd root 435 0.0 0.4 143144 32720 ?? I 3:42PM 0:05.83 /usr/local/bin/php root 540 0.0 0.0 16776 4124 ?? Ss 1Oct13 0:15.90 /usr/local/sbin/racoon -f /var/etc/ipsec/racoon.conf root 658 0.0 0.0 7928 1524 ?? Ss 29Sep13 0:01.89 /usr/sbin/cron -s root 3365 0.0 0.1 15264 7164 ?? Ss 29Sep13 0:26.02 /usr/local/bin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid root 6393 0.0 0.3 28308 24808 ?? Is Mon03PM 0:06.52 /usr/local/sbin/dhcpleases -l /var/dhcpd/var/db/dhcpd.leases -d mrpcap.com -p /var/run/dnsmasq.pid -h /var/etc/hosts nobody 9927 0.0 0.0 10100 2912 ?? S Mon03PM 0:30.61 [dnsmasq] root 9994 0.0 0.0 15268 2580 ?? Is 29Sep13 0:00.01 /usr/sbin/sshd root 10260 0.0 0.0 7036 1272 ?? Is 29Sep13 0:00.02 /usr/local/sbin/sshlockout_pf 15 dhcpd 12656 0.0 0.1 17104 11804 ?? Ss Mon03PM 0:06.78 /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid em1 root 20627 0.0 0.1 23744 5776 ?? Is Mon04PM 0:00.00 /usr/pbi/squid-amd64/sbin/squid -f /usr/pbi/squid-amd64/etc/squid/squid.conf proxy 21192 0.0 0.8 85192 62772 ?? S Mon04PM 24:45.99 (squid) -f /usr/pbi/squid-amd64/etc/squid/squid.conf (squid) proxy 21468 0.0 0.0 7056 1620 ?? I Mon04PM 0:00.00 (unlinkd) (unlinkd) root 22372 0.0 0.1 13488 4440 ?? Ss 29Sep13 0:49.46 /usr/local/sbin/openvpn --config /var/etc/openvpn/server1.conf root 22494 0.0 0.1 13488 4216 ?? Ss 29Sep13 0:04.06 /usr/local/sbin/openvpn --config /var/etc/openvpn/server2.conf root 23250 0.0 0.0 8984 1500 ?? Is 29Sep13 0:00.01 /usr/sbin/inetd -wW -R 0 -a 127.0.0.1 /var/etc/inetd.conf root 25786 0.0 0.0 5780 1424 ?? Ss 29Sep13 1:53.46 /usr/local/sbin/apinger -c /var/etc/apinger.conf root 25813 0.0 0.0 14384 1728 ?? I 29Sep13 0:03.61 /usr/local/bin/rrdtool - root 26185 0.0 0.0 8296 1448 ?? IN 29Sep13 0:09.70 /bin/sh /usr/local/pkg/sqpmon.sh clamav 29306 0.0 0.4 57768 35980 ?? I 9:53AM 0:01.24 /usr/local/sbin/dansguardian clamav 29569 0.0 0.4 55720 35160 ?? S 9:53AM 0:01.23 /usr/local/sbin/dansguardian clamav 29570 0.0 0.4 57768 37220 ?? I 9:53AM 0:01.20 /usr/local/sbin/dansguardian clamav 29812 0.0 0.4 55720 35152 ?? I 9:53AM 0:01.20 /usr/local/sbin/dansguardian clamav 31805 0.0 0.1 22952 7496 ?? S 4:29PM 0:12.22 /usr/local/sbin/dansguardian clamav 31963 0.0 0.1 22952 7960 ?? S 4:29PM 0:06.45 /usr/local/sbin/dansguardian clamav 31991 0.0 0.4 57768 36196 ?? I 4:29PM 0:08.86 /usr/local/sbin/dansguardian clamav 32027 0.0 0.4 57768 37188 ?? I 4:29PM 0:17.31 /usr/local/sbin/dansguardian clamav 32399 0.0 0.4 57768 37120 ?? S 4:29PM 0:11.53 /usr/local/sbin/dansguardian clamav 32676 0.0 0.4 57768 36076 ?? I 4:29PM 0:07.82 /usr/local/sbin/dansguardian clamav 33294 0.0 0.4 57768 36232 ?? S 4:29PM 0:06.85 /usr/local/sbin/dansguardian clamav 33629 0.0 0.4 57768 36292 ?? I 4:29PM 0:07.69 /usr/local/sbin/dansguardian clamav 33711 0.0 0.4 55720 35576 ?? I 4:29PM 0:04.72 /usr/local/sbin/dansguardian clamav 33954 0.0 0.4 55720 35532 ?? I 4:29PM 0:02.17 /usr/local/sbin/dansguardian clamav 34172 0.0 0.4 55720 35660 ?? S 4:29PM 0:04.46 /usr/local/sbin/dansguardian clamav 34368 0.0 0.4 57768 35916 ?? I 4:35PM 0:00.72 /usr/local/sbin/dansguardian clamav 34554 0.0 0.4 55720 35548 ?? I 4:35PM 0:00.92 /usr/local/sbin/dansguardian clamav 34660 0.0 0.4 57768 35556 ?? I 4:29PM 0:05.49 /usr/local/sbin/dansguardian clamav 34671 0.0 0.4 57768 37120 ?? I 4:29PM 0:04.31 /usr/local/sbin/dansguardian clamav 34888 0.0 0.4 55720 35164 ?? I 4:35PM 0:00.70 /usr/local/sbin/dansguardian clamav 34979 0.0 0.4 57768 36180 ?? S 4:29PM 0:04.81 /usr/local/sbin/dansguardian clamav 35071 0.0 0.4 55720 35164 ?? I 4:35PM 0:00.85 /usr/local/sbin/dansguardian clamav 35305 0.0 0.4 55720 35276 ?? I 4:35PM 0:00.67 /usr/local/sbin/dansguardian clamav 35465 0.0 0.4 55720 35356 ?? S 4:35PM 0:00.58 /usr/local/sbin/dansguardian clamav 35664 0.0 0.4 57768 36200 ?? S 4:35PM 0:00.60 /usr/local/sbin/dansguardian clamav 35730 0.0 0.4 55720 35324 ?? S 4:35PM 0:00.50 /usr/local/sbin/dansguardian clamav 35977 0.0 0.4 55720 35312 ?? S 4:35PM 0:00.82 /usr/local/sbin/dansguardian clamav 36009 0.0 0.4 55720 35048 ?? S 4:35PM 0:00.45 /usr/local/sbin/dansguardian clamav 36208 0.0 0.4 55720 34916 ?? I 8:39AM 0:00.01 /usr/local/sbin/dansguardian clamav 36222 0.0 0.4 55720 34924 ?? I 8:39AM 0:00.01 /usr/local/sbin/dansguardian clamav 36248 0.0 0.4 55720 35408 ?? I 4:35PM 0:00.54 /usr/local/sbin/dansguardian clamav 36327 0.0 0.4 55720 34916 ?? I 8:39AM 0:00.01 /usr/local/sbin/dansguardian clamav 36594 0.0 0.4 55720 35028 ?? I 4:35PM 0:00.59 /usr/local/sbin/dansguardian clamav 36642 0.0 0.4 55720 34928 ?? I 8:39AM 0:00.05 /usr/local/sbin/dansguardian clamav 36665 0.0 0.4 55720 35392 ?? I 4:29PM 0:06.44 /usr/local/sbin/dansguardian clamav 36740 0.0 0.4 55720 34916 ?? I 8:39AM 0:00.02 /usr/local/sbin/dansguardian clamav 36799 0.0 0.4 57768 35760 ?? I 4:35PM 0:00.50 /usr/local/sbin/dansguardian clamav 36803 0.0 0.4 55720 34976 ?? I 4:35PM 0:00.38 /usr/local/sbin/dansguardian clamav 36929 0.0 0.4 55720 34916 ?? I 8:39AM 0:00.02 /usr/local/sbin/dansguardian clamav 36951 0.0 0.4 57768 35700 ?? I 4:35PM 0:00.45 /usr/local/sbin/dansguardian clamav 37008 0.0 0.4 57768 35608 ?? I 4:29PM 0:04.74 /usr/local/sbin/dansguardian clamav 37105 0.0 0.4 57768 36188 ?? S 4:29PM 0:04.20 /usr/local/sbin/dansguardian clamav 37226 0.0 0.4 55720 34916 ?? I 8:39AM 0:00.02 /usr/local/sbin/dansguardian clamav 37232 0.0 0.4 55720 34980 ?? I 4:35PM 0:00.38 /usr/local/sbin/dansguardian clamav 37302 0.0 0.4 57768 35784 ?? S 4:29PM 0:01.80 /usr/local/sbin/dansguardian clamav 37363 0.0 0.4 57768 36200 ?? I 4:29PM 0:05.56 /usr/local/sbin/dansguardian clamav 37485 0.0 0.4 55720 34972 ?? I 4:35PM 0:00.51 /usr/local/sbin/dansguardian clamav 37594 0.0 0.4 57768 35576 ?? I 4:29PM 0:03.61 /usr/local/sbin/dansguardian clamav 37625 0.0 0.4 57768 36072 ?? S 4:29PM 0:04.70 /usr/local/sbin/dansguardian clamav 37739 0.0 0.4 55720 35620 ?? I 4:35PM 0:00.56 /usr/local/sbin/dansguardian clamav 37909 0.0 0.4 55720 35416 ?? I 4:35PM 0:00.43 /usr/local/sbin/dansguardian clamav 38165 0.0 0.4 55720 35212 ?? I 4:35PM 0:00.53 /usr/local/sbin/dansguardian clamav 38358 0.0 0.4 57768 36044 ?? S 4:29PM 0:03.52 /usr/local/sbin/dansguardian clamav 38619 0.0 0.4 57768 35536 ?? S 4:29PM 0:03.15 /usr/local/sbin/dansguardian clamav 38740 0.0 0.4 57768 36816 ?? I 4:29PM 0:03.75 /usr/local/sbin/dansguardian clamav 38897 0.0 0.4 57768 35360 ?? I 4:29PM 0:25.72 /usr/local/sbin/dansguardian clamav 38899 0.0 0.4 55720 35428 ?? S 4:29PM 0:02.14 /usr/local/sbin/dansguardian clamav 38905 0.0 0.4 55720 35364 ?? S 4:29PM 0:03.22 /usr/local/sbin/dansguardian clamav 39087 0.0 0.4 57768 36248 ?? I 4:29PM 0:03.17 /usr/local/sbin/dansguardian clamav 39388 0.0 0.4 57768 36156 ?? I 4:29PM 0:03.02 /usr/local/sbin/dansguardian clamav 39572 0.0 0.4 55720 35312 ?? I 4:35PM 0:00.32 /usr/local/sbin/dansguardian clamav 39664 0.0 0.4 55720 35392 ?? I 4:29PM 0:03.83 /usr/local/sbin/dansguardian clamav 39695 0.0 0.4 55720 35288 ?? I 4:35PM 0:00.64 /usr/local/sbin/dansguardian clamav 39877 0.0 0.4 57768 35652 ?? I 4:35PM 0:00.39 /usr/local/sbin/dansguardian clamav 39963 0.0 0.4 55720 35404 ?? I 4:29PM 0:03.20 /usr/local/sbin/dansguardian clamav 40058 0.0 0.4 55720 35000 ?? I 4:35PM 0:00.37 /usr/local/sbin/dansguardian clamav 40236 0.0 0.4 57768 35584 ?? I 4:35PM 0:00.39 /usr/local/sbin/dansguardian clamav 40395 0.0 0.4 55720 35168 ?? I 4:35PM 0:00.32 /usr/local/sbin/dansguardian clamav 40434 0.0 0.4 55720 35416 ?? I 4:35PM 0:00.32 /usr/local/sbin/dansguardian clamav 40465 0.0 0.4 57768 35296 ?? I 4:29PM 0:03.44 /usr/local/sbin/dansguardian clamav 40713 0.0 0.4 57768 35660 ?? I 4:35PM 0:00.35 /usr/local/sbin/dansguardian clamav 40835 0.0 0.4 57768 36096 ?? S 4:29PM 0:03.06 /usr/local/sbin/dansguardian clamav 41012 0.0 0.4 55720 34968 ?? I 4:35PM 0:00.65 /usr/local/sbin/dansguardian clamav 41156 0.0 0.4 55720 35668 ?? I 4:29PM 0:03.92 /usr/local/sbin/dansguardian clamav 41273 0.0 0.4 55720 35196 ?? I 4:35PM 0:00.28 /usr/local/sbin/dansguardian clamav 41354 0.0 0.4 55720 35004 ?? I 4:35PM 0:00.25 /usr/local/sbin/dansguardian clamav 41471 0.0 0.4 55720 35492 ?? I 4:29PM 0:03.16 /usr/local/sbin/dansguardian clamav 41525 0.0 0.4 55720 34968 ?? I 4:35PM 0:00.37 /usr/local/sbin/dansguardian clamav 41556 0.0 0.4 55720 34960 ?? I 4:35PM 0:00.30 /usr/local/sbin/dansguardian clamav 41596 0.0 0.4 57768 36020 ?? S 4:29PM 0:03.44 /usr/local/sbin/dansguardian clamav 41813 0.0 0.4 57768 36104 ?? I 4:29PM 0:02.76 /usr/local/sbin/dansguardian clamav 41869 0.0 0.4 57768 35876 ?? S 4:35PM 0:00.33 /usr/local/sbin/dansguardian clamav 41951 0.0 0.4 55720 35112 ?? I 4:35PM 0:00.37 /usr/local/sbin/dansguardian clamav 42127 0.0 0.4 57768 35644 ?? I 4:29PM 0:03.53 /usr/local/sbin/dansguardian clamav 42209 0.0 3.1 298568 260464 ?? Ss 4:29PM 2:33.38 /usr/pbi/dansguardian-amd64/sbin/clamd clamav 42297 0.0 0.4 57768 36168 ?? I 4:35PM 0:00.33 /usr/local/sbin/dansguardian clamav 42335 0.0 0.4 57768 36272 ?? I 4:29PM 0:02.82 /usr/local/sbin/dansguardian clamav 42535 0.0 0.4 55720 35440 ?? S 4:29PM 0:02.75 /usr/local/sbin/dansguardian clamav 42547 0.0 0.4 55720 34952 ?? I 4:35PM 0:00.33 /usr/local/sbin/dansguardian clamav 42594 0.0 0.4 55720 34988 ?? I 4:35PM 0:00.30 /usr/local/sbin/dansguardian clamav 42709 0.0 0.4 57768 36152 ?? I 4:29PM 0:02.36 /usr/local/sbin/dansguardian clamav 42731 0.0 0.4 57768 36200 ?? S 4:29PM 0:03.19 /usr/local/sbin/dansguardian clamav 42776 0.0 0.4 57768 35668 ?? I 4:35PM 0:00.38 /usr/local/sbin/dansguardian clamav 42879 0.0 0.4 55720 35388 ?? S 4:29PM 0:02.09 /usr/local/sbin/dansguardian clamav 42897 0.0 0.4 55720 35336 ?? I 4:35PM 0:00.57 /usr/local/sbin/dansguardian clamav 43158 0.0 0.4 57768 36104 ?? I 4:29PM 0:03.36 /usr/local/sbin/dansguardian clamav 43218 0.0 0.4 55720 34968 ?? I 4:29PM 0:00.57 /usr/local/sbin/dansguardian clamav 43449 0.0 0.4 55720 35392 ?? I 4:29PM 0:02.77 /usr/local/sbin/dansguardian clamav 43607 0.0 0.4 57768 35748 ?? I 4:29PM 0:02.52 /usr/local/sbin/dansguardian clamav 43790 0.0 0.4 57768 35992 ?? I 4:29PM 0:02.76 /usr/local/sbin/dansguardian clamav 44115 0.0 0.4 55720 35384 ?? I 4:29PM 0:02.34 /usr/local/sbin/dansguardian clamav 44312 0.0 0.4 57768 36132 ?? I 4:29PM 0:02.52 /usr/local/sbin/dansguardian clamav 44394 0.0 0.4 55720 35312 ?? I 4:29PM 0:00.77 /usr/local/sbin/dansguardian clamav 44550 0.0 0.4 55720 35580 ?? I 4:29PM 0:02.11 /usr/local/sbin/dansguardian clamav 44802 0.0 0.4 57768 36252 ?? I 4:29PM 0:01.98 /usr/local/sbin/dansguardian clamav 44907 0.0 0.4 55720 35604 ?? I 4:29PM 0:02.26 /usr/local/sbin/dansguardian clamav 44935 0.0 0.4 55720 35288 ?? I 4:29PM 0:00.69 /usr/local/sbin/dansguardian clamav 45257 0.0 0.4 57768 36624 ?? I 4:29PM 0:02.64 /usr/local/sbin/dansguardian clamav 45589 0.0 0.4 57768 35164 ?? I 4:29PM 0:02.26 /usr/local/sbin/dansguardian clamav 46536 0.0 0.4 55720 35024 ?? I 4:36PM 0:00.36 /usr/local/sbin/dansguardian clamav 46826 0.0 0.4 55720 34952 ?? I 4:36PM 0:00.23 /usr/local/sbin/dansguardian clamav 46978 0.0 0.4 55720 35140 ?? I 4:36PM 0:00.25 /usr/local/sbin/dansguardian clamav 47073 0.0 0.4 55720 34964 ?? I 4:36PM 0:00.26 /usr/local/sbin/dansguardian clamav 47373 0.0 0.4 55720 35204 ?? I 4:36PM 0:00.37 /usr/local/sbin/dansguardian clamav 47680 0.0 0.4 55720 34976 ?? I 4:36PM 0:00.22 /usr/local/sbin/dansguardian clamav 47697 0.0 0.4 55720 34952 ?? I 4:36PM 0:00.27 /usr/local/sbin/dansguardian clamav 47918 0.0 0.4 57768 35692 ?? I 4:36PM 0:00.37 /usr/local/sbin/dansguardian clamav 47980 0.0 0.4 55720 34956 ?? I 4:36PM 0:00.15 /usr/local/sbin/dansguardian clamav 48013 0.0 0.4 55720 35144 ?? S 4:36PM 0:00.20 /usr/local/sbin/dansguardian clamav 48300 0.0 0.4 57768 36204 ?? I 4:29PM 0:01.86 /usr/local/sbin/dansguardian clamav 48304 0.0 0.4 57768 36684 ?? I 4:36PM 0:00.21 /usr/local/sbin/dansguardian clamav 48372 0.0 0.4 57768 35664 ?? I 4:36PM 0:00.42 /usr/local/sbin/dansguardian clamav 48428 0.0 0.4 57768 35980 ?? I 4:29PM 0:02.27 /usr/local/sbin/dansguardian clamav 48437 0.0 0.4 57768 35736 ?? I 4:29PM 0:01.84 /usr/local/sbin/dansguardian clamav 48537 0.0 0.4 55720 34952 ?? I 4:36PM 0:00.15 /usr/local/sbin/dansguardian clamav 48716 0.0 0.4 57768 37060 ?? I 4:29PM 0:02.47 /usr/local/sbin/dansguardian clamav 48812 0.0 0.4 55720 35000 ?? I 4:36PM 0:00.13 /usr/local/sbin/dansguardian clamav 48878 0.0 0.4 57768 36048 ?? S 4:29PM 0:02.80 /usr/local/sbin/dansguardian clamav 49151 0.0 0.4 55720 34972 ?? I 4:36PM 0:00.22 /usr/local/sbin/dansguardian clamav 49155 0.0 0.4 57768 36612 ?? S 4:29PM 0:01.86 /usr/local/sbin/dansguardian clamav 49269 0.0 0.4 55720 34964 ?? I 4:36PM 0:00.23 /usr/local/sbin/dansguardian clamav 49364 0.0 0.4 57768 35504 ?? I 4:29PM 0:02.03 /usr/local/sbin/dansguardian clamav 49492 0.0 0.4 55720 34960 ?? I 4:36PM 0:00.17 /usr/local/sbin/dansguardian clamav 49598 0.0 0.4 55720 34952 ?? I 4:36PM 0:00.16 /usr/local/sbin/dansguardian clamav 49639 0.0 0.4 57768 36120 ?? I 4:29PM 0:02.26 /usr/local/sbin/dansguardian clamav 49912 0.0 0.4 57768 36112 ?? I 4:36PM 0:00.28 /usr/local/sbin/dansguardian clamav 49933 0.0 0.4 57768 36168 ?? I 4:29PM 0:02.11 /usr/local/sbin/dansguardian clamav 50165 0.0 0.4 55720 35368 ?? I 4:36PM 0:00.17 /usr/local/sbin/dansguardian clamav 50176 0.0 0.4 57768 35380 ?? I 4:29PM 0:01.67 /usr/local/sbin/dansguardian clamav 50211 0.0 0.4 57768 36176 ?? I 4:29PM 0:03.94 /usr/local/sbin/dansguardian clamav 50254 0.0 0.4 57768 36020 ?? S 4:29PM 0:01.62 /usr/local/sbin/dansguardian clamav 50404 0.0 0.4 55720 35640 ?? I 4:29PM 0:01.92 /usr/local/sbin/dansguardian clamav 50631 0.0 0.4 57768 36076 ?? I 4:29PM 0:01.68 /usr/local/sbin/dansguardian clamav 50905 0.0 0.4 55720 35544 ?? I 4:29PM 0:01.58 /usr/local/sbin/dansguardian clamav 51233 0.0 0.4 57768 35816 ?? I 4:29PM 0:03.95 /usr/local/sbin/dansguardian clamav 51490 0.0 0.4 55720 35256 ?? S 4:29PM 0:01.64 /usr/local/sbin/dansguardian clamav 51669 0.0 0.4 55720 35308 ?? I 4:29PM 0:02.07 /usr/local/sbin/dansguardian clamav 51775 0.0 0.4 55720 35292 ?? S 4:29PM 0:01.79 /usr/local/sbin/dansguardian clamav 51920 0.0 0.4 57768 35768 ?? I 4:29PM 0:02.29 /usr/local/sbin/dansguardian clamav 52027 0.0 0.4 57768 36200 ?? S 10:18AM 0:01.08 /usr/local/sbin/dansguardian clamav 52042 0.0 0.4 57768 36304 ?? I 4:29PM 0:01.53 /usr/local/sbin/dansguardian clamav 52126 0.0 0.4 57768 35068 ?? S 4:29PM 0:01.54 /usr/local/sbin/dansguardian clamav 52487 0.0 0.4 57768 36256 ?? I 4:29PM 0:01.62 /usr/local/sbin/dansguardian clamav 52693 0.0 0.4 55720 35228 ?? I 4:29PM 0:01.51 /usr/local/sbin/dansguardian clamav 52968 0.0 0.4 55720 35456 ?? I 4:29PM 0:01.30 /usr/local/sbin/dansguardian clamav 53215 0.0 0.4 57768 35664 ?? I 4:29PM 0:01.52 /usr/local/sbin/dansguardian clamav 53364 0.0 0.4 55720 35624 ?? I 4:29PM 0:01.45 /usr/local/sbin/dansguardian clamav 53416 0.0 0.4 57768 35128 ?? I 4:29PM 0:01.73 /usr/local/sbin/dansguardian root 53585 0.0 0.0 5784 1096 ?? Is 29Sep13 0:00.00 /usr/local/bin/minicron 240 /var/run/ping_hosts.pid /usr/local/bin/ping_hosts.sh clamav 53725 0.0 0.4 55720 35420 ?? I 4:29PM 0:01.56 /usr/local/sbin/dansguardian root 53849 0.0 0.0 5784 1144 ?? I 29Sep13 0:00.50 minicron: helper /usr/local/bin/ping_hosts.sh (minicron) clamav 53890 0.0 0.4 57768 35472 ?? I 4:29PM 0:01.27 /usr/local/sbin/dansguardian clamav 54038 0.0 0.4 57768 36748 ?? I 4:29PM 0:01.48 /usr/local/sbin/dansguardian root 54046 0.0 0.0 5784 1096 ?? Is 29Sep13 0:00.00 /usr/local/bin/minicron 3600 /var/run/expire_accounts.pid /etc/rc.expireaccounts clamav 54107 0.0 0.4 57768 35756 ?? S 4:29PM 0:01.56 /usr/local/sbin/dansguardian clamav 54152 0.0 0.4 55720 35632 ?? S 4:29PM 0:02.10 /usr/local/sbin/dansguardian root 54269 0.0 0.0 5784 1144 ?? I 29Sep13 0:00.04 minicron: helper /etc/rc.expireaccounts (minicron) clamav 54508 0.0 0.4 55720 35332 ?? I 4:29PM 0:01.62 /usr/local/sbin/dansguardian root 54534 0.0 0.0 5784 1096 ?? Is 29Sep13 0:00.00 /usr/local/bin/minicron 86400 /var/run/update_alias_url_data.pid /etc/rc.update_alias_url_data root 54891 0.0 0.0 5784 1144 ?? I 29Sep13 0:00.00 minicron: helper /etc/rc.update_alias_url_data (minicron) clamav 55534 0.0 0.4 55720 35560 ?? I 7:09AM 0:03.12 /usr/local/sbin/dansguardian clamav 55543 0.0 0.4 57768 37232 ?? I 7:09AM 0:04.92 /usr/local/sbin/dansguardian clamav 55554 0.0 0.4 57768 36956 ?? S 7:09AM 0:03.13 /usr/local/sbin/dansguardian clamav 55802 0.0 0.4 57768 36096 ?? I 7:09AM 0:02.79 /usr/local/sbin/dansguardian root 55834 0.0 5.9 952676 491184 ?? Ss 29Sep13 166:18.81 /usr/local/bin/ntop -i em0,em4,em1,em3 -u root -d -4 -M -x 8102 -X 8192 root 55967 0.0 0.0 9084 1324 ?? Is 29Sep13 0:00.02 /usr/local/sbin/sshlockout_pf 15 clamav 56138 0.0 0.4 57768 35716 ?? I 7:09AM 0:04.63 /usr/local/sbin/dansguardian clamav 56288 0.0 0.4 57768 35720 ?? S 7:09AM 0:02.83 /usr/local/sbin/dansguardian root 56290 0.0 0.0 7928 1560 ?? I 4:00PM 0:00.00 cron: running job (cron) clamav 56334 0.0 0.4 57768 36180 ?? S 7:09AM 0:02.53 /usr/local/sbin/dansguardian clamav 56510 0.0 0.4 55720 35572 ?? S 7:09AM 0:02.41 /usr/local/sbin/dansguardian clamav 56788 0.0 0.4 57768 36180 ?? I 7:09AM 0:03.75 /usr/local/sbin/dansguardian clamav 57059 0.0 0.4 55720 35612 ?? I 7:09AM 0:02.76 /usr/local/sbin/dansguardian clamav 57338 0.0 0.4 55720 35548 ?? I 7:09AM 0:02.66 /usr/local/sbin/dansguardian root 57347 0.0 0.5 155432 45808 ?? Is 4:00PM 0:00.16 /usr/local/bin/php /usr/local/www/sarg.php 0 clamav 57376 0.0 0.4 55720 35404 ?? I 7:09AM 0:02.94 /usr/local/sbin/dansguardian clamav 57391 0.0 0.4 55720 35644 ?? S 7:09AM 0:02.70 /usr/local/sbin/dansguardian clamav 57511 0.0 0.4 57768 36068 ?? I 7:09AM 0:02.51 /usr/local/sbin/dansguardian clamav 57611 0.0 0.4 57768 35348 ?? S 7:09AM 0:01.65 /usr/local/sbin/dansguardian clamav 57910 0.0 0.4 57768 36056 ?? S 7:09AM 0:03.86 /usr/local/sbin/dansguardian clamav 58028 0.0 0.4 55720 35260 ?? S 7:09AM 0:00.38 /usr/local/sbin/dansguardian clamav 58372 0.0 0.4 55720 35276 ?? I 7:09AM 0:04.12 /usr/local/sbin/dansguardian clamav 58650 0.0 0.4 57768 35708 ?? I 7:09AM 0:02.02 /usr/local/sbin/dansguardian clamav 58899 0.0 0.4 55720 35524 ?? S 7:09AM 0:03.82 /usr/local/sbin/dansguardian clamav 59068 0.0 0.4 57768 36124 ?? I 7:09AM 0:01.95 /usr/local/sbin/dansguardian clamav 59267 0.0 0.4 55720 35320 ?? I 7:09AM 0:04.04 /usr/local/sbin/dansguardian clamav 59330 0.0 0.4 55720 35576 ?? I 7:09AM 0:02.07 /usr/local/sbin/dansguardian clamav 59511 0.0 0.4 55720 35612 ?? I 7:09AM 0:02.14 /usr/local/sbin/dansguardian clamav 59687 0.0 0.4 55720 34964 ?? I 7:09AM 0:00.97 /usr/local/sbin/dansguardian clamav 59771 0.0 0.4 55720 35640 ?? I 7:09AM 0:01.78 /usr/local/sbin/dansguardian clamav 60009 0.0 0.4 55720 35464 ?? I 7:09AM 0:02.08 /usr/local/sbin/dansguardian clamav 60339 0.0 0.4 57768 35804 ?? I 7:09AM 0:02.01 /usr/local/sbin/dansguardian clamav 60342 0.0 0.4 55720 35196 ?? I 4:33PM 0:01.22 /usr/local/sbin/dansguardian clamav 60510 0.0 0.4 57768 36084 ?? S 7:09AM 0:01.86 /usr/local/sbin/dansguardian clamav 60664 0.0 0.4 57768 35652 ?? I 4:33PM 0:00.96 /usr/local/sbin/dansguardian clamav 60704 0.0 0.4 55720 34948 ?? I 7:09AM 0:00.07 /usr/local/sbin/dansguardian clamav 60773 0.0 0.4 55720 34944 ?? I 7:09AM 0:00.07 /usr/local/sbin/dansguardian clamav 60828 0.0 0.4 55720 35464 ?? I 4:33PM 0:01.18 /usr/local/sbin/dansguardian clamav 60917 0.0 0.4 55720 35380 ?? I 4:33PM 0:01.16 /usr/local/sbin/dansguardian clamav 60946 0.0 0.4 55720 34932 ?? I 7:09AM 0:00.06 /usr/local/sbin/dansguardian clamav 61101 0.0 0.4 55720 34928 ?? I 7:09AM 0:00.04 /usr/local/sbin/dansguardian clamav 61421 0.0 0.4 55720 34920 ?? I 7:09AM 0:00.04 /usr/local/sbin/dansguardian clamav 61506 0.0 0.4 55720 35344 ?? S 4:33PM 0:01.01 /usr/local/sbin/dansguardian clamav 61748 0.0 0.4 55720 34928 ?? I 7:09AM 0:00.04 /usr/local/sbin/dansguardian clamav 61790 0.0 0.4 55720 34844 ?? Ss Thu12PM 7:30.87 /usr/local/sbin/dansguardian clamav 61814 0.0 0.4 55720 35636 ?? I 4:33PM 0:01.36 /usr/local/sbin/dansguardian clamav 61867 0.0 0.4 57768 36320 ?? I 4:33PM 0:00.93 /usr/local/sbin/dansguardian clamav 61994 0.0 0.4 55720 34940 ?? I 7:09AM 0:00.05 /usr/local/sbin/dansguardian clamav 62010 0.0 0.4 55720 34936 ?? I 7:09AM 0:00.06 /usr/local/sbin/dansguardian clamav 62019 0.0 0.4 57768 35028 ?? I 4:33PM 0:01.44 /usr/local/sbin/dansguardian clamav 62083 0.0 0.4 55720 35324 ?? S 4:33PM 0:00.94 /usr/local/sbin/dansguardian clamav 62192 0.0 0.4 55720 34920 ?? I 7:09AM 0:00.06 /usr/local/sbin/dansguardian clamav 62380 0.0 0.4 55720 35232 ?? S 4:33PM 0:01.59 /usr/local/sbin/dansguardian clamav 62539 0.0 0.4 55720 34940 ?? I 7:09AM 0:00.04 /usr/local/sbin/dansguardian clamav 62671 0.0 0.4 55720 35576 ?? I 4:33PM 0:01.12 /usr/local/sbin/dansguardian clamav 62717 0.0 0.4 55720 34920 ?? I 7:09AM 0:00.03 /usr/local/sbin/dansguardian clamav 62762 0.0 0.4 55720 35336 ?? I 4:33PM 0:00.89 /usr/local/sbin/dansguardian clamav 62853 0.0 0.4 55720 35188 ?? I 7:09AM 0:00.05 /usr/local/sbin/dansguardian clamav 63081 0.0 0.4 55720 34936 ?? I 7:09AM 0:00.04 /usr/local/sbin/dansguardian clamav 63086 0.0 0.4 55720 34932 ?? I 7:09AM 0:00.04 /usr/local/sbin/dansguardian clamav 63106 0.0 0.4 55720 35464 ?? I 4:33PM 0:00.99 /usr/local/sbin/dansguardian clamav 63254 0.0 0.4 55720 34916 ?? I 7:09AM 0:00.04 /usr/local/sbin/dansguardian clamav 63407 0.0 0.4 57768 36160 ?? S 4:33PM 0:01.07 /usr/local/sbin/dansguardian clamav 63551 0.0 0.4 55720 34936 ?? I 7:09AM 0:00.07 /usr/local/sbin/dansguardian clamav 63609 0.0 0.4 55720 35456 ?? I 4:33PM 0:00.82 /usr/local/sbin/dansguardian clamav 63713 0.0 0.4 57768 35832 ?? I 4:33PM 0:01.16 /usr/local/sbin/dansguardian clamav 63794 0.0 0.4 55720 34928 ?? I 7:09AM 0:00.07 /usr/local/sbin/dansguardian clamav 63912 0.0 0.4 57768 35044 ?? I 4:33PM 0:01.03 /usr/local/sbin/dansguardian clamav 64082 0.0 0.4 55720 34920 ?? I 7:09AM 0:00.04 /usr/local/sbin/dansguardian clamav 64150 0.0 0.4 55720 35636 ?? I 4:33PM 0:00.87 /usr/local/sbin/dansguardian clamav 64317 0.0 0.4 55720 34940 ?? I 7:09AM 0:00.07 /usr/local/sbin/dansguardian clamav 64441 0.0 0.4 55720 34924 ?? I 7:09AM 0:00.04 /usr/local/sbin/dansguardian clamav 64448 0.0 0.4 57768 35980 ?? I 4:33PM 0:00.92 /usr/local/sbin/dansguardian clamav 64656 0.0 0.4 55720 34932 ?? I 7:09AM 0:00.29 /usr/local/sbin/dansguardian clamav 64660 0.0 0.4 57768 36300 ?? I 4:33PM 0:00.91 /usr/local/sbin/dansguardian clamav 64793 0.0 0.4 55720 34972 ?? I 4:33PM 0:00.91 /usr/local/sbin/dansguardian clamav 64898 0.0 0.4 55720 34932 ?? I 7:09AM 0:00.04 /usr/local/sbin/dansguardian clamav 65010 0.0 0.4 55720 35092 ?? I 4:33PM 0:00.89 /usr/local/sbin/dansguardian clamav 65157 0.0 0.4 55720 34936 ?? I 7:09AM 0:00.05 /usr/local/sbin/dansguardian clamav 65187 0.0 0.4 55720 35632 ?? I 4:33PM 0:00.67 /usr/local/sbin/dansguardian clamav 65299 0.0 0.4 55720 34920 ?? I 7:09AM 0:00.04 /usr/local/sbin/dansguardian clamav 65308 0.0 0.4 55720 34932 ?? I 7:09AM 0:00.04 /usr/local/sbin/dansguardian clamav 65317 0.0 0.4 57768 36716 ?? I 4:33PM 0:01.36 /usr/local/sbin/dansguardian clamav 65510 0.0 0.4 55720 35292 ?? I 4:33PM 0:00.84 /usr/local/sbin/dansguardian clamav 65556 0.0 0.4 57768 37116 ?? I 4:33PM 0:00.70 /usr/local/sbin/dansguardian clamav 65576 0.0 0.4 55720 34944 ?? I 7:09AM 0:00.06 /usr/local/sbin/dansguardian clamav 65625 0.0 0.4 57768 35092 ?? I 4:33PM 0:00.81 /usr/local/sbin/dansguardian clamav 65664 0.0 0.4 55720 35604 ?? S 4:33PM 0:00.83 /usr/local/sbin/dansguardian clamav 65676 0.0 0.4 55720 35116 ?? I 7:09AM 0:00.04 /usr/local/sbin/dansguardian clamav 65791 0.0 0.4 57768 36936 ?? I 4:33PM 0:00.94 /usr/local/sbin/dansguardian clamav 65962 0.0 0.4 55720 34920 ?? I 7:09AM 0:00.03 /usr/local/sbin/dansguardian root 66036 0.0 0.1 13796 4884 ?? S Mon04PM 0:03.97 /usr/sbin/tcpdump -s 256 -v -S -l -n -e -ttt -i pflog0 clamav 66059 0.0 0.4 55720 34916 ?? I 7:09AM 0:00.02 /usr/local/sbin/dansguardian clamav 66078 0.0 0.4 57768 35988 ?? I 4:33PM 0:00.87 /usr/local/sbin/dansguardian clamav 66109 0.0 0.4 55720 35572 ?? I 4:33PM 0:00.61 /usr/local/sbin/dansguardian clamav 66146 0.0 0.4 55720 34920 ?? I 7:09AM 0:00.03 /usr/local/sbin/dansguardian clamav 66237 0.0 0.4 55720 34920 ?? I 7:09AM 0:00.02 /usr/local/sbin/dansguardian root 66306 0.0 0.0 5780 1060 ?? S Mon04PM 0:03.35 logger -t pf -p local0.info clamav 66356 0.0 0.4 55720 34952 ?? I 7:09AM 0:00.04 /usr/local/sbin/dansguardian clamav 66451 0.0 0.4 55720 35288 ?? I 4:33PM 0:00.72 /usr/local/sbin/dansguardian clamav 66629 0.0 0.4 55720 34924 ?? I 7:09AM 0:00.05 /usr/local/sbin/dansguardian clamav 66691 0.0 0.4 55720 35356 ?? I 4:33PM 0:01.12 /usr/local/sbin/dansguardian clamav 66741 0.0 0.4 57768 36184 ?? I 4:33PM 0:00.68 /usr/local/sbin/dansguardian clamav 66843 0.0 0.4 55720 35188 ?? I 4:33PM 0:00.68 /usr/local/sbin/dansguardian clamav 66867 0.0 0.4 55720 35268 ?? I 4:33PM 0:00.54 /usr/local/sbin/dansguardian clamav 66928 0.0 0.4 55720 34924 ?? I 7:09AM 0:00.05 /usr/local/sbin/dansguardian clamav 67056 0.0 0.4 55720 35388 ?? I 4:33PM 0:00.61 /usr/local/sbin/dansguardian clamav 67202 0.0 0.4 55720 35596 ?? I 4:33PM 0:00.60 /usr/local/sbin/dansguardian clamav 67215 0.0 0.4 55720 34924 ?? I 7:09AM 0:00.04 /usr/local/sbin/dansguardian clamav 67298 0.0 0.4 55720 34936 ?? I 7:09AM 0:00.04 /usr/local/sbin/dansguardian clamav 67332 0.0 0.4 55720 34916 ?? I 7:09AM 0:00.03 /usr/local/sbin/dansguardian clamav 67488 0.0 0.4 55720 34920 ?? I 7:09AM 0:00.04 /usr/local/sbin/dansguardian clamav 67812 0.0 0.4 55720 34924 ?? I 7:09AM 0:00.04 /usr/local/sbin/dansguardian clamav 68075 0.0 0.4 55720 34936 ?? I 7:09AM 0:00.03 /usr/local/sbin/dansguardian clamav 68288 0.0 0.4 55720 34916 ?? I 7:09AM 0:00.03 /usr/local/sbin/dansguardian clamav 68315 0.0 0.4 55720 34916 ?? I 7:09AM 0:00.02 /usr/local/sbin/dansguardian clamav 68348 0.0 0.4 55720 34916 ?? I 7:09AM 0:00.02 /usr/local/sbin/dansguardian clamav 68408 0.0 0.4 55720 34920 ?? I 7:09AM 0:00.02 /usr/local/sbin/dansguardian clamav 68565 0.0 0.4 55720 34916 ?? I 7:09AM 0:00.03 /usr/local/sbin/dansguardian clamav 68746 0.0 0.4 55720 34956 ?? I 7:09AM 0:00.04 /usr/local/sbin/dansguardian clamav 68970 0.0 0.4 55720 34928 ?? I 7:09AM 0:00.03 /usr/local/sbin/dansguardian clamav 69198 0.0 0.4 55720 34916 ?? I 7:09AM 0:00.02 /usr/local/sbin/dansguardian clamav 69431 0.0 0.4 55720 34916 ?? I 7:09AM 0:00.03 /usr/local/sbin/dansguardian clamav 69489 0.0 0.4 55720 34916 ?? I 7:09AM 0:00.02 /usr/local/sbin/dansguardian clamav 69492 0.0 0.4 55720 34952 ?? I 7:09AM 0:00.03 /usr/local/sbin/dansguardian clamav 69795 0.0 0.4 55720 35204 ?? I 7:09AM 0:00.03 /usr/local/sbin/dansguardian clamav 69988 0.0 0.4 55720 34932 ?? I 7:09AM 0:00.06 /usr/local/sbin/dansguardian clamav 70328 0.0 0.4 55720 35204 ?? I 7:09AM 0:00.03 /usr/local/sbin/dansguardian clamav 70565 0.0 0.4 55720 34916 ?? I 7:09AM 0:00.02 /usr/local/sbin/dansguardian clamav 70674 0.0 0.4 55720 34920 ?? I 7:09AM 0:00.03 /usr/local/sbin/dansguardian clamav 70753 0.0 0.4 55720 34916 ?? I 7:09AM 0:00.02 /usr/local/sbin/dansguardian clamav 70928 0.0 0.4 55720 34920 ?? I 7:09AM 0:00.02 /usr/local/sbin/dansguardian clamav 71077 0.0 0.4 55720 34916 ?? I 7:09AM 0:00.02 /usr/local/sbin/dansguardian clamav 71100 0.0 0.4 55720 34916 ?? I 7:09AM 0:00.02 /usr/local/sbin/dansguardian clamav 71224 0.0 0.4 55720 34920 ?? I 7:09AM 0:00.04 /usr/local/sbin/dansguardian clamav 71556 0.0 0.4 55720 34916 ?? I 7:09AM 0:00.03 /usr/local/sbin/dansguardian clamav 71767 0.0 0.4 55720 34928 ?? I 7:09AM 0:00.02 /usr/local/sbin/dansguardian clamav 71933 0.0 0.4 55720 34916 ?? I 7:09AM 0:00.02 /usr/local/sbin/dansguardian clamav 72045 0.0 0.4 55720 34920 ?? I 7:09AM 0:00.03 /usr/local/sbin/dansguardian clamav 72357 0.0 0.4 55720 34916 ?? I 7:09AM 0:00.02 /usr/local/sbin/dansguardian clamav 72596 0.0 0.4 55720 34916 ?? I 7:09AM 0:00.02 /usr/local/sbin/dansguardian clamav 72726 0.0 0.4 55720 34916 ?? I 7:09AM 0:00.01 /usr/local/sbin/dansguardian clamav 72734 0.0 0.4 55720 34920 ?? I 7:09AM 0:00.01 /usr/local/sbin/dansguardian clamav 73046 0.0 0.4 55720 34916 ?? I 7:09AM 0:00.02 /usr/local/sbin/dansguardian clamav 73277 0.0 0.4 55720 35356 ?? I 4:36PM 0:00.23 /usr/local/sbin/dansguardian clamav 73308 0.0 0.4 55720 35132 ?? I 4:36PM 0:00.15 /usr/local/sbin/dansguardian clamav 73599 0.0 0.4 55720 34960 ?? I 4:36PM 0:00.14 /usr/local/sbin/dansguardian clamav 73766 0.0 0.4 55720 34976 ?? I 4:36PM 0:00.14 /usr/local/sbin/dansguardian root 73802 0.0 0.5 147240 40232 ?? I 3:56PM 0:06.84 /usr/local/bin/php clamav 74055 0.0 0.4 55720 35404 ?? I 4:36PM 0:00.17 /usr/local/sbin/dansguardian clamav 74077 0.0 0.4 55720 34968 ?? I 4:36PM 0:00.19 /usr/local/sbin/dansguardian clamav 74301 0.0 0.4 55720 34944 ?? I 4:36PM 0:00.17 /usr/local/sbin/dansguardian clamav 74428 0.0 0.4 55720 35168 ?? I 4:36PM 0:00.19 /usr/local/sbin/dansguardian clamav 74539 0.0 0.4 55720 34948 ?? I 4:36PM 0:00.24 /usr/local/sbin/dansguardian clamav 74865 0.0 0.4 55720 34964 ?? I 4:36PM 0:00.21 /usr/local/sbin/dansguardian clamav 75153 0.0 0.4 55720 35116 ?? I 4:36PM 0:00.15 /usr/local/sbin/dansguardian clamav 75218 0.0 0.4 57768 35516 ?? I 4:36PM 0:00.15 /usr/local/sbin/dansguardian clamav 75265 0.0 0.4 57768 35776 ?? I 4:36PM 0:00.25 /usr/local/sbin/dansguardian clamav 75406 0.0 0.4 55720 34932 ?? I 4:36PM 0:00.10 /usr/local/sbin/dansguardian clamav 75457 0.0 0.4 55720 34944 ?? I 4:36PM 0:00.20 /usr/local/sbin/dansguardian clamav 75469 0.0 0.4 55720 35140 ?? I 4:36PM 0:00.23 /usr/local/sbin/dansguardian clamav 75606 0.0 0.4 55720 34948 ?? I 4:36PM 0:00.13 /usr/local/sbin/dansguardian clamav 75898 0.0 0.4 55720 35120 ?? I 4:36PM 0:00.14 /usr/local/sbin/dansguardian clamav 76065 0.0 0.4 55720 34952 ?? I 4:36PM 0:00.26 /usr/local/sbin/dansguardian clamav 76354 0.0 0.4 55720 35056 ?? I 4:36PM 0:00.11 /usr/local/sbin/dansguardian clamav 76672 0.0 0.4 55720 34948 ?? I 4:36PM 0:00.12 /usr/local/sbin/dansguardian clamav 76886 0.0 0.4 55720 34928 ?? I 4:36PM 0:00.12 /usr/local/sbin/dansguardian clamav 77061 0.0 0.4 55720 34932 ?? I 4:36PM 0:00.11 /usr/local/sbin/dansguardian clamav 77330 0.0 0.4 55720 34932 ?? I 4:36PM 0:00.13 /usr/local/sbin/dansguardian clamav 77594 0.0 0.4 57768 35508 ?? I 4:36PM 0:00.09 /usr/local/sbin/dansguardian clamav 77685 0.0 0.4 55720 35436 ?? I 4:36PM 0:00.08 /usr/local/sbin/dansguardian clamav 77906 0.0 0.4 55720 35152 ?? I 4:36PM 0:00.09 /usr/local/sbin/dansguardian clamav 78001 0.0 0.4 55720 34952 ?? I 4:36PM 0:00.11 /usr/local/sbin/dansguardian clamav 78334 0.0 0.4 57768 35636 ?? I 4:36PM 0:00.12 /usr/local/sbin/dansguardian clamav 78485 0.0 0.4 55720 34948 ?? I 4:36PM 0:00.11 /usr/local/sbin/dansguardian clamav 78513 0.0 0.4 55720 34932 ?? I 4:36PM 0:00.09 /usr/local/sbin/dansguardian clamav 78770 0.0 0.4 55720 34944 ?? I 4:36PM 0:00.31 /usr/local/sbin/dansguardian root 78906 0.0 0.1 32412 7372 ?? S 4:28PM 0:16.21 /usr/local/sbin/lighttpd -f /var/etc/lighty-webConfigurator.conf clamav 78981 0.0 0.4 55720 34956 ?? I 4:36PM 0:00.10 /usr/local/sbin/dansguardian clamav 79057 0.0 0.4 55720 34940 ?? I 4:36PM 0:00.06 /usr/local/sbin/dansguardian root 79384 0.0 0.2 139048 18336 ?? Is 4:28PM 0:00.05 /usr/local/bin/php root 80970 0.0 0.3 49924 25120 ?? Ss 29Sep13 18:44.36 /usr/sbin/bsnmpd -c /var/etc/snmpd.conf -p /var/run/snmpd.pid root 81369 0.0 0.0 2716 880 ?? IN 4:11PM 0:00.00 sleep 60 root 81701 0.0 0.0 26168 3584 ?? Ss 4:11PM 0:00.02 sshd: root@pts/1 (sshd) clamav 83032 0.0 0.4 55720 34960 ?? I 1:55PM 0:00.38 /usr/local/sbin/dansguardian clamav 83164 0.0 0.4 55720 35168 ?? I 1:55PM 0:00.38 /usr/local/sbin/dansguardian clamav 83269 0.0 0.4 55720 34964 ?? I 1:55PM 0:00.38 /usr/local/sbin/dansguardian clamav 83413 0.0 0.4 55720 34916 ?? I 1:55PM 0:00.01 /usr/local/sbin/dansguardian clamav 83498 0.0 0.4 55720 34920 ?? I 1:55PM 0:00.01 /usr/local/sbin/dansguardian clamav 83504 0.0 0.4 55720 34904 ?? I 1:55PM 0:00.01 /usr/local/sbin/dansguardian clamav 83645 0.0 0.4 55720 34904 ?? I 1:55PM 0:00.01 /usr/local/sbin/dansguardian clamav 83666 0.0 0.4 55720 34912 ?? I 1:55PM 0:00.01 /usr/local/sbin/dansguardian clamav 83737 0.0 0.4 55720 34916 ?? I 1:55PM 0:00.01 /usr/local/sbin/dansguardian clamav 83785 0.0 0.4 55720 34912 ?? I 1:55PM 0:00.01 /usr/local/sbin/dansguardian clamav 83893 0.0 0.4 55720 34904 ?? I 1:55PM 0:00.01 /usr/local/sbin/dansguardian clamav 84016 0.0 0.4 55720 34904 ?? I 1:55PM 0:00.01 /usr/local/sbin/dansguardian clamav 84079 0.0 0.4 55720 34916 ?? I 1:55PM 0:00.01 /usr/local/sbin/dansguardian clamav 84201 0.0 0.4 55720 34904 ?? I 1:55PM 0:00.00 /usr/local/sbin/dansguardian root 84433 0.0 0.0 2716 880 ?? IN 4:11PM 0:00.00 sleep 55 clamav 84436 0.0 0.4 55720 34912 ?? I 1:55PM 0:00.01 /usr/local/sbin/dansguardian root 91780 0.0 0.2 139048 18336 ?? Is 4:28PM 0:00.08 /usr/local/bin/php root 95972 0.0 0.0 6956 1552 ?? Ss 29Sep13 1:44.09 /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -f /var/etc/syslog.conf root 55672 0.0 0.0 19480 1696 v0 Is 29Sep13 0:00.02 login [pam] (login) root 56282 0.0 0.0 8296 1460 v0 I 29Sep13 0:00.00 -sh (sh) root 58194 0.0 0.0 8296 1460 v0 I+ 29Sep13 0:00.00 /bin/sh /etc/rc.initial root 79823 0.0 0.0 8296 1716 0- IN 4:28PM 0:12.21 /bin/sh /var/db/rrd/updaterrd.sh root 84550 0.0 0.0 8296 1644 1 Is 4:11PM 0:00.00 -sh (sh) root 85190 0.0 0.0 8296 1648 1 I 4:11PM 0:00.00 /bin/sh /etc/rc.initial root 87858 0.0 0.0 8268 2436 1 S 4:11PM 0:00.01 /bin/tcsh root 88539 0.0 0.0 7992 1844 1 R+ 4:12PM 0:00.00 ps uxawww
-
We are not running sipproxd or any other add-on package for that matter. We have about 20 phones that use tftp to download configuration from an Internet-based SIP provider, so we use tftp-proxy a bit. Other than that most of our traffic is SIP/RTP, and plain old Internet browsing (DNS, HTTP, HTTPS, etc.).
It's not a very complicated setup. We had some traffic shaping set up for the VOIP traffic, and we used the "Single Wan multi Lan" wizard to set it up. VOIP traffic is on a different VLAN than the rest of the traffic, so that wizard seemed appropriate. I have since disabled the traffic shaper in an attempt to narrow down the problem.
-
Strange - My memory utilization has creeped up to over 80% now, but I suspect it won't go much higher and I think its pure squid usage. What do your states look like?
-
States almost never go above 2000. They're pretty consistent too. There aren't spikes in the state table size when the problem occurs.
-
It's also interesting to note that when the firewall starts having issues, neither the processor usage nor the memory usage is above 50%.
-
kejianshi,
The dansguardian processes were in a post from timthetortoise who says he's not having issues. I (Jason, the OP) am not running dansguardian or ClamAV, but I am having a memory leak issue. My process lists is in the original post.
Thanks for your help though!
-
HAHA - Yea. I saw that after I posted and realized my mistake and deleted it… I was hoping no one saw it. You caught me!
Question. Is this system upgraded or is it a fresh clean install?
I've been told repeatedly that that shouldn't matter, but I'm not sure how true that is.
-
The system was upgraded to 2.1–not a clean install. The problem existed both on 2.0.2 and continues on 2.1.
-
Is a new clean install out of the question?
-
At this point, nothing is out of the question. Of course a new install is a pain, but we have to solve the problem. The last thing we did was turn off the traffic shaper all together. The folks behind this firewall don't seem to consume their Internet pipe (10Mb symmetrical), so the traffic shaper isn't that big of a deal. Now I'm just waiting for the memory leak to start again. Sometimes it takes a few days; others it takes a month or more.
We're in a virtualized environment, so I've considered setting up a separate gateway in a high-availability configuration. It'd give me somewhere to move traffic when the problem occurs assuming the memory leak doesn't show up on both firewalls at the same time. That's a few hours work and I have to make sure I have extra Internet IPs available, but it would reduce the risk of having to incur a middle-of-the-workday outage.
-
"traffic shaper" - Is that no longer running?
-
The traffic shaper is no longer running.
-
I'd wipe the drive and install clean. Unless your rules are special in some way, I would not even do a restore of settings. I'd redo them by hand to be sure you didn't import trouble. Is this a 64 or 32 bit install? Because I'd also prefer 32bit.
-
For your information, I think this problem has followed us through several iterations of pfSense from 1.2.3 (i think) on an ALIX board to 2.0.1 on a PC to 2.0.2 on a virtual server to 2.1 on the same server. I do think the same XML config has been ported through all of those environments–both 32- and 64-bit. I guess I'm in agreement that a scratch rebuild might be the next good thing to do.
It's an amd64 install today. I'm curious, why do you prefer 32-bit? Does anyone else have thoughts on 32-bit vs 64-bit?
-
Because with less than 4GB ram you don't NEED 64bit version and I've seen more issues with 64 than 32. Some would say its my imagination, but I don't think so. I think the 32bit version is just working better under VMs. I've experienced that anyway. Seems others have also, but thats more of something I've noticed than a scientific study of the issue.
-
I have a customer on a 64 bit version and it behaves stupendously. Not one issue! :)
-
We also have at least six other 64-bit installs that are working flawlessly. A couple of them have fairly complicated configurations with ~10 interfaces, ~20 IPsec tunnels, 100s of firewall & NAT rules, traffic shaping, etc. With all of that, they're still able to sustain 100Mb/s throughput easily. I'm curious if there would be performance degradation moving from 64-bit to 32-bit.
-
The problems I personally experienced were on 64bit version on ESXi 5.
Same install, nothing else modified and it worked fine. Tried several times.
I couldn't solve it, so I just stayed with 32bit version.
Sounds like your hypervisor isn't same, so probably depends.
Also, the problems I had were mainly either with NTP or with nothing working well with 5 or more interfaces.
There are just too many variables in hardware and network to say anthing for sure for everybody, but as a rule, if I don't need 64bit, I use 32bit where I can. As for it being a point of contention for this or that person, I say do whatever floats your boat and works for you.
There won't be any performance losses. 64bit is just able to address more RAM. Thats all.
-
Jason: Did you ever solve this?
I ended up just re-installing the firmware over the original install via the firmware page and that seems to have solved it for me here.
My other watchguard "customer" had his unit do the same thing starting last Wednesday and build up to over 30% before we caught it. Restarted Apinger and it has been normal since…
-
Since we've been through at least four firmware updates on this firewall and the problem still persists, I'm doubting a firmware reinstall will fix the problem. I'm also pretty sure apinger is not the problem as I believe that would cause memory usage to spike in user space. I'm seeing wired kernel memory balloon.
My plan at this point is to build another near-identical firewall through the web gui, download the XML config from the new and old ones, and compare. Maybe that will reveal some anomaly. This XML has been pulled up through the 1.2 versions to 2.1. Maybe something bad slipped through.
-
My memory does exactly the same thing. Looks like utilization climbs and climbs and then it hits about 90%, teeters there for a while and then will drop back down to about 30% all by its self, with no reset, reboot or anything and no ill effects and then the cycle starts again. I don't worry about it.
-
Im actually still seeing this intermittently on a couple of boxes. Though last time it happened to me here it actually used up all memory before it reset itself.
But since Ive reset the webgui from a console it has not occurred. Re-booting the box never solved it. In fact it would simply do the same right after the re-boot.
-
When the memory usage is high, have you checked the process list to see what is using the RAM?
-
I'll chuck this in this mix as its running in a VM.
Brute force
http://www.run-virtual.com/?p=695then how to reset the VM password which might be sat in a data centre and doesnt get checked often
http://blog.geekdiary.me/?p=21because it might be running auto failover type of software like this:
http://www.veeam.com/virtual-machine-backup-solution-free.html
-
When the memory usage is high, have you checked the process list to see what is using the RAM?
Just this- https://forum.pfsense.org/index.php/topic,67703.msg370496.html#msg370496
It hasn't happened since the webgui reset from the console so I cant look now. Im watching for it.
-
After some testing it seems the LCDProc package is causing this for me.
When I do a webgui reset the display goes to a screen that shows Cli: 0 Scr: 0 with no backlight. I didn't notice it till days after my initial webgui reset. Once I restarted the LCD package the memory leak started again. Reset tonight and Im going to leave the package in the inop state for a few weeks.
Running the dev package.
