Wired memory usage slowy increasing
-
Hi!
I've had a pfsense box running for a couple of months as our company's internet gateway and I absolutely love it! So powerful… anyway, the basics of the setup are as follows:
AMD Phenom II X2 225 CPU w. 4GB DDR3 on an Asus M4N689T-M board, with an APC UPS plugged in via USB. There are four NICs:
nfe0 which is our LAN on 192.168.0.0/16 (LAN1)
A multi-wan setup on rl0 and rl1 involving PPPoE connections to our two ADSL modems
em0 which is separate LAN on 10.0.0.0/24 (LAN2)I've installed the Apcupsd package, Snort and the OpenVPN export utility (though the VPN is currently disabled)
We have the usual traffic between our LAN and the multi-wan load balancing gateway. The separate LAN is not physically secure; as such I've used the firewall/NAT to drop any inbound traffic to our LAN but allow connections from LAN1 to LAN2. LAN2 traffic is very high as it contains the (network block device, encrypted) offsite backup server and an IP camera monitoring the room that it's in.
All was seemingly fine until a recently when after a power failure and subsequent reboot the wired memory usage grew and grew until swapping occurred; after having a tinker I restarted the box and it was fine for a week and now the same thing has happened again. It seems that devbuf's memory usage grows over a couple of days until we run out of memory - aargh!
I'm not too familiar with BSD, my day job is Linux - but to me this looks like some sort of memory leak. I've not yet filed a bug as I don't think I've got enough useful information yet - I've included things I thought may be useful in the post, hopefully someone can point me where to dig next!
One odd thing I noticed was that dmesg shows me some sort of USB error. I found an old post on tinterwebs showing a similar thing to do with a USB1 hob - could this be related?
The system is currently still up and running and in this funny state - I reckon I've got about a day before it needs restarting, so if anything else useful can be gleaned let me know!
dmesg.boot
Copyright (c) 1992-2012 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.3-RELEASE-p16 #0: Mon Aug 25 08:27:11 EDT 2014 root@pf2_1_1_amd64.pfsense.org:/usr/obj.amd64/usr/pfSensesrc/src/sys/pfSense_SMP.8 amd64 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: AMD Phenom(tm) II X2 555 Processor (3214.64-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x100f43 Family = 10 Model = 4 Stepping = 3 Features=0x178bfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,mmx,fxsr,sse,sse2,htt>Features2=0x802009 <sse3,mon,cx16,popcnt>AMD Features=0xee500800 <syscall,nx,mmx+,ffxsr,page1gb,rdtscp,lm,3dnow!+,3dnow!>AMD Features2=0x37ff <lahf,cmp,svm,extapic,cr8,abm,sse4a,mas,prefetch,osvw,ibs,skinit,wdt>TSC: P-state invariant real memory = 4294967296 (4096 MB) avail memory = 4058845184 (3870 MB) ACPI APIC Table: <061210 APIC1416> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 1 package(s) x 2 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 ioapic0 <version 1.1=""> irqs 0-23 on motherboard wlan: mac acl policy registered ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/. ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff804abaf0, 0) error 1 ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/. ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff804abb90, 0) error 1 ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/. ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf. module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff804abc30, 0) error 1 kbd1 at kbdmux0 cryptosoft0: <software crypto=""> on motherboard padlock0: No ACE support. acpi0: <061210 XSDT1416> on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of fec00000, fed40000 (3) failed acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, bdf00000 (3) failed Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x508-0x50b on acpi0 cpu0: <acpi cpu=""> on acpi0 cpu1: <acpi cpu=""> on acpi0 pcib0: <acpi host-pci="" bridge=""> port 0xcf8-0xcff on acpi0 pci0: <acpi pci="" bus=""> on pcib0 pci0: <memory, ram=""> at device 0.0 (no driver attached) isab0: <pci-isa bridge=""> port 0x900-0x9ff at device 1.0 on pci0 isa0: <isa bus=""> on isab0 pci0: <serial bus,="" smbus=""> at device 1.1 (no driver attached) pci0: <memory, ram=""> at device 1.2 (no driver attached) ohci0: <nvidia nforce="" mcp61="" usb="" controller=""> mem 0xdfdfb000-0xdfdfbfff irq 21 at device 2.0 on pci0 ohci0: [ITHREAD] usbus0: <nvidia nforce="" mcp61="" usb="" controller=""> on ohci0 ehci0: <nvidia nforce="" mcp61="" usb="" 2.0="" controller=""> mem 0xdfdfac00-0xdfdfacff irq 22 at device 2.1 on pci0 ehci0: [ITHREAD] usbus1: EHCI version 1.0 usbus1: <nvidia nforce="" mcp61="" usb="" 2.0="" controller=""> on ehci0 pcib1: <acpi pci-pci="" bridge=""> at device 4.0 on pci0 pci1: <acpi pci="" bus=""> on pcib1 rl0: <realtek 10="" 8139="" 100basetx=""> port 0xd800-0xd8ff mem 0xdfeffc00-0xdfeffcff irq 16 at device 6.0 on pci1 miibus0: <mii bus=""> on rl0 rlphy0: <realtek internal="" media="" interface=""> PHY 0 on miibus0 rlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto rl0: [ITHREAD] rl1: <realtek 10="" 8139="" 100basetx=""> port 0xd400-0xd4ff mem 0xdfeff800-0xdfeff8ff irq 17 at device 7.0 on pci1 miibus1: <mii bus=""> on rl1 rlphy1: <realtek internal="" media="" interface=""> PHY 0 on miibus1 rlphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto rl1: [ITHREAD] atapci0: <nvidia nforce="" mcp61="" udma133="" controller=""> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 6.0 on pci0 ata0: <ata channel=""> at channel 0 on atapci0 ata0: [ITHREAD] ata1: <ata channel=""> at channel 1 on atapci0 ata1: [ITHREAD] nfe0: <nvidia nforce="" mcp61="" networking="" adapter=""> port 0xc480-0xc487 mem 0xdfdf9000-0xdfdf9fff irq 23 at device 7.0 on pci0 miibus2: <mii bus=""> on nfe0 rgephy0: <rtl8169s 8110s="" 8211b="" media="" interface=""> PHY 1 on miibus2 rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow nfe0: [FILTER] nfe0: [FILTER] nfe0: [FILTER] nfe0: [FILTER] nfe0: [FILTER] nfe0: [FILTER] nfe0: [FILTER] nfe0: [FILTER] atapci1: <nvidia nforce="" mcp61="" sata300="" controller=""> port 0xc400-0xc407,0xc080-0xc083,0xc000-0xc007,0xbc00-0xbc03,0xb880-0xb88f mem 0xdfdf8000-0xdfdf8fff irq 20 at device 8.0 on pci0 atapci1: [ITHREAD] ata2: <ata channel=""> at channel 0 on atapci1 ata2: [ITHREAD] ata3: <ata channel=""> at channel 1 on atapci1 ata3: [ITHREAD] atapci2: <nvidia nforce="" mcp61="" sata300="" controller=""> port 0xb800-0xb807,0xb480-0xb483,0xb400-0xb407,0xb080-0xb083,0xb000-0xb00f mem 0xdfdf3000-0xdfdf3fff irq 21 at device 8.1 on pci0 atapci2: [ITHREAD] ata4: <ata channel=""> at channel 0 on atapci2 ata4: [ITHREAD] ata5: <ata channel=""> at channel 1 on atapci2 ata5: [ITHREAD] pcib2: <acpi pci-pci="" bridge=""> at device 9.0 on pci0 pci2: <acpi pci="" bus=""> on pcib2 pcib3: <acpi pci-pci="" bridge=""> at device 11.0 on pci0 pci3: <acpi pci="" bus=""> on pcib3 em0: <intel(r) 1000="" pro="" network="" connection="" 7.3.8=""> port 0xec00-0xec1f mem 0xdffe0000-0xdfffffff,0xdff00000-0xdff7ffff,0xdffdc000-0xdffdffff irq 18 at device 0.0 on pci3 em0: Using MSIX interrupts with 3 vectors em0: [ITHREAD] em0: [ITHREAD] em0: [ITHREAD] pcib4: <acpi pci-pci="" bridge=""> at device 12.0 on pci0 pci4: <acpi pci="" bus=""> on pcib4 vgapci0: <vga-compatible display=""> mem 0xde000000-0xdeffffff,0xc0000000-0xcfffffff,0xdd000000-0xddffffff irq 22 at device 13.0 on pci0 acpi_button0: <power button=""> on acpi0 atrtc0: <at realtime="" clock=""> port 0x70-0x71 irq 8 on acpi0 ppc0: <parallel port=""> port 0x378-0x37f irq 7 on acpi0 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode ppc0: [ITHREAD] ppbus0: <parallel port="" bus=""> on ppc0 plip0: <plip network="" interface=""> on ppbus0 plip0: [ITHREAD] lpt0: <printer> on ppbus0 lpt0: [ITHREAD] lpt0: Interrupt-driven port ppi0: <parallel i="" o=""> on ppbus0 acpi_hpet0: <high precision="" event="" timer=""> iomem 0xfed00000-0xfed00fff on acpi0 Timecounter "HPET" frequency 25000000 Hz quality 900 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] uart0: console (115200,n,8,1) orm0: <isa option="" rom=""> at iomem 0xcf000-0xcffff on isa0 sc0: <system console=""> at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: <generic isa="" vga=""> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 atkbdc0: <keyboard controller="" (i8042)=""> at port 0x60,0x64 on isa0 atkbd0: <at keyboard=""> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] hwpstate0: <cool`n'quiet 2.0=""> on cpu0 Timecounters tick every 1.000 msec IPsec: Initialized Security Association Processing. usbus0: 12Mbps Full Speed USB v1.0 usbus1: 480Mbps High Speed USB v2.0 ad4: 476940MB <st500dm002 1bd142="" kc45=""> at ata2-master UDMA100 SATA 1.5Gb/s ugen0.1: <nvidia> at usbus0 uhub0: <nvidia 1="" 9="" ohci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr=""> on usbus0 ugen1.1: <nvidia> at usbus1 uhub1: <nvidia 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr=""> on usbus1 SMP: AP CPU #1 Launched! Root mount waiting for: usbus1 usbus0 uhub0: 10 ports with 10 removable, self powered Root mount waiting for: usbus1 Root mount waiting for: usbus1 Root mount waiting for: usbus1 uhub1: 10 ports with 10 removable, self powered Trying to mount root from ufs:/dev/ad4s1a ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present; to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf. ZFS WARNING: Recommended minimum kmem_size is 512MB; expect unstable behavior. Consider tuning vm.kmem_size and vm.kmem_size_max in /boot/loader.conf. ZFS filesystem version 5 ZFS storage pool version 28</nvidia></nvidia></nvidia></nvidia></st500dm002></cool`n'quiet></at></keyboard></generic></system></isa></high></parallel></printer></plip></parallel></parallel></at></power></vga-compatible></acpi></acpi></intel(r)></acpi></acpi></acpi></acpi></ata></ata></nvidia></ata></ata></nvidia></rtl8169s></mii></nvidia></ata></ata></nvidia></realtek></mii></realtek></realtek></mii></realtek></acpi></acpi></nvidia></nvidia></nvidia></nvidia></memory,></serial></isa></pci-isa></memory,></acpi></acpi></acpi></acpi></software></version></lahf,cmp,svm,extapic,cr8,abm,sse4a,mas,prefetch,osvw,ibs,skinit,wdt></syscall,nx,mmx+,ffxsr,page1gb,rdtscp,lm,3dnow!+,3dnow!></sse3,mon,cx16,popcnt></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,mmx,fxsr,sse,sse2,htt>recent dmesg output, repeated many times
ugen0.2: <unknown> at usbus0 (disconnected) uhub_reattach_port: could not allocate new device uhub_reattach_port: giving up port reset - device vanished usb_alloc_device: set address 2 failed (USB_ERR_IOERROR, ignored) usbd_setup_device_desc: getting device descriptor at addr 2 failed, USB_ERR_IOERROR usbd_req_re_enumerate: addr=2, set address failed! (USB_ERR_IOERROR, ignored) usbd_setup_device_desc: getting device descriptor at addr 2 failed, USB_ERR_IOERROR usbd_req_re_enumerate: addr=2, set address failed! (USB_ERR_IOERROR, ignored) usbd_setup_device_desc: getting device descriptor at addr 2 failed, USB_ERR_IOERROR</unknown>ps aux
USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAND root 11 196.7 0.0 0 32 ?? RL 15Oct14 25120:14.22 [idle] root 0 2.5 0.0 0 208 ?? DLs 15Oct14 115:45.87 [kernel] root 12 2.3 0.0 0 368 ?? WL 15Oct14 137:48.44 [intr] root 1 0.0 0.0 3200 588 ?? ILs 15Oct14 0:00.61 /sbin/init -- root 2 0.0 0.0 0 16 ?? DL 15Oct14 0:00.00 [g_event] root 3 0.0 0.0 0 16 ?? DL 15Oct14 1:43.97 [g_up] root 4 0.0 0.0 0 16 ?? DL 15Oct14 1:00.65 [g_down] root 5 0.0 0.0 0 16 ?? DL 15Oct14 0:00.00 [crypto] root 6 0.0 0.0 0 16 ?? DL 15Oct14 0:00.00 [crypto returns] root 7 0.0 0.0 0 16 ?? DL 15Oct14 0:00.01 [sctp_iterator] root 8 0.0 0.0 0 16 ?? DL 15Oct14 0:09.19 [pfpurge] root 9 0.0 0.0 0 16 ?? DL 15Oct14 0:00.00 [xpt_thrd] root 10 0.0 0.0 0 16 ?? DL 15Oct14 0:00.00 [audit] root 13 0.0 0.0 0 32 ?? DL 15Oct14 0:01.09 [ng_queue] root 14 0.0 0.0 0 16 ?? DL 15Oct14 3:49.03 [yarrow] root 15 0.0 0.0 0 128 ?? DL 15Oct14 20:35.75 [usb] root 16 0.0 0.0 0 16 ?? DL 15Oct14 0:00.47 [pagedaemon] root 17 0.0 0.0 0 16 ?? DL 15Oct14 0:00.00 [vmdaemon] root 18 0.0 0.0 0 16 ?? DL 15Oct14 0:00.01 [pagezero] root 19 0.0 0.0 0 16 ?? DL 15Oct14 0:00.68 [idlepoll] root 20 0.0 0.0 0 16 ?? DL 15Oct14 0:01.88 [bufdaemon] root 21 0.0 0.0 0 16 ?? DL 15Oct14 1:12.54 [syncer] root 22 0.0 0.0 0 16 ?? DL 15Oct14 0:02.07 [vnlru] root 23 0.0 0.0 0 16 ?? DL 15Oct14 0:02.22 [softdepflush] root 36 0.0 0.0 0 32 ?? DL 15Oct14 0:04.70 [zfskern] root 66 0.0 0.0 0 16 ?? DL 15Oct14 0:08.36 [md0] root 258 0.0 0.0 6908 1404 ?? INs 15Oct14 213:09.37 /usr/local/sbin/check_reload_status root 260 0.0 0.0 6908 1288 ?? IN 15Oct14 0:00.00 check_reload_status: Monitoring daemon of check_reload_status root 269 0.0 0.1 5248 3148 ?? Ss 15Oct14 1:25.81 /sbin/devd root 753 0.0 0.0 6960 1664 ?? Ss 15Oct14 0:09.00 /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -f /var/etc/syslog.conf root 1227 0.0 0.0 7928 1620 ?? Is 15Oct14 0:00.85 /usr/sbin/cron -s root 6912 0.0 0.1 25028 4300 ?? Ss 15Oct14 0:07.09 /usr/local/sbin/mpd5 -b -k -d /var/etc -f mpd_wan.conf -p /var/run/pppoe_wan.pid -s ppp pppoeclient root 10176 0.0 0.1 15264 3700 ?? INs 15Oct14 0:00.00 /usr/sbin/sshd root 11287 0.0 0.1 25028 4604 ?? Ss 15Oct14 0:04.18 /usr/local/sbin/mpd5 -b -k -d /var/etc -f mpd_opt1.conf -p /var/run/pppoe_opt1.pid -s ppp pppoeclient root 17344 0.0 2.8 442348 116352 ?? Ss 9:49AM 0:04.06 /usr/pbi/snort-amd64/bin/snort -R 8376 -D -q -l /var/log/snort/snort_pppoe08376 --pid-path /var/run --nolock-pidfile -G 8376 -c /usr/pbi/snort-amd64/etc/snort/snort_8376_ppp nobody 17656 0.0 0.0 6836 1228 ?? Is 10:36AM 0:00.00 nc -w 2000 1***.***.***.*** 143 root 26684 0.0 0.0 8984 1720 ?? Is 15Oct14 0:27.76 /usr/sbin/inetd -wW -R 0 -a 127.0.0.1 /var/etc/inetd.conf nobody 26761 0.0 0.0 6836 1232 ?? Is 8:27AM 0:00.01 nc -w 2000 ***.***.***.*** 143 nobody 27056 0.0 0.0 6836 1228 ?? Is 8:27AM 0:00.00 nc -w 2000 ***.***.***.*** 143 nobody 27093 0.0 0.0 6836 1228 ?? Is 8:27AM 0:00.01 nc -w 2000 ***.***.***.*** 143 root 38658 0.0 2.8 440300 114216 ?? Ss 9:49AM 0:01.58 /usr/pbi/snort-amd64/bin/snort -R 52612 -D -q -l /var/log/snort/snort_pppoe152612 --pid-path /var/run --nolock-pidfile -G 52612 -c /usr/pbi/snort-amd64/etc/snort/snort_52612 root 40746 0.0 0.6 147776 22920 ?? Is 15Oct14 0:00.07 /usr/local/bin/php root 41374 0.0 0.6 147776 22920 ?? Is 15Oct14 0:00.21 /usr/local/bin/php nobody 49725 0.0 0.0 6836 1232 ?? Is 8:33AM 0:00.02 nc -w 2000 ***.***.***.*** 143 nobody 54102 0.0 0.0 6836 1232 ?? Is 9:41AM 0:00.01 nc -w 2000 ***.***.***.*** 143 root 68045 0.0 0.0 5784 1500 ?? SNs 15Oct14 2:30.73 /usr/local/sbin/apinger -c /var/etc/apinger.conf root 68409 0.0 0.0 14392 2004 ?? IN 15Oct14 0:01.72 /usr/local/bin/rrdtool - nobody 71710 0.0 0.0 6836 1232 ?? Is 9:37AM 0:00.01 nc -w 2000 ***.***.***.*** 143 root 71728 0.0 1.0 151872 41152 ?? I 10:18AM 0:05.35 /usr/local/bin/php nobody 74208 0.0 0.0 6836 1228 ?? Is 10:06AM 0:00.00 nc -w 2000 ***.***.***.*** 143 root 85187 0.0 0.0 2716 964 ?? IN 10:49AM 0:00.00 sleep 60 root 85239 0.0 0.1 26164 4184 ?? SNs 10:49AM 0:00.02 sshd: admin@pts/0 (sshd) nobody 85396 0.0 0.1 16040 5328 ?? S 15Oct14 2:00.46 /usr/local/sbin/dnsmasq --all-servers --rebind-localhost-ok --stop-dns-rebind --listen-address=***.***.***.*** --dns-forward-max=5000 --cache-size=10000 --local-ttl=1 nobody 85704 0.0 0.0 6836 1228 ?? Is 10:39AM 0:00.00 nc -w 2000 ***.***.***.*** 143 nobody 85882 0.0 0.0 6836 1228 ?? Is 10:39AM 0:00.00 nc -w 2000 ***.***.***.*** 143 nobody 85917 0.0 0.0 6836 1228 ?? Is 10:39AM 0:00.00 nc -w 2000 ***.***.***.*** 143 root 86346 0.0 0.0 7036 1392 ?? Is 15Oct14 0:00.01 /usr/local/sbin/sshlockout_pf 15 nobody 89988 0.0 0.0 6836 1228 ?? Is 10:44AM 0:00.01 nc -w 2000 ***.***.***.*** 143 nobody 90335 0.0 0.0 6836 1228 ?? Is 10:44AM 0:00.00 nc -w 2000 ***.***.***.*** 143 root 90944 0.0 0.2 12032 6988 ?? SNs 10:52PM 0:01.18 /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid dhcpd 92639 0.0 0.3 19148 12952 ?? Ss 15Oct14 0:35.54 /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid nfe0 em0 root 95229 0.0 0.9 149824 35476 ?? I 10:24AM 0:00.41 /usr/local/bin/php root 96045 0.0 0.0 5784 1180 ?? Is 15Oct14 0:00.00 /usr/local/bin/minicron 240 /var/run/ping_hosts.pid /usr/local/bin/ping_hosts.sh root 96126 0.0 0.0 5784 1228 ?? I 15Oct14 0:00.27 minicron: helper /usr/local/bin/ping_hosts.sh (minicron) root 96414 0.0 0.0 5784 1180 ?? Is 15Oct14 0:00.00 /usr/local/bin/minicron 3600 /var/run/expire_accounts.pid /etc/rc.expireaccounts root 96869 0.0 0.0 5784 1228 ?? I 15Oct14 0:00.02 minicron: helper /etc/rc.expireaccounts (minicron) root 96873 0.0 0.0 5784 1180 ?? Is 15Oct14 0:00.00 /usr/local/bin/minicron 86400 /var/run/update_alias_url_data.pid /etc/rc.update_alias_url_data root 97116 0.0 0.0 5784 1228 ?? I 15Oct14 0:00.00 minicron: helper /etc/rc.update_alias_url_data (minicron) root 98012 0.0 0.2 26280 6548 ?? SN 15Oct14 1:01.23 /usr/local/sbin/lighttpd -f /var/etc/lighty-webConfigurator.conf root 98653 0.0 0.0 8296 1856 ?? IN 15Oct14 0:57.91 /bin/sh /var/db/rrd/updaterrd.sh root 300 0.0 0.0 19480 1844 u0 Is 15Oct14 0:00.01 login [pam] (login) root 344 0.0 0.0 8296 1796 u0 I 15Oct14 0:00.00 -sh (sh) root 2900 0.0 0.0 8296 1760 u0 I+ 15Oct14 0:00.00 /bin/sh /etc/rc.initial root 22505 0.0 0.1 11744 2548 u0- S 15Oct14 0:05.05 /usr/sbin/tcpdump -s 256 -v -S -l -n -e -ttt -i pflog0 root 22506 0.0 0.0 5780 1096 u0- I 15Oct14 0:01.40 logger -t pf -p local0.info root 214 0.0 0.0 19480 1844 v0 Is 15Oct14 0:00.01 login [pam] (login) root 540 0.0 0.0 8296 1796 v0 I 15Oct14 0:00.00 -sh (sh) root 3231 0.0 0.0 8296 1760 v0 I+ 15Oct14 0:00.00 /bin/sh /etc/rc.initial root 85777 0.0 0.0 8296 1808 0 Ss 10:49AM 0:00.00 /bin/sh /etc/rc.initial root 87784 0.0 0.1 8268 2784 0 S 10:49AM 0:00.01 /bin/tcsh root 88555 0.0 0.0 7992 1524 0 R+ 10:49AM 0:00.00 ps auxvmstat -mh
$ vmstat -mh Type InUse MemUse HighUse Requests Size(s) cdev 11 3K - 11 256 sigio 1 1K - 2 64 filedesc 113 73K - 5124517 16,512,1024,2048 kenv 78 11K - 88 16,32,64,128 kqueue 6 6K - 112753 256,512,2048 proc-args 53 5K - 2849369 16,32,64,128,256 hhook 2 1K - 2 128 CAM dev queue 1 1K - 1 128 ithread 100 16K - 100 32,128,256 ata_generic 1 1K - 226 16,512,1024 KTRACE 100 13K - 100 128 UART 3 2K - 3 16,512,1024 linker 233 856K - 210311 16,32,64,128,256,512,1024,2048,4096 lockf 20 3K - 59407951 64,128 kbdmux 6 10K - 6 16,512,1024,2048,4096 ip6ndp 20 2K - 766 64,128 ip6opt 13 4K - 17521 256 temp 75 691K - 14256975 16,32,64,128,256,512,1024,2048,4096 devbuf 621591 1263690K - 4974322 16,32,64,128,256,512,1024,2048,4096 LED 2 1K - 2 16,128 module 537 68K - 540 128 mtx_pool 2 16K - 2 osd 3 1K - 4 16,64 subproc 243 402K - 2583762 512,4096 proc 2 16K - 2 session 38 5K - 11586 128 pgrp 43 6K - 11599 128 cred 74 12K - 67439438 64,256 uidinfo 4 3K - 377 128,2048 plimit 22 6K - 61781 256 acpidev 72 5K - 72 64 md_sectors 6 24K - 6 4096 md_disk 7 3K - 7 32,2048 sysctltmp 0 0K - 1435113 16,32,64,256 sysctloid 4203 207K - 4300 16,32,64,128 sysctl 0 0K - 3625820 16,32,64 callout 1 512K - 1 umtx 336 42K - 336 128 p1003.1b 1 1K - 1 16 SWAP 2 1097K - 2 64 bus-sc 83 147K - 5207 16,32,64,128,256,512,1024,2048,4096 bus 1269 107K - 213329 16,32,64,128,256,512,1024 devstat 4 9K - 4 32,4096 eventhandler 97 8K - 97 64,128 kobj 326 1304K - 387 4096 Per-cpu 1 1K - 1 32 USBdev 96 47K - 35979 64,128,512,1024,2048 USB 14 5K - 121426 16,32,64,128,256,2048 rman 210 26K - 705 16,32,128 ad_driver 1 1K - 1 32 sbuf 0 0K - 143276 16,32,64,128,256 ar_driver 0 0K - 6 512,2048 taskqueue 23 2K - 23 16,32,128 Unitno 16 1K - 112314 32,64 iov 0 0K - 1044615 16,64,128,256 select 236 30K - 236 128 ioctlops 0 0K - 46990235 16,32,64,128,256,512,1024,2048,4096 msg 4 89K - 4 1024 sem 4 11K - 4 512,1024 shm 1 20K - 1 tty 21 21K - 22 1024 pts 1 1K - 2 256 mbuf_tag 0 0K - 1940714252 32,64 ksem 1 8K - 1 shmfd 1 8K - 1 pcb 83 160K - 369462 16,32,64,128,1024,2048,4096 soname 15 2K - 8479922 16,32,128 acl 0 0K - 36 4096 biobuf 209 418K - 32846 2048 vfscache 1 1024K - 1 cl_savebuf 0 0K - 40840 64,128 vfs_hash 1 512K - 1 vnodes 2 1K - 2 256 vnodemarker 0 0K - 93244 512 mount 42 2K - 69 16,32,64,128,256 fadvise 0 0K - 3765690 32 BPF 32 213K - 20800 16,128,512,4096 ether_multi 112 6K - 4718 16,32,64 ifaddr 146 36K - 1629 32,64,256,512,2048,4096 ifnet 13 25K - 13 128,2048 clone 15 60K - 15 4096 arpcom 4 1K - 4 16 lltable 840 216K - 3558 256,512 tun 1 1K - 1 256 DEVFS1 84 42K - 33798 512 DEVFS3 198 50K - 3254 256 DEVFS2 82 2K - 84 16 routetbl 112 23K - 43995401 32,64,128,256,512 DEVFS 21 1K - 1537 16,128 DEVFSP 5 1K - 21051 64 netgraph_msg 0 0K - 310528 64,128,256,512,1024,2048 netgraph_node 24 6K - 10147 256 netgraph_hook 24 3K - 36 128 netgraph 8 1K - 10 64,256 netgraph_iface 2 1K - 2 128 netgraph_mppc 0 0K - 2 1024 netgraph_ppp 2 24K - 2 netgraph_pppoe 4 25K - 6 64,512 netgraph_sock 26 4K - 20270 32,128 netgraph_path 0 0K - 155285 16,32 igmp 12 3K - 12 256 ipid 2 24K - 2 ip_moptions 8 2K - 7404 64,256 in_multi 5 2K - 745 256 in_mfilter 4 4K - 3702 1024 encap_export_host 4 4K - 4 1024 mroutetbl 1 1K - 1 16 sctp_iter 0 0K - 1487 256 sctp_ifn 5 1K - 745 128 sctp_ifa 9 2K - 1489 128 sctp_vrf 1 1K - 1 64 sctp_a_it 0 0K - 1487 16 hostcache 1 28K - 1 syncache 1 96K - 1 nullfs_hash 1 1K - 1 128 in6_multi 60 8K - 4574 32,256 pfs_nodes 21 6K - 21 256 mld 12 2K - 12 128 inpcbpolicy 94 3K - 11879699 32 ipsecpolicy 188 47K - 23924378 256 NFS FHA 1 2K - 1 2048 crypto 1 1K - 1 512 xform 0 0K - 239127 16,32 rpc 2 9K - 2 256 audit_evclass 172 6K - 211 32 newblk 1 1K - 1 512 inodedep 1 512K - 1 pagedep 1 128K - 1 ufs_dirhash 57 11K - 189 16,32,64,128,256,512 ufs_mount 6 65K - 6 512,2048,4096 UMAHash 2 1K - 2 512 vm_pgdata 2 129K - 2 128 ddb_capture 1 48K - 1 pci_link 36 3K - 36 16,128 io_apic 1 2K - 1 2048 memdesc 1 4K - 1 4096 acpi_perf 2 1K - 2 128 GEOM 48 16K - 357 16,32,64,128,256,512,1024 atkbddev 2 1K - 2 64 acpica 3866 396K - 89568 16,32,64,128,256,512,1024,2048 acpitask 1 2K - 1 2048 spkr 0 0K - 5 512 CAM queue 3 1K - 7 16 acpisem 19 3K - 19 128 ppbusdev 3 1K - 3 256 entropy 1024 64K - 1024 64 CAM SIM 1 1K - 1 256 CAM periph 2 1K - 12 16,32,64,128,256 isadev 6 1K - 6 128 CAM XPT 12 3K - 26 32,64,128,2048 msi 12 2K - 12 32,128 nexusdev 5 1K - 5 16 solaris 103 8711K - 105 16,64,128,1024 kstat_data 4 1K - 4 64 dummynet 6 2K - 6 256,512 dummynet 3 3K - 3 512,1024 ```  -
I've installed the Apcupsd package …
All was seemingly fine until a recently when after a power failure and subsequent reboot the wired memory usage grew and grew until swapping occurred ...
One odd thing I noticed was that dmesg shows me some sort of USB error ...
recent dmesg output, repeated many times:
ugen0.2: <unknown> at usbus0 (disconnected) uhub_reattach_port: could not allocate new device uhub_reattach_port: giving up port reset - device vanished usb_alloc_device: set address 2 failed (USB_ERR_IOERROR, ignored) usbd_setup_device_desc: getting device descriptor at addr 2 failed, USB_ERR_IOERROR usbd_req_re_enumerate: addr=2, set address failed! (USB_ERR_IOERROR, ignored) usbd_setup_device_desc: getting device descriptor at addr 2 failed, USB_ERR_IOERROR usbd_req_re_enumerate: addr=2, set address failed! (USB_ERR_IOERROR, ignored) usbd_setup_device_desc: getting device descriptor at addr 2 failed, USB_ERR_IOERROR</unknown>Did the USB errors started showing up after the power outage?
At first glance I would try disconnecting USB from your UPS, and perhaps temporarily remove the apcpusd package. Maybe the power outage damaged the USB port on the UPS (how ironic!). It's not clear to me how that could cause a memory leak, but a lot is possible with damaged hardware.
-
Thanks charliem! I'm starting to think that USB is the root of the issue.
I fired up the apcupsd status panel and got the "could not connect to service" error; according to the system logs apcupsd was not started as the UPS could not be found. I unplugged the UPS, plugged it in again, checked dmesg and got the following:
ugen0.2: <unknown>at usbus0 (disconnected) uhub_reattach_port: could not allocate new device ugen0.2: <american power="" conversion="">at usbus0</american></unknown>I went into the apcupsd panel and hit "Save" on the configuration page then clicked over to the "Status" tab and lo and behold, the UPS service had been started and the status was printed out. This was a couple of hours ago; I've just checked the RRD graphs / vmstat and the wired memory has stopped increasing and leveled off, and no funny USB related stuff in dmesg either.
I've checked the UPS' USB port on my linux dev machine and all seems just fine, my guess is that there is an issue with the USB driver and the way it is initialising the USB hub which is hung off the controller on the motherboard. I've detached the UPS' USB for the time being, will restart it and see what happens. Incidentally I plan to upgrade the hardware to a more modern Haswell CPU / board in order to save some power so I'll do the same set of experiments once that's up and running in a couple of weeks.
Many thanks for your help! If things start to get wierd again I'll append this thread with an update as it does seem there may be an underlying issue with the kernel and this USB configuration.