Wired memory usage slowy increasing



  • Hi!

    I've had a pfsense box running for a couple of months as our company's internet gateway and I absolutely love it! So powerful… anyway, the basics of the setup are as follows:

    AMD Phenom II X2 225 CPU w. 4GB DDR3 on an Asus M4N689T-M board, with an APC UPS plugged in via USB. There are four NICs:
    nfe0 which is our LAN on 192.168.0.0/16 (LAN1)
    A multi-wan setup on rl0 and rl1 involving PPPoE connections to our two ADSL modems
    em0 which is separate LAN on 10.0.0.0/24 (LAN2)

    I've installed the Apcupsd package, Snort and the OpenVPN export utility (though the VPN is currently disabled)

    We have the usual traffic between our LAN and the multi-wan load balancing gateway. The separate LAN is not physically secure; as such I've used the firewall/NAT to drop any inbound traffic to our LAN but allow connections from LAN1 to LAN2. LAN2 traffic is very high as it contains the (network block device, encrypted) offsite backup server and an IP camera monitoring the room that it's in.

    All was seemingly fine until a recently when after a power failure and subsequent reboot the wired memory usage grew and grew until swapping occurred; after having a tinker I restarted the box and it was fine for a week and now the same thing has happened again. It seems that devbuf's memory usage grows over a couple of days until we run out of memory - aargh!

    I'm not too familiar with BSD, my day job is Linux - but to me this looks like some sort of memory leak. I've not yet filed a bug as I don't think I've got enough useful information yet - I've included things I thought may be useful in the post, hopefully someone can point me where to dig next!

    One odd thing I noticed was that dmesg shows me some sort of USB error. I found an old post on tinterwebs showing a similar thing to do with a USB1 hob - could this be related?

    The system is currently still up and running and in this funny state - I reckon I've got about a day before it needs restarting, so if anything else useful can be gleaned let me know!

    dmesg.boot

    Copyright (c) 1992-2012 The FreeBSD Project.
    Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    	The Regents of the University of California. All rights reserved.
    FreeBSD is a registered trademark of The FreeBSD Foundation.
    FreeBSD 8.3-RELEASE-p16 #0: Mon Aug 25 08:27:11 EDT 2014
        root@pf2_1_1_amd64.pfsense.org:/usr/obj.amd64/usr/pfSensesrc/src/sys/pfSense_SMP.8 amd64
    Timecounter "i8254" frequency 1193182 Hz quality 0
    CPU: AMD Phenom(tm) II X2 555 Processor (3214.64-MHz K8-class CPU)
      Origin = "AuthenticAMD"  Id = 0x100f43  Family = 10  Model = 4  Stepping = 3
      Features=0x178bfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,mmx,fxsr,sse,sse2,htt>Features2=0x802009 <sse3,mon,cx16,popcnt>AMD Features=0xee500800 <syscall,nx,mmx+,ffxsr,page1gb,rdtscp,lm,3dnow!+,3dnow!>AMD Features2=0x37ff <lahf,cmp,svm,extapic,cr8,abm,sse4a,mas,prefetch,osvw,ibs,skinit,wdt>TSC: P-state invariant
    real memory  = 4294967296 (4096 MB)
    avail memory = 4058845184 (3870 MB)
    ACPI APIC Table: <061210 APIC1416>
    FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
    FreeBSD/SMP: 1 package(s) x 2 core(s)
     cpu0 (BSP): APIC ID:  0
     cpu1 (AP): APIC ID:  1
    ioapic0 <version 1.1=""> irqs 0-23 on motherboard
    wlan: mac acl policy registered
    ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
    ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff804abaf0, 0) error 1
    ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
    ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff804abb90, 0) error 1
    ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
    ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
    module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff804abc30, 0) error 1
    kbd1 at kbdmux0
    cryptosoft0: <software crypto=""> on motherboard
    padlock0: No ACE support.
    acpi0: <061210 XSDT1416> on motherboard
    acpi0: [ITHREAD]
    acpi0: Power Button (fixed)
    acpi0: reservation of fec00000, fed40000 (3) failed
    acpi0: reservation of 0, a0000 (3) failed
    acpi0: reservation of 100000, bdf00000 (3) failed
    Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
    acpi_timer0: <24-bit timer at 3.579545MHz> port 0x508-0x50b on acpi0
    cpu0: <acpi cpu=""> on acpi0
    cpu1: <acpi cpu=""> on acpi0
    pcib0: <acpi host-pci="" bridge=""> port 0xcf8-0xcff on acpi0
    pci0: <acpi pci="" bus=""> on pcib0
    pci0: <memory, ram=""> at device 0.0 (no driver attached)
    isab0: <pci-isa bridge=""> port 0x900-0x9ff at device 1.0 on pci0
    isa0: <isa bus=""> on isab0
    pci0: <serial bus,="" smbus=""> at device 1.1 (no driver attached)
    pci0: <memory, ram=""> at device 1.2 (no driver attached)
    ohci0: <nvidia nforce="" mcp61="" usb="" controller=""> mem 0xdfdfb000-0xdfdfbfff irq 21 at device 2.0 on pci0
    ohci0: [ITHREAD]
    usbus0: <nvidia nforce="" mcp61="" usb="" controller=""> on ohci0
    ehci0: <nvidia nforce="" mcp61="" usb="" 2.0="" controller=""> mem 0xdfdfac00-0xdfdfacff irq 22 at device 2.1 on pci0
    ehci0: [ITHREAD]
    usbus1: EHCI version 1.0
    usbus1: <nvidia nforce="" mcp61="" usb="" 2.0="" controller=""> on ehci0
    pcib1: <acpi pci-pci="" bridge=""> at device 4.0 on pci0
    pci1: <acpi pci="" bus=""> on pcib1
    rl0: <realtek 10="" 8139="" 100basetx=""> port 0xd800-0xd8ff mem 0xdfeffc00-0xdfeffcff irq 16 at device 6.0 on pci1
    miibus0: <mii bus=""> on rl0
    rlphy0: <realtek internal="" media="" interface=""> PHY 0 on miibus0
    rlphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
    rl0: [ITHREAD]
    rl1: <realtek 10="" 8139="" 100basetx=""> port 0xd400-0xd4ff mem 0xdfeff800-0xdfeff8ff irq 17 at device 7.0 on pci1
    miibus1: <mii bus=""> on rl1
    rlphy1: <realtek internal="" media="" interface=""> PHY 0 on miibus1
    rlphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
    rl1: [ITHREAD]
    atapci0: <nvidia nforce="" mcp61="" udma133="" controller=""> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 6.0 on pci0
    ata0: <ata channel=""> at channel 0 on atapci0
    ata0: [ITHREAD]
    ata1: <ata channel=""> at channel 1 on atapci0
    ata1: [ITHREAD]
    nfe0: <nvidia nforce="" mcp61="" networking="" adapter=""> port 0xc480-0xc487 mem 0xdfdf9000-0xdfdf9fff irq 23 at device 7.0 on pci0
    miibus2: <mii bus=""> on nfe0
    rgephy0: <rtl8169s 8110s="" 8211b="" media="" interface=""> PHY 1 on miibus2
    rgephy0:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
    nfe0: [FILTER]
    nfe0: [FILTER]
    nfe0: [FILTER]
    nfe0: [FILTER]
    nfe0: [FILTER]
    nfe0: [FILTER]
    nfe0: [FILTER]
    nfe0: [FILTER]
    atapci1: <nvidia nforce="" mcp61="" sata300="" controller=""> port 0xc400-0xc407,0xc080-0xc083,0xc000-0xc007,0xbc00-0xbc03,0xb880-0xb88f mem 0xdfdf8000-0xdfdf8fff irq 20 at device 8.0 on pci0
    atapci1: [ITHREAD]
    ata2: <ata channel=""> at channel 0 on atapci1
    ata2: [ITHREAD]
    ata3: <ata channel=""> at channel 1 on atapci1
    ata3: [ITHREAD]
    atapci2: <nvidia nforce="" mcp61="" sata300="" controller=""> port 0xb800-0xb807,0xb480-0xb483,0xb400-0xb407,0xb080-0xb083,0xb000-0xb00f mem 0xdfdf3000-0xdfdf3fff irq 21 at device 8.1 on pci0
    atapci2: [ITHREAD]
    ata4: <ata channel=""> at channel 0 on atapci2
    ata4: [ITHREAD]
    ata5: <ata channel=""> at channel 1 on atapci2
    ata5: [ITHREAD]
    pcib2: <acpi pci-pci="" bridge=""> at device 9.0 on pci0
    pci2: <acpi pci="" bus=""> on pcib2
    pcib3: <acpi pci-pci="" bridge=""> at device 11.0 on pci0
    pci3: <acpi pci="" bus=""> on pcib3
    em0: <intel(r) 1000="" pro="" network="" connection="" 7.3.8=""> port 0xec00-0xec1f mem 0xdffe0000-0xdfffffff,0xdff00000-0xdff7ffff,0xdffdc000-0xdffdffff irq 18 at device 0.0 on pci3
    em0: Using MSIX interrupts with 3 vectors
    em0: [ITHREAD]
    em0: [ITHREAD]
    em0: [ITHREAD]
    pcib4: <acpi pci-pci="" bridge=""> at device 12.0 on pci0
    pci4: <acpi pci="" bus=""> on pcib4
    vgapci0: <vga-compatible display=""> mem 0xde000000-0xdeffffff,0xc0000000-0xcfffffff,0xdd000000-0xddffffff irq 22 at device 13.0 on pci0
    acpi_button0: <power button=""> on acpi0
    atrtc0: <at realtime="" clock=""> port 0x70-0x71 irq 8 on acpi0
    ppc0: <parallel port=""> port 0x378-0x37f irq 7 on acpi0
    ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
    ppc0: [ITHREAD]
    ppbus0: <parallel port="" bus=""> on ppc0
    plip0: <plip network="" interface=""> on ppbus0
    plip0: [ITHREAD]
    lpt0: <printer> on ppbus0
    lpt0: [ITHREAD]
    lpt0: Interrupt-driven port
    ppi0: <parallel i="" o=""> on ppbus0
    acpi_hpet0: <high precision="" event="" timer=""> iomem 0xfed00000-0xfed00fff on acpi0
    Timecounter "HPET" frequency 25000000 Hz quality 900
    uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
    uart0: [FILTER]
    uart0: console (115200,n,8,1)
    orm0: <isa option="" rom=""> at iomem 0xcf000-0xcffff on isa0
    sc0: <system console=""> at flags 0x100 on isa0
    sc0: VGA <16 virtual consoles, flags=0x300>
    vga0: <generic isa="" vga=""> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
    atkbdc0: <keyboard controller="" (i8042)=""> at port 0x60,0x64 on isa0
    atkbd0: <at keyboard=""> irq 1 on atkbdc0
    kbd0 at atkbd0
    atkbd0: [GIANT-LOCKED]
    atkbd0: [ITHREAD]
    hwpstate0: <cool`n'quiet 2.0=""> on cpu0
    Timecounters tick every 1.000 msec
    IPsec: Initialized Security Association Processing.
    usbus0: 12Mbps Full Speed USB v1.0
    usbus1: 480Mbps High Speed USB v2.0
    ad4: 476940MB <st500dm002 1bd142="" kc45=""> at ata2-master UDMA100 SATA 1.5Gb/s
    ugen0.1: <nvidia> at usbus0
    uhub0: <nvidia 1="" 9="" ohci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr=""> on usbus0
    ugen1.1: <nvidia> at usbus1
    uhub1: <nvidia 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr=""> on usbus1
    SMP: AP CPU #1 Launched!
    Root mount waiting for: usbus1 usbus0
    uhub0: 10 ports with 10 removable, self powered
    Root mount waiting for: usbus1
    Root mount waiting for: usbus1
    Root mount waiting for: usbus1
    uhub1: 10 ports with 10 removable, self powered
    Trying to mount root from ufs:/dev/ad4s1a
    ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
                to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
    ZFS WARNING: Recommended minimum kmem_size is 512MB; expect unstable behavior.
                 Consider tuning vm.kmem_size and vm.kmem_size_max
                 in /boot/loader.conf.
    ZFS filesystem version 5
    ZFS storage pool version 28</nvidia></nvidia></nvidia></nvidia></st500dm002></cool`n'quiet></at></keyboard></generic></system></isa></high></parallel></printer></plip></parallel></parallel></at></power></vga-compatible></acpi></acpi></intel(r)></acpi></acpi></acpi></acpi></ata></ata></nvidia></ata></ata></nvidia></rtl8169s></mii></nvidia></ata></ata></nvidia></realtek></mii></realtek></realtek></mii></realtek></acpi></acpi></nvidia></nvidia></nvidia></nvidia></memory,></serial></isa></pci-isa></memory,></acpi></acpi></acpi></acpi></software></version></lahf,cmp,svm,extapic,cr8,abm,sse4a,mas,prefetch,osvw,ibs,skinit,wdt></syscall,nx,mmx+,ffxsr,page1gb,rdtscp,lm,3dnow!+,3dnow!></sse3,mon,cx16,popcnt></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,mmx,fxsr,sse,sse2,htt>
    

    recent dmesg output, repeated many times

    ugen0.2: <unknown> at usbus0 (disconnected)
    uhub_reattach_port: could not allocate new device
    uhub_reattach_port: giving up port reset - device vanished
    usb_alloc_device: set address 2 failed (USB_ERR_IOERROR, ignored)
    usbd_setup_device_desc: getting device descriptor at addr 2 failed, USB_ERR_IOERROR
    usbd_req_re_enumerate: addr=2, set address failed! (USB_ERR_IOERROR, ignored)
    usbd_setup_device_desc: getting device descriptor at addr 2 failed, USB_ERR_IOERROR
    usbd_req_re_enumerate: addr=2, set address failed! (USB_ERR_IOERROR, ignored)
    usbd_setup_device_desc: getting device descriptor at addr 2 failed, USB_ERR_IOERROR</unknown>
    

    ps aux

    USER     PID %CPU %MEM   VSZ   RSS  TT  STAT STARTED      TIME COMMAND
    root      11 196.7  0.0     0    32  ??  RL   15Oct14 25120:14.22 [idle]
    root       0  2.5  0.0     0   208  ??  DLs  15Oct14 115:45.87 [kernel]
    root      12  2.3  0.0     0   368  ??  WL   15Oct14 137:48.44 [intr]
    root       1  0.0  0.0  3200   588  ??  ILs  15Oct14   0:00.61 /sbin/init --
    root       2  0.0  0.0     0    16  ??  DL   15Oct14   0:00.00 [g_event]
    root       3  0.0  0.0     0    16  ??  DL   15Oct14   1:43.97 [g_up]
    root       4  0.0  0.0     0    16  ??  DL   15Oct14   1:00.65 [g_down]
    root       5  0.0  0.0     0    16  ??  DL   15Oct14   0:00.00 [crypto]
    root       6  0.0  0.0     0    16  ??  DL   15Oct14   0:00.00 [crypto returns]
    root       7  0.0  0.0     0    16  ??  DL   15Oct14   0:00.01 [sctp_iterator]
    root       8  0.0  0.0     0    16  ??  DL   15Oct14   0:09.19 [pfpurge]
    root       9  0.0  0.0     0    16  ??  DL   15Oct14   0:00.00 [xpt_thrd]
    root      10  0.0  0.0     0    16  ??  DL   15Oct14   0:00.00 [audit]
    root      13  0.0  0.0     0    32  ??  DL   15Oct14   0:01.09 [ng_queue]
    root      14  0.0  0.0     0    16  ??  DL   15Oct14   3:49.03 [yarrow]
    root      15  0.0  0.0     0   128  ??  DL   15Oct14  20:35.75 [usb]
    root      16  0.0  0.0     0    16  ??  DL   15Oct14   0:00.47 [pagedaemon]
    root      17  0.0  0.0     0    16  ??  DL   15Oct14   0:00.00 [vmdaemon]
    root      18  0.0  0.0     0    16  ??  DL   15Oct14   0:00.01 [pagezero]
    root      19  0.0  0.0     0    16  ??  DL   15Oct14   0:00.68 [idlepoll]
    root      20  0.0  0.0     0    16  ??  DL   15Oct14   0:01.88 [bufdaemon]
    root      21  0.0  0.0     0    16  ??  DL   15Oct14   1:12.54 [syncer]
    root      22  0.0  0.0     0    16  ??  DL   15Oct14   0:02.07 [vnlru]
    root      23  0.0  0.0     0    16  ??  DL   15Oct14   0:02.22 [softdepflush]
    root      36  0.0  0.0     0    32  ??  DL   15Oct14   0:04.70 [zfskern]
    root      66  0.0  0.0     0    16  ??  DL   15Oct14   0:08.36 [md0]
    root     258  0.0  0.0  6908  1404  ??  INs  15Oct14 213:09.37 /usr/local/sbin/check_reload_status
    root     260  0.0  0.0  6908  1288  ??  IN   15Oct14   0:00.00 check_reload_status: Monitoring daemon of check_reload_status
    root     269  0.0  0.1  5248  3148  ??  Ss   15Oct14   1:25.81 /sbin/devd
    root     753  0.0  0.0  6960  1664  ??  Ss   15Oct14   0:09.00 /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -f /var/etc/syslog.conf
    root    1227  0.0  0.0  7928  1620  ??  Is   15Oct14   0:00.85 /usr/sbin/cron -s
    root    6912  0.0  0.1 25028  4300  ??  Ss   15Oct14   0:07.09 /usr/local/sbin/mpd5 -b -k -d /var/etc -f mpd_wan.conf -p /var/run/pppoe_wan.pid -s ppp pppoeclient
    root   10176  0.0  0.1 15264  3700  ??  INs  15Oct14   0:00.00 /usr/sbin/sshd
    root   11287  0.0  0.1 25028  4604  ??  Ss   15Oct14   0:04.18 /usr/local/sbin/mpd5 -b -k -d /var/etc -f mpd_opt1.conf -p /var/run/pppoe_opt1.pid -s ppp pppoeclient
    root   17344  0.0  2.8 442348 116352  ??  Ss    9:49AM   0:04.06 /usr/pbi/snort-amd64/bin/snort -R 8376 -D -q -l /var/log/snort/snort_pppoe08376 --pid-path /var/run --nolock-pidfile -G 8376 -c /usr/pbi/snort-amd64/etc/snort/snort_8376_ppp
    nobody 17656  0.0  0.0  6836  1228  ??  Is   10:36AM   0:00.00 nc -w 2000 1***.***.***.*** 143
    root   26684  0.0  0.0  8984  1720  ??  Is   15Oct14   0:27.76 /usr/sbin/inetd -wW -R 0 -a 127.0.0.1 /var/etc/inetd.conf
    nobody 26761  0.0  0.0  6836  1232  ??  Is    8:27AM   0:00.01 nc -w 2000 ***.***.***.*** 143
    nobody 27056  0.0  0.0  6836  1228  ??  Is    8:27AM   0:00.00 nc -w 2000 ***.***.***.*** 143
    nobody 27093  0.0  0.0  6836  1228  ??  Is    8:27AM   0:00.01 nc -w 2000 ***.***.***.*** 143
    root   38658  0.0  2.8 440300 114216  ??  Ss    9:49AM   0:01.58 /usr/pbi/snort-amd64/bin/snort -R 52612 -D -q -l /var/log/snort/snort_pppoe152612 --pid-path /var/run --nolock-pidfile -G 52612 -c /usr/pbi/snort-amd64/etc/snort/snort_52612
    root   40746  0.0  0.6 147776 22920  ??  Is   15Oct14   0:00.07 /usr/local/bin/php
    root   41374  0.0  0.6 147776 22920  ??  Is   15Oct14   0:00.21 /usr/local/bin/php
    nobody 49725  0.0  0.0  6836  1232  ??  Is    8:33AM   0:00.02 nc -w 2000 ***.***.***.*** 143
    nobody 54102  0.0  0.0  6836  1232  ??  Is    9:41AM   0:00.01 nc -w 2000 ***.***.***.*** 143
    root   68045  0.0  0.0  5784  1500  ??  SNs  15Oct14   2:30.73 /usr/local/sbin/apinger -c /var/etc/apinger.conf
    root   68409  0.0  0.0 14392  2004  ??  IN   15Oct14   0:01.72 /usr/local/bin/rrdtool -
    nobody 71710  0.0  0.0  6836  1232  ??  Is    9:37AM   0:00.01 nc -w 2000 ***.***.***.*** 143
    root   71728  0.0  1.0 151872 41152  ??  I    10:18AM   0:05.35 /usr/local/bin/php
    nobody 74208  0.0  0.0  6836  1228  ??  Is   10:06AM   0:00.00 nc -w 2000 ***.***.***.*** 143
    root   85187  0.0  0.0  2716   964  ??  IN   10:49AM   0:00.00 sleep 60
    root   85239  0.0  0.1 26164  4184  ??  SNs  10:49AM   0:00.02 sshd: admin@pts/0 (sshd)
    nobody 85396  0.0  0.1 16040  5328  ??  S    15Oct14   2:00.46 /usr/local/sbin/dnsmasq --all-servers --rebind-localhost-ok --stop-dns-rebind --listen-address=***.***.***.*** --dns-forward-max=5000 --cache-size=10000 --local-ttl=1
    nobody 85704  0.0  0.0  6836  1228  ??  Is   10:39AM   0:00.00 nc -w 2000 ***.***.***.*** 143
    nobody 85882  0.0  0.0  6836  1228  ??  Is   10:39AM   0:00.00 nc -w 2000 ***.***.***.*** 143
    nobody 85917  0.0  0.0  6836  1228  ??  Is   10:39AM   0:00.00 nc -w 2000 ***.***.***.*** 143
    root   86346  0.0  0.0  7036  1392  ??  Is   15Oct14   0:00.01 /usr/local/sbin/sshlockout_pf 15
    nobody 89988  0.0  0.0  6836  1228  ??  Is   10:44AM   0:00.01 nc -w 2000 ***.***.***.*** 143
    nobody 90335  0.0  0.0  6836  1228  ??  Is   10:44AM   0:00.00 nc -w 2000 ***.***.***.*** 143
    root   90944  0.0  0.2 12032  6988  ??  SNs  10:52PM   0:01.18 /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid
    dhcpd  92639  0.0  0.3 19148 12952  ??  Ss   15Oct14   0:35.54 /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid nfe0 em0
    root   95229  0.0  0.9 149824 35476  ??  I    10:24AM   0:00.41 /usr/local/bin/php
    root   96045  0.0  0.0  5784  1180  ??  Is   15Oct14   0:00.00 /usr/local/bin/minicron 240 /var/run/ping_hosts.pid /usr/local/bin/ping_hosts.sh
    root   96126  0.0  0.0  5784  1228  ??  I    15Oct14   0:00.27 minicron: helper /usr/local/bin/ping_hosts.sh  (minicron)
    root   96414  0.0  0.0  5784  1180  ??  Is   15Oct14   0:00.00 /usr/local/bin/minicron 3600 /var/run/expire_accounts.pid /etc/rc.expireaccounts
    root   96869  0.0  0.0  5784  1228  ??  I    15Oct14   0:00.02 minicron: helper /etc/rc.expireaccounts  (minicron)
    root   96873  0.0  0.0  5784  1180  ??  Is   15Oct14   0:00.00 /usr/local/bin/minicron 86400 /var/run/update_alias_url_data.pid /etc/rc.update_alias_url_data
    root   97116  0.0  0.0  5784  1228  ??  I    15Oct14   0:00.00 minicron: helper /etc/rc.update_alias_url_data  (minicron)
    root   98012  0.0  0.2 26280  6548  ??  SN   15Oct14   1:01.23 /usr/local/sbin/lighttpd -f /var/etc/lighty-webConfigurator.conf
    root   98653  0.0  0.0  8296  1856  ??  IN   15Oct14   0:57.91 /bin/sh /var/db/rrd/updaterrd.sh
    root     300  0.0  0.0 19480  1844  u0  Is   15Oct14   0:00.01 login [pam] (login)
    root     344  0.0  0.0  8296  1796  u0  I    15Oct14   0:00.00 -sh (sh)
    root    2900  0.0  0.0  8296  1760  u0  I+   15Oct14   0:00.00 /bin/sh /etc/rc.initial
    root   22505  0.0  0.1 11744  2548  u0- S    15Oct14   0:05.05 /usr/sbin/tcpdump -s 256 -v -S -l -n -e -ttt -i pflog0
    root   22506  0.0  0.0  5780  1096  u0- I    15Oct14   0:01.40 logger -t pf -p local0.info
    root     214  0.0  0.0 19480  1844  v0  Is   15Oct14   0:00.01 login [pam] (login)
    root     540  0.0  0.0  8296  1796  v0  I    15Oct14   0:00.00 -sh (sh)
    root    3231  0.0  0.0  8296  1760  v0  I+   15Oct14   0:00.00 /bin/sh /etc/rc.initial
    root   85777  0.0  0.0  8296  1808   0  Ss   10:49AM   0:00.00 /bin/sh /etc/rc.initial
    root   87784  0.0  0.1  8268  2784   0  S    10:49AM   0:00.01 /bin/tcsh
    root   88555  0.0  0.0  7992  1524   0  R+   10:49AM   0:00.00 ps aux
    
    

    vmstat -mh

    $ vmstat -mh
             Type InUse MemUse HighUse Requests  Size(s)
             cdev    11     3K       -       11  256
            sigio     1     1K       -        2  64
         filedesc   113    73K       -  5124517  16,512,1024,2048
             kenv    78    11K       -       88  16,32,64,128
           kqueue     6     6K       -   112753  256,512,2048
        proc-args    53     5K       -  2849369  16,32,64,128,256
            hhook     2     1K       -        2  128
    CAM dev queue     1     1K       -        1  128
          ithread   100    16K       -      100  32,128,256
      ata_generic     1     1K       -      226  16,512,1024
           KTRACE   100    13K       -      100  128
             UART     3     2K       -        3  16,512,1024
           linker   233   856K       -   210311  16,32,64,128,256,512,1024,2048,4096
            lockf    20     3K       - 59407951  64,128
           kbdmux     6    10K       -        6  16,512,1024,2048,4096
           ip6ndp    20     2K       -      766  64,128
           ip6opt    13     4K       -    17521  256
             temp    75   691K       - 14256975  16,32,64,128,256,512,1024,2048,4096
           devbuf 621591 1263690K       -  4974322  16,32,64,128,256,512,1024,2048,4096
              LED     2     1K       -        2  16,128
           module   537    68K       -      540  128
         mtx_pool     2    16K       -        2  
              osd     3     1K       -        4  16,64
          subproc   243   402K       -  2583762  512,4096
             proc     2    16K       -        2  
          session    38     5K       -    11586  128
             pgrp    43     6K       -    11599  128
             cred    74    12K       - 67439438  64,256
          uidinfo     4     3K       -      377  128,2048
           plimit    22     6K       -    61781  256
          acpidev    72     5K       -       72  64
       md_sectors     6    24K       -        6  4096
          md_disk     7     3K       -        7  32,2048
        sysctltmp     0     0K       -  1435113  16,32,64,256
        sysctloid  4203   207K       -     4300  16,32,64,128
           sysctl     0     0K       -  3625820  16,32,64
          callout     1   512K       -        1  
             umtx   336    42K       -      336  128
         p1003.1b     1     1K       -        1  16
             SWAP     2  1097K       -        2  64
           bus-sc    83   147K       -     5207  16,32,64,128,256,512,1024,2048,4096
              bus  1269   107K       -   213329  16,32,64,128,256,512,1024
          devstat     4     9K       -        4  32,4096
     eventhandler    97     8K       -       97  64,128
             kobj   326  1304K       -      387  4096
          Per-cpu     1     1K       -        1  32
           USBdev    96    47K       -    35979  64,128,512,1024,2048
              USB    14     5K       -   121426  16,32,64,128,256,2048
             rman   210    26K       -      705  16,32,128
        ad_driver     1     1K       -        1  32
             sbuf     0     0K       -   143276  16,32,64,128,256
        ar_driver     0     0K       -        6  512,2048
        taskqueue    23     2K       -       23  16,32,128
           Unitno    16     1K       -   112314  32,64
              iov     0     0K       -  1044615  16,64,128,256
           select   236    30K       -      236  128
         ioctlops     0     0K       - 46990235  16,32,64,128,256,512,1024,2048,4096
              msg     4    89K       -        4  1024
              sem     4    11K       -        4  512,1024
              shm     1    20K       -        1  
              tty    21    21K       -       22  1024
              pts     1     1K       -        2  256
         mbuf_tag     0     0K       - 1940714252  32,64
             ksem     1     8K       -        1  
            shmfd     1     8K       -        1  
              pcb    83   160K       -   369462  16,32,64,128,1024,2048,4096
           soname    15     2K       -  8479922  16,32,128
              acl     0     0K       -       36  4096
           biobuf   209   418K       -    32846  2048
         vfscache     1  1024K       -        1  
       cl_savebuf     0     0K       -    40840  64,128
         vfs_hash     1   512K       -        1  
           vnodes     2     1K       -        2  256
      vnodemarker     0     0K       -    93244  512
            mount    42     2K       -       69  16,32,64,128,256
          fadvise     0     0K       -  3765690  32
              BPF    32   213K       -    20800  16,128,512,4096
      ether_multi   112     6K       -     4718  16,32,64
           ifaddr   146    36K       -     1629  32,64,256,512,2048,4096
            ifnet    13    25K       -       13  128,2048
            clone    15    60K       -       15  4096
           arpcom     4     1K       -        4  16
          lltable   840   216K       -     3558  256,512
              tun     1     1K       -        1  256
           DEVFS1    84    42K       -    33798  512
           DEVFS3   198    50K       -     3254  256
           DEVFS2    82     2K       -       84  16
         routetbl   112    23K       - 43995401  32,64,128,256,512
            DEVFS    21     1K       -     1537  16,128
           DEVFSP     5     1K       -    21051  64
     netgraph_msg     0     0K       -   310528  64,128,256,512,1024,2048
    netgraph_node    24     6K       -    10147  256
    netgraph_hook    24     3K       -       36  128
         netgraph     8     1K       -       10  64,256
    netgraph_iface     2     1K       -        2  128
    netgraph_mppc     0     0K       -        2  1024
     netgraph_ppp     2    24K       -        2  
    netgraph_pppoe     4    25K       -        6  64,512
    netgraph_sock    26     4K       -    20270  32,128
    netgraph_path     0     0K       -   155285  16,32
             igmp    12     3K       -       12  256
             ipid     2    24K       -        2  
      ip_moptions     8     2K       -     7404  64,256
         in_multi     5     2K       -      745  256
       in_mfilter     4     4K       -     3702  1024
    encap_export_host     4     4K       -        4  1024
        mroutetbl     1     1K       -        1  16
        sctp_iter     0     0K       -     1487  256
         sctp_ifn     5     1K       -      745  128
         sctp_ifa     9     2K       -     1489  128
         sctp_vrf     1     1K       -        1  64
        sctp_a_it     0     0K       -     1487  16
        hostcache     1    28K       -        1  
         syncache     1    96K       -        1  
      nullfs_hash     1     1K       -        1  128
        in6_multi    60     8K       -     4574  32,256
        pfs_nodes    21     6K       -       21  256
              mld    12     2K       -       12  128
      inpcbpolicy    94     3K       - 11879699  32
      ipsecpolicy   188    47K       - 23924378  256
          NFS FHA     1     2K       -        1  2048
           crypto     1     1K       -        1  512
            xform     0     0K       -   239127  16,32
              rpc     2     9K       -        2  256
    audit_evclass   172     6K       -      211  32
           newblk     1     1K       -        1  512
         inodedep     1   512K       -        1  
          pagedep     1   128K       -        1  
      ufs_dirhash    57    11K       -      189  16,32,64,128,256,512
        ufs_mount     6    65K       -        6  512,2048,4096
          UMAHash     2     1K       -        2  512
        vm_pgdata     2   129K       -        2  128
      ddb_capture     1    48K       -        1  
         pci_link    36     3K       -       36  16,128
          io_apic     1     2K       -        1  2048
          memdesc     1     4K       -        1  4096
        acpi_perf     2     1K       -        2  128
             GEOM    48    16K       -      357  16,32,64,128,256,512,1024
         atkbddev     2     1K       -        2  64
           acpica  3866   396K       -    89568  16,32,64,128,256,512,1024,2048
         acpitask     1     2K       -        1  2048
             spkr     0     0K       -        5  512
        CAM queue     3     1K       -        7  16
          acpisem    19     3K       -       19  128
         ppbusdev     3     1K       -        3  256
          entropy  1024    64K       -     1024  64
          CAM SIM     1     1K       -        1  256
       CAM periph     2     1K       -       12  16,32,64,128,256
           isadev     6     1K       -        6  128
          CAM XPT    12     3K       -       26  32,64,128,2048
              msi    12     2K       -       12  32,128
         nexusdev     5     1K       -        5  16
          solaris   103  8711K       -      105  16,64,128,1024
       kstat_data     4     1K       -        4  64
         dummynet     6     2K       -        6  256,512
         dummynet     3     3K       -        3  512,1024
    
    ```![status_rrd_graph_img.php.png](/public/_imported_attachments_/1/status_rrd_graph_img.php.png)
    ![status_rrd_graph_img.php.png_thumb](/public/_imported_attachments_/1/status_rrd_graph_img.php.png_thumb)


  • @jpetpet:

    I've installed the Apcupsd package …

    All was seemingly fine until a recently when after a power failure and subsequent reboot the wired memory usage grew and grew until swapping occurred ...

    One odd thing I noticed was that dmesg shows me some sort of USB error ...

    recent dmesg output, repeated many times:

    ugen0.2: <unknown> at usbus0 (disconnected)
    uhub_reattach_port: could not allocate new device
    uhub_reattach_port: giving up port reset - device vanished
    usb_alloc_device: set address 2 failed (USB_ERR_IOERROR, ignored)
    usbd_setup_device_desc: getting device descriptor at addr 2 failed, USB_ERR_IOERROR
    usbd_req_re_enumerate: addr=2, set address failed! (USB_ERR_IOERROR, ignored)
    usbd_setup_device_desc: getting device descriptor at addr 2 failed, USB_ERR_IOERROR
    usbd_req_re_enumerate: addr=2, set address failed! (USB_ERR_IOERROR, ignored)
    usbd_setup_device_desc: getting device descriptor at addr 2 failed, USB_ERR_IOERROR</unknown>
    

    Did the USB errors started showing up after the power outage?

    At first glance I would try disconnecting USB from your UPS, and perhaps temporarily remove the apcpusd package.  Maybe the power outage damaged the USB port on the UPS (how ironic!).  It's not clear to me how that could cause a memory leak, but a lot is possible with damaged hardware.



  • Thanks charliem! I'm starting to think that USB is the root of the issue.

    I fired up the apcupsd status panel and got the "could not connect to service" error; according to the system logs apcupsd was not started as the UPS could not be found. I unplugged the UPS, plugged it in again, checked dmesg and got the following:

    ugen0.2: <unknown>at usbus0 (disconnected)
    uhub_reattach_port: could not allocate new device
    ugen0.2: <american power="" conversion="">at usbus0</american></unknown> 
    

    I went into the apcupsd panel and hit "Save" on the configuration page then clicked over to the "Status" tab and lo and behold, the UPS service had been started and the status was printed out. This was a couple of hours ago; I've just checked the RRD graphs / vmstat and the wired memory has stopped increasing and leveled off, and no funny USB related stuff in dmesg either.

    I've checked the UPS' USB port on my linux dev machine and all seems just fine, my guess is that there is an issue with the USB driver and the way it is initialising the USB hub which is hung off the controller on the motherboard. I've detached the UPS' USB for the time being, will restart it and see what happens. Incidentally I plan to upgrade the hardware to a more modern Haswell CPU / board in order to save some power so I'll do the same set of experiments once that's up and running in a couple of weeks.

    Many thanks for your help! If things start to get wierd again I'll append this thread with an update as it does seem there may be an underlying issue with the kernel and this USB configuration.