Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Wired memory usage slowy increasing

    Scheduled Pinned Locked Moved General pfSense Questions
    3 Posts 2 Posters 1.6k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • J
      jpetpet
      last edited by

      Hi!

      I've had a pfsense box running for a couple of months as our company's internet gateway and I absolutely love it! So powerful… anyway, the basics of the setup are as follows:

      AMD Phenom II X2 225 CPU w. 4GB DDR3 on an Asus M4N689T-M board, with an APC UPS plugged in via USB. There are four NICs:
      nfe0 which is our LAN on 192.168.0.0/16 (LAN1)
      A multi-wan setup on rl0 and rl1 involving PPPoE connections to our two ADSL modems
      em0 which is separate LAN on 10.0.0.0/24 (LAN2)

      I've installed the Apcupsd package, Snort and the OpenVPN export utility (though the VPN is currently disabled)

      We have the usual traffic between our LAN and the multi-wan load balancing gateway. The separate LAN is not physically secure; as such I've used the firewall/NAT to drop any inbound traffic to our LAN but allow connections from LAN1 to LAN2. LAN2 traffic is very high as it contains the (network block device, encrypted) offsite backup server and an IP camera monitoring the room that it's in.

      All was seemingly fine until a recently when after a power failure and subsequent reboot the wired memory usage grew and grew until swapping occurred; after having a tinker I restarted the box and it was fine for a week and now the same thing has happened again. It seems that devbuf's memory usage grows over a couple of days until we run out of memory - aargh!

      I'm not too familiar with BSD, my day job is Linux - but to me this looks like some sort of memory leak. I've not yet filed a bug as I don't think I've got enough useful information yet - I've included things I thought may be useful in the post, hopefully someone can point me where to dig next!

      One odd thing I noticed was that dmesg shows me some sort of USB error. I found an old post on tinterwebs showing a similar thing to do with a USB1 hob - could this be related?

      The system is currently still up and running and in this funny state - I reckon I've got about a day before it needs restarting, so if anything else useful can be gleaned let me know!

      dmesg.boot

      Copyright (c) 1992-2012 The FreeBSD Project.
      Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
      	The Regents of the University of California. All rights reserved.
      FreeBSD is a registered trademark of The FreeBSD Foundation.
      FreeBSD 8.3-RELEASE-p16 #0: Mon Aug 25 08:27:11 EDT 2014
          root@pf2_1_1_amd64.pfsense.org:/usr/obj.amd64/usr/pfSensesrc/src/sys/pfSense_SMP.8 amd64
      Timecounter "i8254" frequency 1193182 Hz quality 0
      CPU: AMD Phenom(tm) II X2 555 Processor (3214.64-MHz K8-class CPU)
        Origin = "AuthenticAMD"  Id = 0x100f43  Family = 10  Model = 4  Stepping = 3
        Features=0x178bfbff <fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,mmx,fxsr,sse,sse2,htt>Features2=0x802009 <sse3,mon,cx16,popcnt>AMD Features=0xee500800 <syscall,nx,mmx+,ffxsr,page1gb,rdtscp,lm,3dnow!+,3dnow!>AMD Features2=0x37ff <lahf,cmp,svm,extapic,cr8,abm,sse4a,mas,prefetch,osvw,ibs,skinit,wdt>TSC: P-state invariant
      real memory  = 4294967296 (4096 MB)
      avail memory = 4058845184 (3870 MB)
      ACPI APIC Table: <061210 APIC1416>
      FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
      FreeBSD/SMP: 1 package(s) x 2 core(s)
       cpu0 (BSP): APIC ID:  0
       cpu1 (AP): APIC ID:  1
      ioapic0 <version 1.1=""> irqs 0-23 on motherboard
      wlan: mac acl policy registered
      ipw_bss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
      ipw_bss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
      module_register_init: MOD_LOAD (ipw_bss_fw, 0xffffffff804abaf0, 0) error 1
      ipw_ibss: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
      ipw_ibss: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
      module_register_init: MOD_LOAD (ipw_ibss_fw, 0xffffffff804abb90, 0) error 1
      ipw_monitor: You need to read the LICENSE file in /usr/share/doc/legal/intel_ipw/.
      ipw_monitor: If you agree with the license, set legal.intel_ipw.license_ack=1 in /boot/loader.conf.
      module_register_init: MOD_LOAD (ipw_monitor_fw, 0xffffffff804abc30, 0) error 1
      kbd1 at kbdmux0
      cryptosoft0: <software crypto=""> on motherboard
      padlock0: No ACE support.
      acpi0: <061210 XSDT1416> on motherboard
      acpi0: [ITHREAD]
      acpi0: Power Button (fixed)
      acpi0: reservation of fec00000, fed40000 (3) failed
      acpi0: reservation of 0, a0000 (3) failed
      acpi0: reservation of 100000, bdf00000 (3) failed
      Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
      acpi_timer0: <24-bit timer at 3.579545MHz> port 0x508-0x50b on acpi0
      cpu0: <acpi cpu=""> on acpi0
      cpu1: <acpi cpu=""> on acpi0
      pcib0: <acpi host-pci="" bridge=""> port 0xcf8-0xcff on acpi0
      pci0: <acpi pci="" bus=""> on pcib0
      pci0: <memory, ram=""> at device 0.0 (no driver attached)
      isab0: <pci-isa bridge=""> port 0x900-0x9ff at device 1.0 on pci0
      isa0: <isa bus=""> on isab0
      pci0: <serial bus,="" smbus=""> at device 1.1 (no driver attached)
      pci0: <memory, ram=""> at device 1.2 (no driver attached)
      ohci0: <nvidia nforce="" mcp61="" usb="" controller=""> mem 0xdfdfb000-0xdfdfbfff irq 21 at device 2.0 on pci0
      ohci0: [ITHREAD]
      usbus0: <nvidia nforce="" mcp61="" usb="" controller=""> on ohci0
      ehci0: <nvidia nforce="" mcp61="" usb="" 2.0="" controller=""> mem 0xdfdfac00-0xdfdfacff irq 22 at device 2.1 on pci0
      ehci0: [ITHREAD]
      usbus1: EHCI version 1.0
      usbus1: <nvidia nforce="" mcp61="" usb="" 2.0="" controller=""> on ehci0
      pcib1: <acpi pci-pci="" bridge=""> at device 4.0 on pci0
      pci1: <acpi pci="" bus=""> on pcib1
      rl0: <realtek 10="" 8139="" 100basetx=""> port 0xd800-0xd8ff mem 0xdfeffc00-0xdfeffcff irq 16 at device 6.0 on pci1
      miibus0: <mii bus=""> on rl0
      rlphy0: <realtek internal="" media="" interface=""> PHY 0 on miibus0
      rlphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
      rl0: [ITHREAD]
      rl1: <realtek 10="" 8139="" 100basetx=""> port 0xd400-0xd4ff mem 0xdfeff800-0xdfeff8ff irq 17 at device 7.0 on pci1
      miibus1: <mii bus=""> on rl1
      rlphy1: <realtek internal="" media="" interface=""> PHY 0 on miibus1
      rlphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
      rl1: [ITHREAD]
      atapci0: <nvidia nforce="" mcp61="" udma133="" controller=""> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 6.0 on pci0
      ata0: <ata channel=""> at channel 0 on atapci0
      ata0: [ITHREAD]
      ata1: <ata channel=""> at channel 1 on atapci0
      ata1: [ITHREAD]
      nfe0: <nvidia nforce="" mcp61="" networking="" adapter=""> port 0xc480-0xc487 mem 0xdfdf9000-0xdfdf9fff irq 23 at device 7.0 on pci0
      miibus2: <mii bus=""> on nfe0
      rgephy0: <rtl8169s 8110s="" 8211b="" media="" interface=""> PHY 1 on miibus2
      rgephy0:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
      nfe0: [FILTER]
      nfe0: [FILTER]
      nfe0: [FILTER]
      nfe0: [FILTER]
      nfe0: [FILTER]
      nfe0: [FILTER]
      nfe0: [FILTER]
      nfe0: [FILTER]
      atapci1: <nvidia nforce="" mcp61="" sata300="" controller=""> port 0xc400-0xc407,0xc080-0xc083,0xc000-0xc007,0xbc00-0xbc03,0xb880-0xb88f mem 0xdfdf8000-0xdfdf8fff irq 20 at device 8.0 on pci0
      atapci1: [ITHREAD]
      ata2: <ata channel=""> at channel 0 on atapci1
      ata2: [ITHREAD]
      ata3: <ata channel=""> at channel 1 on atapci1
      ata3: [ITHREAD]
      atapci2: <nvidia nforce="" mcp61="" sata300="" controller=""> port 0xb800-0xb807,0xb480-0xb483,0xb400-0xb407,0xb080-0xb083,0xb000-0xb00f mem 0xdfdf3000-0xdfdf3fff irq 21 at device 8.1 on pci0
      atapci2: [ITHREAD]
      ata4: <ata channel=""> at channel 0 on atapci2
      ata4: [ITHREAD]
      ata5: <ata channel=""> at channel 1 on atapci2
      ata5: [ITHREAD]
      pcib2: <acpi pci-pci="" bridge=""> at device 9.0 on pci0
      pci2: <acpi pci="" bus=""> on pcib2
      pcib3: <acpi pci-pci="" bridge=""> at device 11.0 on pci0
      pci3: <acpi pci="" bus=""> on pcib3
      em0: <intel(r) 1000="" pro="" network="" connection="" 7.3.8=""> port 0xec00-0xec1f mem 0xdffe0000-0xdfffffff,0xdff00000-0xdff7ffff,0xdffdc000-0xdffdffff irq 18 at device 0.0 on pci3
      em0: Using MSIX interrupts with 3 vectors
      em0: [ITHREAD]
      em0: [ITHREAD]
      em0: [ITHREAD]
      pcib4: <acpi pci-pci="" bridge=""> at device 12.0 on pci0
      pci4: <acpi pci="" bus=""> on pcib4
      vgapci0: <vga-compatible display=""> mem 0xde000000-0xdeffffff,0xc0000000-0xcfffffff,0xdd000000-0xddffffff irq 22 at device 13.0 on pci0
      acpi_button0: <power button=""> on acpi0
      atrtc0: <at realtime="" clock=""> port 0x70-0x71 irq 8 on acpi0
      ppc0: <parallel port=""> port 0x378-0x37f irq 7 on acpi0
      ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
      ppc0: [ITHREAD]
      ppbus0: <parallel port="" bus=""> on ppc0
      plip0: <plip network="" interface=""> on ppbus0
      plip0: [ITHREAD]
      lpt0: <printer> on ppbus0
      lpt0: [ITHREAD]
      lpt0: Interrupt-driven port
      ppi0: <parallel i="" o=""> on ppbus0
      acpi_hpet0: <high precision="" event="" timer=""> iomem 0xfed00000-0xfed00fff on acpi0
      Timecounter "HPET" frequency 25000000 Hz quality 900
      uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
      uart0: [FILTER]
      uart0: console (115200,n,8,1)
      orm0: <isa option="" rom=""> at iomem 0xcf000-0xcffff on isa0
      sc0: <system console=""> at flags 0x100 on isa0
      sc0: VGA <16 virtual consoles, flags=0x300>
      vga0: <generic isa="" vga=""> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
      atkbdc0: <keyboard controller="" (i8042)=""> at port 0x60,0x64 on isa0
      atkbd0: <at keyboard=""> irq 1 on atkbdc0
      kbd0 at atkbd0
      atkbd0: [GIANT-LOCKED]
      atkbd0: [ITHREAD]
      hwpstate0: <cool`n'quiet 2.0=""> on cpu0
      Timecounters tick every 1.000 msec
      IPsec: Initialized Security Association Processing.
      usbus0: 12Mbps Full Speed USB v1.0
      usbus1: 480Mbps High Speed USB v2.0
      ad4: 476940MB <st500dm002 1bd142="" kc45=""> at ata2-master UDMA100 SATA 1.5Gb/s
      ugen0.1: <nvidia> at usbus0
      uhub0: <nvidia 1="" 9="" ohci="" root="" hub,="" class="" 0,="" rev="" 1.00="" 1.00,="" addr=""> on usbus0
      ugen1.1: <nvidia> at usbus1
      uhub1: <nvidia 1="" 9="" ehci="" root="" hub,="" class="" 0,="" rev="" 2.00="" 1.00,="" addr=""> on usbus1
      SMP: AP CPU #1 Launched!
      Root mount waiting for: usbus1 usbus0
      uhub0: 10 ports with 10 removable, self powered
      Root mount waiting for: usbus1
      Root mount waiting for: usbus1
      Root mount waiting for: usbus1
      uhub1: 10 ports with 10 removable, self powered
      Trying to mount root from ufs:/dev/ad4s1a
      ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
                  to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
      ZFS WARNING: Recommended minimum kmem_size is 512MB; expect unstable behavior.
                   Consider tuning vm.kmem_size and vm.kmem_size_max
                   in /boot/loader.conf.
      ZFS filesystem version 5
      ZFS storage pool version 28</nvidia></nvidia></nvidia></nvidia></st500dm002></cool`n'quiet></at></keyboard></generic></system></isa></high></parallel></printer></plip></parallel></parallel></at></power></vga-compatible></acpi></acpi></intel(r)></acpi></acpi></acpi></acpi></ata></ata></nvidia></ata></ata></nvidia></rtl8169s></mii></nvidia></ata></ata></nvidia></realtek></mii></realtek></realtek></mii></realtek></acpi></acpi></nvidia></nvidia></nvidia></nvidia></memory,></serial></isa></pci-isa></memory,></acpi></acpi></acpi></acpi></software></version></lahf,cmp,svm,extapic,cr8,abm,sse4a,mas,prefetch,osvw,ibs,skinit,wdt></syscall,nx,mmx+,ffxsr,page1gb,rdtscp,lm,3dnow!+,3dnow!></sse3,mon,cx16,popcnt></fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,mmx,fxsr,sse,sse2,htt>
      

      recent dmesg output, repeated many times

      ugen0.2: <unknown> at usbus0 (disconnected)
      uhub_reattach_port: could not allocate new device
      uhub_reattach_port: giving up port reset - device vanished
      usb_alloc_device: set address 2 failed (USB_ERR_IOERROR, ignored)
      usbd_setup_device_desc: getting device descriptor at addr 2 failed, USB_ERR_IOERROR
      usbd_req_re_enumerate: addr=2, set address failed! (USB_ERR_IOERROR, ignored)
      usbd_setup_device_desc: getting device descriptor at addr 2 failed, USB_ERR_IOERROR
      usbd_req_re_enumerate: addr=2, set address failed! (USB_ERR_IOERROR, ignored)
      usbd_setup_device_desc: getting device descriptor at addr 2 failed, USB_ERR_IOERROR</unknown>
      

      ps aux

      USER     PID %CPU %MEM   VSZ   RSS  TT  STAT STARTED      TIME COMMAND
      root      11 196.7  0.0     0    32  ??  RL   15Oct14 25120:14.22 [idle]
      root       0  2.5  0.0     0   208  ??  DLs  15Oct14 115:45.87 [kernel]
      root      12  2.3  0.0     0   368  ??  WL   15Oct14 137:48.44 [intr]
      root       1  0.0  0.0  3200   588  ??  ILs  15Oct14   0:00.61 /sbin/init --
      root       2  0.0  0.0     0    16  ??  DL   15Oct14   0:00.00 [g_event]
      root       3  0.0  0.0     0    16  ??  DL   15Oct14   1:43.97 [g_up]
      root       4  0.0  0.0     0    16  ??  DL   15Oct14   1:00.65 [g_down]
      root       5  0.0  0.0     0    16  ??  DL   15Oct14   0:00.00 [crypto]
      root       6  0.0  0.0     0    16  ??  DL   15Oct14   0:00.00 [crypto returns]
      root       7  0.0  0.0     0    16  ??  DL   15Oct14   0:00.01 [sctp_iterator]
      root       8  0.0  0.0     0    16  ??  DL   15Oct14   0:09.19 [pfpurge]
      root       9  0.0  0.0     0    16  ??  DL   15Oct14   0:00.00 [xpt_thrd]
      root      10  0.0  0.0     0    16  ??  DL   15Oct14   0:00.00 [audit]
      root      13  0.0  0.0     0    32  ??  DL   15Oct14   0:01.09 [ng_queue]
      root      14  0.0  0.0     0    16  ??  DL   15Oct14   3:49.03 [yarrow]
      root      15  0.0  0.0     0   128  ??  DL   15Oct14  20:35.75 [usb]
      root      16  0.0  0.0     0    16  ??  DL   15Oct14   0:00.47 [pagedaemon]
      root      17  0.0  0.0     0    16  ??  DL   15Oct14   0:00.00 [vmdaemon]
      root      18  0.0  0.0     0    16  ??  DL   15Oct14   0:00.01 [pagezero]
      root      19  0.0  0.0     0    16  ??  DL   15Oct14   0:00.68 [idlepoll]
      root      20  0.0  0.0     0    16  ??  DL   15Oct14   0:01.88 [bufdaemon]
      root      21  0.0  0.0     0    16  ??  DL   15Oct14   1:12.54 [syncer]
      root      22  0.0  0.0     0    16  ??  DL   15Oct14   0:02.07 [vnlru]
      root      23  0.0  0.0     0    16  ??  DL   15Oct14   0:02.22 [softdepflush]
      root      36  0.0  0.0     0    32  ??  DL   15Oct14   0:04.70 [zfskern]
      root      66  0.0  0.0     0    16  ??  DL   15Oct14   0:08.36 [md0]
      root     258  0.0  0.0  6908  1404  ??  INs  15Oct14 213:09.37 /usr/local/sbin/check_reload_status
      root     260  0.0  0.0  6908  1288  ??  IN   15Oct14   0:00.00 check_reload_status: Monitoring daemon of check_reload_status
      root     269  0.0  0.1  5248  3148  ??  Ss   15Oct14   1:25.81 /sbin/devd
      root     753  0.0  0.0  6960  1664  ??  Ss   15Oct14   0:09.00 /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -f /var/etc/syslog.conf
      root    1227  0.0  0.0  7928  1620  ??  Is   15Oct14   0:00.85 /usr/sbin/cron -s
      root    6912  0.0  0.1 25028  4300  ??  Ss   15Oct14   0:07.09 /usr/local/sbin/mpd5 -b -k -d /var/etc -f mpd_wan.conf -p /var/run/pppoe_wan.pid -s ppp pppoeclient
      root   10176  0.0  0.1 15264  3700  ??  INs  15Oct14   0:00.00 /usr/sbin/sshd
      root   11287  0.0  0.1 25028  4604  ??  Ss   15Oct14   0:04.18 /usr/local/sbin/mpd5 -b -k -d /var/etc -f mpd_opt1.conf -p /var/run/pppoe_opt1.pid -s ppp pppoeclient
      root   17344  0.0  2.8 442348 116352  ??  Ss    9:49AM   0:04.06 /usr/pbi/snort-amd64/bin/snort -R 8376 -D -q -l /var/log/snort/snort_pppoe08376 --pid-path /var/run --nolock-pidfile -G 8376 -c /usr/pbi/snort-amd64/etc/snort/snort_8376_ppp
      nobody 17656  0.0  0.0  6836  1228  ??  Is   10:36AM   0:00.00 nc -w 2000 1***.***.***.*** 143
      root   26684  0.0  0.0  8984  1720  ??  Is   15Oct14   0:27.76 /usr/sbin/inetd -wW -R 0 -a 127.0.0.1 /var/etc/inetd.conf
      nobody 26761  0.0  0.0  6836  1232  ??  Is    8:27AM   0:00.01 nc -w 2000 ***.***.***.*** 143
      nobody 27056  0.0  0.0  6836  1228  ??  Is    8:27AM   0:00.00 nc -w 2000 ***.***.***.*** 143
      nobody 27093  0.0  0.0  6836  1228  ??  Is    8:27AM   0:00.01 nc -w 2000 ***.***.***.*** 143
      root   38658  0.0  2.8 440300 114216  ??  Ss    9:49AM   0:01.58 /usr/pbi/snort-amd64/bin/snort -R 52612 -D -q -l /var/log/snort/snort_pppoe152612 --pid-path /var/run --nolock-pidfile -G 52612 -c /usr/pbi/snort-amd64/etc/snort/snort_52612
      root   40746  0.0  0.6 147776 22920  ??  Is   15Oct14   0:00.07 /usr/local/bin/php
      root   41374  0.0  0.6 147776 22920  ??  Is   15Oct14   0:00.21 /usr/local/bin/php
      nobody 49725  0.0  0.0  6836  1232  ??  Is    8:33AM   0:00.02 nc -w 2000 ***.***.***.*** 143
      nobody 54102  0.0  0.0  6836  1232  ??  Is    9:41AM   0:00.01 nc -w 2000 ***.***.***.*** 143
      root   68045  0.0  0.0  5784  1500  ??  SNs  15Oct14   2:30.73 /usr/local/sbin/apinger -c /var/etc/apinger.conf
      root   68409  0.0  0.0 14392  2004  ??  IN   15Oct14   0:01.72 /usr/local/bin/rrdtool -
      nobody 71710  0.0  0.0  6836  1232  ??  Is    9:37AM   0:00.01 nc -w 2000 ***.***.***.*** 143
      root   71728  0.0  1.0 151872 41152  ??  I    10:18AM   0:05.35 /usr/local/bin/php
      nobody 74208  0.0  0.0  6836  1228  ??  Is   10:06AM   0:00.00 nc -w 2000 ***.***.***.*** 143
      root   85187  0.0  0.0  2716   964  ??  IN   10:49AM   0:00.00 sleep 60
      root   85239  0.0  0.1 26164  4184  ??  SNs  10:49AM   0:00.02 sshd: admin@pts/0 (sshd)
      nobody 85396  0.0  0.1 16040  5328  ??  S    15Oct14   2:00.46 /usr/local/sbin/dnsmasq --all-servers --rebind-localhost-ok --stop-dns-rebind --listen-address=***.***.***.*** --dns-forward-max=5000 --cache-size=10000 --local-ttl=1
      nobody 85704  0.0  0.0  6836  1228  ??  Is   10:39AM   0:00.00 nc -w 2000 ***.***.***.*** 143
      nobody 85882  0.0  0.0  6836  1228  ??  Is   10:39AM   0:00.00 nc -w 2000 ***.***.***.*** 143
      nobody 85917  0.0  0.0  6836  1228  ??  Is   10:39AM   0:00.00 nc -w 2000 ***.***.***.*** 143
      root   86346  0.0  0.0  7036  1392  ??  Is   15Oct14   0:00.01 /usr/local/sbin/sshlockout_pf 15
      nobody 89988  0.0  0.0  6836  1228  ??  Is   10:44AM   0:00.01 nc -w 2000 ***.***.***.*** 143
      nobody 90335  0.0  0.0  6836  1228  ??  Is   10:44AM   0:00.00 nc -w 2000 ***.***.***.*** 143
      root   90944  0.0  0.2 12032  6988  ??  SNs  10:52PM   0:01.18 /usr/local/sbin/ntpd -g -c /var/etc/ntpd.conf -p /var/run/ntpd.pid
      dhcpd  92639  0.0  0.3 19148 12952  ??  Ss   15Oct14   0:35.54 /usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf -pf /var/run/dhcpd.pid nfe0 em0
      root   95229  0.0  0.9 149824 35476  ??  I    10:24AM   0:00.41 /usr/local/bin/php
      root   96045  0.0  0.0  5784  1180  ??  Is   15Oct14   0:00.00 /usr/local/bin/minicron 240 /var/run/ping_hosts.pid /usr/local/bin/ping_hosts.sh
      root   96126  0.0  0.0  5784  1228  ??  I    15Oct14   0:00.27 minicron: helper /usr/local/bin/ping_hosts.sh  (minicron)
      root   96414  0.0  0.0  5784  1180  ??  Is   15Oct14   0:00.00 /usr/local/bin/minicron 3600 /var/run/expire_accounts.pid /etc/rc.expireaccounts
      root   96869  0.0  0.0  5784  1228  ??  I    15Oct14   0:00.02 minicron: helper /etc/rc.expireaccounts  (minicron)
      root   96873  0.0  0.0  5784  1180  ??  Is   15Oct14   0:00.00 /usr/local/bin/minicron 86400 /var/run/update_alias_url_data.pid /etc/rc.update_alias_url_data
      root   97116  0.0  0.0  5784  1228  ??  I    15Oct14   0:00.00 minicron: helper /etc/rc.update_alias_url_data  (minicron)
      root   98012  0.0  0.2 26280  6548  ??  SN   15Oct14   1:01.23 /usr/local/sbin/lighttpd -f /var/etc/lighty-webConfigurator.conf
      root   98653  0.0  0.0  8296  1856  ??  IN   15Oct14   0:57.91 /bin/sh /var/db/rrd/updaterrd.sh
      root     300  0.0  0.0 19480  1844  u0  Is   15Oct14   0:00.01 login [pam] (login)
      root     344  0.0  0.0  8296  1796  u0  I    15Oct14   0:00.00 -sh (sh)
      root    2900  0.0  0.0  8296  1760  u0  I+   15Oct14   0:00.00 /bin/sh /etc/rc.initial
      root   22505  0.0  0.1 11744  2548  u0- S    15Oct14   0:05.05 /usr/sbin/tcpdump -s 256 -v -S -l -n -e -ttt -i pflog0
      root   22506  0.0  0.0  5780  1096  u0- I    15Oct14   0:01.40 logger -t pf -p local0.info
      root     214  0.0  0.0 19480  1844  v0  Is   15Oct14   0:00.01 login [pam] (login)
      root     540  0.0  0.0  8296  1796  v0  I    15Oct14   0:00.00 -sh (sh)
      root    3231  0.0  0.0  8296  1760  v0  I+   15Oct14   0:00.00 /bin/sh /etc/rc.initial
      root   85777  0.0  0.0  8296  1808   0  Ss   10:49AM   0:00.00 /bin/sh /etc/rc.initial
      root   87784  0.0  0.1  8268  2784   0  S    10:49AM   0:00.01 /bin/tcsh
      root   88555  0.0  0.0  7992  1524   0  R+   10:49AM   0:00.00 ps aux
      
      

      vmstat -mh

      $ vmstat -mh
               Type InUse MemUse HighUse Requests  Size(s)
               cdev    11     3K       -       11  256
              sigio     1     1K       -        2  64
           filedesc   113    73K       -  5124517  16,512,1024,2048
               kenv    78    11K       -       88  16,32,64,128
             kqueue     6     6K       -   112753  256,512,2048
          proc-args    53     5K       -  2849369  16,32,64,128,256
              hhook     2     1K       -        2  128
      CAM dev queue     1     1K       -        1  128
            ithread   100    16K       -      100  32,128,256
        ata_generic     1     1K       -      226  16,512,1024
             KTRACE   100    13K       -      100  128
               UART     3     2K       -        3  16,512,1024
             linker   233   856K       -   210311  16,32,64,128,256,512,1024,2048,4096
              lockf    20     3K       - 59407951  64,128
             kbdmux     6    10K       -        6  16,512,1024,2048,4096
             ip6ndp    20     2K       -      766  64,128
             ip6opt    13     4K       -    17521  256
               temp    75   691K       - 14256975  16,32,64,128,256,512,1024,2048,4096
             devbuf 621591 1263690K       -  4974322  16,32,64,128,256,512,1024,2048,4096
                LED     2     1K       -        2  16,128
             module   537    68K       -      540  128
           mtx_pool     2    16K       -        2  
                osd     3     1K       -        4  16,64
            subproc   243   402K       -  2583762  512,4096
               proc     2    16K       -        2  
            session    38     5K       -    11586  128
               pgrp    43     6K       -    11599  128
               cred    74    12K       - 67439438  64,256
            uidinfo     4     3K       -      377  128,2048
             plimit    22     6K       -    61781  256
            acpidev    72     5K       -       72  64
         md_sectors     6    24K       -        6  4096
            md_disk     7     3K       -        7  32,2048
          sysctltmp     0     0K       -  1435113  16,32,64,256
          sysctloid  4203   207K       -     4300  16,32,64,128
             sysctl     0     0K       -  3625820  16,32,64
            callout     1   512K       -        1  
               umtx   336    42K       -      336  128
           p1003.1b     1     1K       -        1  16
               SWAP     2  1097K       -        2  64
             bus-sc    83   147K       -     5207  16,32,64,128,256,512,1024,2048,4096
                bus  1269   107K       -   213329  16,32,64,128,256,512,1024
            devstat     4     9K       -        4  32,4096
       eventhandler    97     8K       -       97  64,128
               kobj   326  1304K       -      387  4096
            Per-cpu     1     1K       -        1  32
             USBdev    96    47K       -    35979  64,128,512,1024,2048
                USB    14     5K       -   121426  16,32,64,128,256,2048
               rman   210    26K       -      705  16,32,128
          ad_driver     1     1K       -        1  32
               sbuf     0     0K       -   143276  16,32,64,128,256
          ar_driver     0     0K       -        6  512,2048
          taskqueue    23     2K       -       23  16,32,128
             Unitno    16     1K       -   112314  32,64
                iov     0     0K       -  1044615  16,64,128,256
             select   236    30K       -      236  128
           ioctlops     0     0K       - 46990235  16,32,64,128,256,512,1024,2048,4096
                msg     4    89K       -        4  1024
                sem     4    11K       -        4  512,1024
                shm     1    20K       -        1  
                tty    21    21K       -       22  1024
                pts     1     1K       -        2  256
           mbuf_tag     0     0K       - 1940714252  32,64
               ksem     1     8K       -        1  
              shmfd     1     8K       -        1  
                pcb    83   160K       -   369462  16,32,64,128,1024,2048,4096
             soname    15     2K       -  8479922  16,32,128
                acl     0     0K       -       36  4096
             biobuf   209   418K       -    32846  2048
           vfscache     1  1024K       -        1  
         cl_savebuf     0     0K       -    40840  64,128
           vfs_hash     1   512K       -        1  
             vnodes     2     1K       -        2  256
        vnodemarker     0     0K       -    93244  512
              mount    42     2K       -       69  16,32,64,128,256
            fadvise     0     0K       -  3765690  32
                BPF    32   213K       -    20800  16,128,512,4096
        ether_multi   112     6K       -     4718  16,32,64
             ifaddr   146    36K       -     1629  32,64,256,512,2048,4096
              ifnet    13    25K       -       13  128,2048
              clone    15    60K       -       15  4096
             arpcom     4     1K       -        4  16
            lltable   840   216K       -     3558  256,512
                tun     1     1K       -        1  256
             DEVFS1    84    42K       -    33798  512
             DEVFS3   198    50K       -     3254  256
             DEVFS2    82     2K       -       84  16
           routetbl   112    23K       - 43995401  32,64,128,256,512
              DEVFS    21     1K       -     1537  16,128
             DEVFSP     5     1K       -    21051  64
       netgraph_msg     0     0K       -   310528  64,128,256,512,1024,2048
      netgraph_node    24     6K       -    10147  256
      netgraph_hook    24     3K       -       36  128
           netgraph     8     1K       -       10  64,256
      netgraph_iface     2     1K       -        2  128
      netgraph_mppc     0     0K       -        2  1024
       netgraph_ppp     2    24K       -        2  
      netgraph_pppoe     4    25K       -        6  64,512
      netgraph_sock    26     4K       -    20270  32,128
      netgraph_path     0     0K       -   155285  16,32
               igmp    12     3K       -       12  256
               ipid     2    24K       -        2  
        ip_moptions     8     2K       -     7404  64,256
           in_multi     5     2K       -      745  256
         in_mfilter     4     4K       -     3702  1024
      encap_export_host     4     4K       -        4  1024
          mroutetbl     1     1K       -        1  16
          sctp_iter     0     0K       -     1487  256
           sctp_ifn     5     1K       -      745  128
           sctp_ifa     9     2K       -     1489  128
           sctp_vrf     1     1K       -        1  64
          sctp_a_it     0     0K       -     1487  16
          hostcache     1    28K       -        1  
           syncache     1    96K       -        1  
        nullfs_hash     1     1K       -        1  128
          in6_multi    60     8K       -     4574  32,256
          pfs_nodes    21     6K       -       21  256
                mld    12     2K       -       12  128
        inpcbpolicy    94     3K       - 11879699  32
        ipsecpolicy   188    47K       - 23924378  256
            NFS FHA     1     2K       -        1  2048
             crypto     1     1K       -        1  512
              xform     0     0K       -   239127  16,32
                rpc     2     9K       -        2  256
      audit_evclass   172     6K       -      211  32
             newblk     1     1K       -        1  512
           inodedep     1   512K       -        1  
            pagedep     1   128K       -        1  
        ufs_dirhash    57    11K       -      189  16,32,64,128,256,512
          ufs_mount     6    65K       -        6  512,2048,4096
            UMAHash     2     1K       -        2  512
          vm_pgdata     2   129K       -        2  128
        ddb_capture     1    48K       -        1  
           pci_link    36     3K       -       36  16,128
            io_apic     1     2K       -        1  2048
            memdesc     1     4K       -        1  4096
          acpi_perf     2     1K       -        2  128
               GEOM    48    16K       -      357  16,32,64,128,256,512,1024
           atkbddev     2     1K       -        2  64
             acpica  3866   396K       -    89568  16,32,64,128,256,512,1024,2048
           acpitask     1     2K       -        1  2048
               spkr     0     0K       -        5  512
          CAM queue     3     1K       -        7  16
            acpisem    19     3K       -       19  128
           ppbusdev     3     1K       -        3  256
            entropy  1024    64K       -     1024  64
            CAM SIM     1     1K       -        1  256
         CAM periph     2     1K       -       12  16,32,64,128,256
             isadev     6     1K       -        6  128
            CAM XPT    12     3K       -       26  32,64,128,2048
                msi    12     2K       -       12  32,128
           nexusdev     5     1K       -        5  16
            solaris   103  8711K       -      105  16,64,128,1024
         kstat_data     4     1K       -        4  64
           dummynet     6     2K       -        6  256,512
           dummynet     3     3K       -        3  512,1024
      
      ```![status_rrd_graph_img.php.png](/public/_imported_attachments_/1/status_rrd_graph_img.php.png)
      ![status_rrd_graph_img.php.png_thumb](/public/_imported_attachments_/1/status_rrd_graph_img.php.png_thumb)
      1 Reply Last reply Reply Quote 0
      • C
        charliem
        last edited by

        @jpetpet:

        I've installed the Apcupsd package …

        All was seemingly fine until a recently when after a power failure and subsequent reboot the wired memory usage grew and grew until swapping occurred ...

        One odd thing I noticed was that dmesg shows me some sort of USB error ...

        recent dmesg output, repeated many times:

        ugen0.2: <unknown> at usbus0 (disconnected)
        uhub_reattach_port: could not allocate new device
        uhub_reattach_port: giving up port reset - device vanished
        usb_alloc_device: set address 2 failed (USB_ERR_IOERROR, ignored)
        usbd_setup_device_desc: getting device descriptor at addr 2 failed, USB_ERR_IOERROR
        usbd_req_re_enumerate: addr=2, set address failed! (USB_ERR_IOERROR, ignored)
        usbd_setup_device_desc: getting device descriptor at addr 2 failed, USB_ERR_IOERROR
        usbd_req_re_enumerate: addr=2, set address failed! (USB_ERR_IOERROR, ignored)
        usbd_setup_device_desc: getting device descriptor at addr 2 failed, USB_ERR_IOERROR</unknown>
        

        Did the USB errors started showing up after the power outage?

        At first glance I would try disconnecting USB from your UPS, and perhaps temporarily remove the apcpusd package.  Maybe the power outage damaged the USB port on the UPS (how ironic!).  It's not clear to me how that could cause a memory leak, but a lot is possible with damaged hardware.

        1 Reply Last reply Reply Quote 0
        • J
          jpetpet
          last edited by

          Thanks charliem! I'm starting to think that USB is the root of the issue.

          I fired up the apcupsd status panel and got the "could not connect to service" error; according to the system logs apcupsd was not started as the UPS could not be found. I unplugged the UPS, plugged it in again, checked dmesg and got the following:

          ugen0.2: <unknown>at usbus0 (disconnected)
          uhub_reattach_port: could not allocate new device
          ugen0.2: <american power="" conversion="">at usbus0</american></unknown> 
          

          I went into the apcupsd panel and hit "Save" on the configuration page then clicked over to the "Status" tab and lo and behold, the UPS service had been started and the status was printed out. This was a couple of hours ago; I've just checked the RRD graphs / vmstat and the wired memory has stopped increasing and leveled off, and no funny USB related stuff in dmesg either.

          I've checked the UPS' USB port on my linux dev machine and all seems just fine, my guess is that there is an issue with the USB driver and the way it is initialising the USB hub which is hung off the controller on the motherboard. I've detached the UPS' USB for the time being, will restart it and see what happens. Incidentally I plan to upgrade the hardware to a more modern Haswell CPU / board in order to save some power so I'll do the same set of experiments once that's up and running in a couple of weeks.

          Many thanks for your help! If things start to get wierd again I'll append this thread with an update as it does seem there may be an underlying issue with the kernel and this USB configuration.

          1 Reply Last reply Reply Quote 0
          • First post
            Last post
          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.