Issue with PHP_errors.log filling /tmp

Sn3ak

I just tried logging into my network via IPsec and couldn't get in. Which may be unrelated.. Anyways, Upon opening System logs to try and determine why that was. I got to
see fun messages like:

kernel: pid 35766 (php), uid 0 inumber 73 on /tmp: filesystem full

Upon logging in via ssh and running du -h -d1 I see PHP_errors.log is 500+MB.

Other than one other IPsec error, the file is full of:

03-Nov-2014 18:12:47 America/Los_Angeles] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 32 bytes) in /usr/local/sbin/prefixes.php on line 9
[03-Nov-2014 18:12:47 America/Los_Angeles] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 71 bytes) in /usr/local/sbin/prefixes.php on line 9
[03-Nov-2014 18:12:48 America/Los_Angeles] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 32 bytes) in /usr/local/sbin/prefixes.php on line 9
[03-Nov-2014 18:12:48 America/Los_Angeles] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 3 bytes) in /usr/local/sbin/prefixes.php on line 9
[03-Nov-2014 18:12:49 America/Los_Angeles] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 71 bytes) in /usr/local/sbin/prefixes.php on line 9
[03-Nov-2014 18:12:49 America/Los_Angeles] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 71 bytes) in /usr/local/sbin/prefixes.php on line 9
[03-Nov-2014 18:12:50 America/Los_Angeles] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 71 bytes) in /usr/local/sbin/prefixes.php on line 9
[03-Nov-2014 18:12:50 America/Los_Angeles] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 32 bytes) in /usr/local/sbin/prefixes.php on line 9
[03-Nov-2014 18:12:50 America/Los_Angeles] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 71 bytes) in /usr/local/sbin/prefixes.php on line 9
[03-Nov-2014 18:12:51 America/Los_Angeles] PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 71 bytes) in /usr/local/sbin/prefixes.php on line 9

The log started in late September. (Which is probably when I replaced the old failing router with a much nicer one, with a new config from scratch)

Any idea what I can do to track this down?

I see it has something to do with dhcpd6.leases. my dhcpd6.leases is 22M although I don't know why it is, I shouldn't have that much pulling dhcp6.

cmb

That seems like a really big leases file, what's in it?

Sn3ak

an excessive amount of leases?! (Sorry, not really sure how to answer that question)

I don't know enough about the format of the lease file for ipv6.

I see numerous (20+) entries for the same ip with different in-na declarations. (this count is only reading the last ~25 ia-na declarations at the bottom of the file)


ia-na "\002\000\000\000\000\003\000\001\000\300\267[\235\326" {
  cltt 2 2014/11/04 02:37:27;
  iaaddr 2001:470:1f05:xxx::f00 {
    binding state active;
    preferred-life 4500;
    max-life 7200;
    ends 2 2014/11/04 04:37:27;
  }
}

ia-na "\002\000\000\000\000\003\000\001\000\300\267\263\211\003" {
  cltt 2 2014/11/04 02:37:27;
  iaaddr 2001:470:1f05:xxx::f00 {
    binding state active;
    preferred-life 4500;
    max-life 7200;
    ends 2 2014/11/04 04:37:27;
  }
}

[/code]
[code]/root(4): grep "2001:470:1f05:xxx::f00" /var/dhcpd/var/db/dhcpd6.leases | wc -l
   24545
[/code]

I also forgot to mention in my original post, I'm running 2.1.5 on amd64 for this box.

For my dhcpv6 server settings, everything is left at defaults, other than setting up a scope of 100 ips, a dns server and a ntp server.

Sn3ak

for giggles, I stopped the dhcpd6 service, moved the old lease file to leases.old, and started up dhcpv6.

Within a minute the file is over 600kb, with 20-30 leases for the same ip.

cmb

That still happening? If so, I'd recommend trying 2.2 to see if it's the same there, and if it is, we can check into it.

Sn3ak

Yes, it is still happening with 2.2-Release.

I can watch the dhcp log getting 10s of hits per second.

I've been working on arranging my network so I can cut the majority of my devices off to see if I can find out if it's
a bad client causing the issue. Otherwise I'm not sure how to tract the culprit down.

Sn3ak

Tracked down the issue to an Apple Time Capsule. Once removed, things seem to be behaving themselves.

Sn3ak

@Sn3ak:

Tracked down the issue to an Apple Time Capsule. Once removed, things seem to be behaving themselves.

Sadly, I spoke too soon. This problem is back.

Any advice on tracking it down would be great, otherwise I'll take another stab at removing things off my network to find out if it is another noisy client.

cmb

Could you PM me a copy of your dhcpd6.leases files? Can scp it off from /var/dhcpd/var/ directory, or download /var/dhcpd/var/dhcpd6.leases from Diag>Command.