Issues with /usr/local/sbin/dhcpleases

Jason Crowley

On several occasions I've had problems with the dhcpleases process. It appears to develop a memory leak. The problem has occurred on multiple pfSense instances. None of them have any additional modules installed. Each time the problem has occurred, I've killed and restarted the process, and the problem has gone away.

This morning I started collecting data so that I can hopefully find the root cause of the problem and eliminate it. Note in the output below that dhcpleases is using over 95MB of RAM.

[2.1-RELEASE][root@pfsense.domain.com]/root(3): ps auxwww | grep 'USER\dhcpleases'
USER     PID %CPU %MEM   VSZ   RSS  TT  STAT STARTED      TIME COMMAND
root   26227  0.0  9.0 95892 92640  ??  Ss    7Jul14   3:13.87 /usr/local/sbin/dhcpleases -l /var/dhcpd/var/db/dhcpd.leases -d domain.com -p /var/run/dnsmasq.pid -h /var/etc/hosts

To ensure that the files loaded by dhcpleases were not huge and consuming the RAM, I checked them.

[2.1-RELEASE][root@pfsense.domain.com]/root(13): ls -l /var/dhcpd/var/db/dhcpd.leases /var/run/dnsmasq.pid /var/etc/hosts
-rw-r--r--  1 dhcpd  _dhcp  24614 Dec  9 07:29 /var/dhcpd/var/db/dhcpd.leases
-rw-r--r--  1 root   wheel   6674 Dec  9 07:29 /var/etc/hosts
-rw-r--r--  1 root   wheel      6 Jul  7 09:54 /var/run/dnsmasq.pid

They looked good, so I restarted dhcpleases and everything appears to be fine now. I had no idea what the dhcpleases process did, so I looked up the source code. There's not a lot of commentary in the source or anywhere else I could find, but this is what I think it does. Will someone tell me if I'm right?

_The dhcpleases process waits for a kevent to tell it the dhcpd.leases file hass been updated. When that file is updated, it writes a new /var/etc/hosts file with IPs and host names for all DHCP leases. Then it sends a HUP signal to dnsmasq to get it to reread /var/etc/hosts and produce DNS results for host names associated with DHCP leases. Finally it loops and waits for another kevent. _

Now that I think I understand the purpose of dhcpleases, I want to find out why its memory usage sometimes blows up. I have memory monitoring for the firewalls I manage, so I get notified before anything gets too crazy. Does anyone have suggestions for what I should look at when I find that dhcpleases is consuming too much memory?

Jason Crowley

I encountered the same problem again today on a different firewall. In this case there was some bad data in the /var/etc/hosts file like what you see in the link below.

https://redmine.pfsense.org/issues/3176

Today's firewall has about 300 devices behind it, and the hosts file was much bigger. I opened it to explore, and I found that restarting the dhcpleases process the way I describe above causes duplicate entries in the hosts file.

This time killed dhcpleases. Then I deleted the following line and everything below it from the hosts file and restarted dhcpleases.

# dhcpleases automatically entered

That seems to have cleared up the problem for now. I still don't know why it's occurring, but I'll post here if I figure it out.

cmb

I suspect that's the file handle leaking issue that some have seen. That's been fixed in 2.2, I'd encourage you to try 2.2-RC. Your description of what it does is correct.