DHCP Server fails after sometime- Filesystem full

fsantaana

I am running pfsense 2.0.2 with Captive Portal and FreeRadius2 using embedded 2G image on CF card. After several hours with no problems the device stops providing dhcp addresses. What i discover is that it's telling me that the filesystem is full. My DHCP lease is 86400 and the range is 172.16.0.10-172.16.255.250/16 subnet.

Please see the logs below. When i restart the system everything goes back to normal and all is working again. Is it a way to increase the database size or reduce the logs or what is written? I'm not sure why this is getting filled.

$ df -i
Filesystem        512-blocks   Used   Avail Capacity iused  ifree %iused  Mounted on
/dev/ufs/pfsense0    1859358 689885 1020725    40%   16635 102499   14%   /
devfs                      2      2       0   100%       0      0  100%   /dev
/dev/md0               78812   1424   71084     2%     136   5238    3%   /tmp
/dev/md1              118492 118388   -9372   109%     149   7785    2%   /var
/dev/ufs/cf           101055   2226   90745     2%      21   6505    0%   /cf
devfs                      2      2       0   100%       0      0  100%   /var/dhcpd/dev

 Jan 22 09:43:05	kernel: pid 12591 (dhcpd), uid 1002 inumber 22 on /var: filesystem full
Jan 22 09:43:08	kernel: pid 12591 (dhcpd), uid 1002 inumber 23 on /var: filesystem full
Jan 22 09:43:16	kernel: pid 12591 (dhcpd), uid 1002 inumber 22 on /var: filesystem full
Jan 22 09:43:31	kernel: pid 12591 (dhcpd), uid 1002 inumber 23 on /var: filesystem full
Jan 22 09:44:24	kernel: pid 12591 (dhcpd), uid 1002 inumber 22 on /var: filesystem full
Jan 22 09:48:53	kernel: pid 12591 (dhcpd), uid 1002 inumber 23 on /var: filesystem full
Jan 22 09:48:58	kernel: pid 12591 (dhcpd), uid 1002 inumber 22 on /var: filesystem full
Jan 22 09:49:06	kernel: pid 12591 (dhcpd), uid 1002 inumber 23 on /var: filesystem full
Jan 22 09:49:23	kernel: pid 12591 (dhcpd), uid 1002 inumber 22 on /var: filesystem full
Jan 22 09:54:38	kernel: pid 12591 (dhcpd), uid 1002 inumber 23 on /var: filesystem full
Jan 22 09:54:42	kernel: pid 12591 (dhcpd), uid 1002 inumber 22 on /var: filesystem full
Jan 22 09:54:49	kernel: pid 12591 (dhcpd), uid 1002 inumber 23 on /var: filesystem full
Jan 22 09:55:05	kernel: pid 12591 (dhcpd), uid 1002 inumber 22 on /var: filesystem full

 Jan 22 09:54:37	dhcpd: DHCPDISCOVER from 00:26:55:57:4b:14 (CGASWS9001) via em1
Jan 22 09:54:38	dhcpd: unexpected ICMP Echo Reply from 10.1.3.254
Jan 22 09:54:38	dhcpd: DHCPOFFER on 172.16.134.175 to 00:26:55:57:4b:14 (CGASWS9001) via em1
Jan 22 09:54:38	dhcpd: write_lease: unable to write lease 172.16.0.16
Jan 22 09:54:38	dhcpd: DHCPREQUEST for 172.16.134.175 (172.16.0.1) from 00:26:55:57:4b:14 (CGASWS9001) via em1: database update failed
Jan 22 09:54:42	dhcpd: write_lease: unable to write lease 172.16.0.16
Jan 22 09:54:42	dhcpd: DHCPREQUEST for 172.16.134.175 (172.16.0.1) from 00:26:55:57:4b:14 (CGASWS9001) via em1: database update failed
Jan 22 09:54:49	dhcpd: write_lease: unable to write lease 172.16.0.16
Jan 22 09:54:49	dhcpd: DHCPREQUEST for 172.16.134.175 (172.16.0.1) from 00:26:55:57:4b:14 (CGASWS9001) via em1: database update failed
Jan 22 09:55:05	dhcpd: write_lease: unable to write lease 172.16.0.16
Jan 22 09:55:05	dhcpd: DHCPREQUEST for 172.16.134.175 (172.16.0.1) from 00:26:55:57:4b:14 (CGASWS9001) via em1: database update failed

wallabybob

The /var file system where DHCP leases are being written is clearly full. On an embedded system /var is a "RAM disk".

Your a parameters appear to allow a large number of concurrent leases. This MIGHT be beyond the capacity of the embedded system unless you are prepared to tweak it to significantly increase the size of the RAM disk used for /var.

A number of system logs are also written to /var, so it not just DHCP leases causing the problem.

dhatz

You could try searching from the CLI for files larger than x, e.g. :

find /var -size +800k -print

fsantaana

i found out that it was the FR2 log daily log files that were filling up the space. I changed the logging form the radius.log files to system logs and this should resolve the issue. Also i increased the size of the /var filesystem in the rc.embedded to 700MB since i'm running 2 GB of RAM on the system.

I am going to replace the embedded image i have with a full pfsense image. i'm just waiting on the new industrial CF cards to arrive so i don't have to worry about my R/W limits. :)