dhcpd.log file is ~10GB, filling my disk up every couple days and taking my network down

ryanrozich

@ryanrozich Sounds like first order of business is to get log rotation turned on, if I could get instructions for that I'd be grateful.

Separately, does it seem strange that its been less than 24 hours since I manually truncated my log files yesterday and I am already back up to 7GB usage (of 13GB on my netgate 6100) from dhcpd? This is my home network - family of 5, I work from home - 3 kids with ipads and an array of iot devices (set top box, ring cameras, etc).

I have not intentionally turned off log rotation, so I'm not sure if it was ever on. Could this be some problem with one of my devices on the network? or some issue with my recent pfsense+ update?

jimp

I suspect there is a bug in there similar to https://redmine.pfsense.org/issues/14283 where it isn't doing rotation because the log settings are empty (all at defaults and the entire section is missing from the config.xml file).

If you make some non-default change to the log settings, such as changing the compression to 'none' (which is probably a good move anyhow), that should kick it in.

As to why DHCP is logging so much, that is very odd but it's hard to say why without knowing what is in the log. Something on your local network could be hitting the service hard even with very few devices. It just takes one broken thing going nuts to cause trouble.

Gertjan

@ryanrozich said in dhcpd.log file is ~10GB, filling my disk up every couple days and taking my network down:

and I am already back up to 7GB usage

Well .... don't keep us waiting : what is happening ?
Have a look at the last 300 ~ 500 lines of that file.

tail -n 500 /var/log/dhcpd.log

To follow it

tail -f /var/log/dhcpd.log

Hit Ctrl-C to end.

What kind of lines are added ?

My guess :
Some device uses a Wifi connection that is on the edge of 'out of reach' so every time the Wifi connections comes up again, dhpc is activated, and loads of DHCP transactions are logged.
Or, some 'made in ****' device has pretty broken dhcp client that chain guns the DHCP server.

Solution : wast-bin the offending device and done.

Major side effect : the DHCP log fills up fast.
If also issue 14283 (see Jimp's post above) is in play, the the file system fills up and that breaks everything.

jimp

Confirmed my suspicion and reproduced it in a lab system:

https://redmine.pfsense.org/issues/14517

I also pushed a fix ( 892de1ecdaa23b164f6b2a2251d7538eee2199ea ).

But the simplest workaround is to save on the syslog settings as I mentioned. No need to patch anything if you do that, it will just start to work immediately.

ryanrozich

@Gertjan

This is what the tail of that log looks like

ryanrozich

@jimp Thank You! I just truncated the logs again and then made the change from bzip to none compression. I'll keep an eye on it

Does this mean that in a future pfsense release there should be a fix?

johnpoz

@ryanrozich you have a problem child there - that box ending in b54f how many times is he going to ask for renew IP?

No wonder your logs are HUGE..

Looks from that log that all happened in 1 second?

jimp

@ryanrozich said in dhcpd.log file is ~10GB, filling my disk up every couple days and taking my network down:

@jimp Thank You! I just truncated the logs again and then made the change from bzip to none compression. I'll keep an eye on it

Does this mean that in a future pfsense release there should be a fix?

Yes, future releases will have the fix.

Gertjan

@ryanrozich said in dhcpd.log file is ~10GB, filling my disk up every couple days and taking my network down:

Does this mean that in a future pfsense release there should be a fix?

Why wait if you can have a good permanent solution right now ?
The patches come from the official source.

The first patch = https://github.com/pfsense/pfsense/commit/892de1ecdaa23b164f6b2a2251d7538eee2199ea.patch

Second patch = https://github.com/pfsense/pfsense/commit/77e168861ba43b3d6290df07fc04481c09174b28.patch

johnpoz

So fixing log rotation issues is only going to mask the issue of that one client constantly asking for renewal.

ryanrozich

@johnpoz agreed. I switched my printer from wired to wireless networking and that seemed to fix that issue.

However if this hadn’t taken down my home network I wouldn’t have known about it. Is there any alerting that I could enable in pfsense that would warn me of problems like this?

Gertjan

@ryanrozich said in dhcpd.log file is ~10GB, filling my disk up every couple days and taking my network down:

from wired to wireless networking

That's original, as normally, it's the wireless connection that has a very limited (bad) connection, so it get reconstructed again and again, and that introduces a DHCP sequence on every 'link up'.

If a wired connection does this : I'll bet you have a bad NIC on one side, or a bad cable.

Or the printer has a very bad DHCP client implementation, like : forcing the the DHCP lease duration to 10 seconds or so.

@ryanrozich said in dhcpd.log file is ~10GB, filling my disk up every couple days and taking my network down:

However if this hadn’t taken down my home network I wouldn’t have known about it. Is there any alerting that I could enable in pfsense that would warn me of problems like this?

That is actually the reason why pfSense is not some AI driven device that you power up, hook up and walk away. Like a switch.
pfSense needs the human type of admin, in this case : you. And 99 % of the time you won't be looking at the dashboard, but you're somewhere in the Status menu.
The most favorite one is all the log files.
And no, I'm not kidding

But I have a tip : when you add a 'new' device to your network, you should have a look at your log files (System, DHCP, DNS) a couple of times.
Things can always go bad, cable get cut, wifi gets destroyed by the new AP the neighbor bought (or the new micro wave that "works just fine with the door open").