dhcpd.log file is ~10GB, filling my disk up every couple days and taking my network down

ryanrozich

Hi there, my network went down this weekend and it took some troubleshooting to figure out it was because my disk was full

and the culprit was dhcpd.log

I truncated the log files

and everything was back to normal, less than two days later my network is down again for the same reason. So I truncated again, but it seems like in the past week something has caused these logs to grow out of control (I thought pfsense rotated logs by default - I haven't changed any default settings)

In the last week (I cant remember the day) I did upgrade pfSense software to this most recent version

Not sure if these problems are related to this upgrade or not - but prior to this > 1yr with this device - I've never had a problem with disk filling up.

Can anyone help? I dont want to have to manually truncate my logs every day. I dont ever check the dhcpd.log files so if needed I can just turn off dhcpd logging if someone can point me how to do that. Can anyone help?

jimp

Did you maybe disable log rotation somehow? It's enabled by default and would normally prevent that from happening.

Look in /etc/crontab and see if you see a line like this:

*/1     *       *       *       *       root    /usr/sbin/newsyslog

Now look at the configuration for newsyslog in /var/etc/newsyslog.conf.d/pfSense.conf:

It should have a line for every log file, including the DHCP log file, for example:

/var/log/dhcpd.log              root:wheel      600     7       500     *       C

Then look at Status > System Logs, Settings tab and check the Log Rotation Options section. What exactly is set in the three options in that section?

ryanrozich

looks like this file does not exist /var/etc/newsyslog.conf.d/pfSense.conf - so looks like you are right. However I dont know how to restore that file. Replies below:

@jimp said in dhcpd.log file is ~10GB, filling my disk up every couple days and taking my network down:

Did you maybe disable log rotation somehow? It's enabled by default and would normally prevent that from happening.

Look in /etc/crontab and see if you see a line like this:

*/1 * * * * root /usr/sbin/newsyslog

yes, thats there

Now look at the configuration for newsyslog in /var/etc/newsyslog.conf.d/pfSense.conf:
It should have a line for every log file, including the DHCP log file, for example:

/var/log/dhcpd.log root:wheel 600 7 500 * C

That file does not exist, that directory is empty - see below:

[23.05-RELEASE][admin@pfSense.rozich.com]/root: ls /var/etc/newsyslog.conf.d/
[23.05-RELEASE][admin@pfSense.rozich.com]/root:

Then look at Status > System Logs, Settings tab and check the Log Rotation Options section. What exactly is set in the three options in that section?

See below - note that the values in the first and third boxes are in grey (probably indicating the default values for those fields)

ryanrozich

@ryanrozich Sounds like first order of business is to get log rotation turned on, if I could get instructions for that I'd be grateful.

Separately, does it seem strange that its been less than 24 hours since I manually truncated my log files yesterday and I am already back up to 7GB usage (of 13GB on my netgate 6100) from dhcpd? This is my home network - family of 5, I work from home - 3 kids with ipads and an array of iot devices (set top box, ring cameras, etc).

I have not intentionally turned off log rotation, so I'm not sure if it was ever on. Could this be some problem with one of my devices on the network? or some issue with my recent pfsense+ update?

jimp

I suspect there is a bug in there similar to https://redmine.pfsense.org/issues/14283 where it isn't doing rotation because the log settings are empty (all at defaults and the entire section is missing from the config.xml file).

If you make some non-default change to the log settings, such as changing the compression to 'none' (which is probably a good move anyhow), that should kick it in.

As to why DHCP is logging so much, that is very odd but it's hard to say why without knowing what is in the log. Something on your local network could be hitting the service hard even with very few devices. It just takes one broken thing going nuts to cause trouble.

Gertjan

@ryanrozich said in dhcpd.log file is ~10GB, filling my disk up every couple days and taking my network down:

and I am already back up to 7GB usage

Well .... don't keep us waiting : what is happening ?
Have a look at the last 300 ~ 500 lines of that file.

tail -n 500 /var/log/dhcpd.log

To follow it

tail -f /var/log/dhcpd.log

Hit Ctrl-C to end.

What kind of lines are added ?

My guess :
Some device uses a Wifi connection that is on the edge of 'out of reach' so every time the Wifi connections comes up again, dhpc is activated, and loads of DHCP transactions are logged.
Or, some 'made in ****' device has pretty broken dhcp client that chain guns the DHCP server.

Solution : wast-bin the offending device and done.

Major side effect : the DHCP log fills up fast.
If also issue 14283 (see Jimp's post above) is in play, the the file system fills up and that breaks everything.

jimp

Confirmed my suspicion and reproduced it in a lab system:

https://redmine.pfsense.org/issues/14517

I also pushed a fix ( 892de1ecdaa23b164f6b2a2251d7538eee2199ea ).

But the simplest workaround is to save on the syslog settings as I mentioned. No need to patch anything if you do that, it will just start to work immediately.

ryanrozich

@Gertjan

This is what the tail of that log looks like

ryanrozich

@jimp Thank You! I just truncated the logs again and then made the change from bzip to none compression. I'll keep an eye on it

Does this mean that in a future pfsense release there should be a fix?

johnpoz

@ryanrozich you have a problem child there - that box ending in b54f how many times is he going to ask for renew IP?

No wonder your logs are HUGE..

Looks from that log that all happened in 1 second?

jimp

@ryanrozich said in dhcpd.log file is ~10GB, filling my disk up every couple days and taking my network down:

@jimp Thank You! I just truncated the logs again and then made the change from bzip to none compression. I'll keep an eye on it

Does this mean that in a future pfsense release there should be a fix?

Yes, future releases will have the fix.

Gertjan

@ryanrozich said in dhcpd.log file is ~10GB, filling my disk up every couple days and taking my network down:

Does this mean that in a future pfsense release there should be a fix?

Why wait if you can have a good permanent solution right now ?
The patches come from the official source.

The first patch = https://github.com/pfsense/pfsense/commit/892de1ecdaa23b164f6b2a2251d7538eee2199ea.patch

Second patch = https://github.com/pfsense/pfsense/commit/77e168861ba43b3d6290df07fc04481c09174b28.patch

johnpoz

So fixing log rotation issues is only going to mask the issue of that one client constantly asking for renewal.

ryanrozich

@johnpoz agreed. I switched my printer from wired to wireless networking and that seemed to fix that issue.

However if this hadn’t taken down my home network I wouldn’t have known about it. Is there any alerting that I could enable in pfsense that would warn me of problems like this?

Gertjan

@ryanrozich said in dhcpd.log file is ~10GB, filling my disk up every couple days and taking my network down:

from wired to wireless networking

That's original, as normally, it's the wireless connection that has a very limited (bad) connection, so it get reconstructed again and again, and that introduces a DHCP sequence on every 'link up'.

If a wired connection does this : I'll bet you have a bad NIC on one side, or a bad cable.

Or the printer has a very bad DHCP client implementation, like : forcing the the DHCP lease duration to 10 seconds or so.

@ryanrozich said in dhcpd.log file is ~10GB, filling my disk up every couple days and taking my network down:

However if this hadn’t taken down my home network I wouldn’t have known about it. Is there any alerting that I could enable in pfsense that would warn me of problems like this?

That is actually the reason why pfSense is not some AI driven device that you power up, hook up and walk away. Like a switch.
pfSense needs the human type of admin, in this case : you. And 99 % of the time you won't be looking at the dashboard, but you're somewhere in the Status menu.
The most favorite one is all the log files.
And no, I'm not kidding

But I have a tip : when you add a 'new' device to your network, you should have a look at your log files (System, DHCP, DNS) a couple of times.
Things can always go bad, cable get cut, wifi gets destroyed by the new AP the neighbor bought (or the new micro wave that "works just fine with the door open").