Disk usage slowly increasing

keyser

I can confirm that changing pfBlockerNG 3.0.16 from “Unbound Python mode” to “Unbound mode” removes the continious write I/O from unbound to disk. Everything still works as expected, but there is much less write I/O done on the system.

The memory usage on my SG-2100 also went down from 15% to 10% by making this change.

I will report back tomorrow to report if the disk filling issue really is stopped by using this workaround.
I will also report on whether the memory use stays permanently lower.

keyser

I can now also confirm the filling filesystem issue is gone once pfBlockerNG is changed to "Unbound Mode" instead of python mode.

So this will serve as workaround until the issue with Python mode filling the filesystem is solved:

NOTE: It seems my pfBlockerNG stopped logging DNSBL hits once I changed to Unbound mode.
The counters in the widget no longer increases, and no hits are registered in the DNSBL report.
But DNSBL is still active and working.

Gertjan

@keyser said in Disk usage slowly increasing:

NOTE: It seems my pfBlockerNG stopped logging DNSBL hits once I changed to Unbound mode.
The counters in the widget no longer increases, and no hits are registered in the DNSBL report.
But DNSBL is still active and working.

That's one of the reasons the 'python mode' exist : it can interact with the DNS traffic, and log.

Btw : the pyhon module logs here : /var/log/pfblockerng/

Their max size (or lines size) can be set with :

keyser

@gertjan Yes I know, but then something is wrong in the script that does the logging. Because the filesystem fills until pfSense crashes regardless of the limit set there. And the log file size does not show it uses several Gb of storage.

Gertjan

These are my files :

-rw-------  1 unbound  unbound  14534320 Jul  2 15:10 dns_reply.log
-rw-------  1 unbound  unbound   2713595 Jul  2 15:10 dnsbl.log
-rw-r--r--  1 root     wheel           0 Dec  9  2020 dnsbl_error.log
-rw-------  1 root     wheel       53752 Jul  2 12:00 dnsbl_parsed_error.log
-rw-------  1 root     wheel       19383 Jul  2 12:00 extras.log
-rw-------  1 root     wheel     3325212 Jul  2 15:10 ip_block.log
-rw-r--r--  1 root     wheel         119 Jun 10 02:00 maxmind_ver
-rw-------  1 root     wheel     1728760 Jul  2 12:16 pfblockerng.log
-rw-r--r--  1 unbound  unbound         0 May 21 12:20 py_error.log
-rw-------  1 unbound  unbound  14606143 Jul  2 15:10 unified.log

The number of lines in the file "dns_reply.log" is

wc -l dns_reply.log
157894 dns_reply.log

That's not "20 000" but "157 894" (lines !)

Changing from 20 000 to 40 000 didn't change the file size.

But a force reload of pfBlokcerng-devel does the job.

Now I see this :

wc -l dns_reply.log
40000 dns_reply.log

Did you lose your disk space with the files in /var/log/pfblockerng ?
Or else where ?
Other packages installed ?

keyser

@gertjan Interesting…

I just noticed that pfBlockerNG - when in unbound python mode - creates several extra filesystems that are mounted different places under /var

When in regular - unbound mode - those filesystems are not there, and the log is just placed in /var/log/pfBlockerNG in the root filesystem.

Anyhow, your dns_reply file does report a “big” size, so you can see that’s the “culprit” that’s using up filesystem space. I could not locate such a file that used up all the space.

But I’m a *nix/bsd novice so this may be a dumb question:
Does a “du” command only parse the current filesystem you are in - and not others mounted in subfolders? Because then I might have missed big log files since they were located in one of the new filesystems.

No files reported that kind of diskspace usage to “du -h” on my system when doing it in the “/“ filesystem.

anthonys

I have an SG-1100, upgraded from 2.4.1p1 to 21.05 about 2 weeks ago. Only extension is pfBlocker-NG-devel, and changed it from unbound to python mode at the same time as the upgrade.

Same symptom here, since above changes, disk usage slowly increasing ~3% per day. Rebooted 2 days ago, mainly because RAM usage had also climbed to 92%.

Have started collecting more logging to try to narrow down the problem/s, and will do so for a few more days, then switch back to unbound mode to compare.

keyser

@anthonys Thank you for reporting back. Kind of nice to know it’s a more widespread issue across different hardware.

Makes me wonder if this is a general issue with 21.05 and pfB-NG 3.0.0.16 and people just haven’t upgraded yet since no more reports are coming in?

anthonys

@keyser an interim update.
After reboot, disk usage was 22%, after running 7 days 28%,
then executed the command

du -s /var/unbound

which returned the disk reported disk usage back to 22%.
Continuing work to locate which sub-directory in /var/unbound
But given that executing du repairs disk usage (df -hi /), then my first reaction is a kernel bug?

anthonys

Quick update, have narrowed down to /var/unbound/usr/local/lib/python3.8

Shell Output - df -hi /;du -s /var/unbound/usr/local/lib/python3.8; df -hi /
Filesystem                     Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/ufsid/5cdd3c209fe36da7    7.0G    2.0G    4.5G    31%     27k  935k    3%   /
274800	/var/unbound/usr/local/lib/python3.8
Filesystem                     Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/ufsid/5cdd3c209fe36da7    7.0G    1.4G    5.1G    22%     27k  935k    3%   /