Unbound python mode in combination with pfBlockerNG wrote more than 40 TB (!) (was: What wrote 47 TB (!) in two years? 2.7.2)
-
@keyser said in What wrote 47 TB (!) in two years? 2.7.2:
version and config of pfBlockerNG (obviously also the number of clients and traffic)
I guess you are right, unbound and with it pfBlockerNG 3.2.0_8 (?) is the problem.
Any hint to disable the logging?top -m io show:
Enable remote logging is enabled to external syslog server.
pfSense Gold subscription
-
Found it, as soon I switch from "unbound python mode" to "unbound mode" unbound stop permanent writing. I can confirm this on two different pfSense boxes with 2.7.2 CE.
As @keyser wrote, this is a huge problem.
https://forum.netgate.com/post/1027554 -
S slu referenced this topic on
-
@slu
32TB writes since 2019 and no, in my case it's not unbound in first place, enabled python mode since it was introduced in pfblockerng btw. -
@w0w
thanks for your reply, that's interesting.On my smaller network (only ~ 8 systems online) I see write throughput of approximately 10. But on my bigger network (screenshot above) its continuity around 1400 write throughput.
pfSense Gold subscription
-
@slu said in Unbound python mode in combination with pfBlockerNG wrote more than 40 TB (!) (was: What wrote 47 TB (!) in two years? 2.7.2):
Do SNORT write on the disk to check the data flow @bmeeks ?
No, Snort only writes what you see in the alert logs. It does no temporary writes other than when downloading and unpacking rules files updates. Those happen under
/tmpand are cleaned up when the rules update completes. Snort logs are under/var/log/snort/. -
@bmeeks said in Unbound python mode in combination with pfBlockerNG wrote more than 40 TB (!) (was: What wrote 47 TB (!) in two years? 2.7.2):
No, Snort only writes what you see in the alert logs.
Thank you @bmeeks that confirmed what I see in top -m io with the Snort process.
-
@slu
Long-term monitoring of disk writes still showed that a significant portion of them is performed unbound. I'm not sure what percentage of the records are unbound, but it is clear that it's substantial, though I don't know how to track it precisely. The Samsung SSD 860 PRO 256GB drives are in a ZFS mirror, and 88% of the resource remains, which is generally non-critical, but... -
@slu said in Unbound python mode in combination with pfBlockerNG wrote more than 40 TB (!) (was: What wrote 47 TB (!) in two years? 2.7.2):
Any hint to disable the logging?
Just checking :
= unbound log level setting : Right ?
Level 3 and above logs a lot, and is only useful for temporary debug sessions. Setting it back to '1' is not 'optional'.
Btw : logs files, also the /var/log/resolver.log file, are rotated by pfSense.
My pfBlockerng log files (most of them are here /var/unbound/var/log/pfblockerng) are also rotated.
I've never, over a decade now, saw a Tbytes file size on my pfSense ... -
@Gertjan said in Unbound python mode in combination with pfBlockerNG wrote more than 40 TB (!) (was: What wrote 47 TB (!) in two years? 2.7.2):
= unbound log level setting : Right ?
Yes setting is "Level 1", I tried also "Level 0" but unbound write anyway according to "top -m io".
Edit:
@Gertjan said in Unbound python mode in combination with pfBlockerNG wrote more than 40 TB (!) (was: What wrote 47 TB (!) in two years? 2.7.2):I've never, over a decade now, saw a Tbytes file size on my pfSense ...
I do not have Tbytes file/log size on my pfSense, only see the SSD write TB's...
-
@slu
I know this might not solve the root problem, but what about looking for a really long-lasting SSD? Maybe some of the older, reliable MLC or SLC variants? Which form factor do you have? -
@slu Or perhaps a RAM disk.
pfBlocker has some options that are on by default such as DNS Reply Logging to log all non-blocked queries (i.e. all valid DNS). Much like Suricata's HTTP request logging we disable that.
-
@SteveITS said in Unbound python mode in combination with pfBlockerNG wrote more than 40 TB (!) (was: What wrote 47 TB (!) in two years? 2.7.2):
pfBlocker has some options that are on by default such as DNS Reply Logging to log all non-blocked queries (i.e. all valid DNS).
I tried to disable this settings and reload pfBlockerNG + reboot pfSense, but unbound write and write again in python mode.
-
@slu I don't know, have not noticed high disk writes. I have seen posts over the years though...here are a couple.
https://www.reddit.com/r/pfBlockerNG/comments/13di9c2/dnsbl_python_mode_and_disk_writes/
https://forum.netgate.com/topic/165993/should-i-be-using-unbound-python-mode-is-it-stable -
-
@slu
The settings can affect it. I just checked how often it updates. The Cron settings are set to once a day. This is probably significantly reducing the number of writes. Maybe there are some other settings that are affecting it as well. -
New topic here to investigate the issue since pfBlockerNG is not the cause.
https://forum.netgate.com/topic/189820/how-do-i-find-out-what-write-continuously-on-my-pfsense-ssd
