Unbound python mode in combination with pfBlockerNG wrote more than 40 TB (!) (was: What wrote 47 TB (!) in two years? 2.7.2)
-
@slu Snort might be the obvious culprit, but let me assure you, depending on your version and config of pfBlockerNG (obviously also the number of clients and traffic), it can absolutely lay waste to SSD’s if you fx. Ask it to do replylogging and have a lot of blocklists with standard logging turned on.
I had a config where pfBlockerNG did around 400 KB/s sustained writes to my SSD 24/7 with just 50 clients on the network.
That will total about 13 TB writes a year from pfBlockerNG alone. -
@keyser
ok so I must check snort as next step.Available Spare: 100% Available Spare Threshold: 10% Percentage Used: 67% Data Units Read: 22,535 [11.5 GB] Data Units Written: 92,432,066 [47.3 TB] Host Read Commands: 4,378,418 Host Write Commands: 4,303,452,653 Controller Busy Time: 42,328 Power Cycles: 25 Power On Hours: 13,114 -
@slu said in What wrote 47 TB (!) in two years? 2.7.2:
SMART show total write of 47 TB (!) in two years, no wonder my SSDs are die.
The question is now, what write so much?Hi, if you dig in the Netgate Forum you'll find some threads concerning the huge amount of writing to SSDs/Disk ...
Regards,
fireodo -
@keyser said in What wrote 47 TB (!) in two years? 2.7.2:
version and config of pfBlockerNG (obviously also the number of clients and traffic)
I guess you are right, unbound and with it pfBlockerNG 3.2.0_8 (?) is the problem.
Any hint to disable the logging?top -m io show:
Enable remote logging is enabled to external syslog server.
pfSense Gold subscription
-
Found it, as soon I switch from "unbound python mode" to "unbound mode" unbound stop permanent writing. I can confirm this on two different pfSense boxes with 2.7.2 CE.
As @keyser wrote, this is a huge problem.
https://forum.netgate.com/post/1027554 -
S slu referenced this topic on
-
@slu
32TB writes since 2019 and no, in my case it's not unbound in first place, enabled python mode since it was introduced in pfblockerng btw. -
@w0w
thanks for your reply, that's interesting.On my smaller network (only ~ 8 systems online) I see write throughput of approximately 10. But on my bigger network (screenshot above) its continuity around 1400 write throughput.
pfSense Gold subscription
-
@slu said in Unbound python mode in combination with pfBlockerNG wrote more than 40 TB (!) (was: What wrote 47 TB (!) in two years? 2.7.2):
Do SNORT write on the disk to check the data flow @bmeeks ?
No, Snort only writes what you see in the alert logs. It does no temporary writes other than when downloading and unpacking rules files updates. Those happen under
/tmpand are cleaned up when the rules update completes. Snort logs are under/var/log/snort/. -
@bmeeks said in Unbound python mode in combination with pfBlockerNG wrote more than 40 TB (!) (was: What wrote 47 TB (!) in two years? 2.7.2):
No, Snort only writes what you see in the alert logs.
Thank you @bmeeks that confirmed what I see in top -m io with the Snort process.
-
@slu
Long-term monitoring of disk writes still showed that a significant portion of them is performed unbound. I'm not sure what percentage of the records are unbound, but it is clear that it's substantial, though I don't know how to track it precisely. The Samsung SSD 860 PRO 256GB drives are in a ZFS mirror, and 88% of the resource remains, which is generally non-critical, but... -
@slu said in Unbound python mode in combination with pfBlockerNG wrote more than 40 TB (!) (was: What wrote 47 TB (!) in two years? 2.7.2):
Any hint to disable the logging?
Just checking :
= unbound log level setting : Right ?
Level 3 and above logs a lot, and is only useful for temporary debug sessions. Setting it back to '1' is not 'optional'.
Btw : logs files, also the /var/log/resolver.log file, are rotated by pfSense.
My pfBlockerng log files (most of them are here /var/unbound/var/log/pfblockerng) are also rotated.
I've never, over a decade now, saw a Tbytes file size on my pfSense ... -
@Gertjan said in Unbound python mode in combination with pfBlockerNG wrote more than 40 TB (!) (was: What wrote 47 TB (!) in two years? 2.7.2):
= unbound log level setting : Right ?
Yes setting is "Level 1", I tried also "Level 0" but unbound write anyway according to "top -m io".
Edit:
@Gertjan said in Unbound python mode in combination with pfBlockerNG wrote more than 40 TB (!) (was: What wrote 47 TB (!) in two years? 2.7.2):I've never, over a decade now, saw a Tbytes file size on my pfSense ...
I do not have Tbytes file/log size on my pfSense, only see the SSD write TB's...
-
@slu
I know this might not solve the root problem, but what about looking for a really long-lasting SSD? Maybe some of the older, reliable MLC or SLC variants? Which form factor do you have? -
@slu Or perhaps a RAM disk.
pfBlocker has some options that are on by default such as DNS Reply Logging to log all non-blocked queries (i.e. all valid DNS). Much like Suricata's HTTP request logging we disable that.
-
@SteveITS said in Unbound python mode in combination with pfBlockerNG wrote more than 40 TB (!) (was: What wrote 47 TB (!) in two years? 2.7.2):
pfBlocker has some options that are on by default such as DNS Reply Logging to log all non-blocked queries (i.e. all valid DNS).
I tried to disable this settings and reload pfBlockerNG + reboot pfSense, but unbound write and write again in python mode.
-
@slu I don't know, have not noticed high disk writes. I have seen posts over the years though...here are a couple.
https://www.reddit.com/r/pfBlockerNG/comments/13di9c2/dnsbl_python_mode_and_disk_writes/
https://forum.netgate.com/topic/165993/should-i-be-using-unbound-python-mode-is-it-stable -
-
@slu
The settings can affect it. I just checked how often it updates. The Cron settings are set to once a day. This is probably significantly reducing the number of writes. Maybe there are some other settings that are affecting it as well. -
New topic here to investigate the issue since pfBlockerNG is not the cause.
https://forum.netgate.com/topic/189820/how-do-i-find-out-what-write-continuously-on-my-pfsense-ssd
