Unbound python mode in combination with pfBlockerNG wrote more than 40 TB (!) (was: What wrote 47 TB (!) in two years? 2.7.2)

[slu]

Hi,

since my third SSD died in different pfSense setups we check the SMART status of an two year old pfSense system with the following packages installed:

arpwatch
Cron
freeradius3
nmap
openvpn-client-export
pfBlockerNG
Service_Watchdog
snort
Status_Traffic_Totals
System_Patches
zabbix-agent6

SMART show total write of 47 TB (!) in two years, no wonder my SSDs are die.
The question is now, what write so much?

Do SNORT write on the disk to check the data flow @bmeeks ?

Edit: Modify topic subject.

[keyser]

@slu Snort might be the obvious culprit, but let me assure you, depending on your version and config of pfBlockerNG (obviously also the number of clients and traffic), it can absolutely lay waste to SSD’s if you fx. Ask it to do replylogging and have a lot of blocklists with standard logging turned on.
I had a config where pfBlockerNG did around 400 KB/s sustained writes to my SSD 24/7 with just 50 clients on the network.
That will total about 13 TB writes a year from pfBlockerNG alone.

[slu]

@keyser
ok so I must check snort as next step.

Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    67%
Data Units Read:                    22,535 [11.5 GB]
Data Units Written:                 92,432,066 [47.3 TB]
Host Read Commands:                 4,378,418
Host Write Commands:                4,303,452,653
Controller Busy Time:               42,328
Power Cycles:                       25
Power On Hours:                     13,114

[fireodo]

@slu said in What wrote 47 TB (!) in two years? 2.7.2:

SMART show total write of 47 TB (!) in two years, no wonder my SSDs are die.
The question is now, what write so much?

Hi, if you dig in the Netgate Forum you'll find some threads concerning the huge amount of writing to SSDs/Disk ...

Regards,
fireodo

[slu]

@keyser said in What wrote 47 TB (!) in two years? 2.7.2:

version and config of pfBlockerNG (obviously also the number of clients and traffic)

I guess you are right, unbound and with it pfBlockerNG 3.2.0_8 (?) is the problem.
Any hint to disable the logging?

top -m io show:

Enable remote logging is enabled to external syslog server.

[slu]

Found it, as soon I switch from "unbound python mode" to "unbound mode" unbound stop permanent writing. I can confirm this on two different pfSense boxes with 2.7.2 CE.

As @keyser wrote, this is a huge problem.
https://forum.netgate.com/post/1027554

[w0w]

@slu
32TB writes since 2019 and no, in my case it's not unbound in first place, enabled python mode since it was introduced in pfblockerng btw.

[slu]

@w0w
thanks for your reply, that's interesting.

On my smaller network (only ~ 8 systems online) I see write throughput of approximately 10. But on my bigger network (screenshot above) its continuity around 1400 write throughput.

[bmeeks]

@slu said in Unbound python mode in combination with pfBlockerNG wrote more than 40 TB (!) (was: What wrote 47 TB (!) in two years? 2.7.2):

Do SNORT write on the disk to check the data flow @bmeeks ?

No, Snort only writes what you see in the alert logs. It does no temporary writes other than when downloading and unpacking rules files updates. Those happen under /tmp and are cleaned up when the rules update completes. Snort logs are under /var/log/snort/.

[slu]

@bmeeks said in Unbound python mode in combination with pfBlockerNG wrote more than 40 TB (!) (was: What wrote 47 TB (!) in two years? 2.7.2):

No, Snort only writes what you see in the alert logs.

Thank you @bmeeks that confirmed what I see in top -m io with the Snort process.

[w0w]

@slu
Long-term monitoring of disk writes still showed that a significant portion of them is performed unbound. I'm not sure what percentage of the records are unbound, but it is clear that it's substantial, though I don't know how to track it precisely. The Samsung SSD 860 PRO 256GB drives are in a ZFS mirror, and 88% of the resource remains, which is generally non-critical, but...

[Gertjan]

@slu said in Unbound python mode in combination with pfBlockerNG wrote more than 40 TB (!) (was: What wrote 47 TB (!) in two years? 2.7.2):

Any hint to disable the logging?

Just checking :

= unbound log level setting : Right ?

Level 3 and above logs a lot, and is only useful for temporary debug sessions. Setting it back to '1' is not 'optional'.

Btw : logs files, also the /var/log/resolver.log file, are rotated by pfSense.
My pfBlockerng log files (most of them are here /var/unbound/var/log/pfblockerng) are also rotated.
I've never, over a decade now, saw a Tbytes file size on my pfSense ...

[slu]

@Gertjan said in Unbound python mode in combination with pfBlockerNG wrote more than 40 TB (!) (was: What wrote 47 TB (!) in two years? 2.7.2):

= unbound log level setting : Right ?

Yes setting is "Level 1", I tried also "Level 0" but unbound write anyway according to "top -m io".

Edit:
@Gertjan said in Unbound python mode in combination with pfBlockerNG wrote more than 40 TB (!) (was: What wrote 47 TB (!) in two years? 2.7.2):

I've never, over a decade now, saw a Tbytes file size on my pfSense ...

I do not have Tbytes file/log size on my pfSense, only see the SSD write TB's...

[w0w]

@slu
I know this might not solve the root problem, but what about looking for a really long-lasting SSD? Maybe some of the older, reliable MLC or SLC variants? Which form factor do you have?

[SteveITS]

@slu Or perhaps a RAM disk.

pfBlocker has some options that are on by default such as DNS Reply Logging to log all non-blocked queries (i.e. all valid DNS). Much like Suricata's HTTP request logging we disable that.

[slu]

@SteveITS said in Unbound python mode in combination with pfBlockerNG wrote more than 40 TB (!) (was: What wrote 47 TB (!) in two years? 2.7.2):

pfBlocker has some options that are on by default such as DNS Reply Logging to log all non-blocked queries (i.e. all valid DNS).

I tried to disable this settings and reload pfBlockerNG + reboot pfSense, but unbound write and write again in python mode.

[SteveITS]

@slu I don't know, have not noticed high disk writes. I have seen posts over the years though...here are a couple.

https://www.reddit.com/r/pfBlockerNG/comments/13di9c2/dnsbl_python_mode_and_disk_writes/
https://forum.netgate.com/topic/165993/should-i-be-using-unbound-python-mode-is-it-stable

[slu]

@SteveITS
thanks @SteveITS for the links.

Still not sure whats the root cause for this massive writes on the SSD, maybe I'm on the wrong way with pfBlockerNG and Unbound Python mode...

[w0w]

@slu
The settings can affect it. I just checked how often it updates. The Cron settings are set to once a day. This is probably significantly reducing the number of writes. Maybe there are some other settings that are affecting it as well.

[slu]

New topic here to investigate the issue since pfBlockerNG is not the cause.
https://forum.netgate.com/topic/189820/how-do-i-find-out-what-write-continuously-on-my-pfsense-ssd