Sustained Unbound write I/O
-
@keyser said in Sustained Unbound write I/O:
Is unbound the command with all the write IO’s after each screen refresh?
Its mostly unbound
-
@fireodo Well there will be some depending on the logging level you have activated (combined with the number and activitylevel of clients).
-
The problem seams more complex - look here: Average Disk writes
-
@fireodo Ahh, you’re running ZFS as well. That will in itself generate more because of the optimizations and “no modify of blocks” that ZFS uses.
But the Unbound activity is also a large contributor as I understood it. Try and disable pfBlockerNG-Devel - does Unbound then stop writing so much? If it does it’s probably the logging levels you have configured that causes the heavy writing.
-
@keyser said in Sustained Unbound write I/O:
Try and disable pfBlockerNG-Devel - does Unbound then stop writing so much?
I have done that - no significant change. Maybe that I use CE 2.5.2 and you the 21.05.1 is also playing a role ...
-
@fireodo Could be, or maybe some of your other packages are causing unbound to log a lot of activity as well.
-
@keyser said in Sustained Unbound write I/O:
maybe some of your other packages are causing unbound to log a lot of activity as well
Its only pfblockerNG that interacts with unbound - the other packages have nothing to do with it.
Kettop Mi4300YL CPU: i5-4300Y @ 1.60GHz RAM: 8GB Ethernet Ports: 4
SSD: SanDisk pSSD-S2 16GB (ZFS) WiFi: WLE200NX
pfsense 2.7.2 CE
Packages: Apcupsd Cron Iftop Iperf LCDproc Nmap pfBlockerNG RRD_Summary Shellcmd Snort Speedtest System_Patches. -
@fireodo Have you tried to stop unbound briefly and see if there still is a unbound proces writing to disk? Perhaps some deadlocked scripts running in a loop that does not stop/respond to changes? (While pfBlockerNG is also disabled)
Love the no fuss of using the official appliances :-)
-
@fireodo Maybe try a full reboot while pfBlocker is disabled
-
@keyser said in Sustained Unbound write I/O:
Have you tried to stop unbound briefly and see if there still is a unbound proces writing to disk?
Yes, I have stopped almost every stoppable process on the firewall - as stated in the other thread it seams that a process called
zpool-zroot{txg_thread_enter}
is doing very much writing (much more than unbound). I saw that when I use
top -SH -o write (and after that "m") -
@fireodo Got it.
ZFS can be a bit hard on SSD’s because of the way it handles disk writes and in particular existing block modify’s (which it doesn’t do - it allocates a new block to write the change, and then modifies the file block pointer).
That strategy makes A LOT of sense when using Raid and in particular when the filesystem supports snapshots. But it does come at an increased write IO penalty which impacts very small SSD’s.
-
@keyser I guess that will cause some trouble on little enclosures with build in eMMC ...
Kettop Mi4300YL CPU: i5-4300Y @ 1.60GHz RAM: 8GB Ethernet Ports: 4
SSD: SanDisk pSSD-S2 16GB (ZFS) WiFi: WLE200NX
pfsense 2.7.2 CE
Packages: Apcupsd Cron Iftop Iperf LCDproc Nmap pfBlockerNG RRD_Summary Shellcmd Snort Speedtest System_Patches. -
@fireodo Yep, which I’m sure is why Netgate does not deliver the desktop series SG boxes installed with ZFS :-)
-
@keyser Just as a follow up I think I have now tried EVERY possible setting in my pfBlockerNG setup in terms of making it “quiet” on disk writes.
Nothing sticks - No lists, no logging enabled anywhere, nothing - and still I have a UNBOUND process that writes on average about 380 Kb/s to disk if python mode is enabled.
If I disable Python mode or disable pfBlockerNG in general, UNBOUND no longer does it’s sustained disk writing.So what could cause that behavior?
-
@keyser I'm sorry that I'm ressurecting my old thread on this topic again, but I just installed 22.01 (ZFS reinstall) on my 6100 and that in turn updated pfBlockerNG to the latest version 3.1.0_1 version.
I'm sorry to report that has brought back the Unbound disk write issue (with the same config).
My box went from doing about 130KB/s writes to about 550KB/s now. About 30% of that comes curtesy of the ZFS filesystem, but still it's at least a tripeling of Unbound diskwrites...I'm beginning to wonder if Netgate considered the wear ZFS would cause on the small eMMC. appliances. If the write endurance rating is "industry standard" on the built-in eMMc, this level of write IO will kill my box within 2 years (And that's only because it's a 16Gb SG-6100).
The same config in my SG-2100 will kill the eMMC within a year.I think we have a huge problem here....
-
@keyser
did you find any solution?Same here, disable all I can find, unbound still writing.
-
@slu No, not specifically. It is related to doing python mode integration to unbound as you discovered.
In my case, I eventually removed pfBlockerNG without saving the config.
After reinstalling the latest 3.2.0.x and configuring it from scratch, the deadly writing levels never reappeared. So I sort of concluded it might have been crud from previous installs/upgrades that caused pfBlocker to behave like that. -
@keyser said in Sustained Unbound write I/O:
After reinstalling the latest 3.2.0.x and configuring it from scratch, the deadly writing levels never reappeared.
With python unbound mode or without the deadly writing never reappeared?
-
@slu said in Sustained Unbound write I/O:
With python unbound mode or without the deadly writing never reappeared?
The issue did not reappear with Python mode enabled - which is what you want as Python mode has some nice advantages.
-
@keyser said in Sustained Unbound write I/O:
The issue did not reappear with Python mode enabled [...]
That's very interesting, no idea who to find out what trigger this issue.
