Sustained Unbound write I/O

fireodo

@keyser said in Sustained Unbound write I/O:

Try doing “iostat -d 5 6”

iostat -d 5 6

This is the output after switching to unbound mode:
md0 ada0 pass0
KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s
0.00 0 0.00 15.54 14 0.21 0.38 0 0.00
0.00 0 0.00 16.06 13 0.21 0.00 0 0.00
0.00 0 0.00 14.90 15 0.21 0.00 0 0.00
0.00 0 0.00 17.24 14 0.23 0.00 0 0.00
0.00 0 0.00 13.61 15 0.20 0.00 0 0.00
0.00 0 0.00 27.76 20 0.55 0.00 0 0.00
and this is the output in python mode:
md0 ada0 pass0
KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s
0.00 0 0.00 15.55 14 0.21 0.38 0 0.00
0.00 0 0.00 12.77 15 0.18 0.00 0 0.00
0.00 0 0.00 16.11 14 0.23 0.00 0 0.00
0.00 0 0.00 14.23 15 0.21 0.00 0 0.00
0.00 0 0.00 15.54 14 0.21 0.00 0 0.00
0.00 0 0.00 12.58 18 0.22 0.00 0 0.00

keyser

@fireodo said in Sustained Unbound write I/O:

s
0.00 0 0.00 15.54 14 0.21 0.38 0 0.00
0.00 0 0.00 16.06 13 0.21 0.00 0 0.00
0.00 0 0.00 14.90 15 0.21 0.00 0 0.00
0.00 0 0.00 17.24 14 0.23 0.00 0 0.00

Ohh, yes that is different than mine. Your writing continues.
Is it still done by unbound? (Try: “top -m io”)

Is unbound the command with all the write IO’s after each screen refresh?

fireodo

@keyser said in Sustained Unbound write I/O:

Is unbound the command with all the write IO’s after each screen refresh?

Its mostly unbound

keyser

@fireodo Well there will be some depending on the logging level you have activated (combined with the number and activitylevel of clients).

fireodo

@keyser

The problem seams more complex - look here: Average Disk writes

keyser

@fireodo Ahh, you’re running ZFS as well. That will in itself generate more because of the optimizations and “no modify of blocks” that ZFS uses.

But the Unbound activity is also a large contributor as I understood it. Try and disable pfBlockerNG-Devel - does Unbound then stop writing so much? If it does it’s probably the logging levels you have configured that causes the heavy writing.

fireodo

@keyser said in Sustained Unbound write I/O:

Try and disable pfBlockerNG-Devel - does Unbound then stop writing so much?

I have done that - no significant change. Maybe that I use CE 2.5.2 and you the 21.05.1 is also playing a role ...

keyser

@fireodo Could be, or maybe some of your other packages are causing unbound to log a lot of activity as well.

fireodo

@keyser said in Sustained Unbound write I/O:

maybe some of your other packages are causing unbound to log a lot of activity as well

Its only pfblockerNG that interacts with unbound - the other packages have nothing to do with it.

keyser

@fireodo Have you tried to stop unbound briefly and see if there still is a unbound proces writing to disk? Perhaps some deadlocked scripts running in a loop that does not stop/respond to changes? (While pfBlockerNG is also disabled)

keyser

@fireodo Maybe try a full reboot while pfBlocker is disabled

fireodo

@keyser said in Sustained Unbound write I/O:

Have you tried to stop unbound briefly and see if there still is a unbound proces writing to disk?

Yes, I have stopped almost every stoppable process on the firewall - as stated in the other thread it seams that a process called

zpool-zroot{txg_thread_enter}

is doing very much writing (much more than unbound). I saw that when I use
top -SH -o write (and after that "m")

keyser

@fireodo Got it.

ZFS can be a bit hard on SSD’s because of the way it handles disk writes and in particular existing block modify’s (which it doesn’t do - it allocates a new block to write the change, and then modifies the file block pointer).

That strategy makes A LOT of sense when using Raid and in particular when the filesystem supports snapshots. But it does come at an increased write IO penalty which impacts very small SSD’s.

fireodo

@keyser I guess that will cause some trouble on little enclosures with build in eMMC ...

keyser

@fireodo Yep, which I’m sure is why Netgate does not deliver the desktop series SG boxes installed with ZFS :-)

keyser

@keyser Just as a follow up I think I have now tried EVERY possible setting in my pfBlockerNG setup in terms of making it “quiet” on disk writes.
Nothing sticks - No lists, no logging enabled anywhere, nothing - and still I have a UNBOUND process that writes on average about 380 Kb/s to disk if python mode is enabled.
If I disable Python mode or disable pfBlockerNG in general, UNBOUND no longer does it’s sustained disk writing.

So what could cause that behavior?

keyser

@keyser I'm sorry that I'm ressurecting my old thread on this topic again, but I just installed 22.01 (ZFS reinstall) on my 6100 and that in turn updated pfBlockerNG to the latest version 3.1.0_1 version.

I'm sorry to report that has brought back the Unbound disk write issue (with the same config).
My box went from doing about 130KB/s writes to about 550KB/s now. About 30% of that comes curtesy of the ZFS filesystem, but still it's at least a tripeling of Unbound diskwrites...

I'm beginning to wonder if Netgate considered the wear ZFS would cause on the small eMMC. appliances. If the write endurance rating is "industry standard" on the built-in eMMc, this level of write IO will kill my box within 2 years (And that's only because it's a 16Gb SG-6100).
The same config in my SG-2100 will kill the eMMC within a year.

I think we have a huge problem here....

slu

@keyser
did you find any solution?

Same here, disable all I can find, unbound still writing.

keyser

@slu No, not specifically. It is related to doing python mode integration to unbound as you discovered.
In my case, I eventually removed pfBlockerNG without saving the config.
After reinstalling the latest 3.2.0.x and configuring it from scratch, the deadly writing levels never reappeared. So I sort of concluded it might have been crud from previous installs/upgrades that caused pfBlocker to behave like that.

slu

@keyser said in Sustained Unbound write I/O:

After reinstalling the latest 3.2.0.x and configuring it from scratch, the deadly writing levels never reappeared.

With python unbound mode or without the deadly writing never reappeared?