Sustained Unbound write I/O

keyser

@fireodo Ahh, you’re running ZFS as well. That will in itself generate more because of the optimizations and “no modify of blocks” that ZFS uses.

But the Unbound activity is also a large contributor as I understood it. Try and disable pfBlockerNG-Devel - does Unbound then stop writing so much? If it does it’s probably the logging levels you have configured that causes the heavy writing.

fireodo

@keyser said in Sustained Unbound write I/O:

Try and disable pfBlockerNG-Devel - does Unbound then stop writing so much?

I have done that - no significant change. Maybe that I use CE 2.5.2 and you the 21.05.1 is also playing a role ...

keyser

@fireodo Could be, or maybe some of your other packages are causing unbound to log a lot of activity as well.

fireodo

@keyser said in Sustained Unbound write I/O:

maybe some of your other packages are causing unbound to log a lot of activity as well

Its only pfblockerNG that interacts with unbound - the other packages have nothing to do with it.

keyser

@fireodo Have you tried to stop unbound briefly and see if there still is a unbound proces writing to disk? Perhaps some deadlocked scripts running in a loop that does not stop/respond to changes? (While pfBlockerNG is also disabled)

keyser

@fireodo Maybe try a full reboot while pfBlocker is disabled

fireodo

@keyser said in Sustained Unbound write I/O:

Have you tried to stop unbound briefly and see if there still is a unbound proces writing to disk?

Yes, I have stopped almost every stoppable process on the firewall - as stated in the other thread it seams that a process called

zpool-zroot{txg_thread_enter}

is doing very much writing (much more than unbound). I saw that when I use
top -SH -o write (and after that "m")

keyser

@fireodo Got it.

ZFS can be a bit hard on SSD’s because of the way it handles disk writes and in particular existing block modify’s (which it doesn’t do - it allocates a new block to write the change, and then modifies the file block pointer).

That strategy makes A LOT of sense when using Raid and in particular when the filesystem supports snapshots. But it does come at an increased write IO penalty which impacts very small SSD’s.

fireodo

@keyser I guess that will cause some trouble on little enclosures with build in eMMC ...

keyser

@fireodo Yep, which I’m sure is why Netgate does not deliver the desktop series SG boxes installed with ZFS :-)

keyser

@keyser Just as a follow up I think I have now tried EVERY possible setting in my pfBlockerNG setup in terms of making it “quiet” on disk writes.
Nothing sticks - No lists, no logging enabled anywhere, nothing - and still I have a UNBOUND process that writes on average about 380 Kb/s to disk if python mode is enabled.
If I disable Python mode or disable pfBlockerNG in general, UNBOUND no longer does it’s sustained disk writing.

So what could cause that behavior?

keyser

@keyser I'm sorry that I'm ressurecting my old thread on this topic again, but I just installed 22.01 (ZFS reinstall) on my 6100 and that in turn updated pfBlockerNG to the latest version 3.1.0_1 version.

I'm sorry to report that has brought back the Unbound disk write issue (with the same config).
My box went from doing about 130KB/s writes to about 550KB/s now. About 30% of that comes curtesy of the ZFS filesystem, but still it's at least a tripeling of Unbound diskwrites...

I'm beginning to wonder if Netgate considered the wear ZFS would cause on the small eMMC. appliances. If the write endurance rating is "industry standard" on the built-in eMMc, this level of write IO will kill my box within 2 years (And that's only because it's a 16Gb SG-6100).
The same config in my SG-2100 will kill the eMMC within a year.

I think we have a huge problem here....

slu

@keyser
did you find any solution?

Same here, disable all I can find, unbound still writing.

keyser

@slu No, not specifically. It is related to doing python mode integration to unbound as you discovered.
In my case, I eventually removed pfBlockerNG without saving the config.
After reinstalling the latest 3.2.0.x and configuring it from scratch, the deadly writing levels never reappeared. So I sort of concluded it might have been crud from previous installs/upgrades that caused pfBlocker to behave like that.

slu

@keyser said in Sustained Unbound write I/O:

After reinstalling the latest 3.2.0.x and configuring it from scratch, the deadly writing levels never reappeared.

With python unbound mode or without the deadly writing never reappeared?

keyser

@slu said in Sustained Unbound write I/O:

With python unbound mode or without the deadly writing never reappeared?

The issue did not reappear with Python mode enabled - which is what you want as Python mode has some nice advantages.

slu

@keyser said in Sustained Unbound write I/O:

The issue did not reappear with Python mode enabled [...]

That's very interesting, no idea who to find out what trigger this issue.

keyser

@slu Not that I could find… I sort of concluded that somehow the python script had ended up doing all of its temporary data handling on actual disk rather than in a memory based datastructure. I have nothing to base that conclusion on as I painstaikingly tried to determine what file(s) all the writing was done into. But there was never any files that changed or grew during this - neither did pfSense Swap… so….

slu

@keyser
not easy to debug.

@NOCling you use python mode as well, right?
Do you see this as well?

NOCling

Yes, i use it.
But i run a RAM Disk to prevent Unbound killing my SSD.
I use a UPS and NUT on the pfsense, so no unexpected Power loss and the RAM Disk is safely written to the SSD.