How do I find out what write continuously on my pfSense SSD

slu

My SSD show very high write values for an two year old pfSense CE system.

Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    67%
Data Units Read:                    22,535 [11.5 GB]
Data Units Written:                 92,432,066 [47.3 TB]
Host Read Commands:                 4,378,418
Host Write Commands:                4,303,452,653
Controller Busy Time:               42,328
Power Cycles:                       25
Power On Hours:                     13,114

-- 176 hours --

Available Spare:                    100%
Available Spare Threshold:          10%
Percentage Used:                    68%
Data Units Read:                    23,964 [12.2 GB]
Data Units Written:                 92,948,526 [47.5 TB]
Host Read Commands:                 4,405,841
Host Write Commands:                4,322,349,019
Controller Busy Time:               42,394
Power Cycles:                       25
Power On Hours:                     13,290

176 hour from 47.3TB to 47.5TB = 200GB
200GB / 176h = 1,13GB/h
27,27GB day

iostat -d 5 6
            nda0            pass0            pass1            pass2 
 KB/t  tps  MB/s  KB/t  tps  MB/s  KB/t  tps  MB/s  KB/t  tps  MB/s 
 21.0   26   0.5   0.0    0   0.0   0.0    0   0.0   0.0    0   0.0 
  8.3   40   0.3   0.0    0   0.0   0.0    0   0.0   0.0    0   0.0 
  9.9   27   0.3   0.0    0   0.0   0.0    0   0.0   0.0    0   0.0 
  8.3   33   0.3   0.0    0   0.0   0.0    0   0.0   0.0    0   0.0 
 2386    1   2.3   0.0    0   0.0   0.0    0   0.0   0.0    0   0.0 
  9.6   31   0.3   0.0    0   0.0   0.0    0   0.0   0.0    0   0.0

The first problem I found was pfBlockerNG with Unbound Python Mode, for some reasons it wrote continuously. Switched over to Unbound Mode stop this, but that was not the cause.

top -m io show some values direct at the start and than values (mostly) of 0.
top -m io -o write -s 1 show php-fpm often on the top, wo can I see what php-fpm wrote?

I learned zfs write some data for sync, but can this so much?
How I do find what write so much data on pfSense/FreeBSD?

stephenw10

Try using top with more shown. Maybe: top -aSHm io

What does it actually show?

slu

@stephenw10 said in How do I find out what write continuously on my pfSense SSD:

Maybe: top -aSHm io

top -aSHm io -o write

Shows alternately:

PID USERNAME     VCSW  IVCSW   READ  WRITE  FAULT  TOTAL PERCENT COMMAND
88772 root          10      0      0     10      0     10 100.00% /usr/local/bin/php -f /usr/local/pkg/pfblockerng/pfblockerng.inc queries

and

PID USERNAME     VCSW  IVCSW   READ  WRITE  FAULT  TOTAL PERCENT COMMAND
  6 root          30      1      0     40      0     40 100.00% [zfskern{txg_thread_enter}]

So that would answer what php-fpm wrote, pfBlockerNG, right?

stephenw10

Yes, that's pfBlocker running something.

The other line looks like ZFS updates.

slu

@stephenw10 said in How do I find out what write continuously on my pfSense SSD:

Yes, that's pfBlocker running something.

But why write, is it a bug?
Line 947 https://github.com/pfsense/FreeBSD-ports/blob/devel/net/pfSense-pkg-pfBlockerNG/files/usr/local/pkg/pfblockerng/pfblockerng.inc

I'm not a programmer, but I see if python is enabled the code copy files continuously.
They are a lot of Netgate appliances with small storage's and the write and write until the SSD die.
Or I'm wrong?

@stephenw10 said in How do I find out what write continuously on my pfSense SSD:

The other line looks like ZFS updates.

Maybe as result of the pfBlockerNG wrote?

stephenw10

If you disable DNS-BL in pfBlocker do both processes stop?

bmeeks

@slu said in How do I find out what write continuously on my pfSense SSD:

But why write, is it a bug?
Line 947 https://github.com/pfsense/FreeBSD-ports/blob/devel/net/pfSense-pkg-pfBlockerNG/files/usr/local/pkg/pfblockerng/pfblockerng.inc

If you examine the linked code and follow the execution path, you will see that pfBlockerNG is gathering unbound query stats and writing those to a SQL Lite 3 database. The code at line 947 in the file eventually winds up launching this daemon: https://github.com/pfsense/FreeBSD-ports/blob/02b234b5d0825aa92810a15239b8b398e389ab83/net/pfSense-pkg-pfBlockerNG/files/usr/local/pkg/pfblockerng/pfblockerng.inc#L6486 called pfb_daemon_queries(). That function enters a hard loop and runs at the interval specified by this parameter: $sleep_freq = config_get_path('installedpackages/pfblockerngglobal/widget-dnsblquery', 5);. It runs a loop, then sleeps for the time specified and repeats the loop over and over until the daemon is killed.

I don't know why it's doing that, but that would likely account for the many disk writes (updating the records in that database and maintaining any associated indexes). Then ZFS is coming along behind those database writes and doing "its thing" to maintain the journal and data integrity on the disk.

Gertjan

@slu said in How do I find out what write continuously on my pfSense SSD:

show php-fpm often on the top, wo can I see what php-fpm wrote?

php-fpm write to a ... socket, an 'in memory' file stream or 'pipe' that connects it to nginx ... that pipes its output to the network, to your browser, for example.
php-fpm is also used by pfSense as an ordinary script file engine, like the shell, for maintenance tasks

Gertjan

@bmeeks said in How do I find out what write continuously on my pfSense SSD:

I don't know why it's doing that, but that would likely account for the many disk writes (updating the records in that database and maintaining any associated indexes).

@bmeeks No coffee this morning ?
At line 6530, a SQLite database is queried (read operation) for the 'resolver' field which contains the current PID of unbound.
If it changed = restart of unbound, then that field is updated, and that's a (very small) write operation.

Normally, unbound shouldn't restart a lot ..... My unbound memory usage stats shows ones or twice a week.
That said, if this one is checked, and there are a lot of devices and/or Wifi is present, then this :

will chain gun unbound.

Btw : @slu : This (example) :

looks nice. We all love it.
To make it work, a lot of data needs to be stored and updated.
If you have just one or two small DNSBL lists, or/and your 'network' doesn't try to visit every host name that is listed in these DNSBL, then stats maintenance won't create a lot of write (update) events.
But, if you network(s) try to visit as much as blocked DNSBL as possible, then, yeah, pfBlockerer will write "a lot".

So, its all a matter of 'making choices'.

slu

@stephenw10 said in How do I find out what write continuously on my pfSense SSD:

Yes, that's pfBlocker running something.

Yes the php / pfblockerng.inc queries stop.

@stephenw10 said in How do I find out what write continuously on my pfSense SSD:

The other line looks like ZFS updates.

zfs updates still there, approximately every 5 seconds write of 40.

slu

@Gertjan said in How do I find out what write continuously on my pfSense SSD:

To make it work, a lot of data needs to be stored and updated.
If you have just one or two small DNSBL lists, or/and your 'network' doesn't try to visit every host name that is listed in these DNSBL, then stats maintenance won't create a lot of write (update) events.

Yes you are right and I underestimated that.

I tried now to disable as much as possible (complete UT1 list) but still see continuously pfblockerng.inc queries writes (no python mode).

So at the end of the day RAM disk look like the solution.

slu

I don't activate RAM disk yet (because the reboot), but don't look like there is much data in:

RAM disk 2024-08-27_11-01.png

So if this is my problem, it must write every time the same "1 MB" file.

Gertjan

You have a

@slu said in How do I find out what write continuously on my pfSense SSD:

pfSense CE system

so your SSD is accessible and replaceable ? This was one of the criteria when I picked my Netgate appliance : I could, in theory, change the drive, if needed, as I know pfSense behaves like any OS out there, and when we start adding gadgets, widgets, packages and tools, drives will get solicited.
So, a 4100 was the bare minimum for me.
Ok, it wears out after x years.
The smallest Crucial out there is .. what ... 20 box ?

IMHO : stay away from 'RAM drives'. They solve some issues, by creating other ones.

Classic iron disk don't last 'forever' neither. Some last over 10 years, some died within a couple of yours.

The problem is different but nothing really changed ;)

slu

@Gertjan said in How do I find out what write continuously on my pfSense SSD:

so your SSD is accessible and replaceable ?

Yes no problem to replace the M.2 NVMe SSD.

@Gertjan said in How do I find out what write continuously on my pfSense SSD:

Classic iron disk don't last 'forever' neither. Some last over 10 years, some died within a couple of yours.

That's true.

@Gertjan said in How do I find out what write continuously on my pfSense SSD:

IMHO : stay away from 'RAM drives'. They solve some issues, by creating other ones.

This is not clear to me, what issues can it create?
I have enough RAM (16 GB), so it will not be an issue.

On say use RAM disk, the other say not, I'm really unsure at the moment.

stephenw10

Enable it and see what happens. I usually start out at double the minimum values but since you're running some packages etc and have RAM to spare I'd start out at least 4x. So 160 + 240 MB. If somethings fail to start at boot double it again.

Gertjan

@slu said in How do I find out what write continuously on my pfSense SSD:

This is not clear to me, what issues can it create?

The ones I've never seen before ^^
Remember : I've a 4100 and I went for the MAX, but that didn't MAXed out the RAM, though.
So with a mere 4 Gbytes, still very ok to run pfSense, I can't go wild with the big resource eating packages, or use options like "pfBlockerng with all feeds selected ^^"

More serious : every non programmed power down will destroy all recent stats ...

Also : It gives a blow to the KIS concept.

stephenw10

Yup you lose the RAM disk content if power is lost. But only since the last time it wrote out.

Bob.Dig

@stephenw10 But loosing its content is what is wanted here anyways. If it would wrote everything to the disk at some point, there is nothing gained from this.

stephenw10

Writing periodically is still far fewer disk writes. And not everything is written out like that only the RRD files. The other ram disk content is only backed up at shutdown.

Gertjan

@Bob-Dig said in How do I find out what write continuously on my pfSense SSD:

there is nothing gained from this.

There is.
During shutdown, the set of files will get written out to the 'real' disk.
But de xxxxx times the files were updated are all pure bonus.