How to find what is writing to disk? posts found not working

MarioG

Please note I have searched for weeks on the web and posts here and pfsense doc, nothing I found others doing seems to work.

I want to know details of what is writing to the SSD. Visually a red light indicates writing, sometimes it’s solid for 40 seconds. I thought this would be easy but nothing seems to work. I have mailreport, ntopng, and pfBlockerNG-devel installed. I know it could be one of them.

I tried “top -aSHm io -o write” in Diagnostics Command Prompt as suggested here: https://forum.netgate.com/topic/189820/how-do-i-find-out-what-write-continuously-on-my-pfsense-ssd/3
All I get is (no details):

last pid: 16210;  load averages:  1.34,  0.95,  0.83  up 14+16:47:27    09:53:33
66 processes:  1 running, 65 sleeping
CPU: 11.2% user,  0.2% nice,  2.5% system,  0.2% interrupt, 85.9% idle
Mem: 142M Active, 1878M Inact, 2388M Wired, 56K Buf, 27G Free
ARC: 584M Total, 85M MFU, 447M MRU, 9852K Anon, 5128K Header, 37M Other
     456M Compressed, 2011M Uncompressed, 4.41:1 Ratio

I did the “iostat -d 5 6” and got:

ada0            pass0
KB/t  tps  MB/s  KB/t  tps  MB/s
65.2   86   5.4   0.5    0   0.0
21.5   85   1.8   0.0    0   0.0
16.5   83   1.3   0.0    0   0.0
17.8   66   1.1   0.0    0   0.0
27.2   56   1.5   0.0    0   0.0
12.5   38   0.5   0.0    0   0.0

Also, I can find no way to get this info (from other post). Could be because I have an MSATA? SMART Status shows everything but this:
Available Spare: 100%
Available Spare Threshold: 10%
Percentage Used: 68%
Data Units Read: 25,221 [12.9 GB]
Data Units Written: 93,370,737 [47.8 TB]
Host Read Commands: 4,428,272
Host Write Commands: 4,340,924,956
Controller Busy Time: 42,507
Power Cycles: 25
Power On Hours: 16,659

The disk utilization on the dashboard shows little used, no problem on size:

/
944M
404G
0% of 404G (zfs)
/cf
924K
403G
0% of 403G (zfs)
/home
128K
403G
0% of 403G (zfs)
/pfSense
96K
403G
0% of 403G (zfs)
/tmp
264K
403G
0% of 403G (zfs)
/var
2.5M
403G
0% of 403G (zfs)

Thanks

SteveITS

@MarioG I don’t have a direct answer to your question but it’s ntopng:
https://www.netgate.com/supported-pfsense-plus-packages

MarioG

@SteveITS Yes I read everything I could find in the pfsense doc, forums, web. I want to know what pfsense, pftopng, etc. are doing. Maybe I could adjust or setting, or have to turn ptfopng off. But I need to know exactly what is going on. I find it strange that no commands I found will display the details in the post mentioned above and others which people seem to have no trouble getting the details. Thanks.

bmeeks

@MarioG said in How to find what is writing to disk? posts found not working:

Yes I read everything I could find in the pfsense doc, forums, web. I want to know what pfsense, pftopng, etc. are doing. Maybe I could adjust or setting, or have to turn ptfopng off. But I need to know exactly what is going on.

It is mostly ZFS. It makes a ton of extra writes to maintain its file system integrity. That's one of the things that makes ZFS much more resilient to "pull the plug" power outages -- it writes a lot of backup information scattered around the drive so that it can recover from problems that would leave older file systems such has UFS dead in the water. That resilience is what sets ZFS apart. If you look in the other main thread on the forum about eMMC wear (https://forum.netgate.com/topic/195990/another-netgate-with-storage-failure-6-in-total-so-far/227), you will see a post detailing how ZFS makes writes at 4 to 6 times the rate of UFS. That's due to the way ZFS duplicates data across the disk. It writes a bunch of blocks and thus the aggregrate total of bytes written per disk transaction is 4 to 6 times greater than UFS.

Add packages that themselves write a lot of log data (ntop, for example) and you wind up with a ton of writes. I'm not sure how much of the internal ZFS stuff is visible or at least is logged by the popular I/O stats utilities. My suspicion is a fair amount of those ZFS housekeeping disk writes do not show up in the stats from the common utilities.

The issues with eMMC wear pretty much coincide with the introduction of ZFS as a default install of pfSense. I'm not knocking ZFS as it has several advantages (the biggest of those being boot environments), but on a firewall with a relatively puny amount of RAM and a small eMMC it's not the ideal file system in my opinion. ZFS loves RAM for caching, but if your firewall is limited with RAM that impacts overall performance. And the other thread details the pitfalls of the increased disk writes on small eMMC devices. Better in my humble opinion to stick with UFS and install a UPS and either nut or apcupsd to monitor the UPS and gracefully shutdown the firewall as the battery nears exhaustion. Or at the very least do that for devices that use eMMC. Maybe if you have a larger SSD and enough RAM, then ZFS makes more sense. Just my view ... .

MarioG

@bmeeks Thanks for the post, but I saw that too. I searched high and low and so far nothing to explain why the “top -aSHm io -o write” command will not show details, only the header. I really need to find how to get the details of what is writing to the SSD (no EMMC here). Many others have done so.