RAM Disk enabled, but still constant writes to disk…



  • Hi everyone.

    I'm running 2.4.3-RELEASE (latest at the time of this writing) on HyperV.  It's a plain vanilla install with no plugins or additional packages – essentially just out of the box config with a few NAT rules.  I recently attempted to enable RAM Disk in the settings like so:

    I've rebooted several times, and I can confirm I can see "/tmp" and "/var" on the main page dashboard under "Disk Usage", however I still see almost constant writes to disk, according to the Hypervisor.  Even when I'm not logged in to the console or web admin pages, I'm still seeing constant writes:

    I should also note that disk usage reported on the dashboard for /tmp is always 0%.

    Is this a bug?  Am I missing something?


  • Rebel Alliance Developer Netgate

    How much RAM/Swap do you have on that VM? Maybe it's swapping to disk because it doesn't have enough RAM.

    You can check programs doing disk i/o with utilities at the shell, like "top -am io"



  • Thanks for the reply.

    I gave this VM 2GB of RAM, which I figured was probably more than enough.  Here's a shot from the dashboard:

    I ran the top command you suggested and watched it for a bit.  I didn't see anything that seemed out of the ordinary, and I never once saw anything but 0s in the "Read' or "Write" columns.  The list jumped around a lot, but here's a screenshot from a random point in time:

    Looks pretty normal but it's quite possible I don't know what I'm looking at.

    Any other suggestions?



  • Edit: The post is incorrect, see below.

    What filesystem is your pfSense box?
    If you picked ZFS it's probably always writing out to the journal.



  • Interesting thought, but it looks like I'm using UFS:

    
    [2.4.3-RELEASE][admin@pfSense]/root: df -T
    Filesystem                   Type  1K-blocks   Used    Avail Capacity  Mounted on
    /dev/ufsid/59877dae5429c116  ufs    16226924 830328 14098444     6%    /
    devfs                        devfs         1      1        0   100%    /dev
    /dev/md0                     ufs       98716    172    90648     0%    /tmp
    /dev/md1                     ufs       98716  18744    72076    21%    /var
    devfs                        devfs         1      1        0   100%    /var/dhcpd/dev
    
    


  • @muppet:

    What filesystem is your pfSense box?
    If you picked ZFS it's probably always writing out to the journal.

    Completely untrue. The so called "journal" (ZIL aka ZFS intent log)  is used only when synchronous writes are requested by an application. Afaik there's very little on a standard pfSense install that would request synchronous writes, most likely nothing.



  • Update: I was searching for old threads about issues with RAM Disk and I found one where jimp suggested using a slightly different top command: "top -aSH -m io -o total".  I tried running this, and with this one I noticed every couple of seconds, I saw [syncer] jump to the top of the list, with some actual writes:

    So, hmm… I'm guessing [syncer] is a process that actually performs writes that are queued up to disk.  But I figured if I had RAM Disk set, it would be syncing to the ramdisk and not to /.  But that doesn't seem to be what's happening…


  • Rebel Alliance Developer Netgate

    The "100%" in that output can be misleading. It means that of all the writes happening in that moment, 100% are being performed by that process. It doesn't mean that it's using 100% of the disk write capability.

    RAM disks being enabled means that anything writing to /var or /tmp is writing to a RAM disk, not permanent storage. There could still be other unrelated writes happening not to /tmp and /var, however.



  • Thanks, that makes sense.  Any idea how to check what it is writing so frequently to disk outside of /tmp and /var?  I SSH'd in and I ran the following command at / to sort all files by last modified time:

    find . -print0 | xargs -0 -n 100 stat -f"%m %Sm %N" | sort -n|awk '{$1="";print}'
    

    I see a bunch of writes to stuff in /tmp and /var, but the most recent write I see outside of there was almost 25 minutes beforehand (on the hour, when /tmp and /var got flushed to disk).  Still, Hyper-V reports nearly constant writes.  I've got several Linux VMs on the same box running at the same time, but pfSense is always at the top of the list for disk writes.  Very strange.



  • I know this is an old thread, but I was having the exact same problem with no one having a solution. This thread comes up first when investigating, so I'd like to make sure people see the solution.

    I tried all the same stuff, enabling RAM disk, disabling logging, turning off swap in ESXI, etc. The trick to getting it to stop writing to your SSD is to reinstall Pfsense and select 0 as the swap size during the install. Obviously you want to save and then restore your current config. By default it makes a 2GB swap partition and that seems to be what is being written to constantly - despite RAM disk being enabled and regardless of how much free memory Pfsense has. After the reinstall I get next to nothing on disk writes, whereas before I killed one 128GB SSD by writing several TB to it over the course of only a couple months.



  • There is no way that a modern SSD would die after such a short period of time with only several TB written. It was defective.



  • @KOM I mistakenly put several. It was a lot more than that. It was a lower cost one, I suspect it died so fast because it was lacking a DRAM cache, so it was doing thousands of small writes per day (if not more, my disk light was blinking all the time when Pfsense's VM was running). Having to write each block for a pointlessly small write will kill a cell just as fast as a full cell write.

    TLC memory won't take much and when you write dozens of TB to a 128 GB disk over the course of a couple months. Hell, my barely used Samsung 850 Pro 256 GB is already at 61 TB of total writes because of this damn thing - it was barely used before I put it in the server and it literally has: ESXi with FreeNAS, PFsense, and a couple other small VMs. The only one which did any writing to the disk was the Pfsense one.


  • Netgate Administrator

    Make sure your root is mounted 'noatime'. Check mount -p. Edit the fstab and reboot to apply it if it's not.

    https://redmine.pfsense.org/issues/9483

    Hard to imagine it writing that much though.

    Steve



  • @stephenw10 It's not that this issue was writing a lot of data, it was the number of writes per minute of little chunks. I'm 99% sure the zero swap install fixes the problem, and was the root cause. The whole point of having the RAM disk enabled was to save flash based storage, and for some reason it just wrote all the RAM disk info to swap as it came in (I suspect, I didn't look too much into the data being written, other than it was syncer doing the writing and it was all the time). It could be some interaction between ESXi and Pfsense since it doesn't seem to be a very widely reported issue.

    While testing I tried reinstalls (default, with swap), killing all the logging I could find, removing all packages (even VM tools, just in case), changing RAM disk settings, etc. It just kept writing away, 24/7 until I did the no swap install. :D The no swap install used a restore of the exact same config I was using.


  • Netgate Administrator

    Hmm, interesting. I usually install without swap anyway on devices running from flash. However I would not on an SSD and I've never noticed drive writes anywhere near that. Did it actually show swap being used?

    Steve



  • @stephenw10 I'm not actually sure, I was really in the "just try things to get this damn thing to stop" mode so I didn't look too much into what exactly it was doing other than calling syncer constantly. I have no idea why it would care about swap, it had 8 gigs of memory (later 32 GB just to test it), and it was still writing. Note I use ZFS, I believe I tried a UFS install a while back to test and it still did it.

    I made a new VM, installed it. Disk writes. Restored my config and it still was writing to the disk. When I made a VM and installed without swap it went away both before and after I restored my config. Like I said in another reply, I don't think this is very widespread so I would like to he from the original poster since they seemed to have the exact same problem.


  • Netgate Administrator

    Hmm, interesting catch. Yeah 61TB in 2 months is waaaaay outside the range of anything I've seen. Even those systems that were not mounted noatime for a while.

    Steve



  • Just providing some input to this thread because it very nicely captured a problem which I am (have?) experiencing.

    I have a 2.4.4 pfSense system running on a Proxmox Virtual Environment (6.1-3) and I was surprised to see that my SSD's (128GB LiteOn m.2 SATA SSD - consumer grade stuff) Wear Leveling SMART number dropped 4% in about 3 weeks. So I started to investigate optimizing proxmox and pfsense to reduce writes to the drive.

    On the pfSense side, I have been observing regular writes on the hypervisor (via iotop -a --only) which show the kvm of the pfSense system is writing to the disk rather constantly. Proxmox history shows it's around 10k on average:

    0290b7f6-8b83-4b77-a60a-2d89087df645-image.png

    On my 2.4.4 setup I have RAM disk enabled. I have a constant connection to a OpenVPN server and a not much else.

    Based on the observations in this thread, I reinstalled pfSense with a manually partitioned drive where I deleted the swap file (and enabled trim on the virtio based disk).

    After reinstalling I still observed writes on the VM, so I enabled noatime as well on the root mount.

    In this plot you can sort of see the effects of that (it shows up around 22:50-22:55 on the plot).

    17803b98-8f14-42b3-9957-e49d2906812c-image.png

    Now the writes are not so constant, but there are still a few periodically.
    Based on the change observed with the noatime setting, vs reinstalling without swap, I'm not sure if the swap was the problem. It would be interesting to know if PeterBrockie's successful setup did include the noatime mount option.

    Although I am definitely not in the range where the drive use is going to kill my SSD, but I think it's worthwhile noting that such writes do exist, when in theory we might expect no writing to occur. I'm still trying to understand what the other writes are - I'll try not to log into the pfsense system in the next little while and see what the hypervisor detects.

    Thanks to everyone who contributed to this post.


  • Netgate Administrator

    Mmm, interesting. I too have a Proxmox system here with a least two pfSense VMs running in it continually. I also see 20-30k drive writes in each. I've set noatime manually now, I'll let you know.
    The smartctl status from it is interesting:

    === START OF SMART DATA SECTION ===
    SMART overall-health self-assessment test result: PASSED
    
    SMART/Health Information (NVMe Log 0x02)
    Critical Warning:                   0x00
    Temperature:                        32 Celsius
    Available Spare:                    100%
    Available Spare Threshold:          10%
    Percentage Used:                    0%
    Data Units Read:                    696,237 [356 GB]
    Data Units Written:                 1,119,658 [573 GB]
    Host Read Commands:                 3,490,850
    Host Write Commands:                11,895,271
    Controller Busy Time:               66
    Power Cycles:                       10
    Power On Hours:                     341
    Unsafe Shutdowns:                   2
    Media and Data Integrity Errors:    0
    Error Information Log Entries:      0
    Warning  Comp. Temperature Time:    0
    Critical Comp. Temperature Time:    0
    Temperature Sensor 1:               32 Celsius
    Temperature Sensor 2:               32 Celsius
    
    Error Information (NVMe Log 0x01, max 64 entries)
    No Errors Logged
    

    I'm not totally sure about that since the power on hours seem low, I've had that running for significantly longer than 2 weeks.

    Steve



  • @emobo I have a ZFS pool and haven't touched atime. As far as I can tell my specific disk write problem was solely an interaction between swap and the VM. Despite having plenty of free RAM and having all logs go to RAMdisks, it still wrote something to the swap constantly.

    After removing swap:
    Screenshot (44).png



  • @stephenw10
    Ah sounds familiar. Is that smartctl status output from your proxmox Debian host or the pfsense Freebsd. I don't know the right command line arguments to get it from the pfSense VM. I'm not sure if it can be accessed there.

    @PeterBrockie
    Thanks, that is impressive and it's in the last hour so the resolution is high. That is exactly what I would hope to achieve as well. I have no swap now, so I'm unsure what else could be causing it.

    I have more history now on my host log and it still shows activity:
    87402877-484d-40da-871b-a67ab6bd47a8-image.png

    Focusing on the last hour it's still showing periodic writes.
    7d3c7f80-2b02-491a-bfce-18628ca63764-image.png

    There is an old thread here from back in 2012 https://forum.netgate.com/topic/130424/ram-disk-enabled-but-still-constant-writes-to-disk


  • Netgate Administrator

    Anything from 2012 is largely irrelevant at this point.

    I only have 2.4.5 and 2.5 snapshots running but the results were broadly similar. Very basic installs.
    Mounting root noatime produces ~50% decrease in drive writes. ~30kBps to ~18kBps.
    Enabling RAM drives reduces it to 0 most of the time. There are obviously still some writes when the config updates etc.

    Selection_752.png

    Steve



  • Thanks - yes the noatime has a noticeable effect.
    I'm puzzled how PeterBrockie's configuration could be so quiet while the other setups still have regular activity.

    As a test, I tried disabling local logging but it seems to have little to no effect. This makes sense if the logs were being written to the ramdisk anyway.



  • Pardon the interruption, but is this a Proxmox, VM, SSD or swap specific issue?



  • @provels That'd what we are trying to figure out. I am running VMware and it killed a ssd in no time. Disabling swap fixed it for me and not for others, so we are trying to figure out exactly what it is.


  • Netgate Administrator

    It's not VM specific, it's just far easier to see the disk IO in a VM. What the actual cause of the OPs issue where he had to remove SWAP is a mystery. I could not replicate.



  • @stephenw10 I personally didn't have the problem outside a VM. I was running Pfsense for years and years on a small 32GB SSD which would have failed 10 times over at the rate it killed my larger drive. The little drive passed SMART tests, etc and is still going.

    Same config file (although I did test a fresh install).


  • Netgate Administrator

    Without noatime set I have seen some high drive write numbers, much higher than I expected. I've yet to see anything kill a drive though. At least not with drive writes alone.

    With RAM drives enabled I'm seeing effectively 0 drive writes until I save a change etc. I think that's the same as you are pretty much.

    Steve



  • @PeterBrockie Well, FWIW, with noatime, ramdisks, and swap enabled I see no disk activity at all on my pfSense VM VHDX in Hyper-V (2012R2).
    Without noatime, but else same, as below.
    f2a5a30c-02cf-4926-8feb-a7a6f4b7c7f6-image.png



  • It would seem very strange if this was caused by the choice of Hypervisor. I'm less familiar with the other hypervisors - does anyone know if Proxmox is the only one that uses the virtioblock device for the hard disk? If it was VM host related perhaps that could be related?

    @stephenw10 - in the case of the writes I'm curious about - I believe those are not initiated by me directly - I am purposely trying to avoid touching the pfsense system while those writes are occurring. I don't login, or make any changes to the environment - it should be just routing (and logging). I can accept that there will be a few jobs on timers which occur (i.e. the ramdisk is dumped to disk periodically - but I have that set to 24hours) but I am surprised it would be anything so frequent.

    I do find this truly intriguing. To me, this is less about killing SSD's, than it is about not really having a good handle on the what the system is doing. These are security focused platforms so it would be ideal if an administrator can make sense of what's happening.

    I wonder if an experiment like this would work - on a test pfsense install - can we remount the / partition as ro and see what gets upset? It might be time to start breaking out more VM's...


  • Netgate Administrator

    Well what do you have configured on that VM? Any packages?

    I see basically zero writes unless I'm saving a change or as you say it is writing out the RRD data or updating bogons etc.

    Selection_754.png



  • @stephenw10
    Thanks - yeah I don't have any packages except the openvpn client export.
    Configuration wise

    • a few custom firewall rules
    • iot vlan
    • openvpn client interface
    • openvpn server

    Service wise it's the usual suspects (dhcpd, dpinger, ntpd, openvpn x2, sshd, syslogd and unbound).

    Given what you and others have provided above - it must be something in my configuration or traffic.

    I've got a proxmox 6.0 server which I've reinstalled pfsense 2.4.4 with no swap and configured the ssd to have noatime and enabled ramdisk. As soon as I enabled ramdisk it went super quiet (just around 9:30)
    ed06ab54-169c-4060-9e00-689e170a5156-image.png

    Now this setup is not really representative of my live setup (no real wan traffic and no clients) but (unless there is something different in promox 6.0 vs 6.1) it's a good indication that it's something due to the configuration and loading on the live pfsense setup.

    Thanks for your help - I'll keep playing around and keep this post updated if I find anything else.

    Incidentally on my fresh install, I did an iostat comparison between a reboot last night and this morning and it shows it wrote about 20megs to the disk. (Without RAMdisks).



  • Ok so just an update on this investigation. I've been away for a bit so not doing much with the setup.

    During that time my live pfsense machine continues to write frequently to the disk. Over the last 10 days up, iostat shows it has written about 2500MB - which seems like a lot for something that shouldn't be writing anything to the disk.

    Meanwhile, my test pfsense setup with no routing traffic has been very silent on disk.
    I updated it to Promox 6.1-5 (from 6.0) and it was still fine.
    I took my live pfsense XML and restored it on the test configuration (i had to shift around some ip addresses and interfaces to keep things legit) and it continued to be disk silent.

    So it seems like the disk activity requires some network activity, unfortunately my test VM machine doesn't have multiple nics, so I may have to configure some bridges to mimic more network traffic.



  • Adding a client machine to my test network generates some writes on my test installation, which confirms it is related to the existence of client machines. Since it's unlikely related to traffic (as most of that is logged in RAM), I guessed it maybe something related to DHCP leases.

    I used a modified version of the find command listed by BlueScreenOfTOM above to identify some files being written to, and it seems like /etc/hosts is being written to quite regularly.

    I looked at the contents and it seems to be related to the DHCP leases getting written to the /etc/hosts files

    I believe this is caused by "Register DHCP leases in the DNS Resolver" being selected in the DHCP server settings, so I have removed that for now. Given my hostname is not really legit, these are pretty much pointless anyway.

    So far, disabling that has reduced the writes to zero.
    6cf5ea10-5535-45c3-9d71-535d270fbd11-image.png
    So perhaps the mystery is solved? :)


Log in to reply