RAM Disk enabled and crashes

Mission-Ghost

I tried the RAM disk on my 4200 with what research suggested to me was more than enough memory. It filled up within a day and froze my system killing production.

It doesn’t seem to have any logic to fail gracefully when it has problems. I can’t experiment with a production system to find the right size, which seems to be the only way to figure out how big to make it.

I won’t use it again.

Laxarus

@Mission-Ghost yeah, I guess, it is better not to use it at all. The current implementation seems to be broken if it does not have a mechanism in place to gracefully rotate it to disk or do something else when it gets filled.

stephenw10

512MB should be more than enough for most configurations. However if you have packages that require a lot of /var space to update you may need more. Notably pfBlockerNG or Snort/Suricata can use a lot if you have large number of lists or signatures configured.

Laxarus

@stephenw10 hmm I do have pfblocker with a number of lists but not to the point that it will fill the var space with 512MBs.

current usage is not very high

stephenw10

Even when it runs an update? That's when it uses the most space as it expands the new lists.

Laxarus

@stephenw10 did not check that one actually. But in that case, it is very hard to determine how much space you need for that.

There should still be a graceful way to fail even after the space has been exhausted instead of just failing right?

stephenw10

Yup, there should be. I would expect 512MB to be large enough even for pfBlocker unless you really have a lot of large lists.

bmeeks

It can be very difficult for the package code (for example, pfBlockerNG, Snort, Suricata, etc.) to know how much disk space is required for a task.

Take Snort and Suricata as examples. The rules package files come down from the rules vendor website in compressed GZIP form. Neither Snort nor Suricata know how to natively manipulate GZIP files, so they simply pass the compressed file and the path /tmp to the gzip binary utility and let it unzip the file. The gzipped rules archive contains several dozen individual compressed text files of various sizes that get expanded by gzip and written to disk. That's the point where the available space in the RAM disk might get exhausted, and once the space is gone, other critical pieces of pfSense will encounter issues and the whole system may crash or freeze.

The Snort and Suricata packages clean up behind themselves by removing all files they created from /tmp even if the unzip process fails. So, if you look at the directory on the RAM disk after the rules update process completes, it will appear as if there was plenty of space. But during the actual update process while gzip was unzipping the rules archive, space may have been exhausted.

I can't speak for pfBlockerNG since I don't maintain that package, but for Snort or Suricata the use of RAM disks is strongly discouraged. Not only can you hit issues when unzipping the rules archive files for updating your IDS rules, you can also exhaust available space with the logs which will go into /var/log/xxxx. Automatic log size management is also a challenge with these packages as neither of the two binaries offers a perfect built-in solution. Snort is a little better as it can be told to rotate its logs by size. Suricata, however, only provides an option to rotate logs based on a time interval (not size). Time interval is not ideal for log rotation because the file sizes can vary dramatically based on traffic during the time interval.

Laxarus

Hmm, isnt that RAM DISKs inherently flawed feature then? Since in my case, assuming that 512 was not enough and pfblocker has failed the firewall did not recover.
I have also checked the pfblocker logs and nothing indicates a failure of the update during that time.
So, not sure what is going on here.

bmeeks

@Laxarus said in RAM Disk enabled and crashes:

Hmm, isnt that RAM DISKs inherently flawed feature then? Since in my case, assuming that 512 was not enough and pfblocker has failed the firewall did not recover.
I have also checked the pfblocker logs and nothing indicates a failure of the update during that time.
So, not sure what is going on here.

The package logs may or may not reflect the out-of-space condition. The pfSense system log should record an error message, though.

Speaking for my packages, they passed off the unzip task to a separate FreeBSD utility (gzip in this case), and they will only receive an ERROR or NO ERROR result. They don't know what the error was.

bmeeks

@Laxarus said in RAM Disk enabled and crashes:

Hmm, isnt that RAM DISKs inherently flawed feature then?

No, the feature itself is not flawed. It's how users implement the feature -- sometimes making unwise choices -- that causes the problems. As I mentioned above, it's been repeated in numerous posts on these forums that use of RAM Disks and Service Watchdog with the IDS/IPS packages is very strongly discouraged, yet users persist in trying to use RAM disks and Service Watchdog with those packages and return to post about encountering issues .

If you run packages that download things (like rules updates and IP lists) or that log lots of data (like IDS/IPS, ntopng, and even pfBlockerNG), then RAM disks are not a good choice. If you have a bare bones plain vanilla pfSense install, RAM disks have a valid place to conserve the life of eMMC, for example.

Laxarus

@bmeeks said in RAM Disk enabled and crashes:

use of RAM Disks and Service Watchdog with the IDS/IPS packages is very strongly discouraged

I wish what you said here was indicated below the RAM DISK option on the UI or in the docs.

Tracing my System Logs, I see that there is a huge gap until I restarted the firewall.
From Jun 20 03:02:36 to Jun 20 16:48:41.

This is just after the pfblocker update. So, I guess pgblocker just nuked it.

Jun 20 03:02:29 FIREWALL check_reload_status[668]: Syncing firewall
Jun 20 03:02:29 FIREWALL check_reload_status[668]: Reloading filter
Jun 20 03:02:30 FIREWALL dhcpleases[45163]: Could not deliver signal HUP to process 47311: No such process.
Jun 20 03:02:30 FIREWALL php-fpm[64274]: /rc.filter_configure_sync: The gateway: VPNAC_WG is invalid or unknown, not using it.
Jun 20 03:02:34 FIREWALL php-fpm[17524]: /diag_reboot.php: The command '/usr/local/etc/rc.d/wireguardd stop' returned exit code '1', the output was '' 
Jun 20 03:02:34 FIREWALL php-fpm[17524]: /diag_reboot.php: The command '/usr/local/etc/rc.d/pfsense_tailscaled stop' returned exit code '1', the output was 'Stopping tailscaled. Waiting for PIDS: 56646.' 
Jun 20 03:02:34 FIREWALL radiusd[25219]: Signalled to terminate
Jun 20 03:02:34 FIREWALL radiusd[25219]: Exiting normally
Jun 20 03:02:34 FIREWALL mdns-bridge[30401]: exiting on signal 15
Jun 20 03:02:34 FIREWALL kernel: tailscale0: link state changed to DOWN
Jun 20 03:02:35 FIREWALL lighttpd_pfb[52403]: [pfBlockerNG] DNSBL Webserver stopped
Jun 20 03:02:35 FIREWALL tail_pfb[54461]: [pfBlockerNG] Firewall Filter Service stopped
Jun 20 03:02:35 FIREWALL php_pfb[55380]: [pfBlockerNG] filterlog daemon stopped
Jun 20 03:02:36 FIREWALL php-fpm[53932]: /widgets/widgets/interfaces.widget.php: ERROR! ldap_get_groups() could not bind to server Windows Active Directory LDAPS.
Jun 20 16:48:41 FIREWALL syslogd: kernel boot file is /boot/kernel/kernel
Jun 20 16:48:41 FIREWALL kernel: ---<<BOOT>>---
Jun 20 16:48:41 FIREWALL kernel: Copyright (c) 1992-2024 The FreeBSD Project.
Jun 20 16:48:41 FIREWALL kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Jun 20 16:48:41 FIREWALL kernel: 	The Regents of the University of California. All rights reserved.
Jun 20 16:48:41 FIREWALL kernel: FreeBSD is a registered trademark of The FreeBSD Foundation.
Jun 20 16:48:41 FIREWALL kernel: FreeBSD 15.0-CURRENT #1 plus-RELENG_25_03-n256490-5cb88176dc52: Wed Apr  9 22:49:02 UTC 2025
Jun 20 16:48:41 FIREWALL kernel:

bmeeks

@Laxarus said in RAM Disk enabled and crashes:

I wish what you said here was indicated below the RAM DISK option on the UI or in the docs.

I have several times considered testing within the PHP package code of Snort and Suricata for the existence of RAM disks or the presence of Service Watchdog and have both of those packages refuse to start if either condition was true. That's a bit of a big hammer approach and I'm sure would generate complaints, so I have not coded that yet. But both settings can cause really big issues with both of the IDS/IPS packages.

stephenw10

Mmm, we could certainly improve the advice on the page enabling it.

Of course that doesn't help if RAM disks are already enabled before the package is installed.

Laxarus

@stephenw10 Hmm,
maybe a warning here too below the description if it has already been enabled.

SteveITS

@bmeeks FWIW we’ve used RAM disks with Suricata for many years across all clients without issue. And pfBlocker also though we mostly use that for feeds and geoIP not DNSBL lists. I expect setup/configuration can produce different results. But it’s not inherently flawed.

@Laxarus Re temp space, the “adult” pfBlocker list for instance takes over 1 GB to just extract. I don’t know the total amount needed for it to process.

Based on my few decades in IT I would suggest no one expect applications or even OS services to fail gracefully by checking for disk space at every write; as alluded to above it’s problematic anyway. A program installer knows how much space is needed, is the one exception. And even then it’s not uncommon to sit at “100% done” while still installing.

SteveITS

@Laxarus Also I wouldn’t use a RAM disk for performance, but for limiting disk writes. Maybe on eMMC storage but then the disk writing is more important anyway.

provels

@Laxarus said in RAM Disk enabled and crashes:

I have about 32GB of RAM installed which should be more than enough.
I set 512/512 and enabled RAM disks.

Any reason you chose such small RAM disks with so much memory? I've run 2048 tmp and 4096 var on my 16GB sys. I know at least pfB needs plenty of space to update, enough for the old and new data simultaneously.

Laxarus

@provels I thought 512 should be plenty enough but who knew???