RAM Disk Management PR Discuss

NOYB · Jan 6, 2017, 11:03 PM

RAM Disk Management #2902
https://github.com/pfsense/pfsense/pull/2902

Jim,
Have not been able to reproduce the issue you outlined in the PR.
@jim-p:

I applied this patch and it seemed to work at first. The first reboot after applying it picked up the previously saved RAM disk files during upgrade, but then the next reboot the RRD files were gone. Either they didn't save on shutdown or they were not restored on boot.

Start with a fresh VM, update it, activate the RAM disk setting, reboot, apply the patch, reset the VM so it upgrades, check the graph data, then reboot and check the graph data again.

Here are the steps I used.

pfSense 2.4 on VirtualBox.


1) Console option 13) Update from console
(update if not already up to date)

2) Console option 12) PHP shell + pfSense tools
playback gitsync --minimal --diff --show-files git://github.com/pfSense/pfsense.git master
Console option 5) Reboot system / y: reboot normally

3) Web GUI - Status / Monitoring (graph appears fine)

4) Web GUI - System / Advanced / Miscellaneous
Enable "Use RAM Disk"
/tmp size (40; default)
/var size (60; default)
RRD Backup (unset)
DHCP Backup (unset)

System reboots to enable RAM disk.

4b) Web GUI - Status / Monitoring (graph appears fine)
/cf/conf/dhcpleases.tgz appears fine.
/cf/conf/rrd.tgz appears fine.
/cf/conf/RAM_Disk_Store/var/db/aliastables/*.tgz appears fine.

5) Console option 12) PHP shell + pfSense tools
playback gitsync --minimal --diff --show-files git://github.com/pfSense/pfsense.git RAM_Disk_Management
Reset VM

5b) Web GUI - Status / Monitoring (graph appears fine)
/cf/conf/rrd.tgz.old appears fine.
/cf/conf/RAM_Disk_Store/*.tgz appears fine.  (dhcpleases.tgz, rrd.tgz, and url tables *.tgz)

6) Console option 5) Reboot system / y: reboot normally

6b) Web GUI - Status / Monitoring (graph appears fine)
/cf/conf/rrd.tgz.old appears fine.
/cf/conf/RAM_Disk_Store/*.tgz appears fine.  (dhcpleases.tgz, rrd.tgz, and url tables *.tgz)

6c) System Logs Filter on "RAM"
8 Matched General Log Entries. (Maximum 50) 
Time Process PID Message
Jan 6 14:14:25  root   RAM disk restore succeeded: /cf/conf/RAM_Disk_Store/Bogons_IPv4.txt.tgz  
Jan 6 14:14:25  root   RAM disk restore succeeded: /cf/conf/RAM_Disk_Store/Bogons_IPv6.txt.tgz  
Jan 6 14:14:25  root   RAM disk restore succeeded: /cf/conf/RAM_Disk_Store/CIPB_US_Networks.txt.tgz  
Jan 6 14:14:25  root   RAM disk restore succeeded: /cf/conf/RAM_Disk_Store/Spamhaus_DROP.txt.tgz  
Jan 6 14:14:25  root   RAM disk restore succeeded: /cf/conf/RAM_Disk_Store/Spamhaus_EDROP.txt.tgz  
Jan 6 14:14:25  root   RAM disk restore succeeded: /cf/conf/RAM_Disk_Store/dhcpleases.tgz  
Jan 6 14:14:25  root   RAM disk restore succeeded: /cf/conf/RAM_Disk_Store/rrd.tgz  
Jan 6 14:14:25  root   RAM disk restore succeeded.  

7) Console option 6) Halt system
Start VM

7b) Everything still appears fine.

jimp · Jan 6, 2017, 11:27 PM

Try with ZFS

chrcoluk · Jan 7, 2017, 12:02 AM

is there merits to using a ram disk when I have a ssd as the storage? Since the ssd should easily be fast enough for any i/o needs. The cons of needing to restore files after a reboot seen heavier than the pros.

jimp · Jan 7, 2017, 12:03 AM

SSD should be fine, if it's decent quality or recent.

Problem is when people want to run from media with low overall tolerance for writes (CF, SD, thumb drives…)

NOYB · Jan 7, 2017, 7:53 AM

VirtualBox
Fresh install using "pfSense-CE-2.4.0-BETA-amd64-20170106-1042.iso"
Auto (ZFS)
stripe Stripe - No Redundancy

ada0 VBOX HARDISK

Console 2) Assign LAN IP address
Console 14) Enable secure shell
WebGUI Config Wizard (defaults except primary DNS, disable dhcp override, TMZ, allow private and bogon networks)

Let system percolate some RRD data for 5 minutes. (20:43-20:48)
Enable ram disk (auto reboot)
Monitoring graphs and backup files appear fine.

Let system percolate more RRD data for 5 minutes. (20:50-20:56)
GitSync with master:
playback gitsync –minimal --diff --show-files git://github.com/NOYB/pfsense.git master
Shutdown
VM Snapshot
Start VM

Monitoring graphs and backup files appear fine.

Let system percolate more RRD data for 5 minutes. (21:02-21:07)
GitSync with the patch ("RAM_Disk_Management"):
playback gitsync --minimal --diff --show-files git://github.com/NOYB/pfsense.git RAM_Disk_Management
Reset VM so update is applied.

Monitoring graphs have a gap from 20:56 to 21:10 (present boot time).
Believe the gap is VM rest induced due to old shutdown script not being run which would create an up to date rrd backup. Was last backed up at 20:56.

~~Expected console message "Restoring contents of RAM disk store…". Is not seen during the update boot-up. It is seen on subsequent boot-ups though.~~ Nothing in the RAM disk store at this point so no console message is to be expected.
Backup files (new ram disk store) have been updated. Indicating the update script has run.

Let system percolate more RRD data for 5 minutes. (21:10-21:15)
reboot
Expected console message "Restoring contents of RAM disk store…". is now seen during boot-up.
Monitoring graphs and backup files appear fine. (VM reset induced gap but otherwise appears fine.)

Looks like it is working far as I can tell. Can anyone confirm or refute?

jimp · Jan 8, 2017, 3:38 AM

OK, the problem in my case was that the new files created by the patch didn't get set executable. Once I fixed that it started to work.

NOYB · Jan 8, 2017, 11:56 PM

Here's a procedure that may be better for testing this without the VM/hard reset induced gap. Basically run the backups manually from command prompt. Then after applying the patch, halt to stop cleanly. Using halt should be more UFS friendly than doing a dirty/hostile hard reset. Whereas ZFS doesn't seem to mind the hard reset.

Enable RAM Disk
Percolate some monitoring data for a few minutes.
Run the backups manually from console command prompt.
/etc/rc.backup_aliastables.sh
/etc/rc.backup_dhcpleases.sh
/etc/rc.backup_rrd.sh
Apply the patch.
playback gitsync –minimal --diff --show-files git://github.com/NOYB/pfsense.git RAM_Disk_Management
Stop and restart so the system upgrades.
halt
Check upgrade result.
a) Verify monitoring graphs and RAM disk backup files.
b) Reboot
c) Verify monitoring graphs and RAM disk backup files.