Manual Restore
-
I have a unit that fails to boot every time and just hangs. If I boot to Single User mode it can essentially run fsck every few minutes and it finds problems. It also locks up when running programs like vi while trying to edit files to get it back up. My assumption is something wrong with the SSD. I've set up a new unit and want to bring over all the info (system config, system logs, historical usage graphs, suricata logs, etc.) Is there a way to do that manually? I can copy over the config.xml and the suricata log files but is there a better way to make sure I get it all?
-
Could be the SSD or RAM. Either way you'd get similar results.
If the disk is in question, you might not want to trust it for much. config.xml should be pretty easy to verify that it's OK, less likely to be able to verify logs and other data.
I'd just worry about the config and give up on the rest.
Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!
Need help fast? Netgate Global Support!
Do not Chat/PM for help!
-
Yeah if you need long term logging you should exporting it via syslog anyway:
https://docs.netgate.com/pfsense/en/latest/monitoring/copying-logs-to-a-remote-host-with-syslog.htmlYou can include the RRD data in the config to retain monitoring graphs of you need it. Assuming the drive is not damaged beyond extracting it that is.
Steve
-
It looks like I can still extract anything from the drive. I can attach it into another machine and copy without issue. I've moved the config.xml. I can move the Suricata logs. Is there a way to manually move the RRD info and firewall logs? Maybe I can look into a syslog server in the future.
-
@Stewart said in Manual Restore:
Is there a way to manually move the RRD info and firewall logs?
The RDD files can be copied as you did with the config.xml.
You can find them in the RDD folder : /var/db/rrdI wouldn't bother copying over the fixed size clog 'firewall' log files - keep them for later use - if needed (you probably won't).
-
Thanks. I'd just like to move over whatever I can so I don't lose the historical stuff. Ideally I'd love for it to be like running the backup wizard from the gui that gives a file I can move over. Otherwise, just to grab all the stuff I can.
-
I understand.
But the log files on pfSense are **auto rotated files over themselves ** or circular : the only contain the last x hours / days .
So, ok, for the info, but there isn't that much.If there are logs files to maintain you should have being using an external syslogger for long time now.
"Router" in genera have limited disk space and log files become huger or bigger - especially if you use packages like Snort or Suricate - people post often here because these explored the (space on) disk - pfSense dies ....edit : Oh ... you do have Suricata .... ;)
-
@Gertjan said in Manual Restore:
"Router" in genera have limited disk space and log files become huger or bigger - especially if you use packages like Snort or Suricate -
Well, yes and no. I can put in any amount of storage I want. If I reduce costs, albeit barely, and put in a 16GB or 32GB card then I'd be in trouble (especially with my settings). I use either 60GB or a 120GB (simply for cost) but I could easily go with a 1TB for around $100 more. That's plenty of storage.
@Gertjan said in Manual Restore:
- people post often here because these explored the (space on) disk - pfSense dies ....
I can see how that would happen, but the only time Suricata has caused that type of an issue to me is the bug a couple versions back where it didn't follow the set limits. Other than that my units have at least a 60GB mSATA SSD in them and I don't ever have an issue with space. This particular drive that is suspect has about 16GB of space free of the 50GB on the partition. I'm also using Squid and its cache and logs (11GB of usage) so i could easily pare it down if needed. It's been in the field over 2 years now and the space used has been pretty consistent around 65% usage.
If you want to talk about the ability to do something with the data or for archival storage then a separate system is certainly better. We set these up in all sorts of businesses but this company is only 7 employees. What's the point of the syslog server for them? Who will pay for it? Maintain it? Check it? It's much easier to just use local storage. To them, the old box has died and we had them back up with a new box and are happy. I'd just like to bring historical stuff over so it's all there. Suricata logs are nice but to see long term historical trends on data usage is valuable.
-
You can copy off all the logs from /var/log. However unless you have changed the log size they mostly do not cover a very large amount of time. The filter log especially often covers only a few hours in a firewall with a public WAN IP.
Steve
-
@stephenw10
I get that and for installs that need to keep all those logs then a syslog server makes sense. I wish there were a way to dictate how long the logs were stored for locally. The Status -> System Logs -> Settings page let you set the Maximum file size of each system log, but you never really know how long the logs are stored for. In that option, if you wanted the log file to be 1MB instead of the default 500KB, would you enter 1024000 for the Log File Size? -
The rotating format is to limit the file size. When you could install pfSense on a 512MB CF card they had to be limited. They still do if /var is in RAM. Of the time but not the size was specified a very busy log could grow huge potentially causing problems.
Yes the value is in Bytes so setting 1048576 will result in 1MiB files for each log. You can go bigger than that.Steve
-
@jimp said in Manual Restore:
Could be the SSD or RAM. Either way you'd get similar results.
He might try running memtest. It's a memory test utility. It can be installed directly to any bootable device. It's also included in the install media for many Linux distros. Start it up and let it run for several hours, to test the memory.
-
@Gertjan said in Manual Restore:
@Stewart said in Manual Restore:
Is there a way to manually move the RRD info and firewall logs?
The RDD files can be copied as you did with the config.xml.
You can find them in the RDD folder : /var/db/rrdI wouldn't bother copying over the fixed size clog 'firewall' log files - keep them for later use - if needed (you probably won't).
I copied over the /var/db/rrd files and it doesn't appear to have the historical information in there. When I look for the info they are practically empty. Weird. The file sizes range from 145K on WAN quality to 719K on system-memory but most are 384KB. Those feel small but looking at other units it's about the same. Are you certain that's where the info is stored? The unit was some version of 2.4.4.
@JKnott said in Manual Restore:
@jimp said in Manual Restore:
Could be the SSD or RAM. Either way you'd get similar results.
He might try running memtest. It's a memory test utility. It can be installed directly to any bootable device. It's also included in the install media for many Linux distros. Start it up and let it run for several hours, to test the memory.
I'm running the memtest now that's embedded into the APU2. So far 1 pass complete without any issues so we'll see.
-
@Stewart said in Manual Restore:
Are you certain that's where the info is stored?
Well .... test it yourself : on the box that you install, activate the RRD stats - I guess are already running for default interfaces by default.
So, your new pfSense will generate and maintain these RRD files. You will find them here :[2.4.4-RELEASE][admin@pfsense.brit-hotel-fumel.net]/var/db/rrd: ls -al *.rrd total 13560 drwxr-xr-x 2 nobody wheel 1536 May 21 16:40 . drwxr-xr-x 22 root wheel 1536 May 22 16:13 .. -rw-r--r-- 1 nobody wheel 147848 May 22 16:14 WAN_DHCP-quality.rrd -rw-r--r-- 1 nobody wheel 147848 Dec 10 10:26 WAN_PPPOE-quality.rrdx -rw-r--r-- 1 nobody wheel 197752 May 22 16:14 captiveportal-cpzone1-concurrent.rrd -rw-r--r-- 1 nobody wheel 197752 May 22 16:14 captiveportal-cpzone1-loggedin.rrd -rw-r--r-- 1 nobody wheel 393168 May 22 16:14 ipsec-packets.rrd -rw-r--r-- 1 nobody wheel 393168 May 22 16:14 ipsec-traffic.rrd -rw-r--r-- 1 nobody wheel 441864 May 22 16:14 lan-dhcpd.rrd -rw-r--r-- 1 nobody wheel 393168 May 22 16:14 lan-packets.rrd -rw-r--r-- 1 nobody wheel 393168 May 22 16:14 lan-traffic.rrd -rw-r--r-- 1 nobody wheel 882048 May 22 16:14 ntpd.rrd -rw-r--r-- 1 nobody wheel 441864 May 22 16:14 opt1-dhcpd.rrd -rw-r--r-- 1 nobody wheel 393168 May 22 16:14 opt1-packets.rrd -rw-r--r-- 1 nobody wheel 393168 May 22 16:14 opt1-traffic.rrd -rw-r--r-- 1 nobody wheel 393168 May 22 16:14 opt2-packets.rrd -rw-r--r-- 1 nobody wheel 393168 May 22 16:14 opt2-traffic.rrd -rw-r--r-- 1 nobody wheel 393168 May 22 16:14 opt3-packets.rrd -rw-r--r-- 1 nobody wheel 393168 May 22 16:14 opt3-traffic.rrd -rw-r--r-- 1 nobody wheel 393168 May 22 16:14 ovpns1-packets.rrd -rw-r--r-- 1 nobody wheel 393168 May 22 16:14 ovpns1-traffic.rrd -rw-r--r-- 1 nobody wheel 49720 May 22 16:14 ovpns1-vpnusers.rrd -rw-r--r-- 1 nobody wheel 588592 May 22 16:14 system-mbuf.rrd -rw-r--r-- 1 nobody wheel 735320 May 22 16:14 system-memory.rrd -rw-r--r-- 1 nobody wheel 245976 May 22 16:14 system-processor.rrd -rw-r--r-- 1 nobody wheel 245976 May 22 16:14 system-states.rrd -rw-r--r-- 1 nobody wheel 393168 May 22 16:14 wan-packets.rrd -rw-r--r-- 1 nobody wheel 393168 May 22 16:14 wan-traffic.rrdAs you can see, all files a have the same date / time, that's because they are all always updated on the same time.
Will it work ?
The file size difference already tels you one thing : depending on the number of parameters (== axes) like items to log, the file size will change dynamically.
So, if you have changed something yourself in the RRD graphs settings on the old pfSense, then this should be reflected into the new pfSense also, otherwise the RRD system can not use the RRD file you copied in ...
To get this info : see your old config.xml file for hints.By now you also understand why it's better to save a config.xml WITH the RRD files included. In that case, the settings to generated the included RRD files will always match - and the imported RRD files will get used for further updates.
Have a look again at the list above, and spot the "WAN_PPPOE-quality.rrdx" file : I abandoned PPPOE WAN to switch to WAN DHCP. This means that my "WAN_PPPOE-quality.rrdx" is an orphaned file, and I could throw it away - its not used (update, shown) by pfSense any-more..
-
I am trying it myself. I understand what you're saying and in theory I can see it working. The problem is that I've copied over the RRD files but the graphs have no historical data. Just flat lines. Do I need to stop a process before I copy them over? Do I need to do it in single-user mode?
Edit: And I can see many of your .rrd files are exactly the same size as mine. Wouldn't they dynamically expand as they hold more historical information? I know they are round robin databases but the information must be stored somewhere and as there is more and more information I would think that something would need to grow.
[2.4.4-RELEASE][root@pfSense.Domain]/var/db/rrd: ls -la total 5544 drwxr-xr-x 2 nobody wheel 1024 May 1 12:56 . drwxr-xr-x 20 root wheel 1024 May 10 00:27 .. -rw-r--r-- 1 nobody wheel 147848 Aug 25 2017 DEVWANGW-quality.rrd -rw-r--r-- 1 nobody wheel 147848 May 22 11:30 WANGW-quality.rrd -rw-r--r-- 1 nobody wheel 147848 Aug 25 2016 WAN_DHCP-quality.rrd -rw-r--r-- 1 nobody wheel 147848 Jul 5 2016 WAN_DHCP6-quality.rrd -rw-r--r-- 1 nobody wheel 393168 May 22 11:30 ipsec-packets.rrd -rw-r--r-- 1 nobody wheel 393168 May 22 11:30 ipsec-traffic.rrd -rw-r--r-- 1 nobody wheel 393168 May 22 11:30 lan-packets.rrd -rw-r--r-- 1 nobody wheel 393168 May 22 11:30 lan-traffic.rrd -rw-r--r-- 1 nobody wheel 393168 May 22 11:30 opt1-packets.rrd -rw-r--r-- 1 nobody wheel 393168 May 22 11:30 opt1-traffic.rrd -rw-r--r-- 1 nobody wheel 588592 May 22 11:30 system-mbuf.rrd -rw-r--r-- 1 nobody wheel 735320 May 22 11:30 system-memory.rrd -rw-r--r-- 1 nobody wheel 245976 May 22 11:30 system-processor.rrd -rw-r--r-- 1 nobody wheel 245976 May 22 11:30 system-states.rrd -rw-r--r-- 1 root wheel 6913 May 9 15:38 updaterrd.sh -rw-r--r-- 1 nobody wheel 393168 May 22 11:30 wan-packets.rrd -rw-r--r-- 1 nobody wheel 393168 May 22 11:30 wan-traffic.rrd -
It would be better to export the RRD data in the config if that's what you need. That's just an option on the backup/restore page. That way it will be imported to the new system correctly. I'm not sure sure I've ever tried just copying the files as you did there.
Steve
-
When I do a backup/restore I always do that and make sure I grab every option I have and it always moves over. At this point it doesn't really matter all that much as I'd be losing the recent stuff anyway. Now it's just a matter of how it can be done and if it works manually.
