Booting stuck on “Restoring contents from RAM store…”
-
Dear pfSense Gurus!
On one of my pfSense box (bare metal, 2.7.2-REL, 24Gb physical RAM, 3Gb ZFS swap, RAM disks ON, PHP mem limit 512, /tmp 500, /var 16000) suddenly increased /var directory.
And because of low space (1Mb on 124Gb HDD) PHP errors (from haproxy module) quickly start to filling log.
And of course SSH access not working.
P.S.
Suricata, ntopng - started.So…
After manual Reboot (normal mode, not re-root), pfSense box stuck to booting on string:… Restoring contents of RAM store…The quickest way may be just reinstall whole systems, but for some unknown reason pfSense installer from USB-flash cannot be able choose HDD RAID as for install (but successfully read config.xml and SSH keys from old install).
So my question are:
1.
Where and how delete unneeded pfSense caches (and may be some logs also) to be able to start pfSense normally ?How determine source why SWAP is filled so quickly?
Is the reason to SET SEPARATE PARTITION FOR LOGS AND CACHES (when making new installation)?
This mean even pfSense and packages filling this separate “cache” partition to 100%, the whole pfSense/FreeBSD continue to working well (and sending logs to separate local log server.)Thank You so much for detailed answering!
(and sorry for some may be stupid question…)—
CLOSE SKY FOR UKRAINE https://youtu.be/_tU1i8VAdCo !
Help Ukraine to resist, save civilians people’s lives !
(Take an active part in public protests, push on Your country’s politics, congressmans, mass media, leaders of opinion.) -
Check:
/cf/conf/RAM_Disk_StoreThat's what gets restored at boot.
A 10GB RAM disk is extreme. I'm not sure I've ever seen anyone using something that large.
Is that all logs?The only reason SWAP might be getting filled would be crash dumps. Otherwise pfSense should not use SWAP in general. Seeing it used usually indicates something using far too much RAM.
-
@stephenw10 said in Booting stuck on “Restoring contents from RAM store…”:
Check:
/cf/conf/RAM_Disk_StoreThat's what gets restored at boot.
Right now:
237M /cf/conf/RAM_Disk_StoreA 10GB RAM disk is extreme. I'm not sure I've ever seen anyone using something that large.
# du -h -d 1 /var/log 15M /var/log/pfblockerng …. 1.8M /var/log/snort … 692K /var/log/ntp 1.0G /var/log/suricata 2.5G /var/log …Is that all logs?
ls /var/log auth.log resolver.log dhcpd.log resolver.log.0 dhcpd.log.0 resolver.log.1 dhcpd.log.1 resolver.log.10 dhcpd.log.2 resolver.log.11 dhcpd.log.3 resolver.log.12 dhcpd.log.4 resolver.log.13 dhcpd.log.5 resolver.log.14 dmesg.boot resolver.log.15 filter.log resolver.log.16 filter.log.0 resolver.log.17 filter.log.1 resolver.log.18 filter.log.10 resolver.log.19 filter.log.11 resolver.log.2 filter.log.12 resolver.log.20 filter.log.13 resolver.log.21 filter.log.14 resolver.log.22 filter.log.15 resolver.log.23 filter.log.16 resolver.log.24 filter.log.17 resolver.log.25 filter.log.18 resolver.log.26 filter.log.19 resolver.log.27 filter.log.2 resolver.log.28 filter.log.20 resolver.log.29 filter.log.21 resolver.log.3 filter.log.22 resolver.log.30 filter.log.23 resolver.log.31 filter.log.24 resolver.log.32 filter.log.25 resolver.log.33 filter.log.26 resolver.log.34 filter.log.27 resolver.log.35 filter.log.28 resolver.log.36 filter.log.29 resolver.log.37 filter.log.3 resolver.log.38 filter.log.30 resolver.log.39 filter.log.31 resolver.log.4 filter.log.32 resolver.log.40 filter.log.33 resolver.log.41 filter.log.34 resolver.log.42 filter.log.35 resolver.log.43 filter.log.36 resolver.log.44 filter.log.37 resolver.log.45 filter.log.38 resolver.log.46 filter.log.39 resolver.log.47 filter.log.4 resolver.log.48 filter.log.40 resolver.log.49 filter.log.41 resolver.log.5 filter.log.42 resolver.log.50 filter.log.43 resolver.log.51 filter.log.44 resolver.log.52 filter.log.45 resolver.log.53 filter.log.46 resolver.log.54 filter.log.47 resolver.log.55 filter.log.48 resolver.log.56 filter.log.49 resolver.log.57 filter.log.5 resolver.log.58 filter.log.50 resolver.log.59 filter.log.51 resolver.log.6 filter.log.52 resolver.log.60 filter.log.53 resolver.log.61 filter.log.54 resolver.log.62 filter.log.55 resolver.log.63 filter.log.56 resolver.log.64 filter.log.57 resolver.log.65 filter.log.58 resolver.log.66 filter.log.59 resolver.log.67 filter.log.6 resolver.log.68 filter.log.60 resolver.log.69 filter.log.61 resolver.log.7 filter.log.62 resolver.log.70 filter.log.63 resolver.log.71 filter.log.64 resolver.log.72 filter.log.65 resolver.log.73 filter.log.66 resolver.log.74 filter.log.67 resolver.log.75 filter.log.68 resolver.log.76 filter.log.69 resolver.log.77 filter.log.7 resolver.log.78 filter.log.70 resolver.log.79 filter.log.71 resolver.log.8 filter.log.72 resolver.log.80 filter.log.73 resolver.log.81 filter.log.74 resolver.log.82 filter.log.75 resolver.log.83 filter.log.76 resolver.log.84 filter.log.77 resolver.log.85 filter.log.78 resolver.log.86 filter.log.79 resolver.log.87 filter.log.8 resolver.log.88 filter.log.80 resolver.log.89 filter.log.81 resolver.log.9 filter.log.82 resolver.log.90 filter.log.83 resolver.log.91 filter.log.84 resolver.log.92 filter.log.85 resolver.log.93 filter.log.86 resolver.log.94 filter.log.87 resolver.log.95 filter.log.88 resolver.log.96 filter.log.89 resolver.log.97 filter.log.9 resolver.log.98 filter.log.90 restore_ramdisk_store.boot filter.log.91 routing.log filter.log.92 snort filter.log.93 suricata filter.log.94 system.log filter.log.95 system.log.0 filter.log.96 system.log.1 filter.log.97 system.log.10 filter.log.98 system.log.11 gateways.log system.log.12 haproxy.log system.log.13 ipsec.log system.log.2 l2tps.log system.log.3 lastlog system.log.4 nginx system.log.5 nginx.log system.log.6 ntp system.log.7 ntpd.log system.log.8 openvpn.log system.log.9 pfblockerng telegraf.log poes.log tinc.log portalauth.log userlog ppp.log utx.lastlogin radacct utx.log radutmp vpn.log radwtmp wireless.logRight now pfSense server state:
The only reason SWAP might be getting filled would be crash dumps. Otherwise pfSense should not use SWAP in general. Seeing it used usually indicates something using far too much RAM.
So… How to check this properly? By Monitoring menu? ;)
-
You should be using an external syslog server if you need that much logging. That is a huge amount of logs top have on pfSense itself.
-
@stephenw10 said in Booting stuck on “Restoring contents from RAM store…”:
You should be using an external syslog server if you need that much logging. That is a huge amount of logs top have on pfSense itself.
Agree.
But the question was “how to eliminate amount of this logs” without reducing log details to prevent filling /var ?
Only by custom sh/bash croned script?
-
@Sergei_Shablovsky said in Booting stuck on “Restoring contents from RAM store…”:
without reducing log details
How do you mean 'details'?
Not sure what you're trying to achieve. It failed to boot because it got stick trying to restore a 10GB ram drive. You can just clear that file so it boots.
-
@stephenw10 said in Booting stuck on “Restoring contents from RAM store…”:
@Sergei_Shablovsky said in Booting stuck on “Restoring contents from RAM store…”:
without reducing log details
How do you mean 'details'?
I mean not to reduce whole logs amount by switching OFF logging for some services (no matter snort/suricata, fw errors, or fw rules).
Not sure what you're trying to achieve. It failed to boot because it got stick trying to restore a 10GB ram drive. You can just clear that file so it boots.
I try to avoid fw stuck in a future when /var again being totally filled by logs.
Thank You for patience and help!
-
@Sergei_Shablovsky said in Booting stuck on “Restoring contents from RAM store…”:
Where and how delete unneeded pfSense caches (and may be some logs also) to be able to start pfSense normally ?
How determine source why SWAP is filled so quickly?
As always : use the console access !
( Or ssh, and use a SSH client like putty and/or WinSCP)and visit every folder and sub folder in /var/
You'll find out quickly what are the big files, and what are the files that grow rapidly.Btw : the day you've decides to use "Suricata, ntopng" you also signed up a permanent (!), manually ( !) inspection of the folders where these process log. Running out of space with these two - and some others - is a very commun issue.
You have a 124G SSD : why use a RAM disk ?
No "help me" PM's please. Use the forum, the community will thank you.
Edit : and where are the logs ?? -
Yup, I would argue you don't need a RAM disk at all.
There is really no way around it; if you need that level of logging and that much data retention you should be using an external syslog server whether or not you use RAM disks. pfSense was never designed to store logs like that.
-
Glad to read You!
@Gertjan said in Booting stuck on “Restoring contents from RAM store…”:
@Sergei_Shablovsky said in Booting stuck on “Restoring contents from RAM store…”:
Where and how delete unneeded pfSense caches (and may be some logs also) to be able to start pfSense normally ?
How determine source why SWAP is filled so quickly?
As always : use the console access !
( Or ssh, and use a SSH client like putty and/or WinSCP)and visit every folder and sub folder in /var/
You'll find out quickly what are the big files, and what are the files that grow rapidly.I already using SSH, and VERY useful Termius + SSH Editor (iOS/macOS).
VERY useful apps, take a look!And command
du -h -d 1 /var/logto see
and
yes | rm -IPR /var/log/*to remove all (or the same modified for certain dir)
Btw : the day you've decides to use "Suricata, ntopng" you also signed up a permanent (!), manually ( !) inspection of the folders where these process log. Running out of space with these two - and some others - is a very commun issue.
Hm.
If this is CONSTANTLY EXISTED PROBLEM (but for 7+ years I see this is constant problem that pop-up again and again), may be MUCH BETTER to create redmine ticket to ask Netgate to create settings:
Notifications about dusk space:
When free disk space are below [25%…..]
(You may entering amount of disk space in Gb or %. For example “15%”, “40Gb”)
When size of /var/log system and packages logs directory are more than [40Gb]
(You may entering dir size in “Gb” or “Mb”)If pfSense admin are serious about their pfSense installation, BOTH LICAL LOGGING and LIGGING ON REMOTE SERVER ARE MUST HAVE!
You have a 124G SSD : why use a RAM disk ?
Sorry, my mistyping. HDD on this pfSense.
-
@stephenw10 said in Booting stuck on “Restoring contents from RAM store…”:
Yup, I would argue you don't need a RAM disk at all.
But /var on SSD in comparison with RAM are MORE slower.
Am I wrong ?
-
It is slower but it's not usually significantly. It can help a lot on slow storage like CF cards but SSDs are already fast enough that the you're unlikely to see much difference.
-
@stephenw10 said in Booting stuck on “Restoring contents from RAM store…”:
It is slower but it's not usually significantly. It can help a lot on slow storage like CF cards but SSDs are already fast enough that the you're unlikely to see much difference.
Even on 10G throughput with snort/suricata and a lot of logging ?
-
Well as far as I know, yes. But I've never tested that directly. I doubt the logging would be slowing anything there though. Running Snort against that is going to be slower.
-
@stephenw10 said in Booting stuck on “Restoring contents from RAM store…”:
Well as far as I know, yes. But I've never tested that directly. I doubt the logging would be slowing anything there though. Running Snort against that is going to be slower.
Ok, thank You!
When traffic would be rising up. of coarse, I remove Snort/Suricata on SEPARATE server where incoming traffic from all WLANs would be mirrored.
But I need to be sure that Snort/Suricata would be able to keep throughput to be able to instructing pfSense creating BLOCK records in a rules...
Did You know great source with detailed explanation how to make this installation: pfSense on one bare metal server + Snort/Suricata on other bare metal server ?
—
CLOSE SKY FOR UKRAINE https://youtu.be/_tU1i8VAdCo !
Help Ukraine to resist, save civilians people’s lives !
(Take an active part in public protests, push on Your country’s politics, congressmans, mass media, leaders of opinion.) -
Hmm, not sure I've seen that specifically. I'm not sure how you would arrange the block rules to be sent between them. The pfSense Snort package includes some custom code to make that happen.
-
@Sergei_Shablovsky said in Booting stuck on “Restoring contents from RAM store…”:
But I need to be sure that Snort/Suricata would be able to keep throughput to be able to instructing pfSense creating BLOCK records in a rules...
This is not possible. Neither package supports "remote" blocking. The code in the pfSense packages is written to communicate with only the local
pffirewall engine.What you could do with Suricata (or Snort) is use the standard FreeBSD package (not the GUI version provided with pfSense) and configure the package to use Inline IPS Mode on the separate server. All management and interaction would have to be done on the local server through the shell interface as there would be no GUI. You would use two NIC ports on the separate server and connect them inline between the pfSense LAN connection and the master LAN switch. But this configuration is totally outside of pfSense and you would be on your own to configure it. And I would suggest going this route that you use Linux on the separate server and install a Suricata package from the packages tree for the particular Linux distro you choose. With Linux you have the option of using DPDK or AF_PACKET for the IPS mode in Suricata. With Snort you are limited to an older netmap interface.
-
Thank You for detailed answering!
@bmeeks said in Booting stuck on “Restoring contents from RAM store…”:
@Sergei_Shablovsky said in Booting stuck on “Restoring contents from RAM store…”:
But I need to be sure that Snort/Suricata would be able to keep throughput to be able to instructing pfSense creating BLOCK records in a rules...
This is not possible. Neither package supports "remote" blocking. The code in the pfSense packages is written to communicate with only the local
pffirewall engine.Thank You for confirming my toughs.
Also I already have a plan on this site (when overall traffic bandwidth goes up, and tuning IDS/IPS rules would be more complete) to take Snort/Suricata out from pfSense itself and leave pfSense only for FW/ROUTE/VPN needs.
What you could do with Suricata (or Snort) is use the standard FreeBSD package (not the GUI version provided with pfSense) and configure the package to use Inline IPS Mode on the separate server. All management and interaction would have to be done on the local server through the shell interface as there would be no GUI.
Clearly understand. Thank You!
You would use two NIC ports on the separate server and connect them inline between the pfSense LAN connection and the master LAN switch.
Please explain me how to realize this in case when for example pfSense server
- have 4 WANs (working simultaneously, balanced by Tiers);
- have 8 LANs (office, public web services, etc.)
Is this mean on this separate Snort/Suricata server I need 16 (2 x 8, for inspecting traffic) + 1 for SecAdmins management?
But this configuration is totally outside of pfSense and you would be on your own to configure it. And I would suggest going this route that you use Linux on the separate server and install a Suricata package from the packages tree for the particular Linux distro you choose. With Linux you have the option of using DPDK or AF_PACKET for the IPS mode in Suricata. With Snort you are limited to an older netmap interface.
Why exactly Linux (and which one ? RHEL, Debian?) and not FreeBSD ?
Thank You so much for detailed answering!
Have a nice day! -
@Sergei_Shablovsky said in Booting stuck on “Restoring contents from RAM store…”:
Is this mean on this separate Snort/Suricata server I need 16 (2 x 8, for inspecting traffic) + 1 for SecAdmins management?
Yes. It will take two separate NIC ports per pathway to implement. Think of it as a transparent firewall "bridge" of sorts. Suricata sits between two NIC ports (directly) and either forwards or drops particular packets between those two ports.
You could consider splitting the load across two mostly identical servers (4 complete pathways on each server). 8 Suricata instances inspecting a lot of traffic against many rules is going to be resource intensive. Splitting that across multiple servers might work better performance wise. You will want multi-queue NICs and high core-count CPUs and lots of RAM.
@Sergei_Shablovsky said in Booting stuck on “Restoring contents from RAM store…”:
Why exactly Linux (and which one ? RHEL, Debian?) and not FreeBSD ?
Mostly because Suricata is primarily developed and debugged on Linux platforms and thus has excellent support there. While the Suricata team does compile and test on FreeBSD, they must do that manually because none of their automated testing tools work on FreeBSD. And none of them that I know run Suricata on FreeBSD themselves.
Another reason is that the AF_PACKET interface is quite well established on Linux and less buggy than the netmap interface in FreeBSD.
These are the available IPS options on Linux: https://docs.suricata.io/en/suricata-7.0.5/setting-up-ipsinline-for-linux.html.
I don't think it really matters about the Linux distro. Just choose one you might already be familiar with.
-
Thank You for patience and detailed answering!
So, let’s dive in ;)@bmeeks said in Booting stuck on “Restoring contents from RAM store…”:
@Sergei_Shablovsky said in Booting stuck on “Restoring contents from RAM store…”:
Is this mean on this separate Snort/Suricata server I need 16 (2 x 8, for inspecting traffic) + 1 for SecAdmins management?
Yes. It will take two separate NIC ports per pathway to implement. Think of it as a transparent firewall "bridge" of sorts. Suricata sits between two NIC ports (directly) and either forwards or drops particular packets between those two ports.
Adding 2 multi-CPU is not a problem for us.
More important-
You could consider splitting the load across two mostly identical servers (4 complete pathways on each server). 8 Suricata instances inspecting a lot of traffic against many rules is going to be resource intensive. Splitting that across multiple servers might work better performance wise.
You will want multi-queue NICs and high core-count CPUs and lots of RAM.
@Sergei_Shablovsky said in Booting stuck on “Restoring contents from RAM store…”:
Why exactly Linux (and which one ? RHEL, Debian?) and not FreeBSD ?
Mostly because Suricata is primarily developed and debugged on Linux platforms and thus has excellent support there. While the Suricata team does compile and test on FreeBSD, they must do that manually because none of their automated testing tools work on FreeBSD. And none of them that I know run Suricata on FreeBSD themselves.
Another reason is that the AF_PACKET interface is quite well established on Linux and less buggy than the netmap interface in FreeBSD.
These are the available IPS options on Linux: https://docs.suricata.io/en/suricata-7.0.5/setting-up-ipsinline-for-linux.html.
I don't think it really matters about the Linux distro. Just choose one you might already be familiar with.
