Store pfSense (+ packages logs) on external (or internal) USB drive / memory card
-
@jimp said in Store pfSense (+ packages logs) on external (or internal) USB drive / memory card:
If a device is compromised, you can't trust logs from that compromised device. It doesn't matter what level of access you believe they obtained, you have no idea if they escalated from there and covered their tracks.
Absolutely agree with this!
Unless they compromised the syslog server and/or intermediate devices (e.g. took out the network), the logs sent over the network to a syslog server will always be more reliable than logs on the device itself. And in cases of local storage failure, the only way to find that out would be logs sent over the network. The best solution is always local short term logs + long term remote logs. Any logs that happen in the event of a network failure can be obtained from the device itself, but normal logs are viewed/processed centrally.
Exactly about this I am writing: ability to make scheduled encrypted archives + external syslog. Two different procedures for two different purposes: local copy for improving security and in case device main HD/SDD crashed (primary for home users, small campuses, etc...) and remote more powerful log aggregator and analyser (like Splunk) for constantly monitoring and alerting tech stuff.
Local scheduled copying is not for replace remote log aggregator/analyzer, but more like additional security layer for part of users ordinary level.If you want to be pedantic, the only real secure log would be one put on write-only/WORM type storage which isn't really a thing these days. In the old days with low volume logs they might have even been printed line-by-line on a dot matrix style printer (which has its own downsides). But no matter which method you choose, a clever attacker could likely find a way to compromise it or at least render it ineffectual.
No one of us need to be so mad. :) But I remember that times, not so long time ago... :)
Agree.If you want to be lazy and not setup a central log server, just own up to that. Plenty of people are in the same situation. I'm sure there are others who would like to see this request implemented. But you just aren't going to convince anyone who disagrees with this idea in principle to do the work of adding this feature for you. If someone comes along and submits a PR to allow setting a custom log storage/rotation directory on a future version, we'd happily review it.
It's not about "i have no time and passion to install syslog server, but have a time and passion to flooding forum". This tread about "making scheduled creating logs archive on a local USB-drive as a part of standard pfSense feature".
As for me personally - I am "totally on Splunk side". All what I am asking in this thread - making local copy archived logs as standard feature of pfSense. :)From ordinary user side all looks like "Apple magic": they unpack NetGate device, connect cables, go thru the Wizard (or install pfSense on own appliance) -> insert USB-flash -> all working. And all updates, patches makes remotely, not need all time to be physically near appliance.
I am sure that not small part from 600.000+ pfSense users are happy to have a local copy of archived logs.P.S. Sorry for nub question, what is "submits a PR". ? I start to writing script, with cron work but ned more time for this...
-
@jimp said in Store pfSense (+ packages logs) on external (or internal) USB drive / memory card:
The best solution is always local short term logs + long term remote logs. Any logs that happen in the event of a network failure can be obtained from the device itself, but normal logs are viewed/processed centrally.
Need to add that using Automatic Configuration Backup (ACB) with conjunction with remote log aggregator/analyzer is the "remote backup", for purpose of backup archived logs on a local removable media better to use separate feature in pfSense.
And of course "ACB with Gold subscription" = paid remote backup option, when "backup archived logs on a local removable media" = free local option only for logs. (for manual local Configuration Backup already exist separate package where possible to adding any paths/files).
-
It is possible to do this already using the syslog-ng package. You can configure that to store it's logs in any location including some other drive like a memory card. Then just configure the main logs to export to it as well.
The difficulty is that pfSense has no facility for managing additional drives. They are not auto-mounted etc. You can just add new devices to the fstab but what happens if you pull the memory card?
There have been a few scripts written to address this. When NanoBSD was a thing local logs were RAM only so several people wrote stuff to use a separate drive for logging only.
Steve
-
@stephenw10 said in Store pfSense (+ packages logs) on external (or internal) USB drive / memory card:
It is possible to do this already using the syslog-ng package. You can configure that to store it's logs in any location including some other drive like a memory card. Then just configure the main logs to export to it as well.
Please take attention “storing encrypted archives of logs on a local media”. This is different that “just copy logs to another drive”. ;)
The difficulty is that pfSense has no facility for managing additional drives. They are not auto-mounted etc. You can just add new devices to the fstab but what happens if you pull the memory card?
There have been a few scripts written to address this. When NanoBSD was a thing local logs were RAM only so several people wrote stuff to use a separate drive for logging only.
Steve
Please wait, I just starting writing scripts. Cron, daemon, etc... -
Mmm, reading through the syslog-ng manuals I think your need the Premium Edition to store logs encrypted.
Steve
-
@stephenw10 said in Store pfSense (+ packages logs) on external (or internal) USB drive / memory card:
Mmm, reading through the syslog-ng manuals I think your need the Premium Edition to store logs encrypted.
Steve
Sorry, what You mean ?I told about storing encrypted archive with logs LOCALLY. This is the first level for advanced users or home users, that not need store logs in a remote place for various reasons.
-
Indeed, I thought you might be able to do that with syslog-ng since they do have that feature. But it looks to be for PE only so not in the FreeBSD port.
Steve
-
use the syslog package, I am using it to send logs from my AP to pfSense however you can also forward syslogs out of pfsense
-
I was thinking…
What about you mount a drive like a second SSD or a mpcie to m.2 and have a NVMe drive that you mount to something like /root/logs use gpart to make a partition on that other drive and mount to it with fstab automatically, after use it for snort logs and squid or any other package that lets you pick what location you log to… what would cut down a lot. I mean you can copy to usb drive with fat32 partition why not just do the same thing and make it a log partition ? Wouldn’t that work?
I got this monster Optane drive and I am using a small part as an emergency swap location so it has 255gb I could add a new partition to it with gpart and rock that for a log directory. Any thoughts ?
-
Yup that can work. The problem is if you have to re-install or at upgrade it may get overwritten. And then how does the package behave if its unable to reach it's log location.
-
@stephenw10 does it have to be fat32? I can't get it to mount the zfs manually
Shell Output - mount /dev/nda0p2 /root/LOGS_Optane mount: /dev/nda0p2: Integrity check failedShell Output - gpart show nda0 => 40 500118112 nda0 GPT (238G) 40 2008 - free - (1.0M) 2048 16777216 1 freebsd-swap (8.0G) 16779264 482344960 2 freebsd-zfs (230G) 499124224 993928 - free - (485M)I researched it and found how to do it it needs efi for GPT I have learned that way you can make a fat32 that pfSense can allow use of. I think it is restricted to only fat32 for external stuff correct me if I am wrong
It needs to have this done
gpart add -t efi -s 230GB -l LOG nda0 newfs_msdos -F 32 /dev/nda0p2 mount_msdosfs /dev/nda0p2 /root/LOGS_OptaneShell Output - gpart show nda0 => 40 500118112 nda0 GPT (238G) 40 2008 - free - (1.0M) 2048 16777216 1 freebsd-swap (8.0G) 16779264 482344960 2 efi (230G) 499124224 993928 - free - (485M)It works I can mount it send files to it after this you create a cron to mount it
Warning this can break stuff if you do not know what your doing here for others I am doing a new partition inside a second drive that I am also using as swap so be careful to not wipe out your drives
-
Updated my unofficial guide if anyone else wants to try this here is a short guide for you.
https://forum.netgate.com/topic/195843/unofficial-guide-have-package-logs-record-to-a-secondary-ssd-drive-snort-syslog-squid-and-or-squid-cache-system
