Store pfSense (+ packages logs) on external (or internal) USB drive / memory card

Sergei_Shablovsky

Hi, pfSense folks!

All we not care so much about logs BEFORE some troubles happened.
And only after troubles we start seeking thru the logs to find saver on question “what exactly happened”.

Most advanced hardware like Juniper, Extreme, Cisco, F5, and others for a long time have slot for external Memory Card, mostly “Industrial Grade”.

So my request are simple: How to store pfSense (+ packages logs, of course) on external (or internal) USB drive / memory card ?

Logically, log daemon must each time not only

cutting current log;
creating tar/zip archive of previous log;
and also
copy previously created tar/zip to external (or internal) Memory Card;

I am strongly sure that log saving functionality must be as part of pfSense core nope ax external optional package.
And of course during installation “advanced” procedure must be option to choose on which media (different with that pfSense installed on) user want to save logs and parameters of archive

format;
size;
overwrite when media full;
notify admin if media full;
notify admin if writing on media fault / media fault or not existed at all (but previously exist));

P.S. Last years appliance manufacturers try to using USB-flash drive for log saving due lack of small size (128, 512 Mb, even 1 Gb)Memory Card Industrial Grade on a market, but this is not change sense of my question.

Than You for attention!

NogBadTheBad

Most network equipment sends its logs to a syslog server.

I'd suggest you do the same.

jimp

2.5.0 has changed from the binary circular log format over to standard syslog plain text format with compression+rotation : https://redmine.pfsense.org/issues/8350

The log store directory is still not something you can choose to relocate, however. You might be able to script something to copy the archive files over to another drive periodically but it wouldn't be reliable.

The only acceptable solution here is to use an external syslog server. The firewall is a firewall, not a log server and management platform. If you wish to maintain logs long-term, they must be stored centrally in a log server.

Sergei_Shablovsky

@NogBadTheBad said in Store pfSense (+ packages logs) on external (or internal) USB drive / memory card:

Most network equipment sends its logs to a syslog server.

I'd suggest you do the same.

This not mean not to keep local logs copy. Follow Your logic way, Cisco and other are looser and need to cut their appliances to have Memory Card slot. :)

Sergei_Shablovsky

@jimp said in Store pfSense (+ packages logs) on external (or internal) USB drive / memory card:

2.5.0 has changed from the binary circular log format over to standard syslog plain text format with compression+rotation : https://redmine.pfsense.org/issues/8350

The log store directory is still not something you can choose to relocate, however. You might be able to script something to copy the archive files over to another drive periodically but it wouldn't be reliable.

Understand Your point. But why I need to make some workaround (and take care of all of it in case pfSense update each time) instead of having the standard feature like in any other serious firewalls?

The only acceptable solution here is to use an external syslog server. The firewall is a firewall, not a log server and management platform. If you wish to maintain logs long-term, they must be stored centrally in a log server.

Again one time: follow this logic Juniper, F5, Cisco must cutting their Memory Card slot in appliance ? What a reason of this ?

If something happened with Your network (intrusion happened, system fault, cable fault, syslog server fault, etc, etc...) You lost logs and not able to detect what happened. And next time attacks come again. And again You cannot see what happened.

Firewall is Firewall, You are right. And ability to see what exactly happened - is one of the main advantages of system.

Sergei_Shablovsky

@jimp said in Store pfSense (+ packages logs) on external (or internal) USB drive / memory card:

The only acceptable solution here is to use an external syslog server. The firewall is a firewall, not a log server and management platform. If you wish to maintain logs long-term, they must be stored centrally in a log server.

As You able to see most of users here on forum using pfSense in small/middle networks, even home or campus networks, and pushing them to making separate syslog server - may be too much :) Look at the thru: none of them doing this. :)

So the ability to save logs on Memory Card / USB device - great benefit for them. With a fraction of additional work for pfSense developers.

NogBadTheBad

@Sergei_Shablovsky said in Store pfSense (+ packages logs) on external (or internal) USB drive / memory card:

@NogBadTheBad said in Store pfSense (+ packages logs) on external (or internal) USB drive / memory card:

Most network equipment sends its logs to a syslog server.

I'd suggest you do the same.

This not mean not to keep local logs copy. Follow Your logic way, Cisco and other are looser and need to cut their appliances to have Memory Card slot. :)

Exactly what Cisco devices, give me some examples, the memory card slots are for IOS.

Normally Cisco logs are stored in DRAM.

Sergei_Shablovsky

@NogBadTheBad said in Store pfSense (+ packages logs) on external (or internal) USB drive / memory card:

@Sergei_Shablovsky said in Store pfSense (+ packages logs) on external (or internal) USB drive / memory card:

@NogBadTheBad said in Store pfSense (+ packages logs) on external (or internal) USB drive / memory card:

Most network equipment sends its logs to a syslog server.

I'd suggest you do the same.

This not mean not to keep local logs copy. Follow Your logic way, Cisco and other are looser and need to cut their appliances to have Memory Card slot. :)

Exactly what Cisco devices, give me some examples, the memory card slots are for IOS.

I hate to make a tech-war, but if You asking...

Please, for You attention:

ASA5510 series
ASA5520 series
ASA5540 series

Normally Cisco logs are stored in DRAM.

You may understanding, from security manager point of view the logs on a FW (no matter, main gate, or application, or IDS) are only one point where we able to determine some initial details how attacks was made.

Of course, no any sense to controlling .log savings from pfSense freeBSD user rights, because pfSense are on top of FreeBSD.
Only one way reasonable -> copying tar/zip of .log on separate external MemoryCard from root FreeBSD user.

Even hacker/attacker broke pfSense (directly, or thru 3rd party package/extension - more usable way our days), and correct .log files to eliminate his presence, he cannot act as FreeBSD root User and delete/correct copy of archived .log files.

Reasonable?

provels

@Sergei_Shablovsky I think you're a solution in search of a problem. Here's some free syslog servers. https://www.ittsystems.com/best-free-syslog-server-windows/

NogBadTheBad

@Sergei_Shablovsky said in Store pfSense (+ packages logs) on external (or internal) USB drive / memory card:

ASA5540

Those are all eol devices and logs would be written to dram.

Anybody with their head screwed on would log to a central syslog server and then use Splunk / Elasticsearch to drill down into the data.

Sorry but I and may others will fail to see why you need the logs on the router itself.

Sergei_Shablovsky

@provels said in Store pfSense (+ packages logs) on external (or internal) USB drive / memory card:

@Sergei_Shablovsky I think you're a solution in search of a problem. Here's some free syslog servers. https://www.ittsystems.com/best-free-syslog-server-windows/

At the first, thank You for advise.

May be I am not so correctly writing my thoughts. "Problem" is not free or paid syslog server are. Please read carefully all thread: main goal are to have a backup copy of archived logs on separate, industrial grade, media (USB-dongle or Memory Card).

All that I asking for is exactly to make ability to doing logs archive copying on a core of pfSense level rather making some handmade workaround with manual written script, and add crone task.
Making things usable mean give all users ability to use them. :)

Sergei_Shablovsky

@NogBadTheBad said in Store pfSense (+ packages logs) on external (or internal) USB drive / memory card:

@Sergei_Shablovsky said in Store pfSense (+ packages logs) on external (or internal) USB drive / memory card:

ASA5540

Those are all eol devices and logs would be written to dram.

Sorry, I have no time and passion o make some sort of tech battle here and spend time to create list of models, I just write some that I remember. If You stay on Your position, ok, I'l spend a time and create a list especially for You. But I still thinking common sense are better.
Many users here on pfSense forum using eol, old routers to install and running pfSense. So called "old" routers, mostly are very good engineered enterprise hardware dramatically low priced (eBay, etc...). This is reality. :)
Before write, I spend some time to sure that most of questions on this forum during last 2 years not related to very fresh hardware.
Please read my posts with possible attention: I write "last years modern fw/routers/balancers/IDs using USB-flash instead Memory Card". If You read User Manual for most of systems that are stay in front of entire network (FW, balancers, IDs, ...) from Juniper, Extreme, f5, Cisco, they have ability to using USB-dongle in USB port as media for archived log saving. And this is not an option, this are recommended.

Anybody with their head screwed on would log to a central syslog server and then use Splunk / Elasticsearch to drill down into the data.

Sorry but I and may others will fail to see why you need the logs on the router itself.
Splunk are really perfect software for log analyzing. A am strongly stay on that.

You may understanding, from security manager point of view the logs on a FW (no matter, main gate, or application, or IDS) are only one point where we able to determine some initial details how attacks was made.

If attacker/hacker compromise entry FW (balancer, IDs, etc...) on a software level, but not the root, there are only one way to know about his steps: see to system logs. The first things that attacker doing - disabling or correct logging in compromised system. How syslog server help in this?
If attacker not able to delete/correct archived logs - this is really good point to start investigate for system security manager.

Sergei_Shablovsky

@NogBadTheBad said in Store pfSense (+ packages logs) on external (or internal) USB drive / memory card:

Sorry but I and may others will fail to see why you need the logs on the router itself.

Not "logs on a router/firewall", but "archived logs on independent removable industrial grade media like USB-dongle".
Of course, FreeBSD root user rights on write archives, to eliminate possibilities to compromise system thru 3rd packages or pfSense core, or 0-day FreeBSD vulnerabilities.

provels

@Sergei_Shablovsky FWIW, I just setup the free version Kiwi Syslog in about 10 minutes and have real-time live logs going to it, and I'm definitely not the sharpest knife in the drawer. It can send the logs to whatever server/storage device I want for analysis from USB flash to multi-terabyte RAID, no scripting or cron jobs required. Just point pfSense to the syslog server's IP and choose the logs you want sent. That sounds like a little better solution to me. If an attacker is already crawling all over your network changing archived syslog entries, you have very big problems.

jimp

If a device is compromised, you can't trust logs from that compromised device. It doesn't matter what level of access you believe they obtained, you have no idea if they escalated from there and covered their tracks.

Unless they compromised the syslog server and/or intermediate devices (e.g. took out the network), the logs sent over the network to a syslog server will always be more reliable than logs on the device itself. And in cases of local storage failure, the only way to find that out would be logs sent over the network. The best solution is always local short term logs + long term remote logs. Any logs that happen in the event of a network failure can be obtained from the device itself, but normal logs are viewed/processed centrally.

If you want to be pedantic, the only real secure log would be one put on write-only/WORM type storage which isn't really a thing these days. In the old days with low volume logs they might have even been printed line-by-line on a dot matrix style printer (which has its own downsides). But no matter which method you choose, a clever attacker could likely find a way to compromise it or at least render it ineffectual.

If you want to be lazy and not setup a central log server, just own up to that. Plenty of people are in the same situation. I'm sure there are others who would like to see this request implemented. But you just aren't going to convince anyone who disagrees with this idea in principle to do the work of adding this feature for you. If someone comes along and submits a PR to allow setting a custom log storage/rotation directory on a future version, we'd happily review it.

Sergei_Shablovsky

@provels said in Store pfSense (+ packages logs) on external (or internal) USB drive / memory card:

@Sergei_Shablovsky FWIW, I just setup the free version Kiwi Syslog in about 10 minutes and have real-time live logs going to it, and I'm definitely not the sharpest knife in the drawer. It can send the logs to whatever server/storage device I want for analysis from USB flash to multi-terabyte RAID, no scripting or cron jobs required. Just point pfSense to the syslog server's IP and choose the logs you want sent. That sounds like a little better solution to me. If an attacker is already crawling all over your network changing archived syslog entries, you have very big problems.

There are some CONS about Your solution:

Logs = important part of security, only one way to know "what exactly happened BEFORE the problem". More additional software or hardware You using = more points of potential failure You have. From this point of view extra Kiwi Syslog mean "another one bunch of code, another one point in update schedule, another one source of new bugs, another one point of possible misscompability with bunch of other software You have". This is not about Kiwi exactly, this is about ANY extra software you add to Your system.
Why using extra software from 3rd party instead of function that are standard for each serious fw/router software? This is out common sense. Ok, if You make experiments or advanced setup for Your home router, but even in NetGate hardware list only first 2 models intended for small speeds, other hardware are much more powerful and intended to be using in professional environment, where security are much more important than in home/campus setup.

Sergei_Shablovsky

@jimp said in Store pfSense (+ packages logs) on external (or internal) USB drive / memory card:

If a device is compromised, you can't trust logs from that compromised device. It doesn't matter what level of access you believe they obtained, you have no idea if they escalated from there and covered their tracks.

Absolutely agree with this!

Unless they compromised the syslog server and/or intermediate devices (e.g. took out the network), the logs sent over the network to a syslog server will always be more reliable than logs on the device itself. And in cases of local storage failure, the only way to find that out would be logs sent over the network. The best solution is always local short term logs + long term remote logs. Any logs that happen in the event of a network failure can be obtained from the device itself, but normal logs are viewed/processed centrally.

Exactly about this I am writing: ability to make scheduled encrypted archives + external syslog. Two different procedures for two different purposes: local copy for improving security and in case device main HD/SDD crashed (primary for home users, small campuses, etc...) and remote more powerful log aggregator and analyser (like Splunk) for constantly monitoring and alerting tech stuff.
Local scheduled copying is not for replace remote log aggregator/analyzer, but more like additional security layer for part of users ordinary level.

If you want to be pedantic, the only real secure log would be one put on write-only/WORM type storage which isn't really a thing these days. In the old days with low volume logs they might have even been printed line-by-line on a dot matrix style printer (which has its own downsides). But no matter which method you choose, a clever attacker could likely find a way to compromise it or at least render it ineffectual.

No one of us need to be so mad. :) But I remember that times, not so long time ago... :)
Agree.

If you want to be lazy and not setup a central log server, just own up to that. Plenty of people are in the same situation. I'm sure there are others who would like to see this request implemented. But you just aren't going to convince anyone who disagrees with this idea in principle to do the work of adding this feature for you. If someone comes along and submits a PR to allow setting a custom log storage/rotation directory on a future version, we'd happily review it.

It's not about "i have no time and passion to install syslog server, but have a time and passion to flooding forum". This tread about "making scheduled creating logs archive on a local USB-drive as a part of standard pfSense feature".
As for me personally - I am "totally on Splunk side". All what I am asking in this thread - making local copy archived logs as standard feature of pfSense. :)

From ordinary user side all looks like "Apple magic": they unpack NetGate device, connect cables, go thru the Wizard (or install pfSense on own appliance) -> insert USB-flash -> all working. And all updates, patches makes remotely, not need all time to be physically near appliance.
I am sure that not small part from 600.000+ pfSense users are happy to have a local copy of archived logs.

P.S. Sorry for nub question, what is "submits a PR". ? I start to writing script, with cron work but ned more time for this...

Sergei_Shablovsky

@jimp said in Store pfSense (+ packages logs) on external (or internal) USB drive / memory card:

The best solution is always local short term logs + long term remote logs. Any logs that happen in the event of a network failure can be obtained from the device itself, but normal logs are viewed/processed centrally.

Need to add that using Automatic Configuration Backup (ACB) with conjunction with remote log aggregator/analyzer is the "remote backup", for purpose of backup archived logs on a local removable media better to use separate feature in pfSense.

And of course "ACB with Gold subscription" = paid remote backup option, when "backup archived logs on a local removable media" = free local option only for logs. (for manual local Configuration Backup already exist separate package where possible to adding any paths/files).

stephenw10

It is possible to do this already using the syslog-ng package. You can configure that to store it's logs in any location including some other drive like a memory card. Then just configure the main logs to export to it as well.

The difficulty is that pfSense has no facility for managing additional drives. They are not auto-mounted etc. You can just add new devices to the fstab but what happens if you pull the memory card?

There have been a few scripts written to address this. When NanoBSD was a thing local logs were RAM only so several people wrote stuff to use a separate drive for logging only.

Steve

Sergei_Shablovsky

@stephenw10 said in Store pfSense (+ packages logs) on external (or internal) USB drive / memory card:

It is possible to do this already using the syslog-ng package. You can configure that to store it's logs in any location including some other drive like a memory card. Then just configure the main logs to export to it as well.

Please take attention “storing encrypted archives of logs on a local media”. This is different that “just copy logs to another drive”. ;)

The difficulty is that pfSense has no facility for managing additional drives. They are not auto-mounted etc. You can just add new devices to the fstab but what happens if you pull the memory card?

There have been a few scripts written to address this. When NanoBSD was a thing local logs were RAM only so several people wrote stuff to use a separate drive for logging only.

Steve
Please wait, I just starting writing scripts. Cron, daemon, etc...