Store pfSense (+ packages logs) on external (or internal) USB drive / memory card

Sergei_Shablovsky

@jimp said in Store pfSense (+ packages logs) on external (or internal) USB drive / memory card:

The best solution is always local short term logs + long term remote logs. Any logs that happen in the event of a network failure can be obtained from the device itself, but normal logs are viewed/processed centrally.

Need to add that using Automatic Configuration Backup (ACB) with conjunction with remote log aggregator/analyzer is the "remote backup", for purpose of backup archived logs on a local removable media better to use separate feature in pfSense.

And of course "ACB with Gold subscription" = paid remote backup option, when "backup archived logs on a local removable media" = free local option only for logs. (for manual local Configuration Backup already exist separate package where possible to adding any paths/files).

stephenw10

It is possible to do this already using the syslog-ng package. You can configure that to store it's logs in any location including some other drive like a memory card. Then just configure the main logs to export to it as well.

The difficulty is that pfSense has no facility for managing additional drives. They are not auto-mounted etc. You can just add new devices to the fstab but what happens if you pull the memory card?

There have been a few scripts written to address this. When NanoBSD was a thing local logs were RAM only so several people wrote stuff to use a separate drive for logging only.

Steve

Sergei_Shablovsky

@stephenw10 said in Store pfSense (+ packages logs) on external (or internal) USB drive / memory card:

It is possible to do this already using the syslog-ng package. You can configure that to store it's logs in any location including some other drive like a memory card. Then just configure the main logs to export to it as well.

Please take attention “storing encrypted archives of logs on a local media”. This is different that “just copy logs to another drive”. ;)

The difficulty is that pfSense has no facility for managing additional drives. They are not auto-mounted etc. You can just add new devices to the fstab but what happens if you pull the memory card?

There have been a few scripts written to address this. When NanoBSD was a thing local logs were RAM only so several people wrote stuff to use a separate drive for logging only.

Steve
Please wait, I just starting writing scripts. Cron, daemon, etc...

stephenw10

Mmm, reading through the syslog-ng manuals I think your need the Premium Edition to store logs encrypted.

Steve

Sergei_Shablovsky

@stephenw10 said in Store pfSense (+ packages logs) on external (or internal) USB drive / memory card:

Mmm, reading through the syslog-ng manuals I think your need the Premium Edition to store logs encrypted.

Steve
Sorry, what You mean ?

I told about storing encrypted archive with logs LOCALLY. This is the first level for advanced users or home users, that not need store logs in a remote place for various reasons.

stephenw10

Indeed, I thought you might be able to do that with syslog-ng since they do have that feature. But it looks to be for PE only so not in the FreeBSD port.

Steve

JonathanLee

use the syslog package, I am using it to send logs from my AP to pfSense however you can also forward syslogs out of pfsense

JonathanLee

I was thinking…

What about you mount a drive like a second SSD or a mpcie to m.2 and have a NVMe drive that you mount to something like /root/logs use gpart to make a partition on that other drive and mount to it with fstab automatically, after use it for snort logs and squid or any other package that lets you pick what location you log to… what would cut down a lot. I mean you can copy to usb drive with fat32 partition why not just do the same thing and make it a log partition ? Wouldn’t that work?

I got this monster Optane drive and I am using a small part as an emergency swap location so it has 255gb I could add a new partition to it with gpart and rock that for a log directory. Any thoughts ?

stephenw10

Yup that can work. The problem is if you have to re-install or at upgrade it may get overwritten. And then how does the package behave if its unable to reach it's log location.

JonathanLee

@stephenw10 does it have to be fat32? I can't get it to mount the zfs manually

Shell Output - mount /dev/nda0p2 /root/LOGS_Optane
mount: /dev/nda0p2: Integrity check failed

Shell Output - gpart show nda0
=>       40  500118112  nda0  GPT  (238G)
         40       2008        - free -  (1.0M)
       2048   16777216     1  freebsd-swap  (8.0G)
   16779264  482344960     2  freebsd-zfs  (230G)
  499124224     993928        - free -  (485M)

I researched it and found how to do it it needs efi for GPT I have learned that way you can make a fat32 that pfSense can allow use of. I think it is restricted to only fat32 for external stuff correct me if I am wrong

It needs to have this done

gpart add -t efi  -s 230GB -l LOG nda0
newfs_msdos -F 32 /dev/nda0p2
mount_msdosfs /dev/nda0p2 /root/LOGS_Optane

Shell Output - gpart show nda0
=>       40  500118112  nda0  GPT  (238G)
         40       2008        - free -  (1.0M)
       2048   16777216     1  freebsd-swap  (8.0G)
   16779264  482344960     2  efi  (230G)
  499124224     993928        - free -  (485M)

It works I can mount it send files to it after this you create a cron to mount it

Warning this can break stuff if you do not know what your doing here for others I am doing a new partition inside a second drive that I am also using as swap so be careful to not wipe out your drives

JonathanLee

Updated my unofficial guide if anyone else wants to try this here is a short guide for you.

https://forum.netgate.com/topic/195843/unofficial-guide-have-package-logs-record-to-a-secondary-ssd-drive-snort-syslog-squid-and-or-squid-cache-system