Logmail - sending syslogs by mail

sullrich · Apr 23, 2007, 11:26 PM

This should go in the base system I think.

Please continue developing the script. The critical parts now is the auto installation of cron and a simple webConfigurator screen.

uranellus · May 2, 2007, 9:42 PM

Thanks for your reply sullrich,

it looks like I'm half way through .. :)

I've modified:
/usr/local/www/diag_logs_settings.php : added inputs
/usr/inc/system.inc : added system_logmail_setup() -> sets up cronjob ..

I've added:
/usr/inc/smtp.inc : The above mentioned SMTP class
/usr/local/sbin/logmail.php : script that is called by cron ..

It works so far :)

The patch attached, should create all required files ..
Note that this is still NOT production ready! But all testers are very welcomed! :) Thanks dexcs for being the first one :)

Please check my first post for a updated todo/wishlist..

Update:
Bugs in logmail-0.1:

~~remote smtp host is not used, instead the host is automatically queried using dns/mx records..~~ (fixed in 0.3)
fix: change $smtp->direct_delivery = 1; to $smtp->direct_delivery = 0; in /usr/local/sbin/logmail.php:154
todo: the automatic selection of the smtp host will eventually become a build in feature, but later ..
due to the fact that the /etc/inc/pfsense-utils.inc:add_text_to_file(); does add same lines multiple times (though the description states that it doesnt -> bug?),
upon enabling and later changing the settings on diag_logs_settings.php will cause the cronjob to be added more than once in the crontab.
Note that removing all cronjobs works with remove_text_from_file();

BTW: Note the namechange .. logtailmail -> logmail ..

logmail-0.1.patch.txt

hoba · Apr 24, 2007, 7:27 PM

I have a feature request for this: ;)
Could you implement some kind of encryption? Maybe like compressing the log as a zip with userconfigurable password so you can't uncopmpress it without knowing the password? Logs are not meant for everybodies eyes.

sullrich · Apr 24, 2007, 7:27 PM

Good idea. We can include a zip binary if needed for this.

uranellus · May 2, 2007, 7:02 PM

@hoba:

I have a feature request for this: ;)
Could you implement some kind of encryption? Maybe like compressing the log as a zip with userconfigurable password so you can't uncopmpress it without knowing the password? Logs are not meant for everybodies eyes.

Nice idea, though after some thoughts about it: Wouldn't it be nicer to encrypt the mail using the webgui's private key? Found a php based gpg class for this as well as an article on zend dev zone about this. All that's needed is a gpg executable.
What are your thoughts about this?

uranellus · May 3, 2007, 6:24 AM

I'd like to share logmail-0.3:

CHANGES:
logmail-0.3 - 2007-05-02 23:30
added auth mechs: plain, pop3 before smtp
specify the minumum account of logs before sending
moved all the configs to $config['syslog']['logmail']

I've basically added some auth options which NEEDS to be tested!

I've supplied the added/modified scripts a long with a patch. I hope supplying the complete files makes it easier for you to have a look at my work, or maybe you'd even give it a try :)

The complete archive is available here: ~~http://alexanderschaber.de/ablage/logmail-0.3.tar.gz~~ Check below for newer version!

Have fun, and I'm looking forward for your reply :)

sullrich · May 2, 2007, 9:43 PM

Looking good! When will you be ready to commit these to -HEAD and RELENG_1? RELENG_1_2 is frozen so we cannot consider adding it here.

uranellus · May 2, 2007, 9:50 PM

@sullrich:

Looking good! When will you be ready to commit these to -HEAD and RELENG_1? RELENG_1_2 is frozen so we cannot consider adding it here.

Wow thanks for the quick answer :)

I still need to do some testing on the auth mechs (or: I need someone to test it) .. Other than that, it works great for me :)
I'll have my pfSense box in production next week, then I'll be able to do some real testing (currently just a vm) .. So I'd say that in a week or two the initial version should be ready.

sullrich · May 2, 2007, 9:57 PM

Sounds good. I'll issue you a commit bit when you are ready.

uranellus · May 2, 2007, 11:20 PM

Releasing bugfix version:

logmail-0.3.1 - 2007-05-03 01:00 CEST
fixed a blocker bug: no cron job added
fixed a pfsense bug: add_text_to_file() did not filter
duplicates (patch sent to pfsense
devs and included in releng_1/head)
some minor cleanup..

http://alexanderschaber.de/ablage/logmail-0.3.1.tar.gz

Hope this works out better .. :(

cmb · May 2, 2007, 11:59 PM

Maybe I'm jumping in a bit late here, but maybe it would make more sense to have a logging alerts tab or something, and just make the email alerts say that there's a new log alert, with a URL to the webGUI of the pfsense box it came from? Then it's as secure as your webGUI, and avoids the mess of encrypting emails.

If email encryption is the preferred way to go, personally I much prefer S/MIME. Have a field where I can put in my public key on pfsense, and then my MUA would automatically decrypt it since it has my private key.

uranellus · May 3, 2007, 6:14 AM

@cmb:

Maybe I'm jumping in a bit late here, but maybe it would make more sense to have a logging alerts tab or something, and just make the email alerts say that there's a new log alert, with a URL to the webGUI of the pfsense box it came from? Then it's as secure as your webGUI, and avoids the mess of encrypting emails.

Nice idea, I might add that as well later .. Though my initial goal was it to have the actual logs on my mail account, so I hope you don't mind, if I go for that functionality first.
@cmb:

If email encryption is the preferred way to go, personally I much prefer S/MIME. Have a field where I can put in my public key on pfsense, and then my MUA would automatically decrypt it since it has my private key.

Ok, that's another way to do it, though since pfSense has it's own private key for https, I guess I'll rather just use that, instead of adding another key, but I'm open to this as well if there are more people who would prefer this. Or then I might even implement both..

uranellus · May 4, 2007, 12:02 PM

Release: logmail-0.4:


CHANGELOG:
logmail 0.4 - 2007-05-04 13:40 CEST
  fixed bug:            that offsets of last leftoff are written,
                        although mail was not sent since the amount
                        of collected logs is less than the specified
                        minimum log count.
  fixed problems:       with cronjob, which caused the script to stop
                        without notice and without ever actually sending
                        mail -> surpressing output of the script
                        (piping to /dev/null)
  changed input type:   password input is now actually an input type
                        password
  changed default:      the minimum count for logs is now 50 ..
  Note:                 this is the first version that actually works
                        like a charm on our production pfSense box :)

Though still need somebody to test auth mechanisms ..

Download: http://alexanderschaber.de/ablage/logmail-0.4.tar.bz2

uranellus · May 5, 2007, 12:52 PM

Release: logmail-0.5:


CHANGELOG:
logmail 0.5 - 2007-05-05 13:40 CEST
  added enhancement:    select timeinterval for cron job
  added checkbox:       now possible to select wether logmail should
                        wait for a minimum number of new logs before sending
                        or just send without counting ..
  added some cosmetics: button onclick: show auth options,
                        hide/show of minlogcnt with checkbox
                        onenable - logmail - after a js:confirm() popup
                        some nice recomended values are loaded
  changed cron handling:cronjob is now added to $config['cron']['item']
                        and automatically added to crontab with
                        configure_cron(); this should make the cronjob
                        survive a reboot :)
                        also removing a cronjob adjusted to fit the above
                        mentioned changes.
  changed clog exec:    clog is now called with complete path to make sure it
                        works eventhough PATH might not be set correctly
  Note:                 auth mech: pop3 before smtp has been tested and
                        confirmed as working by summat, thanks :)

Some embedded or low-profile hardware around to test the cpu/memory load?

Summat tested on p3-500 and 192mb ram .. the first run was quiet a ressource hog (2 mb mail) but later ones didn't create any problem ..

GeekGod: I guess I'll be done at the end of next week .. Check the todo list in the first post ..

Download: http://alexanderschaber.de/ablage/logmail-0.5.tar.bz2

cmb · May 5, 2007, 6:52 PM

@uranellus:

@cmb:

Maybe I'm jumping in a bit late here, but maybe it would make more sense to have a logging alerts tab or something, and just make the email alerts say that there's a new log alert, with a URL to the webGUI of the pfsense box it came from? Then it's as secure as your webGUI, and avoids the mess of encrypting emails.

Nice idea, I might add that as well later .. Though my initial goal was it to have the actual logs on my mail account, so I hope you don't mind, if I go for that functionality first.

You're the one doing the work, do it however suits you. :)

@uranellus:

@cmb:

If email encryption is the preferred way to go, personally I much prefer S/MIME. Have a field where I can put in my public key on pfsense, and then my MUA would automatically decrypt it since it has my private key.

Ok, that's another way to do it, though since pfSense has it's own private key for https, I guess I'll rather just use that, instead of adding another key, but I'm open to this as well if there are more people who would prefer this. Or then I might even implement both..

I think both would be the way to go (eventually). Not a lot of people use S/MIME, though I think a large number of the people who want this would configure it just for this purpose because it's so convenient. All the emails would show in your MUA as normal messages, but be secure in transit.