System Logs Format (rsyslog)

arrmo

That makes sense. I did a bit of digging, and in BSD v12 syslog does support the official (RFC 5424 format). But not in v11.x.

Not sure I understand your rsyslog comment - can you clarify? rsyslog doesn't exist on pfSense, does it?

Thanks!

stephenw10

Indeed it doesn't. I was suggesting exporting it to something else and converting there before sending it to the log analyser.

I've never tried that myself.

Steve

arrmo

Ahh, OK - NP. Thanks for the idea! Will dig more.

To be honest, if I go to v2.5 of pfSense the issue goes away (i.e. RFC5424 support is there, directly in syslog). Just not sure how stable v2.5 is.

Thanks again.

stephenw10

It's quite stable on x86-64. I've been running it for months on numerous boxes with no issues. Is is still in dev though so the normal precautions apply etc...

Steve

arrmo

No worries, understand the caveats, legal-ize, etc. ... LOL.

Thanks! I may go this way - then of course need to see if I can tweak the output format (i.e. need to modify the syslogd options a bit, to output the needed format).

Thanks again.

arrmo

OK, shifted to v2.5, seem to have the new and improved version of syslogd ... :-). Meaning, the -O format option exists. Perfect!

Now, how to modify the execution script to have this added to the command? I just need to find that.

arrmo

Checked the output, working great now - thanks for all the help!

Need to figure out the next step - would be nice to have this as a (GUI) option ... it's pretty simple. Just need to figure out how / where to suggest it.

Thanks again.

kiokoman

i still see syslogd on my 2.5.0
you can place additional configuration files in /var/etc/syslog.d
best place to ask for new features is https://redmine.pfsense.org/

stephenw10

Yup or pull-requests directly in github: https://github.com/pfsense

Steve

jimp

I made an issue for it here: https://redmine.pfsense.org/issues/9808

Should be simple enough to code, I'll get to it before long, assuming someone doesn't send in a PR first.

arrmo

Thanks! I was going to do that - just hadn't had a chance to yet.

kiokoman

yes ... it was easy to add the gui fuction

the problem is that if i set rfc5424
remote syslog still work

[2.5.0-DEVELOPMENT][root@pfSense.localdomain]/usr/local/www: ps aux | grep syslogd
root  76833   0.0  0.1  11376   2836  -  Ss   23:18       0:00.03 /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /etc/syslog.conf -O rfc5424
root  62262   0.0  0.1  11144   2636  0  S+   23:19       0:00.00 grep syslogd
[2.5.0-DEVELOPMENT][root@pfSense.localdomain]/usr/local/www: ps aux | grep syslogd
root  74853   0.0  0.1  11376   2836  -  Ss   23:20       0:00.07 /usr/sbin/syslogd -s -c -c -l /var/dhcpd/var/run/log -P /var/run/syslog.pid -f /etc/syslog.conf -O rfc3164
root   3527   0.0  0.1  11144   2636  0  S+   23:30       0:00.00 grep syslogd

this are just example
on my rsyslog server, there is only the hostname instead of the ip, it is able to filter the incoming log

Oct  3 23:19:39 pfSense.localdomain radvd[28029] resuming normal operation
Oct  3 23:19:55 pfSense.localdomain radvd[28029] IPv6 forwarding on interface seems to be disabled, but continuing anyway
Oct  3 23:19:55 pfSense.localdomain radvd[28029] message repeated 2 times: [IPv6 forwarding on interface seems to be disabled, but continuing anyway]
Oct  3 23:20:09 172.17.0.254 radvd[28029]: attempting to reread config file
Oct  3 23:20:09 172.17.0.254 radvd[28029]: IPv6 forwarding on interface seems to be disabled, but continuing anyway
Oct  3 23:20:09 172.17.0.254 radvd[28029]: message repeated 5 times: [ IPv6 forwarding on interface seems to be disabled, but continuing anyway]

anyway this is what is written inside pfsense

<190>1 2019-10-03T23:19:39.586931+02:00 pfSense.localdomain dhcpd 57488 - - Listening on Socket/6/ix0/2001:470:26:5dc::/64
<190>1 2019-10-03T23:19:39.586942+02:00 pfSense.localdomain dhcpd 57488 - - Sending on   Socket/6/ix0/2001:470:26:5dc::/64
<190>1 2019-10-03T23:19:39.586942+02:00 pfSense.localdomain dhcpd 57488 - - Sending on   Socket/6/ix0/2001:470:26:5dc::/64
<190>1 2019-10-03T23:19:39.587172+02:00 pfSense.localdomain dhcpd 57488 - - Server starting service.
<190>1 2019-10-03T23:19:39.587172+02:00 pfSense.localdomain dhcpd 57488 - - Server starting service.
Oct  3 23:20:08 pfSense dhcpd[85579]: Internet Systems Consortium DHCP Server 4.4.1
Oct  3 23:20:08 pfSense dhcpd[85579]: Internet Systems Consortium DHCP Server 4.4.1
Oct  3 23:20:08 pfSense dhcpd[85579]: Copyright 2004-2018 Internet Systems Consortium.
Oct  3 23:20:08 pfSense dhcpd[85579]: Copyright 2004-2018 Internet Systems Consortium.
Oct  3 23:20:08 pfSense dhcpd[85579]: All rights reserved.
Oct  3 23:20:08 pfSense dhcpd[85579]: All rights reserved.

but from the gui i'm unable to see any log (i see only rfc3164) , i think that log filters also need to be adjusted based on rfc selected. and ... well ... that it's not easy for me

jimp

Finally had time to pivot back to this one. I pushed a fix that seems to do a decent job of parsing dynamically, even when the logs contain a mix of entries in different formats.

It should show up in snapshots soon.

https://redmine.pfsense.org/issues/9808
https://github.com/pfsense/pfsense/commit/b16c3a12c61c117e9c8140b115efc7f9acea96c5

arrmo

@jimp Awesome, thanks! Will give it a try once it's in a snapshot. Much appreciated!

arrmo

@jimp Seems to be working - thanks so much! Will keep an eye on it, let you know if I come across any issues. Much appreciated!