Syslog format

biggsy · Aug 15, 2014, 2:23 AM

The new 2.2 firewall syslog format is certainly cleaner and briefer than the old but I feel it's become less human-readable.

While doing some cleaning up of old files recently, I happened to find a log from my Smoothwall from many years ago:

00:04:07 IN=eth0 OUT= MAC=00:0b:cd:49:8d:e7:00:14:f1:65:5c:17:08:00 SRC=89.222.153.113 DST=111.222.333.444 LEN=404 TOS=0x00 PREC=0x00 TTL=49 ID=62404 PROTO=UDP SPT=3067 DPT=1434 LEN=384 
00:31:32 IN=eth0 OUT= MAC=00:0b:cd:49:8d:e7:00:14:f1:65:5c:17:08:00 SRC=189.7.68.130 DST=111.222.333.444 LEN=48 TOS=0x00 PREC=0x00 TTL=109 ID=4969 DF PROTO=TCP SPT=3966 DPT=1433 WINDOW=64240 RES=0x00 SYN URGP=0 
00:31:35 IN=eth0 OUT= MAC=00:0b:cd:49:8d:e7:00:14:f1:65:5c:17:08:00 SRC=189.7.68.130 DST=111.222.333.444 LEN=48 TOS=0x00 PREC=0x00 TTL=109 ID=6131 DF PROTO=TCP SPT=3966 DPT=1433 WINDOW=64240 RES=0x00 SYN URGP=0 
01:24:15 IN=eth0 OUT= MAC=00:0b:cd:49:8d:e7:00:14:f1:65:5c:17:08:00 SRC=125.211.198.26 DST=111.222.333.444 LEN=485 TOS=0x00 PREC=0x00 TTL=44 ID=0 DF PROTO=UDP SPT=58577 DPT=1026 LEN=465 
01:24:15 IN=eth0 OUT= MAC=00:0b:cd:49:8d:e7:00:14:f1:65:5c:17:08:00 SRC=125.211.198.26 DST=111.222.333.444 LEN=485 TOS=0x00 PREC=0x00 TTL=44 ID=0 DF PROTO=UDP SPT=58577 DPT=1027 LEN=465 
01:26:01 IN=eth0 OUT= MAC=00:0b:cd:49:8d:e7:00:14:f1:65:5c:17:08:00 SRC=121.28.41.204 DST=111.222.333.444 LEN=48 TOS=0x00 PREC=0x00 TTL=108 ID=31786 PROTO=TCP SPT=64485 DPT=22 WINDOW=65535 RES=0x00 SYN URGP=0 
01:39:05 IN=eth0 OUT= MAC=00:0b:cd:49:8d:e7:00:14:f1:65:5c:17:08:00 SRC=24.64.49.141 DST=111.222.333.444 LEN=512 TOS=0x00 PREC=0x00 TTL=66 ID=37357 PROTO=UDP SPT=16268 DPT=1026 LEN=492 
01:39:05 IN=eth0 OUT= MAC=00:0b:cd:49:8d:e7:00:14:f1:65:5c:17:08:00 SRC=24.64.49.141 DST=111.222.333.444 LEN=512 TOS=0x00 PREC=0x00 TTL=66 ID=37358 PROTO=UDP SPT=16268 DPT=1027 LEN=492 
01:39:05 IN=eth0 OUT= MAC=00:0b:cd:49:8d:e7:00:14:f1:65:5c:17:08:00 SRC=24.64.49.141 DST=111.222.333.444 LEN=512 TOS=0x00 PREC=0x00 TTL=66 ID=37359 PROTO=UDP SPT=16268 DPT=1028 LEN=492 
02:07:09 IN=eth0 OUT= MAC=00:0b:cd:49:8d:e7:00:14:f1:65:5c:17:08:00 SRC=24.64.65.139 DST=111.222.333.444 LEN=512 TOS=0x00 PREC=0x00 TTL=66 ID=41254 PROTO=UDP SPT=19465 DPT=1026 LEN=492 
02:07:09 IN=eth0 OUT= MAC=00:0b:cd:49:8d:e7:00:14:f1:65:5c:17:08:00 SRC=24.64.65.139 DST=111.222.333.444 LEN=512 TOS=0x00 PREC=0x00 TTL=66 ID=41255 PROTO=UDP SPT=19465 DPT=1027 LEN=492 
02:07:09 IN=eth0 OUT= MAC=00:0b:cd:49:8d:e7:00:14:f1:65:5c:17:08:00 SRC=24.64.65.139 DST=111.222.333.444 LEN=512 TOS=0x00 PREC=0x00 TTL=66 ID=41256 PROTO=UDP SPT=19465 DPT=1028 LEN=492 
02:16:34 IN=eth0 OUT= MAC=00:0b:cd:49:8d:e7:00:14:f1:65:5c:17:08:00 SRC=221.206.121.54 DST=111.222.333.444 LEN=485 TOS=0x00 PREC=0x00 TTL=43 ID=0 DF PROTO=UDP SPT=57085 DPT=1026 LEN=465 
02:19:42 IN=eth0 OUT= MAC=00:0b:cd:49:8d:e7:00:14:f1:65:5c:17:08:00 SRC=61.176.216.15 DST=111.222.333.444 LEN=40 TOS=0x00 PREC=0x00 TTL=99 ID=3519 PROTO=TCP SPT=6000 DPT=1433 WINDOW=16384 RES=0x00 SYN URGP=0 
02:53:19 IN=eth0 OUT= MAC=00:0b:cd:49:8d:e7:00:14:f1:65:5c:17:08:00 SRC=125.211.198.10 DST=111.222.333.444 LEN=485 TOS=0x00 PREC=0x00 TTL=44 ID=0 DF PROTO=UDP SPT=45028 DPT=1027 LEN=465 
02:53:19 IN=eth0 OUT= MAC=00:0b:cd:49:8d:e7:00:14:f1:65:5c:17:08:00 SRC=125.211.198.10 DST=111.222.333.444 LEN=485 TOS=0x00 PREC=0x00 TTL=44 ID=0 DF PROTO=UDP SPT=45029 DPT=1026 LEN=465 
02:53:19 IN=eth0 OUT= MAC=00:0b:cd:49:8d:e7:00:14:f1:65:5c:17:08:00 SRC=125.211.198.10 DST=111.222.333.444 LEN=485 TOS=0x00 PREC=0x00 TTL=44 ID=0 DF PROTO=UDP SPT=45029 DPT=1027 LEN=465 
03:15:37 IN=eth0 OUT= MAC=00:0b:cd:49:8d:e7:00:14:f1:65:5c:17:08:00 SRC=89.207.168.25 DST=111.222.333.444 LEN=60 TOS=0x00 PREC=0x00 TTL=47 ID=30271 DF PROTO=TCP SPT=49569 DPT=22 WINDOW=5840 RES=0x00 SYN URGP=0 
03:32:47 IN=eth0 OUT= MAC=00:0b:cd:49:8d:e7:00:14:f1:65:5c:17:08:00 SRC=59.63.157.67 DST=111.222.333.444 LEN=40 TOS=0x00 PREC=0x00 TTL=101 ID=256 PROTO=TCP SPT=6000 DPT=1433 WINDOW=16384 RES=0x00 SYN URGP=0 
03:33:11 IN=eth0 OUT= MAC=00:0b:cd:49:8d:e7:00:14:f1:65:5c:17:08:00 SRC=61.176.216.15 DST=111.222.333.444 LEN=40 TOS=0x00 PREC=0x00 TTL=99 ID=5914 PROTO=TCP SPT=6000 DPT=1433 WINDOW=16384 RES=0x00 SYN URGP=0 
03:37:32 IN=eth0 OUT= MAC=00:0b:cd:49:8d:e7:00:14:f1:65:5c:17:08:00 SRC=24.64.110.53 DST=111.222.333.444 LEN=512 TOS=0x00 PREC=0x00 TTL=66 ID=25644 PROTO=UDP SPT=15625 DPT=1026 LEN=492 
03:37:32 IN=eth0 OUT= MAC=00:0b:cd:49:8d:e7:00:14:f1:65:5c:17:08:00 SRC=24.64.110.53 DST=111.222.333.444 LEN=512 TOS=0x00 PREC=0x00 TTL=66 ID=25645 PROTO=UDP SPT=15625 DPT=1027 LEN=492 
03:37:32 IN=eth0 OUT= MAC=00:0b:cd:49:8d:e7:00:14:f1:65:5c:17:08:00 SRC=24.64.110.53 DST=111.222.333.444 LEN=512 TOS=0x00 PREC=0x00 TTL=66 ID=25646 PROTO=UDP SPT=15625 DPT=1028 LEN=492 
03:47:10 IN=eth2 OUT=eth0 SRC=192.168.0.33 DST=69.32.188.13 LEN=40 TOS=0x00 PREC=0x00 TTL=127 ID=8679 DF PROTO=TCP SPT=49704 DPT=80 WINDOW=0 RES=0x00 ACK RST URGP=0

Much longer records but easier to read and easier to search for things of interest without uploading to a spreadsheet or other external tools.

Also noticed that the new format seems to contain both protocol number and name (e.g., 17,UDP)

eri-- · Aug 15, 2014, 1:32 PM

There is map of its format on tools repository.

Will not going to be smoothwall and i am not sure is useful to see by eye a log as this.
Without any tools its hard to spot anything evn if its in front of your eyes or its format looks nicer.

biggsy · Aug 15, 2014, 10:11 PM

Will not going to be smoothwall …

No problem with that! ;D

biggsy · Sep 23, 2014, 10:52 AM

@ermal:

There is map of its format on tools repository.

I submitted the ICLA, tried navigating git to get a copy of the tools repository and failed miserably. A lack of experience on my part, I admit.

However, when we're talking about a relatively small number of log entries per day it's amazing what stands out using just a text editor.

Let's not assume that if you're not smart enough to be running an automated log parsing/analysis tool, you have no business looking at logs.

Please, can these formats be published somewhere that ordinary people can access or, better still, how about a user-selectable option of "verbose" firewall logging?

jimp · Sep 23, 2014, 4:00 PM

The "raw" firewall logs can be enabled from the settings tab under Status > System Logs ("Show raw filter logs")

The format will be documented closer to release, probably on the doc wiki, it's one of the many things left on our various todo lists.

biggsy · Sep 24, 2014, 7:17 AM

Thanks, jimp.

jimp · Sep 29, 2014, 4:40 PM

Here you go:

https://doc.pfsense.org/index.php/Filter_Log_Format_for_pfSense_2.2

biggsy · Sep 29, 2014, 8:31 PM

Thanks again, jimp.