Snort + barnyard2: full packet capture missing

mkcharlie

Hi,

I'm trying to get full packet data logged to a remote syslog server from barnyard2.
I have the barnyard2 remote to syslog enabled, and I see packets being sent to my syslog server.

–------------------------
My barnyard2 config:


#   barnyard2.conf
#   barnyard2 can be found at http://www.securixlive.com/barnyard2/index.php
#

## General Barnyard2 settings ##
config quiet
config daemon
config decode_data_link
config alert_with_interface_name
config event_cache_size:    8192
config show_year
config dump_payload
config archivedir:          /var/log/snort/snort_igb09814/barnyard2/archive
config reference_file:      /usr/local/etc/snort/snort_9814_igb0/reference.config
config classification_file: /usr/local/etc/snort/snort_9814_igb0/classification.config
config sid_file:            /usr/local/etc/snort/snort_9814_igb0/sid-msg.map
config gen_file:            /usr/local/etc/snort/snort_9814_igb0/gen-msg.map
config hostname:            pfSense.localdomain
config interface:           igb0
config waldo_file:          /var/log/snort/snort_igb09814/barnyard2/9814_igb0.waldo
config logdir:              /var/log/snort/snort_igb09814

## START user pass through ##

## END user pass through ##

## Setup input plugins ##
input unified2

## Setup output plugins ##
# syslog_full: log to a syslog receiver
output alert_syslog_full: sensor_name pfSense.localdomain, server 192.168.2.2, protocol udp, port 5140, operation_mode complete, log_facility LOG_USER, log_priority LOG_INFO

Note:
I have tried both with and without 'config dump_payload' in the config (via the UI).
–-----------------------------------------

The packet that I capture (from my pfsense box to my remote syslog) when browsing to a suspicious .tk website is:

0000   7c 20 5b 53 4e 4f 52 54 49 44 53 5b 41 4c 45 52  | [SNORTIDS[ALER
0010   54 5d 3a 20 5b 70 66 53 65 6e 73 65 2e 6c 6f 63  T]: [pfSense.loc
0020   61 6c 64 6f 6d 61 69 6e 5d 20 5d 20 7c 7c 20 32  aldomain] ] || 2
0030   30 31 38 2d 30 31 2d 31 33 20 32 33 3a 31 31 3a  018-01-13 23:11:
0040   31 30 2e 38 39 35 2b 30 30 31 20 33 20 5b 31 3a  10.895+001 3 [1:
0050   33 39 38 36 37 3a 33 5d 20 49 4e 44 49 43 41 54  39867:3] INDICAT
0060   4f 52 2d 43 4f 4d 50 52 4f 4d 49 53 45 20 53 75  OR-COMPROMISE Su
0070   73 70 69 63 69 6f 75 73 20 2e 74 6b 20 64 6e 73  spicious .tk dns
0080   20 71 75 65 72 79 20 7c 7c 20 6d 69 73 63 2d 61   query || misc-a
0090   63 74 69 76 69 74 79 20 7c 7c 20 31 37 20 31 39  ctivity || 17 19
00a0   32 2e 31 36 38 2e 33 2e 31 30 30 20 31 39 32 2e  2.168.3.100 192.
00b0   31 36 38 2e 33 2e 31 20 7c 7c 20 35 38 31 38 35  168.3.1 || 58185
00c0   20 35 33 20 7c 7c 20 0a 20 7c 00                  53 || . |.

–--------------------------------------------

As you see, it does not contain any payload data, just the alert. I have read the following related topics, but no solution is provided:

http://seclists.org/snort/2014/q4/17
https://forum.pfsense.org/index.php?topic=111050.0
https://forum.pfsense.org/index.php?topic=77952.0

Perhaps there is a difference between the setting 'alert_syslog_full' and 'log_syslog_full' in barnyard config, but that's not something I can enable via the UI.
If I do NOT enable barnyard2, the packet data IS logged in ASCII format:

Log directory BEFORE enabling barnyard2 (notice size of the log file containing packet data):
#/var/log/snort/snort_igb09814: ls -alhrt

-rw–-----  1 root  wheel   2.3K Jan 13 21:27 snort.log.1515755290
-rw-r--r--  1 root  wheel   360K Jan 13 21:27 alert

Log directory AFTER enabling barnyard2:

-rw–-----  1 root  wheel   216K Jan 13 22:39 alert
-rw-------  1 root  wheel     0B Jan 13 22:39 snort_9814_igb0.u2.1515879592

So before enabling barnyard, the packet data was logged in a readable file. After enabling barnyard, I'm not sure what happened, but the file seems empty.
Is this expected behaviour?

mkcharlie

Update:

I tried doing the same by sending the logs to the PFSense system logs, and then forwarding the logs to the remote server. Same issue (tested with portscan and suspicious tk dns query).

NogBadTheBad

I've had a few goes at getting packets to syslog working but never been able to.

Does any more show when doing a u2spewfoo on the router ?

[2.4.2-RELEASE][admin@pfsense]/var/log/snort/snort_igb0.256577: u2spewfoo snort_56577_igb0.2.u2.1515802084

(IPv6 Event)
sensor id: 0 event id: 3707830273 event second: 1515927039 event microsecond: 247445
sig id: 2025146 gen id: 1 revision: 1 classification: 5
priority: 2 ip source: 2a02:XXXX.XXXX:2::14 ip destination: 2a02:XXXX.XXXX:2::1
src port: 52064 dest port: 53 protocol: 17 impact_flag: 0 blocked: 0
mpls label: 0 vland id: 0 policy id: 0

(ExtraDataHdr)
event type: 4 event length: 48

(ExtraData)
sensor id: 0 event id: 3707830273 event second: 1515927039
type: 11 datatype: 1 bloblength: 24 IPv6 Source Address: 2a02:XXXX.XXXX:2::14

(ExtraDataHdr)
event type: 4 event length: 48

(ExtraData)
sensor id: 0 event id: 3707830273 event second: 1515927039
type: 12 datatype: 1 bloblength: 24 IPv6 Destination Address: 2a02:XXXX.XXXX:2::1

Packet
sensor id: 0 event id: 3707830273 event second: 1515927039
packet second: 1515927039 packet microsecond: 247445
linktype: 1 packet_length: 91
[ 0] 00 08 A2 0A 9D CB 00 3E E1 C1 AF 07 86 DD 60 0E …....>......`.

CB 60 00 35 00 25 64 F0 00 F6 .......`.5.%d...
[ 64] 01 00 00 01 00 00 00 00 00 00 04 66 72 65 64 02 …........fred.
[ 80] 67 72 03 63 6F 6D 00 00 01 00 01 gr.com…..
[2.4.2-RELEASE][admin@pfsense]/var/log/snort/snort_igb0.256577:

mkcharlie

Yes, that contains something:

[2.4.2-RELEASE][admin@pfSense.localdomain]/var/log/snort/snort_igb09814: u2spewfoo snort_9814_igb0.u2.1516273671

(Event)
sensor id: 0 event id: 643170305 event second: 1516290836 event microsecond: 59309
sig id: 5 gen id: 122 revision: 1 classification: 3
priority: 2 ip source: xx ip destination: xx
src port: 0 dest port: 0 protocol: 6 impact_flag: 0 blocked: 0

Packet
sensor id: 0 event id: 643170305 event second: 1516290836
packet second: 1516290836 packet microsecond: 59309
linktype: 1 packet_length: 179
[ 0] 00 0D B9 45 C9 D0 00 00 5E 00 01 04 08 00 45 30 …E....^.....E0
[ 16] 00 A5 00 00 40 00 3A FF 3B 18 5E 6D 74 34 5E 6E ….@.:.;.^mt4^n
[ 32] D4 60 50 72 69 6F 72 69 74 79 20 43 6F 75 6E 74 .`Priority Count
[ 48] 3A 20 30 0A 43 6F 6E 6E 65 63 74 69 6F 6E 20 43 : 0.Connection C
[ 64] 6F 75 6E 74 3A 20 32 30 30 0A 49 50 20 43 6F 75 ount: 200.IP Cou
[ 80] 6E 74 3A 20 33 0A 53 63 61 6E 6E 65 72 20 49 50 nt: 3.Scanner IP
[ 96] 20 52 61 6E 67 65 3A 20 35 2E 31 38 38 2E 38 36 Range: 5.188.86
[ 112] 2E 32 35 3A 31 38 31 2E 32 31 34 2E 38 37 2E 32 .25:181.214.87.2
[ 128] 35 31 0A 50 6F 72 74 2F 50 72 6F 74 6F 20 43 6F 51.Port/Proto Co
[ 144] 75 6E 74 3A 20 32 30 31 0A 50 6F 72 74 2F 50 72 unt: 201.Port/Pr
[ 160] 6F 74 6F 20 52 61 6E 67 65 3A 20 35 32 3A 34 32 oto Range: 52:42
[ 176] 38 31 0A 81.

Packet
sensor id: 0 event id: 643170305 event second: 1516290836
packet second: 1516290836 packet microsecond: 59310
linktype: 1 packet_length: 48
[ 0] 00 0D B9 45 C9 D0 00 00 5E 00 01 04 08 00 45 30 …E....^.....E0
[ 16] 00 22 00 00 40 00 3A FF 3B 18 5E 6D 74 34 5E 6E ."..@.:.;.^mt4^n
[ 32] D4 60 4F 70 65 6E 20 50 6F 72 74 3A 20 38 30 0A .`Open Port: 80.

–-----------
Anything else you think I can try?

Also, it seems that the alerts are still logged to the plain-text alert file as well. I would think that snort would only output to U2, so that it does not waste time by parsing the alerts to a human-readable file.
Is this another setting I should disable?

Edit, so if I do ps aux | grep snort, i get the following:


[2.4.2-RELEASE][admin@pfSense.localdomain]/: ps auxww | grep snort
root    41804   0.0  5.4 680972 221892  -  SNs  12:07       1:30.41 /usr/local/bin/snort -R 9814 -D -q --suppress-config-log -l /var/log/snort/snort_igb09814 --pid-path /var/run --nolock-pidfile -G 9814 -c /usr/local/etc/snort/snort_9814_igb0/snort.conf -i igb0
root    42308   0.0  0.3  47768  12328  -  SN   12:07       0:15.94 /usr/local/bin/barnyard2 -r 9814 -f snort_9814_igb0.u2 --pid-path /var/run --nolock-pidfile -c /usr/local/etc/snort/snort_9814_igb0/barnyard2.conf -d /var/log/snort/snort_igb09814 -D -q
root    42750   0.0  5.3 680972 217296  -  SNs  12:07       1:29.13 /usr/local/bin/snort -R 13317 -D -q --suppress-config-log -l /var/log/snort/snort_igb1.21013317 --pid-path /var/run --nolock-pidfile -G 13317 -c /usr/local/etc/snort/snort_13317_igb1.210/snort.conf -i igb1.210
root    42888   0.0  0.3  47768  12320  -  SN   12:07       0:16.26 /usr/local/bin/barnyard2 -r 13317 -f snort_13317_igb1.210.u2 --pid-path /var/run --nolock-pidfile -c /usr/local/etc/snort/snort_13317_igb1.210/barnyard2.conf -d /var/log/snort/snort_igb1.21013317 -D -q

And the 'output' section of one of the config files:

# Snort Output Logs #
output alert_csv: alert timestamp,sig_generator,sig_id,sig_rev,msg,proto,src,srcport,dst,dstport,id,classification,priority 500K

output unified2: filename snort_9814_igb0.u2, limit 128K
output alert_pf: /usr/local/etc/snort/snort_9814_igb0/default,snort2c,both,kill

And the output section of one of the barnyard2 config files:


## Setup output plugins ##
# syslog_full: log to a syslog receiver
output alert_syslog_full: sensor_name pfSense.localdomain, local, log_facility LOG_USER, log_priority LOG_INFO

If I compare that with the snort manual, it seems there might be several issues:

the -l command instructs snort to log to a readable output file
the snort config contains an output to csv, no idea why that is here
the snort config contains an output alert_pf, I assume that is what pfsense use to block ips which have triggered an alert?

However, all that would mean snort always needs to log to a readable file, and then I wonder why barnyard should be used (as the output will still be parsed by snort, something we could prevent if we only would enable binary u2 output).

Looking at the barnyard2 config file, it seems something is definitely missing. As mentioned in https://forum.pfsense.org/index.php?topic=111050.0 and https://github.com/firnsy/barnyard2/blob/master/etc/barnyard2.conf, the output line in the barnyard2.conf file should contain "operation_mode complete".

mkcharlie

I think I found it,

when manually changing alert_syslog_full to log_syslog_full in the barnyard2 config file, I get the packet details.
edit: this was not enough. The full line is:

output log_syslog_full: sensor_name pfSense.localdomain, local, log_facility LOG_USER, log_priority LOG_INFO, operation_mode complete

So it is a combination fo the log_syslog_full and the operation_mode complete.
I don't think it can be set via the UI (despite that the operation mode is set to 'complete' in the barnyard settings tab) , so config is overwritten on each restart.

Should I file a bug for the inability to log raw packet data, which is solved by manually setting log_syslog_full? The https://doc.pfsense.org/index.php/Bug_reporting says that I should first discuss it here to see if it is a legitimate bug
The fact that snort outputs to alert_csv and alert_pf seems to thwart the use of Barnyard a bit to me (as the processing still occurs by Snort). Should I worry about this?

NogBadTheBad

Think it's Bill ( bmeeks ) that maintains Snort , might be worth running it past him via a PM.

https://forum.pfsense.org/index.php?action=profile;u=17262

bmeeks

Replied to OP via PM. This bug will be fixed in the next Snort GUI update.

Bill

mkcharlie

Should anyone be interested, the following grok pattern matches most of the packets when using barnyard's log_syslog_full with complete operation_mode. Don't know if it works for all alerts yet.

SENSORNAME ([0-9a-zA-Z\.]*)
BARNYARD_PREAMBLE \[SNORTIDS\[%{DATA:logtype}\]\: \[%{SENSORNAME:sensor_name}\] \]
BARNYARD_SNORT_HEADER %{INT:ids_priority} \[%{INT:gid}\:%{INT:sid}\:%{INT:rev}\] %{GREEDYDATA:snort_msg_v}
BARNYARD_IP_HEADER_DATA %{INT:proto} %{IP:src_ip} %{IP:dst_ip} %{INT:ip_version} %{INT:header_length} %{INT:type_of_service} %{INT:ip_packet_length} %{INT:ip_id} %{INT:flag} %{INT:offset} %{INT:ip_checksum} %{INT:unknown1}
BARNYARD_SEGMENT_DATA %{BARNYARD_UDP_DATA}|%{BARNYARD_TCP_DATA}|%{BARNYARD_ICMP_DATA}
BARNYARD_UDP_DATA %{INT:src_port} %{INT:dst_port} %{INT:transport_length} %{INT:transport_checksum}
BARNYARD_TCP_DATA %{INT:src_port} %{INT:dst_port} %{INT:sequence} %{INT:ack} %{INT:tcp_offset} %{INT:tcp_reserved} %{INT:tcp_flags} %{INT:tcp_window} %{INT:tcp_checksum} %{INT:tcp_urgent}
BARNYARD_ICMP_DATA %{INT:icmp_type} %{INT:icmp_code} %{INT:icmp_checksum} %{INT:id} %{INT:icmp_sequence}

\| %{BARNYARD_PREAMBLE} \|\| %{TIMESTAMP_ISO8601:iso8601_time} %{BARNYARD_SNORT_HEADER} \|\| %{DATA:ids_classification} \|\| %{BARNYARD_IP_HEADER_DATA} \|\| (%{BARNYARD_SEGMENT_DATA} \|\| )?%{INT:frame_length} %{BASE16NUM:raw_packet}(?:\s+\|\|   \|)?

Based on https://groups.google.com/forum/#!topic/barnyard2-users/7Qnpmc3VT_w , though there seemed to be a difference in the protocol data (ip_header_data in the above list).

mkcharlie

Upgraded to the latest Snort version.

Now it seems like the full_syslog_full is working, but ONLY if you send it to a remote syslog server. So you can't send full events to a local syslog. I just modified my grok rules to be able to catch syslog events forwarded from the local system to a remote syslog (via status => system logs), but that format is different when you send them directly from snort to a remote syslog.

I think it should work, I'll modify the grok rules later today. However, to improve consistency with other logs forwarded via the system, perhaps it is a good idea to also allow "local full syslog complete" .

In other words, now I have:

output log_syslog_full: sensor_name snort.WAN, server 192.168.2.2, protocol udp, port 514, operation_mode complete, payload_encoding hex, log_facility LOG_USER, log_priority LOG_INFO

Which gives a different format on my remote syslog server then when using the following in combination with remote logging via the status => system logs setting:

output log_syslog_full: sensor_name snort.WAN, operation_mode complete, payload_encoding hex, log_facility LOG_USER, log_priority LOG_INFO

Kr,
Michael

bmeeks

@MichaelB:

Upgraded to the latest Snort version.

Now it seems like the full_syslog_full is working, but ONLY if you send it to a remote syslog server. So you can't send full events to a local syslog. I just modified my grok rules to be able to catch syslog events forwarded from the local system to a remote syslog (via status => system logs), but that format is different when you send them directly from snort to a remote syslog.

I think it should work, I'll modify the grok rules later today. However, to improve consistency with other logs forwarded via the system, perhaps it is a good idea to also allow "local full syslog complete" .

In other words, now I have:
output log_syslog_full: sensor_name snort.WAN, server 192.168.2.2, protocol udp, port 514, operation_mode complete, payload_encoding hex, log_facility LOG_USER, log_priority LOG_INFO
Which gives a different format on my remote syslog server then when using the following in combination with remote logging via the status => system logs setting:
output log_syslog_full: sensor_name snort.WAN, operation_mode complete, payload_encoding hex, log_facility LOG_USER, log_priority LOG_INFO
Kr,
Michael

I did in fact code it so that full packet capture only works to a remote syslog server. It is not a good idea to fill up your local firewall system log with full packet captures, so I elected to not make that available.

Bill