Snort + barnyard2: full packet capture missing
-
Hi,
I'm trying to get full packet data logged to a remote syslog server from barnyard2.
I have the barnyard2 remote to syslog enabled, and I see packets being sent to my syslog server.–------------------------
My barnyard2 config:# barnyard2.conf # barnyard2 can be found at http://www.securixlive.com/barnyard2/index.php # ## General Barnyard2 settings ## config quiet config daemon config decode_data_link config alert_with_interface_name config event_cache_size: 8192 config show_year config dump_payload config archivedir: /var/log/snort/snort_igb09814/barnyard2/archive config reference_file: /usr/local/etc/snort/snort_9814_igb0/reference.config config classification_file: /usr/local/etc/snort/snort_9814_igb0/classification.config config sid_file: /usr/local/etc/snort/snort_9814_igb0/sid-msg.map config gen_file: /usr/local/etc/snort/snort_9814_igb0/gen-msg.map config hostname: pfSense.localdomain config interface: igb0 config waldo_file: /var/log/snort/snort_igb09814/barnyard2/9814_igb0.waldo config logdir: /var/log/snort/snort_igb09814 ## START user pass through ## ## END user pass through ## ## Setup input plugins ## input unified2 ## Setup output plugins ## # syslog_full: log to a syslog receiver output alert_syslog_full: sensor_name pfSense.localdomain, server 192.168.2.2, protocol udp, port 5140, operation_mode complete, log_facility LOG_USER, log_priority LOG_INFO
Note:
I have tried both with and without 'config dump_payload' in the config (via the UI).
–-----------------------------------------The packet that I capture (from my pfsense box to my remote syslog) when browsing to a suspicious .tk website is:
0000 7c 20 5b 53 4e 4f 52 54 49 44 53 5b 41 4c 45 52 | [SNORTIDS[ALER 0010 54 5d 3a 20 5b 70 66 53 65 6e 73 65 2e 6c 6f 63 T]: [pfSense.loc 0020 61 6c 64 6f 6d 61 69 6e 5d 20 5d 20 7c 7c 20 32 aldomain] ] || 2 0030 30 31 38 2d 30 31 2d 31 33 20 32 33 3a 31 31 3a 018-01-13 23:11: 0040 31 30 2e 38 39 35 2b 30 30 31 20 33 20 5b 31 3a 10.895+001 3 [1: 0050 33 39 38 36 37 3a 33 5d 20 49 4e 44 49 43 41 54 39867:3] INDICAT 0060 4f 52 2d 43 4f 4d 50 52 4f 4d 49 53 45 20 53 75 OR-COMPROMISE Su 0070 73 70 69 63 69 6f 75 73 20 2e 74 6b 20 64 6e 73 spicious .tk dns 0080 20 71 75 65 72 79 20 7c 7c 20 6d 69 73 63 2d 61 query || misc-a 0090 63 74 69 76 69 74 79 20 7c 7c 20 31 37 20 31 39 ctivity || 17 19 00a0 32 2e 31 36 38 2e 33 2e 31 30 30 20 31 39 32 2e 2.168.3.100 192. 00b0 31 36 38 2e 33 2e 31 20 7c 7c 20 35 38 31 38 35 168.3.1 || 58185 00c0 20 35 33 20 7c 7c 20 0a 20 7c 00 53 || . |.
–--------------------------------------------
As you see, it does not contain any payload data, just the alert. I have read the following related topics, but no solution is provided:
-
http://seclists.org/snort/2014/q4/17
-
https://forum.pfsense.org/index.php?topic=111050.0
-
https://forum.pfsense.org/index.php?topic=77952.0
Perhaps there is a difference between the setting 'alert_syslog_full' and 'log_syslog_full' in barnyard config, but that's not something I can enable via the UI.
If I do NOT enable barnyard2, the packet data IS logged in ASCII format:Log directory BEFORE enabling barnyard2 (notice size of the log file containing packet data):
#/var/log/snort/snort_igb09814: ls -alhrt-rw–----- 1 root wheel 2.3K Jan 13 21:27 snort.log.1515755290 -rw-r--r-- 1 root wheel 360K Jan 13 21:27 alert
Log directory AFTER enabling barnyard2:
-rw–----- 1 root wheel 216K Jan 13 22:39 alert -rw------- 1 root wheel 0B Jan 13 22:39 snort_9814_igb0.u2.1515879592
So before enabling barnyard, the packet data was logged in a readable file. After enabling barnyard, I'm not sure what happened, but the file seems empty.
Is this expected behaviour? -
-
Update:
I tried doing the same by sending the logs to the PFSense system logs, and then forwarding the logs to the remote server. Same issue (tested with portscan and suspicious tk dns query).
-
I've had a few goes at getting packets to syslog working but never been able to.
Does any more show when doing a u2spewfoo on the router ?
[2.4.2-RELEASE][admin@pfsense]/var/log/snort/snort_igb0.256577: u2spewfoo snort_56577_igb0.2.u2.1515802084
(IPv6 Event)
sensor id: 0 event id: 3707830273 event second: 1515927039 event microsecond: 247445
sig id: 2025146 gen id: 1 revision: 1 classification: 5
priority: 2 ip source: 2a02:XXXX.XXXX:2::14 ip destination: 2a02:XXXX.XXXX:2::1
src port: 52064 dest port: 53 protocol: 17 impact_flag: 0 blocked: 0
mpls label: 0 vland id: 0 policy id: 0(ExtraDataHdr)
event type: 4 event length: 48(ExtraData)
sensor id: 0 event id: 3707830273 event second: 1515927039
type: 11 datatype: 1 bloblength: 24 IPv6 Source Address: 2a02:XXXX.XXXX:2::14(ExtraDataHdr)
event type: 4 event length: 48(ExtraData)
sensor id: 0 event id: 3707830273 event second: 1515927039
type: 12 datatype: 1 bloblength: 24 IPv6 Destination Address: 2a02:XXXX.XXXX:2::1Packet
sensor id: 0 event id: 3707830273 event second: 1515927039
packet second: 1515927039 packet microsecond: 247445
linktype: 1 packet_length: 91
[ 0] 00 08 A2 0A 9D CB 00 3E E1 C1 AF 07 86 DD 60 0E …....>......`.CB 60 00 35 00 25 64 F0 00 F6 .......`.5.%d...
[ 64] 01 00 00 01 00 00 00 00 00 00 04 66 72 65 64 02 …........fred.
[ 80] 67 72 03 63 6F 6D 00 00 01 00 01 gr.com…..
[2.4.2-RELEASE][admin@pfsense]/var/log/snort/snort_igb0.256577: -
Yes, that contains something:
[2.4.2-RELEASE][admin@pfSense.localdomain]/var/log/snort/snort_igb09814: u2spewfoo snort_9814_igb0.u2.1516273671
(Event)
sensor id: 0 event id: 643170305 event second: 1516290836 event microsecond: 59309
sig id: 5 gen id: 122 revision: 1 classification: 3
priority: 2 ip source: xx ip destination: xx
src port: 0 dest port: 0 protocol: 6 impact_flag: 0 blocked: 0Packet
sensor id: 0 event id: 643170305 event second: 1516290836
packet second: 1516290836 packet microsecond: 59309
linktype: 1 packet_length: 179
[ 0] 00 0D B9 45 C9 D0 00 00 5E 00 01 04 08 00 45 30 …E....^.....E0
[ 16] 00 A5 00 00 40 00 3A FF 3B 18 5E 6D 74 34 5E 6E ….@.:.;.^mt4^n
[ 32] D4 60 50 72 69 6F 72 69 74 79 20 43 6F 75 6E 74 .`Priority Count
[ 48] 3A 20 30 0A 43 6F 6E 6E 65 63 74 69 6F 6E 20 43 : 0.Connection C
[ 64] 6F 75 6E 74 3A 20 32 30 30 0A 49 50 20 43 6F 75 ount: 200.IP Cou
[ 80] 6E 74 3A 20 33 0A 53 63 61 6E 6E 65 72 20 49 50 nt: 3.Scanner IP
[ 96] 20 52 61 6E 67 65 3A 20 35 2E 31 38 38 2E 38 36 Range: 5.188.86
[ 112] 2E 32 35 3A 31 38 31 2E 32 31 34 2E 38 37 2E 32 .25:181.214.87.2
[ 128] 35 31 0A 50 6F 72 74 2F 50 72 6F 74 6F 20 43 6F 51.Port/Proto Co
[ 144] 75 6E 74 3A 20 32 30 31 0A 50 6F 72 74 2F 50 72 unt: 201.Port/Pr
[ 160] 6F 74 6F 20 52 61 6E 67 65 3A 20 35 32 3A 34 32 oto Range: 52:42
[ 176] 38 31 0A 81.Packet
sensor id: 0 event id: 643170305 event second: 1516290836
packet second: 1516290836 packet microsecond: 59310
linktype: 1 packet_length: 48
[ 0] 00 0D B9 45 C9 D0 00 00 5E 00 01 04 08 00 45 30 …E....^.....E0
[ 16] 00 22 00 00 40 00 3A FF 3B 18 5E 6D 74 34 5E 6E ."..@.:.;.^mt4^n
[ 32] D4 60 4F 70 65 6E 20 50 6F 72 74 3A 20 38 30 0A .`Open Port: 80.–-----------
Anything else you think I can try?Also, it seems that the alerts are still logged to the plain-text alert file as well. I would think that snort would only output to U2, so that it does not waste time by parsing the alerts to a human-readable file.
Is this another setting I should disable?
Edit, so if I do ps aux | grep snort, i get the following:
[2.4.2-RELEASE][admin@pfSense.localdomain]/: ps auxww | grep snort root 41804 0.0 5.4 680972 221892 - SNs 12:07 1:30.41 /usr/local/bin/snort -R 9814 -D -q --suppress-config-log -l /var/log/snort/snort_igb09814 --pid-path /var/run --nolock-pidfile -G 9814 -c /usr/local/etc/snort/snort_9814_igb0/snort.conf -i igb0 root 42308 0.0 0.3 47768 12328 - SN 12:07 0:15.94 /usr/local/bin/barnyard2 -r 9814 -f snort_9814_igb0.u2 --pid-path /var/run --nolock-pidfile -c /usr/local/etc/snort/snort_9814_igb0/barnyard2.conf -d /var/log/snort/snort_igb09814 -D -q root 42750 0.0 5.3 680972 217296 - SNs 12:07 1:29.13 /usr/local/bin/snort -R 13317 -D -q --suppress-config-log -l /var/log/snort/snort_igb1.21013317 --pid-path /var/run --nolock-pidfile -G 13317 -c /usr/local/etc/snort/snort_13317_igb1.210/snort.conf -i igb1.210 root 42888 0.0 0.3 47768 12320 - SN 12:07 0:16.26 /usr/local/bin/barnyard2 -r 13317 -f snort_13317_igb1.210.u2 --pid-path /var/run --nolock-pidfile -c /usr/local/etc/snort/snort_13317_igb1.210/barnyard2.conf -d /var/log/snort/snort_igb1.21013317 -D -q
And the 'output' section of one of the config files:
# Snort Output Logs # output alert_csv: alert timestamp,sig_generator,sig_id,sig_rev,msg,proto,src,srcport,dst,dstport,id,classification,priority 500K output unified2: filename snort_9814_igb0.u2, limit 128K output alert_pf: /usr/local/etc/snort/snort_9814_igb0/default,snort2c,both,kill
And the output section of one of the barnyard2 config files:
## Setup output plugins ## # syslog_full: log to a syslog receiver output alert_syslog_full: sensor_name pfSense.localdomain, local, log_facility LOG_USER, log_priority LOG_INFO
If I compare that with the snort manual, it seems there might be several issues:
-
the -l command instructs snort to log to a readable output file
-
the snort config contains an output to csv, no idea why that is here
-
the snort config contains an output alert_pf, I assume that is what pfsense use to block ips which have triggered an alert?
However, all that would mean snort always needs to log to a readable file, and then I wonder why barnyard should be used (as the output will still be parsed by snort, something we could prevent if we only would enable binary u2 output).
Looking at the barnyard2 config file, it seems something is definitely missing. As mentioned in https://forum.pfsense.org/index.php?topic=111050.0 and https://github.com/firnsy/barnyard2/blob/master/etc/barnyard2.conf, the output line in the barnyard2.conf file should contain "operation_mode complete".
-
-
I think I found it,
when manually changing alert_syslog_full to log_syslog_full in the barnyard2 config file, I get the packet details.
edit: this was not enough. The full line is:output log_syslog_full: sensor_name pfSense.localdomain, local, log_facility LOG_USER, log_priority LOG_INFO, operation_mode complete
So it is a combination fo the log_syslog_full and the operation_mode complete.
I don't think it can be set via the UI (despite that the operation mode is set to 'complete' in the barnyard settings tab) , so config is overwritten on each restart.-
Should I file a bug for the inability to log raw packet data, which is solved by manually setting log_syslog_full? The https://doc.pfsense.org/index.php/Bug_reporting says that I should first discuss it here to see if it is a legitimate bug
-
The fact that snort outputs to alert_csv and alert_pf seems to thwart the use of Barnyard a bit to me (as the processing still occurs by Snort). Should I worry about this?
-
-
Think it's Bill ( bmeeks ) that maintains Snort , might be worth running it past him via a PM.
https://forum.pfsense.org/index.php?action=profile;u=17262
-
Replied to OP via PM. This bug will be fixed in the next Snort GUI update.
Bill
-
Should anyone be interested, the following grok pattern matches most of the packets when using barnyard's log_syslog_full with complete operation_mode. Don't know if it works for all alerts yet.
SENSORNAME ([0-9a-zA-Z\.]*) BARNYARD_PREAMBLE \[SNORTIDS\[%{DATA:logtype}\]\: \[%{SENSORNAME:sensor_name}\] \] BARNYARD_SNORT_HEADER %{INT:ids_priority} \[%{INT:gid}\:%{INT:sid}\:%{INT:rev}\] %{GREEDYDATA:snort_msg_v} BARNYARD_IP_HEADER_DATA %{INT:proto} %{IP:src_ip} %{IP:dst_ip} %{INT:ip_version} %{INT:header_length} %{INT:type_of_service} %{INT:ip_packet_length} %{INT:ip_id} %{INT:flag} %{INT:offset} %{INT:ip_checksum} %{INT:unknown1} BARNYARD_SEGMENT_DATA %{BARNYARD_UDP_DATA}|%{BARNYARD_TCP_DATA}|%{BARNYARD_ICMP_DATA} BARNYARD_UDP_DATA %{INT:src_port} %{INT:dst_port} %{INT:transport_length} %{INT:transport_checksum} BARNYARD_TCP_DATA %{INT:src_port} %{INT:dst_port} %{INT:sequence} %{INT:ack} %{INT:tcp_offset} %{INT:tcp_reserved} %{INT:tcp_flags} %{INT:tcp_window} %{INT:tcp_checksum} %{INT:tcp_urgent} BARNYARD_ICMP_DATA %{INT:icmp_type} %{INT:icmp_code} %{INT:icmp_checksum} %{INT:id} %{INT:icmp_sequence} \| %{BARNYARD_PREAMBLE} \|\| %{TIMESTAMP_ISO8601:iso8601_time} %{BARNYARD_SNORT_HEADER} \|\| %{DATA:ids_classification} \|\| %{BARNYARD_IP_HEADER_DATA} \|\| (%{BARNYARD_SEGMENT_DATA} \|\| )?%{INT:frame_length} %{BASE16NUM:raw_packet}(?:\s+\|\| \|)?
Based on https://groups.google.com/forum/#!topic/barnyard2-users/7Qnpmc3VT_w , though there seemed to be a difference in the protocol data (ip_header_data in the above list).
-
Upgraded to the latest Snort version.
Now it seems like the full_syslog_full is working, but ONLY if you send it to a remote syslog server. So you can't send full events to a local syslog. I just modified my grok rules to be able to catch syslog events forwarded from the local system to a remote syslog (via status => system logs), but that format is different when you send them directly from snort to a remote syslog.
I think it should work, I'll modify the grok rules later today. However, to improve consistency with other logs forwarded via the system, perhaps it is a good idea to also allow "local full syslog complete" .
In other words, now I have:
output log_syslog_full: sensor_name snort.WAN, server 192.168.2.2, protocol udp, port 514, operation_mode complete, payload_encoding hex, log_facility LOG_USER, log_priority LOG_INFO
Which gives a different format on my remote syslog server then when using the following in combination with remote logging via the status => system logs setting:
output log_syslog_full: sensor_name snort.WAN, operation_mode complete, payload_encoding hex, log_facility LOG_USER, log_priority LOG_INFO
Kr,
Michael -
@MichaelB:
Upgraded to the latest Snort version.
Now it seems like the full_syslog_full is working, but ONLY if you send it to a remote syslog server. So you can't send full events to a local syslog. I just modified my grok rules to be able to catch syslog events forwarded from the local system to a remote syslog (via status => system logs), but that format is different when you send them directly from snort to a remote syslog.
I think it should work, I'll modify the grok rules later today. However, to improve consistency with other logs forwarded via the system, perhaps it is a good idea to also allow "local full syslog complete" .
In other words, now I have:
output log_syslog_full: sensor_name snort.WAN, server 192.168.2.2, protocol udp, port 514, operation_mode complete, payload_encoding hex, log_facility LOG_USER, log_priority LOG_INFO
Which gives a different format on my remote syslog server then when using the following in combination with remote logging via the status => system logs setting:
output log_syslog_full: sensor_name snort.WAN, operation_mode complete, payload_encoding hex, log_facility LOG_USER, log_priority LOG_INFO
Kr,
MichaelI did in fact code it so that full packet capture only works to a remote syslog server. It is not a good idea to fill up your local firewall system log with full packet captures, so I elected to not make that available.
Bill