Snort + barnyard2: full packet capture missing



  • Hi,

    I'm trying to get full packet data logged to a remote syslog server from barnyard2.
    I have the barnyard2 remote to syslog enabled, and I see packets being sent to my syslog server.

    –------------------------
    My barnyard2 config:

    
    #   barnyard2.conf
    #   barnyard2 can be found at http://www.securixlive.com/barnyard2/index.php
    #
    
    ## General Barnyard2 settings ##
    config quiet
    config daemon
    config decode_data_link
    config alert_with_interface_name
    config event_cache_size:    8192
    config show_year
    config dump_payload
    config archivedir:          /var/log/snort/snort_igb09814/barnyard2/archive
    config reference_file:      /usr/local/etc/snort/snort_9814_igb0/reference.config
    config classification_file: /usr/local/etc/snort/snort_9814_igb0/classification.config
    config sid_file:            /usr/local/etc/snort/snort_9814_igb0/sid-msg.map
    config gen_file:            /usr/local/etc/snort/snort_9814_igb0/gen-msg.map
    config hostname:            pfSense.localdomain
    config interface:           igb0
    config waldo_file:          /var/log/snort/snort_igb09814/barnyard2/9814_igb0.waldo
    config logdir:              /var/log/snort/snort_igb09814
    
    ## START user pass through ##
    
    ## END user pass through ##
    
    ## Setup input plugins ##
    input unified2
    
    ## Setup output plugins ##
    # syslog_full: log to a syslog receiver
    output alert_syslog_full: sensor_name pfSense.localdomain, server 192.168.2.2, protocol udp, port 5140, operation_mode complete, log_facility LOG_USER, log_priority LOG_INFO
    
    

    Note:
    I have tried both with and without 'config dump_payload' in the config (via the UI).
    –-----------------------------------------

    The packet that I capture (from my pfsense box to my remote syslog) when browsing to a suspicious .tk website is:

    0000   7c 20 5b 53 4e 4f 52 54 49 44 53 5b 41 4c 45 52  | [SNORTIDS[ALER
    0010   54 5d 3a 20 5b 70 66 53 65 6e 73 65 2e 6c 6f 63  T]: [pfSense.loc
    0020   61 6c 64 6f 6d 61 69 6e 5d 20 5d 20 7c 7c 20 32  aldomain] ] || 2
    0030   30 31 38 2d 30 31 2d 31 33 20 32 33 3a 31 31 3a  018-01-13 23:11:
    0040   31 30 2e 38 39 35 2b 30 30 31 20 33 20 5b 31 3a  10.895+001 3 [1:
    0050   33 39 38 36 37 3a 33 5d 20 49 4e 44 49 43 41 54  39867:3] INDICAT
    0060   4f 52 2d 43 4f 4d 50 52 4f 4d 49 53 45 20 53 75  OR-COMPROMISE Su
    0070   73 70 69 63 69 6f 75 73 20 2e 74 6b 20 64 6e 73  spicious .tk dns
    0080   20 71 75 65 72 79 20 7c 7c 20 6d 69 73 63 2d 61   query || misc-a
    0090   63 74 69 76 69 74 79 20 7c 7c 20 31 37 20 31 39  ctivity || 17 19
    00a0   32 2e 31 36 38 2e 33 2e 31 30 30 20 31 39 32 2e  2.168.3.100 192.
    00b0   31 36 38 2e 33 2e 31 20 7c 7c 20 35 38 31 38 35  168.3.1 || 58185
    00c0   20 35 33 20 7c 7c 20 0a 20 7c 00                  53 || . |.
    

    –--------------------------------------------

    As you see, it does not contain any payload data, just the alert. I have read the following related topics, but no solution is provided:

    Perhaps there is a difference between the setting 'alert_syslog_full' and 'log_syslog_full' in barnyard config, but that's not something I can enable via the UI.
    If I do NOT enable barnyard2, the packet data IS logged in ASCII format:

    Log directory BEFORE enabling barnyard2 (notice size of the log file containing packet data):
    #/var/log/snort/snort_igb09814: ls -alhrt

    -rw–-----  1 root  wheel   2.3K Jan 13 21:27 snort.log.1515755290
    -rw-r--r--  1 root  wheel   360K Jan 13 21:27 alert
    
    

    Log directory AFTER enabling barnyard2:

    -rw–-----  1 root  wheel   216K Jan 13 22:39 alert
    -rw-------  1 root  wheel     0B Jan 13 22:39 snort_9814_igb0.u2.1515879592
    
    

    So before enabling barnyard, the packet data was logged in a readable file. After enabling barnyard, I'm not sure what happened, but the file seems empty.
    Is this expected behaviour?



  • Update:

    I tried doing the same by sending the logs to the PFSense system logs, and then forwarding the logs to the remote server. Same issue (tested with portscan and suspicious tk dns query).


  • Galactic Empire

    I've had a few goes at getting packets to syslog working but never been able to.

    Does any more show when doing a u2spewfoo on the router ?

    [2.4.2-RELEASE][admin@pfsense]/var/log/snort/snort_igb0.256577: u2spewfoo snort_56577_igb0.2.u2.1515802084

    (IPv6 Event)
    sensor id: 0 event id: 3707830273 event second: 1515927039 event microsecond: 247445
    sig id: 2025146 gen id: 1 revision: 1 classification: 5
    priority: 2 ip source: 2a02:XXXX.XXXX:2::14 ip destination: 2a02:XXXX.XXXX:2::1
    src port: 52064 dest port: 53 protocol: 17 impact_flag: 0 blocked: 0
    mpls label: 0 vland id: 0 policy id: 0

    (ExtraDataHdr)
    event type: 4 event length: 48

    (ExtraData)
    sensor id: 0 event id: 3707830273 event second: 1515927039
    type: 11 datatype: 1 bloblength: 24 IPv6 Source Address: 2a02:XXXX.XXXX:2::14

    (ExtraDataHdr)
    event type: 4 event length: 48

    (ExtraData)
    sensor id: 0 event id: 3707830273 event second: 1515927039
    type: 12 datatype: 1 bloblength: 24 IPv6 Destination Address: 2a02:XXXX.XXXX:2::1

    Packet
    sensor id: 0 event id: 3707830273 event second: 1515927039
    packet second: 1515927039 packet microsecond: 247445
    linktype: 1 packet_length: 91
    [    0] 00 08 A2 0A 9D CB 00 3E E1 C1 AF 07 86 DD 60 0E  …....>......`.

    CB 60 00 35 00 25 64 F0 00 F6  .......`.5.%d...
    [  64] 01 00 00 01 00 00 00 00 00 00 04 66 72 65 64 02  …........fred.
    [  80] 67 72 03 63 6F 6D 00 00 01 00 01                gr.com…..
    [2.4.2-RELEASE][admin@pfsense]/var/log/snort/snort_igb0.256577:



  • Yes, that contains something:

    [2.4.2-RELEASE][admin@pfSense.localdomain]/var/log/snort/snort_igb09814: u2spewfoo snort_9814_igb0.u2.1516273671

    (Event)
            sensor id: 0    event id: 643170305    event second: 1516290836        event microsecond: 59309
            sig id: 5      gen id: 122    revision: 1      classification: 3
            priority: 2    ip source: xx        ip destination: xx
            src port: 0    dest port: 0    protocol: 6    impact_flag: 0  blocked: 0

    Packet
            sensor id: 0    event id: 643170305    event second: 1516290836
            packet second: 1516290836      packet microsecond: 59309
            linktype: 1    packet_length: 179
    [    0] 00 0D B9 45 C9 D0 00 00 5E 00 01 04 08 00 45 30  …E....^.....E0
    [  16] 00 A5 00 00 40 00 3A FF 3B 18 5E 6D 74 34 5E 6E  ….@.:.;.^mt4^n
    [  32] D4 60 50 72 69 6F 72 69 74 79 20 43 6F 75 6E 74  .`Priority Count
    [  48] 3A 20 30 0A 43 6F 6E 6E 65 63 74 69 6F 6E 20 43  : 0.Connection C
    [  64] 6F 75 6E 74 3A 20 32 30 30 0A 49 50 20 43 6F 75  ount: 200.IP Cou
    [  80] 6E 74 3A 20 33 0A 53 63 61 6E 6E 65 72 20 49 50  nt: 3.Scanner IP
    [  96] 20 52 61 6E 67 65 3A 20 35 2E 31 38 38 2E 38 36  Range: 5.188.86
    [  112] 2E 32 35 3A 31 38 31 2E 32 31 34 2E 38 37 2E 32  .25:181.214.87.2
    [  128] 35 31 0A 50 6F 72 74 2F 50 72 6F 74 6F 20 43 6F  51.Port/Proto Co
    [  144] 75 6E 74 3A 20 32 30 31 0A 50 6F 72 74 2F 50 72  unt: 201.Port/Pr
    [  160] 6F 74 6F 20 52 61 6E 67 65 3A 20 35 32 3A 34 32  oto Range: 52:42
    [  176] 38 31 0A                                        81.

    Packet
            sensor id: 0    event id: 643170305    event second: 1516290836
            packet second: 1516290836      packet microsecond: 59310
            linktype: 1    packet_length: 48
    [    0] 00 0D B9 45 C9 D0 00 00 5E 00 01 04 08 00 45 30  …E....^.....E0
    [  16] 00 22 00 00 40 00 3A FF 3B 18 5E 6D 74 34 5E 6E  ."..@.:.;.^mt4^n
    [  32] D4 60 4F 70 65 6E 20 50 6F 72 74 3A 20 38 30 0A  .`Open Port: 80.

    –-----------
    Anything else you think I can try?

    Also, it seems that the alerts are still logged to the plain-text alert file as well. I would think that snort would only output to U2, so that it does not waste time by parsing the alerts to a human-readable file.
    Is this another setting I should disable?


    Edit, so if I do ps aux | grep snort, i get the following:

    
    [2.4.2-RELEASE][admin@pfSense.localdomain]/: ps auxww | grep snort
    root    41804   0.0  5.4 680972 221892  -  SNs  12:07       1:30.41 /usr/local/bin/snort -R 9814 -D -q --suppress-config-log -l /var/log/snort/snort_igb09814 --pid-path /var/run --nolock-pidfile -G 9814 -c /usr/local/etc/snort/snort_9814_igb0/snort.conf -i igb0
    root    42308   0.0  0.3  47768  12328  -  SN   12:07       0:15.94 /usr/local/bin/barnyard2 -r 9814 -f snort_9814_igb0.u2 --pid-path /var/run --nolock-pidfile -c /usr/local/etc/snort/snort_9814_igb0/barnyard2.conf -d /var/log/snort/snort_igb09814 -D -q
    root    42750   0.0  5.3 680972 217296  -  SNs  12:07       1:29.13 /usr/local/bin/snort -R 13317 -D -q --suppress-config-log -l /var/log/snort/snort_igb1.21013317 --pid-path /var/run --nolock-pidfile -G 13317 -c /usr/local/etc/snort/snort_13317_igb1.210/snort.conf -i igb1.210
    root    42888   0.0  0.3  47768  12320  -  SN   12:07       0:16.26 /usr/local/bin/barnyard2 -r 13317 -f snort_13317_igb1.210.u2 --pid-path /var/run --nolock-pidfile -c /usr/local/etc/snort/snort_13317_igb1.210/barnyard2.conf -d /var/log/snort/snort_igb1.21013317 -D -q
    
    

    And the 'output' section of one of the config files:

    # Snort Output Logs #
    output alert_csv: alert timestamp,sig_generator,sig_id,sig_rev,msg,proto,src,srcport,dst,dstport,id,classification,priority 500K
    
    output unified2: filename snort_9814_igb0.u2, limit 128K
    output alert_pf: /usr/local/etc/snort/snort_9814_igb0/default,snort2c,both,kill
    

    And the output section of one of the barnyard2 config files:

    
    ## Setup output plugins ##
    # syslog_full: log to a syslog receiver
    output alert_syslog_full: sensor_name pfSense.localdomain, local, log_facility LOG_USER, log_priority LOG_INFO
    
    

    If I compare that with the snort manual, it seems there might be several issues:

    • the -l command instructs snort to log to a readable output file

    • the snort config contains an output to csv, no idea why that is here

    • the snort config contains an output alert_pf, I assume that is what pfsense use to block ips which have triggered an alert?

    However, all that would mean snort always needs to log to a readable file, and then I wonder why barnyard should be used (as the output will still be parsed by snort, something we could prevent if we only would enable binary u2 output).

    Looking at the barnyard2 config file, it seems something is definitely missing. As mentioned in https://forum.pfsense.org/index.php?topic=111050.0 and https://github.com/firnsy/barnyard2/blob/master/etc/barnyard2.conf, the output line in the barnyard2.conf file should contain "operation_mode complete".



  • I think I found it,

    when manually changing  alert_syslog_full to log_syslog_full in the barnyard2 config file, I get the packet details.
    edit: this was not enough. The full line is:

    output log_syslog_full: sensor_name pfSense.localdomain, local, log_facility LOG_USER, log_priority LOG_INFO, operation_mode complete
    

    So it is a combination fo the log_syslog_full and the operation_mode complete.
    I don't think it can be set via the UI (despite that the operation mode is set to 'complete' in the barnyard settings tab) , so config is overwritten on each restart.

    • Should I file a bug for the inability to log raw packet data, which is solved by manually setting log_syslog_full? The https://doc.pfsense.org/index.php/Bug_reporting says that I should first discuss it here to see if it is a legitimate bug

    • The fact that snort outputs to alert_csv and alert_pf seems to thwart the use of Barnyard a bit to me (as the processing still occurs by Snort). Should I worry about this?


  • Galactic Empire

    Think it's Bill ( bmeeks ) that maintains Snort , might be worth running it past him via a PM.

    https://forum.pfsense.org/index.php?action=profile;u=17262



  • Replied to OP via PM.  This bug will be fixed in the next Snort GUI update.

    Bill



  • Should anyone be interested, the following grok pattern matches most of the packets when using barnyard's log_syslog_full with complete operation_mode. Don't know if it works for all alerts yet.

    SENSORNAME ([0-9a-zA-Z\.]*)
    BARNYARD_PREAMBLE [SNORTIDS[%{DATA:logtype}]\: [%{SENSORNAME:sensor_name}] ]
    BARNYARD_SNORT_HEADER %{INT:ids_priority} [%{INT:gid}\:%{INT:sid}\:%{INT:rev}] %{GREEDYDATA:snort_msg_v}
    BARNYARD_IP_HEADER_DATA %{INT:proto} %{IP:src_ip} %{IP:dst_ip} %{INT:ip_version} %{INT:header_length} %{INT:type_of_service} %{INT:ip_packet_length} %{INT:ip_id} %{INT:flag} %{INT:offset} %{INT:ip_checksum} %{INT:unknown1}
    BARNYARD_SEGMENT_DATA %{BARNYARD_UDP_DATA}|%{BARNYARD_TCP_DATA}|%{BARNYARD_ICMP_DATA}
    BARNYARD_UDP_DATA %{INT:src_port} %{INT:dst_port} %{INT:transport_length} %{INT:transport_checksum}
    BARNYARD_TCP_DATA %{INT:src_port} %{INT:dst_port} %{INT:sequence} %{INT:ack} %{INT:tcp_offset} %{INT:tcp_reserved} %{INT:tcp_flags} %{INT:tcp_window} %{INT:tcp_checksum} %{INT:tcp_urgent}
    BARNYARD_ICMP_DATA %{INT:icmp_type} %{INT:icmp_code} %{INT:icmp_checksum} %{INT:id} %{INT:icmp_sequence}
    
    \| %{BARNYARD_PREAMBLE} \|\| %{TIMESTAMP_ISO8601:iso8601_time} %{BARNYARD_SNORT_HEADER} \|\| %{DATA:ids_classification} \|\| %{BARNYARD_IP_HEADER_DATA} \|\| (%{BARNYARD_SEGMENT_DATA} \|\| )?%{INT:frame_length} %{BASE16NUM:raw_packet}(?:\s+\|\|   \|)?
    
    

    Based on https://groups.google.com/forum/#!topic/barnyard2-users/7Qnpmc3VT_w , though there seemed to be a difference in the protocol data (ip_header_data in the above list).



  • Upgraded to the latest Snort version.

    Now it seems like the full_syslog_full is working, but ONLY if you send it to a remote syslog server. So you can't send full events to a local syslog. I just modified my grok rules to be able to catch syslog events forwarded from the local system to a remote syslog (via status => system logs), but that format is different when you send them directly from snort to a remote syslog.

    I think it should work, I'll modify the grok rules later today. However, to improve consistency with other logs forwarded via the system, perhaps it is a good idea to also allow "local full syslog complete" .

    In other words, now I have:

    output log_syslog_full: sensor_name snort.WAN, server 192.168.2.2, protocol udp, port 514, operation_mode complete, payload_encoding hex, log_facility LOG_USER, log_priority LOG_INFO
    

    Which gives a different format on my remote syslog server then when using the following in combination with remote logging via the status => system logs setting:

    output log_syslog_full: sensor_name snort.WAN, operation_mode complete, payload_encoding hex, log_facility LOG_USER, log_priority LOG_INFO
    

    Kr,
    Michael



  • @MichaelB:

    Upgraded to the latest Snort version.

    Now it seems like the full_syslog_full is working, but ONLY if you send it to a remote syslog server. So you can't send full events to a local syslog. I just modified my grok rules to be able to catch syslog events forwarded from the local system to a remote syslog (via status => system logs), but that format is different when you send them directly from snort to a remote syslog.

    I think it should work, I'll modify the grok rules later today. However, to improve consistency with other logs forwarded via the system, perhaps it is a good idea to also allow "local full syslog complete" .

    In other words, now I have:

    output log_syslog_full: sensor_name snort.WAN, server 192.168.2.2, protocol udp, port 514, operation_mode complete, payload_encoding hex, log_facility LOG_USER, log_priority LOG_INFO
    

    Which gives a different format on my remote syslog server then when using the following in combination with remote logging via the status => system logs setting:

    output log_syslog_full: sensor_name snort.WAN, operation_mode complete, payload_encoding hex, log_facility LOG_USER, log_priority LOG_INFO
    

    Kr,
    Michael

    I did in fact code it so that full packet capture only works to a remote syslog server.  It is not a good idea to fill up your local firewall system log with full packet captures, so I elected to not make that available.

    Bill