Dshield pfsense log parser now available

  • For those of you who are interested in submitting your firewall logs to dshield (http://www.dshield.org/), they now have a pfsense firewall log parser available.

    See "pfSense Firewall" under the DShield 'Framework' Linux and UNIX clients section at http://www.dshield.org/howto.html.

    Raman Gupta

  • One small change to make it work with 1.2 RC2

    --- pfsense.pl  2008-10-28 06:22:35.000000000 -0600
    +++ pfsense.pl.new      2008-11-30 16:52:26.000000000 -0700
    @@ -1037,9 +1037,9 @@
            # This regex should work for TCP and UDP.
            # Apr 28 11:27:57.262651 rule 5/0(match): block in on tun0: > R [tcp sum ok]  180649940:180649940(0) ack 2761221649 win 0 (ttl 59, id 51314)
            # Field description:                                                                                                         Month              Day    HH    MM    SS        Extra    SrcIP              SrcPrt        DstIP              DstPort    Flags      Protocol
    -    if (($month,$day,$hour,$minute,$second,$extra, $protocol,$ipSource,$portSource,$ipDestin,$portDestin,$flags) = ($line =~ m/^([A-Z][a-z]{2}) +(\d{1,2}) (\d+):(\d+):(\d+) +(.*): (TCP|UDP) \(\d{1,2}\), length: \d+\) (\d+\.\d+\.\d+\.\d+)\.(\d*) +\> +(\d+\.\d+\.\d+\.\d+)\.(\d*): ([A-Z]*)?/ )) {
    +    if (($month,$day,$hour,$minute,$second,$extra, $protocol,$ipSource,$portSource,$ipDestin,$portDestin,$flags) = ($line =~ m/^([A-Z][a-z]{2}) +(\d{1,2}) (\d+):(\d+):(\d+) +(.*) (TCP|UDP) \(\d{1,2}\), length \d+\) (\d+\.\d+\.\d+\.\d+)\.(\d*) +\> +(\d+\.\d+\.\d+\.\d+)\.(\d*): ([A-Z]*)?/ )) {
            $flags = '' if $protocol eq 'UDP';
    -       #Aug 24 10:03:58 alix pf: 110\. 682955 rule 402/0(match): block in on ng0: (tos 0x0, ttl  52, id 21033, offset 0, flags [DF], proto: TCP (6), length: 638) > FP 586:1172(586) ack 1 win 95 <nop,nop,timestamp 173275264[|tcp]="">
    +       #Aug 24 10:03:58 alix pf: 110\. 682955 rule 402/0(match): block in on ng0: (tos 0x0, ttl  52, id 21033, offset 0, flags [DF], proto TCP (6), length 638) > FP 586:1172(586) ack 1 win 95 <nop,nop,timestamp 173275264[|tcp]="">
         } elsif (($month,$day,$hour,$minute,$second,$extra,$ipSource,$portSource,$ipDestin,$portDestin,$flags,$protocol) = ($line =~ m/^([A-Z][a-z]{2}) +(\d{1,2}) (\d+):(\d+):(\d+)\.\d* +(.*): +(\d+\.\d+\.\d+\.\d+)\.(\d*) +\> +(\d+\.\d+\.\d+\.\d+)\.(\d*): +([A-Z]*) +[(tcp|udp+)/ )) {

  • im more interested in getting the worst offender ip's from dshield and log-blocking them in the FW, might stop some spam and other attacks..

  • Agreed. It would be nice if I could use dshield's HPB, though you do need to provide them with reports in order for them to create a HPB list for you.

  • This patch might have worked on 1.2 RC2, BUT on 1.2.2, the output of the logs looks quite different from what I can tell.  The regex used in the sub parse cannot match with a single line of my logs.

    Here is the log sample line from the pfsense.pl script that you can download from dshield.org

    # Apr 29 03:27:17.466144 rule 8/0(match): block in on tun0: > icmp: echo request (id:3 seq:51426) (ttl 124, id 39118)

    and here is what an ICMP line looks like taken from my log file

    Feb 16 23:11:21 fw pf: 1\. 157100 rule 77/0(match): block in on fxp1: (tos 0x0, ttl 51, id 23360, offset 0, flags [none], proto ICMP (1), length 159) > ICMP host unreachable, length 139

    I am getting the DShield script to read the log file at /var/log/filter.log.  Is this correct?  Can someone confirm that I'm getting the log data from the right place?

    Has anyone gotten the DShield script to work on pfSense 1.2.2?


  • I'm running Dshield and 1.2.2 without issue. The patch I made for 1.2RC2 works fine with 1.2.2, though I do use a remote syslog server to insure I'm parsing the complete log. The sample you posted, does not look correct, but your line does. Here is one from my log.

    Feb 17 08:24:38 firewall pf: 052974 rule 382/0(match): block in on fxp1: (tos 0x0, ttl 6, id 62012, offset 0, flags [none], proto ICMP (1), length 64) x.x.x.x > y.y.y.y: ICMP echo request, id 29540, seq 14080, length 44

  • Ah, I'm glad to see it is me then.  I'm wondering here if anybody has actually gotten this to work directly on a pfSense box as opposed to on the syslog server where the log ends-up.  I can see some issues with going directly against the pfSense box since I understand that the logging is limited as to it maximum size (anyone knows what that size is and whether it can be changed?) and it is a circular log file.  On a busy firewall, it probably would overwrite logs entries pretty quickly.

    As for the sample lines I supplied in my previous post, I got that straight from the DShield download for pfSense (http://www.dshield.org/clients/framework/pfsense.tar.gz).

    I'm in a small network and do not have a syslog server setup.  Does syslog modify the logs that are passed to it from pfSense?

    Any suggestions as to what I can try next?

    Thanks for your help.

  • I had it installed on my pfsense box to begin with, but if I remember correctly it, it did not work as expected because of the way the log file exists on the firewall. Since it is fixed at 512k, I was losing entries as well. My recommendation would be to setup a remote syslog server. It is a trivial task.