Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Dshield send pfsense log - issue with reading log entry time?

    Scheduled Pinned Locked Moved General pfSense Questions
    2 Posts 2 Posters 851 Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • johnpozJ
      johnpoz LAYER 8 Global Moderator
      last edited by

      So stumbled upon an updated script to send pfsense logs to dshields

      https://www.dshield.org/forums/diary/Updated+PFSense+Client/20937/1

      And I have gotten it to work, had a few bumps for sure somehow when I had pasted it got a CR in the line format so logs were being sent in 2 lines and dshield not putting anything in.  I also had a weird setup with my email notification where I was sending to gmail via port 587 and that was working for actual notifications, but this script was choking.  Changed to port 465 and checked Enable SMTP over SSL/TLS and now this script will send the email.

      And it does send email with the listing from the logs.

      2016-09-10 05:43:46 +00:00      94snipped56      1      186.118.8.122  48326  24.13.snipped    23      TCP    S
      2016-09-10 05:43:49 +00:00      94snipped56      1      186.118.8.122  48326  24.13.snipped    23      TCP    S
      2016-09-10 05:44:26 +00:00      94snipped56      1      166.250.144.100 28403  24.13.snipped    23      TCP    S
      2016-09-10 05:44:50 +00:00      94snipped56      1      168.1.128.50    22101  24.13.snipped    995    TCP    S
      2016-09-10 05:49:22 +00:00      94snipped56      1      106.187.97.102  35024  24.13.snipped    995    TCP    S
      2016-09-10 05:53:19 +00:00      94snipped56      1      45.33.116.208  37018  24.13.snipped    5904    TCP    S

      I had changed my TZ to UTC, normally I run it in Central time, but was wondering if that was causing a issue with how it figures out when last entry was sent, etc.  So above is an example of how it sends the logs.. Where my UID been snipped out some of it, and my public IP I have snipped out… But this works and get notifications back from dshield

      Errors-To: bounces@dshield.org
      Reply-To: info@dshield.org
      From: admin@dshield.org

      Authorized Userid: 94snipped56
                                Format: DSHIELD
                              Timezone: +00:00

      Lines in file: 565
                        Lines rejected: none
      Unique lines written to database: 565
        identical lines are added up on import.

      Lines written to database (up to 10):
      2016-09-10 05:43:46 +00:00 94snipped56 1 186.118.8.122 48326 24.13.snipped 23 TCP S
      2016-09-10 05:43:49 +00:00 94snipped56 1 186.118.8.122 48326 24.13.snipped 23 TCP S
      2016-09-10 05:44:26 +00:00 94snipped56 1 166.250.144.100 28403 24.13.snipped 23 TCP S
      2016-09-10 05:44:50 +00:00 94snipped56 1 168.1.128.50 22101 24.13.snipped 995 TCP S
      2016-09-10 05:49:22 +00:00 94snipped56 1 106.187.97.102 35024 24.13.snipped 995 TCP S

      So that looks like great everything is working..  But the problem is its not sending when it should.  So I have cron job running every 30 mins, at 11 and 41 past the hour.. What it does is say

      Sep 11 13:41:03 php /root/bin/dshieldpfsense.php: no new lines added to log since last run OK
      Sep 11 13:11:03 php /root/bin/dshieldpfsense.php: no new lines added to log since last run OK
      Sep 11 12:41:04 php /root/bin/dshieldpfsense.php: no new lines added to log since last run OK
      Sep 11 12:11:03 php /root/bin/dshieldpfsense.php: no new lines added to log since last run OK
      Sep 11 11:41:03 php /root/bin/dshieldpfsense.php: no new lines added to log since last run OK
      Sep 11 11:11:04 php /root/bin/dshieldpfsense.php: no new lines added to log since last run OK

      But clearly if you look there have been multiple blocks in the log that should be sent up. Between times when script last ran

      So you can view the script directly here
      https://isc.sans.edu/clients/dshieldpfsense.txt or you can find the link in the above link article about it, etc.

      I have emailed the author, but its been a week and nothing back..  And I posted on the above article and nothing.  My guess is it has something to do with how pfsense actual uses clog and maybe is not posting the info into the filter.log its opening so when the script run it really just isn't seeing anything new?  And have to wait til something fills up for new stuff to get posted to the log based upon the log file size?

      Here is snip of the code that looks at last time, runs thru log and then depending errors and exits, etc..

      
      # check when we ran last.
      if ( file_exists('/var/run/dshieldlastts') ) {
        $lasttime=file_get_contents('/var/run/dshieldlastts');
      }
      
      # read the log
      $log=fopen("/var/log/filter.log","r");
      while(!feof($log)) {
              $line = fgets($log);
              $line = rtrim($line);
      # the name of this function changed in Pfsense 2.3
              if ( $config['version']>=15 ) {
      	        $flent = parse_firewall_log_line(trim($line));
      	} else {
      	        $flent = parse_filter_line(trim($line));
              }
      
      # eliminating ICMP (we don't log that) and TCP with FA and RA flags as these are usually false positives
      
              if ($flent != "" && in_array($flent['interface'],$interfaces) && $flent['proto']!='ICMP' && $flent['tcpflags']!='FA' && $flent['tcpflags']!='RA' ) {
        	   $time=strtotime($flent['time']);
      
      # check if this log line is newer then the last one we processesed.
         if ( $time>$lasttime) {
            $linesout.=date("Y-m-d H:i:s P",$time)."\t$uid\t1\t{$flent['srcip']}\t{$flent['srcport']}\t{$flent['dstip']}\t{$flent['dstport']}\t{$flent['proto']}\t{$flent['tcpflags']}\n";
         $flent='';
         $linecnt++;
      }
              }
      }
      fclose($log);
      
      # done reading the log
      
      # dealing with errors
      if ( $lasttime>=$time ) {
        log_error("no new lines added to log since last run OK");
        exit();
      }
      if ( $linecnt==0 ){
         log_error("no new lines found to submit to dshield OK");
         exit();
      }
      
      # safe the "last run" time stamp for the next time we will run.
      
      file_put_contents('/var/run/dshieldlastts',$time);
      
      

      How to correct?  What I would expect to happen is how often I run this script it should send up the amount of hits in the log since the last time it ran be it 1 hour, 2 hours, 30 mins etc.. so normally you would see it submit small numbers of entries every so often, not like 900 entries in 1 email, and then not run submit again for 12 some hours, etc.

      An intelligent man is sometimes forced to be drunk to spend time with his fools
      If you get confused: Listen to the Music Play
      Please don't Chat/PM me for help, unless mod related
      SG-4860 24.11 | Lab VMs 2.8, 24.11

      1 Reply Last reply Reply Quote 0
      • G
        grandrivers
        last edited by

        just a wild thought this might be a nice add to pfblockerng or both ids/ips packages

        pfsense plus 25.03 super micro A1SRM-2558F
        C2558 32gig ECC  60gig SSD

        1 Reply Last reply Reply Quote 0
        • First post
          Last post
        Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.