Suricata 1.4.6 pkg v1.0.2 – Update Release Notes



  • Suricata 1.4.6 pkg v1.0.2

    This update for the Suricata package fixes five bugs and adds one new feature.

    Bug Fixes:

    1.  Saved changes to auto-refresh parameters on the ALERTS tab are not reflected in the GUI display.

    2.  Suricata will install but not startup on pfSense 2.2 snapshot systems.

    3.  Call-time pass-by-reference is now deprecated in PHP 5.5 used on pfSense 2.2 installs.

    4.  Security vulnerability exists in LOGS BROWSER tab code.

    5.  Barnyard2 sensor name parameter defaults back to hostname when user sets it to blank.

    New Features:

    1.  The ability to create a new Suricata interface instance by "duplicating" an existing instance is now available. This feature allows an admin to duplicate the settings for an existing interface over to a new Suricata instance on a different physical interface. All settings are duplicated with the exception of: (1) the interface name, (2) the interface description, (3) the Home Net, External Net, Pass List and Suppress List values. The Home Net, External Net, Pass List and Suppress List values for the new instance are set to "default". The "duplicate" feature is activated by clicking the plus (+) icon next to the desired "source" instance.


  • Moderator

    Great Job Bill,

    When do you think we can expect the next Snort Package release?



  • @BBcan17:

    Great Job Bill,

    When do you think we can expect the next Snort Package release?

    It's posted for review.  The Core Team guys were busy with BSDCan stuff recently, but I think they are all back to work now.  I'm hoping this week, but I am at the mercy of the team to review and merge… ;)

    As soon as the posted update is approved and merged, I will get the Snort binary updated to 2.9.6.1.  That version is now posted on Freshports.  After that, I will be working on moving Suricata to the 2.0 binary code tree.

    Bill


  • Moderator

    Hopefully they all didn't end up at the post-conf party… wont see much till the week after?  :)

    I noticed that I can't sort some of the Alerts Page Columns in the existing Snort and the new Suricata Pakage. I tried it in Chrome and IE10.

    (These can be sorted)
    Date Pri Proto Class

    (These cannot be sorted)
    Src SPort Dst DPort SID Desription



  • @BBcan17:

    Hopefully they all didn't end up at the post-conf party… wont see much till the week after?  :)

    I noticed that I can't sort some of the Alerts Page Columns in the existing Snort and the new Suricata Pakage. I tried it in Chrome and IE10.

    (These can be sorted)
    Date Pri Proto Class

    (These cannot be sorted)
    Src SPort Dst DPort SID Desription

    I don't know at the moment what's wrong.  Nothing changed in the package code.  Looks like something changed with the CSS in pfSense itself maybe.  Not sure.  I will research deeper.  I validated the behavior as you described.  This simply uses a pre-defined CSS class.  For the columns you mentioned that no longer sort, a custom key is used.  That has not changed, but apparently the way it is interpreted by the CSS in the latest pfSense updates has changed.  I have a 2.0.3 VM I just tested in, and the exact same Snort GUI code works there just fine (sorts all the columns properly).

    Bill



  • Hi Bill, firstly great job.

    And would like to ask if the PPPoE issue still ongoing in Suricata?



  • @val:

    Hi Bill, firstly great job.

    And would like to ask if the PPPoE issue still ongoing in Suricata?

    The underlying binary is unchanged for this release, so yeah it probably still won't recognize the interface type.  I am working now on moving to the 2.0 code branch for the Suricata binary.  Maybe that will work better with a PPPoE interface.

    Bill



  • @bmeeks:

    @val:

    Hi Bill, firstly great job.

    And would like to ask if the PPPoE issue still ongoing in Suricata?

    The underlying binary is unchanged for this release, so yeah it probably still won't recognize the interface type.  I am working now on moving to the 2.0 code branch for the Suricata binary.  Maybe that will work better with a PPPoE interface.

    Bill

    Righto, Thanks, I will keep an eye out once you updated the underlying binary and I will test it again.

    Once again, thank you very much.



  • Started testing suricata, already found a bug.
    Interface variables for ports, do not allow ports to be entered, with an error of "Only aliases are allowed"

    HTTP_PORTS box contains (without quotes): "80,443"

    When saving, the above mentioned error is shown.

    Edit:

    Interface variables, add FTP_SERVERS
    Interface variables, add SSH_SERVERS
    add option to change the syslog facility
    blocked tabs does not update on new alerts, host shown as blocked on the alerts tab. On that note, where is the suricata blocked hosts table??? had to restart the box, ok now
    automatically generated Pass List not recognizing IPv6 addresses
    in edit rules tab, the icons do not behave as expected. disabling a user enabled rule, makes it red (normal or faded, depending on how upstream handles it), which means if upstream enables it/changes it, it could start generating false positives. Clicking the disable button on the >ALERTS< tab though, behaves correctly, the icon is faded yellow (user disabled). I noticed this a couple of months back in snort also, it makes it very hard to permanently disable rules from the rule edit tab.

    Feature request: sync to CARP slave


  • Moderator

    I think the error means you need to create an "alias" with those ports listed, and than add the "alias" to the Suricata variables boxes. This is similar to Snort variables.

    So don't try to directly enter the ports in the variable boxes.



  • Assign a variable to a variable? that's counterintuitive. It needs to behave the same way as the hosts variables above it.


  • Moderator

    Have to keep you on your toes!!  :)

    I agree, but that's how it works…



  • Here's a screenshot from snort, I'm already on my toes, don't need any more anxiety by switching over  :P

    insert Jobs speech "You can change it!"

    They need to accept both aliases and values in those fields, it would make setting it up easier.

    Notes on screenshot: 28days removal, almost no false positives, based on custom rules. I do believe I hold the forum record for most hosts blocked by snort, but then again I broke the snort package by enabling too many rules  ;D




  • @jflsakfja:

    Assign a variable to a variable? that's counterintuitive. It needs to behave the same way as the hosts variables above it.

    This is in keeping with the pfSense overall philosophy.  I have mentioned this analogy before, but many of the big "commercial pay" firewalls follow the same idea.  Checkpoint is one I am quite familiar with as I managed those for many years.  They use "objects" instead of "aliases", but the concept and operation is exactly the same.

    If you approach this with an open mind and think about the entire firewall configuration, it can actually make quite good sense.  Suppose you have a web server farm and use the standard 80 and 443 ports and maybe one non-standard port for something.  You would want your firewall and Snort or Suricata rules to match up with respect to inspecting those ports.  You could manually enter port numbers, but then as your rules get more complex you can easily make a change in one place and forget it in two others.  With an alias, you sort of can't make that error.  You create the alias with the ports, then use that alias in all places where port numbers are required.  If you need to change a port 2 years from now, you edit the alias and the new port is correct everywhere… :D

    When I first started using Checkpoint products I felt like you did -- just let me enter a physical port number.  But as I gained experience and starting creating and managing much more complicated rule sets, I began to appreciate the real power of "objects in Checkpoint" or "aliases in pfSense".  Don't forget that starting in 2.0 of pfSense you can nest aliases.  This can be helpful in some situations.

    Bill



  • @bmeeks:

    This is in keeping with the pfSense overall philosophy.  I have mentioned this analogy before, but many of the big "commercial pay" firewalls follow the same idea.  Checkpoint is one I am quite familiar with as I managed those for many years.  They use "objects" instead of "aliases", but the concept and operation is exactly the same.

    If you approach this with an open mind and think about the entire firewall configuration, it can actually make quite good sense.  Suppose you have a web server farm and use the standard 80 and 443 ports and maybe one non-standard port for something.  You would want your firewall and Snort or Suricata rules to match up with respect to inspecting those ports.  You could manually enter port numbers, but then as your rules get more complex you can easily make a change in one place and forget it in two others.  With an alias, you sort of can't make that error.  You create the alias with the ports, then use that alias in all places where port numbers are required.  If you need to change a port 2 years from now, you edit the alias and the new port is correct everywhere… :D

    When I first started using Checkpoint products I felt like you did -- just let me enter a physical port number.  But as I gained experience and starting creating and managing much more complicated rule sets, I began to appreciate the real power of "objects in Checkpoint" or "aliases in pfSense".  Don't forget that starting in 2.0 of pfSense you can nest aliases.  This can be helpful in some situations.

    Bill

    If you can't beat them, join them  ;D
    I'll try it out and see how it works. Bill, have you seen my other post above? Those need fixing



  • @jflsakfja:

    @bmeeks:

    This is in keeping with the pfSense overall philosophy.  I have mentioned this analogy before, but many of the big "commercial pay" firewalls follow the same idea.  Checkpoint is one I am quite familiar with as I managed those for many years.  They use "objects" instead of "aliases", but the concept and operation is exactly the same.

    If you approach this with an open mind and think about the entire firewall configuration, it can actually make quite good sense.  Suppose you have a web server farm and use the standard 80 and 443 ports and maybe one non-standard port for something.  You would want your firewall and Snort or Suricata rules to match up with respect to inspecting those ports.  You could manually enter port numbers, but then as your rules get more complex you can easily make a change in one place and forget it in two others.  With an alias, you sort of can't make that error.  You create the alias with the ports, then use that alias in all places where port numbers are required.  If you need to change a port 2 years from now, you edit the alias and the new port is correct everywhere… :D

    When I first started using Checkpoint products I felt like you did -- just let me enter a physical port number.  But as I gained experience and starting creating and managing much more complicated rule sets, I began to appreciate the real power of "objects in Checkpoint" or "aliases in pfSense".  Don't forget that starting in 2.0 of pfSense you can nest aliases.  This can be helpful in some situations.

    Bill

    If you can't beat them, join them  ;D
    I'll try it out and see how it works. Bill, have you seen my other post above? Those need fixing

    Yes, I saw the other reports.  I will take a look.  I have a really hard (well, almost impossible) time testing IPv6 stuff because my ISP does not provide or allow it.  Anything I do has to all be whatever VMware can emulate/simulate.

    Which syslog do you want to change the facility for?  The alerts output option or the general Suricata operations logging to syslog?  There is an option for changing the syslog facility if you use the Barnyard2 output plugin.

    The thing with the forced enable/disable icons was done as part of fixing it so you could revert back to "default" for a particular rule.  Previously you only had the option of "forcing it on" or "forcing it off".  You could never click back around to just the default state.  If you have a rule "forced", it will stay in whatever state you selected because it stores that SID info in the config.xml file.

    Bill



  • Ideally I would like to change both syslog facilities. The auth facility used by suricata now is miles from being relevant to it. It's for logins and elevating user priviledges, not an IDS/IPS. I believe that's for the alerts. I think I saw local5 mentioned somewhere, that could be for the daemon part of suricata, which should be…well... daemon.

    If a rule is default enabled (red), clicking it sets it to user disabled (pale yellow). Enabling all the rules though, and then clicking a particular rule does not set it to pale yellow, but what ever the default was. If I misunderstood something, please correct me.



  • @jflsakfja:

    Ideally I would like to change both syslog facilities. The auth facility used by suricata now is miles from being relevant to it. It's for logins and elevating user priviledges, not an IDS/IPS. I believe that's for the alerts. I think I saw local5 mentioned somewhere, that could be for the daemon part of suricata, which should be…well... daemon.

    If a rule is default enabled (red), clicking it sets it to user disabled (pale yellow). Enabling all the rules though, and then clicking a particular rule does not set it to pale yellow, but what ever the default was. If I misunderstood something, please correct me.

    I need to go back and look at how things are configured in pfSense, but off the top of my head I seem to remember that only AUTH facility messages would wind up in the system log.  Other facilities are hard-coded directed to some other files if I recall.  I can allow the Suricata facility to be anything (both alerts and general output are configurable), but only certain settings will actually cause the messages to show up in the system log on pfSense.  I chose the defaults the way I did simply to insure the output showed up in the system log on pfSense in the expected file.  If you really want custom facility outputs, I suggest using the Barnyard2 output options and feed the data to a remote syslog server.

    I see what you mean about the rule icon colors.  As I mentioned, the idea is to now "default" the rule back when clicked a second time.  So if the rule is default disabled (pale red) and you click it, then it is forced to enabled and turns yellow.  If it was forced "enabled" and you click it a second time, it reverts to its default state (that is, any "forced state" is simply removed).  I have been thinking about the best way to incorporate the functionality of enablesid, disablesid and modifysid into the package (like Pulled Pork and some other third-party tools do).  That would solve your problem of wanting to make sure a rule stays "always disabled".

    Bill



  • @bmeeks:

    I have a really hard (well, almost impossible) time testing IPv6 stuff because my ISP does not provide or allow it.  Anything I do has to all be whatever VMware can emulate/simulate.

    You could use an IPv6 Tunnel Broker like HE.NET. I've started to use them ever since IPv6 was added to pfSense testing.



  • @bmeeks:

    I need to go back and look at how things are configured in pfSense, but off the top of my head I seem to remember that only AUTH facility messages would wind up in the system log.  Other facilities are hard-coded directed to some other files if I recall.  I can allow the Suricata facility to be anything (both alerts and general output are configurable), but only certain settings will actually cause the messages to show up in the system log on pfSense.  I chose the defaults the way I did simply to insure the output showed up in the system log on pfSense in the expected file.  If you really want custom facility outputs, I suggest using the Barnyard2 output options and feed the data to a remote syslog server.

    I've been using snort with local0 for a few years now (added via advanced options) and the logs showed up both on the pfsense log page, and the remote syslog.

    As it stands now, the logs do get sent to the remote syslog, but they are tagged with the wrong facility (auth). On the remote syslog I can just direct them based on their tags, but it's not ideal. As I said, the auth facility should be used for logging user logins/logouts/users elevating priviledges through sudo (for example). Barnyard2 shouldn't be necessary, since the logs are already pushed to syslog, but just tagged wrongly.

    WRT IPv6, I too highly recommend HE.net's tunnel service. It provides everything you need to experiment with IPv6.



  • @jflsakfja:

    @bmeeks:

    I need to go back and look at how things are configured in pfSense, but off the top of my head I seem to remember that only AUTH facility messages would wind up in the system log.  Other facilities are hard-coded directed to some other files if I recall.  I can allow the Suricata facility to be anything (both alerts and general output are configurable), but only certain settings will actually cause the messages to show up in the system log on pfSense.  I chose the defaults the way I did simply to insure the output showed up in the system log on pfSense in the expected file.  If you really want custom facility outputs, I suggest using the Barnyard2 output options and feed the data to a remote syslog server.

    I've been using snort with local0 for a few years now (added via advanced options) and the logs showed up both on the pfsense log page, and the remote syslog.

    As it stands now, the logs do get sent to the remote syslog, but they are tagged with the wrong facility (auth). On the remote syslog I can just direct them based on their tags, but it's not ideal. As I said, the auth facility should be used for logging user logins/logouts/users elevating priviledges through sudo (for example). Barnyard2 shouldn't be necessary, since the logs are already pushed to syslog, but just tagged wrongly.

    WRT IPv6, I too highly recommend HE.net's tunnel service. It provides everything you need to experiment with IPv6.

    I will put altering the syslog facility in the next Suricata release.  I'm hoping that coincides with the 2.0.1 Suricata binary as well.  That's my plan at this point (update the binary to 2.0.1 and add the necessary bits to the GUI to support the additional features).

    The Suricata guys are also working on Netmap support.  They have not published a release date or version yet, but it is showing as 50% done on their work schedule.  This will allow high speed IPS operation assuming the pfSense guys will include the required kernel module in their builds.

    As for the IPv6 trick, thanks for the tip and recommendation.  I will check it out.

    Bill


  • Moderator

    The Suricata guys are also working on Netmap support.  They have not published a release date or version yet, but it is showing as 50% done on their work schedule.  This will allow high speed IPS operation assuming the pfSense guys will include the required kernel module in their builds.

    Do the pfSense Devs support moving to netmap also? I assume this will also work for the Snort package? I wonder what they expect the max throughput to be?



  • Another bug: IPv6 addresses do not have an unblock button on the alerts page, although they are correctly added to the blocked table (snort2c) and the blocked page.



  • @BBcan17:

    The Suricata guys are also working on Netmap support.  They have not published a release date or version yet, but it is showing as 50% done on their work schedule.  This will allow high speed IPS operation assuming the pfSense guys will include the required kernel module in their builds.

    Do the pfSense Devs support moving to netmap also? I assume this will also work for the Snort package? I wonder what they expect the max throughput to be?

    I believe they do.  Haven't heard any estimates of throughput, but it should be pretty good.

    Bill



  • @jflsakfja:

    Another bug: IPv6 addresses do not have an unblock button on the alerts page, although they are correctly added to the blocked table (snort2c) and the blocked page.

    Now that I have my own IPv6 setup working with the Hurricane Electric tunnel broker, I can do a bit more testing.  The way Suricata (and Snort) get the list of currently blocked IPs is by querying pf using pfctl.  Curious they showed up in the <snort2c>table and still did not have an unblock icon.

    Bill</snort2c>



  • Great news Bill. Welcome to IPv6, maybe its me but its seems to be a lonely place right now…

    1: Dk if this is a bug but I turned on pfSense notifications and I'm receiving these emails:

    
    X-Cron-Env: <shell= bin="" sh="">X-Cron-Env: <home= root="">X-Cron-Env: <path= usr="" bin:="" bin="">X-Cron-Env: <logname=root>X-Cron-Env: <user=root>Warning: filesize(): stat failed for /var/log/suricata/suricata_em339811/stats.log in /usr/local/pkg/suricata/suricata_check_cron_misc.inc on line 129
    
    Warning: filesize(): stat failed for /var/log/suricata/suricata_em231600/stats.log in /usr/local/pkg/suricata/suricata_check_cron_misc.inc on line 129</user=root></logname=root></path=></home=></shell=> 
    

    I didn't have statistics enabled which I believe is why this popped up. I've enabled it, and the emails seem to have gone away.

    2: In Log mgt, I've changed Alerts to have NO LIMIT, upon saving; it goes back to 500KB

    3: Also I noticed 75% of the time, my second sensor wont start on its on. Either a re-boot, package re-start because of dhcp or wan flap. I'll see if I can capture a log of when this happens if that will help. I can't reproduce if I manually stop/start the service



  • I'm having a problem with Suricata running on my LAN interface. I have 2 instances running, WAN & LAN and WAN seems fine but anytime the rules update the LAN fails to restart. Manually starting it seems to work fine.

    Suricata.log for the interface

    30/5/2014 -- 02:31:30 - <info> -- Signal Received.  Stopping engine.
    30/5/2014 -- 02:31:30 - <info> -- 0 new flows, 0 established flows were timed out, 0 flows in closed state
    30/5/2014 -- 02:31:30 - <info> -- time elapsed 66110.758s
    30/5/2014 -- 02:31:31 - <info> -- (RxPcapem01) Packets 4399339, bytes 2794681616
    30/5/2014 -- 02:31:31 - <info> -- (RxPcapem01) Pcap Total:4399388 Recv:4399339 Drop:49 (0.0%).
    30/5/2014 -- 02:31:31 - <info> -- AutoFP - Total flow handler queues - 6
    30/5/2014 -- 02:31:31 - <info> -- AutoFP - Queue 0  - pkts: 1315978      flows: 131283      
    30/5/2014 -- 02:31:31 - <info> -- AutoFP - Queue 1  - pkts: 1361933      flows: 94964       
    30/5/2014 -- 02:31:31 - <info> -- AutoFP - Queue 2  - pkts: 809788       flows: 45693       
    30/5/2014 -- 02:31:31 - <info> -- AutoFP - Queue 3  - pkts: 358030       flows: 15976       
    30/5/2014 -- 02:31:31 - <info> -- AutoFP - Queue 4  - pkts: 297870       flows: 7314        
    30/5/2014 -- 02:31:31 - <info> -- AutoFP - Queue 5  - pkts: 270079       flows: 4121        
    30/5/2014 -- 02:31:31 - <info> -- Stream TCP processed 707300 TCP packets
    30/5/2014 -- 02:31:31 - <info> -- alert-pf output inserted 3 IP address blocks
    30/5/2014 -- 02:31:31 - <info> -- alert-pf output wrote 3 alerts
    30/5/2014 -- 02:31:31 - <info> -- Fast log output wrote 7 alerts
    30/5/2014 -- 02:31:31 - <info> -- HTTP logger logged 2706 requests
    30/5/2014 -- 02:31:31 - <info> -- Stream TCP processed 954694 TCP packets
    30/5/2014 -- 02:31:31 - <info> -- alert-pf output inserted 3 IP address blocks
    30/5/2014 -- 02:31:31 - <info> -- alert-pf output wrote 3 alerts
    30/5/2014 -- 02:31:31 - <info> -- Fast log output wrote 7 alerts
    30/5/2014 -- 02:31:31 - <info> -- HTTP logger logged 2200 requests
    30/5/2014 -- 02:31:31 - <info> -- Stream TCP processed 479062 TCP packets
    30/5/2014 -- 02:31:31 - <info> -- alert-pf output inserted 3 IP address blocks
    30/5/2014 -- 02:31:31 - <info> -- alert-pf output wrote 3 alerts
    30/5/2014 -- 02:31:31 - <info> -- Fast log output wrote 7 alerts
    30/5/2014 -- 02:31:31 - <info> -- HTTP logger logged 1843 requests
    30/5/2014 -- 02:31:31 - <info> -- Stream TCP processed 95982 TCP packets
    30/5/2014 -- 02:31:31 - <info> -- alert-pf output inserted 3 IP address blocks
    30/5/2014 -- 02:31:31 - <info> -- alert-pf output wrote 3 alerts
    30/5/2014 -- 02:31:31 - <info> -- Fast log output wrote 7 alerts
    30/5/2014 -- 02:31:31 - <info> -- HTTP logger logged 1389 requests
    30/5/2014 -- 02:31:31 - <info> -- Stream TCP processed 52257 TCP packets
    30/5/2014 -- 02:31:31 - <info> -- alert-pf output inserted 3 IP address blocks
    30/5/2014 -- 02:31:31 - <info> -- alert-pf output wrote 3 alerts
    30/5/2014 -- 02:31:31 - <info> -- Fast log output wrote 7 alerts
    30/5/2014 -- 02:31:31 - <info> -- HTTP logger logged 1090 requests
    30/5/2014 -- 02:31:31 - <info> -- Stream TCP processed 30060 TCP packets
    30/5/2014 -- 02:31:31 - <info> -- alert-pf output inserted 3 IP address blocks
    30/5/2014 -- 02:31:31 - <info> -- alert-pf output wrote 3 alerts
    30/5/2014 -- 02:31:31 - <info> -- Fast log output wrote 7 alerts
    30/5/2014 -- 02:31:31 - <info> -- HTTP logger logged 883 requests
    30/5/2014 -- 02:31:31 - <info> -- host memory usage: 194304 bytes, maximum: 16777216
    30/5/2014 -- 02:31:31 - <info> -- cleaning up signature grouping structure... complete
    30/5/2014 -- 02:31:32 - <error> -- [ERRCODE: UNKNOWN_ERROR(87)] - Child died unexpectedly</error></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info>
    


  • On 2.2 alpha I am getting an email notifications every 4 minutes about a cronjob error.  I have 'Auto Log Management' enabled.

    2.2-ALPHA (i386)
    built on Thu May 29 06:53:30 CDT 2014
    FreeBSD 10.0-STABLE

    Subject: Cron root@pfsense/usr/bin/nice -n20 /usr/local/bin/php -f /usr/local/pkg/suricata/suricata_check_cron_misc.inc

    X-Cron-Env: <shell= bin="" sh="">
    X-Cron-Env: <path= etc:="" bin:="" sbin:="" usr="" sbin="">
    X-Cron-Env: <home= var="" log="">
    X-Cron-Env: <logname=root>
    X-Cron-Env: <user=root>
    
    Warning: filesize(): stat failed for /var/log/suricata/suricata_rl034283/files-json.log in /usr/local/pkg/suricata/suricata_check_cron_misc.inc on line 129
    
    Warning: filesize(): stat failed for /var/log/suricata/suricata_rl034283/tls.log in /usr/local/pkg/suricata/suricata_check_cron_misc.inc on line 129</user=root></logname=root></home=></path=></shell=>
    ```</root@pfsense>


  • i'm betting your dont have have Enable 'Tracked-Files Log' and 'Enable TLS Log' turned on



  • And you would be correct.  I didn't want to log those.  I went ahead and enabled them just to get rid of the constant emails :).  Thanks for the workaround.



  • @Cino:

    Great news Bill. Welcome to IPv6, maybe its me but its seems to be a lonely place right now…

    1: Dk if this is a bug but I turned on pfSense notifications and I'm receiving these emails:

    
    X-Cron-Env: <shell= bin="" sh="">X-Cron-Env: <home= root="">X-Cron-Env: <path= usr="" bin:="" bin="">X-Cron-Env: <logname=root>X-Cron-Env: <user=root>Warning: filesize(): stat failed for /var/log/suricata/suricata_em339811/stats.log in /usr/local/pkg/suricata/suricata_check_cron_misc.inc on line 129
    
    Warning: filesize(): stat failed for /var/log/suricata/suricata_em231600/stats.log in /usr/local/pkg/suricata/suricata_check_cron_misc.inc on line 129</user=root></logname=root></path=></home=></shell=> 
    

    I didn't have statistics enabled which I believe is why this popped up. I've enabled it, and the emails seem to have gone away.

    2: In Log mgt, I've changed Alerts to have NO LIMIT, upon saving; it goes back to 500KB

    3: Also I noticed 75% of the time, my second sensor wont start on its on. Either a re-boot, package re-start because of dhcp or wan flap. I'll see if I can capture a log of when this happens if that will help. I can't reproduce if I manually stop/start the service

    I can take care of that filesize() stat message.  Just need to check that the file exists before looking at its size.  I'll get that handled in the next update.

    I will also look at the NO LIMIT no saving.  I was working back and forth a month or so back trying to get both Suricata and Snort updates out concurrently, and I missed some things in my Suricata testing.

    Bill



  • @adam65535:

    And you would be correct.  I didn't want to log those.  I went ahead and enabled them just to get rid of the constant emails :).  Thanks for the workaround.

    I will fix this in the next update.  The cron script is checking size prior to verifying the file exists.  I can post a quick fix that affected folks can manually apply if they want to edit a file.  I will post up something in a bit.

    UPDATE EDIT:
    Here is the fix.  Edit the file /usr/local/pkg/suricata/suricata_check_cron_misc.inc. I've included context around the change to help in locating the proper section.  The maroon-colored text (two lines) is what needs to be added.

    // Check the current log to see if it needs rotating.
    // If it does, rotate it and put the current time
    // on the end of the filename as UNIX timestamp.
    if (!file_exists($log_file))
    return;
    if (($log_limit > 0) && (filesize($log_file) >= $log_limit)) {
    $newfile = $log_file . "." . strval(time());
    try {
    copy($log_file, $newfile);
    file_put_contents($log_file, "");
    } catch (Exception $e) {
    log_error("[Suricata] Failed to rotate file '{$log_file}' – error was {$e->getMessage()}");
    }
    }

    Sorry,
    Bill



  • @DigitalDeviant:

    I'm having a problem with Suricata running on my LAN interface. I have 2 instances running, WAN & LAN and WAN seems fine but anytime the rules update the LAN fails to restart. Manually starting it seems to work fine.

    Suricata.log for the interface

    30/5/2014 -- 02:31:30 - <info> -- Signal Received.  Stopping engine.
    30/5/2014 -- 02:31:30 - <info> -- 0 new flows, 0 established flows were timed out, 0 flows in closed state
    30/5/2014 -- 02:31:30 - <info> -- time elapsed 66110.758s
    30/5/2014 -- 02:31:31 - <info> -- (RxPcapem01) Packets 4399339, bytes 2794681616
    30/5/2014 -- 02:31:31 - <info> -- (RxPcapem01) Pcap Total:4399388 Recv:4399339 Drop:49 (0.0%).
    30/5/2014 -- 02:31:31 - <info> -- AutoFP - Total flow handler queues - 6
    30/5/2014 -- 02:31:31 - <info> -- AutoFP - Queue 0  - pkts: 1315978      flows: 131283      
    30/5/2014 -- 02:31:31 - <info> -- AutoFP - Queue 1  - pkts: 1361933      flows: 94964       
    30/5/2014 -- 02:31:31 - <info> -- AutoFP - Queue 2  - pkts: 809788       flows: 45693       
    30/5/2014 -- 02:31:31 - <info> -- AutoFP - Queue 3  - pkts: 358030       flows: 15976       
    30/5/2014 -- 02:31:31 - <info> -- AutoFP - Queue 4  - pkts: 297870       flows: 7314        
    30/5/2014 -- 02:31:31 - <info> -- AutoFP - Queue 5  - pkts: 270079       flows: 4121        
    30/5/2014 -- 02:31:31 - <info> -- Stream TCP processed 707300 TCP packets
    30/5/2014 -- 02:31:31 - <info> -- alert-pf output inserted 3 IP address blocks
    30/5/2014 -- 02:31:31 - <info> -- alert-pf output wrote 3 alerts
    30/5/2014 -- 02:31:31 - <info> -- Fast log output wrote 7 alerts
    30/5/2014 -- 02:31:31 - <info> -- HTTP logger logged 2706 requests
    30/5/2014 -- 02:31:31 - <info> -- Stream TCP processed 954694 TCP packets
    30/5/2014 -- 02:31:31 - <info> -- alert-pf output inserted 3 IP address blocks
    30/5/2014 -- 02:31:31 - <info> -- alert-pf output wrote 3 alerts
    30/5/2014 -- 02:31:31 - <info> -- Fast log output wrote 7 alerts
    30/5/2014 -- 02:31:31 - <info> -- HTTP logger logged 2200 requests
    30/5/2014 -- 02:31:31 - <info> -- Stream TCP processed 479062 TCP packets
    30/5/2014 -- 02:31:31 - <info> -- alert-pf output inserted 3 IP address blocks
    30/5/2014 -- 02:31:31 - <info> -- alert-pf output wrote 3 alerts
    30/5/2014 -- 02:31:31 - <info> -- Fast log output wrote 7 alerts
    30/5/2014 -- 02:31:31 - <info> -- HTTP logger logged 1843 requests
    30/5/2014 -- 02:31:31 - <info> -- Stream TCP processed 95982 TCP packets
    30/5/2014 -- 02:31:31 - <info> -- alert-pf output inserted 3 IP address blocks
    30/5/2014 -- 02:31:31 - <info> -- alert-pf output wrote 3 alerts
    30/5/2014 -- 02:31:31 - <info> -- Fast log output wrote 7 alerts
    30/5/2014 -- 02:31:31 - <info> -- HTTP logger logged 1389 requests
    30/5/2014 -- 02:31:31 - <info> -- Stream TCP processed 52257 TCP packets
    30/5/2014 -- 02:31:31 - <info> -- alert-pf output inserted 3 IP address blocks
    30/5/2014 -- 02:31:31 - <info> -- alert-pf output wrote 3 alerts
    30/5/2014 -- 02:31:31 - <info> -- Fast log output wrote 7 alerts
    30/5/2014 -- 02:31:31 - <info> -- HTTP logger logged 1090 requests
    30/5/2014 -- 02:31:31 - <info> -- Stream TCP processed 30060 TCP packets
    30/5/2014 -- 02:31:31 - <info> -- alert-pf output inserted 3 IP address blocks
    30/5/2014 -- 02:31:31 - <info> -- alert-pf output wrote 3 alerts
    30/5/2014 -- 02:31:31 - <info> -- Fast log output wrote 7 alerts
    30/5/2014 -- 02:31:31 - <info> -- HTTP logger logged 883 requests
    30/5/2014 -- 02:31:31 - <info> -- host memory usage: 194304 bytes, maximum: 16777216
    30/5/2014 -- 02:31:31 - <info> -- cleaning up signature grouping structure... complete
    30/5/2014 -- 02:31:32 - <error> -- [ERRCODE: UNKNOWN_ERROR(87)] - Child died unexpectedly</error></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info>
    

    It might be that you have two identical processes running.  What is the output of this command?

    ps -ax |grep suricata
    

    Bill



  • @bmeeks:

    It might be that you have two identical processes running.  What is the output of this command?

    ps -ax |grep suricata
    

    Bill

    $ ps -ax |grep suricata
    33297  ??  IN    0:00.00 /bin/sh /usr/local/etc/rc.d/suricata.sh start
    34081  ??  SN    0:01.27 /usr/pbi/suricata-amd64/bin/suricata -i em1 -D -c /us
    34249  ??  SNs  18:56.87 /usr/pbi/suricata-amd64/bin/suricata -i em1 -D -c /us
    79307  ??  S      0:00.00 sh -c ps -ax |grep suricata 2>&1
    79773  ??  S      0:00.00 grep suricata
    97563  ??  Ss    14:09.82 /usr/pbi/suricata-amd64/bin/suricata -i em0 -D -c /us


  • Moderator

    @DigitalDeviant:

    @bmeeks:

    It might be that you have two identical processes running.  What is the output of this command?

    ps -ax |grep suricata
    

    Bill

    $ ps -ax |grep suricata
    33297  ??  IN    0:00.00 /bin/sh /usr/local/etc/rc.d/suricata.sh start
    34081  ??  SN    0:01.27 /usr/pbi/suricata-amd64/bin/suricata -i em1 -D -c /us
    34249  ??  SNs  18:56.87 /usr/pbi/suricata-amd64/bin/suricata -i em1 -D -c /us

    79307  ??  S      0:00.00 sh -c ps -ax |grep suricata 2>&1
    79773  ??  S      0:00.00 grep suricata
    97563  ??  Ss    14:09.82 /usr/pbi/suricata-amd64/bin/suricata -i em0 -D -c /us

    You can see that for Interface "em1" that there are two Processes Running (PIDS 34081 and 34249).

    If you run this command from the shell, it will kill all Running Suricata sessions. After that, goto services and re-Start the Suricata Service.

    pkill suricata

    You can try to kill just the one "em1" pid, but Its cleaner to kill them all and restart cleanly.

    EDIT:
    To do individually (example)

    kill 34249



  • I was looking at the log directory and noticed a bunch of stats.log.xxxxxxxxx files (where xxxx is a big number).  What is interesting is that the file size grows for every new stat file.  That doesn't seem right to me.  Is that expected behavior?

    -rw-r--r--  1 root  wheel   5545829 May 30 09:43 stats.log.1401457203
    -rw-r--r--  1 root  wheel   5650619 May 30 09:48 stats.log.1401457503
    -rw-r--r--  1 root  wheel   5755409 May 30 09:53 stats.log.1401457803
    -rw-r--r--  1 root  wheel   5860199 May 30 09:58 stats.log.1401458103
    -rw-r--r--  1 root  wheel   5964989 May 30 10:03 stats.log.1401458403
    -rw-r--r--  1 root  wheel   6069779 May 30 10:08 stats.log.1401458703
    -rw-r--r--  1 root  wheel   6174569 May 30 10:13 stats.log.1401459003
    -rw-r--r--  1 root  wheel   6279359 May 30 10:18 stats.log.1401459303
    -rw-r--r--  1 root  wheel   6384149 May 30 10:23 stats.log.1401459603
    -rw-r--r--  1 root  wheel   6488939 May 30 10:28 stats.log.1401459903
    -rw-r--r--  1 root  wheel   6593729 May 30 10:33 stats.log.1401460203
    -rw-r--r--  1 root  wheel   6698519 May 30 10:38 stats.log.1401460503
    -rw-r--r--  1 root  wheel   6803309 May 30 10:44 stats.log.1401460803
    -rw-r--r--  1 root  wheel   6908099 May 30 10:48 stats.log.1401461103
    -rw-r--r--  1 root  wheel   7012889 May 30 10:54 stats.log.1401461403
    -rw-r--r--  1 root  wheel   7117679 May 30 10:59 stats.log.1401461703
    -rw-r--r--  1 root  wheel   7222469 May 30 11:04 stats.log.1401462003
    -rw-r--r--  1 root  wheel   7327259 May 30 11:09 stats.log.1401462303
    -rw-r--r--  1 root  wheel   7432049 May 30 11:14 stats.log.1401462603
    -rw-r--r--  1 root  wheel   7536839 May 30 11:19 stats.log.1401462903
    -rw-r--r--  1 root  wheel   7641629 May 30 11:24 stats.log.1401463203
    


  • @adam65535:

    I was looking at the log directory and noticed a bunch of stats.log.xxxxxxxxx files (where xxxx is a big number).  What is interesting is that the file size grows for every new stat file.  That doesn't seem right to me.  Is that expected behavior?

    -rw-r--r--  1 root  wheel   5545829 May 30 09:43 stats.log.1401457203
    -rw-r--r--  1 root  wheel   5650619 May 30 09:48 stats.log.1401457503
    -rw-r--r--  1 root  wheel   5755409 May 30 09:53 stats.log.1401457803
    -rw-r--r--  1 root  wheel   5860199 May 30 09:58 stats.log.1401458103
    -rw-r--r--  1 root  wheel   5964989 May 30 10:03 stats.log.1401458403
    -rw-r--r--  1 root  wheel   6069779 May 30 10:08 stats.log.1401458703
    -rw-r--r--  1 root  wheel   6174569 May 30 10:13 stats.log.1401459003
    -rw-r--r--  1 root  wheel   6279359 May 30 10:18 stats.log.1401459303
    -rw-r--r--  1 root  wheel   6384149 May 30 10:23 stats.log.1401459603
    -rw-r--r--  1 root  wheel   6488939 May 30 10:28 stats.log.1401459903
    -rw-r--r--  1 root  wheel   6593729 May 30 10:33 stats.log.1401460203
    -rw-r--r--  1 root  wheel   6698519 May 30 10:38 stats.log.1401460503
    -rw-r--r--  1 root  wheel   6803309 May 30 10:44 stats.log.1401460803
    -rw-r--r--  1 root  wheel   6908099 May 30 10:48 stats.log.1401461103
    -rw-r--r--  1 root  wheel   7012889 May 30 10:54 stats.log.1401461403
    -rw-r--r--  1 root  wheel   7117679 May 30 10:59 stats.log.1401461703
    -rw-r--r--  1 root  wheel   7222469 May 30 11:04 stats.log.1401462003
    -rw-r--r--  1 root  wheel   7327259 May 30 11:09 stats.log.1401462303
    -rw-r--r--  1 root  wheel   7432049 May 30 11:14 stats.log.1401462603
    -rw-r--r--  1 root  wheel   7536839 May 30 11:19 stats.log.1401462903
    -rw-r--r--  1 root  wheel   7641629 May 30 11:24 stats.log.1401463203
    

    The package is manually rotating the file every 5 minutes.  This is not optimum, but since the Suricata binary only offers log rotation for the unified2 file used by Barnyard2, that's the best I could come up with.  What happens is the file can grow beyond the "rotate threshold size" in between checks.  A cron job runs every 5 minutes and checks the file size against the set value.  If the file is equal to or larger than the set threshold, it should get rotated.  The number at the end is the UNIX timestamp of when the file was rotated.  The size it grows to in between checks is a function of the stats update interval and the amount of traffic on your network.

    I'm not sure why in your case the file seems to grow linearly.  Take a look at say the last three or four rotated files and see if you can spot something to give a clue.  I let this run during development in a VM and the rotated files tended to vary in size, but all were generally larger than the set threshold due to the 5 minute interval for the cron job.

    Bill



  • I have Suricata running on several interfaces and I noticed that after some time, the process stops on random interfaces. The only thing I see in the logs are:

    
    4/6/2014 -- 00:30:38 - <info>-- Signal Received.  Stopping engine.
    4/6/2014 -- 00:30:38 - <info>-- 0 new flows, 0 established flows were timed out, 0 flows in closed state
    4/6/2014 -- 00:30:39 - <info>-- time elapsed 8834.827s
    4/6/2014 -- 00:30:39 - <info>-- (RxPcapem11) Packets 706687, bytes 459350916
    4/6/2014 -- 00:30:39 - <info>-- (RxPcapem11) Pcap Total:706699 Recv:706699 Drop:0 (0.0%).
    4/6/2014 -- 00:30:39 - <info>-- AutoFP - Total flow handler queues - 6
    4/6/2014 -- 00:30:39 - <info>-- AutoFP - Queue 0  - pkts: 133565       flows: 2838        
    4/6/2014 -- 00:30:39 - <info>-- AutoFP - Queue 1  - pkts: 143185       flows: 1865        
    4/6/2014 -- 00:30:39 - <info>-- AutoFP - Queue 2  - pkts: 107166       flows: 1203        
    4/6/2014 -- 00:30:39 - <info>-- AutoFP - Queue 3  - pkts: 110217       flows: 556         
    4/6/2014 -- 00:30:39 - <info>-- AutoFP - Queue 4  - pkts: 105277       flows: 423         
    4/6/2014 -- 00:30:39 - <info>-- AutoFP - Queue 5  - pkts: 108398       flows: 380         
    4/6/2014 -- 00:30:39 - <info>-- Stream TCP processed 33241 TCP packets
    4/6/2014 -- 00:30:39 - <info>-- Fast log output wrote 12 alerts
    4/6/2014 -- 00:30:39 - <info>-- HTTP logger logged 595 requests
    4/6/2014 -- 00:30:39 - <info>-- (Detect1) Alerts 12
    4/6/2014 -- 00:30:39 - <info>-- Stream TCP processed 46221 TCP packets
    4/6/2014 -- 00:30:39 - <info>-- Fast log output wrote 12 alerts
    4/6/2014 -- 00:30:39 - <info>-- HTTP logger logged 401 requests
    4/6/2014 -- 00:30:39 - <info>-- (Detect2) Alerts 12
    4/6/2014 -- 00:30:39 - <info>-- Stream TCP processed 8799 TCP packets
    4/6/2014 -- 00:30:39 - <info>-- Fast log output wrote 12 alerts
    4/6/2014 -- 00:30:39 - <info>-- HTTP logger logged 314 requests
    4/6/2014 -- 00:30:39 - <info>-- (Detect3) Alerts 12
    4/6/2014 -- 00:30:39 - <info>-- Stream TCP processed 15513 TCP packets
    4/6/2014 -- 00:30:39 - <info>-- Fast log output wrote 12 alerts
    4/6/2014 -- 00:30:39 - <info>-- HTTP logger logged 164 requests
    4/6/2014 -- 00:30:39 - <info>-- (Detect4) Alerts 12
    4/6/2014 -- 00:30:39 - <info>-- Stream TCP processed 10770 TCP packets
    4/6/2014 -- 00:30:39 - <info>-- Fast log output wrote 12 alerts
    4/6/2014 -- 00:30:39 - <info>-- HTTP logger logged 116 requests
    4/6/2014 -- 00:30:39 - <info>-- (Detect5) Alerts 12
    4/6/2014 -- 00:30:39 - <info>-- Stream TCP processed 13990 TCP packets
    4/6/2014 -- 00:30:39 - <info>-- Fast log output wrote 12 alerts
    4/6/2014 -- 00:30:39 - <info>-- HTTP logger logged 120 requests
    4/6/2014 -- 00:30:39 - <info>-- (Detect6) Alerts 12
    4/6/2014 -- 00:30:39 - <info>-- host memory usage: 194304 bytes, maximum: 16777216
    4/6/2014 -- 00:30:39 - <info>-- cleaning up signature grouping structure... complete</info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info> 
    


  • @GoldServe:

    I have Suricata running on several interfaces and I noticed that after some time, the process stops on random interfaces. The only thing I see in the logs are:

    
    4/6/2014 -- 00:30:38 - <info>-- Signal Received.  Stopping engine.
    4/6/2014 -- 00:30:38 - <info>-- 0 new flows, 0 established flows were timed out, 0 flows in closed state
    4/6/2014 -- 00:30:39 - <info>-- time elapsed 8834.827s
    4/6/2014 -- 00:30:39 - <info>-- (RxPcapem11) Packets 706687, bytes 459350916
    4/6/2014 -- 00:30:39 - <info>-- (RxPcapem11) Pcap Total:706699 Recv:706699 Drop:0 (0.0%).
    4/6/2014 -- 00:30:39 - <info>-- AutoFP - Total flow handler queues - 6
    4/6/2014 -- 00:30:39 - <info>-- AutoFP - Queue 0  - pkts: 133565       flows: 2838        
    4/6/2014 -- 00:30:39 - <info>-- AutoFP - Queue 1  - pkts: 143185       flows: 1865        
    4/6/2014 -- 00:30:39 - <info>-- AutoFP - Queue 2  - pkts: 107166       flows: 1203        
    4/6/2014 -- 00:30:39 - <info>-- AutoFP - Queue 3  - pkts: 110217       flows: 556         
    4/6/2014 -- 00:30:39 - <info>-- AutoFP - Queue 4  - pkts: 105277       flows: 423         
    4/6/2014 -- 00:30:39 - <info>-- AutoFP - Queue 5  - pkts: 108398       flows: 380         
    4/6/2014 -- 00:30:39 - <info>-- Stream TCP processed 33241 TCP packets
    4/6/2014 -- 00:30:39 - <info>-- Fast log output wrote 12 alerts
    4/6/2014 -- 00:30:39 - <info>-- HTTP logger logged 595 requests
    4/6/2014 -- 00:30:39 - <info>-- (Detect1) Alerts 12
    4/6/2014 -- 00:30:39 - <info>-- Stream TCP processed 46221 TCP packets
    4/6/2014 -- 00:30:39 - <info>-- Fast log output wrote 12 alerts
    4/6/2014 -- 00:30:39 - <info>-- HTTP logger logged 401 requests
    4/6/2014 -- 00:30:39 - <info>-- (Detect2) Alerts 12
    4/6/2014 -- 00:30:39 - <info>-- Stream TCP processed 8799 TCP packets
    4/6/2014 -- 00:30:39 - <info>-- Fast log output wrote 12 alerts
    4/6/2014 -- 00:30:39 - <info>-- HTTP logger logged 314 requests
    4/6/2014 -- 00:30:39 - <info>-- (Detect3) Alerts 12
    4/6/2014 -- 00:30:39 - <info>-- Stream TCP processed 15513 TCP packets
    4/6/2014 -- 00:30:39 - <info>-- Fast log output wrote 12 alerts
    4/6/2014 -- 00:30:39 - <info>-- HTTP logger logged 164 requests
    4/6/2014 -- 00:30:39 - <info>-- (Detect4) Alerts 12
    4/6/2014 -- 00:30:39 - <info>-- Stream TCP processed 10770 TCP packets
    4/6/2014 -- 00:30:39 - <info>-- Fast log output wrote 12 alerts
    4/6/2014 -- 00:30:39 - <info>-- HTTP logger logged 116 requests
    4/6/2014 -- 00:30:39 - <info>-- (Detect5) Alerts 12
    4/6/2014 -- 00:30:39 - <info>-- Stream TCP processed 13990 TCP packets
    4/6/2014 -- 00:30:39 - <info>-- Fast log output wrote 12 alerts
    4/6/2014 -- 00:30:39 - <info>-- HTTP logger logged 120 requests
    4/6/2014 -- 00:30:39 - <info>-- (Detect6) Alerts 12
    4/6/2014 -- 00:30:39 - <info>-- host memory usage: 194304 bytes, maximum: 16777216
    4/6/2014 -- 00:30:39 - <info>-- cleaning up signature grouping structure... complete</info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info></info> 
    

    According to the very first line in the log output you provided, something told it to shutdown.  Here is the line I'm talking about:

    4/6/2014 -- 00:30:38 - <info> -- Signal Received.  Stopping engine.</info>
    

    If Suricata had died or crashed, you would not see that line ("signal received" means a SIGTERM was sent to the process).  So look for a cron job or something in your configuration that is stopping Suricata (or maybe stopping many or all processes).  For the daily rule update check, do you have it set to "warm restart" or "cold restart".  This is an option on the GENERAL SETTINGS tab.

    Bill



  • Bug:

    Blocked tab:
    Warning: inet_pton(): Unrecognized address TCP in /usr/local/www/suricata/suricata_blocked.php on line 247 Warning: inet_pton(): Unrecognized address TCP in /usr/local/www/suricata/suricata_blocked.php on line 247

    Rule that caused it:
    files.rules:
    1:22 FILE pdf claimed, but not pdf <<< fires up when spiders (eg googlebot) try to download a part of a pdf, therefore DELETE UPSTREAM (part after <<< is in the upcoming suricata topic)

    It also breaks the alerts tab, with text all over the place.


Log in to reply