Suricata repeatedfails to start on lan interface because of recurring stale file



  • Suricata repeatedly stops working on my lan interface, the log indicates that there is a problem with a stale file.

    19/9/2017 – 10:43:24 - <error>-- [ERRCODE: SC_ERR_INITIALIZATION(45)] - pid file '/var/run/suricata_re152829.pid' exists but appears stale. Make sure Suricata is not running and then remove /var/run/suricata_re152829.pid. Aborting!</error>

    Yes, removing the file solves the problem and suricata starts again on the interface. However after some time the problem appears again.



  • Does this happen after the restart from the daily rules update process?  What other messages are in the firewall's system log around the same time?  My first suspicion is two instances of Suricata trying to run and/or start on the same interface.  That can happen if the firewall issues a "restart all packages" command during the same time Suricata is self-restarting after a rules update.

    Also, are you using a RAMDISK?  If so, check and make sure it is not running out of space.

    Bill



  • The stale log file may just be a symptom that it didn't shut down properly, or didn't fully start up on a previous attempt.

    My experience from today - Suricata 3.x was running fine, but then I updated to Suricata to 4.0.0_1, and then it failed to restart, even after removing the /var/run/suricata*.pid files, it would immediately fail again, so I checked the log:

    (Suricata –-> Logs View -- suricata.log)

    The log showed the well known STREAMS memory allocation error:

    19/9/2017 -- 08:56:07 - <error>-- [ERRCODE: SC_ERR_POOL_INIT(66)] - alloc error
    19/9/2017 – 08:56:07 - <error>-- [ERRCODE: SC_ERR_POOL_INIT(66)] - pool grow failed

    So, I doubled stream memory cap and halved preallocated sessions as a starting point (note - this may not be an optimal configuration)

    Stream Memory Cap 134217728
    Preallocated Sessions 16384

    The result was a successful startup (PFSense XG-1540):
    Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz
    Current: 2100 MHz, Max: 2101 MHz
    16 CPUs: 1 package(s) x 8 core(s) x 2 SMT threads

    19/9/2017 – 09:03:26 - <notice>-- all 17 packet processing threads, 2 management threads initialized, engine started.

    So...after removing the .pid files, check the logs to see if there may be an underlying issue if it fails again, and there might have been something that changed in the 4.0 update</notice></error></error>



  • I don't recall off the top of my head, but there is a forumula that uses the number of cores and memory to compute optimal stream memory settings.  I think some adjustments to that area of the binary also happened between the 3.2.x and 4.0.0 versions of the code, so that might account for the fact of not having that exact error using 3.2.x.

    Bill



  • @bmeeks:

    I don't recall off the top of my head, but there is a forumula that uses the number of cores and memory to compute optimal stream memory settings.  I think some adjustments to that area of the binary also happened between the 3.2.x and 4.0.0 versions of the code, so that might account for the fact of not having that exact error using 3.2.x.

    Bill

    Seems like I am now (after fixing other issues) having the same issue as op.

    my supermicro a1sri-2758f has 8 cores and no threads, its non-standard core count I guess. Might explain the buggy multi threading  issues I am experiencing.



  • @strangegopher:

    @bmeeks:

    I don't recall off the top of my head, but there is a forumula that uses the number of cores and memory to compute optimal stream memory settings.  I think some adjustments to that area of the binary also happened between the 3.2.x and 4.0.0 versions of the code, so that might account for the fact of not having that exact error using 3.2.x.

    Bill

    Seems like I am now (after fixing other issues) having the same issue as op.

    my supermicro a1sri-2758f has 8 cores and no threads, its non-standard core count I guess. Might explain the buggy multi threading  issues I am experiencing.

    See my reply in the other thread you linked.  You need to at least double the Stream Mem Cap setting, and possible increase it even more.

    Bill


Log in to reply