Suricata ~ Updates Killing Network Connections



  • Since updating to the last official 2.4.2 P1 release every single day at 12:30 AM and 6:30 PM my entire network shuts down. Upon further audit and review I  found when ever the Suricata program is updating its signatures the system will be locked up doing something which literally kills all network connections in my home.  😠

    Since I update based on six hour intervals starting at 12:30 AM each day it was easy to track. As seen in this image capture when Suricata is updating its data base the load increases several watts from its base 8 watts RMS at 12:30 AM & 6:30 PM.

    I have since changed the setting to 28 days until the next scheduled update until such time root cause has been identified and resolved by the development team.

    Moving the update period has completely solved this network down issue for me . . .

    Thank You!

    ![Suricata Energy Draw.PNG](/public/imported_attachments/1/Suricata Energy Draw.PNG)
    ![Suricata Energy Draw.PNG_thumb](/public/imported_attachments/1/Suricata Energy Draw.PNG_thumb)
    ![Suricata Update Interval.PNG](/public/imported_attachments/1/Suricata Update Interval.PNG)
    ![Suricata Update Interval.PNG_thumb](/public/imported_attachments/1/Suricata Update Interval.PNG_thumb)



  • Hi.

    Is 0:30 the best update hour? mine is on 4 or 5 AM.

    Have you enable or disabled “Live Rule Swap on Update”? on Global settings? what are your interface “IPS Mode” (Goto and edit interface, section “Alert and Block Settings”)?



  • @mais_um:

    Hi.

    Is 0:30 the best update hour? mine is on 4 or 5 AM.

    Have you enable or disabled “Live Rule Swap on Update”? on Global settings? what are your interface “IPS Mode” (Goto and edit interface, section “Alert and Block Settings”)?

    The time could really be what ever I simply left it at the default value. The problem is not just because the network goes out but when it happens I have several Alpha / Beta pieces of hardware that have a really hard time coming back on line once this update happens. I literally have to hard reboot these two devices by removing power remotely via my web hosted switch.

    This was NEVER an issue prior to this release  . . .

    I did note the option of Live Rule Swap on Update but again I never used it in the past. I see no reason to use this as a solution given the previous release operated just fine. If no one is going to take this issue seriously and address it obviously I will have to use that option moving forward.

    what are your interface “IPS Mode” (Goto and edit interface, section “Alert and Block Settings”)?

    Can you provide a image capture as to where this is I can’t find it.

    Thank You!



  • Hi,

    Suricata, when updating uses some Watts … never saw stats that showed  increased system usage expressed by Watts before.
    What about process usage ? RAM usage ? Do they follow the Watt usage ?

    Although firewalls like this https://store.netgate.com/SG-1000.aspx shouldn’t maybe not using heavy packages as Suricate (I might be wrong here), I wonder what you use for hardware.



  • @Gertran is on the right track.  Suricata needs a good bit of CPU horsepower, and the more rules you enable the more horsepower it needs.  That needed horsepower includes a pretty fast and capable CPU along with plenty of RAM.  I would say 2GB is cutting it close on RAM.  I would rather have at least 4GB of RAM for Suricata with a lot of rules enabled.

    Are any other packages running on this firewall?  That can further add to load, and if you have another package that needs to download daily updates (such as IP lists or something), then perhaps there is a conflict with the update jobs ???

    Bill



  • @bmeeks:

    @Gertran is on the right track.  Suricata needs a good bit of CPU horsepower, and the more rules you enable the more horsepower it needs.  That needed horsepower includes a pretty fast and capable CPU along with plenty of RAM.  I would say 2GB is cutting it close on RAM.  I would rather have at least 4GB of RAM for Suricata with a lot of rules enabled.

    Bill

    This is way i point a bad hour to make updates. To me 0:30 is to soon, is this the default?. This updates should be made when less people is connected.

    @Teken:


    Can you provide a image capture as to where this is I can’t find it.

    Image annexed.

    @bmeeks is here you should have a good feedback, is the maintainer and the expert.

    ![Suricata - inline mode.jpg](/public/imported_attachments/1/Suricata - inline mode.jpg)
    ![Suricata - inline mode.jpg_thumb](/public/imported_attachments/1/Suricata - inline mode.jpg_thumb)



  • @mais_um:

    This is way i point a bad hour to make updates. To me 0:30 is to soon, is this the default?. This updates should be made when less people is connected.

    I can’t remember the package default off the top of my head, but since the package is used all over the world and in different time zones, there is no “perfect” default value.  The creators of the rule packages each have their own update posting schedules based on their local time.  But their local time is not the same as someone in a different part of the world.  So you just need to experiment.  For me, I set my times to 1:30 AM US Eastern and 1:30 PM Eastern and have not had any issues.  I did have issues in the past with the old default of midnight US Eastern and noon US Eastern.

    Bill



  • @Gertjan:

    Hi,

    Suricata, when updating uses some Watts … never saw stats that showed  increased system usage expressed by Watts before.
    What about process usage ? RAM usage ? Do they follow the Watt usage ?

    Although firewalls like this https://store.netgate.com/SG-1000.aspx shouldn’t maybe not using heavy packages as Suricate (I might be wrong here), I wonder what you use for hardware.

    I haven’t been around when the system shuts down the network due to work flow. In the near future I’ll schedule it for a time I’m around and confirm what the process / RAM usage is. I’m sure lots of folks haven’t ever seen a energy chart placed on this forum to describe a problem. I only did so because it helped illustrate the factual data of when the time event happen and the correlation was a increase in power due to more processing power being called upon.

    I know what every single device and circuit consumes, for how long, frequency of, and if there are any out of band readings in the home. If there is my systems shut them down and send alerts to me indicating when and where.



  • @bmeeks:

    @Gertran is on the right track.  Suricata needs a good bit of CPU horsepower, and the more rules you enable the more horsepower it needs.  That needed horsepower includes a pretty fast and capable CPU along with plenty of RAM.  I would say 2GB is cutting it close on RAM.  I would rather have at least 4GB of RAM for Suricata with a lot of rules enabled.

    Are any other packages running on this firewall?  That can further add to load, and if you have another package that needs to download daily updates (such as IP lists or something), then perhaps there is a conflict with the update jobs ???

    Bill

    Hello Bill,

    This is a brand new Micro PC I propped up in late 2017 and the hardware specification are these:

    Intel® Atom™ CPU E3845 @ 1.91GHz
    4 CPUs: 1 package(s) x 4 core(s)
    AES-NI CPU Crypto: Yes (active)



  • @mais_um:

    @bmeeks:

    @Gertran is on the right track.  Suricata needs a good bit of CPU horsepower, and the more rules you enable the more horsepower it needs.  That needed horsepower includes a pretty fast and capable CPU along with plenty of RAM.  I would say 2GB is cutting it close on RAM.  I would rather have at least 4GB of RAM for Suricata with a lot of rules enabled.

    Bill

    This is way i point a bad hour to make updates. To me 0:30 is to soon, is this the default?. This updates should be made when less people is connected.

    @Teken:


    Can you provide a image capture as to where this is I can’t find it.

    Image annexed.

    @bmeeks is here you should have a good feedback, is the maintainer and the expert.

    I do no see IPS Mode in the 2.4.2 P1 firmware release.



  • @Teken:

    @bmeeks:

    @Gertran is on the right track.  Suricata needs a good bit of CPU horsepower, and the more rules you enable the more horsepower it needs.  That needed horsepower includes a pretty fast and capable CPU along with plenty of RAM.  I would say 2GB is cutting it close on RAM.  I would rather have at least 4GB of RAM for Suricata with a lot of rules enabled.

    Are any other packages running on this firewall?  That can further add to load, and if you have another package that needs to download daily updates (such as IP lists or something), then perhaps there is a conflict with the update jobs ???

    Bill

    Hello Bill,

    This is a brand new Micro PC I propped up in late 2017 and the hardware specification are these:

    Intel® Atom™ CPU E3845 @ 1.91GHz
    4 CPUs: 1 package(s) x 4 core(s)
    AES-NI CPU Crypto: Yes (active)

    The amount of RAM might be an issue.  It really depends on the number of enabled rules.  When the scheduled updates run, Suricata basically has to load both sets of rules into memory at the same time, then when everything is loaded up it switches over to using the new rules in RAM and discards the old ones.  So for a brief period of time you need almost twice as much RAM as compared to the rest of the running time.  With a limited amount of RAM to start with, this could result in memory paging (the swapping in and out to disk of some RAM content).  Your power spike is simply the physical evidence of the much higher CPU workload during the task.  A higher CPU workload is normal for rule updates.

    Bill



  • @bmeeks:

    @Teken:

    @bmeeks:

    @Gertran is on the right track.  Suricata needs a good bit of CPU horsepower, and the more rules you enable the more horsepower it needs.  That needed horsepower includes a pretty fast and capable CPU along with plenty of RAM.  I would say 2GB is cutting it close on RAM.  I would rather have at least 4GB of RAM for Suricata with a lot of rules enabled.

    Are any other packages running on this firewall?  That can further add to load, and if you have another package that needs to download daily updates (such as IP lists or something), then perhaps there is a conflict with the update jobs ???

    Bill

    Hello Bill,

    This is a brand new Micro PC I propped up in late 2017 and the hardware specification are these:

    Intel® Atom™ CPU E3845 @ 1.91GHz
    4 CPUs: 1 package(s) x 4 core(s)
    AES-NI CPU Crypto: Yes (active)

    The amount of RAM might be an issue.  It really depends on the number of enabled rules.  When the scheduled updates run, Suricata basically has to load both sets of rules into memory at the same time, then when everything is loaded up it switches over to using the new rules in RAM and discards the old ones.  So for a brief period of time you need almost twice as much RAM as compared to the rest of the running time.  With a limited amount of RAM to start with, this could result in memory paging (the swapping in and out to disk of some RAM content).  Your power spike is simply the physical evidence of the much higher CPU workload during the task.  A higher CPU workload is normal for rule updates.

    Bill

    Hello Bill,

    My apologies I didn’t state how much RAM I have on board. This Micro PC has 8 GB of RAM which should be more than plenty to run pfSense.



  • @Teken:

    @bmeeks:

    @Teken:

    @bmeeks:

    @Gertran is on the right track.  Suricata needs a good bit of CPU horsepower, and the more rules you enable the more horsepower it needs.  That needed horsepower includes a pretty fast and capable CPU along with plenty of RAM.  I would say 2GB is cutting it close on RAM.  I would rather have at least 4GB of RAM for Suricata with a lot of rules enabled.

    Are any other packages running on this firewall?  That can further add to load, and if you have another package that needs to download daily updates (such as IP lists or something), then perhaps there is a conflict with the update jobs ???

    Bill

    Hello Bill,

    This is a brand new Micro PC I propped up in late 2017 and the hardware specification are these:

    Intel® Atom™ CPU E3845 @ 1.91GHz
    4 CPUs: 1 package(s) x 4 core(s)
    AES-NI CPU Crypto: Yes (active)

    The amount of RAM might be an issue.  It really depends on the number of enabled rules.  When the scheduled updates run, Suricata basically has to load both sets of rules into memory at the same time, then when everything is loaded up it switches over to using the new rules in RAM and discards the old ones.  So for a brief period of time you need almost twice as much RAM as compared to the rest of the running time.  With a limited amount of RAM to start with, this could result in memory paging (the swapping in and out to disk of some RAM content).  Your power spike is simply the physical evidence of the much higher CPU workload during the task.  A higher CPU workload is normal for rule updates.

    Bill

    Hello Bill,

    My apologies I didn’t state how much RAM I have on board. This Micro PC has 8 GB of RAM which should be more than plenty to run pfSense.

    OK, 8 GB should be plenty of RAM.  A spike in CPU usage and power consumption would be normal, but losing network connectivity is not normal.  That has not been reported by others so far as I can tell, so it appears to be isolated to your case.  Is there anything unusual about your network card?  Is it a USB device or a standard port on the motherboard?  Are you using blocking mode for Suricata or just the default IDS mode?  This is set on the INTERFACE SETTINGS tab for each configured Suricata interface.  You have a choice of two blocking modes when you enable “Block Offenders”.  Those two modes are Legacy Mode and Inline IPS Mode.  If you have blocking configured, which of those two modes do you have enabled?

    Bill



  • Block Offenders is not checked on either LAN / WAN interface. The Micro PC has four on board Intel WG82583 NIC’s.

    Thank You!



  • @Teken:

    Block Offenders is not checked on either LAN / WAN interface. The Micro PC has four on board Intel WG82583 NIC’s.

    Thank You!

    With that basic setup (no blocking), I really can’t imagine a scenario where Suricata could break your network connectivity.  In that default setup you have, it only launches libpcap to get copies of packets coming through the interface so it can analyze them.  That’s it.  Are you 100% positive Suricata is the issue?  I know of nothing within the binary that can break your network connectivity, especially with blocking not enabled.  In the GUI, even if that code got really intense, the worst that should happen is the GUI responsivness would suck for a few seconds.

    When you say the network “breaks”, does it self recover?  In other words, will connectivity come back if you do nothing?  If not, what do you do to restore connectivity?  Those can be troubleshooting hints.

    Bill



  • Hello Bill,

    When the system is updating all networking is halted specifically anything to do with WiFi. If I’m watching Netflix on my LG smart TV the only way to restore the connection is to cycle power to the TV. This same behavior is seen on two smart weather stations I am beta testing.

    On those specific devices I can see the hubs showing a red LED.

    A red LED indicates on those specific pieces of hardware that the hub is not able to connect to the WiFi network / communicate to the weather servers.

    The only remedy is to cycle power to those two hubs to establish a WiFi connection. Since I started this thread this problem has not reappeared due to the fact I have pushed the Suricata update to 29 days etc.

    Any further insight is greatly appreciated!



  • More insight is going to take some additional info from logs and recreating the event more often than every 29 days.

    If you are game, change the interval back to every 6 or 12 hours and change the time for the update to coincide with a period when you are available to login to the firewall and see what’s happening and grab some log info during the issue.  Specifically I need to see the system log entries during the time the loss of connectivity is happening along with your suricata.log file for the interface.

    The system log for pfSense is under STATUS >> SYSTEM LOGS.  You can find the Suricata log I need under the LOGS VIEW tab and then select the suricata.log file in the drop-down selector.  The system log will likely contain the most relevant info.  Just remember to capture the info during the time the connectivity problem is happening.  Feel free to obfuscate IP addresses if you want to.

    Bill



  • @bmeeks:

    More insight is going to take some additional info from logs and recreating the event more often than every 29 days.

    If you are game, change the interval back to every 6 or 12 hours and change the time for the update to coincide with a period when you are available to login to the firewall and see what’s happening and grab some log info during the issue.  Specifically I need to see the system log entries during the time the loss of connectivity is happening along with your suricata.log file for the interface.

    The system log for pfSense is under STATUS >> SYSTEM LOGS.  You can find the Suricata log I need under the LOGS VIEW tab and then select the suricata.log file in the drop-down selector.  The system log will likely contain the most relevant info.  Just remember to capture the info during the time the connectivity problem is happening.  Feel free to obfuscate IP addresses if you want to.

    Bill

    Hi Bill,

    Apologies for the tardy reply I’ve been on the road for work for several weeks. Upon my return I shall follow your suggestions and it should be noted since moving the update interval to 29 days. Nothing bad has happen to any network appliance in the home. Since my last reply that very much affirms this issue is directly related to the update.

    Thank You!



  • I am having issues with apparently random killing all network connections with 2.4.2p1 also, but I’m not sure it is Suricata doing it.  For my system it happens at a seemingly random time from 1-10 days.  I have been unable to correlate it to any cron job, and the logs do not show anything except the gateway is down when it happens.  For mine the network interface outage usually lasts 1-2 minutes, though at least once I had to reboot.  The system at the console thinks all is fine, and there are no crashes.  I’ve tried different hardware including swapping servers and network cards, but the issue persists.  Kinda hard to troubleshoot with the frequency so far apart, and no known way of inducing the error.

    Hardware:  HP DL380 G6, 48 Gig memory, Xeon 5650x2, Intel i340 4 port network cards x 2.
    Packages:  pfBlockerNG 2.1.2_2, mailreport 3.1, Suricata 4.0.3_1


  • Netgate

    So why is the gateway down? Do you have State killing on gateway failure enabled?



  • The gateways appear to go down because all of the Interfaces stop passing traffic (not just the WAN gateways).  I currently have 4 physical interfaces configured. 2 WAN, 1 LAN, and 1 LAN that is just for management of the firewall (no NAT, no Suricata).  The 2 LAN interfaces are configured as VLANS.  When the issue occurs, I lose connectivity to the management LAN as well (no Suricata on that interface).  Typically I will start to see packet loss on 1 gateway and within seconds the other, then all go down including the LAN’s for about a minute.  I know the LAN’s go down as I have a separate Zabbix server pinging, and I cannot reach the web GUI until back up.  I have verified that when this occurs the switch and server Ethernet ports do not lose physical connectivity.

    As for “State killing on gateway failure enabled?”, in Suricata I do have the “Kill States” checked which is the default.  However, that should have no effect on the Interfaces that are not set up to be monitored by Suricata, and those go down as well.  Do you recommend turning “Kill States” off?

    Also in my case if it helps, using Suricata in Legacy mode.

    Another anomaly which may/may not be related:  After a server restart Suricata does not show any blocks.  It shows alerts, but no blocks.  If I restart the Suricata service, it works normally and shows blocks (I am only blocking mostly p2p).  This morning I updated to Suricata 4.0.3_2 but it did not change this behavior.

    I don’t know if this is the same issue as  Teken’s or not, but it looked similar so I thought I’d throw this out there.



  • @RichH:

    Another anomaly which may/may not be related:  After a server restart Suricata does not show any blocks.  It shows alerts, but no blocks.  If I restart the Suricata service, it works normally and shows blocks (I am only blocking mostly p2p).  This morning I updated to Suricata 4.0.3_2 but it did not change this behavior.

    I don’t know if this is the same issue as  Teken’s or not, but it looked similar so I thought I’d throw this out there.

    What server are talking about restarting?  Is it the box hosting pfSense and Suricata?  When you say you “restart Suricata”, is it showing as running on the interface when you click the restart icon or is it showing as not running and you click the start icon?

    In regards to “showing alerts but no blocks”, you do realize that the ALERTS tab is reading from a log file.  So if that log file is not erased during a reboot of the firewall the alerts in the log file will be displayed on the ALERTS tab after the reboot.  New alerts at the top and older ones at the bottom.  The BLOCKS tab, on the other hand, literally reads the contents of the packet filter firewall’s snort2c table and displays those.  After a reboot, the snort2c table is cleared out until network traffic triggers more alerts which the result in blocks.  So after a firewall reboot and before any new alerts have been received, the BLOCKS tab will be empty.

    Bill



  • The “server” is the HP G6 box running pfSense, sorry I didn’t make that clear.  As for Suricata “showing alerts but no blocks” – they are not logs but current alerts that can be seen by the date and time.  I even tested going to do a P2P download (which is set to be blocked by Suricata) just to be sure.  It is logged in the alerts, but not in the “Blocks” section.  I restart the Suricata service from Status-Services and then do the test again.  The Block now shows.  It is a reproducible issue.  I just wish the killed network connections was something I knew how to reproduce, as I presume once it can be reproduced it can be fixed!

    Thanks much!



  • @RichH:

    The “server” is the HP G6 box running pfSense, sorry I didn’t make that clear.  As for Suricata “showing alerts but no blocks” – they are not logs but current alerts that can be seen by the date and time.  I even tested going to do a P2P download (which is set to be blocked by Suricata) just to be sure.  It is logged in the alerts, but not in the “Blocks” section.  I restart the Suricata service from Status-Services and then do the test again.  The Block now shows.  It is a reproducible issue.  I just wish the killed network connections was something I knew how to reproduce, as I presume once it can be reproduced it can be fixed!

    Thanks much!

    Yes, they are logs.  Every single alert shown on that tab is read from an alert log file in /var/logs/suricata and sub-directories present there for each interface.  The PHP code on the ALERTS tab opens the current log file and reads it into memory and displays the contents.  That is the only place alerts are pulled from (the log files).  As for blocks, when the BLOCKS tab loads and renders in browser, PHP code is executed that reads the snort2c table from pfSense, loads its contents (IP addresses) into an array and displays the IP addresses it found.  Those are the “blocks” displayed.  Here is a snippet of the actual code that does that:

    exec("/sbin/pfctl -t snort2c -T show", $blocked_ips);
    

    I’m having a very difficult time reconciling what you describe with how the internals of the code actually works.  Something is not quite right in your setup perhaps.  For starters, when you see this behavior and go to the INTERFACES tab, what icon is showing for each of the Suricata-enabled interfaces?  Is it a green check or is it a red X?  Green means Suricata is running, red X means it is not or has died.  Do you run any other packages on this firewall?  One package known to cause issues with Snort and Suricata both is the Service Watchdog package.  It will start multiple copies of Suricata that then fight with each other.

    Bill



  • Thanks Bill. The 2 Suricata Interfaces I have set up to use are each showing a green check (1 WAN and 1 LAN), and have been green even when I have had issues.  I presume Suricata is working as the current alerts show up even when the blocks do not.  For packages I have only these 3:

    pfBlockerNG 2.1.2_2
    mailreport 3.1
    Suricata 4.0.3_2 (though I see an update to 4.04 is available which I’ll apply later today)

    I had to restart this morning but did not get to check right away; when I did it was working so maybe it is not consistent or takes a while to start working again or something.


 

© Copyright 2002 - 2018 Rubicon Communications, LLC | Privacy Policy