Suricata 4.1.X interface stopping [Sorted by going back to Snort]
-
I have moved pfSense from Virtual apliance to HW, did clean install and imported settings. Previously I did use snort but I want to setup suricata, so I did.
I have 6 interafces that surricata is running on, 2 WAN connections and rest is LAN and WIFI. Only the WAN interfaces are curently configured to block trafic. Everything is working fine except my main LAN interface which will just randomly stop enywhere between 2-12 hours and cannot be started again until Suricata is restarted (or pid file deleted, see below).
There are no obvious errors why surricata stopped (in my opinion, errors are only related to signatures).
Last entries for suricata.log on LAN between rule reloads:... 10/1/2020 -- 09:44:19 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET any -> $HOME_NET $HTTP_PORTS (msg:"SERVER-WEBAPP Lexmark Markvision Enterprise LibraryFileUploadServlet directory traversal attempt"; flow:to_server,established; content:"/mve/upload/library"; fast_pattern:only; http_uri; file_data; content:"PK|03 04|"; depth:4; byte_extract:2,22,filename_len,relative,little; content:"..|5C|"; within:filename_len; distance:2; metadata:policy max-detect-ips drop, policy security-ips drop, service http; reference:bugtraq,72726; reference:cve,2014-9375; classtype:web-application-attack; sid:34056; rev:3;)" from file /usr/local/etc/suricata/suricata_20029_bge0/rules/suricata.rules at line 34018 10/1/2020 -- 09:44:19 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - Can't use file_data with flow:to_server or flow:from_client with http. 10/1/2020 -- 09:44:19 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET any -> $HOME_NET $HTTP_PORTS (msg:"SERVER-WEBAPP Lexmark Markvision Enterprise LibraryFileUploadServlet directory traversal attempt"; flow:to_server,established; content:"/mve/upload/library"; fast_pattern:only; http_uri; file_data; content:"PK|03 04|"; depth:4; byte_extract:2,22,filename_len,relative,little; content:"../"; within:filename_len; distance:2; metadata:policy max-detect-ips drop, policy security-ips drop, service http; reference:bugtraq,72726; reference:cve,2014-9375; classtype:web-application-attack; sid:34055; rev:3;)" from file /usr/local/etc/suricata/suricata_20029_bge0/rules/suricata.rules at line 34019 10/1/2020 -- 09:44:19 - <Error> -- [ERRCODE: SC_ERR_RULE_KEYWORD_UNKNOWN(102)] - unknown rule keyword 'http_raw_cookie'. 10/1/2020 -- 09:44:19 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET any -> $HOME_NET $HTTP_PORTS (msg:"SERVER-WEBAPP Watchguard XCS compose.php SQL injection attempt"; flow:to_server,established; content:"/borderpost/imp/compose.php"; fast_pattern:only; http_uri; content:"sid="; nocase; http_raw_cookie; content:"%3B"; distance:0; nocase; http_raw_cookie; pcre:"/sid=[^\x3b]*?%3B/Ki"; metadata:policy max-detect-ips drop, policy security-ips drop, service http; reference:url,www.watchguard.com/support/release-notes/xcs/index.aspx; classtype:attempted-admin; sid:35573; rev:2;)" from file /usr/local/etc/suricata/suricata_20029_bge0/rules/suricata.rules at line 34145 10/1/2020 -- 09:44:20 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - Can't use file_data with flow:to_server or flow:from_client with http. 10/1/2020 -- 09:44:20 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET any -> $HOME_NET $HTTP_PORTS (msg:"SERVER-WEBAPP ManageEngine ServiceDesk ExportImport.do directory traversal attempt"; flow:to_server,established; content:"/ExportImport.do"; fast_pattern:only; http_uri; file_data; content:"PK|03 04|"; byte_extract:2,22,filename_len,relative,little; content:"..|5C|"; within:filename_len; distance:2; metadata:policy max-detect-ips drop, policy security-ips drop, service http; reference:url,www.manageengine.com/products/service-desk/service-packs.html; classtype:web-application-attack; sid:36102; rev:3;)" from file /usr/local/etc/suricata/suricata_20029_bge0/rules/suricata.rules at line 34171 10/1/2020 -- 09:44:20 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - Can't use file_data with flow:to_server or flow:from_client with http. 10/1/2020 -- 09:44:20 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET any -> $HOME_NET $HTTP_PORTS (msg:"SERVER-WEBAPP ManageEngine ServiceDesk ExportImport.do directory traversal attempt"; flow:to_server,established; content:"/ExportImport.do"; fast_pattern:only; http_uri; file_data; content:"PK|03 04|"; byte_extract:2,22,filename_len,relative,little; content:"../"; within:filename_len; distance:2; metadata:policy max-detect-ips drop, policy security-ips drop, service http; reference:url,www.manageengine.com/products/service-desk/service-packs.html; classtype:web-application-attack; sid:36101; rev:3;)" from file /usr/local/etc/suricata/suricata_20029_bge0/rules/suricata.rules at line 34172 10/1/2020 -- 09:44:20 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - previous keyword has a fast_pattern:only; set. Can't have relative keywords around a fast_pattern only content 10/1/2020 -- 09:44:20 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET any -> $HOME_NET $HTTP_PORTS (msg:"SERVER-WEBAPP Foscam IP Camera multipart boundary stack buffer overflow attempt"; flow:to_server,established; content:"/cgi-bin/CGIProxy.fcgi"; nocase; http_uri; content:"usrBeatHeart"; fast_pattern:only; content:"Boundary="; nocase; http_raw_header; isdataat:256,relative; content:!"|0A|"; within:256; http_raw_header; metadata:policy max-detect-ips drop, policy security-ips drop, service http; reference:cve,2017-2830; reference:url,www.talosintelligence.com/reports/TALOS-2017-0331/; classtype:web-application-attack; sid:42437; rev:3;)" from file /usr/local/etc/suricata/suricata_20029_bge0/rules/suricata.rules at line 34661 10/1/2020 -- 09:44:20 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - previous keyword has a fast_pattern:only; set. Can't have relative keywords around a fast_pattern only content 10/1/2020 -- 09:44:20 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET any -> $HOME_NET $HTTP_PORTS (msg:"SERVER-WEBAPP DotNetNuke DNNPersonalization remote code execution attempt"; flow:to_server,established; content:"DNNPersonalization"; fast_pattern:only; content:"DNNPersonalization"; http_cookie; content:"System.Data.Services.Internal.ExpandedWrapper"; within:100; http_cookie; content:"System.Windows.Data.ObjectDataProvider"; within:200; http_cookie; metadata:policy max-detect-ips drop, policy security-ips drop, service http; reference:cve,2017-9822; reference:url,www.dnnsoftware.com/community/security/security-center; classtype:attempted-admin; sid:45414; rev:2;)" from file /usr/local/etc/suricata/suricata_20029_bge0/rules/suricata.rules at line 34802 10/1/2020 -- 09:44:20 - <Error> -- [ERRCODE: SC_ERR_RULE_KEYWORD_UNKNOWN(102)] - unknown rule keyword 'http_raw_cookie'. 10/1/2020 -- 09:44:20 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET any -> $HOME_NET $HTTP_PORTS (msg:"SERVER-WEBAPP Multiple products DVR admin password leak attempt"; flow:to_server,established; content:"/device.rsp"; fast_pattern:only; http_uri; content:"uid="; http_raw_cookie; content:"cmd=list"; metadata:policy balanced-ips drop, policy max-detect-ips drop, policy security-ips drop, service http; reference:cve,2018-9995; classtype:web-application-attack; sid:46825; rev:1;)" from file /usr/local/etc/suricata/suricata_20029_bge0/rules/suricata.rules at line 34945 10/1/2020 -- 09:44:20 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - previous keyword has a fast_pattern:only; set. Can't have relative keywords around a fast_pattern only content 10/1/2020 -- 09:44:20 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $EXTERNAL_NET any -> $HOME_NET $HTTP_PORTS (msg:"SERVER-WEBAPP GitList searchTree git grep arbitrary command execution attempt"; flow:to_server,established; content:"query=--open-files-in-pager"; fast_pattern:only; content:"/tree/"; http_uri; content:"/search"; distance:0; http_uri; metadata:policy max-detect-ips drop, policy security-ips drop, service http; classtype:attempted-user; sid:47599; rev:1;)" from file /usr/local/etc/suricata/suricata_20029_bge0/rules/suricata.rules at line 35044 10/1/2020 -- 09:44:21 - <Info> -- 2 rule files processed. 35960 rules successfully loaded, 84 rules failed 10/1/2020 -- 09:44:21 - <Info> -- Threshold config parsed: 31 rule(s) found 10/1/2020 -- 09:44:23 - <Info> -- 35964 signatures processed. 1081 are IP-only rules, 6498 are inspecting packet payload, 22560 inspect application layer, 103 are decoder event only 10/1/2020 -- 09:44:23 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'file.zip&file.silverlight' is checked but not set. Checked in 28579 and 6 other sigs 10/1/2020 -- 09:44:23 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'file.swf|file.ole' is checked but not set. Checked in 25676 and 1 other sigs 10/1/2020 -- 09:44:23 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'file.jpeg|file.tiff' is checked but not set. Checked in 25346 and 1 other sigs 10/1/2020 -- 09:44:23 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'file.jpeg|file.xps' is checked but not set. Checked in 41202 and 1 other sigs 10/1/2020 -- 09:44:23 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'file.jar|file.class|file.ttf' is checked but not set. Checked in 24701 and 1 other sigs 10/1/2020 -- 09:44:23 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'file.class|file.jar' is checked but not set. Checked in 31540 and 1 other sigs 10/1/2020 -- 09:44:23 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'file.tiff|file.doc' is checked but not set. Checked in 28464 and 1 other sigs 10/1/2020 -- 09:44:23 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'file.rtf|file.ole' is checked but not set. Checked in 37559 and 1 other sigs 10/1/2020 -- 09:44:23 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'file.doc|file.docx' is checked but not set. Checked in 45370 and 1 other sigs 10/1/2020 -- 09:44:23 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'file.ole|file.doc' is checked but not set. Checked in 30533 and 3 other sigs 10/1/2020 -- 09:44:23 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'file.corel|file.doc' is checked but not set. Checked in 36500 and 1 other sigs 10/1/2020 -- 09:44:23 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'file.pdf&file.ttf' is checked but not set. Checked in 28585 and 1 other sigs 10/1/2020 -- 09:44:23 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'file.doc|file.rtf' is checked but not set. Checked in 45519 and 1 other sigs 10/1/2020 -- 09:44:23 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'file.xls&file.ole' is checked but not set. Checked in 30990 and 1 other sigs 10/1/2020 -- 09:44:23 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'file.doc|file.xls' is checked but not set. Checked in 44559 and 1 other sigs 10/1/2020 -- 09:44:23 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'file.pyc|file.zip' is checked but not set. Checked in 45477 and 1 other sigs 10/1/2020 -- 09:44:23 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'file.doc|file.docm' is checked but not set. Checked in 43975 and 1 other sigs 10/1/2020 -- 09:44:23 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'file.ppsx&file.zip' is checked but not set. Checked in 26068 and 1 other sigs 10/1/2020 -- 09:44:23 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'file.docm|file.docx|file.ppsx|file.pptx|file.xlsx' is checked but not set. Checked in 26066 and 1 other sigs 10/1/2020 -- 09:44:43 - <Info> -- cleaning up signature grouping structure... complete 10/1/2020 -- 09:44:43 - <Notice> -- rule reload completeIt seams that after some of the reloads interface just refuses to start again. If I try to start it again it gives me:
10/1/2020 -- 10:59:33 - <Notice> -- This is Suricata version 4.1.6 RELEASE 10/1/2020 -- 10:59:33 - <Info> -- CPUs/cores online: 4 10/1/2020 -- 10:59:33 - <Info> -- HTTP memcap: 67108864 10/1/2020 -- 10:59:34 - <Notice> -- using flow hash instead of active packets 10/1/2020 -- 10:59:34 - <Error> -- [ERRCODE: SC_ERR_INITIALIZATION(45)] - pid file '/var/run/suricata_bge020029.pid' exists but appears stale. Make sure Suricata is not running and then remove /var/run/suricata_bge020029.pid. Aborting!Of course if I delete the pid file (or restart Suricata), it will start again but it also will "crash" again in few hours.
I tried checksum offloading both enabled and disabled and because I have a lot of RAM in this system I assigned a lot of it to Stream Memory Cap: 805306400 just to be shure that it is not a cause.
I originally setup suricata v 4.1.5 and updated this morning to 4.1.6 so problem is not related to latest or previous version in my case.
What steps would you recomend to take or which logs could shine some light onto this problem?
-
Look in the pfSense system log to see what errors might be getting logged there. The fact you have a leftover, or "stale", PID file indicates the Suricata binary is crashing hard and not cleaning up behind itself. Look in the pfSense log for any messages related to Suricata around the time the interface does down.
What kind of hardware platform are you running pfSense on now? Is it Intel/AMD64 or ARM?
Those rule parsing errors are expected when you run the Snort Subscriber Rules with Suricata. Snort has a number of rule options and keywords that Suricata does not support. However, those parsing errors are not the cause of Suricata stopping. When Suricata does not understand a rule, it logs an error and just skips loading that rule completely.
Post back with anything you find in the pfSense system log.
From the log entry showing the rules parsing errors, it appears Suricata might be crashing during the live reload process (if you have that enabled). You could try turning that off if enabled. Also make sure you DO NOT have the Service Watchdog package configured to monitor Suricata. That will cause crashes because Service Watchdog does not understand how to correctly monitor Suricata or Snort.
-
Live reload is disabled and I do not use Service Watchdog
Should I be looking at "System -> General" under logs?
Only mention of Suricata there is: (LAN = bge0)
Jan 10 13:58:37 SuricataStartup 45038 Suricata START for LAN(20029_bge0)... Jan 10 13:56:36 SuricataStartup 29563 Suricata START for LAN(20029_bge0)... Jan 10 12:58:56 SuricataStartup 31927 Suricata START for LAN(20029_bge0)... Jan 10 08:31:49 SuricataStartup 98333 Suricata START for WAN2(2355_igb1)... Jan 10 08:31:48 SuricataStartup 96262 Suricata START for WIFI-GUEST(59582_bge1.101)... Jan 10 08:31:46 SuricataStartup 94669 Suricata START for WIFI-EMP(54693_bge1.102)... Jan 10 08:31:45 SuricataStartup 93549 Suricata START for WIFI(21461_bge1)... Jan 10 08:31:44 SuricataStartup 91898 Suricata START for LAN(20029_bge0)... Jan 10 08:31:42 SuricataStartup 37751 Suricata START for WAN1(24374_igb0)... Jan 10 08:29:20 SuricataStartup 50897 Suricata STOP for WAN2(2355_igb1)... Jan 10 08:29:18 SuricataStartup 47441 Suricata STOP for WIFI-GUEST(59582_bge1.101)... Jan 10 08:29:16 SuricataStartup 42601 Suricata STOP for WIFI-EMP(54693_bge1.102)... Jan 10 08:29:13 SuricataStartup 36724 Suricata STOP for WIFI(21461_bge1)... Jan 10 08:29:10 SuricataStartup 18941 Suricata STOP for LAN(20029_bge0)... Jan 10 08:29:08 SuricataStartup 14207 Suricata STOP for WAN1(24374_igb0)... Jan 10 08:17:06 SuricataStartup 16942 Suricata START for WAN2(2355_igb1)... Jan 10 08:17:04 SuricataStartup 4625 Suricata START for WIFI-GUEST(59582_bge1.101)... Jan 10 08:17:02 SuricataStartup 95564 Suricata START for WIFI-EMP(54693_bge1.102)... Jan 10 08:17:01 SuricataStartup 85993 Suricata START for WIFI(21461_bge1)... Jan 10 08:16:59 SuricataStartup 79421 Suricata START for LAN(20029_bge0)... Jan 10 08:16:58 SuricataStartup 77445 Suricata START for WAN1(24374_igb0)... Jan 10 08:16:56 SuricataStartup 66239 Suricata STOP for WAN2(2355_igb1)... Jan 10 08:16:54 SuricataStartup 56979 Suricata STOP for WIFI-GUEST(59582_bge1.101)... Jan 10 08:16:52 SuricataStartup 50785 Suricata STOP for WIFI-EMP(54693_bge1.102)... Jan 10 08:16:49 SuricataStartup 36495 Suricata STOP for WIFI(21461_bge1)... Jan 10 08:16:48 SuricataStartup 31261 Suricata STOP for LAN(20029_bge0)... Jan 10 08:16:46 SuricataStartup 19439 Suricata STOP for WAN1(24374_igb0)...System is Intel/AMD64 on AMD Opteron(tm) X3418 APU With 8 GB of RAM
-
@r43K9o said in Suricata 4.1.X interface stopping:
Live reload is disabled and I do not use Service Watchdog
Should I be looking at "System -> General" under logs?
Only mention of Suricata there is: (LAN = bge0)
Jan 10 13:58:37 SuricataStartup 45038 Suricata START for LAN(20029_bge0)... Jan 10 13:56:36 SuricataStartup 29563 Suricata START for LAN(20029_bge0)... Jan 10 12:58:56 SuricataStartup 31927 Suricata START for LAN(20029_bge0)... Jan 10 08:31:49 SuricataStartup 98333 Suricata START for WAN2(2355_igb1)... Jan 10 08:31:48 SuricataStartup 96262 Suricata START for WIFI-GUEST(59582_bge1.101)... Jan 10 08:31:46 SuricataStartup 94669 Suricata START for WIFI-EMP(54693_bge1.102)... Jan 10 08:31:45 SuricataStartup 93549 Suricata START for WIFI(21461_bge1)... Jan 10 08:31:44 SuricataStartup 91898 Suricata START for LAN(20029_bge0)... Jan 10 08:31:42 SuricataStartup 37751 Suricata START for WAN1(24374_igb0)... Jan 10 08:29:20 SuricataStartup 50897 Suricata STOP for WAN2(2355_igb1)... Jan 10 08:29:18 SuricataStartup 47441 Suricata STOP for WIFI-GUEST(59582_bge1.101)... Jan 10 08:29:16 SuricataStartup 42601 Suricata STOP for WIFI-EMP(54693_bge1.102)... Jan 10 08:29:13 SuricataStartup 36724 Suricata STOP for WIFI(21461_bge1)... Jan 10 08:29:10 SuricataStartup 18941 Suricata STOP for LAN(20029_bge0)... Jan 10 08:29:08 SuricataStartup 14207 Suricata STOP for WAN1(24374_igb0)... Jan 10 08:17:06 SuricataStartup 16942 Suricata START for WAN2(2355_igb1)... Jan 10 08:17:04 SuricataStartup 4625 Suricata START for WIFI-GUEST(59582_bge1.101)... Jan 10 08:17:02 SuricataStartup 95564 Suricata START for WIFI-EMP(54693_bge1.102)... Jan 10 08:17:01 SuricataStartup 85993 Suricata START for WIFI(21461_bge1)... Jan 10 08:16:59 SuricataStartup 79421 Suricata START for LAN(20029_bge0)... Jan 10 08:16:58 SuricataStartup 77445 Suricata START for WAN1(24374_igb0)... Jan 10 08:16:56 SuricataStartup 66239 Suricata STOP for WAN2(2355_igb1)... Jan 10 08:16:54 SuricataStartup 56979 Suricata STOP for WIFI-GUEST(59582_bge1.101)... Jan 10 08:16:52 SuricataStartup 50785 Suricata STOP for WIFI-EMP(54693_bge1.102)... Jan 10 08:16:49 SuricataStartup 36495 Suricata STOP for WIFI(21461_bge1)... Jan 10 08:16:48 SuricataStartup 31261 Suricata STOP for LAN(20029_bge0)... Jan 10 08:16:46 SuricataStartup 19439 Suricata STOP for WAN1(24374_igb0)...System is Intel/AMD64 on AMD Opteron(tm) X3418 APU With 8 GB of RAM
Yes, you are looking at the correct system log, but you might need to let Suricata run until it crashes and then check the system log immediately so that any logged event does not get "rolled off" due to the way
clogon pfSense only keeps the most current events.Another question would be why are your interfaces restarting so often? Suricata generally only should look for rules updates at most twice per day. And really once per day is sufficient. Do you have interfaces flapping, or is something on your pfSense box causing it to issue the "restart all packages" command often?
Also scour the system log for any "out of memory" events. That many interfaces with only 8 GB might be a stretch depending on the number of enabled rules. During a rule update/swap, Suricata will briefly need almost double the amount of normal RAM because it has to keep the old and new versions of the rules in memory at the same time.
-
@bmeeks Ok, sorry I'm dumb process is running under PHP... This is everything what happened from update... As far as i know interface monitoring "crashed once during that time"
(I had to upload it as a file because othervise it was flaget as spam for some reason)
Suricata was restarted once in the morning because LAN interface monitoring was down and then shortly after I upgraded to 4.6.1 which I assume causes restart.
-
@r43K9o said in Suricata 4.1.X interface stopping:
@bmeeks Ok, sorry I'm dumb process is running under PHP... This is everything what happened from update... As far as i know interface monitoring "crashed once during that time"
(I had to upload it as a file because othervise it was flaget as spam for some reason)
Suricata was restarted once in the morning because LAN interface monitoring was down and then shortly after I upgraded to 4.6.1 which I assume causes restart.
Your LAN interface is the one crashing. Here is the log entry:
Jan 10 10:12:41 kernel pid 92275 (suricata), uid 0: exited on signal 11 (core dumped)And when that interface instance crashes, it will leave its PID file in
/var/run/and thus you get the subsequent startup error about a stale PID file.Why it crashed is not logged (and thus not known). Very well could be a rule that is causing it. The Suricata binary has had issues from time to time with buggy code because they maintain a fairly rapid update/release schedule as compared to Snort. So bugs come and go with new versions of the binary.
I would disable all of the LAN rules temporarily and then start adding them back one category at the time to see if you can pinpoint which rule category might be causing the issue. For now, a rule problem would be my first guess as to what's happening.
-
@bmeeks Ok, thank you for your time I will have a look. I did apply Security IPS Policy so there is quite few rules even thou I disabled few of them already.
I would just like to ask one question that came to my mind after your respnse. Is it possible that surricata could kill the interface in some way that it woulnd not manage to come up again?
It happened to me twice already (on one of the WAN interface) That it just lost a link and until I restarted the whole system - it just refused to accept any trafic. I doubt that it is HW problem because I have Intel NICs but who knows, in the worst case I will go back to snort which I have a good long experince with... -
@r43K9o said in Suricata 4.1.X interface stopping:
@bmeeks Ok, thank you for your time I will have a look. I did apply Security IPS Policy so there is quite few rules even thou I disabled few of them already.
I would just like to ask one question that came to my mind after your respnse. Is it possible that surricata could kill the interface in some way that it woulnd not manage to come up again?
It happened to me twice already (on one of the WAN interface) That it just lost a link and until I restarted the whole system - it just refused to accept any trafic. I doubt that it is HW problem because I have Intel NICs but who knows, in the worst case I will go back to snort which I have a good long experince with...Not likely unless you are trying to use the Inline IPS Mode. That mode uses
netmap, and there are quite a number of NIC drivers that do not work well at all with thenetmapdevice module. That's a FreeBSD issue and not a direct Suricata issue. I see that your LAN and WIFI links appear to be Broadcom NICs. That brand does not work withnetmapmode very well.The default blocking mode is Legacy Blocking Mode. That mode uses libpcap, which should not cause an issue with any NIC driver.
-
@bmeeks Yes I use Legacy mode, newer touch the Inline IPS Mode... Thank you.
-
@r43K9o said in Suricata 4.1.X interface stopping:
@bmeeks It happened to me twice already (on one of the WAN interface) That it just lost a link and until I restarted the whole system - it just refused to accept any trafic.
You could have some kind of hardware issue on the box. You said you migrated from a virtual appliance to actual hardware. Did all the NIC drivers get updated in
config.xml?Did you start fresh or did you import the old configuration from the VM?Something may need changing if you imported from virtual hardware onto actual hardware.Certainly won't hurt anything to try Snort, though. If you do, report back on the results.
Edit: read your original post again after posting this reply and see that one of my questions was already answered.
-
@bmeeks HW is based on HPE ProLiant MicroServer Gen10
Onboard NICs used for local network are Broadcom 5720 which should use driver bge and intel NICs are Intel
82 576 which are igb so I assume that drivers are correct.System sees about 45% CPU usage at peeks and it uses about 30 % of 8GB RAM...
-
Do the instances of stopped traffic flow happen with Suricata disabled? You need to start some systematic troubleshooting by elminating variables and then slowly adding them back one-by-one to see what might be the cause.
Eliminating all packages would be the first option. Let the system run as a basic firewall and see how stable it is. Then starting adding packages back. When you get to Suricata, just activate one interface at a time. Let each one run for some period of time (maybe hours or even a day or two) to see how stable the firewall is.
-
@bmeeks I started to discus the crashing link with IS provider because it was quite weird state, both machines were sending packets but neither of them received anything. so we changed some cables did some config checks but neither of us could find anything wrong with either device because both machines worked quite happily with other HW but not with each other, everything went quiet after I noticed that in ARP table the link/mac address of gateway on WAN expired randomly which of course was followed by lost connection so I set the mac address of the gateway as permanent and the problem did not repeat since. I will wait for another week or so to confirm that this fixes the problem before I inform the ISP. But I know that he uses that gateway for large number of other customers, without problem so I doubt that he will be able to help me with anything.
I disabled Suricata and installed snort and I will be adding functionality slowly back. -
Ok, so after 5 days of running snort with same rulset as suricata without single problem I would say that suricata was a problem. So I will keep using snort as stability is more important for me.
Thank you for help!
