Suricata 4.1.X interface stopping [Sorted by going back to Snort]
-
@r43K9o said in Suricata 4.1.X interface stopping:
Live reload is disabled and I do not use Service Watchdog
Should I be looking at "System -> General" under logs?
Only mention of Suricata there is: (LAN = bge0)
Jan 10 13:58:37 SuricataStartup 45038 Suricata START for LAN(20029_bge0)... Jan 10 13:56:36 SuricataStartup 29563 Suricata START for LAN(20029_bge0)... Jan 10 12:58:56 SuricataStartup 31927 Suricata START for LAN(20029_bge0)... Jan 10 08:31:49 SuricataStartup 98333 Suricata START for WAN2(2355_igb1)... Jan 10 08:31:48 SuricataStartup 96262 Suricata START for WIFI-GUEST(59582_bge1.101)... Jan 10 08:31:46 SuricataStartup 94669 Suricata START for WIFI-EMP(54693_bge1.102)... Jan 10 08:31:45 SuricataStartup 93549 Suricata START for WIFI(21461_bge1)... Jan 10 08:31:44 SuricataStartup 91898 Suricata START for LAN(20029_bge0)... Jan 10 08:31:42 SuricataStartup 37751 Suricata START for WAN1(24374_igb0)... Jan 10 08:29:20 SuricataStartup 50897 Suricata STOP for WAN2(2355_igb1)... Jan 10 08:29:18 SuricataStartup 47441 Suricata STOP for WIFI-GUEST(59582_bge1.101)... Jan 10 08:29:16 SuricataStartup 42601 Suricata STOP for WIFI-EMP(54693_bge1.102)... Jan 10 08:29:13 SuricataStartup 36724 Suricata STOP for WIFI(21461_bge1)... Jan 10 08:29:10 SuricataStartup 18941 Suricata STOP for LAN(20029_bge0)... Jan 10 08:29:08 SuricataStartup 14207 Suricata STOP for WAN1(24374_igb0)... Jan 10 08:17:06 SuricataStartup 16942 Suricata START for WAN2(2355_igb1)... Jan 10 08:17:04 SuricataStartup 4625 Suricata START for WIFI-GUEST(59582_bge1.101)... Jan 10 08:17:02 SuricataStartup 95564 Suricata START for WIFI-EMP(54693_bge1.102)... Jan 10 08:17:01 SuricataStartup 85993 Suricata START for WIFI(21461_bge1)... Jan 10 08:16:59 SuricataStartup 79421 Suricata START for LAN(20029_bge0)... Jan 10 08:16:58 SuricataStartup 77445 Suricata START for WAN1(24374_igb0)... Jan 10 08:16:56 SuricataStartup 66239 Suricata STOP for WAN2(2355_igb1)... Jan 10 08:16:54 SuricataStartup 56979 Suricata STOP for WIFI-GUEST(59582_bge1.101)... Jan 10 08:16:52 SuricataStartup 50785 Suricata STOP for WIFI-EMP(54693_bge1.102)... Jan 10 08:16:49 SuricataStartup 36495 Suricata STOP for WIFI(21461_bge1)... Jan 10 08:16:48 SuricataStartup 31261 Suricata STOP for LAN(20029_bge0)... Jan 10 08:16:46 SuricataStartup 19439 Suricata STOP for WAN1(24374_igb0)...System is Intel/AMD64 on AMD Opteron(tm) X3418 APU With 8 GB of RAM
Yes, you are looking at the correct system log, but you might need to let Suricata run until it crashes and then check the system log immediately so that any logged event does not get "rolled off" due to the way
clogon pfSense only keeps the most current events.Another question would be why are your interfaces restarting so often? Suricata generally only should look for rules updates at most twice per day. And really once per day is sufficient. Do you have interfaces flapping, or is something on your pfSense box causing it to issue the "restart all packages" command often?
Also scour the system log for any "out of memory" events. That many interfaces with only 8 GB might be a stretch depending on the number of enabled rules. During a rule update/swap, Suricata will briefly need almost double the amount of normal RAM because it has to keep the old and new versions of the rules in memory at the same time.
-
@bmeeks Ok, sorry I'm dumb process is running under PHP... This is everything what happened from update... As far as i know interface monitoring "crashed once during that time"
(I had to upload it as a file because othervise it was flaget as spam for some reason)
Suricata was restarted once in the morning because LAN interface monitoring was down and then shortly after I upgraded to 4.6.1 which I assume causes restart.
-
@r43K9o said in Suricata 4.1.X interface stopping:
@bmeeks Ok, sorry I'm dumb process is running under PHP... This is everything what happened from update... As far as i know interface monitoring "crashed once during that time"
(I had to upload it as a file because othervise it was flaget as spam for some reason)
Suricata was restarted once in the morning because LAN interface monitoring was down and then shortly after I upgraded to 4.6.1 which I assume causes restart.
Your LAN interface is the one crashing. Here is the log entry:
Jan 10 10:12:41 kernel pid 92275 (suricata), uid 0: exited on signal 11 (core dumped)And when that interface instance crashes, it will leave its PID file in
/var/run/and thus you get the subsequent startup error about a stale PID file.Why it crashed is not logged (and thus not known). Very well could be a rule that is causing it. The Suricata binary has had issues from time to time with buggy code because they maintain a fairly rapid update/release schedule as compared to Snort. So bugs come and go with new versions of the binary.
I would disable all of the LAN rules temporarily and then start adding them back one category at the time to see if you can pinpoint which rule category might be causing the issue. For now, a rule problem would be my first guess as to what's happening.
-
@bmeeks Ok, thank you for your time I will have a look. I did apply Security IPS Policy so there is quite few rules even thou I disabled few of them already.
I would just like to ask one question that came to my mind after your respnse. Is it possible that surricata could kill the interface in some way that it woulnd not manage to come up again?
It happened to me twice already (on one of the WAN interface) That it just lost a link and until I restarted the whole system - it just refused to accept any trafic. I doubt that it is HW problem because I have Intel NICs but who knows, in the worst case I will go back to snort which I have a good long experince with... -
@r43K9o said in Suricata 4.1.X interface stopping:
@bmeeks Ok, thank you for your time I will have a look. I did apply Security IPS Policy so there is quite few rules even thou I disabled few of them already.
I would just like to ask one question that came to my mind after your respnse. Is it possible that surricata could kill the interface in some way that it woulnd not manage to come up again?
It happened to me twice already (on one of the WAN interface) That it just lost a link and until I restarted the whole system - it just refused to accept any trafic. I doubt that it is HW problem because I have Intel NICs but who knows, in the worst case I will go back to snort which I have a good long experince with...Not likely unless you are trying to use the Inline IPS Mode. That mode uses
netmap, and there are quite a number of NIC drivers that do not work well at all with thenetmapdevice module. That's a FreeBSD issue and not a direct Suricata issue. I see that your LAN and WIFI links appear to be Broadcom NICs. That brand does not work withnetmapmode very well.The default blocking mode is Legacy Blocking Mode. That mode uses libpcap, which should not cause an issue with any NIC driver.
-
@bmeeks Yes I use Legacy mode, newer touch the Inline IPS Mode... Thank you.
-
@r43K9o said in Suricata 4.1.X interface stopping:
@bmeeks It happened to me twice already (on one of the WAN interface) That it just lost a link and until I restarted the whole system - it just refused to accept any trafic.
You could have some kind of hardware issue on the box. You said you migrated from a virtual appliance to actual hardware. Did all the NIC drivers get updated in
config.xml?Did you start fresh or did you import the old configuration from the VM?Something may need changing if you imported from virtual hardware onto actual hardware.Certainly won't hurt anything to try Snort, though. If you do, report back on the results.
Edit: read your original post again after posting this reply and see that one of my questions was already answered.
-
@bmeeks HW is based on HPE ProLiant MicroServer Gen10
Onboard NICs used for local network are Broadcom 5720 which should use driver bge and intel NICs are Intel
82 576 which are igb so I assume that drivers are correct.System sees about 45% CPU usage at peeks and it uses about 30 % of 8GB RAM...
-
Do the instances of stopped traffic flow happen with Suricata disabled? You need to start some systematic troubleshooting by elminating variables and then slowly adding them back one-by-one to see what might be the cause.
Eliminating all packages would be the first option. Let the system run as a basic firewall and see how stable it is. Then starting adding packages back. When you get to Suricata, just activate one interface at a time. Let each one run for some period of time (maybe hours or even a day or two) to see how stable the firewall is.
-
@bmeeks I started to discus the crashing link with IS provider because it was quite weird state, both machines were sending packets but neither of them received anything. so we changed some cables did some config checks but neither of us could find anything wrong with either device because both machines worked quite happily with other HW but not with each other, everything went quiet after I noticed that in ARP table the link/mac address of gateway on WAN expired randomly which of course was followed by lost connection so I set the mac address of the gateway as permanent and the problem did not repeat since. I will wait for another week or so to confirm that this fixes the problem before I inform the ISP. But I know that he uses that gateway for large number of other customers, without problem so I doubt that he will be able to help me with anything.
I disabled Suricata and installed snort and I will be adding functionality slowly back. -
Ok, so after 5 days of running snort with same rulset as suricata without single problem I would say that suricata was a problem. So I will keep using snort as stability is more important for me.
Thank you for help!
