Suricata keeps crashing since 2.4.2 upgrade
-
Suricata will not stay running whenever I restart the service it just crashes again and I have tried reinstalling the package and such.
-
Ever since I updated to 2.4.2 I( noticed Suricata won't stay on for more than a few seconds without crashing. I had it setup and running perfectly since 2.3.4, updated to 2.3.5, 2.4.0, and 2.4.1 with no problems. I even tried backing up and restoring a config file. Only thing that's changed is I loaded the SSD and NIC cards into a new boxand had to reorder the NIC assignments em0 used to be WAN now it's em4 em1 used to be LAN now it''s em0 DMZ used to be em2 now it's em1 but everything else is the same. I tried reinstalling Suricata as well to no availin fixing my issue
-
Ever since I updated to 2.4.2 I( noticed Suricata won't stay on for more than a few seconds without crashing. I had it setup and running perfectly since 2.3.4, updated to 2.3.5, 2.4.0, and 2.4.1 with no problems. I even tried backing up and restoring a config file. Only thing that's changed is I loaded the SSD and NIC cards into a new boxand had to reorder the NIC assignments em0 used to be WAN now it's em4 em1 used to be LAN now it''s em0 DMZ used to be em2 now it's em1 but everything else is the same. I tried reinstalling Suricata as well to no availin fixing my issue
New interface names should not make it crash so long as the underlying NIC driver is essentially the same. However, changing interface names could mess up your rule assignments as what you might have on LAN is now maybe DMZ or WAN, for example.
What kind of messages are you getting in the suricata.log file? You can view that log using the LOGS VIEW tab. Also, is there anything logged in the pfSense system log?
Are you using Legacy Mode for blocking or Inline IPS Mode?
Bill
-
I am using Legacy mode.
Here is what Suricata.log is saying
http://dumptext.com/LxBvOu58Nothing really shows up in the system log that is out of the ordinary. Just says it is starting Suricata and nothing else
Dec 4 23:08:33 bandwidthd Monitoring subnet 192.168.1.0 with netmask 255.255.255.0
Dec 4 23:08:33 bandwidthd Monitoring subnet 192.168.10.0 with netmask 255.255.255.0
Dec 4 23:08:33 bandwidthd Monitoring subnet 65.60.240.0 with netmask 255.255.252.0
Dec 4 23:08:33 bandwidthd Monitoring subnet 192.168.1.0 with netmask 255.255.255.0
Dec 4 23:08:33 bandwidthd Monitoring subnet 192.168.10.0 with netmask 255.255.255.0
Dec 4 23:08:33 bandwidthd Monitoring subnet 65.60.240.0 with netmask 255.255.252.0
Dec 4 23:08:33 bandwidthd Opening em0
Dec 4 23:08:33 bandwidthd Opening em0
Dec 4 23:08:33 bandwidthd Packet Encoding: Ethernet
Dec 4 23:08:33 bandwidthd Opening em0
Dec 4 23:08:33 bandwidthd Opening em0
Dec 4 23:08:33 bandwidthd Packet Encoding: Ethernet
Dec 4 23:08:33 bandwidthd Opening em0
Dec 4 23:08:33 bandwidthd Packet Encoding: Ethernet
Dec 4 23:08:33 bandwidthd Opening em0
Dec 4 23:08:33 bandwidthd Packet Encoding: Ethernet
Dec 4 23:08:33 bandwidthd Opening em0
Dec 4 23:08:33 bandwidthd Opening em0
Dec 4 23:08:33 bandwidthd Packet Encoding: Ethernet
Dec 4 23:08:33 bandwidthd Packet Encoding: Ethernet
Dec 4 23:08:33 bandwidthd Packet Encoding: Ethernet
Dec 4 23:08:33 bandwidthd Packet Encoding: Ethernet
Dec 4 23:08:33 SuricataStartup 55863 Suricata START for WAN(45069_em4)…
Dec 4 23:08:34 SuricataStartup 60882 Suricata START for LAN(42126_em0)...
Dec 4 23:08:35 SuricataStartup 63724 Suricata START for VoIP(10756_em1)...
Dec 4 23:09:00 php-cgi servicewatchdog_cron.php: Service Watchdog detected service suricata stopped. Restarting suricata (Suricata IDS/IPS Daemon)
Dec 4 23:09:00 SuricataStartup 13977 Suricata START for WAN(45069_em4)...
Dec 4 23:09:01 SuricataStartup 15477 Suricata START for LAN(42126_em0)...
Dec 4 23:09:02 SuricataStartup 18308 Suricata START for VoIP(10756_em1)...
Dec 4 23:10:00 php-cgi servicewatchdog_cron.php: Service Watchdog detected service suricata stopped. Restarting suricata (Suricata IDS/IPS Daemon)
Dec 4 23:10:00 SuricataStartup 44977 Suricata START for WAN(45069_em4)...
Dec 4 23:10:01 SuricataStartup 50292 Suricata START for LAN(42126_em0)...
Dec 4 23:10:02 SuricataStartup 50890 Suricata START for VoIP(10756_em1)...
Dec 4 23:11:00 php-cgi servicewatchdog_cron.php: Service Watchdog detected service suricata stopped. Restarting suricata (Suricata IDS/IPS Daemon)
Dec 4 23:11:00 SuricataStartup 78995 Suricata START for WAN(45069_em4)...
Dec 4 23:11:01 SuricataStartup 80028 Suricata START for LAN(42126_em0)...
Dec 4 23:11:02 SuricataStartup 80695 Suricata START for VoIP(10756_em1)...
Dec 4 23:12:00 php-cgi servicewatchdog_cron.php: Service Watchdog detected service suricata stopped. Restarting suricata (Suricata IDS/IPS Daemon)
Dec 4 23:12:00 SuricataStartup 6914 Suricata START for WAN(45069_em4)...
Dec 4 23:12:01 SuricataStartup 8617 Suricata START for LAN(42126_em0)...
Dec 4 23:12:02 SuricataStartup 10044 Suricata START for VoIP(10756_em1)...
Dec 4 23:12:58 SuricataStartup 73963 Suricata START for WAN(45069_em4)...
Dec 4 23:12:59 SuricataStartup 80831 Suricata START for LAN(42126_em0)...
Dec 4 23:13:00 php-cgi servicewatchdog_cron.php: Service Watchdog detected service suricata stopped. Restarting suricata (Suricata IDS/IPS Daemon)
Dec 4 23:13:00 SuricataStartup 82613 Ignoring additional START command since Suricata is already starting...
Dec 4 23:13:00 SuricataStartup 84841 Suricata START for VoIP(10756_em1)...
Dec 4 23:14:00 php-cgi servicewatchdog_cron.php: Service Watchdog detected service suricata stopped. Restarting suricata (Suricata IDS/IPS Daemon)
Dec 4 23:14:00 SuricataStartup 33805 Suricata START for WAN(45069_em4)...
Dec 4 23:14:01 SuricataStartup 34953 Suricata START for LAN(42126_em0)...
Dec 4 23:14:02 SuricataStartup 35531 Suricata START for VoIP(10756_em1)... -
I am using Legacy mode.
Here is what Suricata.log is saying
http://dumptext.com/LxBvOu58Nothing really shows up in the system log that is out of the ordinary. Just says it is starting Suricata and nothing else
Dec 4 23:08:33 bandwidthd Monitoring subnet 192.168.1.0 with netmask 255.255.255.0
Dec 4 23:08:33 bandwidthd Monitoring subnet 192.168.10.0 with netmask 255.255.255.0
Dec 4 23:08:33 bandwidthd Monitoring subnet 65.60.240.0 with netmask 255.255.252.0
Dec 4 23:08:33 bandwidthd Monitoring subnet 192.168.1.0 with netmask 255.255.255.0
Dec 4 23:08:33 bandwidthd Monitoring subnet 192.168.10.0 with netmask 255.255.255.0
Dec 4 23:08:33 bandwidthd Monitoring subnet 65.60.240.0 with netmask 255.255.252.0
Dec 4 23:08:33 bandwidthd Opening em0
Dec 4 23:08:33 bandwidthd Opening em0
Dec 4 23:08:33 bandwidthd Packet Encoding: Ethernet
Dec 4 23:08:33 bandwidthd Opening em0
Dec 4 23:08:33 bandwidthd Opening em0
Dec 4 23:08:33 bandwidthd Packet Encoding: Ethernet
Dec 4 23:08:33 bandwidthd Opening em0
Dec 4 23:08:33 bandwidthd Packet Encoding: Ethernet
Dec 4 23:08:33 bandwidthd Opening em0
Dec 4 23:08:33 bandwidthd Packet Encoding: Ethernet
Dec 4 23:08:33 bandwidthd Opening em0
Dec 4 23:08:33 bandwidthd Opening em0
Dec 4 23:08:33 bandwidthd Packet Encoding: Ethernet
Dec 4 23:08:33 bandwidthd Packet Encoding: Ethernet
Dec 4 23:08:33 bandwidthd Packet Encoding: Ethernet
Dec 4 23:08:33 bandwidthd Packet Encoding: Ethernet
Dec 4 23:08:33 SuricataStartup 55863 Suricata START for WAN(45069_em4)…
Dec 4 23:08:34 SuricataStartup 60882 Suricata START for LAN(42126_em0)...
Dec 4 23:08:35 SuricataStartup 63724 Suricata START for VoIP(10756_em1)...
Dec 4 23:09:00 php-cgi servicewatchdog_cron.php: Service Watchdog detected service suricata stopped. Restarting suricata (Suricata IDS/IPS Daemon)
Dec 4 23:09:00 SuricataStartup 13977 Suricata START for WAN(45069_em4)...
Dec 4 23:09:01 SuricataStartup 15477 Suricata START for LAN(42126_em0)...
Dec 4 23:09:02 SuricataStartup 18308 Suricata START for VoIP(10756_em1)...
Dec 4 23:10:00 php-cgi servicewatchdog_cron.php: Service Watchdog detected service suricata stopped. Restarting suricata (Suricata IDS/IPS Daemon)
Dec 4 23:10:00 SuricataStartup 44977 Suricata START for WAN(45069_em4)...
Dec 4 23:10:01 SuricataStartup 50292 Suricata START for LAN(42126_em0)...
Dec 4 23:10:02 SuricataStartup 50890 Suricata START for VoIP(10756_em1)...
Dec 4 23:11:00 php-cgi servicewatchdog_cron.php: Service Watchdog detected service suricata stopped. Restarting suricata (Suricata IDS/IPS Daemon)
Dec 4 23:11:00 SuricataStartup 78995 Suricata START for WAN(45069_em4)...
Dec 4 23:11:01 SuricataStartup 80028 Suricata START for LAN(42126_em0)...
Dec 4 23:11:02 SuricataStartup 80695 Suricata START for VoIP(10756_em1)...
Dec 4 23:12:00 php-cgi servicewatchdog_cron.php: Service Watchdog detected service suricata stopped. Restarting suricata (Suricata IDS/IPS Daemon)
Dec 4 23:12:00 SuricataStartup 6914 Suricata START for WAN(45069_em4)...
Dec 4 23:12:01 SuricataStartup 8617 Suricata START for LAN(42126_em0)...
Dec 4 23:12:02 SuricataStartup 10044 Suricata START for VoIP(10756_em1)...
Dec 4 23:12:58 SuricataStartup 73963 Suricata START for WAN(45069_em4)...
Dec 4 23:12:59 SuricataStartup 80831 Suricata START for LAN(42126_em0)...
Dec 4 23:13:00 php-cgi servicewatchdog_cron.php: Service Watchdog detected service suricata stopped. Restarting suricata (Suricata IDS/IPS Daemon)
Dec 4 23:13:00 SuricataStartup 82613 Ignoring additional START command since Suricata is already starting...
Dec 4 23:13:00 SuricataStartup 84841 Suricata START for VoIP(10756_em1)...
Dec 4 23:14:00 php-cgi servicewatchdog_cron.php: Service Watchdog detected service suricata stopped. Restarting suricata (Suricata IDS/IPS Daemon)
Dec 4 23:14:00 SuricataStartup 33805 Suricata START for WAN(45069_em4)...
Dec 4 23:14:01 SuricataStartup 34953 Suricata START for LAN(42126_em0)...
Dec 4 23:14:02 SuricataStartup 35531 Suricata START for VoIP(10756_em1)...Do not run the Service Watchdog package against Suricata or Snort. It can cause crashing. The Watchdog package does not properly account for the multiple Suricata instances (one running process per configured interface). It also does not understand that Suricata stops and restarts itself as part of rule updates and such. Remove Suricata from the Service Watchdog list and I bet it will work for you. From your logs you can see that the Watchdog package is sending a START command to Suricata even while Suricata is already starting up. Multiple start commands on the same interface spells trouble.
Edit: I failed to notice another error in your suricata.log until later, so posting this edit with additional info. This error is why your startup is failing:
1/12/2017 -- 23:08:59 - <error>-- [ERRCODE: SC_ERR_POOL_INIT(66)] - alloc error 1/12/2017 -- 23:08:59 - <error>-- [ERRCODE: SC_ERR_POOL_INIT(66)] - pool grow failed 1/12/2017 -- 23:08:59 - <error>-- [ERRCODE: SC_ERR_MEM_ALLOC(1)] - failed to setup/expand stream session pool. Expand stream.memcap?</error></error></error>
This suggested fix is in the error message. Expand the size of your Stream Memcap. I believe the default is either 32 MB or 64 MB. Users with higher CPU core counts (or hyperthreading) frequently need 128 MB or even 256 MB of stream.memcap.
Bill
-
So I followed your advice, increased the memcap to 256MB ( 268,435,456 ) and it also had an update to 4.0.1_1 so now it is working as it should be. Thank you!
-
So I followed your advice, increased the memcap to 256MB ( 268,435,456 ) and it also had an update to 4.0.1_1 so now it is working as it should be. Thank you!
You are welcome. I saw in your suricata.log file that your CPU has 8 cores. That's why a significant increase in Stream Memory is needed. I think there were also some changes to that part of the Suricata binary from upstream back when the 4.0 series was released.
I still recommend strongly that you do not use Service Watchdog with Suricata (or Snort, for you Snort users reading this thread).
Bill
-
same problem here. sorry I just posted under beta forum for 2.4 i followed all advise from Bill.
-
same problem here. sorry I just posted under beta forum for 2.4 i followed all advise from Bill.
If boosting your stream memcap value did not help, post the output of the suricata.log file. You can view under LOGS VIEW within the package GUI. Any error will be in that file.
Bill
-
Crash report begins. Anonymous machine information:
amd64
11.1-RELEASE-p6
FreeBSD 11.1-RELEASE-p6 #421 r313908+a5b33c9d1c4(RELENG_2_4): Tue Dec 12 09:20:59 CST 2017 root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSenseCrash report details:
PHP Errors:
[13-Dec-2017 16:03:42 America/Los_Angeles] PHP Warning: filesize(): stat failed for /usr/local/etc/suricata/suricata_18353_em0/rules/suricata.rules in /usr/local/pkg/suricata/suricata_generate_yaml.php on line 855
[13-Dec-2017 16:03:42 America/Los_Angeles] PHP Warning: filesize(): stat failed for /usr/local/etc/suricata/suricata_18353_em0/rules/flowbit-required.rules in /usr/local/pkg/suricata/suricata_generate_yaml.php on line 857
[13-Dec-2017 16:03:42 America/Los_Angeles] PHP Warning: filesize(): stat failed for /usr/local/etc/suricata/suricata_18353_em0/rules/custom.rules in /usr/local/pkg/suricata/suricata_generate_yaml.php on line 859
[13-Dec-2017 16:03:42 America/Los_Angeles] PHP Warning: filesize(): stat failed for /usr/local/etc/suricata/suricata_57646_em1/rules/suricata.rules in /usr/local/pkg/suricata/suricata_generate_yaml.php on line 855
[13-Dec-2017 16:03:42 America/Los_Angeles] PHP Warning: filesize(): stat failed for /usr/local/etc/suricata/suricata_57646_em1/rules/flowbit-required.rules in /usr/local/pkg/suricata/suricata_generate_yaml.php on line 857
[13-Dec-2017 16:03:42 America/Los_Angeles] PHP Warning: filesize(): stat failed for /usr/local/etc/suricata/suricata_57646_em1/rules/custom.rules in /usr/local/pkg/suricata/suricata_generate_yaml.php on line 859Filename: /var/crash/minfree
2048this happens after I reinstall the whole package
-
Crash report begins. Anonymous machine information:
amd64
11.1-RELEASE-p6
FreeBSD 11.1-RELEASE-p6 #421 r313908+a5b33c9d1c4(RELENG_2_4): Tue Dec 12 09:20:59 CST 2017 root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSenseCrash report details:
PHP Errors:
[13-Dec-2017 16:03:42 America/Los_Angeles] PHP Warning: filesize(): stat failed for /usr/local/etc/suricata/suricata_18353_em0/rules/suricata.rules in /usr/local/pkg/suricata/suricata_generate_yaml.php on line 855
[13-Dec-2017 16:03:42 America/Los_Angeles] PHP Warning: filesize(): stat failed for /usr/local/etc/suricata/suricata_18353_em0/rules/flowbit-required.rules in /usr/local/pkg/suricata/suricata_generate_yaml.php on line 857
[13-Dec-2017 16:03:42 America/Los_Angeles] PHP Warning: filesize(): stat failed for /usr/local/etc/suricata/suricata_18353_em0/rules/custom.rules in /usr/local/pkg/suricata/suricata_generate_yaml.php on line 859
[13-Dec-2017 16:03:42 America/Los_Angeles] PHP Warning: filesize(): stat failed for /usr/local/etc/suricata/suricata_57646_em1/rules/suricata.rules in /usr/local/pkg/suricata/suricata_generate_yaml.php on line 855
[13-Dec-2017 16:03:42 America/Los_Angeles] PHP Warning: filesize(): stat failed for /usr/local/etc/suricata/suricata_57646_em1/rules/flowbit-required.rules in /usr/local/pkg/suricata/suricata_generate_yaml.php on line 857
[13-Dec-2017 16:03:42 America/Los_Angeles] PHP Warning: filesize(): stat failed for /usr/local/etc/suricata/suricata_57646_em1/rules/custom.rules in /usr/local/pkg/suricata/suricata_generate_yaml.php on line 859Filename: /var/crash/minfree
2048this happens after I reinstall the whole package
What type of hardware is this? Those errors indicate problems within the file system. Another possibility, if you have recently upgraded your hardware and imported an old config, is the interface names have changed (the em1 part of the error path). So for example if your NIC driver is now say igb1 instead of em1, then you will get this error. To fix it you will need to either delete the interface and recreate it from scratch, or manually go into your config.xml file and change all the instances of the strings "em0" and "em1" to match whatever the new name is for your physical interfaces.