Suricata bug

DiskWizard

2.2-RC (amd64)
built on Tue Dec 23 05:11:07 CST 2014

Still crashing

bmeeks

@DiskWizard:

2.2-RC (amd64)
built on Tue Dec 23 05:11:07 CST 2014

Still crashing

Need some additional information such as:

(1) was this a working 2.0.4 install and just recently broke with perhaps an overnight rule update?

(2) or was this a working 2.0.3 install that broke upon the upgrade to 2.0.4?

If #1 is true, then a new or recently enabled rule is at fault. You would need to find and disable it until it is fixed by the rule author. Another possibility is you are experiencing the random LibHTP segfault bug that is reported to be fixed in the new 2.0.5 release of the Suricata binary. I am currently working on getting that new version ready for pfSense, but it will be a few more days.

Bill

bmeeks

I have tested the latest Suricata update two different ways and cannot reproduce this problem. I first performed an upgrade of the package on a current 2.2-RC VM. Next, I wiped the Suricata configuration completely and performed a clean install with no previous configuration. In both cases Suricata performed as expected.

The tests were performed on a December 23rd snapshot of pfSense 2.2-RC on a virtual machine.

Bill

DiskWizard

It is definitely working now 2.1.5 bare metal
Unfortunately I know nothing of previous versions

DiskWizard

I am ready for more instructed questions

DiskWizard

Different domains maybe ? Mine is xxxxxxxxx.xxxxxxxxx.xxxxx

bmeeks

@DiskWizard:

It is definitely working now 2.1.5 bare metal

This statement confuses me unless it is a typo. Are you saying it is working now, or did you leave out the word "not" in the statement?

When you say you know nothing about previous versions, does that none were installed, or did you inherit this firewall and don't know what might have been installed previously?

Bill

DiskWizard

Whoa ! The reports are from test system, production runs 2.1.5

DiskWizard

24/12/2014 – 20:20:19 - <error>- [ERRCODE: SC_ERR_FATAL(171)] - failed to open file: /usr/local/etc/suricata//suricata.yaml: No such file or directory

Double slash troubles ? /usr/local/etc/suricata//suricata.yaml</error>

bmeeks

@DiskWizard:

24/12/2014 – 20:20:19 - <error>- [ERRCODE: SC_ERR_FATAL(171)] - failed to open file: /usr/local/etc/suricata//suricata.yaml: No such file or directory

Double slash troubles ? /usr/local/etc/suricata//suricata.yaml</error>

There is something seriously wrong with the config on the box throwing this error. That is not even the correct path. It should be /usr/pbi/suricata_amd64/…

Have you tried totally wiping this box and reinstalling pfSense 2.2 from scratch on it using the full-install image?

Bill

DiskWizard

This is exactly what I did 4 days ago. Gonna give it another fresh install.

tcsac

I'm also seeing:
kernel: pid 22127 (suricata), uid 0: exited on signal 4 (core dumped)

When I try to start suricata.

Where exactly are you seeing this:
24/12/2014 – 20:20:19 - <error>- [ERRCODE: SC_ERR_FATAL(171)] - failed to open file: /usr/local/etc/suricata//suricata.yaml: No such file or directory</error>

DiskWizard

8 Shell

suricata -T

bmeeks

I have confirmed that on some virtual machine installs Suricata will core dump on an illegal instruction. The problem happens due, I think, to some kind of bug in the C compiler on FreeBSD 10.1. I have not confirmed this.

For you folks seeing a Suricata core dump, can you try running this from the command line and post back what you get?

suricata --build-info

Normally that line should print out a series of lines providing the build information and compiled options. If you are experiencing something else, hopefully it will print a little bit of a hint in the error message (like the "illegal instruction" message I see on some VMware virtual machines).

Bill

tcsac

This is Suricata version 2.0.4 RELEASE
Features: IPFW PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 HAVE_PACKET_FANOUT LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LIBJANSSON
SIMD support: SSE_3
Atomic intrisics: 1 2 4 8 16 byte(s)
64-bits, Little-endian architecture
GCC version 4.2.1 Compatible FreeBSD Clang 3.4 (tags/RELEASE_34/final 197956), C version 199901
compiled with -fstack-protector
compiled with _FORTIFY_SOURCE=2
L1 cache line size (CLS)=64
compiled with LibHTP v0.5.15, linked against LibHTP v0.5.15
Suricata Configuration:
AF_PACKET support: no
PF_RING support: no
NFQueue support: no
NFLOG support: no
IPFW support: yes
DAG enabled: no
Napatech enabled: no
Unix socket enabled: yes
Detection enabled: yes

libnss support: yes
libnspr support: yes
libjansson support: yes
Prelude support: no
PCRE jit: yes
LUA support: no
libluajit: no
libgeoip: yes
Non-bundled htp: no
Old barnyard2 support: no
CUDA enabled: no

Suricatasc install: no

Unit tests enabled: no
Debug output enabled: no
Debug validation enabled: no
Profiling enabled: no
Profiling locks enabled: no
Coccinelle / spatch: no

Generic build parameters:
Installation prefix (–prefix): /usr/local
Configuration directory (--sysconfdir): /usr/local/etc/suricata/
Log directory (--localstatedir) : /var/log/suricata/

Host: amd64-portbld-freebsd10.0
GCC binary: cc
GCC Protect enabled: yes
GCC march native enabled: yes
GCC Profile enabled: no

suricata -T provides the same thing:
25/12/2014 -- 10:37:22 - <error>- [ERRCODE: SC_ERR_FATAL(171)] - failed to open file: /usr/local/etc/suricata//suricata.yaml: No such file or directory</error>

bmeeks

@tcsac:

suricata -T provides the same thing:
25/12/2014 – 10:37:22 - <error>- [ERRCODE: SC_ERR_FATAL(171)] - failed to open file: /usr/local/etc/suricata//suricata.yaml: No such file or directory</error>

On pfSense, you can't just run "suricata -T" without also providing the proper YAML config file path. PBI packages on pfSense are configured to use a special path.

How did you install Suricata on this box? Was is it via System…Packages...Available Packages?

What prints in the system log when you attempt to start the Suricata service from Services…Suricata by clicking the red X icon?

~~Also, the paths for these settings are incorrect:

Installation prefix (–prefix): /usr/local
Configuration directory (--sysconfdir): /usr/local/etc/suricata/

They should read /usr/pbi/suricata_amd64/ instead of /usr/local.~~

Last edit to scratch the statements above … the paths are apparently different on 2.2 versus 2.1 (which I was comparing to). The /usr/local prefix is OK on 2.2 as that is what is showing on my currently working 2.2-RC virtual machine.

Bill

tcsac

Yes installed from system packages.

Ack #133 (Req-Sent)
Dec 25 11:59:19 php-fpm[92494]: /suricata/suricata_interfaces.php: Toggle (suricata starting) for WAN(WAN)…
Dec 25 11:59:19 php-fpm[92494]: /suricata/suricata_interfaces.php: [Suricata] Updating rules configuration for: WAN …
Dec 25 11:59:28 php-fpm[92494]: /suricata/suricata_interfaces.php: [Suricata] Enabling any flowbit-required rules for: WAN…
Dec 25 11:59:29 php-fpm[92494]: /suricata/suricata_interfaces.php: [Suricata] Building new sid-msg.map file for WAN…
Dec 25 11:59:31 suricata: 25/12/2014 -- 11:59:31 - <notice>-- This is Suricata version 2.0.4 RELEASE
Dec 25 11:59:31 barnyard2[19148]: Found pid path directive (/var/run)
Dec 25 11:59:31 barnyard2[19148]: Running in Continuous mode
Dec 25 11:59:31 barnyard2[19148]:
Dec 25 11:59:31 barnyard2[19148]: –== Initializing Barnyard2 ==--
Dec 25 11:59:31 barnyard2[19148]: Initializing Input Plugins!
Dec 25 11:59:31 barnyard2[19148]: Initializing Output Plugins!
Dec 25 11:59:31 barnyard2[19148]: Found pid path directive (/var/run)
Dec 25 11:59:31 barnyard2[19148]: +[ Signature Suppress list ]+ –--------------------------
Dec 25 11:59:31 barnyard2[19148]: +[No entry in Signature Suppress List]+
Dec 25 11:59:31 barnyard2[19148]: –-------------------------- +[ Signature Suppress list ]+
Dec 25 11:59:31 kernel: pid 18635 (suricata), uid 0: exited on signal 4 (core dumped)</notice>

bmeeks

@tcsac:

Yes installed from system packages.

Ack #133 (Req-Sent)
Dec 25 11:59:19 php-fpm[92494]: /suricata/suricata_interfaces.php: Toggle (suricata starting) for WAN(WAN)…
Dec 25 11:59:19 php-fpm[92494]: /suricata/suricata_interfaces.php: [Suricata] Updating rules configuration for: WAN …
Dec 25 11:59:28 php-fpm[92494]: /suricata/suricata_interfaces.php: [Suricata] Enabling any flowbit-required rules for: WAN…
Dec 25 11:59:29 php-fpm[92494]: /suricata/suricata_interfaces.php: [Suricata] Building new sid-msg.map file for WAN…
Dec 25 11:59:31 suricata: 25/12/2014 -- 11:59:31 - <notice>-- This is Suricata version 2.0.4 RELEASE
Dec 25 11:59:31 barnyard2[19148]: Found pid path directive (/var/run)
Dec 25 11:59:31 barnyard2[19148]: Running in Continuous mode
Dec 25 11:59:31 barnyard2[19148]:
Dec 25 11:59:31 barnyard2[19148]: –== Initializing Barnyard2 ==--
Dec 25 11:59:31 barnyard2[19148]: Initializing Input Plugins!
Dec 25 11:59:31 barnyard2[19148]: Initializing Output Plugins!
Dec 25 11:59:31 barnyard2[19148]: Found pid path directive (/var/run)
Dec 25 11:59:31 barnyard2[19148]: +[ Signature Suppress list ]+ –--------------------------
Dec 25 11:59:31 barnyard2[19148]: +[No entry in Signature Suppress List]+
Dec 25 11:59:31 barnyard2[19148]: –-------------------------- +[ Signature Suppress list ]+
Dec 25 11:59:31 kernel: pid 18635 (suricata), uid 0: exited on signal 4 (core dumped)</notice>

Have you ever installed Suricata on this firewall before? In other words, is there a existing configuration?

Can you provide the output of suricata.log from the LOGS VIEW tab?

Bill

tcsac

Before 2.0.4? Yes, I believe I installed suricata at an earlier release and let it upgrade. Tried uninstalling and re-installing but it made no difference.

25/12/2014 – 11:59:31 - <notice>-- This is Suricata version 2.0.4 RELEASE
25/12/2014 -- 11:59:31 - <info>-- CPUs/cores online: 4
25/12/2014 -- 11:59:31 - <info>-- Live rule reloads enabled
25/12/2014 -- 11:59:31 - <info>-- 'default' server has 'request-body-minimal-inspect-size' set to 33882 and 'request-body-inspect-window' set to 4053 after randomization.
25/12/2014 -- 11:59:31 - <info>-- 'default' server has 'response-body-minimal-inspect-size' set to 33695 and 'response-body-inspect-window' set to 4218 after randomization.
25/12/2014 -- 11:59:31 - <info>-- HTTP memcap: 67108864
25/12/2014 -- 11:59:31 - <info>-- DNS request flood protection level: 500
25/12/2014 -- 11:59:31 - <info>-- DNS per flow memcap (state-memcap): 524288
25/12/2014 -- 11:59:31 - <info>-- DNS global memcap: 16777216</info></info></info></info></info></info></info></info></notice>

tcsac

@bmeeks:

@tcsac:

suricata -T provides the same thing:
25/12/2014 – 10:37:22 - <error>- [ERRCODE: SC_ERR_FATAL(171)] - failed to open file: /usr/local/etc/suricata//suricata.yaml: No such file or directory</error>

On pfSense, you can't just run "suricata -T" without also providing the proper YAML config file path. PBI packages on pfSense are configured to use a special path.

How did you install Suricata on this box? Was is it via System…Packages...Available Packages?

What prints in the system log when you attempt to start the Suricata service from Services…Suricata by clicking the red X icon?

~~Also, the paths for these settings are incorrect:

Installation prefix (–prefix): /usr/local
Configuration directory (--sysconfdir): /usr/local/etc/suricata/

They should read /usr/pbi/suricata_amd64/ instead of /usr/local.~~

Last edit to scratch the statements above … the paths are apparently different on 2.2 versus 2.1 (which I was comparing to). The /usr/local prefix is OK on 2.2 as that is what is showing on my currently working 2.2-RC virtual machine.

Bill

Maybe that's part of the problem? On my box at least, /usr/local/etc/suricata/ doesn't even exist. If I search for suricata.yaml, the only file that I find is:

/usr/pbi/suricata-amd64/local/etc/suricata/suricata_23278_pppoe0/suricata.yaml

I'm guessing that's the problem…?