Suricata service stops/wont restart if blocking mode enabled (Solved)

threatsense · Dec 22, 2018, 7:38 PM

Hi,

Problem:

Suricata service stopping on an interface only when blocking mode is enabled.

Version info:

pfsense version: 2.4.4-RELEASE-p1
Installed package: suricata security 4.0.13_11 (package pependency: suricata-4.0.6)
Hardware: Dell R210ii server

Background info:
Suricata runs fine when in default alerting only mode (albeit some errors in the logs shown below which cause no noticeable issues shown at the end of this post for those interested).

Cause:
If I enable blocking mode on any single interface by making the following configuration change:

Block Offenders (Checking this option will automatically block hosts that generate a Suricata alert)
Legacy mode

The service stops running soon after with one error in the suricata.log file for the interface and it will not restart (unless the .pid file is removed) and will die immediately again after :

suricata.log (when blocking enabled on interface bce0.111 and service has stopped by itself)

21/12/2018 -- 12:49:52 - <Info> -- alert-pf -> Creating automatic firewall interface IP address Pass List.
21/12/2018 -- 12:49:52 - <Info> -- alert-pf -> adding firewall interface bce0 IPv6 address fe80:0000:0000:0000:d267:e5ff:fee9:ba18 to automatic interface IP Pass List.
21/12/2018 -- 12:49:52 - <Info> -- alert-pf -> adding firewall interface bce1 IPv6 address fe80:0000:0000:0000:d267:e5ff:fee9:ba19 to automatic interface IP Pass List.
21/12/2018 -- 12:49:52 - <Info> -- alert-pf -> adding firewall interface lo0 IPv6 address 0000:0000:0000:0000:0000:0000:0000:0001 to automatic interface IP Pass List.
21/12/2018 -- 12:49:52 - <Info> -- alert-pf -> adding firewall interface lo0 IPv6 address fe80:0000:0000:0000:0000:0000:0000:0001 to automatic interface IP Pass List.
21/12/2018 -- 12:49:52 - <Info> -- alert-pf -> adding firewall interface lo0 IPv4 address 127.0.0.1 to automatic interface IP Pass List.
21/12/2018 -- 12:49:52 - <Info> -- alert-pf -> adding firewall interface bce0.1 IPv6 address fe80:0000:0000:0000:d267:e5ff:fee9:ba18 to automatic interface IP Pass List.
21/12/2018 -- 12:49:52 - <Info> -- alert-pf -> adding firewall interface bce0.1 IPv4 address 192.168.1.1 to automatic interface IP Pass List.
21/12/2018 -- 12:49:52 - <Info> -- alert-pf -> adding firewall interface bce0.2 IPv6 address fe80:0000:0000:0000:d267:e5ff:fee9:ba18 to automatic interface IP Pass List.
21/12/2018 -- 12:49:52 - <Info> -- alert-pf -> adding firewall interface bce0.2 IPv4 address 192.168.2.1 to automatic interface IP Pass List.
21/12/2018 -- 12:49:52 - <Info> -- alert-pf -> adding firewall interface bce0.11 IPv6 address fe80:0000:0000:0000:d267:e5ff:fee9:ba18 to automatic interface IP Pass List.
21/12/2018 -- 12:49:52 - <Info> -- alert-pf -> adding firewall interface bce0.11 IPv4 address 192.168.11.1 to automatic interface IP Pass List.
21/12/2018 -- 12:49:52 - <Info> -- alert-pf -> adding firewall interface bce0.12 IPv6 address fe80:0000:0000:0000:d267:e5ff:fee9:ba18 to automatic interface IP Pass List.
21/12/2018 -- 12:49:52 - <Info> -- alert-pf -> adding firewall interface bce0.12 IPv4 address 192.168.12.1 to automatic interface IP Pass List.
21/12/2018 -- 12:49:52 - <Info> -- alert-pf -> adding firewall interface bce0.99 IPv6 address fe80:0000:0000:0000:d267:e5ff:fee9:ba18 to automatic interface IP Pass List.
21/12/2018 -- 12:49:52 - <Info> -- alert-pf -> adding firewall interface bce0.99 IPv4 address 192.168.99.1 to automatic interface IP Pass List.
21/12/2018 -- 12:49:52 - <Info> -- alert-pf -> adding firewall interface bce0.111 IPv6 address fe80:0000:0000:0000:d267:e5ff:fee9:ba18 to automatic interface IP Pass List.
21/12/2018 -- 12:49:52 - <Info> -- alert-pf -> adding firewall interface bce0.111 IPv4 address 10.0.111.1 to automatic interface IP Pass List.
21/12/2018 -- 12:49:52 - <Info> -- alert-pf -> adding firewall interface bce0.199 IPv6 address fe80:0000:0000:0000:d267:e5ff:fee9:ba18 to automatic interface IP Pass List.
21/12/2018 -- 12:49:52 - <Info> -- alert-pf -> adding firewall interface bce0.199 IPv4 address 10.0.199.1 to automatic interface IP Pass List.
21/12/2018 -- 12:49:52 - <Info> -- alert-pf -> adding firewall interface bce0.1611 IPv6 address fe80:0000:0000:0000:d267:e5ff:fee9:ba18 to automatic interface IP Pass List.
21/12/2018 -- 12:49:52 - <Info> -- alert-pf -> adding firewall interface bce0.1611 IPv4 address 172.16.11.1 to automatic interface IP Pass List.
21/12/2018 -- 12:49:52 - <Info> -- alert-pf -> adding firewall interface bce0.1699 IPv6 address fe80:0000:0000:0000:d267:e5ff:fee9:ba18 to automatic interface IP Pass List.
21/12/2018 -- 12:49:52 - <Info> -- alert-pf -> adding firewall interface bce0.1699 IPv4 address 172.16.99.1 to automatic interface IP Pass List.
21/12/2018 -- 12:49:52 - <Info> -- alert-pf -> adding firewall interface bce0.1699 IPv6 address 2a02:8010:61c6:1699:0000:0000:0000:0000 to automatic interface IP Pass List.
21/12/2018 -- 12:49:52 - <Info> -- alert-pf -> adding firewall interface bce0.102 IPv6 address fe80:0000:0000:0000:d267:e5ff:fee9:ba18 to automatic interface IP Pass List.
21/12/2018 -- 12:49:52 - <Info> -- alert-pf -> adding firewall interface bce0.102 IPv4 address 10.0.102.1 to automatic interface IP Pass List.
21/12/2018 -- 12:49:52 - <Info> -- alert-pf -> adding firewall interface pppoe1 IPv6 address fe80:0000:0000:0000:0000:0000:0000:0001 to automatic interface IP Pass List.
21/12/2018 -- 12:49:52 - <Info> -- alert-pf -> adding firewall interface pppoe1 IPv4 address 11.11.111.11 to automatic interface IP Pass List.
21/12/2018 -- 12:49:52 - <Info> -- alert-pf output device (regular) initialized: block.log
21/12/2018 -- 12:49:52 - <Error> -- [ERRCODE: SC_ERR_INVALID_ARGUMENTS(52)] - prefix or user NULL

I have googled but the only post I found with advice talk about bad characters in the suppress list etc. However, the suppress list for this interface is empty in its default default state, as are the block and pass lists.

Has anyone else overcome this issue or is able to help me investigate further to work out what this vague error means?

Many thanks.

Appendix
suricata.log (when blocking not enabled and service runs fine):

21/12/2018 -- 05:32:16 - <Notice> -- This is Suricata version 4.0.6 RELEASE
21/12/2018 -- 05:32:16 - <Info> -- CPUs/cores online: 8
21/12/2018 -- 05:32:16 - <Info> -- HTTP memcap: 67108864
21/12/2018 -- 05:32:16 - <Notice> -- using flow hash instead of active packets
21/12/2018 -- 05:32:44 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - previous keyword has a fast_pattern:only; set. Can't have relative keywords around a fast_pattern only content
21/12/2018 -- 05:32:44 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS (msg:"MALWARE-CNC Win.Trojan.Kazy variant outbound connection"; flow:to_server,established; content:".exe HTTP/1.1|0D 0A|User-Agent: Mozilla/"; fast_pattern:only; content:"|3B| MSIE "; http_header; content:!"Accept"; http_header; content:"|29 0D 0A|Host: "; distance:0; http_header; pcre:"/^GET\x20\x2f[a-z]{1,12}\.exe\x20HTTP\x2f1\.1\r\nUser\x2dAgent\x3a\x20Mozilla\x2f[\x20-\x7e]{10,100}\)\r\nHost\x3a\x20[a-z0-9\x2e\x2d]{6,32}\r\nConnection\x3a\x20Keep\x2dAlive\r\n\r\n$/"; metadata:impact_flag red, policy security-ips drop, ruleset community, service http; reference:url,www.virustotal.com/en/file/a064a1d3d8b9d8ab649686b7fb01e0631e569412388084f5c391722c98660763/analysis/; classtype:trojan-activity; sid:28406; rev:1;)" from file /usr/local/etc/suricata/suricata_49599_bce0.111/rules/suricata.rules at line 18557
21/12/2018 -- 05:32:53 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - "http_client_body" keyword seen with a sticky buffer still set.  Reset sticky buffer with pkt_data before using the modifier.
21/12/2018 -- 05:32:53 - <Error> -- [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - error parsing signature "alert tcp $HOME_NET any -> $EXTERNAL_NET $HTTP_PORTS (msg:"MALWARE-CNC Unix.Trojan.Erebus variant outbound connection"; flow:to_server,established; file_data; urilen:1; content:"h="; depth:2; http_client_body; content:"&v="; within:3; distance:8; http_client_body; content:"&k="; within:3; distance:3; fast_pattern; http_client_body; content:"Expect|3A| 100-continue|0D 0A 0D 0A|"; http_header; content:!"User-Agent"; http_header; metadata:impact_flag red, policy balanced-ips alert, policy security-ips alert, service http; reference:url,virustotal.com/en/file/0b7996bca486575be15e68dba7cbd802b1e5f90436ba23f802da66292c8a055f/analysis/; classtype:trojan-activity; sid:43351; rev:1;)" from file /usr/local/etc/suricata/suricata_49599_bce0.111/rules/suricata.rules at line 28549
21/12/2018 -- 05:32:57 - <Info> -- 2 rule files processed. 32391 rules successfully loaded, 2 rules failed
21/12/2018 -- 05:32:57 - <Info> -- Threshold config parsed: 0 rule(s) found
21/12/2018 -- 05:32:57 - <Info> -- 32395 signatures processed. 1194 are IP-only rules, 7599 are inspecting packet payload, 19015 inspect application layer, 102 are decoder event only
21/12/2018 -- 05:33:10 - <Info> -- fast output device (regular) initialized: alerts.log
21/12/2018 -- 05:33:10 - <Info> -- http-log output device (regular) initialized: http.log
21/12/2018 -- 05:33:10 - <Info> -- dns-log output device (regular) initialized: dns.log
21/12/2018 -- 05:33:10 - <Info> -- dns-log output device (regular) initialized: dns.log
21/12/2018 -- 05:33:10 - <Info> -- Using 1 live device(s).
21/12/2018 -- 05:33:10 - <Info> -- using interface bce0.111
21/12/2018 -- 05:33:10 - <Info> -- Running in 'auto' checksum mode. Detection of interface state will require 1000 packets.
21/12/2018 -- 05:33:10 - <Info> -- Set snaplen to 1518 for 'bce0.111'
21/12/2018 -- 05:33:10 - <Info> -- RunModeIdsPcapAutoFp initialised
21/12/2018 -- 05:33:10 - <Notice> -- all 9 packet processing threads, 2 management threads initialized, engine started.

boobletins · Dec 21, 2018, 5:55 PM

@threatsense said in Suricata service stops/wont restart if blocking mode enabled:

prefix or user NULL

So here's where the error originates:
https://github.com/OISF/suricata/blob/033e756905d134cd085eb67673da95a7f16dba1d/src/util-radix-tree.c#L174-L198

But without more reading than I have time for right now I can't tell you why that would happen in IPS Legacy mode but not IDS mode.

bmeeks · Dec 21, 2018, 8:43 PM

Mostly likely you have either an alias or some IP address on the local firewall that is resolving to NULL. When blocking mode is enabled, the custom blocking plugin within Suricata is active. That plugin uses a Radix Tree object in the Suricata binary to hold the list of IP addresses and subnets found in the Pass List assigned for the interface. When blocking mode is turned off, that custom plugin is not loaded and thus the Radix Tree insertion code is not called.

You need to carefully examine any aliases you may have defined for a Pass List and see if all of those are resolving correctly.

threatsense · Dec 21, 2018, 9:20 PM

Thanks for the responses so far, they may have helped get it working.

I did not have any pass list defined, I checked under Suricata / Pass Lists and it was empty.

In the logs included in my original post they show the entries added to the automatic pass list. Ive included the entries specific to the interface that I enabled blocking mode on (bce0.111) below, as you can see there are two entries which look fine:

21/12/2018 -- 12:49:52 - <Info> -- alert-pf -> Creating automatic firewall interface IP address Pass List.
...
21/12/2018 -- 12:49:52 - <Info> -- alert-pf -> adding firewall interface bce0.111 IPv6 address fe80:0000:0000:0000:d267:e5ff:fee9:ba18 to automatic interface IP Pass List.
21/12/2018 -- 12:49:52 - <Info> -- alert-pf -> adding firewall interface bce0.111 IPv4 address 10.0.111.1 to automatic interface IP Pass List.

However, from the shell when I looked at the passlist for the interface, it contained all Internal interface addresses, one of these was an IPv6 shortened address configured on another interface (not configured with blocking) shown below (IPv6 address masked with 'x'):

cat /usr/local/etc/suricata/suricata_49599_bce0.111/passlist

2x02:8xx0:6xx6:1xx9::/64
...
...
All the other IPv4 addresses

When I Removed the static IPv6 allocation from the other interface, and restarted Suricata on the interface I want blocking on, it started up successfully and the IPv6 entry was removed from the passlist file.

It is early days but the logs look OK so hopefully it will keep running.

My next challenge it to work out how to remove some of the interface addresses from the automatic pass lists as some of these are less trusted so I would want them to be blocked...

Merry Christmas.

bmeeks · Dec 21, 2018, 9:43 PM

You can create your own custom Pass List on the PASS LISTS tab and turn off the automatic inclusion of all locally-attached firewall networks. Then you would need to create an alias within pfSense and assign that alias to your custom Pass List (down at the bottom in the Address textbox). Just be aware that doing it this way requires great care or else you can block a lot of your local LAN network hosts. That can cause quite a headache. Generally speaking you should not alter the default pass list that is applied to an interface. You need to really understand the potential ramifications when you choose to not use the default Pass List and instead generate your own (especially if you are disabling the auto-inclusion of locally-attached networks).

threatsense · Dec 21, 2018, 9:47 PM

Great thanks.

I certainly understand the ramifications!

However my network is designed into segments, for example there is a DMZ, this has an internal addressing scheme. If this zone started poking around other internal zones it would mean there is a breach so I'd want it blocked.