Upgrade Suricata 4.0.3

bmeeks

Hi Guys,

I noticed the new version for Suricata in package manager, so I clicked on update, but the update failed with an error for some missing files, so had to remove the package and reinstall it again.

also I noticed this log:
12/1/2018 – 18:57:24 - <info>-- Invalid IP() parameter provided in Pass List, skipping...

i'm not sure where () came from but the list is an Alias list, and I didn't have that log in Suricata 4.0.1_1

Best Regards</info>

I suspect this error message might be the result of fixing one of the bugs. There was a logic flaw in how HOME_NET was populated in some circumstances. That warning about the slash means you have an alias or an interface IP that is resolving to "nothing" at run time when written to the configuration. The single slash is what would normally be part of a CIDR network string such as "192.168.1.0/24".

Bill

NRgia

Thank you for investigating.

If I don't use PCRE keyword in any of the sid.conf why I am getting this?

12/1/2018 – 22:24:01 - <error>-- [ERRCODE: SC_ERR_PCRE_PARSE(7)] - parse error, ret -1, string 1,=,0x05,6,relative,bitmask 0x14

Or is not related to sid.conf parsing?

Thanks</error>

bmeeks

@NRgia:

Thank you for investigating.

If I don't use PCRE keyword in any of the sid.conf why I am getting this?

12/1/2018 – 22:24:01 - <error>-- [ERRCODE: SC_ERR_PCRE_PARSE(7)] - parse error, ret -1, string 1,=,0x05,6,relative,bitmask 0x14

Or is not related to sid.conf parsing?

Thanks</error>

That is a Suricata binary error code, and from the looks of the text I'm guessing it is choking on some rule text. This error would not be from the sid.conf parsing as only the GUI does that work and the GUI can't write to the suricata.log file. It can only write to the pfSense system log. From the format of the text you posted, it appears to have been pulled from the suricata.log file for the interface.

Bill

micropone

I got an different issue

![suracata failed.png](/public/imported_attachments/1/suracata failed.png)
![suracata failed.png_thumb](/public/imported_attachments/1/suracata failed.png_thumb)

bmeeks

@micropone:

I got an different issue

This is from a problem with the pkg utility itself that is used to install packages on pfSense. I have never seen this error in my testing using pfSense virtual machines. Do you have /var on a RAMDISK by chance?

You can also try to remove the package and then install it again. I believe that uses a different set of routines within the pkg utiilty.

Bill

NRgia

@bmeeks:

@NRgia:

Thank you for investigating.

If I don't use PCRE keyword in any of the sid.conf why I am getting this?

12/1/2018 – 22:24:01 - <error>-- [ERRCODE: SC_ERR_PCRE_PARSE(7)] - parse error, ret -1, string 1,=,0x05,6,relative,bitmask 0x14

Or is not related to sid.conf parsing?

Thanks</error>

That is a Suricata binary error code, and from the looks of the text I'm guessing it is choking on some rule text. This error would not be from the sid.conf parsing as only the GUI does that work and the GUI can't write to the suricata.log file. It can only write to the pfSense system log. From the format of the text you posted, it appears to have been pulled from the suricata.log file for the interface.

Bill

You're correct, the log is from /var/log/suricata/suricata_interface_name/suricata.log , and I think it's about the rules that are not parsed correctly, but I did not saw this error with previous version, and the selected rules are almost the same(taking in account the updates):

12/1/2018 – 20:16:57 - <error>-- [ERRCODE: SC_ERR_PCRE_PARSE(7)] - parse error, ret -1, string 1,=,0x05,6,relative,bitmask 0x14

I attached an unaltered log if you have time to look over it.

If you say it's nothing to worry about, then I can rest easy.

Thank you for your time.

OP @The Sky Heart , sorry for hijacking the thread, did you solve your issue?

suricata.log.txt</error>

bmeeks

@NRgia:

@bmeeks:

@NRgia:

Thank you for investigating.

If I don't use PCRE keyword in any of the sid.conf why I am getting this?

12/1/2018 – 22:24:01 - <error>-- [ERRCODE: SC_ERR_PCRE_PARSE(7)] - parse error, ret -1, string 1,=,0x05,6,relative,bitmask 0x14

Or is not related to sid.conf parsing?

Thanks</error>

That is a Suricata binary error code, and from the looks of the text I'm guessing it is choking on some rule text. This error would not be from the sid.conf parsing as only the GUI does that work and the GUI can't write to the suricata.log file. It can only write to the pfSense system log. From the format of the text you posted, it appears to have been pulled from the suricata.log file for the interface.

Bill

You're correct, the log is from /var/log/suricata/suricata_interface_name/suricata.log , and I think it's about the rules that are not parsed correctly, but I did not saw this error with previous version, and the selected rules are almost the same(taking in account the updates):

12/1/2018 – 20:16:57 - <error>-- [ERRCODE: SC_ERR_PCRE_PARSE(7)] - parse error, ret -1, string 1,=,0x05,6,relative,bitmask 0x14

I attached an unaltered log if you have time to look over it.

If you say it's nothing to worry about, then I can rest easy.

Thank you for your time.

OP @The Sky Heart , sorry for hijacking the thread, did you solve your issue?</error>

The error message does not give the SID of the rule where the PCRE parsing failed, so troubleshooting that is a challenge. However, remember that each update of the Suricata binary means some things changed in the source code. There are two parts of the Suricata package. The main piece is the command-line driven binary. That code is wholly and totally supported by the upstream team at Suricata.org. The secondary piece of the Suricata packge on pfSense is a GUI program that interacts with the user. The GUI code ultimately just winds up writing the suricata.yaml file used to send configuration information to the binary.

So why am I going into this detail? It is to make sure users understand that Suricata, and Snort, along with most every other package on pfSense, has an underlying binary piece that does the real work; and a GUI piece used to gather configuration input and pass it to the underlying binary. Sometimes, based on questions and bug reports I see, it seems users conflate the two pieces of a package and make them into one. For example, I can't do anything about issues within the Suricata binary itself. This particular error is one of those. It is a problem in the binary that probably got introduced in version 4.0.3 of the binary. Previously we were running the older 4.0.1 version of the Suricata binary. Sometimes changes in the binary require changes in the GUI, though. The other error posted about an invalid integer value for "stream.max-sessions" is one of those. The newer 4.0.3 binary no longer wants to see that parameter, but the older version did still want to see it.

Bill

NRgia

Actually this is the idea behind pfSense and GUI packages. The user can do the config by CLI also, but it's more work.

IMHO in SDLC when you test integrated pieces of code you don't always know(from returned errors) were the root cause is, if you're not the one who maintained the GUI or the binary. It could be with a bad config from my part, or maybe the config that is passed to the binary is misinterpreted, due to code change in GUI or binary.

I consider you responded my question, by pointing which of the pieces is the culprit.

For binary issues I can take it on the Suricata redmine, or with the maintainer on Fresh Ports.

Thank you

bmeeks

@NRgia:

I consider you responded my question, by pointing which of the pieces is the culprit.

For binary issues I can take it on the Suricata redmine, or with the maintainer on Fresh Ports.

Thank you

Yes, my opinion is the binary is the cause of your error message. I recommend posting a bug report on Suricata redmine as the Fresh Ports maintainer is really just using the code from Suricata upstream with no changes. If it is a binary bug, it needs to be reported and fixed by the actual Suricata team. I guess there is an outside chance it's a problem with the syntax of some rule, but since the GID:SID is not provided in the error it will be hard to locate the offending rule. Maybe you could try "grep" with some of the content text from the error and locate the rule ???. You can find the actual file of enabled rules here:

/usr/local/etc/suricata/suricata__interface_/rules/suricata.rules

where "interface" will be the physical interface name on your firewall plus a UUID random number.

Bill

drewsaur

Will the update GUI package be coming out anytime soon? I am still showing that 4.0.1_1 is current for my box.

fgro

Well folks, i am a bit disappointing with the current release of suricata 4.0.3 due it has a performance issue since upgrading.

*green shows idle, *yellow shows user

Upgrade to recent version / snort with barnyard2 was enabled
shutdown of snort barnyard2 & reenabled it
shutdown of snort at all and reenabled barnyard2 within suricata package on only 2 interfaces as it would rise up to 99% load at 4 nics

CPU: 47.5% user, 0.0% nice, 16.8% system, 0.2% interrupt, 35.5% idle
Mem: 1608M Active, 1758M Inact, 1011M Wired, 675M Buf, 3488M Free
Swap: 16G Total, 16G Free

PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
32784 root 1 102 0 561M 489M CPU1 1 11:12 88.10% barnyard2
36442 root 1 102 0 555M 483M CPU2 2 11:11 85.92% barnyard2

so is there a way rolling back to previous suricata version?

all it seems to be a barnyard2 issue with the performance… due suricata itself runs at low load. but after deinstalling suricata and only running snort barnyard2 it opt out the load issue with barnyard2. :-(

fgro

@fgro:

Well folks, i am a bit disappointing with the current release of suricata 4.0.3 due it has a performance issue since upgrading.

*green shows idle, *yellow shows user

Upgrade to recent version / snort with barnyard2 was enabled

shutdown of snort barnyard2 & reenabled it

shutdown of snort at all and reenabled barnyard2 within suricata package on only 2 interfaces as it would rise up to 99% load at 4 nics

CPU: 47.5% user, 0.0% nice, 16.8% system, 0.2% interrupt, 35.5% idle
Mem: 1608M Active, 1758M Inact, 1011M Wired, 675M Buf, 3488M Free
Swap: 16G Total, 16G Free

PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
32784 root 1 102 0 561M 489M CPU1 1 11:12 88.10% barnyard2
36442 root 1 102 0 555M 483M CPU2 2 11:11 85.92% barnyard2

so is there a way rolling back to previous suricata version?

all it seems to be a barnyard2 issue with the performance… due suricata itself runs at low load. but after deinstalling suricata and only running snort barnyard2 it opt out the load issue with barnyard2. :-(

Ok, figured it out - i am using barnyard2 with mysql and snorby - so the fact that some old references where in the table i resettet snorby and voila - barnyard2 is running under normal beheavior.

bmeeks

@fgro:

@fgro:

Well folks, i am a bit disappointing with the current release of suricata 4.0.3 due it has a performance issue since upgrading.

*green shows idle, *yellow shows user

Upgrade to recent version / snort with barnyard2 was enabled

shutdown of snort barnyard2 & reenabled it

shutdown of snort at all and reenabled barnyard2 within suricata package on only 2 interfaces as it would rise up to 99% load at 4 nics

CPU: 47.5% user, 0.0% nice, 16.8% system, 0.2% interrupt, 35.5% idle
Mem: 1608M Active, 1758M Inact, 1011M Wired, 675M Buf, 3488M Free
Swap: 16G Total, 16G Free

PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
32784 root 1 102 0 561M 489M CPU1 1 11:12 88.10% barnyard2
36442 root 1 102 0 555M 483M CPU2 2 11:11 85.92% barnyard2

so is there a way rolling back to previous suricata version?

all it seems to be a barnyard2 issue with the performance… due suricata itself runs at low load. but after deinstalling suricata and only running snort barnyard2 it opt out the load issue with barnyard2. :-(

Ok, figured it out - i am using barnyard2 with mysql and snorby - so the fact that some old references where in the table i resettet snorby and voila - barnyard2 is running under normal beheavior.

Barnyard2 needs a lot of "love" from its developer in the area of SQL statements. It did that with me on Snort so often that I just abandoned using it completely. I had the Barnyard2 and Snorby combination, but those high CPU utilizations while running flawed SQL statements finally exasperated me enough to abandon the Barnyard2 idea.

Bill

bmeeks

@drewsaur:

Will the update GUI package be coming out anytime soon? I am still showing that 4.0.1_1 is current for my box.

It is out. What version of pfSense are you running? It should be showing up for all 2.4.x versions. I'm not so sure about 2.3.x versions because there may be other port dependencies that are not satisifed in that older pfSense tree.

Bill

The Sky Heart

@bmeeks:

@The:

Hi Guys,

I noticed the new version for Suricata in package manager, so I clicked on update, but the update failed with an error for some missing files, so had to remove the package and reinstall it again.

also I noticed this log:
12/1/2018 – 18:57:24 - <info>-- Invalid IP() parameter provided in Pass List, skipping...

i'm not sure where () came from but the list is an Alias list, and I didn't have that log in Suricata 4.0.1_1

Best Regards</info>

I suspect this error message might be the result of fixing one of the bugs. There was a logic flaw in how HOME_NET was populated in some circumstances. That warning about the slash means you have an alias or an interface IP that is resolving to "nothing" at run time when written to the configuration. The single slash is what would normally be part of a CIDR network string such as "192.168.1.0/24".

Bill

Hi Bill,
thanks a lot for the reply, and also sorry for the late update,

I'm not sure if I got your explanation about the Invalid IP(), I do have some interfaces with no IP's but those interfaces are not enabled, also the Alias list I have is all Network Alias so even single IP's are added with /32, and if I check the list which is imported by suricata I can see that there is a record with \ which is different from (/), even if I remove it from that list it will be rewritten when I restart Suricata.

one more question please, does the below load look's ok? cause I have Suricata jumps to above 150% and it's only running on WAN interface with no extra settings apart from around 10 ET Rules list enabled, but I do admit that there is alot of traffic passing through this box.

the box is a Dell R430 Server with CPU E5-2603 v4 @ 1.70GHz and 32GB of Memory.


last pid: 25566;  load averages:  2.88,  2.85,  2.96                                                                             
41 processes:  2 running, 39 sleeping
CPU:  1.0% user, 14.9% nice,  5.3% system, 17.8% interrupt, 61.0% idle
Mem: 322M Active, 504M Inact, 2270M Wired, 414M Buf, 28G Free
Swap: 4096M Total, 4096M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
69632 root         11  40   20   768M   687M nanslp  2  95.7H 100.06% suricata
36702 root          1  24    0 12696K  2820K bpf     0  26.6H   7.75% filterlog

I also noticed that I have a lot of the below lines in suricata.log file:


16/1/2018 -- 18:18:56 - <info>-- Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1516123136, ts.tv_usec:1399) flow_spare_q status(): 33% flows at the queue
16/1/2018 -- 18:19:05 - <info>-- Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1516123145, ts.tv_usec:1896) flow_spare_q status(): 39% flows at the queue
16/1/2018 -- 18:19:09 - <info>-- Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1516123149, ts.tv_usec:2995) flow_spare_q status(): 38% flows at the queue
16/1/2018 -- 18:19:14 - <info>-- Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1516123153, ts.tv_usec:997415) flow_spare_q status(): 32% flows at the queue
16/1/2018 -- 18:19:15 - <info>-- Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1516123155, ts.tv_usec:2263) flow_spare_q status(): 42% flows at the queue
16/1/2018 -- 18:19:19 - <info>-- Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1516123159, ts.tv_usec:1779) flow_spare_q status(): 43% flows at the queue</info></info></info></info></info></info>

is this something I have to worry about or need to fix?

Thanks again for the help.

RonpfS

@bmeeks:

It is out. What version of pfSense are you running? It should be showing up for all 2.4.x versions. I'm not so sure about 2.3.x versions because there may be other port dependencies that are not satisifed in that older pfSense tree.

Bill

Under 2.3.5-RELEASE-p1 (amd64) I have "Update available to 4.0.3"

drewsaur

@bmeeks:

@drewsaur:

Will the update GUI package be coming out anytime soon? I am still showing that 4.0.1_1 is current for my box.

It is out. What version of pfSense are you running? It should be showing up for all 2.4.x versions. I'm not so sure about 2.3.x versions because there may be other port dependencies that are not satisifed in that older pfSense tree.

Bill

As of this evening, the update showed up. FWIW, I am using 2.4.x.

Drew

bmeeks

@The:

@bmeeks:

@The:

Hi Guys,

I noticed the new version for Suricata in package manager, so I clicked on update, but the update failed with an error for some missing files, so had to remove the package and reinstall it again.

also I noticed this log:
12/1/2018 – 18:57:24 - <info>-- Invalid IP() parameter provided in Pass List, skipping...

i'm not sure where () came from but the list is an Alias list, and I didn't have that log in Suricata 4.0.1_1

Best Regards</info>

I suspect this error message might be the result of fixing one of the bugs. There was a logic flaw in how HOME_NET was populated in some circumstances. That warning about the slash means you have an alias or an interface IP that is resolving to "nothing" at run time when written to the configuration. The single slash is what would normally be part of a CIDR network string such as "192.168.1.0/24".

Bill

Hi Bill,
thanks a lot for the reply, and also sorry for the late update,

I'm not sure if I got your explanation about the Invalid IP(), I do have some interfaces with no IP's but those interfaces are not enabled, also the Alias list I have is all Network Alias so even single IP's are added with /32, and if I check the list which is imported by suricata I can see that there is a record with \ which is different from (/), even if I remove it from that list it will be rewritten when I restart Suricata.

one more question please, does the below load look's ok? cause I have Suricata jumps to above 150% and it's only running on WAN interface with no extra settings apart from around 10 ET Rules list enabled, but I do admit that there is alot of traffic passing through this box.

the box is a Dell R430 Server with CPU E5-2603 v4 @ 1.70GHz and 32GB of Memory.
last pid: 25566;  load averages:  2.88,  2.85,  2.96                                                                             
41 processes:  2 running, 39 sleeping
CPU:  1.0% user, 14.9% nice,  5.3% system, 17.8% interrupt, 61.0% idle
Mem: 322M Active, 504M Inact, 2270M Wired, 414M Buf, 28G Free
Swap: 4096M Total, 4096M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU COMMAND
69632 root         11  40   20   768M   687M nanslp  2  95.7H 100.06% suricata
36702 root          1  24    0 12696K  2820K bpf     0  26.6H   7.75% filterlog
I also noticed that I have a lot of the below lines in suricata.log file:
16/1/2018 -- 18:18:56 - <info>-- Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1516123136, ts.tv_usec:1399) flow_spare_q status(): 33% flows at the queue
16/1/2018 -- 18:19:05 - <info>-- Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1516123145, ts.tv_usec:1896) flow_spare_q status(): 39% flows at the queue
16/1/2018 -- 18:19:09 - <info>-- Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1516123149, ts.tv_usec:2995) flow_spare_q status(): 38% flows at the queue
16/1/2018 -- 18:19:14 - <info>-- Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1516123153, ts.tv_usec:997415) flow_spare_q status(): 32% flows at the queue
16/1/2018 -- 18:19:15 - <info>-- Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1516123155, ts.tv_usec:2263) flow_spare_q status(): 42% flows at the queue
16/1/2018 -- 18:19:19 - <info>-- Flow emergency mode over, back to normal... unsetting FLOW_EMERGENCY bit (ts.tv_sec: 1516123159, ts.tv_usec:1779) flow_spare_q status(): 43% flows at the queue</info></info></info></info></info></info> 
is this something I have to worry about or need to fix?

Thanks again for the help.

Let me answer the question about flow emergency mode errors first. You need to go read this document (scroll down to the section on Flow): https://www.aldeid.com/wiki/Suricata/Suricata-yaml-configuration-file. The short answer is on a high-traffic interface you are going to have to increase the flow memcap value substantially from the default. Some info is posted at the link given, plus I suggest some additional Google searches for the terms "suricata flow" "suricata flow memcap" and "suricata flow emergency mode". That search should net you additional information.

I'm sorry I confused the backslash and forward slash in my reply. If you indeed have a backslash, and you can see that backslash listed when viewing the Pass List on the INTERFACE SETTINGS tab (using the View List button), then you need to test by removing things from the list to see where the backslash is coming from. I would suspect an alias, if you have those configured, is the source of the backslash.

Bill

The Sky Heart

Hi Guys,

Thanks you so much @bmeeks I was finally able to find out the issue with the passlist, it was this record ::1 causing it, it was manually added to the Alias list, after removing it didn't see the alert in logs anymore, I also fixed the flow alerts by increasing the memory to 512mb.

but I have another issue now, it seem's that after the daily rules update cron pfsense GUI can't figure out if Suricata is running or not, so it's showing in the GUI that it's stopped for the interface, but checking the service and alert and block logs I can see it's Suricata is running, doing a start from the gui shows it's running again.

is anyone else facing this issue?

bmeeks

@The:

Hi Guys,

Thanks you so much @bmeeks I was finally able to find out the issue with the passlist, it was this record ::1 causing it, it was manually added to the Alias list, after removing it didn't see the alert in logs anymore, I also fixed the flow alerts by increasing the memory to 512mb.

but I have another issue now, it seem's that after the daily rules update cron pfsense GUI can't figure out if Suricata is running or not, so it's showing in the GUI that it's stopped for the interface, but checking the service and alert and block logs I can see it's Suricata is running, doing a start from the gui shows it's running again.

is anyone else facing this issue?

With the latest update I made a change to the GUI code on the INTERFACES tab to make it dynamically show service status. I was copying the same code over into the latest Snort update (since the two packages share most of the same GUI code) and found a couple of mistakes in the logic. So could be that the new dynamic GUI code is getting fooled when Suricata restarts from a rules update. It is likely a cosmetic display issue only.

Now one other thing I've seen folks do that I have strongly recommended AGAINST is use the Service Watchdog package. That package is great, but it does not understand how Suricata and Snort work with multiple processes (one per interface). The package also does not understand how to cope with Suricata restarting after a rules update, and the instant Service Watchdog sees one of the Suricata processes die it immediately executes the shell script to restart. That can really confuse things because Suricata is already restarting itself, so you wind up with multiple processes running on the same interface.

Bill