Suricata 4.1.4_2 not blocking hosts

wangel

I just found this thread from a google search. I'm having the exact same issue.

It started yesterday, I noticed Suricata would run, and then just crash. I could restart it and it would work fine for a few hours, crash again.

I am getting the same error messages; [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - Argument "tree" NULL

Last night, I found the thread with you talking about the bug. I uninstalled and reinstalled suricata, and it ran all night and most of today w/ no issues, but it just crashed again.

So, again, I uninstalled Suricata ... waited for it to finish. Made sure it was gone, and then reinstalled it. Once I started Suricata, the error messages showed up again. Here's the output from what you where asking @jchud about.

Also, I hadn't noticed, but yes, it is not blocking hosts now either. It will Alert.

[2.4.4-RELEASE][admin@fw.alteredreality.cc]/root: suricata -V
This is Suricata version 4.1.4 RELEASE
[2.4.4-RELEASE][admin@fw.alteredreality.cc]/root: ls -l /usr/local/bin/suricata
-rwxr-xr-x  1 root  wheel  4499944 Jun 13 17:49 /usr/local/bin/suricata
[2.4.4-RELEASE][admin@fw.alteredreality.cc]/root: md5 -q /usr/local/bin/suricata
c962d5d995867c5baf3136035a34fac7
[2.4.4-RELEASE][admin@fw.alteredreality.cc]/root:

Thanks

kiokoman

when exactly have you reinstalled it? in the last 30 min ? because the new update with the right crc appeared on my pf sense 2.5 only 30 min ago max 1 hour ago

bmeeks

Make sure you have the correct MD5 hash showing for your Suricata binary as per my post above. Apparently something went south with the posting of the updated package and the binary got a new timestamp but did not actually get updated with my bug fix. I think that is the root of all these issues.

ryan_g

@bmeeks said in Suricata 4.1.4_2 not blocking hosts:

md5 -q /usr/local/bin/suricata

My alerts and blocks have starting working again. The result of that command gets me - c962d5d995867c5baf3136035a34fac7

wangel

@ryan_g @bmeeks

Mine is still reporting the wrong md5 and timestamp. See below, I just ran this after an uninstall and reinstall. Literally, 3 minutes ago for reference.

[2.4.4-RELEASE][admin@fw.alteredreality.cc]/root: md5 -q /usr/local/bin/suricata
c962d5d995867c5baf3136035a34fac7
[2.4.4-RELEASE][admin@fw.alteredreality.cc]/root: ls -l /usr/local/bin/suricata
-rwxr-xr-x  1 root  wheel  4499944 Jun 13 17:49 /usr/local/bin/suricata

My understanding is the md5 should be:

cc5200e8369def9268b9e30c0c3f41c6

Thanks

bmeeks

@ryan_g said in Suricata 4.1.4_2 not blocking hosts:

@bmeeks said in Suricata 4.1.4_2 not blocking hosts:

md5 -q /usr/local/bin/suricata

My alerts and blocks have starting working again. The result of that command gets me - c962d5d995867c5baf3136035a34fac7

Okay. That's good. It just dawned on me that the MD5 for folks on the 2.4.4 RELEASE branch of pfSense will be different due to being compiled on FreeBSD 11.2 versus FreeBSD 12.0 on the DEVEL branch. I don't have my own RELEASE builder, so I can't test that MD5. It's easy for me lose track of which poster is on which pfSense version ... .

wangel

Side note. I'm still seeing this;

14/6/2019 -- 21:20:45 - <Error> -- [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - Argument "tree" NULL

bmeeks

@wangel said in Suricata 4.1.4_2 not blocking hosts:

Side note. I'm still seeing this;

14/6/2019 -- 21:20:45 - <Error> -- [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - Argument "tree" NULL

Then you have a "non-fixed" version of the binary. Wait for the pfSense developer team to get it squared away in the Package Manager. Something strange happened there yesterday with the deployment of this package update. It seems the old binary got built but was given the new version number.

In the meantime, just disable Suricata blocking and you will be fine. You will still get alerts.

wangel

@bmeeks Sounds good. I'll just babysit it until then.

Thank you!

wangel

@bmeeks
How do I report this issue (or can you) to pfSense? This is still a problem.
It looks like it goes about 24hrs before crashing. Once it crashes, you can restart the service and it will work ... for another 24hrs etc.

Today, at 4:34pm (EST) it crashed. So I figured I would try to uninstall and fix it. I did, and it is STILL getting the bugged binary.

I then, uninstalled again, went into /var/cache/pkg and removed the suricata files. They are as follows;

[2.4.4-RELEASE][admin@fw.alteredreality.cc]/var/cache/pkg: ls -l suri*
-rw-r--r--  1 root  wheel  1585660 Jun 13 17:49 suricata-4.1.4_2-bd1c8a775c.txz
lrwxr-xr-x  1 root  wheel       31 Jun 17 16:50 suricata-4.1.4_2.txz -> suricata-4.1.4_2-bd1c8a775c.txz
[2.4.4-RELEASE][admin@fw.alteredreality.cc]/var/cache/pkg:

I also looked at the pfsense-pkg-suricata file that had 4.1.4_4 in it ... it did not have a binary file in it =(

After that, I tried to install Suricata again. It appears to grab the same files from the package depot.

At anyrate, I started suricata, and sure enough I am still getting the error message:

17/6/2019 -- 16:52:41 - <Error> -- [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - Argument "tree" NULL
17/6/2019 -- 16:52:41 - <Error> -- [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - Argument "tree" NULL

Let me know if there's anything I can do, other than manually restarting the service when it crashes. It doesn't look like the watchdog package will restart it =(

Thanks

bmeeks

@wangel said in Suricata 4.1.4_2 not blocking hosts:

@bmeeks
How do I report this issue (or can you) to pfSense? This is still a problem.
It looks like it goes about 24hrs before crashing. Once it crashes, you can restart the service and it will work ... for another 24hrs etc.

Today, at 4:34pm (EST) it crashed. So I figured I would try to uninstall and fix it. I did, and it is STILL getting the bugged binary.

I then, uninstalled again, went into /var/cache/pkg and removed the suricata files. They are as follows;
[2.4.4-RELEASE][admin@fw.alteredreality.cc]/var/cache/pkg: ls -l suri*
-rw-r--r--  1 root  wheel  1585660 Jun 13 17:49 suricata-4.1.4_2-bd1c8a775c.txz
lrwxr-xr-x  1 root  wheel       31 Jun 17 16:50 suricata-4.1.4_2.txz -> suricata-4.1.4_2-bd1c8a775c.txz
[2.4.4-RELEASE][admin@fw.alteredreality.cc]/var/cache/pkg:
After that, I tried to install Suricata again. It appears to grab the same files from the package depot.

At anyrate, I started suricata, and sure enough I am still getting the error message:
17/6/2019 -- 16:52:41 - <Error> -- [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - Argument "tree" NULL
17/6/2019 -- 16:52:41 - <Error> -- [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - Argument "tree" NULL
Let me know if there's anything I can do, other than manually restarting the service when it crashes. It doesn't look like the watchdog package will restart it =(

Thanks

I am the developer and maintainer for the Suricata package, so if you report anything to pfSense they will just forward it to me (or actually more likely refer you to this sub-forum for assistance). They do not support packages. Volunteer maintainers support the packages (the bulk of them, anyway).

First, DO NOT use Service Watchdog with Suricata. That package does not understand how Suricata works on multiple interfaces and it will try to restart Suricata during a rules update even though Suricata will restart itself. Then the two restarts can collide and crash Suricata. Not saying that is your problem this time, but DO NOT use Service Watchdog with Suricata or Snort. That has been stated here multiple times.

I want you to try something for me to be double sure any existing Suricata binary is removed.

Go to PACKAGE MANAGER and on the Installed Packages tab click the trash icon to delete Suricata from your firewall. Let that process finish, then proceed to step 2.
Open a shell prompt on the firewall and execute this command:

ps -ax | grep suricata

It should return only a single line showing the grep command you just executed. If you seen any other lines printed containing "suricata", then kill those processes and repeat the shell command until you see no suricata processes running and the only output shown is the single grep command.
Now execute this command at the shell prompt:

ls /usr/local/bin/suricata*

It should find no file by that name. If it does, then delete the file it finds.
Return to PACKAGE MANAGER and on the Available Packages tab find and install Suricata again.

At least by following the steps above we can be 100% sure that you are starting with the same binary that I am testing with.

wangel

@bmeeks said in Suricata 4.1.4_2 not blocking hosts:

@wangel said in Suricata 4.1.4_2 not blocking hosts:
@bmeeks
How do I report this issue (or can you) to pfSense? This is still a problem.
It looks like it goes about 24hrs before crashing. Once it crashes, you can restart the service and it will work ... for another 24hrs etc.

Today, at 4:34pm (EST) it crashed. So I figured I would try to uninstall and fix it. I did, and it is STILL getting the bugged binary.

I then, uninstalled again, went into /var/cache/pkg and removed the suricata files. They are as follows;
[2.4.4-RELEASE][admin@fw.alteredreality.cc]/var/cache/pkg: ls -l suri*
-rw-r--r--  1 root  wheel  1585660 Jun 13 17:49 suricata-4.1.4_2-bd1c8a775c.txz
lrwxr-xr-x  1 root  wheel       31 Jun 17 16:50 suricata-4.1.4_2.txz -> suricata-4.1.4_2-bd1c8a775c.txz
[2.4.4-RELEASE][admin@fw.alteredreality.cc]/var/cache/pkg:
After that, I tried to install Suricata again. It appears to grab the same files from the package depot.

At anyrate, I started suricata, and sure enough I am still getting the error message:
17/6/2019 -- 16:52:41 - <Error> -- [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - Argument "tree" NULL
17/6/2019 -- 16:52:41 - <Error> -- [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - Argument "tree" NULL
Let me know if there's anything I can do, other than manually restarting the service when it crashes. It doesn't look like the watchdog package will restart it =(

Thanks
I am the developer and maintainer for the Suricata package, so if you report anything to pfSense they will just forward it to me (or actually more likely refer you to this sub-forum for assistance). They do not support packages. Volunteer maintainers support the packages (the bulk of them, anyway).

First, DO NOT use Service Watchdog with Suricata. That package does not understand how Suricata works on multiple interfaces and it will try to restart Suricata during a rules update even though Suricata will restart itself. Then the two restarts can collide and crash Suricata. Not saying that is your problem this time, but DO NOT use Service Watchdog with Suricata or Snort. That has been stated here multiple times.

I want you to try something for me to be double sure any existing Suricata binary is removed.

Go to PACKAGE MANAGER and on the Installed Packages tab click the trash icon to delete Suricata from your firewall. Let that process finish, then proceed to step 2.

Open a shell prompt on the firewall and execute this command:
ps -ax | grep suricata
It should return only a single line showing the grep command you just executed. If you seen any other lines printed containing "suricata", then kill those processes and repeat the shell command until you see no suricata processes running and the only output shown is the single grep command.

Now execute this command at the shell prompt:
ls /usr/local/bin/suricata*
It should find no file by that name. If it does, then delete the file it finds.

Return to PACKAGE MANAGER and on the Available Packages tab find and install Suricata again.

At least by following the steps above we can be 100% sure that you are starting with the same binary that I am testing with.

Crap I feel like a total jackwagon.

I only installed and tried the Service Watchdog the first time it started crashing, when it didn't work for me, I removed it. I wasn't aware of the issues it would cause... I should have done more research.

That said, after removing the package, I looked at the processes and saw this:

58501  -  Is       13:18.35 /usr/local/bin/suricata -i em0 -D -c /usr/local/etc
66088  -  Is       13:34.59 /usr/local/bin/suricata -i em1 -D -c /usr/local/etc
60337  0  S+        0:00.00 grep suricata
[2.4.4-RELEASE][admin@fw.alteredreality.cc]/root:

I assume that when it would crash, it would just hang out there, and that explains why the binary wasn't getting overwritten with the new binary.

Sigh ... I feel like a dummy, I should have checked that.

Also, I understand you are the dev and maintainer ... I didn't mean to offend, I thought it might be a problem with the package distro that you didn't have control over.

At anyrate... I killed the 2 hung processes, reinstall the package, and we are back in business. Sheesh I feel dumb =(

Thanks @bmeeks ... where do I send the beer?

bmeeks

@wangel: Glad it's working for you. I was really pretty certain I had fixed the bug. I even went back through the binary code two more times to convince myself. All ways for that error to happen were removed from the code. So in your case, it seems you had what I call zombie Suricata processes out there, and your issues were actually coming from them as they would have been running old copies of the binary (hence the continuing error).