Suricata 4.1.4_2 not blocking hosts
-
@bmeeks The only thing I noted when I stopped the service from the homepage is that it still showed as if it were running even though it had stopped on both the interface. When I restarted the service again the WAN didn't automatically start but the LAN did. And the log for the when did not contain any of those errors after the I started it back up again. Lastly when running the command you mention it tells me I am running version 4.1.4.
-
@jchud: the "still running" indication would be a symptom of a zombie process or a crashed process where the PID file did not get cleaned up. The GUI senses the presence of the PID file with a valid process ID within it to determine what icon to show for Suricatat status.
It sounds much better that you no longer are seeing those errors in the
suricata.log
file.In the future, when upgrading the package, it is always safer to first delete it and then install it again instead of just clicking the "Reinstall" icon. The way PHP and the
pkg
utility work on the system can lead to a mixup of new and old files if you just do the reinstall option. While reinstall will work for some simple updates, other updates need to change files that are sometimes cached and held open by the system during a reinstall. That can prevent some migration actions for the new version from happening. -
@bmeeks I killed the zombie process and got everything to start back up again normally. In the suricata.log I still get the change of IP address thing along with the [ERRCODE: SC_WARN_UNCOMMON(230)], which I am guessing has something to do with the fact that when this is restarted on the WAN side "/rc.newwanip" starts doing some stuff like claiming the IP has changed even though its exactly the same as it was before.
-
@jchud said in Suricata 4.1.4_2 not blocking hosts:
@bmeeks I killed the zombie process and got everything to start back up again normally. In the suricata.log I still get the change of IP address thing along with the [ERRCODE: SC_WARN_UNCOMMON(230)], which I am guessing has something to do with the fact that when this is restarted on the WAN side "/rc.newwanip" starts doing some stuff like claiming the IP has changed even though its exactly the same as it was before.
No, that SC_WARN_UNCOMMON error is definitely not supposed to be there. What kind of interface is your WAN? Is it configured for DHCP or is it a PPPoE connection?
It really sounds like your binary is not the patched one. Look at the file date and time for this file:
/usr/local/bin/suricata
and let me know what it says. Giving the me size in bytes will help as well. The latest file should show a date of June 13, 2019 and have a file size of 4,340,864 bytes.
-
@bmeeks Yes my WAN is configure for DHCP and the date/time for that file is June 13 17:49 and the size in bytes is 4499944.
-
@jchud said in Suricata 4.1.4_2 not blocking hosts:
@bmeeks Yes my WAN is configure for DHCP and the date/time for that file is June 13 17:49 and the size in bytes is 4499944.
I'm working the same issue with another user in a different thread, and he is reporting the same file size. That is certainly different from what is on my machine. Is your hardware an Intel-base AMD64 CPU, or is it an ARM-based Netgate appliance? Trying to figure out why the binary is different.
-
@bmeeks Intel based, built the computer its running on myself
-
@jchud : Good, that means we are on the same CPU. My virtual machine box that works is running on VMware Workstation on a Windows host with an Intel i7 CPU.
Let's compare MD5 hashes to see if your binary is the same as mine. Run this command from a shell prompt on the firewall:
md5 -q /usr/local/bin/suricata
The output should be:
cc5200e8369def9268b9e30c0c3f41c6
Let me know what you get.
-
I just found this thread from a google search. I'm having the exact same issue.
It started yesterday, I noticed Suricata would run, and then just crash. I could restart it and it would work fine for a few hours, crash again.
I am getting the same error messages; [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - Argument "tree" NULL
Last night, I found the thread with you talking about the bug. I uninstalled and reinstalled suricata, and it ran all night and most of today w/ no issues, but it just crashed again.
So, again, I uninstalled Suricata ... waited for it to finish. Made sure it was gone, and then reinstalled it. Once I started Suricata, the error messages showed up again. Here's the output from what you where asking @jchud about.
Also, I hadn't noticed, but yes, it is not blocking hosts now either. It will Alert.
[2.4.4-RELEASE][admin@fw.alteredreality.cc]/root: suricata -V This is Suricata version 4.1.4 RELEASE [2.4.4-RELEASE][admin@fw.alteredreality.cc]/root: ls -l /usr/local/bin/suricata -rwxr-xr-x 1 root wheel 4499944 Jun 13 17:49 /usr/local/bin/suricata [2.4.4-RELEASE][admin@fw.alteredreality.cc]/root: md5 -q /usr/local/bin/suricata c962d5d995867c5baf3136035a34fac7 [2.4.4-RELEASE][admin@fw.alteredreality.cc]/root:
Thanks
-
when exactly have you reinstalled it? in the last 30 min ? because the new update with the right crc appeared on my pf sense 2.5 only 30 min ago max 1 hour ago
-
Make sure you have the correct MD5 hash showing for your Suricata binary as per my post above. Apparently something went south with the posting of the updated package and the binary got a new timestamp but did not actually get updated with my bug fix. I think that is the root of all these issues.
-
@bmeeks said in Suricata 4.1.4_2 not blocking hosts:
md5 -q /usr/local/bin/suricata
My alerts and blocks have starting working again. The result of that command gets me - c962d5d995867c5baf3136035a34fac7
-
Mine is still reporting the wrong md5 and timestamp. See below, I just ran this after an uninstall and reinstall. Literally, 3 minutes ago for reference.
[2.4.4-RELEASE][admin@fw.alteredreality.cc]/root: md5 -q /usr/local/bin/suricata c962d5d995867c5baf3136035a34fac7 [2.4.4-RELEASE][admin@fw.alteredreality.cc]/root: ls -l /usr/local/bin/suricata -rwxr-xr-x 1 root wheel 4499944 Jun 13 17:49 /usr/local/bin/suricata
My understanding is the md5 should be:
cc5200e8369def9268b9e30c0c3f41c6
Thanks
-
@ryan_g said in Suricata 4.1.4_2 not blocking hosts:
@bmeeks said in Suricata 4.1.4_2 not blocking hosts:
md5 -q /usr/local/bin/suricata
My alerts and blocks have starting working again. The result of that command gets me - c962d5d995867c5baf3136035a34fac7
Okay. That's good. It just dawned on me that the MD5 for folks on the 2.4.4 RELEASE branch of pfSense will be different due to being compiled on FreeBSD 11.2 versus FreeBSD 12.0 on the DEVEL branch. I don't have my own RELEASE builder, so I can't test that MD5. It's easy for me lose track of which poster is on which pfSense version ... .
-
Side note. I'm still seeing this;
14/6/2019 -- 21:20:45 - <Error> -- [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - Argument "tree" NULL
-
@wangel said in Suricata 4.1.4_2 not blocking hosts:
Side note. I'm still seeing this;
14/6/2019 -- 21:20:45 - <Error> -- [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - Argument "tree" NULL
Then you have a "non-fixed" version of the binary. Wait for the pfSense developer team to get it squared away in the Package Manager. Something strange happened there yesterday with the deployment of this package update. It seems the old binary got built but was given the new version number.
In the meantime, just disable Suricata blocking and you will be fine. You will still get alerts.
-
@bmeeks Sounds good. I'll just babysit it until then.
Thank you!
-
@bmeeks
How do I report this issue (or can you) to pfSense? This is still a problem.
It looks like it goes about 24hrs before crashing. Once it crashes, you can restart the service and it will work ... for another 24hrs etc.Today, at 4:34pm (EST) it crashed. So I figured I would try to uninstall and fix it. I did, and it is STILL getting the bugged binary.
I then, uninstalled again, went into /var/cache/pkg and removed the suricata files. They are as follows;
[2.4.4-RELEASE][admin@fw.alteredreality.cc]/var/cache/pkg: ls -l suri* -rw-r--r-- 1 root wheel 1585660 Jun 13 17:49 suricata-4.1.4_2-bd1c8a775c.txz lrwxr-xr-x 1 root wheel 31 Jun 17 16:50 suricata-4.1.4_2.txz -> suricata-4.1.4_2-bd1c8a775c.txz [2.4.4-RELEASE][admin@fw.alteredreality.cc]/var/cache/pkg:
I also looked at the pfsense-pkg-suricata file that had 4.1.4_4 in it ... it did not have a binary file in it =(
After that, I tried to install Suricata again. It appears to grab the same files from the package depot.
At anyrate, I started suricata, and sure enough I am still getting the error message:
17/6/2019 -- 16:52:41 - <Error> -- [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - Argument "tree" NULL 17/6/2019 -- 16:52:41 - <Error> -- [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - Argument "tree" NULL
Let me know if there's anything I can do, other than manually restarting the service when it crashes. It doesn't look like the watchdog package will restart it =(
Thanks
-
@wangel said in Suricata 4.1.4_2 not blocking hosts:
@bmeeks
How do I report this issue (or can you) to pfSense? This is still a problem.
It looks like it goes about 24hrs before crashing. Once it crashes, you can restart the service and it will work ... for another 24hrs etc.Today, at 4:34pm (EST) it crashed. So I figured I would try to uninstall and fix it. I did, and it is STILL getting the bugged binary.
I then, uninstalled again, went into /var/cache/pkg and removed the suricata files. They are as follows;
[2.4.4-RELEASE][admin@fw.alteredreality.cc]/var/cache/pkg: ls -l suri* -rw-r--r-- 1 root wheel 1585660 Jun 13 17:49 suricata-4.1.4_2-bd1c8a775c.txz lrwxr-xr-x 1 root wheel 31 Jun 17 16:50 suricata-4.1.4_2.txz -> suricata-4.1.4_2-bd1c8a775c.txz [2.4.4-RELEASE][admin@fw.alteredreality.cc]/var/cache/pkg:
After that, I tried to install Suricata again. It appears to grab the same files from the package depot.
At anyrate, I started suricata, and sure enough I am still getting the error message:
17/6/2019 -- 16:52:41 - <Error> -- [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - Argument "tree" NULL 17/6/2019 -- 16:52:41 - <Error> -- [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - Argument "tree" NULL
Let me know if there's anything I can do, other than manually restarting the service when it crashes. It doesn't look like the watchdog package will restart it =(
Thanks
I am the developer and maintainer for the Suricata package, so if you report anything to pfSense they will just forward it to me (or actually more likely refer you to this sub-forum for assistance). They do not support packages. Volunteer maintainers support the packages (the bulk of them, anyway).
First, DO NOT use Service Watchdog with Suricata. That package does not understand how Suricata works on multiple interfaces and it will try to restart Suricata during a rules update even though Suricata will restart itself. Then the two restarts can collide and crash Suricata. Not saying that is your problem this time, but DO NOT use Service Watchdog with Suricata or Snort. That has been stated here multiple times.
I want you to try something for me to be double sure any existing Suricata binary is removed.
-
Go to PACKAGE MANAGER and on the Installed Packages tab click the trash icon to delete Suricata from your firewall. Let that process finish, then proceed to step 2.
-
Open a shell prompt on the firewall and execute this command:
ps -ax | grep suricata
-
It should return only a single line showing the
grep
command you just executed. If you seen any other lines printed containing "suricata", then kill those processes and repeat the shell command until you see no suricata processes running and the only output shown is the singlegrep
command. -
Now execute this command at the shell prompt:
ls /usr/local/bin/suricata*
-
It should find no file by that name. If it does, then delete the file it finds.
-
Return to PACKAGE MANAGER and on the Available Packages tab find and install Suricata again.
At least by following the steps above we can be 100% sure that you are starting with the same binary that I am testing with.
-
-
@bmeeks said in Suricata 4.1.4_2 not blocking hosts:
@wangel said in Suricata 4.1.4_2 not blocking hosts:
@bmeeks
How do I report this issue (or can you) to pfSense? This is still a problem.
It looks like it goes about 24hrs before crashing. Once it crashes, you can restart the service and it will work ... for another 24hrs etc.Today, at 4:34pm (EST) it crashed. So I figured I would try to uninstall and fix it. I did, and it is STILL getting the bugged binary.
I then, uninstalled again, went into /var/cache/pkg and removed the suricata files. They are as follows;
[2.4.4-RELEASE][admin@fw.alteredreality.cc]/var/cache/pkg: ls -l suri* -rw-r--r-- 1 root wheel 1585660 Jun 13 17:49 suricata-4.1.4_2-bd1c8a775c.txz lrwxr-xr-x 1 root wheel 31 Jun 17 16:50 suricata-4.1.4_2.txz -> suricata-4.1.4_2-bd1c8a775c.txz [2.4.4-RELEASE][admin@fw.alteredreality.cc]/var/cache/pkg:
After that, I tried to install Suricata again. It appears to grab the same files from the package depot.
At anyrate, I started suricata, and sure enough I am still getting the error message:
17/6/2019 -- 16:52:41 - <Error> -- [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - Argument "tree" NULL 17/6/2019 -- 16:52:41 - <Error> -- [ERRCODE: SC_ERR_INVALID_ARGUMENT(13)] - Argument "tree" NULL
Let me know if there's anything I can do, other than manually restarting the service when it crashes. It doesn't look like the watchdog package will restart it =(
Thanks
I am the developer and maintainer for the Suricata package, so if you report anything to pfSense they will just forward it to me (or actually more likely refer you to this sub-forum for assistance). They do not support packages. Volunteer maintainers support the packages (the bulk of them, anyway).
First, DO NOT use Service Watchdog with Suricata. That package does not understand how Suricata works on multiple interfaces and it will try to restart Suricata during a rules update even though Suricata will restart itself. Then the two restarts can collide and crash Suricata. Not saying that is your problem this time, but DO NOT use Service Watchdog with Suricata or Snort. That has been stated here multiple times.
I want you to try something for me to be double sure any existing Suricata binary is removed.
-
Go to PACKAGE MANAGER and on the Installed Packages tab click the trash icon to delete Suricata from your firewall. Let that process finish, then proceed to step 2.
-
Open a shell prompt on the firewall and execute this command:
ps -ax | grep suricata
-
It should return only a single line showing the
grep
command you just executed. If you seen any other lines printed containing "suricata", then kill those processes and repeat the shell command until you see no suricata processes running and the only output shown is the singlegrep
command. -
Now execute this command at the shell prompt:
ls /usr/local/bin/suricata*
-
It should find no file by that name. If it does, then delete the file it finds.
-
Return to PACKAGE MANAGER and on the Available Packages tab find and install Suricata again.
At least by following the steps above we can be 100% sure that you are starting with the same binary that I am testing with.
Crap I feel like a total jackwagon.
I only installed and tried the Service Watchdog the first time it started crashing, when it didn't work for me, I removed it. I wasn't aware of the issues it would cause... I should have done more research.
That said, after removing the package, I looked at the processes and saw this:
58501 - Is 13:18.35 /usr/local/bin/suricata -i em0 -D -c /usr/local/etc 66088 - Is 13:34.59 /usr/local/bin/suricata -i em1 -D -c /usr/local/etc 60337 0 S+ 0:00.00 grep suricata [2.4.4-RELEASE][admin@fw.alteredreality.cc]/root:
I assume that when it would crash, it would just hang out there, and that explains why the binary wasn't getting overwritten with the new binary.
Sigh ... I feel like a dummy, I should have checked that.
Also, I understand you are the dev and maintainer ... I didn't mean to offend, I thought it might be a problem with the package distro that you didn't have control over.
At anyrate... I killed the 2 hung processes, reinstall the package, and we are back in business. Sheesh I feel dumb =(
Thanks @bmeeks ... where do I send the beer?
-