Suricata Crashed After PHP Error
-
Do you have a Suricata interface configured but with no rules defined at all? That's about the only way I can imagine that error message getting generated.
-
@bmeeks said in Suricata Crashed After Update:
Do you have a Suricata interface configured but with no rules defined at all? That's about the only way I can imagine that error message getting generated.
No ... rules had been defined and configured as well as running up to Saturday ... then after that day's update that included updates for both Snort and Suricata as well as Squid and OpenVPN-Client Export if I remembered correctly, Suricata would not start, nor the interface available in service.
So, I just completely removed the package, update pfSense to today's latest release and reinstalled Suricata only to get the same issue and crash report. Could it be PHP issue?
-
No, not a PHP issue or nothing would work as the entire pfSense GUI needs PHP to be functioning. You have something else wrong. What that is, I have no idea and no way to determine what it is unless I had your actual physical firewall and its full configuration to examine.
This statement might be key: "Suricata would not start, nor the interface available in service"
Do you still have the Apple hardware with the Thunderbolt NIC thing? If your interfaces are disappearing on updates/reboots, that can most certainly confuse packages. Or if your interfaces are coming up late (as in the driver for them is loading late), that will also confuse packages.
-
@bmeeks said in Suricata Crashed After Update:
No, not a PHP issue or nothing would work as the entire pfSense GUI needs PHP to be functioning. You have something else wrong. What that is, I have no idea and no way to determine what it is unless I had your actual physical firewall and its full configuration to examine.
This statement might be key: "Suricata would not start, nor the interface available in service"
Do you still have the Apple hardware with the Thunderbolt NIC thing? If your interfaces are disappearing on updates/reboots, that can most certainly confuse packages. Or if your interfaces are coming up late (as in the driver for them is loading late), that will also confuse packages.
Yes, still have the Apple Mac Mini with the Thunderbolt enclosure and with the Intel I350 NIC installed. Also, yes, I thought it was strange that at least Suricata should have been available in service. But, I have no issue receiving WAN IP ... so, now I am wondering whether the port is failing or as you said, issue with the driver interaction ... very strange indeed. Really wished I could have the latest Intell v25 driver installed.
Where are the rules stored ... maybe I could try moving to the correct folder.
Just looked and saw Snort's VRT rules been deprecated; however, I had no Snort rules whatsoever ... only ET Emerging rules ... wondered why is that? Can I remove?
-
No, you can't move any files around to fix that error. Those "missing" files are actually created as Suricata runs and builds your interface configuration files on-the-fly during startup. That is step is either not happening at all or is getting seriously messed up timing wise.
The Snort rules message is normal. The Snort team a couple of years or so back renamed lots of their categories, and to keep from breaking existing configurations that used the old names, they left "empty" files in their archive. All that option in the Suricata and Snort packages does is remove those empty categories. That is in no way related to your problem. That file you see is part of the package and is simply the list of deprecated and empty files. If you don't use Snort rules, then this file is never looked at in your setup. For folks that do use Snort rules with Suricata, this file is used.
-
@bmeeks said in Suricata Crashed After Update:
No, you can't move any files around to fix that error. Those "missing" files are actually created as Suricata runs and builds your interface configuration files on-the-fly during startup. That is step is either not happening at all or is getting seriously messed up timing wise.
The Snort rules message is normal. The Snort team a couple of years or so back renamed lots of their categories, and to keep from breaking existing configurations that used the old names, they left "empty" files in their archive. All that option in the Suricata and Snort packages does is remove those empty categories. That is in no way related to your problem. That file you see is part of the package and is simply the list of deprecated and empty files. If you don't use Snort rules, then this file is never looked at in your setup. For folks that do use Snort rules with Suricata, this file is used.
Okay ... now I am puzzled and what makes it worse is I cannot look at Suricata interface. Here is the package, and it's not running ...wondering whether it has an old PID ... the crash report would say that though.
-
@NollipfSense:
Why are you posting the directory contents of/usr/local/pkg/suricata? Do you not know that this is only a portion of the Suricata package files? That list of files in no way represents anything really meaningful for the package installation. -
@bmeeks said in Suricata Crashed After Update:
@NollipfSense:
Why are you posting the directory contents of/usr/local/pkg/suricata? Do you not know that this is only a portion of the Suricata package files? That list of files in no way represents anything really meaningful for the package installation.It's just for cross reference to demonstrate. Also, saw this in the logs ... what you make of it? Not sure why it is saying invalid license key ... I reissued a key on June 3, 2020. Also noticed that Suricata was running even after I had removed it and before I installed it again.
Jun 29 09:51:04 SuricataStartup 69115 Suricata START for WAN(61129_igb0)...
Jun 29 09:52:35 pkg-static 1348 suricata-5.0.2_3 installed
Jun 29 09:52:35 php 26247 /etc/rc.packages: Beginning package installation for suricata .
Jun 29 09:52:35 check_reload_status 575 Syncing firewall
Jun 29 09:52:35 php 26247 [Suricata] Installing free GeoLite2 country IP database file in /usr/local/share/suricata/GeoLite2/...
Jun 29 09:52:35 php 26247 [Suricata] Checking for updated MaxMind GeoLite2 IP database file...
Jun 29 09:52:35 php 26247 [Suricata] ALERT: The Account ID or License Key for MaxMind GeoLite2 is invalid.
Jun 29 09:52:35 php 26247 [Suricata] ERROR: GeoLite2-Country IP database update check failed. The GeoIP database was not updated!
Jun 29 09:52:35 pkg-static 1348 pfSense-pkg-suricata-5.0.2_3 installed -
@NollipfSense said in Suricata Crashed After Update:
@bmeeks said in Suricata Crashed After Update:
@NollipfSense:
Why are you posting the directory contents of/usr/local/pkg/suricata? Do you not know that this is only a portion of the Suricata package files? That list of files in no way represents anything really meaningful for the package installation.It's just for cross reference to demonstrate. Also, saw this in the logs ... what you make of it? Not sure why it is saying invalid license key ... I reissued a key on June 3, 2020. Also noticed that Suricata was running even after I had removed it and before I installed it again.
Jun 29 09:51:04 SuricataStartup 69115 Suricata START for WAN(61129_igb0)...
Jun 29 09:52:35 pkg-static 1348 suricata-5.0.2_3 installed
Jun 29 09:52:35 php 26247 /etc/rc.packages: Beginning package installation for suricata .
Jun 29 09:52:35 check_reload_status 575 Syncing firewall
Jun 29 09:52:35 php 26247 [Suricata] Installing free GeoLite2 country IP database file in /usr/local/share/suricata/GeoLite2/...
Jun 29 09:52:35 php 26247 [Suricata] Checking for updated MaxMind GeoLite2 IP database file...
Jun 29 09:52:35 php 26247 [Suricata] ALERT: The Account ID or License Key for MaxMind GeoLite2 is invalid.
Jun 29 09:52:35 php 26247 [Suricata] ERROR: GeoLite2-Country IP database update check failed. The GeoIP database was not updated!
Jun 29 09:52:35 pkg-static 1348 pfSense-pkg-suricata-5.0.2_3 installedThe error message is self-explanatory --
Jun 29 09:52:35 php 26247 [Suricata] ALERT: The Account ID or License Key for MaxMind GeoLite2 is invalid.Your MaxMind DB key is either invalid or has been entered incorrectly (a typo maybe) into the textbox in Suricata.
So MaxMind GeoLite2 database downloads are failing, but that will not impact Suricata starting up nor its basic functioning. The only issue from that GeoLite2 error is that any geoip rules you might have won't work. A missing database should not result in Suricata not starting.
-
@bmeeks said in Suricata Crashed After Update:
@NollipfSense said in Suricata Crashed After Update:
@bmeeks said in Suricata Crashed After Update:
@NollipfSense:
Why are you posting the directory contents of/usr/local/pkg/suricata? Do you not know that this is only a portion of the Suricata package files? That list of files in no way represents anything really meaningful for the package installation.It's just for cross reference to demonstrate. Also, saw this in the logs ... what you make of it? Not sure why it is saying invalid license key ... I reissued a key on June 3, 2020. Also noticed that Suricata was running even after I had removed it and before I installed it again.
Jun 29 09:51:04 SuricataStartup 69115 Suricata START for WAN(61129_igb0)...
Jun 29 09:52:35 pkg-static 1348 suricata-5.0.2_3 installed
Jun 29 09:52:35 php 26247 /etc/rc.packages: Beginning package installation for suricata .
Jun 29 09:52:35 check_reload_status 575 Syncing firewall
Jun 29 09:52:35 php 26247 [Suricata] Installing free GeoLite2 country IP database file in /usr/local/share/suricata/GeoLite2/...
Jun 29 09:52:35 php 26247 [Suricata] Checking for updated MaxMind GeoLite2 IP database file...
Jun 29 09:52:35 php 26247 [Suricata] ALERT: The Account ID or License Key for MaxMind GeoLite2 is invalid.
Jun 29 09:52:35 php 26247 [Suricata] ERROR: GeoLite2-Country IP database update check failed. The GeoIP database was not updated!
Jun 29 09:52:35 pkg-static 1348 pfSense-pkg-suricata-5.0.2_3 installedThe error message is self-explanatory --
Jun 29 09:52:35 php 26247 [Suricata] ALERT: The Account ID or License Key for MaxMind GeoLite2 is invalid.Your MaxMind DB key is either invalid or has been entered incorrectly (a typo maybe) into the textbox in Suricata.
So MaxMind GeoLite2 database downloads are failing, but that will not impact Suricata starting up nor its basic functioning. The only issue from that GeoLite2 error is that any geoip rules you might have won't work. A missing database should not result in Suricata not starting.
I understand however, I had only applied Maxmind to pfBlockerNG with a valid reissued key and not to Suricata unless the key is universally applied ... also what about Suricata starting despite it been uninstalled? Where again do I check to determine whether there is a stale copy of Suricata?
-
@NollipfSense said in Suricata Crashed After Update:
@bmeeks said in Suricata Crashed After Update:
@NollipfSense said in Suricata Crashed After Update:
@bmeeks said in Suricata Crashed After Update:
@NollipfSense:
Why are you posting the directory contents of/usr/local/pkg/suricata? Do you not know that this is only a portion of the Suricata package files? That list of files in no way represents anything really meaningful for the package installation.It's just for cross reference to demonstrate. Also, saw this in the logs ... what you make of it? Not sure why it is saying invalid license key ... I reissued a key on June 3, 2020. Also noticed that Suricata was running even after I had removed it and before I installed it again.
Jun 29 09:51:04 SuricataStartup 69115 Suricata START for WAN(61129_igb0)...
Jun 29 09:52:35 pkg-static 1348 suricata-5.0.2_3 installed
Jun 29 09:52:35 php 26247 /etc/rc.packages: Beginning package installation for suricata .
Jun 29 09:52:35 check_reload_status 575 Syncing firewall
Jun 29 09:52:35 php 26247 [Suricata] Installing free GeoLite2 country IP database file in /usr/local/share/suricata/GeoLite2/...
Jun 29 09:52:35 php 26247 [Suricata] Checking for updated MaxMind GeoLite2 IP database file...
Jun 29 09:52:35 php 26247 [Suricata] ALERT: The Account ID or License Key for MaxMind GeoLite2 is invalid.
Jun 29 09:52:35 php 26247 [Suricata] ERROR: GeoLite2-Country IP database update check failed. The GeoIP database was not updated!
Jun 29 09:52:35 pkg-static 1348 pfSense-pkg-suricata-5.0.2_3 installedThe error message is self-explanatory --
Jun 29 09:52:35 php 26247 [Suricata] ALERT: The Account ID or License Key for MaxMind GeoLite2 is invalid.Your MaxMind DB key is either invalid or has been entered incorrectly (a typo maybe) into the textbox in Suricata.
So MaxMind GeoLite2 database downloads are failing, but that will not impact Suricata starting up nor its basic functioning. The only issue from that GeoLite2 error is that any geoip rules you might have won't work. A missing database should not result in Suricata not starting.
I understand however, I had only applied Maxmind to pfBlockerNG with a valid reissued key and not to Suricata unless the key is universally applied ... also what about Suricata starting despite it been uninstalled? Where again do I check to determine whether there is a stale copy of Suricata?
Suricata and pfBlockerNG do not share anything related to the GeoLite2 databases. Each package works totally independently in that area. It is Suricata complaining about the MaxMind key. It could be that you once had that GeoIP download option enabled in Suricata; but not anymore, and your key is blank. At any rate, you can simply go to the GLOBAL SETTINGS tab in Suricata and disable the GeoIP database downloads there if you do not wish to use that option.
I don't understand what you are asking me here --
"also what about Suricata starting despite it been uninstalled?"
-
@bmeeks said in Suricata Crashed After Update:
It could be that you once had that GeoIP download option enabled in Suricata; but not anymore, and your key is blank.
No, never had Maxmind with Suricata ... only pfBlockerNG. All I had with Suricata is the Emerging Threat.
@bmeeks said in Suricata Crashed After Update:
I don't understand what you are asking me here --
"also what about Suricata starting despite it been uninstalled?"Also noticed that Suricata was running even after I had removed it and before I installed it again.
Jun 29 09:51:04 SuricataStartup 69115 Suricata START for WAN(61129_igb0)...
Jun 29 09:52:35 pkg-static 1348 suricata-5.0.2_3 installed
Jun 29 09:52:35 php 26247 /etc/rc.packages: Beginning package installation for suricata .
Jun 29 09:52:35 check_reload_status 575 Syncing firewall
Jun 29 09:52:35 php 26247 [Suricata] Installing free GeoLite2 country IP database file in /usr/local/share/suricata/GeoLite2/...
Jun 29 09:52:35 php 26247 [Suricata] Checking for updated MaxMind GeoLite2 IP database file...
Jun 29 09:52:35 php 26247 [Suricata] ALERT: The Account ID or License Key for MaxMind GeoLite2 is invalid.
Jun 29 09:52:35 php 26247 [Suricata] ERROR: GeoLite2-Country IP database update check failed. The GeoIP database was not updated!
Jun 29 09:52:35 pkg-static 1348 pfSense-pkg-suricata-5.0.2_3 installed -
You had a duplicate Suricata process.
Do you by chance have the Service Watchdog package installed and configured to monitor Suricata?
-
@bmeeks said in Suricata Crashed After Update:
You had a duplicate Suricata process.
Do you by chance have the Service Watchdog package installed and configured to monitor Suricata?
No, no service watchdog installed ... that's why I am thinking a have a stale state that is preventing Suricata.
-
@NollipfSense said in Suricata Crashed After Update:
@bmeeks said in Suricata Crashed After Update:
You had a duplicate Suricata process.
Do you by chance have the Service Watchdog package installed and configured to monitor Suricata?
No, no service watchdog installed ... that's why I am thinking a have a stale state that is preventing Suricata.
Stop Suricata on all interfaces using the GUI icons on the INTERFACES tab.
Then open a shell prompt on the firewall console either directly or via SSH and issue this command:ps -ax | grep suricataIf it returns any running instances of suricata, note the Process ID <pid> of the process and run this command:
kill -9 <pid>where <pid> is the Process ID obtained from the first command. Repeat the previous two steps until there are no Suricata processes remaining.
Now delete the Suricata package using SYSTEM > PACKAGE MANAGER and clicking the Delete icon beside Suricata on the Installed Packages tab.
Let the deletion steps complete. Wait for the green "success" bar.
Now, on the Available Packages tab of PACKAGE MANAGER, find Suricata and install it again. Wait for the installation to 100% complete. It will take some time. Again, there will be a green "success bar" and message when the installation is done.
-
@bmeeks said in Suricata Crashed After Update:
ps -ax | grep suricata
I get this when I SSH using console: [2.5.0-DEVELOPMENT][admin@NollipfSense.nollipfsense.lan]/root: ps -ax | grep suricata
23066 0 S+ 0:00.00 grep suricataWhen I used the GUI, I get this:
Suricata is only on the WAN interface and doesn't show under Service tab.
-
@NollipfSense said in Suricata Crashed After Update:
@bmeeks said in Suricata Crashed After Update:
ps -ax | grep suricata
I get this when I SSH using console: [2.5.0-DEVELOPMENT][admin@NollipfSense.nollipfsense.lan]/root: ps -ax | grep suricata
23066 0 S+ 0:00.00 grep suricataWhen I used the GUI, I get this:
Suricata is only on the WAN interface and doesn't show under Service tab.
If it doesn't show under SERVICES, then it is not installed (or at least not properly). I thought you said it was installed but not starting.
I feel like we are going around and around in circles here ... ???
You have posted with an awful lot of problems over the last few months with pfSense and its packages. Running pfSense and its packages on a Mac Mini is not ideal; especially with a Thunderbolt interface to hold your network interfaces. Here is an analogy. I can probably haul a yard of loose sand in a Honda Civic, but that vehicle is certainly not optimized for that task, and I can expect some significant difficulties when doing so. In my view, the same thing applies to your hardware setup. You really should move to something more conventional to mimic the hardware pretty much everyone else is using for pfSense.
-
@bmeeks said in Suricata Crashed After Update:
@NollipfSense said in Suricata Crashed After Update:
@bmeeks said in Suricata Crashed After Update:
ps -ax | grep suricata
I get this when I SSH using console: [2.5.0-DEVELOPMENT][admin@NollipfSense.nollipfsense.lan]/root: ps -ax | grep suricata
23066 0 S+ 0:00.00 grep suricataWhen I used the GUI, I get this:
Suricata is only on the WAN interface and doesn't show under Service tab.
If it doesn't show under SERVICES, then it is not installed (or at least not properly). I thought you said it was installed but not starting.
I feel like we are going around and around in circles here ... ???
You have posted with an awful lot of problems over the last few months with pfSense and its packages. Running pfSense and its packages on a Mac Mini is not ideal; especially with a Thunderbolt interface to hold your network interfaces. Here is an analogy. I can probably haul a yard of loose sand in a Honda Civic, but that vehicle is certainly not optimized for that task, and I can expect some significant difficulties when doing so. In my view, the same thing applies to your hardware setup. You really should move to something more conventional to mimic the hardware pretty much everyone else is using for pfSense.
Had hoped you would offer some light on what appears three different process IDs. There are no lots of awful problems ... remember pfSense 2.5 is still under development, and issues arise along the process especially during the installation of daily built. Remember Mac hardware is already optimized for FreeBSD ... it's based foundation and super optimized for Apple OS.
Six months of running pfSense 2.5 with three issues: the first was after the introduction of FreeBSD 12.1 ... that was sorted with a clean install since I had upgraded from pfSense 2.4.4 to 2.5 with FreeBSD 12.0 FreeBDS 12.1 also caused traffic graph (widget) to show because of the new deployment of Netmap ... the second was the maxmind issue again after an update and eventually had to reissue a new key to resolve ... and now the third again after an update and Suricata won't start while the package is referring again to an invalid Maxmind key when I didn't use Maxmind with the IDS/IPS package.
Prior to installing pfSense 2.5-devel, the same hardware had run pfSense 2.4.4 for almost a year without any issues. So, I cannot buy your premise. Things are not expected to run smoothly on a development software continually ... hiccups are bound to happen. It seems time for another clean install of pfSense 2.5-devel.
-
@bmeeks said in Suricata Crashed After Update:
I thought you said it was installed but not starting.
To clarify, I had Suricata installed and running up to this passed Saturday, then the update for Saturday broke Suricata and I share here. The update for Sunday didn't fix the issue nor the update for today. I uninstall Suricata and then reinstall it again. The GUI above shows Suricata and all its resources are together in a folder, yet Suricata is not running nor appear under the Service tab.
-
After more investigation, it appears that the Suricata crash was PHP error indeed. A clean install of the latest pfSense 2.5 and a restored configuration revealed that the GeoLite2 DB update was enabled on Suricata's global settings. However, last May there was a similar PHP error that resulted invalid MaxMind key. That was changed when a reissued key had been applied on June 3rd. So, there should not have been an invalid key on June 27th resulting in crippling Suricata. Since the 27th, there has been issues with PHP and yesterday there had been five or six snapshots each with issues ... others have mentioned the same here: https://forum.netgate.com/topic/154898/2-5-latest-update-issue and here: https://forum.netgate.com/topic/154936/gateway-automatic-seems-to-have-quit-working-saturday-june-27-working-on-later-snap
So, I have disabled GeoLite2 on Suricata because I did not realize I had enabled it and it should not have crippled Suricata. My firewall is back to normal development testing with MaxMind enabled only on pfBlockerNG. I believe the developers might be pressured (self induced) to have a release candidate; however, the update installer should not be posing issues at this stage. The last snapshot update late last night had issues.
