Suricata crashes almost instantly after startup
-
Good to know, thanks! I've had very limited experience with ARM systems, but I can see (first hand, atm) that being a problem.
Anyway, I uninstalled Suricata, and Installed Snort. The latter is running, but having some minor issues (keeps loosing config). However, I still have Suricata listed as an option in the Services menu. Is there a way for me to clear that out?
-
@CyberMinion said in Suricata crashes almost instantly after startup:
Good to know, thanks! I've had very limited experience with ARM systems, but I can see (first hand, atm) that being a problem.
Anyway, I uninstalled Suricata, and Installed Snort. The latter is running, but having some minor issues (keeps loosing config). However, I still have Suricata listed as an option in the Services menu. Is there a way for me to clear that out?
Sounds like you still have some sort of fundamental problem with your pfSense installation on that box. Snort should not ever lose its configuration. That, coupled with the fact you say you "removed" Suricata but it still shows up in the SERVICES menu, leads me to suspect some issue with the basic configuration of your box.
How did you "remove" Suriata? Did you go to SYSTEM > PACKAGE MANAGER and then on the Installed Packages tab clicked the trash can icon to delete the package? If so, that process should have completed and shown you a green bar on the page. That will remove the entry under SERVICES. If an entry for Suricata remains under SERVICES, then the package removal did not complete.
Or else you may have restored an older
config.xml
configuration from a backup made while Suricata was still installed. Have you perhaps done that? And if you are restoring some olderconfig.xml
backup taken before you installed the Snort package, that could account for your Snort configuration "disappearing". Everything all packages need for their configuration lives in theconfig.xml
file on the firewall. -
How did you "remove" Suriata? Did you go to SYSTEM > PACKAGE MANAGER and then on the Installed Packages tab clicked the trash can icon to delete the package? If so, that process should have completed and shown you a green bar on the page. That will remove the entry under SERVICES. If an entry for Suricata remains under SERVICES, then the package removal did not complete.
Yes, I used the package manager to uninstall it, and it reported success. Suricata is no longer listed as an installed package, either.
Or else you may have restored an older
config.xml
configuration from a backup made while Suricata was still installed. Have you perhaps done that? And if you are restoring some olderconfig.xml
backup taken before you installed the Snort package, that could account for your Snort configuration "disappearing". Everything all packages need for their configuration lives in theconfig.xml
file on the firewall.I have not restored anything on this device in the past year, although I did recently set up automatic backups, just in case.
So I set up Snort on the WAN0 interface, added my auth codes, etc. and started it up. It ran for two days, and started issuing some alerts (I'm fine-tuning now). Then two days later, it just lost all of the config. According to the logs, it was updating regularly, as configured for that time, then it just stopped...no new log entries from Snort. All of the config was blank, as if I had just installed it. So, I went through and did it all again, and started it up. It initialized, and started posting alerts again. Then, I added a few alerts to the suppress list (less than 5, defiantly not too many). Three days later, I checked on it again, and saw the same type of alerts I had previously suppressed! So, I checked the suppression tab, and found that the suppression list was gone. So, I created a new one, and also manually made another backup.. I just checked again, and everything still as it should be.
I suppose I could run a factory reset, but I would prefer not to.
-
@CyberMinion said in Suricata crashes almost instantly after startup:
So, I went through and did it all again, and started it up. It initialized, and started posting alerts again. Then, I added a few alerts to the suppress list (less than 5, defiantly not too many). Three days later, I checked on it again, and saw the same type of alerts I had previously suppressed! So, I checked the suppression tab, and found that the suppression list was gone. So, I created a new one, and also manually made another backup.. I just checked again, and everything still as it should be.
That is just plain weird. Snort simply calls a pfSense system function to write information to the
config.xml
file. That is where all configuration information for the entire firewall and any installed packages is stored.Have you looked in the pfSense system log to see if anything unusual is being logged? I have never heard of any package just randomly losing all of its configuration information from the configuration file.
-
@bmeeks
Ok, I may have found something. Snort was working fine, checking for updates periodically.May 8 16:18:25 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] The Rules update has finished.
Then, the next log entry is this:
May 8 17:21:25 php-cgi rc.bootup: Beginning package installation for snort .
Which is odd, since it was already installed. Maybe it was an auto-update? I didn't tell it to do anything.
May 8 17:24:29 php-cgi rc.bootup: [Snort] Package post-installation tasks completed...
May 8 17:24:32 php-cgi rc.bootup: Successfully installed package: snort.After that, I eventually found it offline, and went to check and see if I had a recent backup. I didn't so I told it to make one. Then I configured it again, and saved it again. The log dump is below. Oddly enough, it seems to show multiple successful backups for each time It ran the process. After this, it seemed to keep working, except for the loss of suppress rules, which I think I noticed on the 14th (yesterday). There are no relevant log entries for that which I could find.
May 12 22:56:08 php-fpm 447 /snort/snort_interfaces_edit.php: Beginning configuration backup to https://acb.netgate.com/save
May 12 22:56:13 php-fpm 447 /snort/snort_interfaces_edit.php: End of configuration backup to https://acb.netgate.com/save (success).
May 12 22:56:16 check_reload_status Syncing firewall
May 12 22:56:16 php-fpm 447 /snort/snort_interfaces_edit.php: Beginning configuration backup to https://acb.netgate.com/save
May 12 22:56:21 php-fpm 447 /snort/snort_interfaces_edit.php: End of configuration backup to https://acb.netgate.com/save (success).
May 12 22:56:23 check_reload_status Syncing firewall
May 12 22:56:23 php-fpm 447 /snort/snort_interfaces_edit.php: Beginning configuration backup to https://acb.netgate.com/save
May 12 22:56:27 php-fpm 447 /snort/snort_interfaces_edit.php: End of configuration backup to https://acb.netgate.com/save (success).
May 12 22:56:30 check_reload_status Syncing firewall
May 12 22:56:30 php-fpm 447 /snort/snort_interfaces_edit.php: Beginning configuration backup to https://acb.netgate.com/save
May 12 22:56:35 php-fpm 447 /snort/snort_interfaces_edit.php: End of configuration backup to https://acb.netgate.com/save (success).
May 12 23:01:12 php-fpm 13038 /snort/snort_interfaces.php: Starting Snort on LAN(mvneta0.4091) per user request...
May 12 23:01:13 php /tmp/snort_mvneta0.409158189_startcmd.php: [Snort] Updating rules configuration for: LAN ...
May 12 23:01:13 php /tmp/snort_mvneta0.409158189_startcmd.php: [Snort] Warning - no text rules or IPS-Policy selected for: LAN ...
May 12 23:01:13 php /tmp/snort_mvneta0.409158189_startcmd.php: [Snort] Building new sid-msg.map file for LAN...
May 12 23:01:13 php /tmp/snort_mvneta0.409158189_startcmd.php: [Snort] Snort START for LAN(mvneta0.4091)...
May 12 23:01:13 kernel mvneta0: promiscuous mode enabled
May 12 23:01:13 kernel mvneta0.4091: promiscuous mode enabled
May 12 23:03:01 php-fpm 447 /snort/snort_interfaces_global.php: Beginning configuration backup to https://acb.netgate.com/save
May 12 23:03:01 check_reload_status Syncing firewall
May 12 23:03:05 php-fpm 447 /snort/snort_interfaces_global.php: End of configuration backup to https://acb.netgate.com/save (success).
May 12 23:03:06 php-fpm 57637 /snort/snort_interfaces.php: Restarting Snort on LAN(mvneta0.4091) per user request...
May 12 23:03:06 php-fpm 57637 /snort/snort_interfaces.php: [Snort] Snort STOP for LAN(mvneta0.4091)...
May 12 23:03:07 snort 80845 *** Caught Term-Signal
May 12 23:03:07 kernel mvneta0: promiscuous mode disabled
May 12 23:03:07 kernel mvneta0.4091: promiscuous mode disabled
May 12 23:03:09 php /tmp/snort_mvneta0.409158189_startcmd.php: [Snort] Updating rules configuration for: LAN ...
>May 12 23:03:09 php /tmp/snort_mvneta0.409158189_startcmd.php: [Snort] Warning - no text rules or IPS-Policy selected for: LAN ...
May 12 23:03:09 php /tmp/snort_mvneta0.409158189_startcmd.php: [Snort] Building new sid-msg.map file for LAN...
May 12 23:03:11 php-fpm 447 /snort/snort_interfaces_global.php: Beginning configuration backup to https://acb.netgate.com/save
May 12 23:03:11 check_reload_status Syncing firewall
May 12 23:03:16 php-fpm 447 /snort/snort_interfaces_global.php: End of configuration backup to https://acb.netgate.com/save (success).
May 12 23:03:16 check_reload_status Syncing firewall
May 12 23:03:16 php /tmp/snort_mvneta0.409158189_startcmd.php: Beginning configuration backup to https://acb.netgate.com/save
May 12 23:03:21 php /tmp/snort_mvneta0.409158189_startcmd.php: End of configuration backup to https://acb.netgate.com/save (success).
May 12 23:03:21 php-fpm 447 /snort/snort_interfaces_global.php: Beginning configuration backup to https://acb.netgate.com/save
May 12 23:03:21 check_reload_status Syncing firewall
May 12 23:03:24 php /tmp/snort_mvneta0.409158189_startcmd.php: Beginning configuration backup to https://acb.netgate.com/save
May 12 23:03:24 check_reload_status Syncing firewall
May 12 23:03:26 php-fpm 447 /snort/snort_interfaces_global.php: End of configuration backup to https://acb.netgate.com/save (success).
May 12 23:03:29 php-fpm 447 /snort/snort_interfaces_global.php: Beginning configuration backup to https://acb.netgate.com/save
May 12 23:03:29 check_reload_status Syncing firewall
May 12 23:03:29 php /tmp/snort_mvneta0.409158189_startcmd.php: End of configuration backup to https://acb.netgate.com/save (success).
May 12 23:03:32 php /tmp/snort_mvneta0.409158189_startcmd.php: Beginning configuration backup to https://acb.netgate.com/save
May 12 23:03:32 check_reload_status Syncing firewall
May 12 23:03:33 php-fpm 447 /snort/snort_interfaces_global.php: End of configuration backup to https://acb.netgate.com/save (success).
May 12 23:03:36 php /tmp/snort_mvneta0.409158189_startcmd.php: End of configuration backup to https://acb.netgate.com/save (success).
May 12 23:03:36 php /tmp/snort_mvneta0.409158189_startcmd.php: [Snort] Snort START for LAN(mvneta0.4091)...
May 12 23:03:37 kernel mvneta0: promiscuous mode enabled
May 12 23:03:37 kernel mvneta0.4091: promiscuous mode enabled
May 12 23:03:55 php-cgi snort_check_for_rule_updates.php: [Snort] There is a new set of Snort Subscriber rules posted. Downloading snortrules-snapshot-29160.tar.gz...
May 12 23:04:17 php-cgi snort_check_for_rule_updates.php: [Snort] Snort Subscriber rules file update downloaded successfully
May 12 23:04:19 php-cgi snort_check_for_rule_updates.php: [Snort] Snort OpenAppID detectors are up to date...
May 12 23:04:19 php-cgi snort_check_for_rule_updates.php: [Snort] There is a new set of Snort OpenAppID RULES detectors posted. Downloading appid_rules.tar.gz...
May 12 23:04:19 php-cgi snort_check_for_rule_updates.php: [Snort] Snort OpenAppID RULES detectors file update downloaded successfully
May 12 23:04:20 php-cgi snort_check_for_rule_updates.php: [Snort] There is a new set of Snort GPLv2 Community Rules posted. Downloading community-rules.tar.gz...
May 12 23:04:21 php-cgi snort_check_for_rule_updates.php: [Snort] Snort GPLv2 Community Rules file update downloaded successfully
May 12 23:04:21 php-cgi snort_check_for_rule_updates.php: [Snort] There is a new set of Emerging Threats Open rules posted. Downloading emerging.rules.tar.gz...
May 12 23:04:22 php-cgi snort_check_for_rule_updates.php: [Snort] Emerging Threats Open rules file update downloaded successfully
May 12 23:08:39 php-cgi snort_check_for_rule_updates.php: [Snort] Updating rules configuration for: LAN ...
May 12 23:08:39 php-cgi snort_check_for_rule_updates.php: [Snort] Warning - no text rules or IPS-Policy selected for: LAN ...
May 12 23:08:39 php-cgi snort_check_for_rule_updates.php: [Snort] Building new sid-msg.map file for LAN...
May 12 23:08:39 php-cgi snort_check_for_rule_updates.php: [Snort] Snort STOP for LAN(mvneta0.4091)...
May 12 23:08:40 snort 79413 *** Caught Term-Signal
May 12 23:08:40 kernel mvneta0: promiscuous mode disabled
May 12 23:08:40 kernel mvneta0.4091: promiscuous mode disabled
May 12 23:08:43 php-cgi snort_check_for_rule_updates.php: [Snort] Snort START for LAN(mvneta0.4091)...
May 12 23:08:43 kernel mvneta0: promiscuous mode enabled
May 12 23:08:43 kernel mvneta0.4091: promiscuous mode enabled
May 12 23:08:46 php-cgi snort_check_for_rule_updates.php: [Snort] Snort has restarted with your new set of rules...
May 12 23:08:46 php-cgi snort_check_for_rule_updates.php: [Snort] The Rules update has finished. -
Snort never ever auto-updates its own code base. Only the pfSense package manager subsystem would do that, and only after an update to the pfSense version (like updating from 2.4.4 to 2.4.5) or if a user manually selects to upgrade a package. Snort never would do such a thing after a rules update. In fact, there is not even such PHP code within the entire package.
So you have something very weird going on. It's almost like maybe you have some kind of hung pfSense upgrade process happening. Have you tried rebooting the firewall? Whatever is happening is what is wiping your Snort configuration out.
-
@bmeeks said in Suricata crashes almost instantly after startup:
Snort never ever auto-updates its own code base. Only the pfSense package manager subsystem would do that, and only after an update to the pfSense version (like updating from 2.4.4 to 2.4.5) or if a user manually selects to upgrade a package. Snort never would do such a thing after a rules update. In fact, there is not even such PHP code within the entire package.
I didn't expect it would...that's good to know for sure. I have not recently updated pfSense.
So you have something very weird going on. It's almost like maybe you have some kind of hung pfSense upgrade process happening. Have you tried rebooting the firewall? Whatever is happening is what is wiping your Snort configuration out.
I rebooted it a week ago, but not for troubleshooting purposes. I may try that, although now I'm wondering if the config with "stick" or not if I leave it alone.
Speaking of weird, do you know what this log entry might mean?
May 8 17:24:53 login login on ttyu0 as root
This shouldn't be any actual user (I'm the only human managing this device), but is it some internal service account? I don't actually know what "ttyu0" is. -
@CyberMinion said in Suricata crashes almost instantly after startup:
Speaking of weird, do you know what this log entry might mean?
May 8 17:24:53 login login on ttyu0 as root
This shouldn't be any actual user (I'm the only human managing this device), but is it some internal service account? I don't actually know what "ttyu0" is.That is the serial port console for the box. Do you have something connected serially to the firewall? That entry indicates something or someone logged into the firewall console as root. Is there a cable connected to the console USB port?
-
@bmeeks said in Suricata crashes almost instantly after startup:
That is the serial port console for the box. Do you have something connected serially to the firewall? That entry indicates something or someone logged into the firewall console as root. Is there a cable connected to the console USB port?
Well that's odd. No, I do not have anything connected to a serial port or USB. The only external connections are power, WAN ethernet, and LAN ethernet. This box does not have an external serial port, although I have not dismantled it; there could be a serial port on the board. I certainly have not attached anything to it internally, either.
-
@CyberMinion said in Suricata crashes almost instantly after startup:
Well that's odd. No, I do not have anything connected to a serial port or USB. The only external connections are power, WAN ethernet, and LAN ethernet. This box does not have an external serial port, although I have not dismantled it; there could be a serial port on the board. I certainly have not attached anything to it internally, either.
The small USB port is the "serial port" for the console. It's where you should have hooked up the USB/Serial Console cable when you first configured the box. The Netgate appliances don't have a convential DB9 serial port connector. They use the mini-USB port called "Console" as their serial port.
-
@bmeeks Ok. I see MicroUSB port on the back, and two USB-A ports on the front. None are in use, nor have I ever used any.
This unit shipped with pfSense already installed and with base configuration. Although I am aware of the console option, I have never used it.