Suricata crashes almost instantly after startup
-
@CyberMinion said in Suricata crashes almost instantly after startup:
@bmeeks Yup, you're right.
May 6 18:17:09 kernel pid 95026 (suricata), jid 0, uid 0: exited on signal 4 (core dumped)
That's unfortunate, but good to know. Thank you! I had snort installed, but when pfsense updated, snort also requred an update. That update refused to install, so I was stuck with nothing. I figured I'd give Suricata a try. I guess I just can't do an IDS an sg-1100 anymore? You'd think they would care a bit more about their paying customers...oh well.
Bro doesn't work on pfsense, does it?
Not sure about Bro. There certainly is no GUI for it. You would have to install it from a third-party repo and that risks breaking your system. Also note the SG-1100 is NOT an Intel or AMD platform. You must use binary packages compiled for the aarch64 architecture.
The problem with Suricata is the forced inclusion of Rust by the upstream Suricata team. They made Rust mandatory starting with Suricata 5.x, and Rust has lots of issues on non-Intel/AMD hardware. The Rust library is probably where the illegal instruction thing is happening. I hate Rust and I hate that the upstream Suricata team adopted it and made it mandatory for version 5.x and up. But they didn't ask me about it ... .
-
@bmeeks said in Suricata crashes almost instantly after startup:
Not sure about Bro. There certainly is no GUI for it. You would have to install it from a third-party repo and that risks breaking your system. Also note the SG-1100 is NOT an Intel or AMD platform. You must use binary packages compiled for the aarch64 architecture.
The problem with Suricata is the forced inclusion of Rust by the upstream Suricata team. They made Rust mandatory starting with Suricata 5.x, and Rust has lots of issues on non-Intel/AMD hardware. The Rust library is probably where the illegal instruction thing is happening. I hate Rust and I hate that the upstream Suricata team adopted it and made it mandatory for version 5.x and up. But they didn't ask me about it ... .
Oh, I didn't realize it was using Rust now. I know a lot of devs who are thrilled about Rust, so I'm not really surprised. I suppose they see this as an unimportant edge case. It is possible to get Suricata 4.x running on here instead?
Do I assume correctly that Suricata wouldn't work any better on Raspbery Pi (4b)? It's still ARM, so I'm assuming the same problems would occur, unless there's a band-aid solution.
-
@CyberMinion said in Suricata crashes almost instantly after startup:
@bmeeks said in Suricata crashes almost instantly after startup:
Not sure about Bro. There certainly is no GUI for it. You would have to install it from a third-party repo and that risks breaking your system. Also note the SG-1100 is NOT an Intel or AMD platform. You must use binary packages compiled for the aarch64 architecture.
The problem with Suricata is the forced inclusion of Rust by the upstream Suricata team. They made Rust mandatory starting with Suricata 5.x, and Rust has lots of issues on non-Intel/AMD hardware. The Rust library is probably where the illegal instruction thing is happening. I hate Rust and I hate that the upstream Suricata team adopted it and made it mandatory for version 5.x and up. But they didn't ask me about it ... .
Oh, I didn't realize it was using Rust now. I know a lot of devs who are thrilled about Rust, so I'm not really surprised. I suppose they see this as an unimportant edge case. It is possible to get Suricata 4.x running on here instead?
Do I assume correctly that Suricata wouldn't work any better on Raspbery Pi (4b)? It's still ARM, so I'm assuming the same problems would occur, unless there's a band-aid solution.
Yes, Rust is the new "cool" language to use. However, I fail to see the real use case. It takes for-damn-ever to compile the Rust environment, then you have to use that to compile the actual application using Rust (in this case, Suricata). In my view it is simply the new "Java", or "Ruby" or "go" or pick your poison. Each of these languages promises the moon and swears it will revolutionize programming. But I'm still waiting ... what do you hear about Ruby or Java today?
Okay, rant off --
Rust is the main problem. It was optional in Suricata 4.x, but upstream decided to make it mandatory in 5.x and later.
We created a Suricata4 package that retains the Suricata 4.x binary without Rust. That package is, for now, limited to just the 32-bit ARM platform used in the SG-3100 and similar appliances. The idea was to transition the 64-bit platforms to Suricata 5.x with Rust.
Another wrinkle in this is the llvm compiler used to produce ARM code. It seemingly has issues when optimization is enabled and the C source code has some unusual pointer casts in it. This was an issue even in the Suricata 4.x and Snort packages on 32-bit ARM platforms. The source of that problem is Intel and AMD processors will perform automatic fix-up of memory accesses that are not word-aligned. There is a tiny performance penalty for doing so, but at least all code will compile and execute properly. The ARM processors do that automatic fix-up for most but not ALL instructions. Thus a compiler, when optimizing a section of code, may choose to use one of those instructions that will not perform automatic fix-up on non-word aligned memory access. In that case, the executing binary will crash. That was the optimization issue I was speaking of earlier.
So the long sad story is hardware and appliance vendors should stick to Intel/AMD CPU platforms if they want to guarantee that all software out there will run on their box. That's just my humble opinion ... . Using hardware (CPU mostly) that does not operate the same as Intel/AMD invites potential troubles when you try to run all of the sea of C source-code binaries out there where programmers over the years have taken various "liberties" with their coding (in particular with the use of pointer casts). Since Intel owns most of the hardware world, and the flaky source code compiles and runs fine on Intel platforms, the developers of that source code have little or no incentive to spend the large amount of time it would take to find all the possible "gotchas" with pointer casting, fix them, and test to be sure nothing new was broken as a result of monkeying with the pointers.
-
Good to know, thanks! I've had very limited experience with ARM systems, but I can see (first hand, atm) that being a problem.
Anyway, I uninstalled Suricata, and Installed Snort. The latter is running, but having some minor issues (keeps loosing config). However, I still have Suricata listed as an option in the Services menu. Is there a way for me to clear that out?
-
@CyberMinion said in Suricata crashes almost instantly after startup:
Good to know, thanks! I've had very limited experience with ARM systems, but I can see (first hand, atm) that being a problem.
Anyway, I uninstalled Suricata, and Installed Snort. The latter is running, but having some minor issues (keeps loosing config). However, I still have Suricata listed as an option in the Services menu. Is there a way for me to clear that out?
Sounds like you still have some sort of fundamental problem with your pfSense installation on that box. Snort should not ever lose its configuration. That, coupled with the fact you say you "removed" Suricata but it still shows up in the SERVICES menu, leads me to suspect some issue with the basic configuration of your box.
How did you "remove" Suriata? Did you go to SYSTEM > PACKAGE MANAGER and then on the Installed Packages tab clicked the trash can icon to delete the package? If so, that process should have completed and shown you a green bar on the page. That will remove the entry under SERVICES. If an entry for Suricata remains under SERVICES, then the package removal did not complete.
Or else you may have restored an older
config.xml
configuration from a backup made while Suricata was still installed. Have you perhaps done that? And if you are restoring some olderconfig.xml
backup taken before you installed the Snort package, that could account for your Snort configuration "disappearing". Everything all packages need for their configuration lives in theconfig.xml
file on the firewall. -
How did you "remove" Suriata? Did you go to SYSTEM > PACKAGE MANAGER and then on the Installed Packages tab clicked the trash can icon to delete the package? If so, that process should have completed and shown you a green bar on the page. That will remove the entry under SERVICES. If an entry for Suricata remains under SERVICES, then the package removal did not complete.
Yes, I used the package manager to uninstall it, and it reported success. Suricata is no longer listed as an installed package, either.
Or else you may have restored an older
config.xml
configuration from a backup made while Suricata was still installed. Have you perhaps done that? And if you are restoring some olderconfig.xml
backup taken before you installed the Snort package, that could account for your Snort configuration "disappearing". Everything all packages need for their configuration lives in theconfig.xml
file on the firewall.I have not restored anything on this device in the past year, although I did recently set up automatic backups, just in case.
So I set up Snort on the WAN0 interface, added my auth codes, etc. and started it up. It ran for two days, and started issuing some alerts (I'm fine-tuning now). Then two days later, it just lost all of the config. According to the logs, it was updating regularly, as configured for that time, then it just stopped...no new log entries from Snort. All of the config was blank, as if I had just installed it. So, I went through and did it all again, and started it up. It initialized, and started posting alerts again. Then, I added a few alerts to the suppress list (less than 5, defiantly not too many). Three days later, I checked on it again, and saw the same type of alerts I had previously suppressed! So, I checked the suppression tab, and found that the suppression list was gone. So, I created a new one, and also manually made another backup.. I just checked again, and everything still as it should be.
I suppose I could run a factory reset, but I would prefer not to.
-
@CyberMinion said in Suricata crashes almost instantly after startup:
So, I went through and did it all again, and started it up. It initialized, and started posting alerts again. Then, I added a few alerts to the suppress list (less than 5, defiantly not too many). Three days later, I checked on it again, and saw the same type of alerts I had previously suppressed! So, I checked the suppression tab, and found that the suppression list was gone. So, I created a new one, and also manually made another backup.. I just checked again, and everything still as it should be.
That is just plain weird. Snort simply calls a pfSense system function to write information to the
config.xml
file. That is where all configuration information for the entire firewall and any installed packages is stored.Have you looked in the pfSense system log to see if anything unusual is being logged? I have never heard of any package just randomly losing all of its configuration information from the configuration file.
-
@bmeeks
Ok, I may have found something. Snort was working fine, checking for updates periodically.May 8 16:18:25 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] The Rules update has finished.
Then, the next log entry is this:
May 8 17:21:25 php-cgi rc.bootup: Beginning package installation for snort .
Which is odd, since it was already installed. Maybe it was an auto-update? I didn't tell it to do anything.
May 8 17:24:29 php-cgi rc.bootup: [Snort] Package post-installation tasks completed...
May 8 17:24:32 php-cgi rc.bootup: Successfully installed package: snort.After that, I eventually found it offline, and went to check and see if I had a recent backup. I didn't so I told it to make one. Then I configured it again, and saved it again. The log dump is below. Oddly enough, it seems to show multiple successful backups for each time It ran the process. After this, it seemed to keep working, except for the loss of suppress rules, which I think I noticed on the 14th (yesterday). There are no relevant log entries for that which I could find.
May 12 22:56:08 php-fpm 447 /snort/snort_interfaces_edit.php: Beginning configuration backup to https://acb.netgate.com/save
May 12 22:56:13 php-fpm 447 /snort/snort_interfaces_edit.php: End of configuration backup to https://acb.netgate.com/save (success).
May 12 22:56:16 check_reload_status Syncing firewall
May 12 22:56:16 php-fpm 447 /snort/snort_interfaces_edit.php: Beginning configuration backup to https://acb.netgate.com/save
May 12 22:56:21 php-fpm 447 /snort/snort_interfaces_edit.php: End of configuration backup to https://acb.netgate.com/save (success).
May 12 22:56:23 check_reload_status Syncing firewall
May 12 22:56:23 php-fpm 447 /snort/snort_interfaces_edit.php: Beginning configuration backup to https://acb.netgate.com/save
May 12 22:56:27 php-fpm 447 /snort/snort_interfaces_edit.php: End of configuration backup to https://acb.netgate.com/save (success).
May 12 22:56:30 check_reload_status Syncing firewall
May 12 22:56:30 php-fpm 447 /snort/snort_interfaces_edit.php: Beginning configuration backup to https://acb.netgate.com/save
May 12 22:56:35 php-fpm 447 /snort/snort_interfaces_edit.php: End of configuration backup to https://acb.netgate.com/save (success).
May 12 23:01:12 php-fpm 13038 /snort/snort_interfaces.php: Starting Snort on LAN(mvneta0.4091) per user request...
May 12 23:01:13 php /tmp/snort_mvneta0.409158189_startcmd.php: [Snort] Updating rules configuration for: LAN ...
May 12 23:01:13 php /tmp/snort_mvneta0.409158189_startcmd.php: [Snort] Warning - no text rules or IPS-Policy selected for: LAN ...
May 12 23:01:13 php /tmp/snort_mvneta0.409158189_startcmd.php: [Snort] Building new sid-msg.map file for LAN...
May 12 23:01:13 php /tmp/snort_mvneta0.409158189_startcmd.php: [Snort] Snort START for LAN(mvneta0.4091)...
May 12 23:01:13 kernel mvneta0: promiscuous mode enabled
May 12 23:01:13 kernel mvneta0.4091: promiscuous mode enabled
May 12 23:03:01 php-fpm 447 /snort/snort_interfaces_global.php: Beginning configuration backup to https://acb.netgate.com/save
May 12 23:03:01 check_reload_status Syncing firewall
May 12 23:03:05 php-fpm 447 /snort/snort_interfaces_global.php: End of configuration backup to https://acb.netgate.com/save (success).
May 12 23:03:06 php-fpm 57637 /snort/snort_interfaces.php: Restarting Snort on LAN(mvneta0.4091) per user request...
May 12 23:03:06 php-fpm 57637 /snort/snort_interfaces.php: [Snort] Snort STOP for LAN(mvneta0.4091)...
May 12 23:03:07 snort 80845 *** Caught Term-Signal
May 12 23:03:07 kernel mvneta0: promiscuous mode disabled
May 12 23:03:07 kernel mvneta0.4091: promiscuous mode disabled
May 12 23:03:09 php /tmp/snort_mvneta0.409158189_startcmd.php: [Snort] Updating rules configuration for: LAN ...
>May 12 23:03:09 php /tmp/snort_mvneta0.409158189_startcmd.php: [Snort] Warning - no text rules or IPS-Policy selected for: LAN ...
May 12 23:03:09 php /tmp/snort_mvneta0.409158189_startcmd.php: [Snort] Building new sid-msg.map file for LAN...
May 12 23:03:11 php-fpm 447 /snort/snort_interfaces_global.php: Beginning configuration backup to https://acb.netgate.com/save
May 12 23:03:11 check_reload_status Syncing firewall
May 12 23:03:16 php-fpm 447 /snort/snort_interfaces_global.php: End of configuration backup to https://acb.netgate.com/save (success).
May 12 23:03:16 check_reload_status Syncing firewall
May 12 23:03:16 php /tmp/snort_mvneta0.409158189_startcmd.php: Beginning configuration backup to https://acb.netgate.com/save
May 12 23:03:21 php /tmp/snort_mvneta0.409158189_startcmd.php: End of configuration backup to https://acb.netgate.com/save (success).
May 12 23:03:21 php-fpm 447 /snort/snort_interfaces_global.php: Beginning configuration backup to https://acb.netgate.com/save
May 12 23:03:21 check_reload_status Syncing firewall
May 12 23:03:24 php /tmp/snort_mvneta0.409158189_startcmd.php: Beginning configuration backup to https://acb.netgate.com/save
May 12 23:03:24 check_reload_status Syncing firewall
May 12 23:03:26 php-fpm 447 /snort/snort_interfaces_global.php: End of configuration backup to https://acb.netgate.com/save (success).
May 12 23:03:29 php-fpm 447 /snort/snort_interfaces_global.php: Beginning configuration backup to https://acb.netgate.com/save
May 12 23:03:29 check_reload_status Syncing firewall
May 12 23:03:29 php /tmp/snort_mvneta0.409158189_startcmd.php: End of configuration backup to https://acb.netgate.com/save (success).
May 12 23:03:32 php /tmp/snort_mvneta0.409158189_startcmd.php: Beginning configuration backup to https://acb.netgate.com/save
May 12 23:03:32 check_reload_status Syncing firewall
May 12 23:03:33 php-fpm 447 /snort/snort_interfaces_global.php: End of configuration backup to https://acb.netgate.com/save (success).
May 12 23:03:36 php /tmp/snort_mvneta0.409158189_startcmd.php: End of configuration backup to https://acb.netgate.com/save (success).
May 12 23:03:36 php /tmp/snort_mvneta0.409158189_startcmd.php: [Snort] Snort START for LAN(mvneta0.4091)...
May 12 23:03:37 kernel mvneta0: promiscuous mode enabled
May 12 23:03:37 kernel mvneta0.4091: promiscuous mode enabled
May 12 23:03:55 php-cgi snort_check_for_rule_updates.php: [Snort] There is a new set of Snort Subscriber rules posted. Downloading snortrules-snapshot-29160.tar.gz...
May 12 23:04:17 php-cgi snort_check_for_rule_updates.php: [Snort] Snort Subscriber rules file update downloaded successfully
May 12 23:04:19 php-cgi snort_check_for_rule_updates.php: [Snort] Snort OpenAppID detectors are up to date...
May 12 23:04:19 php-cgi snort_check_for_rule_updates.php: [Snort] There is a new set of Snort OpenAppID RULES detectors posted. Downloading appid_rules.tar.gz...
May 12 23:04:19 php-cgi snort_check_for_rule_updates.php: [Snort] Snort OpenAppID RULES detectors file update downloaded successfully
May 12 23:04:20 php-cgi snort_check_for_rule_updates.php: [Snort] There is a new set of Snort GPLv2 Community Rules posted. Downloading community-rules.tar.gz...
May 12 23:04:21 php-cgi snort_check_for_rule_updates.php: [Snort] Snort GPLv2 Community Rules file update downloaded successfully
May 12 23:04:21 php-cgi snort_check_for_rule_updates.php: [Snort] There is a new set of Emerging Threats Open rules posted. Downloading emerging.rules.tar.gz...
May 12 23:04:22 php-cgi snort_check_for_rule_updates.php: [Snort] Emerging Threats Open rules file update downloaded successfully
May 12 23:08:39 php-cgi snort_check_for_rule_updates.php: [Snort] Updating rules configuration for: LAN ...
May 12 23:08:39 php-cgi snort_check_for_rule_updates.php: [Snort] Warning - no text rules or IPS-Policy selected for: LAN ...
May 12 23:08:39 php-cgi snort_check_for_rule_updates.php: [Snort] Building new sid-msg.map file for LAN...
May 12 23:08:39 php-cgi snort_check_for_rule_updates.php: [Snort] Snort STOP for LAN(mvneta0.4091)...
May 12 23:08:40 snort 79413 *** Caught Term-Signal
May 12 23:08:40 kernel mvneta0: promiscuous mode disabled
May 12 23:08:40 kernel mvneta0.4091: promiscuous mode disabled
May 12 23:08:43 php-cgi snort_check_for_rule_updates.php: [Snort] Snort START for LAN(mvneta0.4091)...
May 12 23:08:43 kernel mvneta0: promiscuous mode enabled
May 12 23:08:43 kernel mvneta0.4091: promiscuous mode enabled
May 12 23:08:46 php-cgi snort_check_for_rule_updates.php: [Snort] Snort has restarted with your new set of rules...
May 12 23:08:46 php-cgi snort_check_for_rule_updates.php: [Snort] The Rules update has finished. -
Snort never ever auto-updates its own code base. Only the pfSense package manager subsystem would do that, and only after an update to the pfSense version (like updating from 2.4.4 to 2.4.5) or if a user manually selects to upgrade a package. Snort never would do such a thing after a rules update. In fact, there is not even such PHP code within the entire package.
So you have something very weird going on. It's almost like maybe you have some kind of hung pfSense upgrade process happening. Have you tried rebooting the firewall? Whatever is happening is what is wiping your Snort configuration out.
-
@bmeeks said in Suricata crashes almost instantly after startup:
Snort never ever auto-updates its own code base. Only the pfSense package manager subsystem would do that, and only after an update to the pfSense version (like updating from 2.4.4 to 2.4.5) or if a user manually selects to upgrade a package. Snort never would do such a thing after a rules update. In fact, there is not even such PHP code within the entire package.
I didn't expect it would...that's good to know for sure. I have not recently updated pfSense.
So you have something very weird going on. It's almost like maybe you have some kind of hung pfSense upgrade process happening. Have you tried rebooting the firewall? Whatever is happening is what is wiping your Snort configuration out.
I rebooted it a week ago, but not for troubleshooting purposes. I may try that, although now I'm wondering if the config with "stick" or not if I leave it alone.
Speaking of weird, do you know what this log entry might mean?
May 8 17:24:53 login login on ttyu0 as root
This shouldn't be any actual user (I'm the only human managing this device), but is it some internal service account? I don't actually know what "ttyu0" is. -
@CyberMinion said in Suricata crashes almost instantly after startup:
Speaking of weird, do you know what this log entry might mean?
May 8 17:24:53 login login on ttyu0 as root
This shouldn't be any actual user (I'm the only human managing this device), but is it some internal service account? I don't actually know what "ttyu0" is.That is the serial port console for the box. Do you have something connected serially to the firewall? That entry indicates something or someone logged into the firewall console as root. Is there a cable connected to the console USB port?
-
@bmeeks said in Suricata crashes almost instantly after startup:
That is the serial port console for the box. Do you have something connected serially to the firewall? That entry indicates something or someone logged into the firewall console as root. Is there a cable connected to the console USB port?
Well that's odd. No, I do not have anything connected to a serial port or USB. The only external connections are power, WAN ethernet, and LAN ethernet. This box does not have an external serial port, although I have not dismantled it; there could be a serial port on the board. I certainly have not attached anything to it internally, either.
-
@CyberMinion said in Suricata crashes almost instantly after startup:
Well that's odd. No, I do not have anything connected to a serial port or USB. The only external connections are power, WAN ethernet, and LAN ethernet. This box does not have an external serial port, although I have not dismantled it; there could be a serial port on the board. I certainly have not attached anything to it internally, either.
The small USB port is the "serial port" for the console. It's where you should have hooked up the USB/Serial Console cable when you first configured the box. The Netgate appliances don't have a convential DB9 serial port connector. They use the mini-USB port called "Console" as their serial port.
-
@bmeeks Ok. I see MicroUSB port on the back, and two USB-A ports on the front. None are in use, nor have I ever used any.
This unit shipped with pfSense already installed and with base configuration. Although I am aware of the console option, I have never used it.