Snort exit with signal 10 or 11 when doing certain updates
-
Hi,
I have noticed for a while that Snort exit with signal 11 and recently with signal 10 when he is making a certain kind of greater update.
snort 3.2.9.11
Jun 13 02:32:05 kernel: pid 42834 (snort), jid 0, uid 0: exited on signal 11
Starting rules update... Time: 2020-06-13 02:30:33
Downloading Snort Subscriber rules md5 file snortrules-snapshot-29160.tar.gz.md5...
Checking Snort Subscriber rules md5 file...
There is a new set of Snort Subscriber rules posted.
Downloading file 'snortrules-snapshot-29160.tar.gz'...
Done downloading rules file.
Downloading Emerging Threats Open rules md5 file emerging.rules.tar.gz.md5...
Checking Emerging Threats Open rules md5 file...
There is a new set of Emerging Threats Open rules posted.
Downloading file 'emerging.rules.tar.gz'...
Done downloading rules file.
Extracting and installing Snort Subscriber Ruleset...
Using Snort Subscriber precompiled SO rules for FreeBSD-11 ...
Installation of Snort Subscriber rules completed.
Extracting and installing Emerging Threats Open rules...
Installation of Emerging Threats Open rules completed.
Copying new config and map files...
Updating rules configuration for: DSL ...
Restarting Snort to activate the new set of rules...
Snort has restarted with your new set of rules.
The Rules update has finished. Time: 2020-06-13 02:32:10snort 3.2.9.12
Jun 17 02:31:58 kernel: pid 71836 (snort), jid 0, uid 0: exited on signal 10
Starting rules update... Time: 2020-06-17 02:30:27
Downloading Snort Subscriber rules md5 file snortrules-snapshot-29160.tar.gz.md5...
Checking Snort Subscriber rules md5 file...
There is a new set of Snort Subscriber rules posted.
Downloading file 'snortrules-snapshot-29160.tar.gz'...
Done downloading rules file.
Downloading Emerging Threats Open rules md5 file emerging.rules.tar.gz.md5...
Checking Emerging Threats Open rules md5 file...
There is a new set of Emerging Threats Open rules posted.
Downloading file 'emerging.rules.tar.gz'...
Done downloading rules file.
Extracting and installing Snort Subscriber Ruleset...
Using Snort Subscriber precompiled SO rules for FreeBSD-11 ...
Installation of Snort Subscriber rules completed.
Extracting and installing Emerging Threats Open rules...
Installation of Emerging Threats Open rules completed.
Copying new config and map files...
Updating rules configuration for: DSL ...
Restarting Snort to activate the new set of rules...
Snort has restarted with your new set of rules.
The Rules update has finished. Time: 2020-06-17 02:32:04I have searched the forum but I couldn't find any clou whats the cause. After that exit Snort is starting and working OK.
Regards,
fireodoPS. Timestamp is CEST
-
Hi,
Have you tried 3.2.9.12?
@bmeeks Bill has recently made corrections...I don't know how this relates to your case, read it here (if you haven't seen this post yet): https://forum.netgate.com/topic/154373/snort-v3-2-9-12-update-for-pfsense-2-4-5-release-notes
-
@DaddyGo said in Snort exit with signal 10 or 11 when doing certain updates:
Hi,
Have you tried 3.2.9.12?Look at my post - the second entry is with snort 3.2.9.12!
@bmeeks Bill has recently made corrections...
I don't know how this relates to your case, read it here (if you haven't seen this post yet): https://forum.netgate.com/topic/154373/snort-v3-2-9-12-update-for-pfsense-2-4-5-release-notes
Yes I have read it.
-
@fireodo
I slipped over it
-
@DaddyGo said in Snort exit with signal 10 or 11 when doing certain updates:
@fireodo
I slipped over it no problem 
-
@fireodo said in Snort exit with signal 10 or 11 when doing certain updates:
@DaddyGo said in Snort exit with signal 10 or 11 when doing certain updates:
@fireodo
I slipped over it no problem Is the hardware listed in your signature the same as what you are currently running Snort on?
The Signal 10 error is a memory bus error that is generated when software attempts an unaligned memory access (meaning the code attempted to read a memory address that was not on a word-aligned boundary). This is almost always due to issues caused by the compiler and the instructions it chooses to use coupled with the particular properties of the CPU. All Intel hardware will perform auto-fixup of non-aligned memory access and keep going. ARM hardware frequently will not do auto-fixup depending on the exact binary instruction the compiler chose to use in that section of code. So ARM devices will usually crash with Signal 10. I'm not sure about the specific AMD CPU you list in your signature. I was under the impression that all AMD devices were full Intel clones, but perhaps not in every single case.
At any rate, your Signal 10 issue is not fixable at this point, especially if it is coming from where I suspect. If this only happens when automatically shutting down during rule updates and Snort restarts successfully after the update, then you will have to live with it or else change the hardware to a native Intel CPU platform.
-
@bmeeks said in Snort exit with signal 10 or 11 when doing certain updates:
@fireodo said in Snort exit with signal 10 or 11 when doing certain updates:
@DaddyGo said in Snort exit with signal 10 or 11 when doing certain updates:
@fireodo
I slipped over it no problem Is the hardware listed in your signature the same as what you are currently running Snort on?
Yes the hard&software listed in the signature is correct and up to date.
The Signal 10 error is a memory bus error that is generated when software attempts an unaligned memory access (meaning the code attempted to read a memory address that was not on a word-aligned boundary). This is almost always due to issues caused by the compiler and the instructions it chooses to use coupled with the particular properties of the CPU. All Intel hardware will perform auto-fixup of non-aligned memory access and keep going. ARM hardware frequently will not do auto-fixup depending on the exact binary instruction the compiler chose to use in that section of code. So ARM devices will usually crash with Signal 10. I'm not sure about the specific AMD CPU you list in your signature. I was under the impression that all AMD devices were full Intel clones, but perhaps not in every single case.
Thank you very much for the detailed explanation - I have searched a lot before posting this issue but you cleared it now for me.
At any rate, your Signal 10 issue is not fixable at this point, especially if it is coming from where I suspect. If this only happens when automatically shutting down during rule updates and Snort restarts successfully after the update, then you will have to live with it or else change the hardware to a native Intel CPU platform.
That's what I think too (have to live with it) because it doesn't affect the functionality of snort - but for the peace in my mind its good to know ...
Thanks,
fireodo -
@fireodo said in Snort exit with signal 10 or 11 when doing certain updates:
@bmeeks said in Snort exit with signal 10 or 11 when doing certain updates:
@fireodo said in Snort exit with signal 10 or 11 when doing certain updates:
@DaddyGo said in Snort exit with signal 10 or 11 when doing certain updates:
@fireodo
I slipped over it no problem Is the hardware listed in your signature the same as what you are currently running Snort on?
Yes the hard&software listed in the signature is correct and up to date.
The Signal 10 error is a memory bus error that is generated when software attempts an unaligned memory access (meaning the code attempted to read a memory address that was not on a word-aligned boundary). This is almost always due to issues caused by the compiler and the instructions it chooses to use coupled with the particular properties of the CPU. All Intel hardware will perform auto-fixup of non-aligned memory access and keep going. ARM hardware frequently will not do auto-fixup depending on the exact binary instruction the compiler chose to use in that section of code. So ARM devices will usually crash with Signal 10. I'm not sure about the specific AMD CPU you list in your signature. I was under the impression that all AMD devices were full Intel clones, but perhaps not in every single case.
Thank you very much for the detailed explanation - I have searched a lot before posting this issue but you cleared it now for me.
At any rate, your Signal 10 issue is not fixable at this point, especially if it is coming from where I suspect. If this only happens when automatically shutting down during rule updates and Snort restarts successfully after the update, then you will have to live with it or else change the hardware to a native Intel CPU platform.
That's what I think too (have to live with it) because it doesn't affect the functionality of snort - but for the peace in my mind its good to know ...
Thanks,
fireodoAnd when I say "non fixable at this point", it would be more accurate to say not fixable by the pfSense team. The underlying root cause is poor C code programming practices scattered all over the Snort binary code (incorrect use of pointer casting is usually the cause of unaligned access memory bus errors). This bad C code accumulates in a large binary program such as Snort over the years. Because the code runs fine on genuine Intel hardware (due to the auto-fixup logic within Intel processors), there is no driving incentive on the upstream code maintainers/creators of Snort to invest the time and effort required to ferret out all the incorrect C code and fix it. It is not an easy task as a change you make in one place to fix the error can easily introduce a new bug in another part of the code that happens to reference the code area you changed. It turns into a mess of spaghetti code very quickly. And because the code runs fine on genuine Intel hardware, and the vast majority of users have Intel processors, the bad code lives on.
I am so familiar with this because the same issue has bitten pfSense with the ARM hardware in the SG-1000, SG-1100 and SG-3100 Netgate appliances. Bad C coding in a number of binary packages causes similar issues (Telegraph, Snort, Suricata, FRR and others).
-
@bmeeks said in [Snort exit with signal 10 or 11 when doing certain updates]
And because the code runs fine on genuine Intel hardware, and the vast majority of users have Intel processors, the bad code lives on.
Its sad to read that!
I am so familiar with this because the same issue has bitten pfSense with the ARM hardware in the SG-1000, SG-1100 and SG-3100 Netgate appliances. Bad C coding in a number of binary packages causes similar issues (Telegraph, Snort, Suricata, FRR and others).
I have read the other posts in the forum where you describe a similar issue on a ARM system.
Thanks again!
-
@fireodo said in Snort exit with signal 10 or 11 when doing certain updates:
@bmeeks said in [Snort exit with signal 10 or 11 when doing certain updates]
And because the code runs fine on genuine Intel hardware, and the vast majority of users have Intel processors, the bad code lives on.
Its sad to read that!
I investigated maybe fixing up Snort when it first malfunctioned on the new ARM hardware Netgate introduced, but it quickly became a daunting and somewhat overwhelming task since I was not the original creator of the Snort binary package. There are dozens and dozens of C source code files and nothing is well documented, so you really have no clear picture of the impact elsewhere in the C code of a change you might make.
Just to be sure everyone understands, Snort on pfSense is two separate things. There is a PHP GUI code package that I do maintain, and I do fully understand the code in that package. But all that package code does is wrap all the CLI Snort configuration information with a set of GUI screens for the user to enter data on. All that GUI info winds up getting written to a single
snort.conffile in the interface sub-directory for the Snort-enabled interface. A Snort binary program then reads that config file and uses the info to actually perform the traffic inspection. That binary module comes from the Snort programming team (as those dozens of C source code modules I mentioned earlier). When you install Snort on pfSense, there are actually two separate packages installed. One is the binary portion from upstream and the other is the PHP GUI package that creates the screens you enter configuration information on. The Signal 10 error is occuring in the code from the binary portion of Snort. -
@bmeeks said in [Snort exit with signal 10 or 11 when doing certain updates]
That binary module comes from the Snort programming team (as those dozens of C source code modules I mentioned earlier). When you install Snort on pfSense, there are actually two separate packages installed. One is the binary portion from upstream and the other is the PHP GUI package that creates the screens you enter configuration information on. The Signal 10 error is occuring in the code from the binary portion of Snort.
Is the source code for that binary a adaptation of the source-code that can be downloaded from Snort https://www.snort.org/downloads# or is it completely rewritten?
-
@fireodo said in Snort exit with signal 10 or 11 when doing certain updates:
@bmeeks said in [Snort exit with signal 10 or 11 when doing certain updates]
That binary module comes from the Snort programming team (as those dozens of C source code modules I mentioned earlier). When you install Snort on pfSense, there are actually two separate packages installed. One is the binary portion from upstream and the other is the PHP GUI package that creates the screens you enter configuration information on. The Signal 10 error is occuring in the code from the binary portion of Snort.
Is the source code for that binary a adaptation of the source-code that can be downloaded from Snort https://www.snort.org/downloads# or is it completely rewritten?
It is not an adaption, it is the actual source code. More correctly, that source package is incorporated into the FreeBSD ports binary package for snort. The only change made on pfSense is the inclusion of a custom output plugin for the blocking function. But that module is in no way related to the Signal 10 error. That is happening in a completely different area of the code.
When you compile the binary snort executable on FreeBSD, the build system will download those source files and compile from them.
-
@bmeeks said in [Snort exit with signal 10 or 11 when doing certain
But that module is in no way related to the Signal 10 error. That is happening in a completely different area of the code.
I suppose that the problematic code is in the code area that is close to the platform (Intel, AMD, ARM) architecture.
When you compile the binary snort executable on FreeBSD, the build system will download those source files and compile from them.
I understand.
-
@fireodo said in Snort exit with signal 10 or 11 when doing certain updates:
@bmeeks said in [Snort exit with signal 10 or 11 when doing certain
But that module is in no way related to the Signal 10 error. That is happening in a completely different area of the code.
I suppose that the problematic code is in the code area that is close to the platform (Intel, AMD, ARM) architecture.
It is a bit more detailed than that. In the case of an issue on ARM packages, the problem in Snort was the way a particular subroutine call used a cast memory pointer to access the content of a memory location. That pointer cast resulted in an unaligned memory access. Things get tricky and more complicated down in the compiler which is the program that turns the C source code into actual CPU op-codes (or instructions). That memory access using a pointer in C code has to be translated by the compiler into a long series of CPU op-codes. These op-codes are very specific to a given processor. In the case of the 32-bit ARM chip used in some Netgate appliances, that means converting the memory access into a series of 32-bit register loads in CPU binary instructions. The compiler has a choice of two different register load instructions in the ARMv7 CPU. One of those register loads performs an auto-fixup like Intel processors, but the other does not. The auto-fixup instruction is slightly slower than the non-fixup instruction. So when you turn on compiler optimizations for the llvm compiler used in FreeBSD, it will want to choose the faster instruction. Unfortunately that instruction won't perform the auto-fixup on unaligned memory access and thus the resulting binary code will crash with a Signal 10 when the register load instruction is executed. The solution in that case was to turn off all compiler optimizations. That results in bigger, slower binary code, but at least it would execute through that instruction.
Variations of this problem occur elsewhere in the C source for Snort (and to some degree, in Suricata as well). Other binary packages have similar issues (the ones I mentioned in my earlier post). Turning off compiler optimizations may fix some issues, but not necessarily all of them everywhere in the code. It is a very daunting problem! And the finger pointing can be extreme between the compiler programmers and the application programmers. The compiler folks scream "fix your darn code!", and the application people say "just have your compiler make safe choices for op-codes because my code works fine on other platforms!".
-
@bmeeks said in Snort exit with signal 10 or 11 when doing certain updates:
Variations of this problem occur elsewhere in the C source for Snort (and to some degree, in Suricata as well). Other binary packages have similar issues (the ones I mentioned in my earlier post). Turning off compiler optimizations may fix some issues, but not necessarily all of them everywhere in the code. It is a very daunting problem!
Oh I see - like it says in my native language: its a very deep Fountain.
The compiler folks scream "fix your darn code!", and the application people say "just have your compiler make safe choices for op-codes because my code works fine on other platforms!"
