Snort + SG-3100 = exited on signal 10
-
So this may be a possible issue with the brand new SG-3100. I have not done exhaustive troubleshooting YET but this is what I know at this time:
Situation:
-
Brand new SG-3100. Just received and installed November 1, 2017. Pre-order, so I have to imagine I am one of the first to see this issue.
-
Install Snort Package and follow directions here: https://doc.pfsense.org/index.php/Setup_Snort_Package
-
Pay for VRT rules. Snort 100% up to date. (I will update post with specific versions later but up to date as of November 2 1:00AM)
-
Assign WAN interface
-
Apply VRT "Connection" policy instead of choosing specific rules
-
Start Snort
-
Snort errors near immediately with "exited on signal 10" into "exiting promiscuous mode"
What I have noticed thus far:
-
Removing all rules allows snort to continue working (makes me think class issue)
-
PreProcessor rules lack class on many of the default rules ;D
-
Signal 11 is for classification issues. I am getting signal 10. No custom rules installed. Only snort VRT
-
Signal 10 says "According to the FreeBSD docs, Signal 10 is a memory bus error" (TY @bmeeks https://forum.pfsense.org/index.php?topic=138813.0)
-
Re-install and re-configures have no effect
Food for thought from/for the Admins:
On Oct 22 (a week ago) IVOR posted in regards to running snort on the SG-1000 - "No, it's (snort) not enough powerful to run on SG-1000. We added Snort to ARM packages because of SG-3100". This makes me think that there is a bug and this is not something I am doing wrong. I also do not believe that the instructions are missing a step (particularly that you would need to fix classifications on baseline rules) before anything will work.
My PFSense T-Shoot skills are meh but I am more than willing to dump any logs or configs. Just let me know what you guys think. /Ross
-
-
Makes me believe you're having issues with the latest ruleset. Also, those log entries you wrote are not nearly enough of information. Snort should write an actual error if it's having one. Try using community ruleset (built in) instead of paid ones for a test.
Also, enable "Startup/Shutdown Logging" under Snort Global Settings to see more detailed log.
-
Thank you IVOR. I am at work now but will post better error information as soon as I am home.
-
Hmm, I'm also seeing that same issue. Only loading GPL and ET rules. Running a 2.4.2a snapshot.
Snort is not logging anything particularly useful. I'm digging further.
Suricata appears to run just fine though, so that's an option in the mean time.
Steve
-
The SG-3100 uses an ARM CPU I think Signal 10 can also be wrapped up in illegal instruction problem as well. One thing Snort has are those precompiled shared-object rules. They are in binary form. They may not work on ARM platforms, but I'm not sure. But they are generally only present and loaded when using the actual VRT rules. They are not in the ET Open or Community rules, so if using just those rules I would think the SO rules can be ruled out (no pun intended).
EDIT: A Google search found this thread about Shared Object rules and ARM – http://seclists.org/snort/2013/q2/1219
There is a command-line switch for recompile the shared-object rules. Don't know if that would work or not.
Bill
-
Hi,
I'm new here and new to PFSense. I've also recently purchased the SG-3100 which is running nicely other than having this same Snort issue. My box is on 2.4.1 and the Snort package installed using the package manager is 3.2.9.5_3. I've only registered for the free rules.
Snort stops running almost immediately, logs show snort exited on signal 10, then promiscuous mode disabled.
-
Hi,
I'm new here and new to PFSense. I've also recently purchased the SG-3100 which is running nicely other than having this same Snort issue. My box is on 2.4.1 and the Snort package installed using the package manager is 3.2.9.5_3. I've only registered for the free rules.
Snort stops running almost immediately, logs show snort exited on signal 10, then promiscuous mode disabled.
After some Google research, I'm coming to believe Snort and the ARM architecture are currently incompatible. Snort rule sets contain a set of pre-compiled rules called the dynamic shared-object rules. Those are compiled for Intel CPU platforms currently and not ARM. When Snort tries to load and run those shared-object rules, it is generating the Signal 10 error. That error can mean a memory bus problem, but it also is related to code attempting to execute an illegal instruction. In the case of the SO rules, that would be the pre-compiled rule code attempting to execute an Intel CPU instruction on an ARM CPU. That's not going to come out well… ;).
Try running with the Shared Object rules disabled. You do this by making sure none of their categories are checked on the CATEGORIES tab. The Shared Object Rules have their own sub-section on that tab. Make sure none of those categories are checked, click the Save button and then attempt to start Snort on the interface.
Bill
-
Thanks for your help Bill. That news is a little disappointing if that's the case. Perhaps things may change in the future.
I am a newbie so I want to discount the possibility I may be doing something wrong in the setup. I cannot find a separate SO tab, however under the section 'Select the rulesets (Categories) Snort will load at startup', there is a middle column labelled 'Ruleset: Snort SO Rules', with nothing shown below - I assume that means there are no SO rules enabled (problem however still persists).
Thanks again.
-
Thanks for your help Bill. That news is a little disappointing if that's the case. Perhaps things may change in the future.
I am a newbie so I want to discount the possibility I may be doing something wrong in the setup. I cannot find a separate SO tab, however under the section 'Select the rulesets (Categories) Snort will load at startup', there is a middle column labelled 'Ruleset: Snort SO Rules', with nothing shown below - I assume that means there are no SO rules enabled.
Thanks again.
On the GLOBAL SETTINGS page, what rules do you have enabled for download? Do you have a Snort Oinkcode, or are you just using the Emerging Threats or GPLv2 Community Rules? If you don't have an Oinkcode for the Snort VRT rules, then you don't have shared-object rules as they only exist in the Snort VRT package (which you must register for at snort.org and get an Oinkcode for access).
The shared-object rules are definitely one place where ARM architecture can be incompatible with the Snort package, but it's not the only place. There may be some structure/byte alignment issues as well.
Bill
-
Hmm, I'm seeing that with only the GPL rules loaded. It does seem to stay up longer without any emerging rules loaded but still crashes out eventually.
Steve
-
Hmm, I'm seeing that with only the GPL rules loaded. It does seem to stay up longer without any emerging rules loaded but still crashes out eventually.
Steve
Thanks for the feedback. I still think there are some compiler optimizations that may be needed for packages created for the ARM-based systems. While the precompiled shared-object rules can certainly be a problem, I'm betting they are not the only issue here.
I've done some limited Google research on Snort and ARM architecture, but so far have not found a lot of useful information.
Bill
-
On the GLOBAL SETTINGS page, what rules do you have enabled for download? Do you have a Snort Oinkcode, or are you just using the Emerging Threats or GPLv2 Community Rules? If you don't have an Oinkcode for the Snort VRT rules, then you don't have shared-object rules as they only exist in the Snort VRT package (which you must register for at snort.org and get an Oinkcode for access).
The shared-object rules are definitely one place where ARM architecture can be incompatible with the Snort package, but it's not the only place. There may be some structure/byte alignment issues as well.
Bill
Yes I have an Oinkcode and enabled the VRT rules. I've also tried unchecking the VRT rules and selected only the GPLv2 community rules, and then only the emerging threats open rules. In all 3 cases I get the same result, the service stops immediately (one ruleset does not appear to keep the service running any longer than the other).
Jim
-
Question for you guys with SG-3100 systems: have you tried running Snort in pure IDS mode with blocking disabled? That would potentially help narrow down the problem.
I don't have an ARM system to test with, so troubleshooting/fixing this is going to be difficult for me.
Bill
-
Question for you guys with SG-3100 systems: have you tried running Snort in pure IDS mode with blocking disabled? That would potentially help narrow down the problem.
I don't have an ARM system to test with, so troubleshooting/fixing this is going to be difficult for me.
Bill
Good suggestion, I tried unchecking the 'block offenders' option under Settings>Alert Settings, I assume that puts it into IDS mode.
Same result however, service stops :(
-
Same here. I was running in non-blocking mode anyway.
My test box for this sees virtually no traffic to speak of unless I initiate it. Snort definitely remains running for far longer with only the GPL rules loaded.
Steve
-
My suspicion is the differences between ARM CPU architecture and Intel/AMD CPU architecture and instruction sets are the root cause here, and some sequence of steps during parsing of rules triggers the Signal 10. If you are familiar with C programming and have access to the Snort binary source code, you will see lots of #ifdef types of statements in the code to detect various environments and adjust for their peculiarities. These are mostly all related to compilation for different operating systems. Some detective work would be required to figure out what is needed to compile code reliably for an ARM system and add the appropriate #ifdef statements to bound the changes. A key step in doing that is having an ARM platform to test with. I don't have one, and to my knowledge I can't emulate one reliably in a VMware virtual machine either.
Bill
-
UPDATE
An SG-3100 box is graciously being loaned to me for testing, so I will see if I can fix up the Snort package so that it runs reliably on ARM hardware. Give me a little time to get my environment set up and then do some investigation and testing.
Bill
-
UPDATE
An SG-3100 box is graciously being loaned to me for testing, so I will see if I can fix up the Snort package so that it runs reliably on ARM hardware. Give me a little time to get my environment set up and then do some investigation and testing.
Bill
Hi Bill,
Thanks, any progress on this ?. I dug up an old HP N40L Microserver, installed a NC360T dual NIC card and installed PFsense to compare to the Netgate. So far I have got good results with OpenVPN working nicely and Snort service running reliably with the same oinkcode and ruleset selected.
The N40L has an AMD Turion processor which seems compatible. Not sure which device has more grunt but so far its working ok for me. I do however want to retire this box and stick to the SG-3100 to save on power.
-
UPDATE
An SG-3100 box is graciously being loaned to me for testing, so I will see if I can fix up the Snort package so that it runs reliably on ARM hardware. Give me a little time to get my environment set up and then do some investigation and testing.
Bill
Hi Bill,
Thanks, any progress on this ?
Haven't found the offending code yet, but I do have the test/debugging environment set up. It's weird. When I run the standard Snort binary I get the crash pretty much immediately upon startup. However, when I run a Snort binary compiled with debugging symbols it runs just fine and does not crash! Scratching my head over this one … ???.
Bill
-
Ouch! I hate that. Measuring a thing changes it's behaviour. Clearly some sort of quantum behaviour. ;)
Steve
-
Ouch! I hate that. Measuring a thing changes it's behaviour. Clearly some sort of quantum behaviour. ;)
Steve
Yep. Turning on debugging symbols turns off all optimizations done by the compiler. So at least there is a hint there that maybe something in the compiler optimizations are the cause. In some subsequent runs I was able to produce a Signal 10 crash using the Snort binary with debug symbols … but only once so far. The non-debugging version crashes every single time.
Bill
-
Hi, any luck on this matter?
I
ve also bought the VRT ruleset, and my sg-3100 is still being being delivered, once it reaches me, I
ll try to help too.I read that the sg-3100 was tested by all means by Netgate before release, so I believe that they have tested snort.
The question is, which ruleset they have tested? Was VRT rules tested?
If they did test the VRT ruleset, we just need to compare the rules that we have now with the rules they tested to find the offensive code to the ARM.
obs: sorry for my english, it`s not my native language.
-
Hi, any luck on this matter?
I
ve also bought the VRT ruleset, and my sg-3100 is still being being delivered, once it reaches me, I
ll try to help too.I read that the sg-3100 was tested by all means by Netgate before release, so I believe that they have tested snort.
The question is, which ruleset they have tested? Was VRT rules tested?
If they did test the VRT ruleset, we just need to compare the rules that we have now with the rules they tested to find the offensive code to the ARM.
obs: sorry for my english, it`s not my native language.
Working on it along with one of the pfSense kernel developers. It's a complex problem, and there are several errors likely in the Snort binary's source code. Some things were done in the code that are not good programming practice, but Intel processors hide the issue because they silently fix the problem. ARM processors like the armv7 used in the SG-3100 do not silently fix the problem. The issue is unaligned memory access done by portions of the Snort binary code.
We have been able to get Snort to run without the Signal 10 error, but it's not properly decoding some of the TCP packets. It's messing up TCP sequence and ACK numbers for one thing.
Bill
-
Ah man, I knew it! I knew the problem while running on the SG-1000 was more complex than the device doesn't have the processing power for it… I mean, while that may be true, it wouldn't exit with a Signal 10 error on start up. Signal 10 (to me) indicates the ARM chip is running malformed instructions meant for an x86 target.
Good luck @bmeeks ! May the programming gods bless you on this one!
-
Ah man, I knew it! I knew the problem while running on the SG-1000 was more complex than the device doesn't have the processing power for it… I mean, while that may be true, it wouldn't exit with a Signal 10 error on start up. Signal 10 (to me) indicates the ARM chip is running malformed instructions meant for an x86 target.
Good luck @bmeeks ! May the programming gods bless you on this one!
Thanks! This is a tough nut to crack. It's not an illegal instruction that's causing the Signal 10 in this case. Instead, it's a problem with something called unaligned memory access. You can Google that term for details about what it is. It's down all the way to the register level inside the CPU and how hardware memory access has to work. The root cause is what many consider poor or bad form C language programming practice when using pointers to reference data in memory. Intel x86 CPUs swallow these kinds of programmer issues and auto-correct them. In the old days, before tons of CPU on-die cache memory and all the fancy instruction execution pipelines of modern CPUs, there was a peformance penalty each time the CPU "fixed up" a C programmer's mistake. Not so much anymore, though. Modern Intel CPUs just basically instantly fix-up the unaligned memory access and there is no perceivable performance penalty. Thus there has not been a push to fix these problems in legacy C programming code. However, other CPUs such as the armv7 used in the SG-3100 don't perform these auto-fixups by default. So you get the errors. The preferred fix is to find all the poor programming practices in the C code and fix them at the source. That is easier to say that it is to actually do… :(. We're still working on it. The problems are within sections of the Snort binary and have nothing to do with the GUI package.
Bill
-
The preferred fix is to find all the poor programming practices in the C code and fix them at the source. That is easier to say that it is to actually do… :(.
Wow, I feel that pain. :-\
Sending you good vibes! :)
Steve
-
Just jumping into this thread to say that I have the same issue, and await a fix hopefully soon.
I originally thought I had misconfigured something.
-
OP Here.
Thank you guys for looking into this. I look forward to a fix.
-Ross
-
Hey guys, until we wait for the fix, you can use Suricata instead, not a complete replacement due to the lack of appid in my opnion, but it works.
The categories mentioned below worked fine.The only category I got errors inside Suricata.log was from malware-cnc, so I've disabled it.
I'm currently using the VRT paid rules, with the following categories:
snort_blacklist.rules
snort_browser-chrome.rules
snort_browser-firefox.rules
snort_browser-plugins.rules
snort_file-multimedia.rules
snort_file-office.rules
snort_file-pdf.rules
snort_malware-backdoor.rules
snort_os-windows.rulesDisabled the stream-events rules due to huge false positives.
And a few of the decoder-events due to same reason.If you guys have any doubts, just ask.
-
Hmm, interesting. Is Suricata still running OK for you?
I initially thought this also but found Suricata crashed out after some time. However I'm re-testing it now and it's still running….so far.
Steve
-
Yes, it
s working perfectly fine. Didn
t have a single crash so far, running only on my LAN, IPS mode (blocking mode enabled), not inline, didn`t test this yet. -
Hmm, interesting. Is Suricata still running OK for you?
I initially thought this also but found Suricata crashed out after some time. However I'm re-testing it now and it's still running….so far.
Steve
Are you running in inline mode or legacy mode? From what I can tell, inline mode isn't ready yet for the 3100 due to lack of driver support, which the team is working on.
-
Running in non-blocking mode currently. One step at a time ;)
Previously it wasn't running at all from what I could see but now seems good at 24hrs+.
Steve
-
Just checking back in. Any movement getting snort fully functional on SG-3100/ARM? I'm really interested in the new app detection stuff, so running Suricata doesn't scratch the itch. Really happy with my SG-3100 so far (but for this). I'm happy to help test/troubleshoot if my rig can be of assistance.
From the thread, it looks non-trivial based on some old bad programming habits. Not sure how hard that is to track down and fix :(
Thanks for any and all help!
Sean
-
Just checking back in. Any movement getting snort fully functional on SG-3100/ARM? I'm really interested in the new app detection stuff, so running Suricata doesn't scratch the itch. Really happy with my SG-3100 so far (but for this). I'm happy to help test/troubleshoot if my rig can be of assistance.
From the thread, it looks non-trivial based on some old bad programming habits. Not sure how hard that is to track down and fix :(
Thanks for any and all help!
Sean
No firm progress yet. I did manage to find where generally in the code it is failing (at least one point). It appears to be in the loading of the Stream5 preprocessor. Debugging this has proven challenging because when I build Snort with debugging enabled it does not crash! It only crashes with debugging disabled. Without the debugging symbols being enabled, troubleshooting the crash is very difficult.
I've not had much time to troubleshoot over the Christmas holidays. Since those are winding down, I should have more time to devote to the troubleshooting task. I have an SG-3100 appliance I am testing with. It was generously provided by the pfSense team.
Bill
-
Ouch, I literally just bought one of these today because I wanted to get introduced to pfSense and things like Snort. Saw mention elsewhere it didn't work on the SG1000 but missed this about the SG3100. I'm subscribed and best of luck but I think I'm going to put in more research on Qotom.
-
Ouch, I literally just bought one of these today because I wanted to get introduced to pfSense and things like Snort. Saw mention elsewhere it didn't work on the SG1000 but missed this about the SG3100. I'm subscribed and best of luck but I think I'm going to put in more research on Qotom.
Reports from other SG-3100 users indicate Suricata works fine on the hardware. Just use Suricata for now. There is no meaningful security difference between it and Snort. The only functional difference is Snort currently offers OpenAppID while Suricata does not, but then Suricata is multi-threaded and has Inline IPS Mode while Snort does not.
Bill
-
Yes Suricata seems to run fine on the SG-3100.
The only issue with it I have seen is that the package does not survive a firmware update for some reason I've yet to determine. It requires un-installing and then re-installing (not just hitting the reinstall button) after updating.
Not a huge issue unless you're following development snapshots and updating everyday. Like me. ;)
Steve
-
Finally some progress on getting Snort to run on the SG-3100 and other Netgate hardware with the armv6/armv7 processors!
I have a Snort package in testing that runs successfully on an SG-3100. The fix involves turning off compiler optimizations for the armv7 CPU used in the SG-3100. That produces a less efficient machine code binary, but at least it runs. I am discussing options with the pfSense team to get their input on how they would like to proceed with this (meaning use the non-optimized binary package that will run, but a little bit slower than an optimized binary would; or investing time and energy to figure out exactly how to fix the optimization routines in the compiler used for armv6/armv7 packages or else alter all the places in the Snort source code where unaligned access can happen). The root cause is the compiler optimizations appear to replace some byte-oriented instructions with multi-byte equivalents to gain a little speed, but the multi-byte equivalents do not support unaligned memory access and thus cause the crash. The details get a bit geeky from there, but you can read all about unaligned memory access and how the different CPUs handle it using a Google search for "unaligned access".
Bill
-
Great news, thank you bmeeks!