Snort + SG-3100 = exited on signal 10
-
Ouch! I hate that. Measuring a thing changes it's behaviour. Clearly some sort of quantum behaviour. ;)
Steve
-
Ouch! I hate that. Measuring a thing changes it's behaviour. Clearly some sort of quantum behaviour. ;)
Steve
Yep. Turning on debugging symbols turns off all optimizations done by the compiler. So at least there is a hint there that maybe something in the compiler optimizations are the cause. In some subsequent runs I was able to produce a Signal 10 crash using the Snort binary with debug symbols … but only once so far. The non-debugging version crashes every single time.
Bill
-
Hi, any luck on this matter?
I
ve also bought the VRT ruleset, and my sg-3100 is still being being delivered, once it reaches me, I
ll try to help too.I read that the sg-3100 was tested by all means by Netgate before release, so I believe that they have tested snort.
The question is, which ruleset they have tested? Was VRT rules tested?
If they did test the VRT ruleset, we just need to compare the rules that we have now with the rules they tested to find the offensive code to the ARM.
obs: sorry for my english, it`s not my native language.
-
Hi, any luck on this matter?
I
ve also bought the VRT ruleset, and my sg-3100 is still being being delivered, once it reaches me, I
ll try to help too.I read that the sg-3100 was tested by all means by Netgate before release, so I believe that they have tested snort.
The question is, which ruleset they have tested? Was VRT rules tested?
If they did test the VRT ruleset, we just need to compare the rules that we have now with the rules they tested to find the offensive code to the ARM.
obs: sorry for my english, it`s not my native language.
Working on it along with one of the pfSense kernel developers. It's a complex problem, and there are several errors likely in the Snort binary's source code. Some things were done in the code that are not good programming practice, but Intel processors hide the issue because they silently fix the problem. ARM processors like the armv7 used in the SG-3100 do not silently fix the problem. The issue is unaligned memory access done by portions of the Snort binary code.
We have been able to get Snort to run without the Signal 10 error, but it's not properly decoding some of the TCP packets. It's messing up TCP sequence and ACK numbers for one thing.
Bill
-
Ah man, I knew it! I knew the problem while running on the SG-1000 was more complex than the device doesn't have the processing power for it… I mean, while that may be true, it wouldn't exit with a Signal 10 error on start up. Signal 10 (to me) indicates the ARM chip is running malformed instructions meant for an x86 target.
Good luck @bmeeks ! May the programming gods bless you on this one!
-
Ah man, I knew it! I knew the problem while running on the SG-1000 was more complex than the device doesn't have the processing power for it… I mean, while that may be true, it wouldn't exit with a Signal 10 error on start up. Signal 10 (to me) indicates the ARM chip is running malformed instructions meant for an x86 target.
Good luck @bmeeks ! May the programming gods bless you on this one!
Thanks! This is a tough nut to crack. It's not an illegal instruction that's causing the Signal 10 in this case. Instead, it's a problem with something called unaligned memory access. You can Google that term for details about what it is. It's down all the way to the register level inside the CPU and how hardware memory access has to work. The root cause is what many consider poor or bad form C language programming practice when using pointers to reference data in memory. Intel x86 CPUs swallow these kinds of programmer issues and auto-correct them. In the old days, before tons of CPU on-die cache memory and all the fancy instruction execution pipelines of modern CPUs, there was a peformance penalty each time the CPU "fixed up" a C programmer's mistake. Not so much anymore, though. Modern Intel CPUs just basically instantly fix-up the unaligned memory access and there is no perceivable performance penalty. Thus there has not been a push to fix these problems in legacy C programming code. However, other CPUs such as the armv7 used in the SG-3100 don't perform these auto-fixups by default. So you get the errors. The preferred fix is to find all the poor programming practices in the C code and fix them at the source. That is easier to say that it is to actually do… :(. We're still working on it. The problems are within sections of the Snort binary and have nothing to do with the GUI package.
Bill
-
The preferred fix is to find all the poor programming practices in the C code and fix them at the source. That is easier to say that it is to actually do… :(.
Wow, I feel that pain. :-\
Sending you good vibes! :)
Steve
-
Just jumping into this thread to say that I have the same issue, and await a fix hopefully soon.
I originally thought I had misconfigured something.
-
OP Here.
Thank you guys for looking into this. I look forward to a fix.
-Ross
-
Hey guys, until we wait for the fix, you can use Suricata instead, not a complete replacement due to the lack of appid in my opnion, but it works.
The categories mentioned below worked fine.The only category I got errors inside Suricata.log was from malware-cnc, so I've disabled it.
I'm currently using the VRT paid rules, with the following categories:
snort_blacklist.rules
snort_browser-chrome.rules
snort_browser-firefox.rules
snort_browser-plugins.rules
snort_file-multimedia.rules
snort_file-office.rules
snort_file-pdf.rules
snort_malware-backdoor.rules
snort_os-windows.rulesDisabled the stream-events rules due to huge false positives.
And a few of the decoder-events due to same reason.If you guys have any doubts, just ask.
-
Hmm, interesting. Is Suricata still running OK for you?
I initially thought this also but found Suricata crashed out after some time. However I'm re-testing it now and it's still running….so far.
Steve
-
Yes, it
s working perfectly fine. Didn
t have a single crash so far, running only on my LAN, IPS mode (blocking mode enabled), not inline, didn`t test this yet. -
Hmm, interesting. Is Suricata still running OK for you?
I initially thought this also but found Suricata crashed out after some time. However I'm re-testing it now and it's still running….so far.
Steve
Are you running in inline mode or legacy mode? From what I can tell, inline mode isn't ready yet for the 3100 due to lack of driver support, which the team is working on.
-
Running in non-blocking mode currently. One step at a time ;)
Previously it wasn't running at all from what I could see but now seems good at 24hrs+.
Steve
-
Just checking back in. Any movement getting snort fully functional on SG-3100/ARM? I'm really interested in the new app detection stuff, so running Suricata doesn't scratch the itch. Really happy with my SG-3100 so far (but for this). I'm happy to help test/troubleshoot if my rig can be of assistance.
From the thread, it looks non-trivial based on some old bad programming habits. Not sure how hard that is to track down and fix :(
Thanks for any and all help!
Sean
-
Just checking back in. Any movement getting snort fully functional on SG-3100/ARM? I'm really interested in the new app detection stuff, so running Suricata doesn't scratch the itch. Really happy with my SG-3100 so far (but for this). I'm happy to help test/troubleshoot if my rig can be of assistance.
From the thread, it looks non-trivial based on some old bad programming habits. Not sure how hard that is to track down and fix :(
Thanks for any and all help!
Sean
No firm progress yet. I did manage to find where generally in the code it is failing (at least one point). It appears to be in the loading of the Stream5 preprocessor. Debugging this has proven challenging because when I build Snort with debugging enabled it does not crash! It only crashes with debugging disabled. Without the debugging symbols being enabled, troubleshooting the crash is very difficult.
I've not had much time to troubleshoot over the Christmas holidays. Since those are winding down, I should have more time to devote to the troubleshooting task. I have an SG-3100 appliance I am testing with. It was generously provided by the pfSense team.
Bill
-
Ouch, I literally just bought one of these today because I wanted to get introduced to pfSense and things like Snort. Saw mention elsewhere it didn't work on the SG1000 but missed this about the SG3100. I'm subscribed and best of luck but I think I'm going to put in more research on Qotom.
-
Ouch, I literally just bought one of these today because I wanted to get introduced to pfSense and things like Snort. Saw mention elsewhere it didn't work on the SG1000 but missed this about the SG3100. I'm subscribed and best of luck but I think I'm going to put in more research on Qotom.
Reports from other SG-3100 users indicate Suricata works fine on the hardware. Just use Suricata for now. There is no meaningful security difference between it and Snort. The only functional difference is Snort currently offers OpenAppID while Suricata does not, but then Suricata is multi-threaded and has Inline IPS Mode while Snort does not.
Bill
-
Yes Suricata seems to run fine on the SG-3100.
The only issue with it I have seen is that the package does not survive a firmware update for some reason I've yet to determine. It requires un-installing and then re-installing (not just hitting the reinstall button) after updating.
Not a huge issue unless you're following development snapshots and updating everyday. Like me. ;)
Steve
-
Finally some progress on getting Snort to run on the SG-3100 and other Netgate hardware with the armv6/armv7 processors!
I have a Snort package in testing that runs successfully on an SG-3100. The fix involves turning off compiler optimizations for the armv7 CPU used in the SG-3100. That produces a less efficient machine code binary, but at least it runs. I am discussing options with the pfSense team to get their input on how they would like to proceed with this (meaning use the non-optimized binary package that will run, but a little bit slower than an optimized binary would; or investing time and energy to figure out exactly how to fix the optimization routines in the compiler used for armv6/armv7 packages or else alter all the places in the Snort source code where unaligned access can happen). The root cause is the compiler optimizations appear to replace some byte-oriented instructions with multi-byte equivalents to gain a little speed, but the multi-byte equivalents do not support unaligned memory access and thus cause the crash. The details get a bit geeky from there, but you can read all about unaligned memory access and how the different CPUs handle it using a Google search for "unaligned access".
Bill
-
Great news, thank you bmeeks!
-
The fix for Snort on the SG-3100 will be ready soon. It is part of these two Snort updates:
(1) Snort binary update – https://github.com/pfsense/FreeBSD-ports/pull/497
(2) Snort GUI update – https://github.com/pfsense/FreeBSD-ports/pull/496Look for these to appear in Package Manager sometime next week.
Bill
-
I upgraded to the new snort package released in pfSense package manager, 3.2.9.6. I am still seeing issues with crashing. I did a complete removal and reinstall.
Jan 29 16:59:31 kernel mvneta0: promiscuous mode disabled
Jan 29 16:59:31 kernel pid 84107 (snort), uid 0: exited on signal 10
Jan 29 16:58:44 snort 84107 Commencing packet processing (pid=84107) -
@Missionary:
I upgraded to the new snort package released in pfSense package manager, 3.2.9.6. I am still seeing issues with crashing. I did a complete removal and reinstall.
Jan 29 16:59:31 kernel mvneta0: promiscuous mode disabled
Jan 29 16:59:31 kernel pid 84107 (snort), uid 0: exited on signal 10
Jan 29 16:58:44 snort 84107 Commencing packet processing (pid=84107)Could you try going to the interface settings in Snort, then <interface>Preprocs, then under "Stream5 Target-Based Stream Reassembly", uncheck "Track and reassemble TCP sessions. Default is Checked.". Save this and see if the crashes stop… after some troubleshooting with mine this is the only way I could get it to start.</interface>
-
Did you read the release notes?
https://forum.pfsense.org/index.php?topic=143261.0
IMPORTANT INSTALLATION NOTICE
It is strongly recommended that you install this update by removing the Snort package and then installing it again instead of using the "upgrade" icon. This is because a couple of the files in the new update will be cached by the PHP process if you simply "upgrade" using the reinstall icon. The older version of the cached file will be used during the post-install steps and your rules may fail to update properly. If you remove the package completely and then install it again, there will be no cached files issue. So long as you have the "Save Settings" checkbox ticked on the GLOBAL SETTINGS tab, your Snort configuration will be retained when removing the package. That box is checked by default, but if you have ever unchecked it for some reason, be sure to check it before removing the package.If you read this warning afer you've already tried the reinstall icon, then simply manually update your rules on the UPDATES tab, start Snort if it failed to start after the upgrade, and you should be fine.
-
Yes, this is how I originally gave it a try. Now I've gone as far as uninstalling, completely removing the snort section from config.xml to make sure no settings could be carrying over, then reinstalling. I still experience crashing.
-
I tried adjusting the STREAM5 settings. Behavior of crash changed. FATAL ERROR: /usr/local/etc/snort/snort_11522_mvneta0/rules/snort.rules(6083) Unknown rule option: 'stream_size'.
I should note that I have 2 interfaces setup. I have a redundant WAN setup and am trying to set snort to monitor both of these. mvneta2 is the WAN port. mvneta0 is Opt1 which I have labeled WAN2. Prior new package release WAN2 would run but WAN would not. Now my behavior is the exact opposite. WAN will run but WAN2 will not. I did read the release notes. I did a total uninstall of snort and reinstalled.
I am very disappointed with the SG3100. I did not do my research good enough. I have an SG2440 I set up at one of my sites that works great. I went to buy another but it was end of sale. I only bought this because the end of sale page for SG2440 showed this was the recommended replacement. Guess I should have read a little deeper. I will be contacting Netgate to see if we can get the money back. Don't have a big network but need the redundant LAN as I am in Haiti and the internet here is not reliable so we have 2 providers.
-
This crash is likely related to having a rule enabled that needs the preprocessor. I am able to get it to run but only with that option disabled and minimal rules enabled.
-
I just checked my test SG-3100 and Snort is still running with all of the "default enabled" preprocessors enabled. In other words, an out-of-the-box install with several OpenAppID rule categories and the Snort Subscriber Rules "IPS Connectivity" policy enabled.
I have it running on the LAN of this test box and the WAN is not connected. Basically I have the SG-3100 sitting on my LAN. I am getting alerts for the HTTP_INSPECT stuff as I have no suppression list enabled.
Bill
-
Do you have any other packages, or anything else, setup on your test SG-3100? There must be some difference between mine and your's that causes mine to crash. Mine is used as my primary router, so I do have LAN and WAN configured. I also have many other packages installed. If you have any other suggestions I am happy to try anything to get it working.
-
This crash is likely related to having a rule enabled that needs the preprocessor. I am able to get it to run but only with that option disabled and minimal rules enabled.
Let's double-check the binary you have installed. First, are you on an SG-3100 and is it running 2.4.2?
Do
ls -l /usr/local/bin/snort
and you should get a file size of 2112260.
Next, calculate the MD5 of the binary:
md5 /usr/local/bin/snort
you should get: ```
MD5 (snort) = d68fbb7e854e4ed7d16184c0a67d611bLet me know what you have for these checks. Bill
-
Do you have any other packages, or anything else, setup on your test SG-3100? There must be some difference between mine and your's that causes mine to crash. Mine is used as my primary router, so I do have LAN and WAN configured. I also have many other packages installed. If you have any other suggestions I am happy to try anything to get it working.
Nope, no other packages. Just Snort. I was given this box to test with by the Netgate folks, and so I just stuck it on my network while I worked on getting Snort to run.
Bill
-
It looks like I am somehow getting a different binary. I am running 2.4.2_1 of pfSense.
[2.4.2-RELEASE][admin@pfsense]/root: ls -lusr/local/bin/snort
-r-xr-xr-x 1 root wheel 1377676 Jan 25 22:20 /usr/local/bin/snort
[2.4.2-RELEASE][admin@pfsense]/root: md5 /usr/local/bin/snort
MD5 (/usr/local/bin/snort) = 35d9aa2e1e46543242a4c404f015fc8dRunning snort –help gives me this version:
Version 2.9.11.1 GRE (Build 268) FreeBSD
Package manager shows 3.2.9.6 installed with snort-2.9.11.1.
-
It looks like I am somehow getting a different binary. I am running 2.4.2_1 of pfSense.
[2.4.2-RELEASE][admin@pfsense]/root: ls -lusr/local/bin/snort
-r-xr-xr-x 1 root wheel 1377676 Jan 25 22:20 /usr/local/bin/snort
[2.4.2-RELEASE][admin@pfsense]/root: md5 /usr/local/bin/snort
MD5 (/usr/local/bin/snort) = 35d9aa2e1e46543242a4c404f015fc8dRunning snort –help gives me this version:
Version 2.9.11.1 GRE (Build 268) FreeBSD
Package manager shows 3.2.9.6 installed with snort-2.9.11.1.
Yes, your binary is different. Let me investigate that and see what's going on.
Bill
-
OK, the binary that is installing is not correct. I will need to get with the pfSense team to find out why.
In my case, because I had manually installed my "fixed" binary package during testing, when I removed the Snort package from my SG-3100 the actual binary was not getting deleted. Thus even though I was removing the package and installing it fresh during subsequent testing today, my actual binary was not getting changed and my test version binary was being used again. That's why it worked for me. So the fix really works, but for some reason the build of the binary on the Netgate respository is not including my "fix".
EDIT UPDATE: found out after some investigation that one of my patch files got omitted when everything was cherry-picked into the Netgate/pfSense repository. I've notified the pfSense team and they should get things squared away soon. When I get confirmation of the fixed binary being posted, I will post a message to this thread. SG-3100 users can then once again remove and reinstall the Snort package to get the fixed binary.
Sorry for the trouble … ;). I knew it was working on my end, so when I saw reports here to the contrary I was baffled at first. Glad to figure out what actually happened.
Bill
-
OK, the binary that is installing is not correct. I will need to get with the pfSense team to find out why.
In my case, because I had manually installed my "fixed" binary package during testing, when I removed the Snort package from my SG-3100 the actual binary was not getting deleted. Thus even though I was removing the package and installing it fresh during subsequent testing today, my actual binary was not getting changed and my test version binary was being used again. That's why it worked for me. So the fix really works, but for some reason the build of the binary on the Netgate respository is not including my "fix".
EDIT UPDATE: found out after some investigation that one of my patch files got omitted when everything was cherry-picked into the Netgate/pfSense repository. I've notified the pfSense team and they should get things squared away soon. When I get confirmation of the fixed binary being posted, I will post a message to this thread. SG-3100 users can then once again remove and reinstall the Snort package to get the fixed binary.
Sorry for the trouble … ;). I knew it was working on my end, so when I saw reports here to the contrary I was baffled at first. Glad to figure out what actually happened.
Bill
Thanks for the update! I am glad it was something simple and not another issue! :)
-
Patch is ready or not?
-
Patch is ready or not?
The patch has been ready since January 18th, but when my submitted files for the last Snort update got merged into the pfSense repository one of the patch files for the binary was accidentally omitted during the cherry pick process. I notified the pfSense team this past Monday evening of the oversight and provided them another copy of the missing file. The new package is not yet posted, though.
Bill
-
Patch is ready or not?
The patch has been ready since January 18th, but when my submitted files for the last Snort update got merged into the pfSense repository one of the patch files for the binary was accidentally omitted during the cherry pick process. I notified the pfSense team this past Monday evening of the oversight and provided them another copy of the missing file. The new package is not yet posted, though.
Bill
Thanks Bill, I almost installed the previous version, I`ll be waiting, thanks for everything.
Best regards,
-
It will be there soon, apologies for the wait!