Snort running on SG-1100 randomly stops working
-
No, there is no automatic restart process. And DO NOT attempt to use the Service Watchdog package with either Snort or Suricata. That package does not understand the internal workings of the IDS/IPS packages and will not monitor them properly.
Look in your pfSense system log to see if any kind of error messages are being logged relative to Snort crashing. I run Snort on my own system with the "IPS-Balanced" policy and do not ever recall finding a crash, and I have been running it for years across three different hardware platforms.
Are you running a RAM DISK? If so, that is not a good idea with the Snort and Suricata packages.
Do you have tons of other rules enabled besides just the IPS policy?
Check to see if by chance any duplicate Snort process is getting started. Running this command from a shell prompt should show only the single WAN interface process running.
ps -ax | grep snortIf you see additional duplicate lines, then kill those process IDs using this command:
kill -9 <pid> -
@bmeeks Thanks for the response.
There are no logs events that are indicating any problem. I am not using a ram disk and not using any other rules enabled other than the Balanced IPS policy. Only that might be different in terms of rules, is I did pay for the snort subscription, but don't think that matters.
The Netgate SG-1100 is hardware produced and sold by pfSense and has been working great! It's too bad that the Snort service stops without warning. Maybe a future software release will fix it.
-
I also just got this issue, I have been running suricata for a few years with no problem, but now both snort and suricata process stopps on the WAN interface.
I runs for a few minutes.
I don't run a RAM disk.
/root: ps -ax | grep snort
30634 - Ss 0:05.25 /usr/local/bin/snort -R 31246 -E -q --suppress-config-log -l /var/log/snort/snort_vmx031246 --pid-path /var/run --nolock-
65850 - Ss 0:02.73 /usr/local/bin/snort -R 63638 -D -q --suppress-config-log -l /var/log/snort/snort_vmx163638 --pid-path /var/run --nolock-
50746 0 S+ 0:00.00 grep snortone process for each interface.
-
I keep forgetting that the SG-1100 is an ARM-based appliance like the SG-3100. There are some issues with both the Snort and Suricata binaries when running on ARM-based hardware. That may be what is happening to you.
Look in the system log and see if find you any messages about a Signal 10 Bus Error. If you do, then you are hitting the ARM hardware issue and there is nothing to do but either put up with the sporadic failures or stop using the IDS/IPS package for now. I have a bug report in to the upstream Snort developer team about the ARM issue. The problem is something call unaligned access and the fact that ARM processors will not perform an auto-fixup of unaligned access when certain CPU op-codes are selected by the compiler.
-
@bmeeks said in Snort running on SG-1100 randomly stops working:
I keep forgetting that the SG-1100 is an ARM-based appliance like the SG-3100. There are some issues with both the Snort and Suricata binaries when running on ARM-based hardware. That may be what is happening to you.
Look in the system log and see if find you any messages about a Signal 10 Bus Error. If you do, then you are hitting the ARM hardware issue and there is nothing to do but either put up with the sporadic failures or stop using the IDS/IPS package for now. I have a bug report in to the upstream Snort developer team about the ARM issue. The problem is something call unaligned access and the fact that ARM processors will not perform an auto-fixup of unaligned access when certain CPU op-codes are selected by the compiler.
I cant find anything suspicous in the syslogs, no crash or error what I have seen inthe different logs so far.
-
@ekke said in Snort running on SG-1100 randomly stops working:
@bmeeks said in Snort running on SG-1100 randomly stops working:
I keep forgetting that the SG-1100 is an ARM-based appliance like the SG-3100. There are some issues with both the Snort and Suricata binaries when running on ARM-based hardware. That may be what is happening to you.
Look in the system log and see if find you any messages about a Signal 10 Bus Error. If you do, then you are hitting the ARM hardware issue and there is nothing to do but either put up with the sporadic failures or stop using the IDS/IPS package for now. I have a bug report in to the upstream Snort developer team about the ARM issue. The problem is something call unaligned access and the fact that ARM processors will not perform an auto-fixup of unaligned access when certain CPU op-codes are selected by the compiler.
I cant find anything suspicous in the syslogs, no crash or error what I have seen inthe different logs so far.
Are you sure your logs are not rolling over? pfSense uses a circular logger, so older events automatically fall out of the log as new ones are recorded. You would need to be able to see the log entries around the time period where you think the IDS/IPS crashed. If you have a low-activity firewall the logs will generally show a few days worth of data.
The Signal 10 crash is the most likely suspect for your problem. My firewall appliance is the Netgate SG-5100, an Intel-based CPU, and I have had no issues. My prior firewall hardware was also Intel (so not ARM).
-
@bmeeks I don't believe my logs are rolling over. Like the other user that commented, I don't see any even within the logs that would indicate the cause. I setup Snort, basically following the standard prompts.
Maybe it just a software bug?
-
@ekke said in Snort running on SG-1100 randomly stops working:
I have a bug report in to the upstream Snort developer team about the ARM issue.
Given that the Snort developer team is actively working on version 4.0, do you think they will release a software update anytime soon? Snort continues to stop every day. I think it's might be related to when it updates.
-
@costanzo said in Snort running on SG-1100 randomly stops working:
@ekke said in Snort running on SG-1100 randomly stops working:
I have a bug report in to the upstream Snort developer team about the ARM issue.
Given that the Snort developer team is actively working on version 4.0, do you think they will release a software update anytime soon? Snort continues to stop every day. I think it's might be related to when it updates.
I think you are confused about the Snort versions and the difference between the GUI configuration package you see on pfSense and the underlying binary that does the actual work. There is no version 4 of the Snort binary. The upstream Snort team is working on a new multithreaded product they are calling Snort3. The current Snort binary release branch is version 2.9.x (and the current version in that branch is 2.9.14).
What you see on pfSense is simply a GUI helper package that handles the behind-the-scenes configuration of the Snort binary for you. That package has a 3.x branch (in pfSense-2.4-RELEASE) and a beta 4.x branch (in pfSense-2.5-DEVEL). But the GUI package is not Snort itself, and the GUI package is not even being run unless you have an active Snort tab open. What is crashing on you is the underlying Snort binary. All the GUI package does is give you a nice graphical user interface for selecting the configuration parameter values, and then the GUI code writes that to the
snort.conffile the binary requires and then launches the binary. The binary runs as a background process.If it happens only on updates, then you can try turning on the verbose logging on the GLOBAL SETTINGS tab to capture more startup information. You will also need to drastically increase the size (number of displayed entries) of you system log when you do that as Snort is very chatty when starting. After doing all this, then look for any clues in the system log. That's where Snort will log messages related to its operation.
-
@bmeeks Thanks for the detailed explanation.
I haven't turned on verbose logging yet, but I did see the following in my log this morning. Out of swap space? Is that something I can adjust on my SG-1100 or pfSense?
Aug 1 00:08:52 kernel mvneta0.4090: promiscuous mode disabled
Aug 1 00:08:52 kernel mvneta0: promiscuous mode disabled
Aug 1 00:08:52 kernel pid 34675 (snort), uid 0, was killed: out of swap space
Aug 1 00:08:32 kernel pid 55326 (php), uid 0, was killed: out of swap space
Aug 1 00:05:45 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Updating rules configuration for: WAN ...
Aug 1 00:05:17 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Emerging Threats Open rules file update downloaded successfully
Aug 1 00:05:15 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] There is a new set of Emerging Threats Open rules posted. Downloading emerging.rules.tar.gz...
Aug 1 00:05:11 router.domain.net nginx: 2019/08/01 00:05:11 [error] 62756#100109: *104230 open() "/usr/local/www/wpad.dat" failed (2: No such file or directory), client: 192.168.10.2, server: , request: "GET /wpad.dat HTTP/2.0", host: "dyn.domain.net"
Aug 1 00:05:08 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Snort Subscriber rules are up to date... -
@costanzo said in Snort running on SG-1100 randomly stops working:
@bmeeks Thanks for the detailed explanation.
I haven't turned on verbose logging yet, but I did see the following in my log this morning. Out of swap space? Is that something I can adjust on my SG-1100 or pfSense?
Aug 1 00:08:52 kernel mvneta0.4090: promiscuous mode disabled
Aug 1 00:08:52 kernel mvneta0: promiscuous mode disabled
Aug 1 00:08:52 kernel pid 34675 (snort), uid 0, was killed: out of swap space
Aug 1 00:08:32 kernel pid 55326 (php), uid 0, was killed: out of swap space
Aug 1 00:05:45 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Updating rules configuration for: WAN ...
Aug 1 00:05:17 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Emerging Threats Open rules file update downloaded successfully
Aug 1 00:05:15 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] There is a new set of Emerging Threats Open rules posted. Downloading emerging.rules.tar.gz...
Aug 1 00:05:11 router.domain.net nginx: 2019/08/01 00:05:11 [error] 62756#100109: *104230 open() "/usr/local/www/wpad.dat" failed (2: No such file or directory), client: 192.168.10.2, server: , request: "GET /wpad.dat HTTP/2.0", host: "dyn.domain.net"
Aug 1 00:05:08 php /usr/local/pkg/snort/snort_check_for_rule_updates.php: [Snort] Snort Subscriber rules are up to date...You are hitting the RAM ceiling with the number of rules you have enabled. There is only 1 GB of RAM in the SG-1100, and enough enabled Snort rules can eat that rapidly, especially during a rules update because for a period of time Snort keeps two copies of the rules in memory (the old version and the just updated version) while it switches over to the newly downloaded set. It then drops the older set. It actually creates a new copy in RAM of your configuration but the new copy has the new rules in it. Snort then updates all of its internal pointers to reference the new copy and then the old configuration (rules and all) is removed from RAM. This "double-copy" process does not happen when you just start Snort manually from the GUI. That's why it works when you manually restart, but fails during automatic rules updates. There is not enough free RAM to hold two complete copies of the enabled rules.
You can try increasing swap space within pfSense, but if I were you I would switch my IPS policy down to "Connectivity". That provides good security without using as many rules, and so maybe then you would not hit the RAM limit. Also note that anytime you use swap things are going to be much slower, thus you want to avoid using swap space whenever possible.
Another option (well, workaround really) would be to turn off automatic rules updates and instead just set a calendar reminder for yourself to manually kick off the updates on some schedule. The Snort Subscriber rules generally update on Tuesdays and Thursdays of each week. Since you would be doing the update manually, you could easily check and make sure Snort was running after the update.
The SG-1100 is a nice little appliance, and perfect for many applications requiring a small footpriint and low power consumption, but with only 1 GB of RAM it can become a challenge to use an IDS/IPS package on it.
-
@bmeeks Thanks! I will try some of your suggestions. I think I am going to adjust the IPS policy to Connectivity.
I am very happy with the SG-1100. It's perfect for a home firewall application.
Another observation. I rebooted my firewall this morning and noticed the mem usage drop from 66% down to 31%. I am going to monitor it see if it creeps backup.
