Snort 2.9.6.0 - Alerts not being logged
-
Since I upgraded to Snort 2.9.6.0 pkg v3.0.5, alerts are being triggered but not properly logged.
The alert IS being written to the system.log.
The alert IS NOT being written to the /var/log/snort/snort_xx/alert log. Obviously is not being see on the alert tab or the dashboard widget.
I have done an uninstall/install. Restart service. Nothing is kicking in the logging.
This was working before the upgrade. I uninstalled the previous package before installing 2.9.6.0
Any ideas?
==== EDIT: FOUND THE PROBLEM =====
Enabling "IP Reputation Preprocessor" on an interface breaks logging.
This reproducible. Enable "IP Reputation Preprocessor" and logging stops. Disable and logging starts again.
-
Since I upgraded to Snort 2.9.6.0 pkg v3.0.5, alerts are being triggered but not properly logged.
The alert IS being written to the system.log.
The alert IS NOT being written to the /var/log/snort/snort_xx/alert log. Obviously is not being see on the alert tab or the dashboard widget.
I have done an uninstall/install. Restart service. Nothing is kicking in the logging.
This was working before the upgrade. I uninstalled the previous package before installing 2.9.6.0
Any ideas?
==== EDIT: FOUND THE PROBLEM =====
Enabling "IP Reputation Preprocessor" on an interface breaks logging.
This reproducible. Enable "IP Reputation Preprocessor" and logging stops. Disable and logging starts again.
priller:
Is this still a problem for you? I am unable to reproduce in my virtual machine test environment. Specifically I have a 2.1.2 VM running the latest Snort with IP REP enabled. I am still getting full alert log output and ALERTS tab output.
Bill
-
I've re-enable IP REP and will see how the overnight alerts look.
-
Bill …
Confirmed that the problem still exists.
Suricata runs the same rules and there were many events overnight. There were no events LOGGED in Snort.
As I mentioned, Snort is working and putting blocks in place. It's just that alert logging stops when I enable IP Reputation.
I am on 2.1.2-RELEASE (amd64).
-
Bill …
Confirmed that the problem still exists.
Suricata runs the same rules and there were many events overnight. There were no events LOGGED in Snort.
As I mentioned, Snort is working and putting blocks in place. It's just that alert logging stops when I enable IP Reputation.
I am on 2.1.2-RELEASE (amd64).
OK, I will work on reproducing that and testing a fix today. I have all of the other reported bugs fixed. Thanks for reporting it.
Bill
-
I have a box with Snort and Suricata that doesn't have this issue; however, on one box I had Snort running and a Suricata installation that didnt have any interfaces configured. After downloading Suricata, I released I didnt have enough memory on that Box to run both Snort/Suricata as a test comparison.
I did notice the same issue with the IPrep Processor not alerting alerts.
I uninstalled Suricata and it seemed to fix the issue with the IPrep processors alerts.
NB - I posted on another thread for Bill. Maybe it got mixed in with the other topics just an fyi. Sorry for stealing the thread.
https://forum.pfsense.org/index.php?topic=74486.msg410471#msg410471 -
I have tried to reproduce this problem on my production home firewall and on some test VMs and have been unable to replicate the problem with logging stopping after enabling the IP REP preprocessor. All of my machines (and the VMs) are using 2 GB of RAM or greater. My production firewall actually has 16 GB. Maybe it is a memory resource issue? Try trimming the amount of memory reserved by the IP REP preprocessor. The default is 500 MB. You could try 50 MB, 75 MB or 100 MB instead. The setting is on the IP REP tab for an interface. Restart Snort on the interface after making the change.
Bill
-
Well, thanks for checking. But, I can reproduce this reliably. I'll try enabling IP REP on just one interface and see what happens.
Other than that, if somebody else has a sighting of this behavior, please report it.
Not a resource limitation ….
-
Well, thanks for checking. But, I can reproduce this reliably. I'll try enabling IP REP on just one interface and see what happens.
Other than that, if somebody else has a sighting of this behavior, please report it.
Not a resource limitation ….
Do the problem firewalls run both Suricata and Snort, or just Snort? I really can't imagine what the connection would be with IP REP logging being enabled and loss of alert logging.
Bill
-
I normally have both Suricata and Snort running.
I uninstalled Suricata just make sure having both running wasn't somehow related. However, the problem was still there. Did a restart of Snort after Suricata was uninstalled, just to make sure it was 'clean'.
-
I normally have both Suricata and Snort running.
I uninstalled Suricata just make sure having both running wasn't somehow related. However, the problem was still there. Did a restart of Snort after Suricata was uninstalled, just to make sure it was 'clean'.
Just to be sure I am correctly understanding the problem –-
On a given firewall, with a Snort-configured interface, if you enable the IP REP preprocessor for the interface it stops logging new alerts on the ALERTS tab but new blocks show up on the BLOCKED tab. Is that correct? And if you then disable the IP REP preprocessor, new alerts start appearing on the ALERTS tab and also still show up on the BLOCKED tab. Is this last statement also correct?
Bill
-
Just to be sure I am correctly understanding the problem –-
On a given firewall, with a Snort-configured interface, if you enable the IP REP preprocessor for the interface it stops logging new alerts on the ALERTS tab but new blocks show up on the BLOCKED tab. Is that correct? And if you then disable the IP REP preprocessor, new alerts start appearing on the ALERTS tab and also still show up on the BLOCKED tab. Is this last statement also correct?
Correct on all points.
To expand on the 'no alerts', there is also nothing in the system log. So, it's not just a GUI thing.
I did try enabling IP REP on just one interface (out of 3) and alerts stop for all interfaces, not just the one with IP REP enabled.
-
Just to be sure I am correctly understanding the problem –-
On a given firewall, with a Snort-configured interface, if you enable the IP REP preprocessor for the interface it stops logging new alerts on the ALERTS tab but new blocks show up on the BLOCKED tab. Is that correct? And if you then disable the IP REP preprocessor, new alerts start appearing on the ALERTS tab and also still show up on the BLOCKED tab. Is this last statement also correct?
Correct on all points.
To expand on the 'no alerts', there is also nothing in the system log. So, it's not just a GUI thing.
I did try enabling IP REP on just one interface (out of 3) and alerts stop for all interfaces, not just the one with IP REP enabled.
Do you have a VMware or other virtual machine environment where you could try a green field install of pfSense and Snort and see if the problem exists there? I have three interfaces on my home network firewall with Snort running on all three. I have the IP REP preprocessor enabled on the WAN side. My alerts logging is working fine.
I'm not doubting your result, but just am totally baffled about what could be the underlying cause.
EDIT: one other quick question – is this a standard pfSense ISO image on a conventional hard disk, or is this a CF install?
Bill
-
Do you have a VMware or other virtual machine environment where you could try a green field install of pfSense and Snort and see if the problem exists there?
I do. I'll see if I can reproduce there.
EDIT: one other quick question – is this a standard pfSense ISO image on a conventional hard disk, or is this a CF install?
conventional hard disk (SSD)
Also …. If you are interested in seeing the problem live, I'm happy to create a WebEx so you can poke around. Just PM me.
-
Do you have a VMware or other virtual machine environment where you could try a green field install of pfSense and Snort and see if the problem exists there?
I do. I'll see if I can reproduce there.
EDIT: one other quick question – is this a standard pfSense ISO image on a conventional hard disk, or is this a CF install?
conventional hard disk (SSD)
Also …. If you are interested in seeing the problem live, I'm happy to create a WebEx so you can poke around. Just PM me.
Sent you a PM.
Bill
-
As a troubleshooting measure, I rebooted the whole box. Snort is not starting now. Does the following message make sense?
Unable to open rules file "/usr/pbi/snort-amd64/etc/snort/snort_47247_igb1//usr/pbi/snort-amd64/etc/snort/snort_47247_igb1/rules/suricata.rules": No such file or directory.
Should Snort be calling "suricata.rules"?
I have seen this before and the only way to make Snort start is to reinstall it.
–-
Update 1: While Snort was in this FATAL ERROR state (see attached), I uninstalled Suricata and then rebooted again. This time Snort started no problem.I'm starting to have a bad feeling about having both Snort and Suricata installed at the same time.
Now to see if alerts work while having IP REP enabled.
Update 2: The issue of no alerts when IP REP is enabled is still there.
 -
** SOLVED **
Guess I'll take the hit for Operator Error upfront. But, there is some behavior I think is strange.
–
When I enabled IP REP in the interface configuration, I did not go further down and select the IP Blacklist (the only one I had was the emerging-compromised-ips.txt list that was there automatically). While in this configuration, all alerts stopped being written to the alerts file in the interface directories.
Once I assigned the IP Blacklist, alerting started working normally.
I then went back and tried to reproduce the original problem by removing the IP Blacklist while still having IP REP enable. Not only could I not reproduce it, but I kept getting 'packets blacklisted' blocks and alerts without having the blacklist selected (IP REP was still enabled … I did toggle it off/on). Probably gets cached.
--
Another oddity ..... While in the first state of having IP REP enabled and NO blacklist selected, there were still blocks for the IP's in the blacklist, but the Alert Description was blank on the BLOCKED tab.
Now that I've messed around with adding/removing the blacklist, the blocks have the expected "(spp_reputation) packets blacklisted " Alert Description.
Anyway, the root cause was not having an IP Blacklist selected when initially enabling IP REP.
A good feature enhancement would be to have a configuration check to ensure the user has at least one blacklist selected and prompt the user, if there is none.
Still don't understand the FATAL ERROR starting Snort when Suricata was also installed. But, that may be a topic for another thread.
-
** SOLVED **
Guess I'll take the hit for Operator Error upfront. But, there is some behavior I think is strange.
–
When I enabled IP REP in the interface configuration, I did not go further down and select the IP Blacklist (the only one I had was the emerging-compromised-ips.txt list that was there automatically). While in this configuration, all alerts stopped being written to the alerts file in the interface directories.
Once I assigned the IP Blacklist, alerting started working normally.
I then went back and tried to reproduce the original problem by removing the IP Blacklist while still having IP REP enable. Not only could I not reproduce it, but I kept getting 'packets blacklisted' blocks and alerts without having the blacklist selected (IP REP was still enabled … I did toggle it off/on). Probably gets cached.
--
Another oddity ..... While in the first state of having IP REP enabled and NO blacklist selected, there were still blocks for the IP's in the blacklist, but the Alert Description was blank on the BLOCKED tab.
Now that I've messed around with adding/removing the blacklist, the blocks have the expected "(spp_reputation) packets blacklisted " Alert Description.
Anyway, the root cause was not having an IP Blacklist selected when initially enabling IP REP.
A good feature enhancement would be to have a configuration check to ensure the user has at least one blacklist selected and prompt the user, if there is none.
Still don't understand the FATAL ERROR starting Snort when Suricata was also installed. But, that may be a topic for another thread.
Thanks for the detailed write-up. I will review the logic for all the IP REP stuff to be sure the configuration gets written correctly. It should not matter if a blacklist or whitelist is selected or not, but I will test out this condition to see if some problem is in fact lurking there.
Snort and Suricata co-existence should not be a problem. While they use much of the same code base, they store all their files in completely different directories. I'm inclined to believe something unusual happened in your case. I will also test running them both just to be sure.
Bill
-
I then went back and tried to reproduce the original problem by removing the IP Blacklist while still having IP REP enable. Not only could I not reproduce it, but I kept getting 'packets blacklisted' blocks and alerts without having the blacklist selected
Ops, never mind, I was clicking around too fast. Adding the blacklist to the interface 'sticks' without hitting Save, meaning you can leave the interface configuration and when you come back it is still there. You can be 'tricked' into thinking it is doing something.
Only when you hit Save does it trigger the interface config reload.
Same for when you remove a list.
This behavior could have unintended consequences for the user. You continue to see a given blacklist applied (or removed), but it is not doing anything. (Got'a protect dummy users from themselves! :o )
-
I then went back and tried to reproduce the original problem by removing the IP Blacklist while still having IP REP enable. Not only could I not reproduce it, but I kept getting 'packets blacklisted' blocks and alerts without having the blacklist selected
Ops, never mind, I was clicking around too fast. Adding the blacklist to the interface 'sticks' without hitting Save, meaning you can leave the interface configuration and when you come back it is still there. You can be 'tricked' into thinking it is doing something.
Only when you hit Save does it trigger the interface config reload.
Same for when you remove a list.
This behavior could have unintended consequences for the user. You continue to see a given blacklist applied (or removed), but it is not doing anything. (Got'a protect dummy users from themselves! :o )
You're right. Did not think about that. I will update it so changing a blacklist or whitelist does the restart.
Bill
