Snort 2.9.2.3 pkg v. 2.5.0 Issues
-
miles267,
settings do not really matter, but which rules you enable could play a role. Also my pfsense box is not an edge router, so some issues cannot occur if you hook it up o an ISP directly and activate pfsense on the WAN NIC.
Feeling comfortable to tackle the rules updated problem?
-
Hello,
Fesoj: I checked all the messages you send, and my installation is compliant to all that… what I think is that there is some problem with the alert_pf output plugin.I made some test about that. The new version of the Snort package generate a whitelist for the box IPs like:
192.168.0.1/24 (referred to a single IP)
while in the old version (I just compared with a virtual machine I use as text that I didn't update) the IPs are written like:
192.168.0.1
.
So I run this test, I looked in the snort.conf file and I found:
output alert_pf: /usr/local/etc/snort/snort_35034_em2/default,snort2c,both,killI edited the /usr/local/etc/snort/snort_35034_em2/default file in order to change the format (for the single IPs) from:
IP/mask
to
IP
and I restarted Snort (without saving anything in order to avoid the "default" file to be recreated).
After some minutes, and some alerts generated, the only IPs that got blocked were the ones NOT included in the whitelist (as it should be and as it was before the latest package changes).
I made some investigation on the spoink plug-in, and yes, in its documentation the IPs in the whitelist are written without mask.So, I think there is something to fix for the alert_pf package, probably the procedure that generate its whitelist.
Thanks,
Michele -
Michele,
I haven't looked at the details of alert_pf yet. I am still busy with the update problem. It would be helpful if ermal finds some time to validate the latest patches so a standard installation would be at the current "patch level". Too many changes at the same time are typically a recipe for disaster.
As far as whitelisting goes, I would need to do some reading first. Currently I can cope with blocking quite easily, but I could imagine some scenarios where it gets more difficult. Before writing some code I'd like to suggest to collect some reference configurations that include the net topology, the snort interfaces, and the rules (actually sources and destinations would suffice). What I mean is, if you have a rule with ANY as the source and HOME_NET as the destination and you are blocking on the destination side, then this makes not always any sense.
PS: If you search the snort forums, you'll find several places where people have difficulties with IP ranges in whitelists, but I have not looked at the details whether this applies here, or whether these issues have been resolved, or whether the issues were due to a misunderstanding how snort works.
-
Whitelist has always been single IPs. It would be nice to use CDIR but it wasn't built that way. Ever since this change to the snort binary, https://github.com/bsdperimeter/pfsense-tools/commit/34fe38d61ba1f858f3c5bcdec6fa583a74e328a4, snort blocks everything unless its in a whitelist as single IP.
HOME_NET IP/Subnets should not be blocked at all.. If you want the system to be able to block an IP in your HOME_NET NETLIST then create a custom NETLIST that doesn't include IMHO.
But by default, HOME_NET should be exempt from blocking.. But since the binary change, this is not the case!
As far as the other issues, that has to do with the php code that Fesoj seems to be fixing, thanks Fesoj!… I'll test when i'm fully online later this week.
-
Cino,
But by default, HOME_NET should be exempt from blocking..
it actually depends. For example, if there is a company policy that forbids eDonkey and friends, then it could make sense to block machines in the HOME_NET. Sometimes this may be better than reporting the fellow to the management (actually 2 weeks ago I needed to attended a court case, where somebody got fired because of a "Facebook" issue) and sometimes, especially in some European countries, it might even be necessary to avoid legal actions because of excessive liability for interference claims. It all boils down to treat your colleagues or clients like little children, but this is a social problem.
-
PS: If you search the snort forums, you'll find several places where people have difficulties with IP ranges in whitelists, but I have not looked at the details whether this applies here, or whether these issues have been resolved, or whether the issues were due to a misunderstanding how snort works.
Whitelist has always been single IPs. It would be nice to use CDIR but it wasn't built that way. Ever since this change to the snort binary,
ok, we understood what the problem is. Before white lists were generated as "IP lists", now even the "default" whitelist is generated as a "network list" and it does not work anymore.
I am now looking at the Spoink code to see what we can do to restore this functionality… but for the moment I have to turn off the IP blocking or everything gets stuck in few minutes!
Ciao,
Michele -
Michel,
after reading Cino's comment, that is in line with what I had in mind, your analysis is right. CIDR entries in the default file seem to be wrong and don't seem to work as expected. For my box the generated entries contain
(1) the loopback address
(2) the WAN address of the box (which is static in my case, so that's fine) [currently the subnet]
(3) the LAN address of the box [currently the subnet]
(4) and the DNS serversI am wondering what happens if under point 2 the WAN address is dynamic. Would this mean that when blocking is enabled, you cannot generally avoid blocking yourself? Maybe there is an update mechanism.
-
Michel,
after reading Cino's comment, that is in line with what I had in mind, your analysis is right. CIDR entries in the default file seem to be wrong and don't seem to work as expected. For my box the generated entries contain
I am wondering what happens if under point 2 the WAN address is dynamic. Would this mean that when blocking is enabled, you cannot generally avoid blocking yourself? Maybe there is an update mechanism.Thanks! But consider, I have about 80 VIPs, 5 NICs, it would be unconfortable to mantain a IP list aligned for the local IPs…
I mean, managing the HOME_NET with an Alias is awesome, just the whitelists should be managed as before, with an "host alias" for the part the user can decide (probably this should be noticed in the interface), and for the "default" part of the white list as single IP addresses without CIR.Now that the problem is known and identified, I trust in Ermal to fix it... ;) Thanks!!
Michele
-
Imho I think the default home_net list should be exempt. If you need blocking, create a custom netlist.
-
Cino,
understood. As long as a custom list is possible.
-
If you see the following error message in your system log:
FATAL ERROR: /usr/local/etc/snort/snort_56964_re1/preproc_rules/sensitive-data.rules(1) Unknown ClassType: sdf
you can either add the type declaration by hand (see some of the earlier posts), or try the following patch:
https://github.com/bsdperimeter/pfsense-packages/pull/288
The reference classification.config file gets extracted from the rule set package (Snort.org and/or Emergingthreats.net), where the last download overwrites the first one. Currently the Snort.org rules are downloaded and unpacked before the ET rules, where the ET rules don't declare the sdf type. Hence the fatal error. The patch reverses the order of unpacking the archives since the Snort.org rules seem to have all the necessary declarations.
Happy patching.
UPDATE: with package vs. 2.5.1 this is obsolete
-
mdima,
HOME_NET != WHITELIST. On the WAN side, I have the default HOME_NET, but a special whitelist (including the IP address on the WAN side (I don't know what to do in case of an ISP supplied address using DHCP)) and this works fine. It's just the way things are setup. As I said before, it is now easier than ever to shoot oneself in the foot.
That is why the option of add wan ips is there.
It will get the right address when theip changes. at least on 2.0+ -
It's been part of the GUI for years and is still there. For custom Netlist.
-
It will get the right address when theip changes. at least on 2.0+
I checked that and the IP of the WAN nic does not get blocked.
-
ermal,
That is why the option of add wan ips is there.
part of the discussion today was about whether the IP ranges actually work in the whitelist (alert_pf). Essentially, they don't work, but I might be wrong.
-
I pushed some fixes to snort to prevent issues with so rules.
Also changed a bit the upgrade process.Please test again.
Fesoj,
I tested the whitelist features and they work with ranges.
Have you installed teh latest binary? -
If you see the following error message in your system log:
FATAL ERROR: /usr/local/etc/snort/snort_56964_re1/preproc_rules/sensitive-data.rules(1) Unknown ClassType: sdf
The reference classification.config file gets extracted from the rule set package (Snort.org and/or Emergingthreats.net), where the last download overwrites the first one. Currently the Snort.org rules are downloaded and unpacked before the ET rules, where the ET rules don't declare the sdf type. Hence the fatal error.
In looking through the snort_check_for_rule_updates.php file, I see the following lines of code that are, I believe, supposed to prevent the classification.config overwrite when both snort and emgerging threats rules are used.
if ($snortdownload == 'off') { foreach (array("classification.config", "reference.config", "sid-msg.map", "unicode.map") as $file) { if (file_exists("{$snortdir}/rules/{$file}")) @copy("{$snortdir}/rules/{$file}", "{$snortdir}/{$file}"); }
This piece of code is in the Emerging Threats extraction and copy section, and it appears to be designed to prevent overwriting the classification.config and other files extracted earlier in the code when Snort rules are enabled. The value of the variable should be "on" when Snort rules are enabled, and thus the copying of the classification.config and other configuration files from the Emerging Threats rules package is bypassed. This would leave the "correct" versions of these files extracted from the Snort rules package.
Why is this section of code not working?
-
ermal,
I tested the whitelist features and they work with ranges.
give me a few days and I'll present an example, it just takes some time to document it. It is also possible that I am misunderstanding s.th. here. My setup is working, so I am fine. Also, I am using the latest binaries.
-
I removed subnets/cidr mask from auto generated addreses for whitelist and you can put them with the alias you configure if you want.
I bumped the package to 2.5.1
-
bmeeks,
Why is this section of code not working?
well, if you download both rule sets, then the ET part will enter the loop where some files from the rules directory of the unpacked archive are copied into the configuration directory, thereby overwriting the any previous version.
-
bmeeks,
Why is this section of code not working?
well, if you download both rule sets, then the ET part will enter the loop where some files from the rules directory of the unpacked archive are copied into the configuration directory, thereby overwriting the any previous version.
Well, it's sort of a moot point now since I see in the latest update Ermal changed the code a bit so Emerging Threats are done first.
The way I understood the old code, it unpacked each rule set into a temp directory and then copied files into the working directory. The snippet I posted should have been hit before the classification.config file from Emerging Threats was copied over into the working directory. If it saw that Snort downloads were "on", it should have skipped the @copy function where the Emerging Threats classification.config and other configuration files were copied from the temp area where they were extracted into the working directory along with the previously extracted and copied Snort files.
-
2.5.1 throws the following error which was not there before without changing any configuration (did complete reinstall):
FATAL ERROR: /usr/local/etc/snort/snort_4672_em1/rules/snort_botnet-cnc.rules(366) Unknown rule option: 'ssl_state'.
[EDIT] Suprise: the preprocessor page is heavily modified and now we have to enable SSL data separately. It was unchecked and thats where the error came from. Doing further testing now.
-
bmeeks,
https://github.com/Fesoj/pfsense-packages/compare/patch-3 :)
Update: Initially I picked the wrong patch.
-
judex,
FATAL ERROR: /usr/local/etc/snort/snort_4672_em1/rules/snort_botnet-cnc.rules(366) Unknown rule option: 'ssl_state'.
there is a section in the snort user's manual about the dynamic SSL preprocessor. I've never found a reason to activate this one.
On the other hand, did you activate this for the latest version or did the error message appear out of nowhere?
-
Thx for your answer and the investitgation you did during the last days Fesoj!
I edited my post because I found the new preproc settings and enabled SSL data there. Now snort starts without warning.
If I do not enable SSL-data an preproc page I get the errors from botnet-cnc and if disabled from exploit.rules…So I obviousely need it if I want to enable botnet rules etc.
-
I just updated the snort package and now upon restart its giving me
Jul 22 12:38:41 snort[26253]: FATAL ERROR: /usr/local/etc/snort/snort_44959_re0/rules/snort_botnet-cnc.rules(372) Unknown rule option: 'ssl_state'.
Jul 22 12:38:41 snort[26253]: FATAL ERROR: /usr/local/etc/snort/snort_44959_re0/rules/snort_botnet-cnc.rules(372) Unknown rule option: 'ssl_state'.I havent changed my rules/categories. The same as i have been running for some time.
Good news was when I updated I didnt get the errors in the system log like i did before upon completion of the install.
I noticed a new SSL option on the preprocesser Tab.
Enable that option and restarted no issues/errors so far.
I want to say thanks for all the work you all have been doing on this package to keep it updated/adding new features/ resolving bugs.
I cant imagine not having snort up and running. :-)
-
kilthro,
try to deactivate the rules that require the SSL preprocessor and see what happens. Then fiddle around with snort.conf for the associated interface(s). With some luck the snort user's manual contains sufficient info to do that.
Please report when things are running.
-
kilthro,
try to deactivate the rules that require the SSL preprocessor and see what happens. Then fiddle around with snort.conf for the associated interface(s). With some luck the snort user's manual contains sufficient info to do that.
Please report when things are running.
Sorry if my post was misleading. I had the error and I did get it to run fine after enabling the ssl option. I was just posting stating that I had the error, I found the new option and enabled it and all is working just fine now..
Just getting some strange mem cap reached alerts in snort now for pop normalization… Need to look into that.. Other than that, all appears good. Need to try the auto update out later . ;)
Thanks for your hard work with this package.
-
kilthro,
so the "Enable SSL Data" option was not checked? And after enabling it, the snort config was ok?
-
kilthro,
so the "Enable SSL Data" option was not checked? And after enabling it, the snort config was ok?
Correct. It wasnt enabled. Never seen the option there before. So I enabled it and it did fix the problem/error that I was having.
On a side note I am noticing
"(POP) No memory available for decoding. Memcap exceeded" - 07/22-13:01:54
in my alerts.. I dont see a memory setting in the snort package. Is this something now I have to adjust somewhere else? Never had it before until todays update.
-
bmeeks,
https://github.com/Fesoj/pfsense-packages/compare/patch-3 :)
Update: Initially I picked the wrong patch.
I see now, I misunderstood the full flow of the code. I was thinking the entire contents of the rule archive (either Emerging Threats or Snort) were extracted first in a temp directory, and then copied over to the working directory in pieces. That appears to be not true for everything. Looks like by the time the snippet of code I posted earlier gets executed, it's too late because code further up above has already overwritten the files. Changing the order as you did is the best idea for now.
I did have one thought, though. Is it not possible that from time to time there may be legitimate differences in the config files from Emerging Threats and Snort such that simply using the Snort version might cause a different problem? What about a technique that compared the two files and made sure all unique lines from both files got merged into the final classification.config and the other configuration files?
-
kilthro,
I dont see a memory setting in the snort package.
as far as I remember, the some of the snort preprocessors have this option. See the default snort.conf from the package (not the ones in the interface directories). Maybe that helps.
-
bmeeks,
What about a technique that compared the two files…
the best thing is probably to build a union of all type declarations and in case of duplicate entries, maybe the one with the longer description could be chosen. Looks like a freshman homework problem. :)
-
The issue with this snort thingy is that they do not seem to do a good job at keeping things isolated.
If you need preprocessor x for rule x there should be a better linkage of that, or displayed somewhere.Otherwise its very easy to break stuff.
Someone surely could go and identify which preprocessor inserts which rule option and parse the rules for that and do the enable disable during reload!
Is that worth doing?
Probably upstream should fix that rather than workaround that? -
I just updated the snort package and now upon restart its giving me
Jul 22 12:38:41 snort[26253]: FATAL ERROR: /usr/local/etc/snort/snort_44959_re0/rules/snort_botnet-cnc.rules(372) Unknown rule option: 'ssl_state'.
Jul 22 12:38:41 snort[26253]: FATAL ERROR: /usr/local/etc/snort/snort_44959_re0/rules/snort_botnet-cnc.rules(372) Unknown rule option: 'ssl_state'.I havent changed my rules/categories. The same as i have been running for some time.
Good news was when I updated I didnt get the errors in the system log like i did before upon completion of the install.
I noticed a new SSL option on the preprocesser Tab.
Enable that option and restarted no issues/errors so far.
I want to say thanks for all the work you all have been doing on this package to keep it updated/adding new features/ resolving bugs.
I cant imagine not having snort up and running. :-)
Where is this SSL option? I do not see it on any snort preprocessors tab. There is an "Sensitive Data" option although after it's checked, I'm not quite sure how to use or configure it?
-
I confirm that after the installation of the latest package (2.5.1) the issue of blocking whitelisted IPs has been resolved! :)
If I am right the problem with the auto-update is still pending.
@miles267: Please install the latest package and you will find there the SSL data Preprocessor.
-
I just updated the snort package and now upon restart its giving me
Jul 22 12:38:41 snort[26253]: FATAL ERROR: /usr/local/etc/snort/snort_44959_re0/rules/snort_botnet-cnc.rules(372) Unknown rule option: 'ssl_state'.
Jul 22 12:38:41 snort[26253]: FATAL ERROR: /usr/local/etc/snort/snort_44959_re0/rules/snort_botnet-cnc.rules(372) Unknown rule option: 'ssl_state'.I havent changed my rules/categories. The same as i have been running for some time.
Good news was when I updated I didnt get the errors in the system log like i did before upon completion of the install.
I noticed a new SSL option on the preprocesser Tab.
Enable that option and restarted no issues/errors so far.
I want to say thanks for all the work you all have been doing on this package to keep it updated/adding new features/ resolving bugs.
I cant imagine not having snort up and running. :-)
Where is this SSL option? I do not see it on any snort preprocessors tab. There is an "Sensitive Data" option although after it's checked, I'm not quite sure how to use or configure it?
What version are you on? I am on 2.9.2.3 pkg v. 2.5.1
On the preporcessor tab (within the interface your monitoring) right above enable sensitive data is one that says enable ssl data. That is the one I turned on. If the option is not there, I would suggest upgrading or reinstalling the package. -
ermal,
If you need preprocessor x for rule x there should be a better linkage of that, or displayed somewhere.
it's even more complicated, because snort allows to maintain state info between different rules (the "flowbits" stuff). In order to get a consistent set of activated preprocessors and rules, one should also scan all enabled and disabled rules, look at the required preprocessors and the corresponding flowbits statements (set, isset, …). If you view all these items as resources, a dependency graph should be created and then evaluate the graph for all activated items and write the proper configuration files, which might include activating previously disabled rules.
Is it worth it? I typically have only a few manageable rules and hardly need anything more than http_inspect (more or less), so I wouldn't need it. But, it might be a cool feature---you click on a single rule, dependencies are evaluated, and dependent rules and preprocs are "drawn in" by a ghost hand.
-
ermal,
If you need preprocessor x for rule x there should be a better linkage of that, or displayed somewhere.
it's even more complicated, because snort allows to maintain state info between different rules (the "flowbits" stuff). In order to get a consistent set of activated preprocessors and rules, one should also scan all enabled and disabled rules, look at the required preprocessors and the corresponding flowbits statements (set, isset, …). If you view all these items as resources, a dependency graph should be created and then evaluate the graph for all activated items and write the proper configuration files, which might include activating previously disabled rules.
Is it worth it? I typically have only a few manageable rules and hardly need anything more than http_inspect (more or less), so I wouldn't need it. But, it might be a cool feature---you click on a single rule, dependencies are evaluated, and dependent rules and preprocs are "drawn in" by a ghost hand.
I'm still a Snort rookie, but aren't the above tasks exactly what the Pulled Pork help application is supposed to accomplish automatically? If so, is there a way to incorporate it into the pfSense build of Snort? I will do some more research on how Pulled Pork operates, but off the top of my head I seem to remember it was a command-line utility (Perl script, I think) that does the magic.
-
bmeeks,
i am not sure that you can tell pulledpork these are my preprocessors make my snort start!