ok, solved my problem.

Mine was related to network memory buffers (mbuf) as descriped here: https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards#mbuf_.2F_nmbclusters

This seems to be a "known bug"  respectively demand for manual tuning of standard pfsense settings if you have many CPU-cores AND many NICs on your machine (mine has 8 cores and 5 NICs).

After setting it to a Million (1.000.000), everything is fine again, no more suricata crashes (despite the fact that suricata does not handle all VRT-rules, but "that´s another roadworks", as we say in Germany  ;) ).

NanoBSD uses RAM disks for storage.  The default sizes are almost never large enough to provide space for downloading, extracting and installing the vendor rule packages.  When you run out of RAM disk space, very strange things happen.  Lots of times the installation becomes corrupt to the point a reinstall is required.

I do not recommend running either Snort or Suricata on NanoBSD installations.  There are just too many issues with disk space.  The forums here have plenty of posts from NanoBSD users with these kinds of problems.  My advice is to go back to a conventional hard disk.  If you absolutely don't want to do that, then you can try increasing the size of the /tmp and /var partitions to at least 150 MB each (and preferably a lot more!).  Even doing that, be prepared for the occasional weirdness with either of these packages on NanoBSD installs.

Bill

The IPS Policy selection is shown on the RULES tab so you can see which exact rules have been auto-selected by the chosen policy.  You may already know this, but I will repeat it for the benefit of others who may read this thread.  The Snort VRT tags each of their rules with policy keywords (connectivity, balance or security).  Some rules may have all the keywords associated with them, or just one or two.  The IPS Policy option in the Snort package on pfSense examines all the Snort VRT rules and pulls out only the rules marked with the policy keyword you select.  So it is true that the rules in IPS Policy are actually in all of the other category files.

Now on to your question.  You can disable the rules in either place (in the actual Category or when viewing the IPS Policy option).  Rules are recorded for user-defined enable/disable states by their GID:SID number, and those are stored in the config.xml of the firewall.  The last thing Snort processes when building the final rules package for an interface is the list of manual user rule state overrides.  So when you click on the rule icons on the RULES tab to force enable or force disable a rule, that rule's GID:SID is recorded in the firewall's Snort configuration along with the state you toggled it to (enabled or disabled).  Snort will then honor your setting for that rule when building the final rules package file (which by the way, is called snort.rules).  The snort.rules file is built by the Snort GUI package code, and will contain only the rules actually being used for the interface.  Rules you have disabled will not be in the snort.rules file.  Conversely, rules you have explicitly enabled will be in the snort.rules file.

I will also mention that not all rules are enabled in a given category.  This is the way the Snort VRT ships them.  Some rules are commented out (disabled) by default.  Those are displayed in gray on the RULES tab.  You can leave them default disabled, or you can click the toggle icon to force them to the enabled state if you want to use them.  Now here is the tricky part:  when you choose to use the IPS Policy option, any rule tagged with a matching policy keyword (connectivity, balanced or security) is going to be sucked into the final snort.rules file and will get enabled even if it was default disabled in the category file it was pulled from.  So when using IPS Policy, all rules shown will be enabled and used unless you explicitly click the toggle icon to disable one or more of them.

Bill

The portscan preprocessor in Snort seems to have a "hair trigger", and in my opinion at least, it produces more than a few false positives.  There are some tuning tweaks available on the PREPROCESSORS tab for a Snort interface.  You can research what the settings do in the Snort documentation posted at snort.org.  I have greatly dialed down the sensitivity settings on the portscan preprocessor on my home firewall.  That has reduced the false positive rate for me.

One thing that may contribute to the high false-positive rate with Snort on pfSense is the method used to sniff incoming packets.  Snort puts the interface into promiscuous mode.  This means it's going to see everything, including stuff not really meant for that interface.

Bill

@gad_d:

Still didnt find any solution to why the Snort update is blocked by squid
will be happy if anyone has some direction

thanks

Does squid log any message that might give a hint what it does not like about the SSL handshake Snort uses for its updates?

Bill

I have answered part of my own question.  I used nmap to discover that the device in question is an Amazon Fire TV - not sure why it does not show up on the DHCP lease list.

However, I am not sure if these alerts are something to be concerned about?  They seem to be DNS queries for a site with a .pw TLD.  I am not sure why an Amazon Fire TV would be trying to access such a domain.

Thanks.

Yep, that big spike in CPU consumption is exactly what I saw on my home firewall.  Something is weird inside Barnyard2 in my opinion, but like I said, I have not delved into the code to see if I can find out what it is.

Bill

Looking into this issue is next on my list after getting Snort converted to Bootstrap.  That is taking priority, and I'm trying not to make any other PHP code changes or add bug fixes until the GUI is good on pfSense 2.3-BETA and Bootstrap.

It apparently broke when the web server daemon changed in 2.2.6.

Bill

@vingaard:

Would anyone in the forum be able to assist me, so both rules fire a alarm in Snort,  I have a gut feeling that the "flow:established" keyword are the differentiator,
but i would expect that the PFsense Snort would be able to understand this?

many thanks in advance.

Well in that case, why dont you remove the threshold and diagnose only the flow…your second rule should be

alert tcp $HOME_NET any -> $EXTERNAL_NET 80 (msg:"NF - POLICY - malicious web page access"; flow:to_server,established; content:"GET"; nocase; http_method; content:"evilpage.tld"; nocase; http_header; classtype:policy-violation; metadata:NF,25042015; sid:56001811; rev:1;)

F.

Hello,

This might be due to a (unintended) extra space as mention in this post

https://forum.pfsense.org/index.php?topic=106195.0

Well, after testing the options a few times, I am sad to report that the hardware offloading options with the intel pro/1000 pt is not working for me. The network appeared to function fine otherwise, but the amount of those 0 byte packets generated is unnerving. This is on a AMD kabini system, if it makes a difference.

Thanks for the update, we really appreciate it :)
If I can help somehow I'll be glad to.

Br,Greg

If your are testing snort, maybe you should skip the "block for 1h" part for "offenders" or reduce to some minutes.

I'm already aware of how much false positives there are. Let's say my new modus operandi is to add myself to the exclusion list first and foremost  ;-)

Thank you.  My suggestion would be:

1)  add a global option in pfBlockerNG for blocking entire domains where a DNSBL includes say abcdef.com (I think it has to be global option rather than per list, otherwise you run into trouble on de-duping as you say)
2)  do the combination/de-duping/suppression etc as per normal (so people can override the global behaviour on a per domain basis by suppressing e.g. abcdef.com, but adding ads.abcdef.com)
3)  at the point where pfBlockerNG translates the de-duped list into a conf file to pass to Unbound, check each domain being added:
  - if it is a domain: then add it into the conf file with a local zone too - i.e. block the lot
  - if it is a subdomain: then add it into the conf file with just a local data entry - i.e. just block the particular subdomain mentioned

I suspect you may need to build a list of top level domains into pfBlockerNG to do the last part.  You can't just count dots unfortunately, as abc.co.uk has two (but is still a domain), whereas abc.com only has one.

On the crashing point, one way to deal with that is to sort the de-duped list by domain (e.g. invert the character order of each domain string, then sort, then invert back).  Then when you apply the logic in (3), what you'll get is one zone redirect per domain, followed by all the data entries pertaining to that domain - hence no crash.

Just a thought as to how you might do it - I suspect it might be a bit more complicated!

Good luck, and thank you!

Andrew