Added to that : for all this to work, you have to install certs on every client device, certs being used so that client device can use and trust pfSense as a proxy, so pfSense can do the real MITM job.
All this, on paper, is pure dynamite. In reality, its far better then that.
So, I was wrong! It was a "zombie" process. I thought by restarting the Snort service it would kill it, but it did not. I Called Netgate to solve a PHP crashing issue due to Snort and he showed me how to kill a hung process. It is now working!
No, restarting won't kill a zombie process. By definition those will not respond to anything from the GUI. They won't stop nor see any changes to the configuration.
Those can happen if something causes pfSense to rapidly issue a series of "restart all packages' commands. One trigger is your WAN IP changing quickly. Another possibility is Snort in the middle of an auto-restart due to a rules update and during that short interval pfSense issues a "restart all packages" command. That can result in two copies of Snort on the same interface, but one of those copies will not respond to the GUI at all.
The final thing that can lead to zombie Snort processes is the use of the Service Watchdog package to monitor Snort. That package does not understand how the internals of Snort are plumbed, and it therefore does not monitor everything necessary to adequately determine if Snort is running correctly. Also, it does not realize Snort stops and then restarts itself during a rules update (or when the admin manually cycles it via the GUI icons on the INTERFACES tab). When Service Watchdog sees the service down, it immediately calls the shell scrip to restart, but there may already be a restart in progress. Thus you can have two instances running on the same interface.
Lol you are right!
Btw, is there a way to have dnsbl without pfblockerng? Now i have it just as you said, but disabled ip filtering in pfblocker, and snort has a a few et and free vrt rules. Thing are working ok, just some instability (kernel panic sometimes that i have yet to discover why, for now swapped ram but still had a crash when i changed dns to not push dns server from pppoe to clients)
I'm not an expert on the DNSBL thing, but in terms of GUI support you kind of need pfBlockerNG-devel in order to implement the DNSBL feature with unbound. That's because the pfBlockerNG-devel GUI code handles the messy tasks of configuring the Python module and managing other configuration settings required to make DNSBL work.
You could certainly configure all that on your own via the command line, but it would not be as easy as "click this, click that" like it is in the GUI.
As for the instability, that can happen as you burden the firewall with more and more things to keep track of while blocking. Adding millions of IP addresses to block from some list, and comparing each incoming packet against each IP on that list is a lot of CPU work and takes lots of state table entries and RAM. However, a single "deny all" rule is the ultimate in efficiency 😁.
The install is not completing successfully. The very last step of the install is creating the menu and service entries. Because those are missing for you, that means the install is not finishing.
Most likely this is the same PHP crash bug that is impacting 2.5 CE and 21.02_1 boxes (SG-3100 boxes in particular). If this is the same bug, it's not a problem with the package as the same code works fine on other systems. It is something wrong with PHP on the SG-3100 appliances with the 32-bit ARM chip.
bmeeksbmeeks 2 days ago @rloeb said in Snort Package v4.1.3 Update -- Release Notes:
Confirmed that it's not working on SG-3100. Installed succeeded, but it doesn't start (or fails after it starts, although I'm not seeing that in the logs).
The main issue on the SG-3100 is that a portion of the Snort GUI code that runs when you click the Start icon is crashing PHP itself on the firewall. Why that happens has not yet been pinned down. The exact same GUI code runs just fine on everything else (SG-1100, SG-5100 and any other device that has a CPU that is not a 32-bit ARM chip). So that hints the issue is something with PHP itself on 32-bit ARM architecture, but nothing is proven yet.
This crashing of PHP will also likely interfere with the installation of Snort as it calls the same area of code during post-installation configuration. If PHP crashes then, it will likely not complete the last step of the installation which is creating the menu entry under SERVICES.colored text
I have the same problem and more dangerous behaviour from pF after latest ntop update. Even my avahi demon send mdns externally!
I'm on the latest pF 2.5 btw.
I found this is due ntop bag and resolved by turning off hosts discovery in ntop itself.
If you are affected it is easy to check after ntop update by visiting ntop host details page where you will see a lot of errors. This behaviour is discovered even you not using ssh on your pF so changing logging behaviour don't make sense.
I also added all my local networks under the ntop settings in pF.
This stop pF from crazy behaviour with this snort allert, mdns and also fixed host details page in ntop itself.
Don't have time enough to check if all this mess really go outside or just happened on localhost interface with ntop.
If still not updated bug you can contribute on freeBSD forum for it.
I think this ntop bug affecting only folks with WAN enabled under ntop setting in pF but didn't check that.
Annoying thing is that after rebooting your pF you need go to ntop setting page in pf and just clink save all settings again.
Final conclusion is if you have any package wrong configured on your pF then you can become in internet even like an attacker regardless you are reseeding not your own traffic.
Maybe good way to truly test all updates on pF platform :) not simply fork them.
How you run the process is important too because I feel ashamed a bit that my pF firewall became unaware that behave like a worm for resident and friendly network by simply copy redundant traffic across interfaces because one of the distributed packages wasn't test enough.
Form me personal interesting in this is how you are utilise your pF when this can become dodgy for your network. All about is use the tools, analyze the logs and do the tests :)
I love pF btw always my recommendation like you can see in open source we can resolve a lot annoying problems. :)
I need to get more up to speed with this (relatively new), but I was always on the presumption that I needed to put the IDS/IPS on the outside connection (speak WAN) instead of on the internal faced section, in this case LAN. I will work on it, again thanks for the info 👍
That was the old way of thinking, but because the IDS sits immediately after the NIC, it sees inbound traffic before the firewall. So the IDS will alert and respond to all the Internet crap your default drop rule on the firewall is going to block anyway. So why bog down the IDS analyzing all that noise?
Here is a digram that shows how the IDS/IPS fits into the network path for Inline IPS Mode and Legacy Mode.
So notice in either case the IDS is "in front" of the firewall with respect to inbound traffic on an interface. So let the firewall filter the noise on the WAN. pfSense is plenty secure itself, so you aren't protecting the firewall with your IDS/IPS, you are protecting your local networks. They are behind the firewall, so you can put the IDS/IPS there and still protect them just fine. No Internet host can reach a local network host without going through the IDS/IPS on the LAN interface (or any other internal interface you may define).
I just installed the 4.1.3_2 version on a pfSense-2.5.0-RELEASE virtual machine without issue. Here is the package installation log:
>>> Installing pfSense-pkg-snort...
Updating pfSense-core repository catalogue...
pfSense-core repository is up to date.
Updating pfSense repository catalogue...
pfSense repository is up to date.
All repositories are up to date.
The following 5 package(s) will be affected (of 0 checked):
New packages to be INSTALLED:
daq: 2.2.2_3 [pfSense]
libdnet: 1.13_3 [pfSense]
libpcap: 1.9.1_1 [pfSense]
pfSense-pkg-snort: 4.1.3_2 [pfSense]
snort: 2.9.17 [pfSense]
Number of packages to be installed: 5
The process will require 10 MiB more space.
2 MiB to be downloaded.
[1/5] Fetching pfSense-pkg-snort-4.1.3_2.txz: .......... done
[2/5] Fetching snort-2.9.17.txz: .......... done
[3/5] Fetching libdnet-1.13_3.txz: ......... done
[4/5] Fetching daq-2.2.2_3.txz: .......... done
[5/5] Fetching libpcap-1.9.1_1.txz: .......... done
Checking integrity... done (0 conflicting)
[1/5] Installing libdnet-1.13_3...
[1/5] Extracting libdnet-1.13_3: .......... done
[2/5] Installing libpcap-1.9.1_1...
[2/5] Extracting libpcap-1.9.1_1: .......... done
[3/5] Installing daq-2.2.2_3...
[3/5] Extracting daq-2.2.2_3: .......... done
[4/5] Installing snort-2.9.17...
[4/5] Extracting snort-2.9.17: .......... done
[5/5] Installing pfSense-pkg-snort-4.1.3_2...
[5/5] Extracting pfSense-pkg-snort-4.1.3_2: .......... done
Saving updated package information...
Loading package configuration... done.
Configuring package components...
Loading package instructions...
Executing custom_php_install_command()...Saved settings detected.
Migrating settings to new configuration... done.
Downloading configured rule sets. This may take some time...
Downloading Snort Subscriber rules md5 file... done.
Checking Snort Subscriber rules md5 file... done.
There is a new set of Snort Subscriber rules posted.
Downloading snortrules-snapshot-29170.tar.gz... done.
Downloading Snort OpenAppID detectors md5 file... done.
Checking Snort OpenAppID detectors md5 file... done.
There is a new set of Snort OpenAppID detectors posted.
Downloading snort-openappid.tar.gz... done.
Downloading Snort AppID Open Text Rules md5 file... done.
Checking Snort AppID Open Text Rules md5 file... done.
There is a new set of Snort AppID Open Text Rules posted.
Downloading appid_rules.tar.gz... done.
Downloading Snort GPLv2 Community Rules md5 file... done.
Checking Snort GPLv2 Community Rules md5 file... done.
There is a new set of Snort GPLv2 Community Rules posted.
Downloading community-rules.tar.gz... done.
Downloading Emerging Threats Open rules md5 file... done.
Checking Emerging Threats Open rules md5 file... done.
There is a new set of Emerging Threats Open rules posted.
Downloading emerging.rules.tar.gz... done.
Installing Snort Subscriber ruleset...Copying md5 signature to snort directory... done.
Installing Snort OpenAppID detectors...Copying md5 signature to snort directory... done.
Installing Snort OpenAppID Rules...Copying md5 signature to snort directory... done.
Installing Snort GPLv2 Community Rules... done.
Installing Emerging Threats Open rules...Copying md5 signature to snort directory... done.
Updating rules configuration for: WAN ... done.
Updating rules configuration for: OPT1 ... done.
Updating rules configuration for: OPT2 ... done.
Updating rules configuration for: LAN ... done.
Cleaning up temp dirs and files... done.
The Rules update has finished.
Finished downloading and installing configured rules.
Generating snort.conf configuration file from saved settings.
Generating configuration for WAN...
Generating configuration for OPT1...
Generating configuration for OPT2...
Generating configuration for LAN...
Generating snort.sh script in /usr/local/etc/rc.d/... done.
Finished rebuilding Snort configuration files.
Why don't you try removing the Snort package and then installing it again? Don't click the "reinstall icon", instead click the trash can "delete icon" to remove the package. Then go to Available Packages and install it again. You will not lose any settings doing that as they are preserved unless you specifically go to the GLOBAL SETTINGS tab and uncheck that option.
I'm not a Virtualbox user, so I can't help you there. Host networks in workstation-level hypervisors can be tricky. You really need the concept of virtual switches like you can use in ESXi and other hypervisors. That way you can keep things separate.
You don't really need a Pass List with Inline Mode because you are not blocking an IP. You are just dropping individual packets when they match.
Over dinner I read through the snort inline thread and the adjustability of the rules so you can alert or block is huge. I can see so much value with running snort inline. However, I was pretty bummed out to see the limitations with lag and VLANS. I use both of those technologies when architecting highly available networks.
It got me thinking though, that the inline mode and the custom output plugin you wrote really provide two different strategies. Along with some of the other posts you've written I'm starting to question my IPS approach.
My approach has been to identify bad actors in any way possible so that they can be blocked to prevent future harm. So say for example a bad actor is attacking using an ActiveX vulnerability and even though our environment doesn't use ActiveX I still have those rules enabled. This way if the same attacker moves on to use a SQL injection attack they are already blocked, as they don't become unblocked in my case for an extended period of time. At which point I'm hoping they'll just move on.
The limitation with VLANs and LAGG is due to the way the netmap kernel device is plumbed within FreeBSD. It's not a Snort limitation. The netmap idea had great promise when it was introduced a few years ago into FreeBSD and Linux, but some of the grand plans have not taken shape. Thus the various limitations of the technology. You can read up on netmap via Google searches.
As for IDS/IPS strategy, there are as many opinions on what is "right" as there are IDS admins. But generally I favor keeping the workload on my firewall as light as possible while still affording protection. The reality is that almost any firewall today is pretty darn secure. This is especially true if you limit the amount of third-party stuff (such as packages) that you install on it. Remember each installed package brings in a bunch of shared libraries that may, in turn, bring in still more shared libraries. And any of these libraries can harbor vulnerabilities. So the fewer packages, the better.
As for blocking, as I stated, I'm not a fan of putting in specific blocks for the world. Put in Pass rules for explicitly what you want to come in (unsolicited), and then let the default deny rule take care of everything else. If you don't trust your firewall to be secure on its own and by default, why are you using it? (Rhetorical question, not an accusation ... 🙂).
For IDS/IPS, run the rules that protect the exposures you have. Don't waste CPU and RAM resources on rules that protect against threats you are not vulnerable to. Spend your time and effort keeping your internal machines patched with the latest security hotfixes. That is 99% of cyber security right there! And it's much more effective than running every pfBlockerNG IP list or Snort rule in existence.
In general you should NEVER monkey with the HOME_NET or EXTERNAL_NET variables in Snort on pfSense. The defaults are fine for almost every case I can think of.
I think sometimes folks make incorrect assumptions about what those variables actually are. HOME_NET is the IP networks or hosts you want to "Protect". They are where your jewels are stored so to speak. EXTERNAL_NET is where the bad guys live. That is assumed to NOT be inside your protected networks. Thus EXTERNAL_NET defaults to !HOME_NET (or all addresses NOT contained in HOME_NET). That is a logical choice.
Do not confuse HOME_NET or EXTERNAL_NET with Pass Lists. Those are completely separate things! Pass Lists are assigned and used to prevent certain hosts from being blocked. That has nothing to do with HOME_NET or EXTERNAL_NET directly.
The only time I can imagine where you might want to monkey with HOME_NET is if you have downstream networks not directly associated with a pfSense interface that you want to mark as "protected". In this case you would need to create a custom HOME_NET, but you would want to be sure that you included the default networks in that list.
The PASS LIST tab is for creating customized lists. And lists created there are usually assigned as Pass Lists, and thus that's how the tab got its name because there is almost never a reason to monkey with HOME_NET or EXTERNAL_NET. However, for the rare time when you may need to monkey with HOME_NET, you would do so by creating an IP list under the PASS LIST tab and then assigning that list name under the HOME_NET drop-down.
HOME_NET and EXTERNAL_NET are special variables used in many of the Snort rules. If you get the wrong IP addresses or networks defined in there, you can totally neuter your Snort traffic inspection. That's why I tell folks not to mess with the defaults unless you are an IDS/IPS admin expert with lots of experience. You need to know exactly what you are doing when messing with either of those two variables.
Hi thanks for the snort rule. Yeah my test machine can catch it. So it proves that snort works and i have no alternate path to the internet. I see drop alerts in the alert tab and i cannot ping (Request timeout).
Then that indicates the p2p rules you are using are insufficient to stop all of the Bit Torrent stuff. It is catching part of the conversation between client and peer, but not everything, so the client is still able to make the connection and download. It's not a problem with Snort itself. Instead, it is a problem with the rule or rules attempting to detect the traffic. The rules are apparently not picking up everything.
Blocking stuff like this is a whack-a-mole game. The developers of the torrent clients strive to make their traffic indistinguishable from regular network traffic (and thus unblockable). And the IDS/IPS rules creators strive to create new detection rules that trigger on the latest evasion techniques - and around and around it goes .... 🙂.
Something pretty much like you said you needed is coming soon. Look for the update in the pfSense DEVEL snapshots in the near future. Here is a post I made describing the new feature: https://forum.netgate.com/topic/160771/new-often-requested-snort-feature-coming-soon.
@bmeeks Thanks for the information. The SG-7100 is using the ix nics for its 10GB SFP+. Once configured both WAN and LAN to use these inline mode worked correctly with the obvious hardware checksum options enabled.
Thanks for the feedback. It will help others who might have the same question in the future.
@nogamer@bmeeks Ok, I've found my services in "odp/appMapping.data" which was updated in november 2020. So i can create custom rules to block these services in my network.
I hope so that this file is updated in the future.
I think there is some interest in updating the file from another party, but I can't say who for now. Perhaps they will choose to takeover maintaining the OpenAppID text rules going forward.
In the meantime, you can certainly create your own custom OpenAppID rules to supplement those available in the standard archive. You found the proper location for identifying application names (in /usr/local/etc/snort/appid/odp/appMapping.data).
@bmeeks Yup it had me baffled for ages as I couldn't see any lookups from my LAN interface to the offending FQDNS.
Yes, the firewall itself sources the connection for the DNS lookup, and that traffic will exit the WAN on the way to either the DNS root servers or whatever DNS forwarder might be configured. So the IDS rules on the WAN see the traffic and alert. Since it's on the WAN, and the IDS sits beyond the firewall, the IDS sees your firewall's public WAN IP as the "source". The natural inclination is to assume the traffic originated on your LAN, but it actually did not.
@bmeeks, If I had to guess it's the EVE.JSON file which i'm ingesting into ELK for dashboards. In the Log Size and Retention Limits configuration the max size is 10MB, but I currently have a file in there "eve.json" that is 1.2GB
The idea is for the log to rotate and get a new name with a UNIX timestamp appended to it. Then a new empty log file is opened for Suricata. The SIGHUP is supposed to tell Suricata to reopen log files. Unfortunately, the Suricata binary can only rotate certain logs natively. So without the GUI attempting to rotate the others, they will grow to impossibly large sizes.
Do you have any eve.json logs that have a UNIX timestamp on the end? If not, the log rotation is not actually working. That would be why it keeps trying each time the cron task runs (every 5 minutes).
You might have a duplicate Suricata zombie process attempting to use the log file. If you can, stop Suricata on the interface for more than 5 minutes. This will allow the cron task to run and hopefully rotate that huge file. Then restart Suricata on the interface. If stopping Suricata for more than 5 minutes does not result in the file rotating, then manually rename it yourself (the big 1.2 GB file) to something else and then restart Suricata.
We provide leading-edge network security at a fair price - regardless of organizational size or network sophistication. We believe that an open-source security model offers disruptive pricing along with the agility required to quickly address emerging threats.
Subscribe to our Newsletter
Product information, software announcements, and special offers. See our newsletter archive to sign up for future newsletters and to read past announcements.