Snort Package - Potential New Feature Teaser

bmeeks

I have some potentially good news to report about the Snort package and future inline IPS mode operation! This new feature is still in an early stage of development and testing, but initial results are promising.

I have patched the DAQ netmap module used by Snort so that the module supports a more favorable mode of inline IPS operation on pfSense. The new mode allows the specification of the host OS stack as one of the netmap endpoints. This allows Snort to function like Suricata's Inline IPS Mode across a single interface.

The new IPS mode will allow the use of DROP and SDROP actions in Snort rules to actually drop traffic in a true inline IPS mode. I still have to make and test the modifications to the Snort GUI code to support the new operating mode. And before you start asking, the release date is still undetermined at this point. I'm hoping maybe within a month or so, though, if testing goes well.

When released, this will be a pfSense-2.5 and greater feature. It is unlikely to be backported to pfSense-2.4. However, if 2.5 gets delayed for a very long time, perhaps the backport can be revisited. All of my testing and development to date on this feature have been done on pfSense-2.5-DEVEL.

With a true inline IPS mode there is no need to block hosts with firewall rules by putting their IP addresses in the snort2c table. Instead, packets triggering a rule will literally be dropped off the wire while other packets to and from the same host that are not triggering a rule will pass normally. This will be especially useful with the OpenAppID feature in Snort. You will be able to drop traffic matching an OpenAppID rule but not impact other traffic to and from the same host. Today, with the legacy blocking mode, if OpenAppID triggers on a rule, then the host's IP is blocked from all traffic.

bmeeks

Update:

The development work on the new Inline IPS Mode feature for Snort is about 75% complete. The Snort binary code changes are finished and have been tested successfully. These were actually made within the DAQ library used by Snort. I'm working on finishing up the required GUI changes to support the new inline operation mode.

Because I'm able to reuse a lot of the Suricata GUI code for inline operation over in the Snort GUI, things are going pretty fast in terms of modifying the Snort GUI to support inline IPS mode. I am hoping to have a package ready for testing in pfSense-2.5-DEVEL in about one week.

bmeeks

Here some screenshots of the new Snort Inline IPS Mode feature in action on a pfSense-2.5-DEVEL virtual machine.

For users familiar with the Suricata GUI interface, these settings will be very familiar. First, here is the INTERFACE SETTINGS tab showing the configuration for Inline IPS Mode operation for the WAN interface of my test virtual machine. In the Block Settings area check the box to Block Offenders and then choose Inline Mode in the IPS Mode drop-down selector.

Here is a custom DROP rule I created for simple testing. This is a capture from the RULES tab with Custom Rules selected.

From a different Kali Linux virtual machine located in WAN network segment of the Snort machine I executed the following command to trigger the DROP rule. The Snort test firewall was at IP address 192.168.10.20.

Notice there were no replies to the ICMP ping. I executed a CTRL-C to exit the ping utility.

And finally, here is the ALERTS tab of the Snort WAN interface on the test virtual machine. Notice 3 triggers of the custom DROP rule: one for each ping try before I stopped it with CTRL-C. The red thumbs-down icon represents a DROP action while the yellow exclamation point triangle represents an ALERT action.

IP address 192.168.10.21 is the Kali Linux machine where the ping was executed, and address 192.168.10.20 was the WAN IP of the firewall under test.

bmeeks

The new Snort Inline IPS package pull request is posted for review and then merging into the pfSense-2.5-DEVEL tree. Here is a link to the pull request: https://github.com/pfsense/FreeBSD-ports/pull/645.

I will be posting a separate thread once the new package is merged that will include instructions for upgrading along with configuration steps for the new Inline IPS mode.

tman222

@bmeeks - This is fantastic news! I'm very excited to see that inline capabilities will be available for Snort very soon. I assume the process to enable inline capabilities for an existing Snort configuration will be pretty seamless once it becomes officially available with pfSense 2.5? Thank you for all your hard work on this pfSense package and for implementing this feature, I really appreciate it.

bmeeks

@tman222 -- The process to enable inline operation will be pretty easy. It's just one change to a drop-down selector on the INTERFACE SETTINGS page. However, there are two key caveats.

You must have NIC hardware whose FreeBSD driver supports netmap operation. Currently the officially supported NIC families per the FreeBSD netmap documenation are: em, igb, ixgb, ixl, lem, re and cxgbe.
In order to actually block traffic in the Inline IPS mode, you must change the rule actions for the rules which you wish to block traffic from ALERT to DROP. Rules are always shipped with the action set to ALERT by default. There are two ways to change the rule action (and one additional way, making three in total, if you use the Snort Subscriber Rules). You can use the SID MGMT tab features to create a dropsid.conf file and use the special syntax available for that option to select individual rule SIDs, a range of rule SIDs or entire rule categories whose action will be changed to DROP. The other method is to use a new option on the ALERTS and RULES tabs where you can manually force a change to a rule's action. This works the same way as the current method for force disabling or enabling a rule.
Finally, if you use the Snort Subscriber Rules, you have the third option on the CATEGORIES tab of choosing an IPS Policy and then having Snort automatically set certain rules to DROP based on the IPS Policy metadata encoded into the Snort Subscriber Rules.

Important Difference Between Inline Mode and Legacy Mode
The most important difference between the new Inline IPS Mode and the older Legacy Mode blocking is the fact that, by default, unless you change one or more rule actions, nothing will be blocked. With Inline IPS Mode, alerts no longer trigger blocks. They just generate alerts. Only DROP or REJECT rules will actually impact traffic flow! If you want to block traffic without having to modify rule actions, then you should continue to use the older Legacy Mode blocking.

The Snort inline mode will work the same way as Suricata's inline mode. There are a few Sticky Posts at the top of this IDS/IPS sub-forum that explain the blocking modes for Suricata. Snort will work the same with one exception. There is no "Block Drops Only" mode available in Snort due to limitations in the way the Snort binary works.

bmeeks

This new feature is now available for pfSense 2.5-DEVEL snapshot users. Please help test this new Snort package feature.

Release notes for the package update are in this thread.

Information on how to configure and use the new Inline IPS Mode are in this thread.

gfeiner

Good read. I also read this old post of yours regarding inline mode vs legacy in suricata. What I'm curious to know, with regard to it as a new snort feature, is there a performance benefit to be gained in using inline mode instead of legacy mode?

bmeeks

@gfeiner said in Snort Package - Potential New Feature Teaser:

Good read. I also read this old post of yours regarding inline mode vs legacy in suricata. What I'm curious to know, with regard to it as a new snort feature, is there a performance benefit to be gained in using inline mode instead of legacy mode?

Probably not a large performance benefit, but there could be a little bit as netmap is a very high speed mechanism for userspace programs to send and receive network data to and from kernel space. Legacy Blocking Mode in both packages uses the PCAP library to grab a copy of every packet traversing an interface. The IDS/IPS code then analyzes that packet (and perhaps several in a session depending on the particular rule examining the traffic) before making an ALERT decision. If it decides to ALERT, then a custom output plugin within the Snort or Suricata binary is called, and that custom plugin makes the FreeBSD system call to put the offender's IP address into that special snort2c table used by the pf engine that I've talked about multiple times.

The two main advantages of Inline IPS Mode over Legacy Mode are: (1) no leakage of packets occurs while the IDS/IPS engine is examing the traffic to make a decision; and (2) you can selectively have some rules block (or DROP) traffic while other rules just produce alerts (with no block). This second advantage is actually a really big one in most environments. Consider this use case.

You want to block clients from accessing some particular social media applications. OpenAppID on Snort provides the means to detect traffic coming from or going to such applications. However, the means by which that traffic is blocked can create unintended consequences. With Legacy Mode blocking the only option is to put either the client's IP address, the destination server's IP address, or both in a block rule in the firewall. This is actually what putting the IP address into the snort2c table does in the end. Let's assume you blocked the client for the sake of this example. Now that client is blocked from everything because the client's IP is in the snort2c table. That client can't go anywhere until the block is removed.

Contrast the same scenario above with using Inline IPS mode. Now we can set just those particular social media application rules to DROP and leave other rules at ALERT. So when that client attempts to use their social media app, all traffic matching the social media rule (or rules) can be dropped (the packets are simply discarded). So the client can't connect. However, the client can still do other things that do not trigger a DROP rule. So the client is not completely shut down as is the case with Legacy Mode blocking. They can still get to other network hosts so long as it is not social media application traffic. Thus Inline IPS mode offers the security admin lots of granular control over what is blocked.

jake

I just installed the updated Snort package in my test PfSense 2.5 version firewall. I get this error on my PPP DSL WAN interface from the logs:
FATAL ERROR: Invalid pidfile suffix: _pppoe050442. Suffix must less than 11 characters and not have ".." or "/" in the name.

I noticed this was an issue many, many years ago on the forum and the bug tracker.

Thanks for all your work! I'm excited to use IPS mode.

bmeeks

@jake said in Snort Package - Potential New Feature Teaser:

I just installed the updated Snort package in my test PfSense 2.5 version firewall. I get this error on my PPP DSL WAN interface from the logs:
FATAL ERROR: Invalid pidfile suffix: _pppoe050442. Suffix must less than 11 characters and not have ".." or "/" in the name.

I noticed this was an issue many, many years ago on the forum and the bug tracker.

Thanks for all your work! I'm excited to use IPS mode.

Hmm...not sure a PPPoE connection is going to work. I will have to look into that, but I don't have one readily available to test with. It will take creating at least two pfSense VMs to see if I can let one be the "ISP server end" of the PPPoE connection and another be the client end. Never have tried to set up and use PPPoE on pfSense on the server side of things.

Even if that error about the PID file you posted is fixed, I'm thinking the DAQ netmap code is likely to complain next. I will first try to see about fixing the PID filename length error. That is going to require quite a few edits within the GUI package code, so I will need to be thorough in that review so I don't miss anything. That means it is not an easy 10-minute fix.

Edit: OK, after some further research I have a solution to explore. Turns out this may be somewhat self-induced by the GUI package code, although it is exacerbated by what looks to me to be an arbitrary choice by the Snort binary code creator to choose a max-length for a PID filename.

As the older forum post indicates, this problem goes back a long way. All the way back to the original Snort package creator (user "jamesdean") in 2009 or so. I've never made changes to this part of the code as editing it would have ramifications all over the place. May be forced to tackle that now, though.

Whether or not a given user hits the error is dependent first on the type of interface. It will likely only show up on PPPoE or VLAN interfaces with long names. The second part required to trigger the error is more random. The Snort GUI chooses a psuedo-UUID (unique identifier) for tagging each interface. This UUID is just an unique random number between 1 and 65,535. It is unique in that it is not duplicated within any other Snort interface on your firewall. This random UUID is appended to the end of the interface name to create the PID filename. So if your UUID length is 5, and your interface name is long enough (including an underscore to separate interface name and GUID), then this error is triggered.

I will work on a fix, but because so many parts of the GUI code use this UUID and interface name relationship to make decisions, I have to be very thorough and fix all references in all of the GUI code. That's going to take me a little time. I did make a change in the GUI code to support DAQ inline operation with netmap that is making this problem more likely now for users with a PPPoE interface.

tman222

@bmeeks - thanks for this additional information, I really appreciate it. I very much look forward to trying the inline capabilities once pfSense 2.5 is released. Since I (unfortunately) don't have an extra box for testing 2.5, it would be great if this capability could also be made available as an optional upgrade for pfSense 2.4, but I realize that this is probably a lot more work and maybe not worth doing at this point (with 2.5 coming up).

Thank you again for all you hard work on getting this feature implemented in the Snort package.

bmeeks

@tman222 said in Snort Package - Potential New Feature Teaser:

... it would be great if this capability could also be made available as an optional upgrade for pfSense 2.4, but I realize that this is probably a lot more work and maybe not worth doing at this point (with 2.5 coming up).

If pfSense-2.5 is delayed for a long time, then I will see about backporting the new Snort package to 2.4.x. But the new package needs a little time in the pfSense-2.5-DEVEL tree to get some shakedown testing. This is new territory in the Snort package, so I expect a few bugs to show up. Already one has reared its head with the error message user @jake posted up above. I'm looking into getting that problem squared away now. The 2.5-DEVEL tree is designed to shake out bugs and get things in shape for moving to RELEASE.

bmeeks

@jake said in Snort Package - Potential New Feature Teaser:

I just installed the updated Snort package in my test PfSense 2.5 version firewall. I get this error on my PPP DSL WAN interface from the logs:
FATAL ERROR: Invalid pidfile suffix: _pppoe050442. Suffix must less than 11 characters and not have ".." or "/" in the name.

I noticed this was an issue many, many years ago on the forum and the bug tracker.

Thanks for all your work! I'm excited to use IPS mode.

@jake: keep watch for a Snort GUI package update to version 4.0_1 in the very near future. Try that package when it shows up in pfSense-2.5-DEVEL and let me know if it fixes your fatal startup up with the "too long" PID filename.

IMPORTANT: when you do the upgrade, first remove your existing Snort package using the trash can icon on the PACKAGE MANAGER page. Then install Snort again. If you just click the reinstall icon, a critical file will not get immediately updated and you will have issues on the first restart after the update.

I posted the fix in this pull request: https://github.com/pfsense/FreeBSD-ports/pull/646.

The update for the Snort package to correct this startup error should be posted now. It is Snort v4.0_1. Here is a link to the Release Notes thread.

jake

It's starting fine now. Thanks! Right now I'm not getting any alerts or blocked entries but I assume I need to follow the instructions to set IPS mode.

bmeeks

@jake said in Snort Package - Potential New Feature Teaser:

It's starting fine now. Thanks! Right now I'm not getting any alerts or blocked entries but I assume I need to follow the instructions to set IPS mode.

Even though it starts, you may be experiencing the same PPPoE issue that plagues Suricata with inline mode: namely it appears that with netmap on FreeBSD, when you have a PPP connection, the traffic goes around the netmap pipe and not through it. Since the IPS never sees the traffic, it won't alert on it and can't block it.

This is what I meant by my original statement about not being sure PPPoE would work with inline mode. At least on Suricata, it is an operating system limitation and not a problem within the IDS/IPS binary code.

Go ahead and follow the Inline set up instructions I posted in this thread. Please keep me updated on how it works (or does not) for you.

Edit: I assume you have the PPPoE connection on your WAN. If you do not have any open Internet-facing ports on the WAN, then there is no real security benefit of running Snort on the WAN. Just let the default pfSense policy of dropping all unsolicited inbound connections on the WAN take of the security there. You can instead run Snort on your LAN interface (or interfaces in the case of multiple local segments). That will not have a PPPoE interface, so Snort will work fine there.

jake

I've been out of town all week and haven't had time to work on this. I back now and have set my three internal networks in IPS mode and removed Snort from the PPP WAN connection. However, I get this panic message a few minutes after reboot. I've copied it from the console:

Fatal trap 12: page fault while in kernel mode
cpuid = 2; apic id = 04
fault virtual address   = 0x28
fault code              = supervisor read data, page not present
instruction pointer     = 0x20:0xffffffff80e426be
stack pointer           = 0x28:0xfffffe0055d10640
frame pointer           = 0x28:0xfffffe0055d10670
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 84005 (snort)
[ thread pid 84005 tid 100512 ]
Stopped at      ether_8021q_frame+0x2e: movq    ll+0x7(%rax),%rax
db:0:kdb.enter.default> textdump set
textdump set
db:0:kdb.enter.default>  capture on
db:0:kdb.enter.default>  run lockinfo

Could it be my configuration causing this? I'm running the latest development build:
2.5.0-DEVELOPMENT (amd64)
built on Thu Jun 06 06:43:40 EDT 2019

bmeeks

@jake: What kind of NIC hardware is in the box? Inline Mode for both Snort and Suricata utilizes the netmap driver to implement pipes between the NIC interface driver and the kernel network stack. Only these series of drivers on FreeBSD are known to work: em, igb, ixgb, ixl, lem, re or cxgbe. And even some of these "supported" drivers may still need tweaking to run properly.

You say your WAN is a PPPoE connection, that is most likely not going to work with Inline Mode. At least I have not tested it. Another user here also had problems with Inline Mode and PPPoE. Legacy Mode is probably what you should stick with when you have a PPP connection on the interface.

jake

They are all Intel I210-AT NICs. Would the PPPoE still be an issue if I'm running Inline mode only on the LAN side? I also had it Inline Mode on my VLAN interfaces, could that be an issue? Thanks for all your help. I'm happy to do any testing if needed.

bmeeks

@jake said in Snort Package - Potential New Feature Teaser:

They are all Intel I210-AT NICs. Would the PPPoE still be an issue if I'm running Inline mode only on the LAN side? I also had it Inline Mode on my VLAN interfaces, could that be an issue? Thanks for all your help. I'm happy to do any testing if needed.

No, so long as the interface is not using PPPoE, the Intel i210-AT chip should be using the igb or perhaps the em driver. First you need to verify that by looking under STATUS > INTERFACES in the pfSense menu and finding your LAN interface. The type of driver will be shown in parentheses in the header band for each interface. If your LAN is not showing one of the compatible netmap driver families I posted earlier, then inline mode likely won't work for you.

If you have one of the listed NIC driver families, then try altering the settings mentioned in this post: https://forum.netgate.com/topic/138613/configuring-pfsense-netmap-for-suricata-inline-ips-mode-on-em-igb-interfaces.