Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0
-
Thanks @bmeeks - I did end up running a few more speed tests with inline mode enabled on just the WAN (igb) interface and (oddly) saw quite a range of numbers: From as little as 100Mbit/s up/down (speedtest.net) to almost full line speed (Netflix's Fast.com) with several tests coming in between 200 to 300 Mbit/s (similar to what I saw with iperf3). I did only have a handful of ET rules enabled on the WAN interface, but even adding in the Connectivity rule set didn't have too big of an impact on speed (just a slight decrease). I also took some time to study the netmap documentation:
https://www.freebsd.org/cgi/man.cgi?query=netmap&sektion=4&manpath=FreeBSD+11.3-RELEASE
Thanks for providing some additional insight above about how you choose to change DAQ to use a netmap connection to the host stack. I'm trying to visualize how this all fits together so I can narrow down the relevant bottlechecks - would be it be something like this?
NIC --> Netmap --> Snort --> Netmap --> Host Stack
I'm still not convinced yet that a lack of processing power is the primary issue for me - the bottleneck could very well be the single host ring between netmap and the host stack, as the (Chelsio NIC) does support multiple netmap queues on it's virtual interfaces. I did enable preprocessing statistics as well on the WAN (igb) interface and looked at the output, but it's not giving me much insight where the majority of CPU time is being spent (to help narrow down potentially alleviate any bottlenecks). I thought I also read somewhere that there is a way to profile Snort in order to see that - do you know if that's possible to do on the pfSense / FreeBSD implementation?
In any case, I think further exploratory analysis is probably warranted before switching over to Suricata. I do have some additional hardware lying around here to build a test system for pfSense 2.5, including an additional Intel and Chelsio NIC, along with a CPU that will have better single thread performance (Intel i3-8100 3.6GHz quad core). I would first need to test with 2.4.5 to see if the CPU is indeed the primary bottleneck and then move up to 2.5 to see if now having the additional host rings will increase throughput. In the meantime, I will probably explore the netmap tunable parameters a bit more to see if any of them might make a noticeable difference on performance (especially parameters related to buffer and ring sizing).
Thanks again for all your help!
-
@tman222
I don't know the specific setup of youriperf3
testing, but when usingiperf
you should test THROUGH the firewall and not TO the firewall. In other words, do not put either theiperf
client or server on the firewall. They both should instead be on workstations or servers hanging off two different firewall interfaces so that you test through the firewall.When you run
ipferf
on the firewall itself, whether it's the client or server piece, there is some CPU time and potential thoughput chewed up by theiperf
executable.I mention this just because some other users in the past have tested with
iperf
installed on the firewall and have gotten misleading throughput numbers as a result. -
@bmeeks said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
@tman222
I don't know the specific setup of youriperf3
testing, but when usingiperf
you should test THROUGH the firewall and not TO the firewall. In other words, do not put either theiperf
client or server on the firewall. They both should instead be on workstations or servers hanging off two different firewall interfaces so that you test through the firewall.When you run
ipferf
on the firewall itself, whether it's the client or server piece, there is some CPU time and potential thoughput chewed up by theiperf
executable.I mention this just because some other users in the past have tested with
iperf
installed on the firewall and have gotten misleading throughput numbers as a result.Hi @bmeeks - sorry for not being clear in my original post, but the iperf3 test was performed between a (bare metal) Linux host on my home network and a cloud based VM (i.e. I was testing through the firewall).
One thing I forgot to ask - when you have a moment, do you mind asking the folks at Netgate what hardware and any specific sysctl tuning they used that allowed them to get close to 1Gbit/s line rate using Snort in IPS (inline) mode? I realize it was for 2.5, but it would be great to have another data point to calibrate my results to and help me further narrow down what might be causing the performance issues. Thanks again!
-
@bmeeks - well, I might have been wrong. It is starting to look like the CPU may be the bottleneck here. I experimented and reduced the number of rules on the WAN interface and network throughput started to increase. By time I was just using the basic Connectivity IPS policy (and no ET rules), I was able to get very close to line speed using an iperf3 test (again between a local Linux host and cloud based VM). As I started to increase the number of ET rules, throughput dropped again accordingly.
From all this I gather that Snort needs a quite a beefy CPU with great single thread performance to run in IPS (inline) mode and be able to achieve high throughput (I'm actually starting to wonder now whether 1Gbit/s is even possible with a modest rule set and Snort operating in inline mode). Perhaps it's worth trying out Suricata after all to see if the multi-threading would help?
Thanks again for your help and insight.
-
@tman222 said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
@bmeeks - well, I might have been wrong. It is starting to look like the CPU may be the bottleneck here. I experimented and reduced the number of rules on the WAN interface and network throughput started to increase. By time I was just using the basic Connectivity IPS policy (and no ET rules), I was able to get very close to line speed using an iperf3 test (again between a local Linux host and cloud based VM). As I started to increase the number of ET rules, throughput dropped again accordingly.
From all this I gather that Snort needs a quite a beefy CPU with great single thread performance to run in IPS (inline) mode and be able to achieve high throughput (I'm actually starting to wonder now whether 1Gbit/s is even possible with a modest rule set and Snort operating in inline mode). Perhaps it's worth trying out Suricata after all to see if the multi-threading would help?
Thanks again for your help and insight.
The number of enabled rules and exactly what those rules are doing will have a large impact on throughput. That's why I preach to folks to carefully select the rules they enable so that they match the potential threats faced without bogging down the CPU and thus throttling throughput. Rules that are performing complex pattern matching are naturally going to be more processor intensive than some simple rule triggering on an IP or something similarly simple to examine.
One of my favorite examples to illustrate the "enable only what you need to address the attack surfaces you expose" mantra is to not enable the DNS server, web server or email server rules if you are not running such services behind your firewall. If you don't run a DNS server, then attacks against DNS servers are of no concern in your network. And there are other examples from the inventory of rules.
-
@bmeeks I am hitting the pidfile suffix length issue as well:
FATAL ERROR: Invalid pidfile suffix: _vtnet0.100. Suffix must less than 11 characters and not have ".." or "/" in the name.
I suspect it never came up in testing because it appears to require a combination of:
- pfSense be running in a VM with the
virtio
network adapter (this is the Netgate recommended NIC type for Proxmox) - VLANs be handled in the guest, instead of the hypervisor presenting multiple virtual NICs (there are good reasons to do this in some setups)
- A 3 or 4-digit VLAN ID (perhaps most people use short VLAN IDs?)
Unfortunately, it seems it isn't really possible to rename the interface on pfSense, as it gets reverted every reboot. And re-configuring the site's VLANs isn't exactly an ideal solution either.
Instead of changing the pfSense package to create a random pidfile suffix, is the Snort team open to increasing the maximum length? 12 characters would allow 10
virtio
NICs and the maximum VLAN ID of 4095. 13 characters would allow 100virtio
NICs, which "should be enough for anybody" (famous last words). - pfSense be running in a VM with the
-
@DAVe3283 said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
@bmeeks I am hitting the pidfile suffix length issue as well:
FATAL ERROR: Invalid pidfile suffix: _vtnet0.100. Suffix must less than 11 characters and not have ".." or "/" in the name.
I suspect it never came up in testing because it appears to require a combination of:
- pfSense be running in a VM with the
virtio
network adapter (this is the Netgate recommended NIC type for Proxmox) - VLANs be handled in the guest, instead of the hypervisor presenting multiple virtual NICs (there are good reasons to do this in some setups)
- A 3 or 4-digit VLAN ID (perhaps most people use short VLAN IDs?)
Unfortunately, it seems it isn't really possible to rename the interface on pfSense, as it gets reverted every reboot. And re-configuring the site's VLANs isn't exactly an ideal solution either.
Instead of changing the pfSense package to create a random pidfile suffix, is the Snort team open to increasing the maximum length? 12 characters would allow 10
virtio
NICs and the maximum VLAN ID of 4095. 13 characters would allow 100virtio
NICs, which "should be enough for anybody" (famous last words).I think I have a fix. I'm testing now to be sure there are no adverse side effects.
The 11-character limit imposed by the Snort binary stems, most likely, from the 16-character limit on process names in most Unix-based operating systems. The Snort binary prepends "snort" to any PID file suffix the user provides. So that's 5 characters right there, leaving 11 for the user's suffix. As you say, with certain combinations of long interface names and VLAN IDs, the user-supplied suffix can exceed 11 characters.
The only real need is to be able to uniquely identify the PID file for a given interface so the GUI can control the Snort binary. I can accomplish that using a different interface value generated each time you create a Snort interface. That is the UUID field. It's a random value between 1 and 65535, so it will never exceed a 5-character string and thus will be safely within the 11-character Snort limit. It is random when generated at the time the Snort interface is created, but stays static after that.
Look for the fix with this change in the near future. Hopefully later today if the pfSense team can quickly get the change reviewed and merged.
- pfSense be running in a VM with the
-
-
@bmeeks said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
@Beerman and @DAVe3283, plus any others affected by the Snort startup failure with the "PID filename suffix must be less than 11 characters" error.
The fix for this has been posted for review and merging by the pfSense developer team. Look for Snort package version 4.1.2_1 in the near future.
Thx!
-
Both pull requests have been merged. I've already updated my production system, but mh SG-5100 was not affected by the bug.
-
@bmeeks said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
Both pull requests have been merged. I've already updated my production system, but mh SG-5100 was not affected by the bug.
Hi @bmeeks - after this update, Snort will no longer start for me. I see these errors:
/tmp/snort_vcxl0_startcmd.php: The command '/usr/local/bin/snort -R _3137 -D -q --suppress-config-log -Q --daq netmap -l /var/log/snort/snort_vcxl03137 --pid-path /var/run --nolock-pidfile --no-interface-pidfile -G 3137 -c /usr/local/etc/snort/snort_3137_vcxl0/snort.conf -i vcxl0^:vcxl0' returned exit code '1', the output was ''
and
FATAL ERROR: /usr/local/etc/snort/snort_3137_vcxl0/snort.conf(174) => Did not find specified IIS Unicode codemap in the specified IIS Unicode Map file.
Any ideas?
-
@tman222 said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
@bmeeks said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
Both pull requests have been merged. I've already updated my production system, but mh SG-5100 was not affected by the bug.
Hi @bmeeks - after this update, Snort will no longer start for me. I see these errors:
/tmp/snort_vcxl0_startcmd.php: The command '/usr/local/bin/snort -R _3137 -D -q --suppress-config-log -Q --daq netmap -l /var/log/snort/snort_vcxl03137 --pid-path /var/run --nolock-pidfile --no-interface-pidfile -G 3137 -c /usr/local/etc/snort/snort_3137_vcxl0/snort.conf -i vcxl0^:vcxl0' returned exit code '1', the output was ''
and
FATAL ERROR: /usr/local/etc/snort/snort_3137_vcxl0/snort.conf(174) => Did not find specified IIS Unicode codemap in the specified IIS Unicode Map file.
Any ideas?
Easy fix. Your
unicode.map
file got clobbered somehow. Simply delete the Snort package and install it again. Do NOT use the "update" icon. Use the Delete icon and then reinstall from the Available Packages tab.Not sure why some users hit this error. I have never gotten it in all the hundreds of times I've upgraded and/or green-field installed Snort on my test virtual machines.
The only possible explanation might be if you are not using any Snort Subscriber Rules. Those rules contain this file. The Emerging Threats rules do not. So if this file gets clobbered, and you don't have the Snort Subscriber Rules enabled, then the only way to get the file back is to reinstall Snort.
-
@bmeeks - thanks, that worked! After the package installed the first time I actually experienced a system crash - the error pointed to netmap rx queue over-allocation. I had been trying to tune netmap device parameters a bit for performance. I went ahead and reduced the number of netmap rx/tx queues and all has been well since. Not sure if these issues were ultimately related, but could potentially have been a contributing factor. Thanks again for all your help.
-
I went ahead today and changed my entire Snort configuration over from legacy (pcap) mode (IDS) to inline (netmap) mode (IPS) and learned quite a bit along the way. Inline mode does indeed require quite a bit of CPU horsepower to get decent throughput. While I had no trouble hitting 1Gbit/s+ using Snort's legacy (pcap) mode, I could barely scratch 200Mbit/s once I enabled inline mode. By running a variety of tests I concluded that the slowdown can be attributed to a combination of running out of CPU resources and a netmap limitation in FreeBSD 11.3, namely that only one rx/tx host ring exists between netmap and the host stack.
Through some additional tuning and adjustment, I was able to get that throughput number up some - below are some of my notes:
-
I have Snort enabled in inline mode on a Chelsio T540-SO-CR 4 port SFP+ card, where two interfaces are physical only interfaces (no VLAN's) and the other two interfaces that each have three VLAN's on them. Netmap will work natively on the two physical only interfaces as soon as virtual interfaces are enabled on the Chelsio card, as these have netmap support (see my previous post above for more details). Netmap will still work in emulated mode on the other two interfaces that have VLAN's on them (please see https://github.com/luigirizzo/netmap/issues/302). Interestingly, the performance (throughout) is actually better if netmap operates in emulated mode on the virtual interface as opposed to regular interfaces so it is worth changing the VLAN's over to a virtual interfaces as well.
-
I then took a very hard look at the rules I had enabled to try to reduce processing overhead. Some rule sets are particularly taxing on the CPU (I noticed this especially in the Emerging Threats data set, for example). Ultimately, I followed a approach similar to what was suggested here https://forum.ipfire.org/viewtopic.php?t=21521 and used a combination of Snort VRT, Snort Community, and ET Open rule sets. At present I have ~14000 rules active on each interface, which gives me a throughput of approximately 350-400Mbit/s up and down. With a bit more tweaking of the Snort Balanced IPS policy I hope to get this closer to 500Mbit/s. And this may very well be the realistic limit that the 2.2GHz Xeon D-1518 CPU in my firewall can handle. However, since it is a multicore CPU, this is the essentially the throughput limit per interface, i.e. with multiple interfaces sending data simultaneously I am able to max out my 1Gbit fiber circuit (since CPU cores will be dedicated to different Snort instances running on the various interfaces). To get better throughput than this I would probably have to upgrade to faster CPU in the 3.5-4GHz+ range. Having multiple rx/tx host rings in FreeBSD 12 (pfSense 2.5) should also help. I do see netmap transmit errors from time to time when I'm maxing out the throughput on a given interface, but hopefully multiple host rings will help solve this as well.
Overall this quite a humbling experience and really made me appreciate the need to carefully tune IDS/IPS rule sets to get desired performance while at the same time not trading off too much security benefit. Thanks again to @bmeeks for maintaining this package and for all your help and insight - it's truly appreciated.
-
-
@bmeeks said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
@Beerman and @DAVe3283, plus any others affected by the Snort startup failure with the "PID filename suffix must be less than 11 characters" error.
The fix for this has been posted for review and merging by the pfSense developer team. Look for Snort package version 4.1.2_1 in the near future.
The fix is working for me. Thx! :)
-
@bmeeks said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
@Beerman and @DAVe3283, plus any others affected by the Snort startup failure with the "PID filename suffix must be less than 11 characters" error.
The fix for this has been posted for review and merging by the pfSense developer team. Look for Snort package version 4.1.2_1 in the near future.
Updated all my instances and everything seems to be working, thanks!
-
Thanks for all the hard work in bringing this update to us.
Question: Any new setting recommendations/changes for folks using Netgate SG-1100? Is it recommend to use the new Inline IPS Mode? Default is set as Legacy.
Thanks in advance,
-
@costanzo said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
Thanks for all the hard work in bringing this update to us.
Question: Any new setting recommendations/changes for folks using Netgate SG-1100? Is it recommend to use the new Inline IPS Mode? Default is set as Legacy.
Thanks in advance,
I think that depends on your use case - do you need intrusion prevention (IPS) or are you with just intrusion protection (IDS)? Please see this thread for more info on the two different operating modes:
https://forum.netgate.com/topic/143812/snort-package-4-0-inline-ips-mode-introduction-and-configuration-instructions
Having said that, from my experience IPS (inline) mode requires more CPU horsepower than IDS (legacy) mode. Also, I'm not sure whether the SG-1100 supports Netmap natively on its network interfaces. If does not, it may still run in emulated mode, but possibly with a performance penalty. Maybe someone else can chime in here with their experiences using that particular Netgate system. In any case, if you do decide to experiment with IPS mode, be sure to make a backup of your configuration first in the event that issues arise.
Hope this helps.
-
@costanzo said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
Thanks for all the hard work in bringing this update to us.
Question: Any new setting recommendations/changes for folks using Netgate SG-1100? Is it recommend to use the new Inline IPS Mode? Default is set as Legacy.
Thanks in advance,
I would recommend sticking with Legacy Mode on the SG-1100. First because of the extra CPU resources required for Inline IPS Mode at high packet rates. Second, I'm not sure the SG-1100 NICs support native netmap. That would mean the kernel would open an emulated netmap adaptor for them resulting in a substantial performance penalty as @tman222 has already noted.
-
As I have been running Snort in inline (IPS) mode for a few days now, I thought I would share a few more notes based on my observations and tuning / tweaking attempts:
-
Caution must be taken when adjusting sysctl tunables for the netmap device. I've experienced several system crashes (thankfully all recovered fine) due to insufficient (buffer) memory, while trying to increase netmap queue and buffer sizes. The
dev.netmap.buf_num
tunable appears key in particular - make sure to increase it a bit first if you are planning on increasing ring or buffer sizes (please see https://www.freebsd.org/cgi/man.cgi?query=netmap&sektion=4&manpath=FreeBSD+11.3-RELEASE) That being said, if you're in doubt about a settings/tunable change, always make a pfSense configuration backup first. -
When running an iperf3 test between two hosts on different local subnets (LAN) or between a local host and a cloud VM, I'm able to achieve about ~400Mbit/s up and down of throughput. This is with ~10000 active rules enabled on the interfaces. However, when running other speed tests the results can deviate from that. In particular running a speed test that measures buffer bloat (e.g. DSL Reports), I will see significant bloat developing on the upload side and the System Logs will fill up with
netmap_transmit
buffer full errors. I've been able to get around this by now using Limiters and limiting the upload speed to ~150Mbit/s. Download speed, however, can be much higher at ~400-500Mbit/s. Using these settings gets rids of bufferbloat, keeps keeps transmit errors to a minimum (if any show up at all), and results in stable latencies. This is of course not ideal given on a symmetric 1Gbit/s fiber circuit that I currently have, but thankfully is really a per interface limit. This means multiple interfaces can transmit up to 150Mbit/s without any type of bufferbloat developing, assuming a different CPU core is assigned to Snort running on each of the interfaces. I'm not convinced that raw CPU processing speed is the sole issue here, otherwise I would think that I would see lower a throughput limits on the download side as well. The FreeBSD 11 netmap API limit of one host stack rx/tx ring likely has impact as well and could explain the speed disparity: When downloading, packets go from one host stack queue to multiple NIC queues (i.e. "fan out"). When uploading packets go from multiple NIC queues to one host stack queue (i.e. "fan in"). The website from the creators of Netmap has some good diagrams (see poster inset on right side of the page) that illustrate this: http://info.iet.unipi.it/~luigi/netmap/ I'll be very curious to see what impact multiple host stack rx/tx rings will make in pfSense 2.5 / FreeBSD 12. @bmeeks - is there anything special we would have to do to enable multiple host stack rx/tx rings, or will this be set automatically? -
With Snort running inline I've also seen packet latencies go up somewhat (though not in a material way) - my guess is around 200 - 300 microseconds (0.2 - 0.3ms) per Snort sensor (binary). Putting this into context, when pinging between hosts on two different network segments that both have Snort running inline I see an increase of about 0.5 - 0.6ms. This likely a function of the number of active rules as well.
-
I have Snort working inline on VLAN only interfaces, but it will run in emulated mode. So far I've not seen a performance hit by running in emulated mode. These particular interfaces run only VLAN's so there was no need to disable VLAN hw tagging or VLAN hw filtering.
Overall, things have been running pretty well and are stable. I'm not too thrilled about the throughput limit that I'm seeing but I'll take that as a speed / security trade off for the timebeing. Hopefully that will improve in pfSense 2.5 / FreeBSD 12 - unfortunately I'm out of ideas as the moment of what could be tuned further to increase performance in pfSense 2.4.5 / FreeBSD 11 (still researching).
I hope these notes are helpful to those are also looking to implement Snort in inline (IPS) mode.
-