Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0
-
@bmeeks said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
Both pull requests have been merged. I've already updated my production system, but mh SG-5100 was not affected by the bug.
Hi @bmeeks - after this update, Snort will no longer start for me. I see these errors:
/tmp/snort_vcxl0_startcmd.php: The command '/usr/local/bin/snort -R _3137 -D -q --suppress-config-log -Q --daq netmap -l /var/log/snort/snort_vcxl03137 --pid-path /var/run --nolock-pidfile --no-interface-pidfile -G 3137 -c /usr/local/etc/snort/snort_3137_vcxl0/snort.conf -i vcxl0^:vcxl0' returned exit code '1', the output was ''
and
FATAL ERROR: /usr/local/etc/snort/snort_3137_vcxl0/snort.conf(174) => Did not find specified IIS Unicode codemap in the specified IIS Unicode Map file.
Any ideas?
-
@tman222 said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
@bmeeks said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
Both pull requests have been merged. I've already updated my production system, but mh SG-5100 was not affected by the bug.
Hi @bmeeks - after this update, Snort will no longer start for me. I see these errors:
/tmp/snort_vcxl0_startcmd.php: The command '/usr/local/bin/snort -R _3137 -D -q --suppress-config-log -Q --daq netmap -l /var/log/snort/snort_vcxl03137 --pid-path /var/run --nolock-pidfile --no-interface-pidfile -G 3137 -c /usr/local/etc/snort/snort_3137_vcxl0/snort.conf -i vcxl0^:vcxl0' returned exit code '1', the output was ''
and
FATAL ERROR: /usr/local/etc/snort/snort_3137_vcxl0/snort.conf(174) => Did not find specified IIS Unicode codemap in the specified IIS Unicode Map file.
Any ideas?
Easy fix. Your
unicode.map
file got clobbered somehow. Simply delete the Snort package and install it again. Do NOT use the "update" icon. Use the Delete icon and then reinstall from the Available Packages tab.Not sure why some users hit this error. I have never gotten it in all the hundreds of times I've upgraded and/or green-field installed Snort on my test virtual machines.
The only possible explanation might be if you are not using any Snort Subscriber Rules. Those rules contain this file. The Emerging Threats rules do not. So if this file gets clobbered, and you don't have the Snort Subscriber Rules enabled, then the only way to get the file back is to reinstall Snort.
-
@bmeeks - thanks, that worked! After the package installed the first time I actually experienced a system crash - the error pointed to netmap rx queue over-allocation. I had been trying to tune netmap device parameters a bit for performance. I went ahead and reduced the number of netmap rx/tx queues and all has been well since. Not sure if these issues were ultimately related, but could potentially have been a contributing factor. Thanks again for all your help.
-
I went ahead today and changed my entire Snort configuration over from legacy (pcap) mode (IDS) to inline (netmap) mode (IPS) and learned quite a bit along the way. Inline mode does indeed require quite a bit of CPU horsepower to get decent throughput. While I had no trouble hitting 1Gbit/s+ using Snort's legacy (pcap) mode, I could barely scratch 200Mbit/s once I enabled inline mode. By running a variety of tests I concluded that the slowdown can be attributed to a combination of running out of CPU resources and a netmap limitation in FreeBSD 11.3, namely that only one rx/tx host ring exists between netmap and the host stack.
Through some additional tuning and adjustment, I was able to get that throughput number up some - below are some of my notes:
-
I have Snort enabled in inline mode on a Chelsio T540-SO-CR 4 port SFP+ card, where two interfaces are physical only interfaces (no VLAN's) and the other two interfaces that each have three VLAN's on them. Netmap will work natively on the two physical only interfaces as soon as virtual interfaces are enabled on the Chelsio card, as these have netmap support (see my previous post above for more details). Netmap will still work in emulated mode on the other two interfaces that have VLAN's on them (please see https://github.com/luigirizzo/netmap/issues/302). Interestingly, the performance (throughout) is actually better if netmap operates in emulated mode on the virtual interface as opposed to regular interfaces so it is worth changing the VLAN's over to a virtual interfaces as well.
-
I then took a very hard look at the rules I had enabled to try to reduce processing overhead. Some rule sets are particularly taxing on the CPU (I noticed this especially in the Emerging Threats data set, for example). Ultimately, I followed a approach similar to what was suggested here https://forum.ipfire.org/viewtopic.php?t=21521 and used a combination of Snort VRT, Snort Community, and ET Open rule sets. At present I have ~14000 rules active on each interface, which gives me a throughput of approximately 350-400Mbit/s up and down. With a bit more tweaking of the Snort Balanced IPS policy I hope to get this closer to 500Mbit/s. And this may very well be the realistic limit that the 2.2GHz Xeon D-1518 CPU in my firewall can handle. However, since it is a multicore CPU, this is the essentially the throughput limit per interface, i.e. with multiple interfaces sending data simultaneously I am able to max out my 1Gbit fiber circuit (since CPU cores will be dedicated to different Snort instances running on the various interfaces). To get better throughput than this I would probably have to upgrade to faster CPU in the 3.5-4GHz+ range. Having multiple rx/tx host rings in FreeBSD 12 (pfSense 2.5) should also help. I do see netmap transmit errors from time to time when I'm maxing out the throughput on a given interface, but hopefully multiple host rings will help solve this as well.
Overall this quite a humbling experience and really made me appreciate the need to carefully tune IDS/IPS rule sets to get desired performance while at the same time not trading off too much security benefit. Thanks again to @bmeeks for maintaining this package and for all your help and insight - it's truly appreciated.
-
-
@bmeeks said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
@Beerman and @DAVe3283, plus any others affected by the Snort startup failure with the "PID filename suffix must be less than 11 characters" error.
The fix for this has been posted for review and merging by the pfSense developer team. Look for Snort package version 4.1.2_1 in the near future.
The fix is working for me. Thx! :)
-
@bmeeks said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
@Beerman and @DAVe3283, plus any others affected by the Snort startup failure with the "PID filename suffix must be less than 11 characters" error.
The fix for this has been posted for review and merging by the pfSense developer team. Look for Snort package version 4.1.2_1 in the near future.
Updated all my instances and everything seems to be working, thanks!
-
Thanks for all the hard work in bringing this update to us.
Question: Any new setting recommendations/changes for folks using Netgate SG-1100? Is it recommend to use the new Inline IPS Mode? Default is set as Legacy.
Thanks in advance,
-
@costanzo said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
Thanks for all the hard work in bringing this update to us.
Question: Any new setting recommendations/changes for folks using Netgate SG-1100? Is it recommend to use the new Inline IPS Mode? Default is set as Legacy.
Thanks in advance,
I think that depends on your use case - do you need intrusion prevention (IPS) or are you with just intrusion protection (IDS)? Please see this thread for more info on the two different operating modes:
https://forum.netgate.com/topic/143812/snort-package-4-0-inline-ips-mode-introduction-and-configuration-instructions
Having said that, from my experience IPS (inline) mode requires more CPU horsepower than IDS (legacy) mode. Also, I'm not sure whether the SG-1100 supports Netmap natively on its network interfaces. If does not, it may still run in emulated mode, but possibly with a performance penalty. Maybe someone else can chime in here with their experiences using that particular Netgate system. In any case, if you do decide to experiment with IPS mode, be sure to make a backup of your configuration first in the event that issues arise.
Hope this helps.
-
@costanzo said in Upcoming Snort Package Updates for pfSense-2.4.5 and pfSense-2.5.0:
Thanks for all the hard work in bringing this update to us.
Question: Any new setting recommendations/changes for folks using Netgate SG-1100? Is it recommend to use the new Inline IPS Mode? Default is set as Legacy.
Thanks in advance,
I would recommend sticking with Legacy Mode on the SG-1100. First because of the extra CPU resources required for Inline IPS Mode at high packet rates. Second, I'm not sure the SG-1100 NICs support native netmap. That would mean the kernel would open an emulated netmap adaptor for them resulting in a substantial performance penalty as @tman222 has already noted.
-
As I have been running Snort in inline (IPS) mode for a few days now, I thought I would share a few more notes based on my observations and tuning / tweaking attempts:
-
Caution must be taken when adjusting sysctl tunables for the netmap device. I've experienced several system crashes (thankfully all recovered fine) due to insufficient (buffer) memory, while trying to increase netmap queue and buffer sizes. The
dev.netmap.buf_num
tunable appears key in particular - make sure to increase it a bit first if you are planning on increasing ring or buffer sizes (please see https://www.freebsd.org/cgi/man.cgi?query=netmap&sektion=4&manpath=FreeBSD+11.3-RELEASE) That being said, if you're in doubt about a settings/tunable change, always make a pfSense configuration backup first. -
When running an iperf3 test between two hosts on different local subnets (LAN) or between a local host and a cloud VM, I'm able to achieve about ~400Mbit/s up and down of throughput. This is with ~10000 active rules enabled on the interfaces. However, when running other speed tests the results can deviate from that. In particular running a speed test that measures buffer bloat (e.g. DSL Reports), I will see significant bloat developing on the upload side and the System Logs will fill up with
netmap_transmit
buffer full errors. I've been able to get around this by now using Limiters and limiting the upload speed to ~150Mbit/s. Download speed, however, can be much higher at ~400-500Mbit/s. Using these settings gets rids of bufferbloat, keeps keeps transmit errors to a minimum (if any show up at all), and results in stable latencies. This is of course not ideal given on a symmetric 1Gbit/s fiber circuit that I currently have, but thankfully is really a per interface limit. This means multiple interfaces can transmit up to 150Mbit/s without any type of bufferbloat developing, assuming a different CPU core is assigned to Snort running on each of the interfaces. I'm not convinced that raw CPU processing speed is the sole issue here, otherwise I would think that I would see lower a throughput limits on the download side as well. The FreeBSD 11 netmap API limit of one host stack rx/tx ring likely has impact as well and could explain the speed disparity: When downloading, packets go from one host stack queue to multiple NIC queues (i.e. "fan out"). When uploading packets go from multiple NIC queues to one host stack queue (i.e. "fan in"). The website from the creators of Netmap has some good diagrams (see poster inset on right side of the page) that illustrate this: http://info.iet.unipi.it/~luigi/netmap/ I'll be very curious to see what impact multiple host stack rx/tx rings will make in pfSense 2.5 / FreeBSD 12. @bmeeks - is there anything special we would have to do to enable multiple host stack rx/tx rings, or will this be set automatically? -
With Snort running inline I've also seen packet latencies go up somewhat (though not in a material way) - my guess is around 200 - 300 microseconds (0.2 - 0.3ms) per Snort sensor (binary). Putting this into context, when pinging between hosts on two different network segments that both have Snort running inline I see an increase of about 0.5 - 0.6ms. This likely a function of the number of active rules as well.
-
I have Snort working inline on VLAN only interfaces, but it will run in emulated mode. So far I've not seen a performance hit by running in emulated mode. These particular interfaces run only VLAN's so there was no need to disable VLAN hw tagging or VLAN hw filtering.
Overall, things have been running pretty well and are stable. I'm not too thrilled about the throughput limit that I'm seeing but I'll take that as a speed / security trade off for the timebeing. Hopefully that will improve in pfSense 2.5 / FreeBSD 12 - unfortunately I'm out of ideas as the moment of what could be tuned further to increase performance in pfSense 2.4.5 / FreeBSD 11 (still researching).
I hope these notes are helpful to those are also looking to implement Snort in inline (IPS) mode.
-
-
After some further reflection, I decided to go back to running Snort in legacy (pcap) mode for now. I was able to try out Suricata as well for comparison, but after some initial testing in inline mode I saw similar throughput limitations. Given the interesting asymmetry I saw in upload and download speeds, I plan to revisit inline mode in Snort once pfSense 2.5 is released and there is support for multiple host rx/tx rings to see if that will help improve (upload) throughput.