Snort instance using up 100% of one core at 80-85Mbps
-
I'm very new to pfSense, and Snort. I have been running this setup for around a week now, although now is the first time that I have been able to somewhat sit down and test it.
I know that Snort can be very demanding on the hardware depending on several key settings, including how many rules and preprocessors are loaded in.
However, I am unable to even cap my current connection (100Mbps up and down) with Snort using a fairly powerful CPU and a dedicated dual-port server NIC from Intel.I have a good number of categories loaded in (30 or so), and around 8 preprocessors (notice: even though I unload most of the preprocessors, and categories, the thread usage only lowers by around 5-10%). When idling, Snort is consuming around 25-50% of the thread usage in Top. I am using Emerging, Snort and Snort SO rules (non-overlapping).
I would like some suggestions as to where and how to look for improving the performance. Is there possibly a way to make Snort run in several instances, thus load balancing the work between several cores?
As I said, currently it caps out at around 80Mbps (90-95 Mbps with overhead (dropped packets?), 9k pps).
My firewall hardware is:
Intel i5 3570k @ 3.4GHz
8GB 1600MHz RAM
120GB 2.5" 7200 RPM HDD
Dual-Port Intel server NIC (82571GB) -
For advice on tuning Snort, the Snort mailing list is probably your best bet.
However, start by only using the rules that are relevant. For instance, if you're not running a Web server then the IIS rules aren't very relevant. Similarly if you don't care about portscans, disable the portscan pre-processor, and so on.
-
I'm very new to pfSense, and Snort. I have been running this setup for around a week now, although now is the first time that I have been able to somewhat sit down and test it.
I know that Snort can be very demanding on the hardware depending on several key settings, including how many rules and preprocessors are loaded in.
However, I am unable to even cap my current connection (100Mbps up and down) with Snort using a fairly powerful CPU and a dedicated dual-port server NIC from Intel.I have a good number of categories loaded in (30 or so), and around 8 preprocessors (notice: even though I unload most of the preprocessors, and categories, the thread usage only lowers by around 5-10%). When idling, Snort is consuming around 25-50% of the thread usage in Top. I am using Emerging, Snort and Snort SO rules (non-overlapping).
I would like some suggestions as to where and how to look for improving the performance. Is there possibly a way to make Snort run in several instances, thus load balancing the work between several cores?
As I said, currently it caps out at around 80Mbps (90-95 Mbps with overhead (dropped packets?), 9k pps).
My firewall hardware is:
Intel i5 3570k @ 3.4GHz
8GB 1600MHz RAM
120GB 2.5" 7200 RPM HDD
Dual-Port Intel server NIC (82571GB)Well you must have something wrong in your setup.
I am using PFSense 2.0.1 ona small appliance with 1 Intel Atom Dual Core @ 1.8Ghz (4 threads) ULV processor and 3GB of ram.
I run it with snort premium and emerging threats (all but 3 categories checked) on a FTTH link with 150Mbps symmetric bandwidth and I consistently get ~150Mbps throughput.
My setup regarding snort:
-Activate Barnyard2 with a DB on a second box (mysql server)
-Set Snort to use AC (not the default lowmem) as I have lots of RAM
-Activate all preprocessorsWith this setup and my rules it uses a bit more than 1.5GB of ram but that's what it's there for !
Let me know if this helps…
-
@Cry:
For advice on tuning Snort, the Snort mailing list is probably your best bet.
However, start by only using the rules that are relevant. For instance, if you're not running a Web server then the IIS rules aren't very relevant. Similarly if you don't care about portscans, disable the portscan pre-processor, and so on.
Thank you for your reply, Cry Havok! I am spending time reading official channels for Snort, but this doesn't seem to be one of those instances that make sense because the fault seems a bit elusive. I suspect there's an issue somewhere, rather than just the need for performance optimization.
I did indeed start by doing exactly what you're recommending me, even before reading up in detail about Snort, because this made sense to me. I run a fair number of services on my network, and I do need quite a lot of the categories. However, I did notice that I can turn off at least 5-8 categories with a few tweaks to my network, but considering what others seem to be running, this shouldn't be needed. I'm sure there's a bottleneck somewhere, I just don't know how to find it.
Well you must have something wrong in your setup.
I am using PFSense 2.0.1 ona small appliance with 1 Intel Atom Dual Core @ 1.8Ghz (4 threads) ULV processor and 3GB of ram.
I run it with snort premium and emerging threats (all but 3 categories checked) on a FTTH link with 150Mbps symmetric bandwidth and I consistently get ~150Mbps throughput.
My setup regarding snort:
-Activate Barnyard2 with a DB on a second box (mysql server)
-Set Snort to use AC (not the default lowmem) as I have lots of RAM
-Activate all preprocessorsWith this setup and my rules it uses a bit more than 1.5GB of ram but that's what it's there for !
Let me know if this helps…
Thank you for your reply, teknologist.
The difference on my setup would be that I'm not running premium, I have deselected a lot of categories and preprocessors that I do not need, I have 200Mbps symmetric bandwidth, I do not use Barnyard2 (is it that much better to use it instead of having Snort dump to a txt file?).
When I load up Snort, it consumes 27-30% of my RAM (i.e. 27-30% of 8GB), using AC. I tried putting Snort on my local interface and ran a full performance benchmark with iperf (and nothing else). I seemed to have gotten 930Mbps without any real issue and Snort consumed only 70% on its main thread.However, when I set it on my WAN device, something is causing it to consume much more CPU cycles than it should. If I try running something with several different connections (50-60 connections), such as the latest Ubuntu torrent from their official website, I cap out at circa 10.1MB/s (or ~80Mbps), whilst Snort consumes 100% on its main thread. Could this be an issue related to its ability to process packets per second, rather than aggregate bandwidth?
As I've read about other people running pfSense with Snort on much weaker hardware, with more satisfying results, and similar configurations (or even more demanding configurations), I believe it might be a problem elsewhere. This could also be an optimisation issue, in which case I don't know where to turn other than here or the official Snort channels, as I really don't know how to run performance profiling on this system.
I'd appreciate any additional help or advice I can get.
-
@Cry:
For advice on tuning Snort, the Snort mailing list is probably your best bet.
However, start by only using the rules that are relevant. For instance, if you're not running a Web server then the IIS rules aren't very relevant. Similarly if you don't care about portscans, disable the portscan pre-processor, and so on.
Thank you for your reply, Cry Havok! I am spending time reading official channels for Snort, but this doesn't seem to be one of those instances that make sense because the fault seems a bit elusive. I suspect there's an issue somewhere, rather than just the need for performance optimization.
I did indeed start by doing exactly what you're recommending me, even before reading up in detail about Snort, because this made sense to me. I run a fair number of services on my network, and I do need quite a lot of the categories. However, I did notice that I can turn off at least 5-8 categories with a few tweaks to my network, but considering what others seem to be running, this shouldn't be needed. I'm sure there's a bottleneck somewhere, I just don't know how to find it.
Well you must have something wrong in your setup.
I am using PFSense 2.0.1 ona small appliance with 1 Intel Atom Dual Core @ 1.8Ghz (4 threads) ULV processor and 3GB of ram.
I run it with snort premium and emerging threats (all but 3 categories checked) on a FTTH link with 150Mbps symmetric bandwidth and I consistently get ~150Mbps throughput.
My setup regarding snort:
-Activate Barnyard2 with a DB on a second box (mysql server)
-Set Snort to use AC (not the default lowmem) as I have lots of RAM
-Activate all preprocessorsWith this setup and my rules it uses a bit more than 1.5GB of ram but that's what it's there for !
Let me know if this helps…
Thank you for your reply, teknologist.
The difference on my setup would be that I'm not running premium, I have deselected a lot of categories and preprocessors that I do not need, I have 200Mbps symmetric bandwidth, I do not use Barnyard2 (is it that much better to use it instead of having Snort dump to a txt file?).
When I load up Snort, it consumes 27-30% of my RAM (i.e. 27-30% of 8GB), using AC. I tried putting Snort on my local interface and ran a full performance benchmark with iperf (and nothing else). I seemed to have gotten 930Mbps without any real issue and Snort consumed only 70% on its main thread.However, when I set it on my WAN device, something is causing it to consume much more CPU cycles than it should. If I try running something with several different connections (50-60 connections), such as the latest Ubuntu torrent from their official website, I cap out at circa 10.1MB/s (or ~80Mbps), whilst Snort consumes 100% on its main thread. Could this be an issue related to its ability to process packets per second, rather than aggregate bandwidth?
As I've read about other people running pfSense with Snort on much weaker hardware, with more satisfying results, and similar configurations (or even more demanding configurations), I believe it might be a problem elsewhere. This could also be an optimisation issue, in which case I don't know where to turn other than here or the official Snort channels, as I really don't know how to run performance profiling on this system.
I'd appreciate any additional help or advice I can get.
Well snort is single threaded…so If you hit 100% it will limit. Maybe barnyard helps avoiding overload on snort process.
Try with it.
In my case I had tried the same snort setup (categories/rules) with lowmem perf setting and NO barnyard on a Linux box. Bad results as it capped at 50Mbps with 100% CPU use.
I even tried setting up snort with PF_RING to multi thread it and it was a no go.
Then I moved to pfSense (FreeBSD based instead of linux) and setup as I wrote in previous post and everything flies!
Bad hosts get blocked etc. and throughput doesn't suffer at all and this is in a "low end cpu" intel Atom 1.8Ghz Dual core, snort using only core of course.
Also, while snort loads all rules in memory it stays at 100% for a few minutes, that's normal. After it's loaded it goes back to normal...
Please try the exact same setup I described on my post and let me know, especially the "external" DB setup for barnyard2 collector...
-
Well snort is single threaded…so If you hit 100% it will limit. Maybe barnyard helps avoiding overload on snort process.
Try with it.
In my case I had tried the same snort setup (categories/rules) with lowmem perf setting and NO barnyard on a Linux box. Bad results as it capped at 50Mbps with 100% CPU use.
I even tried setting up snort with PF_RING to multi thread it and it was a no go.
Then I moved to pfSense (FreeBSD based instead of linux) and setup as I wrote in previous post and everything flies!
Bad hosts get blocked etc. and throughput doesn't suffer at all and this is in a "low end cpu" intel Atom 1.8Ghz Dual core, snort using only core of course.
Also, while snort loads all rules in memory it stays at 100% for a few minutes, that's normal. After it's loaded it goes back to normal...
Please try the exact same setup I described on my post and let me know, especially the "external" DB setup for barnyard2 collector...
Interesting post!
I have now setup Snort to use Barnyard2, and to connect to a DB on a server that's elsewhere inside my local network.
Snort is using AC (as before).
I have all preprocessors activated.After I activated the last preprocessor (Sensitive data), Snort stops at 98.97% and I cap out at 10.5MB/s (circa 85Mbps, overhead not included) with BitTorrent traffic. Much better, but still really high usage. If I download via HTTP, I get to around 96Mbps with Snort being at 40% or so.
Are you able to try downloading the Ubuntu BitTorrent file from their website and checking on your Snort thread usage? I would very much appreciate it, as I lack a testbed to compare to.
It would be this one (from their Official website): http://releases.ubuntu.com/12.04/ubuntu-12.04.1-desktop-amd64.iso.torrent -
What version of pfsense. Also 32 or 64 bit?
-
What version of pfsense. Also 32 or 64 bit?
Thanks for your reply, tester_02!
I am running 2.1-BETA0. 64-bit (amd64).