Snort configuration on 1Gbps/1Gbps uplink?

msrachelchen

Hi all… I've noticed that Snort is using 100% of CPU all the time... Not sure if the performance is affected by such..

Box Specs (pfSense 2.4.2-RELEASE-p1):


CPU: Intel(R) Xeon(R) CPU E3-1271 v3 @ 3.60GHz
Memory: 16GB (2 x 8GB) DDR3 1600MHz
WAN Uplink: Intel 82599ES (Uplink is 1Gbps/1Gbps LR fiber, with Intel SFP+ transceiver so that's not a problem...)
WANBridge: Intel I210 (No VLAN so no overhead...)

Currently, state table is around 1.6 million - 21% (1685023/8000000)
MBUF usage is low - 3% (28476/1000000)

Bandwidth usage is around 10MB/s (but it's TONS of small packets and TONS of connections).

Snort (3.2.9.5_4) is using AC-BNFA-NQ, Snort GPLv2 and some ET Open rules.

Thanks!

bmeeks

Well, if my math brain is working on all cylinders today, that works out to call it 100 megabits/sec on the link give or take. 10 megabytes/sec X 8-bits per byte = 80 megabits/sec plus overhead and such, so just conservatively call it 100 megabits/sec.

That should generally not result in 100% CPU utilization of a 3.6 GHz Xeon processor unless we are talking about a ton of enabled rules.

Run this CLI command to be sure you have only a single instance of Snort running on each Snort-configured interface:


ps -ax |grep snort

You should see one line of output for each configured and enabled Snort interface. If you see any duplicates, stop Snort on the interface and then kill any Snort process that remains. Any remaining ones would be potential "zombie" types that can get spun up in certain circumstances when pfSense issues a set of "restart all packages" commands really close together.

Bill

msrachelchen


90425  -  Rs    53:10.51 /usr/local/bin/snort -R 26553 -D -q --suppress-config-
90827  -  S      0:00.00 sh -c  ps -ax |grep snort 2>&1
90966  -  S      0:00.00 grep snort

Just one instance… I do have quite a few rules enabled, maybe that’s the culprit?

msrachelchen

more info if this helps…


State Table                          Total             Rate
  current entries                  1948354               
  searches                      1111196684       145807.2/s
  inserts                        165637337        21734.3/s
  removals                       163686949        21478.4/s


Data Summary
Minimum	Average	Maximum	Last	95th Percentile
inpass	242.51 kpps	242.51 kpps	242.51 kpps	242.51 kpps	0.00 pps
outpass	265.01 kpps	265.01 kpps	265.01 kpps	265.01 kpps	0.00 pps
inblock	23.92 pps	23.92 pps	23.92 pps	23.92 pps	
outblock	0.00 pps	0.00 pps	0.00 pps	0.00 pps	
inpass6	759.76 mpps	759.76 mpps	759.76 mpps	759.76 mpps	0.00 pps
outpass6	0.00 pps	0.00 pps	0.00 pps	0.00 pps	0.00 pps
inblock6	0.00 pps	0.00 pps	0.00 pps	0.00 pps	
outblock6	0.00 pps	0.00 pps	0.00 pps	0.00 pps	
inpass total	242.51 kpps	242.51 kpps	242.51 kpps	242.51 kpps	0.00 pps
outpass total	265.01 kpps	265.01 kpps	265.01 kpps	265.01 kpps	0.00 pps

bmeeks

A lot of enabled rules and lots of packets would consume CPU. Have you researched some of the Snort performance tuning options? Here is a link to an older document to get you started.

https://snort-org-site.s3.amazonaws.com/production/document_files/files/000/000/030/original/WhitePaper_Snort_PerformanceTuning_2009.pdf

You can enable the PERFMON preprocessor in the Snort package and then look at the log outputs. Expect your CPU utilization to go even higher while the perfmon preprocessor is enabled. Its use and options are explained in the Snort manual here:

http://manual-snort-org.s3-website-us-east-1.amazonaws.com/node20.html

Bill

msrachelchen

I ended up using Suricata instead. Though with cpu pinning, the load is still pretty intense atm

Looking pretty juicy, 19.82 MB/s. Though most of the “CPU usage” went to NIC queues


State Table: 47% (3987752/8500000) 
Load average: 9.39, 9.96, 9.60
CPU usage: 68%

Nothing is running except Unbound, pfblockerng (DNSBL), IPSec and Suricata. I haven’t started HAProxy and TC yet…

bmeeks

@msrachelchen:

I ended up using Suricata instead. Though with cpu pinning, the load is still pretty intense atm

Looking pretty juicy, 19.82 MB/s. Though most of the “CPU usage” went to NIC queues
State Table: 47% (3987752/8500000) 
Load average: 9.39, 9.96, 9.60
CPU usage: 68%
Nothing is running except Unbound, pfblockerng (DNSBL), IPSec and Suricata. I haven’t started HAProxy and TC yet…

Suricata is multi-threaded whereas the current Snort release is still single-threaded. That can help a little under high loading. The new 3.x BETA versions of Snort are multi-threaded. Once the 3.x tree goes to RELEASE, it will get pulled into pfSense.

Bill