Snort and internet speed problem
-
I have pfsense firewall with Snort in inline mode for good few months and everything was working fine. Few days ago I noticed that my speed test suddenly dropped from 500MBps to 150MBps. After troubleshooting the issue I narrowed it down to Snort, which for some reason started to cut my speed. Can somebody advise what to do with this?
-
@piotrir You have not described your troubleshooting methodology and how you came to that conclusion. Help us understand your process.
-
@nollipfsense
Thank you for your reply.
I'm doing from time to time speedtest.net and always get results around 500MBps but few days ago it drop to around 150MBps. In fact this is my home network as I'm trying to learn pfsense to implement on our customer sites because I really like it. I used hardware which I had, so Lenovo Think Centre M73 with one network adapter. I configured it on the stick and everything was working fine for a good few months. When the speedtest.net showed me wrong result, first thing, I checked ISP but when I connected my laptop only to their router, speed was correct. Last week I also added additional Vlan, so I removed it, but it didn't help. Eventually I've started to disable services, and once I stopped Snort - speed test came back to normal. I've tried to troubleshoot Snort but no luck so far. Changing inline mode to legacy made it even worse. I mean speed test on start was showed correctly, but after a moment the whole firewall stopped to respond, and I had to restart it. Funny thing is, that I tried to use a few different speed tests and nearly all showed 150MBps. However when I did Google speed test, the result was 500MBps. -
@piotrir said in Snort and internet speed problem:
Lenovo Think Centre M73
https://support.lenovo.com/gb/en/solutions/pd029601-detailed-specifications-for-thinkcentre-m73-small-form-factor
Ethernet : Gigabit ethernet port, Realtek RTL8111GN, Wake on LAN
Realtek cards are sub optimal, you'd get better results with an Intel based NIC.
-
@nogbadthebad
Hi Andy,
When I check network adapter details with:
pciconf -lv | grep -A1 -B3 networkIt shows me:
vendor = 'Intel Corporation'
device = 'Ethernet Connection I217-V'
class = network
subclass = ethernetSo it looks I have Intel.
-
@piotrir Ah ok I just did a google to see what NIC card came as default, Realtek cards prove troublesome.
-
@nogbadthebad
I understand, thank you for looking in to it. -
@piotrir You're running the speed tests on a workstation behind pfSense? If not, do that.
If the Google speed test is always 3x faster I'd wonder if it is using HTTPS for the test and thus Snort isn't inspecting those packets. You might try disabling rule sets in Snort to see if that makes a difference. It could be a recent rule update is problematic.
I've seen smaller devices/slower CPUs max out CPU usage during high speed downloads but that would show on the pfSense dashboard.
-
Snort packet processing speed is very closely tied to the number of rules enabled and what specific traffic types are being inspected. As others have replied, certain encrypted traffic is detected early on by Snort, and further testing against the rules is bypassed since encrypted traffic payloads can't be inspected anyway. Bypassing all that other inspection will result in increased speed.
Second thing that influences Snort packet processing is the raw single-core performance of your CPU. Snort versions earlier than the newest 3.x binary version are single-threaded. Thus they can only use a single CPU core. So no matter how many cores you may have, only one is used with Snort. And if that core is say a 2 GHz clocked core, it will likely have subpar performance compared to a 4.2 GHz clocked core. Snort on pfSense is using the 2.9.x binary and is limited to single-thread operation.
The third thing in play here is the recent move to FreeBSD-12 starting with pfSense-2.5.0 and up. FreeBSD-12 made a very large and significant change in the way NIC drivers interact with the kernel. A new API wrapper called
iflibwas introduced. There are still some growing pains with the new API wrapper, and some NICs actually perform worse under FreeBSD-12 than they do under FreeBSD-11. So a given NIC may exhibit slower performance on pfSense-2.5.x as compared to the previous 2.4.x pfSense versions.Lastly, packet payload size has an impact on the results from "speed tests". What really matters at the network layer is raw packets per second. A given hardware/software combination can process a certain number of packets per second. If those are just small packets with 64-byte payloads, then the calculated "bits/second" value will be lower than if the same number of packets per second are processed but having 1500-byte payloads instead. As an example, assume the hardware can process 1000 packets per second (an absurdly low value, but we will use it just for an easy example). If each packet has a 64-byte payload, that equates to a "speed" of 512 kilobits/sec (64 bytes * 8 = 512 bits per packet * 1000 packets/sec = 512 kilobits/sec). But if the same packets are loaded with 1500-byte payloads instead, the throughput becomes 12 megabits/second (1500 * 8 = 12,000 bits/packet * 1000 packets/sec = 12 megabits/sec). So the moral here is that the makeup of the packet payload has a lot to do with the measured bits/second throughput. Actual traffic on the wire in the real world is a mixture of several different payload sizes.
So taking into account all of the above, it is rare indeed to get "line speed" network traffic with Snort enabled and using a fair number of rules. Depending on specific hardware, including the NIC, you might get very close, but probably not full line speed. Of course, in your case, the drop from 500 to 150 is quite surprising. But if I understand you correctly, speed testing to Google seems to work while other speed tests result in slower speeds. I don't really know what to say there. Perhaps it is the traffic used by the speed test code (http versus https, or maybe fixed, large packet sizes as opposed to random sizes ???).
-
I've tried to disable rules but this didn't make any difference. I understand explanations but my problem is, Snort didn't have noticeable impact on speedtest.net since I've installed it. I haven't changed configuration of Snort (I have PfSense version 2.5.1 for good while) nor firewall and suddenly few days ago the issue started...
-
@piotrir said in Snort and internet speed problem:
I've tried to disable rules but this didn't make any difference. I understand explanations but my problem is, Snort didn't have noticeable impact on speedtest.net since I've installed it. I haven't changed configuration of Snort (I have PfSense version 2.5.1 for good while) nor firewall and suddenly few days ago the issue started...
Have you disabled all the Snort rules and then tested? If not, it is quite possible a recent Snort rule update introduced a new poorly performing revision to an existing rule, or a totally new poorly performing rule may have gotten added by the Snort VRT to a category you have enabled. When the rules update job runs (at whatever schedule you've configured), it is always possible that it will download a new rule that is poorly written, or even one that has a syntax error and prevents Snort from restarting. That has happened as well in the past. These rules are created and posted by third parties. In the case of Snort on pfSense, those parties are Emerging Threats (now Proofpoint) or the Snort Vulnerability Research Team (VRT).
The Snort binary has not changed recently, and if you have not changed the pfSense version on your firewall recently, that really does narrow down the possible suspects to perhaps being a specific rule. If you disable all rules and still are having an issue, and you have not added any other package to the firewall or otherwise made any configuration change on the firewall since this problem surfaced, then I'm out of suggestions for you to check.
There is a possibility you have one or more duplicate, zombie Snort processes running on the same interface. You can check for that by first stopping Snort on all interfaces, and then running this command from a shell prompt on the firewall:
ps -ax | grep snortYou should see no active Snort processes. If you see any, kill them. Then restart Snort on your interfaces and test again.
-
Many thanks for your help!
Indeed this must be a rule or some rules. I've disabled "IPS Policy - Balanced" category" and don't have this issue now. I've checked for zombie process and this doesn't look as the case.
I just wander - is any way to find the rule which actually causes the issue without disabling one by one? -
@piotrir said in Snort and internet speed problem:
Many thanks for your help!
Indeed this must be a rule or some rules. I've disabled "IPS Policy - Balanced" category" and don't have this issue now. I've checked for zombie process and this doesn't look as the case.
I just wander - is any way to find the rule which actually causes the issue without disabling one by one?There is a performance stats preprocessor that can be enabled on the PREPROCESSORS tab. I don't have any experience with using it, though. It supposedly gives some "per rule" performance information when you let Snort run for a while. Here is a link to the official Snort documentation from the Snort team: http://manual-snort-org.s3-website-us-east-1.amazonaws.com/node20.html. You could try adding the special configuration commands required to enable "per rule profiling" in the advanced Passthrough Options box at the bottom of the INTERFACE SETTINGS tab.
-
@PiotrIr Thanks for your post and thanks for everyone who replied. Helped me tremendously diagnosing my network performance issues. Performance stats gave me the root cause. I had the "SCADA Preprocessors" (I have no reason to) cause a massive decrease in speed.
...
