SURICATA QUIC failed decrypt - filling my logs
-
@Gblenn said in SURICATA QUIC failed decrypt - filling my logs:
Well, that definitely made a difference in performance... Nearly a 50% drop in throughput.
Yep. Inline IPS Mode can be quite hardware intensive at very fast line rates. Very high-end NICs with multiple queues and well-written hardware drivers coupled with RSS enabled and working in the kernel and a high core count fast clock speed CPU are all required to get top-notch performance. Especially so when the hardware is also busy firewalling and routing in addition to inspecting traffic.
The fact you still got 3.8 Gigabits/sec is a testament to good hardware, but still not surprised at the performance drop. There are several "buried deep down" tuning possibilities with Suricata in terms of CPU affinity, thread counts, etc., that can help. But the GUI package on pfSense does not expose those currently.
It's also not outside the realm of possibility that, when running flat-out with Legacy Blocking Mode, Suricata was dropping packets from the inspection engine and you were not aware. Because Legacy Mode works only with copies of packets, a "dropped" packet in the inspection engine simply means a copy was likely never made of that particular packet. But because the original packet continued on through the firewall engine, the applications ingesting the packets never saw the miss.
These two diagrams show how packets flow with the two modes:
-
@bmeeks Yes I agree, it is quite good performance still, so I can't really complain too much. But it would be interesting to look at what could be done to improve it, if at all possible?
The simplest thing I could do of course is to throw a few more Cores at it... and see if that makes a difference.
-
Still doing some testing and right now I moved back to Legacy mode but have ticked the options to Block On DROP Only,
So now the Alerts tab looks different as I am now seeing all the Dropped items marked in RED, in addition to any Alerts as before.
So this was expected right, but what I don't really understand is why I am also seeing some items show up under the Blocks tab as before??
Why would I see anything in this tab, unless perhaps I have some list I have forgotten about where I have manually put some rules for blocking?? -
@Gblenn said in SURICATA QUIC failed decrypt - filling my logs:
Why would I see anything in this tab, unless perhaps I have some list I have forgotten about where I have manually put some rules for blocking??
The BLOCKS tab is populated by reading the content of the
pfsnort2c table. If you have not cleared that table previously, and if you have not enabled the option under the GLOBAL SETTINGS tab to periodically clear blocked hosts, then IP addresses inserted into that table will persist there until the firewall is rebooted. The table is a RAM construct, so a reboot clears it out.Look again at the diagrams I posted. Legacy Blocking Mode communicates with the
pffirewall engine through a system call to insert IP addresses into the snort2c table that pfSense creates in the firewall engine at boot-up. That's what the "control pathway" means in the diagram. There are hidden firewall rules created by pfSense that block any IP address listed in the snort2c table in both the inbound and outbound directions.Inline IPS Mode is quite different. That creates a literal pipe between the physical NIC and the rest of the pfSense kernel. Packets must flow through that pipe to make it to the kernel and firewall. Inline IPS Mode operates as a gate in that pipe. It either opens to pass a packet along, or if the packet triggers a DROP rule, the packet is simply swallowed by Suricata and not passed on to the firewall engine.
-
@bmeeks Thanks, that part I had understood, but I had both manually cleared the list, and I have a 1h period set, under the GLOBAL SETTINGS tab.
But my understanding was that there shouldn't really be anything in that tab if you set the Block On DROP Only option in legacy mode, or run Inline mode for that matter? Or perhaps I should expect both?
Like right now I'm seeing this:
I get the ET 3CORESec, but the other two, from nearly a month back, what relationship do they have to the recent event?
-
@Gblenn: No, you seem to misunderstand "Block on DROPs Only" mode and how it works.
That is a variation of plain vanilla Legacy Blocking Mode. It still uses libpcap to obtain copies of packets traversing the interface, feeds the copies to Suricata which makes a block/no-block decision, and then feeds that decision to the custom blocking module compiled into the Suricata package on pfSense.
The difference is that Suricata provides a link to the rule's action when it calls my custom blocking module plugin, and thus the plugin can see if the rule's action was ALERT or DROP. It will only make the system call to insert the IP into the snort2c table if the action is DROP (when "Block on DROPs Only" is enabled). Otherwise, it skips entering the IP into the table and simply logs the alert.
The Legacy Blocking Mode custom plugin for Snort can't do this because when my custom plugin is called by Snort it does not provide the alerting rule's action. It only provides the GID:SID. The plugin can't know the action. Thus any ALERT is considered grounds to "block", or insert the offender's IP into the snort2c table.
Originally the custom plugin in Suricata worked the same way as Snort. But when I realized Suricata exposed the rule's action to my plugin, it became easy to let the user optionally simulate Inline IPS Mode by examining the rule's action and only following through with inserting the IP into the snort2c table when the action was DROP. The goal of "Block on DROPs Only" is to provide a solution where some rules could block traffic while others only produced alerts.
Inline IPS Mode is a completely different animal. To start with, it does not use my custom blocking pluging at all. Inline with netmap is a native option in the Suricata binary. And it does not need to interact with the
pffirewall engine at all. The netmap device is a pipeline established by disconnecting the NIC from the kernel stack and inserting netmap as an intermediary in the path. Suricata controls the netmap "gate" and either forwards packets or drops them (really that means "does not forward"). That's how blocking works with Inline IPS Mode. Suricata, as the manager of the netmap path, takes packets from the NIC and examines them. Depending on the verdict (if DROP, for example), the packet is discarded and not forwarded on to the kernel stack. The pfSense firewall is not involved at all. This is completely different from how the Legacy Blocking code works. And is also why I stated in an earlier post that Legacy Blocking Mode leaks packets while Inline IPS Mode does not. With Inline IPS Mode, the packet goes nowhere until Suricata has produced a verdict. That's why Inline IPS Mode is so much more hardware intensive and impacts throughput as much as it does.So, you are seeing Blocks in the BLOCKS tab because "Block on DROPs Only" is still a Legacy Blocking Mode facility, and that facility uses the BLOCKS tab. Inline IPS Mode does not even load my custom blocking plugin and thus does not (and cannot) populate the BLOCKS tab.
-
@bmeeks Ok, I don't think I have completely misunderstood things here. I guess I just thought that when "simulating" inline mode, you would no longer get to see things in the Blocks tab. But I'm perfectly fine having both, in fact the Blocks tab gives me a "filtered" view in a sense...
What I do not understand however, is why I'm seeing those two older items in the block list in the picture above, Both from October 23... which three weeks back!
It would be one thing if I saw the recent ET 3CORESec instance, and any previous one's with the same SID.
If I look in block.log file I see "ET 3CORESec Poor Reputation IP group 26" as the last listing. And I can scroll up and find the two ET CINS, plus two more of the exact same SID.
Why show two, and not the other? Why are they visible at all, they were blocked three weeks ago? -
@Gblenn said in SURICATA QUIC failed decrypt - filling my logs:
What I do not understand however, is why I'm seeing those two older items in the block list in the picture above, Both from October 23... which three weeks back!
Long story requiring a lot of background, but let me see if I can shorten it up.
The custom blocking module writes a primitive
blocks.logfile on the filesystem containing an event time, the blocked IP address and port, and a little bit of additional data about the rule triggering the alert. The BLOCKS tab PHP code first dumps out the content of the snort2c table into an in-memory array, which as described above, contains nothing but the IP addresses currently being blocked by the firewall. No indication of why they are being blocked nor what rule triggered to put them in that table. To put some context to this on the BLOCKS tab, the PHP code cross-correlates the two arrays (IP addresses pulled from snort2c with the IP addresses read in from theblocks.logfile) in order to associate additional information with the currently blocked IP. Those older entries for that IP address are still in theblocks.logfile, and they match up with one of the IP addresses currently in the snort2c table and thus currently being blocked by the hidden firewall rule. Because of the match, they are shown as additional rule triggers below the first one. Notice those additional alerts are sorted by Event Time descending.Without this correlation , all that could be shown on the BLOCKS tab would be just an IP address. No way to associate with any rule that might have produced it. Folks in the past asked for previous blocks to be shown and for matching details for current blocks to also be shown, thus the multiple entries.
-
@bmeeks Aha, makes sense and thanks for clearing up that mystery!
Checking the block.logs file it was the same external IP that triggered a block action three weeks ago. -
@Gblenn said in SURICATA QUIC failed decrypt - filling my logs:
Checking the block.logs file it was the same external IP that triggered a block action three weeks ago.
Yes, and if you examine the event timestamps for those entries it will tell you those events happened in the past, but were associated with the same IP address. Also note that same IP address also triggered a different rule SID in the past.
-
@bmeeks Yes, exactly, that's how I found them, a search for the IP that was listed for the current event...
-
So now I have one more question about this...
@bmeeks said in SURICATA QUIC failed decrypt - filling my logs:
The difference is that Suricata provides a link to the rule's action when it calls my custom blocking module plugin, and thus the plugin can see if the rule's action was ALERT or DROP. It will only make the system call to insert the IP into the snort2c table if the action is DROP (when "Block on DROPs Only" is enabled). Otherwise, it skips entering the IP into the table and simply logs the alert.
Rules with Drop Action results in a block (with Block on DROP Only ticked), that is understood. And in Legacy mode, besides being marked RED in the Alerts tab, all of them also show up under the blocks tab? Including possible earlier blocks related to the same IP that create a match in the log file.
However, now I am looking at the list in the blocks tab, and see two recent blockings like this one:
I can find those exact same items marked RED in the Alerts tab as well:
But then I am also seeing another listing in the Alerts tab, more recent, but this one is NOT visible in the blocks tab?? Is this because it's a request going to pfsense?
Since if I simply type in one of the blocked IP's in the TOR rules list, it shows up in the blocks list..
[EDIT] No that was not it... This is confusing, what is actually going on here?
I do nslookup 'somthing.onion' from my PC and get the following response back:
Server: dns.google
Address: 8.8.8.8*** dns.google can't find something.onion: Non-existent domain
And it shows up under the Alerts tab like so:
And there is nothing in the blocks tab?!?Meaning... I do get a response from google, and looking into the blocks tab, confirms this as it is not listed as actually being blocked...
HOWEVER, looking in the Alerts tab I am led to belive that Drop Action was taken, -
@Gblenn said in SURICATA QUIC failed decrypt - filling my logs:
But then I am also seeing another listing in the Alerts tab, more recent, but this one is NOT visible in the blocks tab?? Is this because it's a request going to pfsense?
Those two IP addresses (192.168.1.115 and 192.168.1.1) are in RFC1918 space, so I assume those are on an internal firewall interface (LAN maybe ??). If true, then they will never result in a block because all local firewall interfaces are in the automatic pass list-- meaning those IP addresses will never be blocked although they will show as alerts. You see the DROP icon simply because that rule's action is DROP. That icon on the ALERTS tab shows you the "action" of the rule as read from the actual rule text. Also, to make them more apparent, rules with the DROP action keyword are highlighted with red text. But that does not necessarily mean each entry in red resulted in an IP address being added to the BLOCKS tab. For those that triggered and resulted in an actual block, you will see an additional red X icon beside whichever IP address (SRC or DST) was blocked.
Additionally, the default Pass List contains all of the local firewall interface subnets except that of the WAN. For the WAN, only the literal single host address of the firewall's WAN port is included. But for other internal interfaces such as the LAN, the entire subnet is pulled from the interface configuration and used in the Pass List. So, for example, likely 192.168.1.0/24 is in the default Pass List.
-
@bmeeks Yes that makes sense of course. And that made med realize that 8.8.8.8 is also in the default pass list so that's probably why my other attempt, bypassing pfsense resolver, were not blocked either...
Tried a different DNS server and now it shows up in the block list.
AND, I suppose I also managed to show the drawback of legacy mode, with "package leakeage".
First attempt, I do get a response back:nslookup something.onion
Server: dns.sse.cisco.com
Address: 208.67.222.222*** dns.sse.cisco.com can't find something.onion: Non-existent domain
Second attempt, fails - blocked:
nslookup something.onion
DNS request timed out.
timeout was 2 seconds.
Server: UnKnown
Address: 208.67.222.222 -
@Gblenn said in SURICATA QUIC failed decrypt - filling my logs:
@bmeeks Yes that makes sense of course. And that made med realize that 8.8.8.8 is also in the default pass list so that's probably why my other attempt, bypassing pfsense resolver, were not blocked either...
Tried a different DNS server and now it shows up in the block list.
AND, I suppose I also managed to show the drawback of legacy mode, with "package leakeage".
First attempt, I do get a response back:nslookup something.onion
Server: dns.sse.cisco.com
Address: 208.67.222.222*** dns.sse.cisco.com can't find something.onion: Non-existent domain
Second attempt, fails - blocked:
nslookup something.onion
DNS request timed out.
timeout was 2 seconds.
Server: UnKnown
Address: 208.67.222.222Yep. The number one drawback of that mode of operation. At least the first packet (and usually several in the flow) get past before the IDS/IPS has enough data to issue a verdict on the traffic. Inline IPS Mode does not have that problem. Nothing is passed on from the NIC until the IDS/IPS has finished analzying and come to a verdict on the flow.
