SURICATA QUIC failed decrypt - filling my logs
-
@Gblenn said in SURICATA QUIC failed decrypt - filling my logs:
However, looking through the lists, I can't find one single rule that has the Drop action assigned to it... So if I were to change the setup to only block when a rule has Drop action, I wouldn't block anything? Or is there some other hidden tag that will change this if I activate the Block On DROP only setting?
You are correct. All of the rules packages are shipped with the default rule action set to ALERT. You as the admin selectively choose which rules you want to actually DROP (or block) traffic and which rules should simply log informational alerts. You can do this rule-by-rule, but the easy and recommended way is to learn how to use the powerful SID Managment feature under the SID MGMT tab. The following Sticky Post at the top of this sub-forum describes how SID MGMT works with Snort. The same process works with Suricata: https://forum.netgate.com/topic/143812/snort-package-4-0-inline-ips-mode-introduction-and-configuration-instructions.
@Gblenn said in SURICATA QUIC failed decrypt - filling my logs:
Also, I'm wondering what might have changed to make these events pop up all of a sudden?
Today I'm also seeing the QUIC error on data in the block list. But it might not have happened earlier since the decrypt got blocked...?Did you recently update your pfSense to maybe 23.09 Plus? There is a new version of Suricata in that update, and the QUIC protocol was added to Suricata by the upstream developers. So, you would not have seen QUIC alerts with an earlier Suricata version.
-
@bmeeks Aha, yes I did update to 23.09 Plus a few days ago, so that makes sense then.
After some research I decided to start testing Inline mode... So far I'm just playing around on a separate VM (running v2.7.1).
One thing I noticed which makes a lot of sense, is in the Snort IPS Policy selection... there is a new option I don't see when running legacy mode? The IPS Policy Mode selection
Selecting Policy here does exactly what I thought would have happened when applying snort policy, also in legacy mode?
Perhaps I was missing something then, or isn't using policy mode supposed to work in legacy mode? Was I supposed to have ticked the box at Bock on DROP Only??Anyway, I'm happy this seems to work, so I'll experiment a bit more, and definitely learn how to use the SID Mgmt configuration. About that btw, as I already have a suppress list which I have built up over time. Is it possible to use that as is or with minimal editing and run it through e.g. the Disable SID List??
Also, besides the snort policies, are there other ways to benefit from what others have done in this area? Or if I wanted to share my settings with someone else running pfsense/suricata?
-
@Gblenn said in SURICATA QUIC failed decrypt - filling my logs:
Perhaps I was missing something then, or isn't using policy mode supposed to work in legacy mode? Was I supposed to have ticked the box at Bock on DROP Only??
Yes, you would need to have "Block on DROPs Only" enabled along with Legacy Mode Blocking in order for the IPS Policy to work the same as it does with Inline IPS Mode.
But if your hardware supports Inline IPS Mode, that is a much superior setting as compared to Legacy Blocking Mode. Legacy Blocking Mode will "leak" packets until the block decision is enforced. Inline IPS Mode does not leak packets. Legacy Blocking Mode uses a custom plugin on pfSense that analyzes copies of packets, makes a decision about the flow, then calls the pfSense kernel code to add the offending IP to an internal
pffirewall table called snort2c. -
@bmeeks I suppose I will move over to Inline IPS mode as soon as I get an opportunity. I'm quite confident my hardware on the main pfsense will also support it.
Will be interesting to see if I will see any degradation in performance... -
@Gblenn said in SURICATA QUIC failed decrypt - filling my logs:
Will be interesting to see if I will see any degradation in performance...
Performance impacts of Inline IPS Mode are dependent upon your particular hardware. The more CPU cores and the faster the clock speed, the less of an impact you will see. Also, a NIC with more queues performs better than one with less queues. But Inline IPS Mode will be slower than Legacy Blocking Mode no matter what hardware you have. It's just that if you have a large enough hardware performance "cushion", then you won't see any real-world impacts on network throughput.
You will also likely see a performance boost to Inline IPS Mode by switching the threading model to workers on the INTERFACE SETTINGS tab. The default is AutoFP, which works fine for Legacy Blocking, but workers can be more performant with Inline IPS Mode and good hardware.
-
@bmeeks Ok I guess I will have to experiment a bit then.
I am running pfsense as a VM on Proxmox with a X520-da2 passed thru, and 4 "cores" allocated from an i5-11400 right now. It certainly copes as it is, giving me up to around 7+ GBit on speedtest. I haven't tested with suricata switched off as it never goes up to more than 50-70% CPU, as reported by pfsense.
It's a home setup and I certainly don't need 10G but I got it for free so why not...
About the legacy mode, perhaps it would be a good idea to add another note, next to the one about the requirement to use snort rules. That you also must have 'Block on DROPs Only' activated for this to actually work.
-
Well, that definitely made a difference in performance... Nearly a 50% drop in throughput.
I guess it should have been expected but now I suppose I have to decide...
I can't really see a use case in a home environment for anything more than what Steam or Blizzard can push, which as far as I have seen recently was around 3 Gbit. And I did get a bit more than that from speedtest (3.8 roughly)...
-
@Gblenn said in SURICATA QUIC failed decrypt - filling my logs:
Well, that definitely made a difference in performance... Nearly a 50% drop in throughput.
Yep. Inline IPS Mode can be quite hardware intensive at very fast line rates. Very high-end NICs with multiple queues and well-written hardware drivers coupled with RSS enabled and working in the kernel and a high core count fast clock speed CPU are all required to get top-notch performance. Especially so when the hardware is also busy firewalling and routing in addition to inspecting traffic.
The fact you still got 3.8 Gigabits/sec is a testament to good hardware, but still not surprised at the performance drop. There are several "buried deep down" tuning possibilities with Suricata in terms of CPU affinity, thread counts, etc., that can help. But the GUI package on pfSense does not expose those currently.
It's also not outside the realm of possibility that, when running flat-out with Legacy Blocking Mode, Suricata was dropping packets from the inspection engine and you were not aware. Because Legacy Mode works only with copies of packets, a "dropped" packet in the inspection engine simply means a copy was likely never made of that particular packet. But because the original packet continued on through the firewall engine, the applications ingesting the packets never saw the miss.
These two diagrams show how packets flow with the two modes:
-
@bmeeks Yes I agree, it is quite good performance still, so I can't really complain too much. But it would be interesting to look at what could be done to improve it, if at all possible?
The simplest thing I could do of course is to throw a few more Cores at it... and see if that makes a difference.
-
Still doing some testing and right now I moved back to Legacy mode but have ticked the options to Block On DROP Only,
So now the Alerts tab looks different as I am now seeing all the Dropped items marked in RED, in addition to any Alerts as before.
So this was expected right, but what I don't really understand is why I am also seeing some items show up under the Blocks tab as before??
Why would I see anything in this tab, unless perhaps I have some list I have forgotten about where I have manually put some rules for blocking?? -
@Gblenn said in SURICATA QUIC failed decrypt - filling my logs:
Why would I see anything in this tab, unless perhaps I have some list I have forgotten about where I have manually put some rules for blocking??
The BLOCKS tab is populated by reading the content of the
pfsnort2c table. If you have not cleared that table previously, and if you have not enabled the option under the GLOBAL SETTINGS tab to periodically clear blocked hosts, then IP addresses inserted into that table will persist there until the firewall is rebooted. The table is a RAM construct, so a reboot clears it out.Look again at the diagrams I posted. Legacy Blocking Mode communicates with the
pffirewall engine through a system call to insert IP addresses into the snort2c table that pfSense creates in the firewall engine at boot-up. That's what the "control pathway" means in the diagram. There are hidden firewall rules created by pfSense that block any IP address listed in the snort2c table in both the inbound and outbound directions.Inline IPS Mode is quite different. That creates a literal pipe between the physical NIC and the rest of the pfSense kernel. Packets must flow through that pipe to make it to the kernel and firewall. Inline IPS Mode operates as a gate in that pipe. It either opens to pass a packet along, or if the packet triggers a DROP rule, the packet is simply swallowed by Suricata and not passed on to the firewall engine.
-
@bmeeks Thanks, that part I had understood, but I had both manually cleared the list, and I have a 1h period set, under the GLOBAL SETTINGS tab.
But my understanding was that there shouldn't really be anything in that tab if you set the Block On DROP Only option in legacy mode, or run Inline mode for that matter? Or perhaps I should expect both?
Like right now I'm seeing this:
I get the ET 3CORESec, but the other two, from nearly a month back, what relationship do they have to the recent event?
-
@Gblenn: No, you seem to misunderstand "Block on DROPs Only" mode and how it works.
That is a variation of plain vanilla Legacy Blocking Mode. It still uses libpcap to obtain copies of packets traversing the interface, feeds the copies to Suricata which makes a block/no-block decision, and then feeds that decision to the custom blocking module compiled into the Suricata package on pfSense.
The difference is that Suricata provides a link to the rule's action when it calls my custom blocking module plugin, and thus the plugin can see if the rule's action was ALERT or DROP. It will only make the system call to insert the IP into the snort2c table if the action is DROP (when "Block on DROPs Only" is enabled). Otherwise, it skips entering the IP into the table and simply logs the alert.
The Legacy Blocking Mode custom plugin for Snort can't do this because when my custom plugin is called by Snort it does not provide the alerting rule's action. It only provides the GID:SID. The plugin can't know the action. Thus any ALERT is considered grounds to "block", or insert the offender's IP into the snort2c table.
Originally the custom plugin in Suricata worked the same way as Snort. But when I realized Suricata exposed the rule's action to my plugin, it became easy to let the user optionally simulate Inline IPS Mode by examining the rule's action and only following through with inserting the IP into the snort2c table when the action was DROP. The goal of "Block on DROPs Only" is to provide a solution where some rules could block traffic while others only produced alerts.
Inline IPS Mode is a completely different animal. To start with, it does not use my custom blocking pluging at all. Inline with netmap is a native option in the Suricata binary. And it does not need to interact with the
pffirewall engine at all. The netmap device is a pipeline established by disconnecting the NIC from the kernel stack and inserting netmap as an intermediary in the path. Suricata controls the netmap "gate" and either forwards packets or drops them (really that means "does not forward"). That's how blocking works with Inline IPS Mode. Suricata, as the manager of the netmap path, takes packets from the NIC and examines them. Depending on the verdict (if DROP, for example), the packet is discarded and not forwarded on to the kernel stack. The pfSense firewall is not involved at all. This is completely different from how the Legacy Blocking code works. And is also why I stated in an earlier post that Legacy Blocking Mode leaks packets while Inline IPS Mode does not. With Inline IPS Mode, the packet goes nowhere until Suricata has produced a verdict. That's why Inline IPS Mode is so much more hardware intensive and impacts throughput as much as it does.So, you are seeing Blocks in the BLOCKS tab because "Block on DROPs Only" is still a Legacy Blocking Mode facility, and that facility uses the BLOCKS tab. Inline IPS Mode does not even load my custom blocking plugin and thus does not (and cannot) populate the BLOCKS tab.
-
@bmeeks Ok, I don't think I have completely misunderstood things here. I guess I just thought that when "simulating" inline mode, you would no longer get to see things in the Blocks tab. But I'm perfectly fine having both, in fact the Blocks tab gives me a "filtered" view in a sense...
What I do not understand however, is why I'm seeing those two older items in the block list in the picture above, Both from October 23... which three weeks back!
It would be one thing if I saw the recent ET 3CORESec instance, and any previous one's with the same SID.
If I look in block.log file I see "ET 3CORESec Poor Reputation IP group 26" as the last listing. And I can scroll up and find the two ET CINS, plus two more of the exact same SID.
Why show two, and not the other? Why are they visible at all, they were blocked three weeks ago? -
@Gblenn said in SURICATA QUIC failed decrypt - filling my logs:
What I do not understand however, is why I'm seeing those two older items in the block list in the picture above, Both from October 23... which three weeks back!
Long story requiring a lot of background, but let me see if I can shorten it up.
The custom blocking module writes a primitive
blocks.logfile on the filesystem containing an event time, the blocked IP address and port, and a little bit of additional data about the rule triggering the alert. The BLOCKS tab PHP code first dumps out the content of the snort2c table into an in-memory array, which as described above, contains nothing but the IP addresses currently being blocked by the firewall. No indication of why they are being blocked nor what rule triggered to put them in that table. To put some context to this on the BLOCKS tab, the PHP code cross-correlates the two arrays (IP addresses pulled from snort2c with the IP addresses read in from theblocks.logfile) in order to associate additional information with the currently blocked IP. Those older entries for that IP address are still in theblocks.logfile, and they match up with one of the IP addresses currently in the snort2c table and thus currently being blocked by the hidden firewall rule. Because of the match, they are shown as additional rule triggers below the first one. Notice those additional alerts are sorted by Event Time descending.Without this correlation , all that could be shown on the BLOCKS tab would be just an IP address. No way to associate with any rule that might have produced it. Folks in the past asked for previous blocks to be shown and for matching details for current blocks to also be shown, thus the multiple entries.
-
@bmeeks Aha, makes sense and thanks for clearing up that mystery!
Checking the block.logs file it was the same external IP that triggered a block action three weeks ago. -
@Gblenn said in SURICATA QUIC failed decrypt - filling my logs:
Checking the block.logs file it was the same external IP that triggered a block action three weeks ago.
Yes, and if you examine the event timestamps for those entries it will tell you those events happened in the past, but were associated with the same IP address. Also note that same IP address also triggered a different rule SID in the past.
-
@bmeeks Yes, exactly, that's how I found them, a search for the IP that was listed for the current event...
-
So now I have one more question about this...
@bmeeks said in SURICATA QUIC failed decrypt - filling my logs:
The difference is that Suricata provides a link to the rule's action when it calls my custom blocking module plugin, and thus the plugin can see if the rule's action was ALERT or DROP. It will only make the system call to insert the IP into the snort2c table if the action is DROP (when "Block on DROPs Only" is enabled). Otherwise, it skips entering the IP into the table and simply logs the alert.
Rules with Drop Action results in a block (with Block on DROP Only ticked), that is understood. And in Legacy mode, besides being marked RED in the Alerts tab, all of them also show up under the blocks tab? Including possible earlier blocks related to the same IP that create a match in the log file.
However, now I am looking at the list in the blocks tab, and see two recent blockings like this one:
I can find those exact same items marked RED in the Alerts tab as well:
But then I am also seeing another listing in the Alerts tab, more recent, but this one is NOT visible in the blocks tab?? Is this because it's a request going to pfsense?
Since if I simply type in one of the blocked IP's in the TOR rules list, it shows up in the blocks list..
[EDIT] No that was not it... This is confusing, what is actually going on here?
I do nslookup 'somthing.onion' from my PC and get the following response back:
Server: dns.google
Address: 8.8.8.8*** dns.google can't find something.onion: Non-existent domain
And it shows up under the Alerts tab like so:
And there is nothing in the blocks tab?!?Meaning... I do get a response from google, and looking into the blocks tab, confirms this as it is not listed as actually being blocked...
HOWEVER, looking in the Alerts tab I am led to belive that Drop Action was taken, -
@Gblenn said in SURICATA QUIC failed decrypt - filling my logs:
But then I am also seeing another listing in the Alerts tab, more recent, but this one is NOT visible in the blocks tab?? Is this because it's a request going to pfsense?
Those two IP addresses (192.168.1.115 and 192.168.1.1) are in RFC1918 space, so I assume those are on an internal firewall interface (LAN maybe ??). If true, then they will never result in a block because all local firewall interfaces are in the automatic pass list-- meaning those IP addresses will never be blocked although they will show as alerts. You see the DROP icon simply because that rule's action is DROP. That icon on the ALERTS tab shows you the "action" of the rule as read from the actual rule text. Also, to make them more apparent, rules with the DROP action keyword are highlighted with red text. But that does not necessarily mean each entry in red resulted in an IP address being added to the BLOCKS tab. For those that triggered and resulted in an actual block, you will see an additional red X icon beside whichever IP address (SRC or DST) was blocked.
Additionally, the default Pass List contains all of the local firewall interface subnets except that of the WAN. For the WAN, only the literal single host address of the firewall's WAN port is included. But for other internal interfaces such as the LAN, the entire subnet is pulled from the interface configuration and used in the Pass List. So, for example, likely 192.168.1.0/24 is in the default Pass List.
