Alert rule triggering Drop rule?

Gblenn

I'm running Suricata in legacy mode with Block on Drop only activated. I'm primarily using SID Mgmt to help structure my rules to Drop, or Deactivate as needed, with some suppress list assistance when desired.

Currently I have all rules in the "ET INFO" list as Alert only, and that is how they show up in the Alert tab as well.

The problem I have been running into is that I seem to have some rule that blocks Google search. And I belive I have narrowed it down to this rule: ET POLICY PE EXE or DLL Windows file download HTTP (SID 1:2018959)
I placed it in the suppress list earler today but then all of a sudden, google is blocked again for the same PC...

And this is what shows up in my Block List:

Looking at the time stamps, it is the ET INFO EXE - Served Attached HTTP that somehow seems to have triggered a Blocking, even though it is only an Alert... The Remove Blocked Hosts Interval is at 3h currently. And I did clear the blocking manually at 9.30 when figuring this out.

Here's the ET INFO rule as it shows up in the Alerts tab:

What is going on here? Am I perhaps misreading the information?? Is there something I should have done in addition to adding the ET POLICY rule to the suppress list, in order to avoid this?

Gblenn

No one else seeing this?
Why are Alert rules ending up as Blocked, when the setting is Block on Drop only?
The same rule appears in both alerts.log and block.log, and in Alerts and Blocks tab in Suricata... and it's not that they show up because and actual block rule happened to target the same IP.

It looks like it is in fact the Alert rule that is what triggers the block action?!

Bob.Dig

@Gblenn I would disable it entirely instead of suppressing it.

bmeeks

@Gblenn said in Alert rule triggering Drop rule?:

No one else seeing this?
Why are Alert rules ending up as Blocked, when the setting is Block on Drop only?
The same rule appears in both alerts.log and block.log, and in Alerts and Blocks tab in Suricata... and it's not that they show up because and actual block rule happened to target the same IP.

It looks like it is in fact the Alert rule that is what triggers the block action?!

Check that you do not have two Suricata instances running on the same interface. That can happen if running Service Watchdog monitoring Suricata, and it can also happen in rare cirumstances if connectivity issues cause pfSense to rapidly issue back-to-back "restart all packages" commands.

To check for multiple processes on an interface, execute this at a shell prompt on the firewall:

ps -ax | grep suricata

You should see exactly one instance per interface. If you see any duplicates, they must be killed. The duplicate process will be a zombie and not respond to any GUI commands.

If you have duplicates, the easiest way to deal with them is to stop all Suricata interfaces in the GUI. That will stop all the regular processes. Then run the previous shell command to see what Suricata processes remain. Kill those individual from a shell prompt on the firewall using:

kill -9 <pid>

The return to the GUI and start Suricata interfaces again.

Gblenn

@bmeeks
I'm actually not using Service Watchdog and greping for suricata gives me:
97088 - Ss 3:23.70 /usr/local/bin/suricata -i ix1 -D -c /usr/local/etc/suricata/suricata_28603_ix1/
49782 0 S+ 0:00.00 grep suricata

Only one instance running...

I did however restart Suricata on the interface some time before I wrote the post today. But I suppose that should not have killed any zombies? And why would the duplicate not have the same properties, like only blocking when Drop?

Gblenn

And there it is again...

The rule that actually is set to Drop here (ET POLICY PE EXE or DLL Windows file download HTTP) is in the Suppress list since a few days back.

bmeeks

@Gblenn said in Alert rule triggering Drop rule?:

And why would the duplicate not have the same properties, like only blocking when Drop?

Because a zombie process will not respond to things the GUI changes. And since certain files are only read once during startup (suppress lists and pass lists being two examples), the zombie process would be totally unaware of anything the user is doing or has previously done in the GUI since the zombie process was started.

bmeeks

@Gblenn said in Alert rule triggering Drop rule?:

The rule that actually is set to Drop here (ET POLICY PE EXE or DLL Windows file download HTTP) is in the Suppress list since a few days back.

Is the Suppress List currently actively assigned to the interface where the block is happening?

And is the Suppress List entry for the SID, or was it created for a particular source or destination IP address? There are three methods for filtering suppress list entries: (1) by source IP; (2) by destination IP; or (3) by SID.

Gblenn

@bmeeks The suppress list is the auto generated list for the only interface which Suricata is running on. The entry is for the SID only.

And that rule (ET POLICY PE EXE or DLL Windows file download HTTP) does not show up since I put it into the suppress list yesterday (30th). You can see it from the time stamps.
Actually the last time it was "used" was 09:11 on the 30th as in the picture in my first post. There is no listing in the block.log file after that. So as far as I can see, the suppress list is working fine.

The block.log file has some 17 entries since this morning. There are a number of ET SCAN, ET CINS, ET POLICY and ET 3CORESec entries. And then there are 7 instances like this one:

12/01/2023-17:10:28.679875 [Block Src] [] [1:2014520:8] ET INFO EXE - Served Attached HTTP [] [Classification: Misc activity] [Priority: 3] {TCP} 34.104.35.123:80

I don't get it... So if it's not a zombie process, what is going on?

bmeeks

@Gblenn said in Alert rule triggering Drop rule?:

I don't get it... So if it's not a zombie process, what is going on?

Is the assigned Suppress List (as shown on the INTERFACE SETTINGS tab) actually getting parsed when the instance starts up? Look in the suricata.log file after starting the interface and you should see an entry about parsing the Suppress List (I think Suricata calls it "Threshold" or something similar). If I recall correctly, it will tell you home many entries it read and parsed from the assigned list.

Gblenn

@bmeeks Yes the suppress list is correctly showing up in Interface Settings. And I can see that it has parsed the right number of rules in suricata.log (Threshold)

I just tried commenting out one of the rules in the suppress list and the number correctly changed in the log file, so that appears to be working as well.

Bob.Dig

I think I have seen this as well, Suppress List was not working for some SID.

Gblenn

@Bob-Dig But I think the suppress list is actually working.

At least I can't see any references in the Blocks tab, or the block.log list to that SID after I entered it into the suppress list.

The strange thing here is the other rule which has Alert Action, but still triggers a Block. It shows up in both logs and both tabs.

This rule has Alert action and should NOT trigger a bock, yet here it is (again today) from the block.log file.
12/02/2023-12:49:51.351046 [Block Src] [] [1:2014520:8] ET INFO EXE - Served Attached HTTP [] [Classification: Misc activity] [Priority: 3] {TCP} 172.217.21.164:80

Gblenn

I'm about to give up on this... having Alert rules triggering Block just doesn't work. I can suppress or disable these Alert rules as well I suppose, But will I have to do that we all Alert rules then? Since any one Alert rule could potentially trigger a Block action...

Just now it was SID 1:2021076, showing up in the Blocked list, blocking the IP which originally was blocked by 1:2018959 (which is now in suppress list).

And here it is in the Alert list, the only place it should be...

bmeeks

@Gblenn: search for the SID in the active rules file and let's see what the real action verb is set for. Do that as follows:

Open a shell prompt on the firewall console either directly (if you have physical access) or via an SSH login from an available client PC (probably easier and what I would suggest).
Navigate to the interface's active rules file at /usr/local/etc/suricata/suricata_xxxx/rules/suricata.rules (where suricata_xxxx is a directory name containing the physical interface descriptor along with a UUID).
Search the active rules file for the target SID as follows:

grep 2021076 suricata.rules

That should print out the line from the file containg that rule and you can examine the actual action verb Suricata is loading for the rule.

Gblenn

@bmeeks Thanks, here's what it looks like, and it's set to alert:

alert http $EXTERNAL_NET any -> $HOME_NET any (msg:"ET HUNTING SUSPICIOUS Dotted Quad Host MZ Response"; flow:established,to_client; flowbits:isset,http.dottedquadhost; file_data; content:"MZ"; within:2; content:"PE|00 00|"; distance:0; classtype:bad-unknown; sid:2021076; rev:2; metadata:created_at 2015_05_08, former_category INFO, updated_at 2015_05_08;)

The other rule is as expected set to drop...

drop http $EXTERNAL_NET any -> $HOME_NET any (msg:"ET POLICY PE EXE or DLL Windows file download HTTP"; flow:established,to_client; flowbits:isnotset,ET.http.binary; flowbits:isnotset,ET.INFO.WindowsUpdate; file_data; content:"MZ"; within:2; byte_jump:4,58,relative,little; content:"PE|00 00|"; distance:-64; within:4; flowbits:set,ET.http.binary; classtype:policy-violation; sid:2018959; rev:4; metadata:created_at 2014_08_19, former_category POLICY, updated_at 2017_02_01;)

bmeeks

@Gblenn said in Alert rule triggering Drop rule?:

@bmeeks Thanks, here's what it looks like, and it's set to alert:

alert http $EXTERNAL_NET any -> $HOME_NET any (msg:"ET HUNTING SUSPICIOUS Dotted Quad Host MZ Response"; flow:established,to_client; flowbits:isset,http.dottedquadhost; file_data; content:"MZ"; within:2; content:"PE|00 00|"; distance:0; classtype:bad-unknown; sid:2021076; rev:2; metadata:created_at 2015_05_08, former_category INFO, updated_at 2015_05_08;)

The other rule is as expected set to drop...

drop http $EXTERNAL_NET any -> $HOME_NET any (msg:"ET POLICY PE EXE or DLL Windows file download HTTP"; flow:established,to_client; flowbits:isnotset,ET.http.binary; flowbits:isnotset,ET.INFO.WindowsUpdate; file_data; content:"MZ"; within:2; byte_jump:4,58,relative,little; content:"PE|00 00|"; distance:-64; within:4; flowbits:set,ET.http.binary; classtype:policy-violation; sid:2018959; rev:4; metadata:created_at 2014_08_19, former_category POLICY, updated_at 2017_02_01;)

Have you restarted Suricata on this interface since changing the rule action?

Also I'm not clear if you changed the rule action using the icons on the ALERTS tab or if you are doing everything solely through SID MGMT tab features.

The custom blocking module code explicitly checks an internal flag set by the Suricata binary before proceeding with a block. Here is the pertinent code section:

/* Don't block on alerts where SIG_FLAG_NOALERT is set for the Signature */
if (pa->s->flags & SIG_FLAG_NOALERT) {
    continue;
}
SC_ATOMIC_ADD(alert_pf_alerts, 1);

 /* If blocking only DROP rules and alert is not from a DROP rule, ignore it */
if (apft->ctx->block_drops && !(PacketCheckAction(p, ACTION_DROP))) {
    continue;
}

The "continue;" statement above means when the checked condition is true the loop continues to process the next alert from the packet (in case multiple rules triggered on a given packet). But it skips the blocking stuff.

Gblenn

@bmeeks About three weeks ago I did some testing with Inline mode but wasn't getting the performance I wanted. So I quickly changed back to legacy mode although this time I set it to Block on Drop Only.

I then started using SID Mgmt and added most of the Emerging Threats rules lists to the drop.conf list.

There were two lists which I did NOT change however, and these were ET INFO and ET HUNTING, which is where these alert rules are found.

I did at one point set all of the ET lists to Drop, when I was still figuring out how SID Mgmt worked... But since then I made a couple of restarts of Suricata as well as pfsense. In addition I did a complete shut down 5 days ago, as well as restarting Suricata 2 days ago... and again just now while writing this.

I use SID Mgmt to Disable QUIC and STREAM lists and a number of individual listings I previously had in the Suppress list, prior to changing to Block on Drop Only.

After having this issue I put sid:2018959 in the Suppress list using the GUI. Meaning I pressed the plus sign in the GID:SID column...

So in the picture above, 2021076 has Alert action and 2018959 is suppressed, yet both show up in the Blocks tab (and block.log).

Greping for any of the rules I currently have in the Suppress list, gives me a result consistent with what I listed above. Meaning I see the info including the expected action Drop or Alert.

Greping for rules which are in the SID Mgmt Disable.conf list, gives me empty result, (not present in suricata.rules).

Yet, none of the other rules which are in the suppress list have generated an Alert or a Block since a week back, or more. And I know at least a few of them would have, since they are related to my sons PC which would trigger both Discord and Minecraft related rules.

Gblenn

Was doing some maintenance today and made a complete shutdown of the machine. Nothing seems to have changed though, still having rules with Alert action triggering blocks...

bmeeks

@Gblenn said in Alert rule triggering Drop rule?:

Was doing some maintenance today and made a complete shutdown of the machine. Nothing seems to have changed though, still having rules with Alert action triggering blocks...

The only possibilty I know of is that internally the Suricata binary is tagging that packet with DROP for some other reason. If you are running an EVE JSON logging configuration, you can search through the EVE JSON log to find any entries related to that SID. EVE provides detailed logging information including "drop reason". A packet might have been marked as "drop" because it matches a closed flow or triggered some other internal Suricata consistency check.

The custom blocking module is not looking at the actual triggered rule's text when determining if a packet should be dropped. It is instead looking for a flag set in a memory structure of alerts related to that rule (SID). There are other reason's why Suricata's internal engine might set that drop flag. Those should be logged in the matching EVE JSON entry.

Here is a list from the Suricata source of the various reasons a packet might get marked with the "drop" flag: https://github.com/OISF/suricata/blob/13cc49388523ba19d3b5b1e72063ca7f0f444be6/src/decode.c#L827. These reasons should be logged in the EVE JSON log when you find the matching alert entry there.