Suricata custom rule alerts but won't block



  • Hello,

    I'm trying to keep hackers and spammers from beating the bejesus out of my mail server.

    Can anyone tell me why this rule will alert successfully, but will not block the remote IP in Suricata? This rule is only a slight modification from one that already exists within the rule sets. I had to create my own custom rule because my mail server responds with "Authentication failed" instead of "Authentication unsuccessful".

    alert tcp $SMTP_SERVERS 25 -> $EXTERNAL_NET any (msg:"Custom SMTP AUTH LOGON brute force attempt"; flow:from_server,established; content:"Authentication failed"; offset:54; nocase; threshold:type threshold, track by_dst, count 3, seconds 60; classtype:suspicious-login; sid:1000001; rev:3;)

    I have block offenders and kill states enabled in the Suricata interface settings.

    Thanks for any help!
    N8


  • LAYER 8

    i'm using fail2ban, it seems like i learned something new today..
    port 25 should be used for SMTP relaying not authentication. 587 is the default mail submission port, coupled with TLS encryption, will ensure that email is submitted securely if you want/can follow the guidelines set out by the IETF.

    anyway check if this can help you
    someone else was doing something similar here
    https://www.reddit.com/r/snort/comments/an5vbc/snort_alerts_and_hydra_brute_force/



  • Thanks but I don't have the means to toss this pfsense box in favor of some new system or software at the moment. Thanks for the link. I'm looking into it now.


  • LAYER 8

    i hope it help, or i'm sure that @bmeeks can help you with this



  • @kiokoman I hope he can respond. As again, that rule I have will alert, but it will not block the remote IP on 3 unsuccessful login attempts in 60 seconds as it should.



  • @RedDelPaPa said in Suricata custom rule alerts but won't block:

    @kiokoman I hope he can respond. As again, that rule I have will alert, but it will not block the remote IP on 3 unsuccessful login attempts in 60 seconds as it should.

    Are you running Suricata in Legacy Blocking Mode or Inline IPS Mode? If using Inline IPS Mode (or if using Legacy Blocking Mode with the Block on DROPS Only option enabled, then you will need to change the action verb of your rule to DROP from ALERT.

    You will also need to be sure that the IP you want to block is not included in your Pass List (if using Legacy Blocking Mode).



  • @bmeeks Hello.

    So I have tried using this rule with drop instead of alert. It still will not block remote IP's.



  • @RedDelPaPa said in Suricata custom rule alerts but won't block:

    @bmeeks Hello.

    So I have tried using this rule with drop instead of alert. It still will not block remote IP's.

    You didn't answer my questions, though.

    1. Are you running with Inline IPS Mode or Legacy Mode blocking?

    2. If using Legacy Mode, do you have the "Block on Drops Only" option checked?

    And one more question, if using Legacy Mode, how is the "Block Which IP" option configured? My guess is the IP you think should be blocked is actually in your Pass List.

    Follow-up edit: after looking again at your rule, I suspect the problem is your rule is backwards. At least what you copied into your post is not correct. The rule you copied in your post says this --

    "if any server whose IP is in the SMTP_SERVERS variable attempts an outbound connection from port 25 to any external host and any port on the Internet, then alert."

    I doubt that is what you want. You are wanting to see traffic inbound to your own mail server I suspect. Your rule should instead look like this if I am correct:

    $EXTERNAL_NET any-> $SMTP_SERVERS 25 (msg:"Custom SMTP AUTH LOGON brute force attempt"; flow:from_server,established; content:"Authentication failed"; offset:54; nocase; threshold:type threshold, track by_dst, count 3, seconds 60; classtype:suspicious-login; sid:1000001; rev:3;)
    

    The way you have your rule written, you are getting an alert when servers in your SMTP_SERVERS variable try to login to external mail servers. I suspect all of the IP addresses in SMTP_SERVERS are likely within a local subnet defined on your firewall (either LAN or DMZ), so these would all be in the default Pass List and therefore not blocked.


  • LAYER 8

    i think that the logic is that you see "Authentication failed" when $SMTP_SERVERS 25 answer back to $EXTERNAL_NET
    and in this case the
    "Which IP to Block" should be set to DST or BOTH after putting $SMTP_SERVERS to the white list



  • @bmeeks Oops, I apologize.

    1 I'm running in legacy mode.

    2 I'm not aware of a "Block on drops only" setting. I will look for it.

    What do you mean by block which IP? How and where is that determined?

    Pretty sure the test public IP I'm trying to block is not in my pass list because I have a different test rule that will block it on alert.

    Ideally, I wanted suricata to block the remote IP when the SMTP server behind pfsense returns an auth failed message to the client. I reckon I could write to block on too many AUTH LOGIN attempts from a client? In fact, I have a test rule for that which does work. So maybe it has something to do with which direction the content message travels?



  • @kiokoman Hello,

    What sets which IP to block? Is that the track by setting in the rule?



  • This post is deleted!




  • @kiokoman said in Suricata custom rule alerts but won't block:

    i think that the logic is that you see "Authentication failed" when $SMTP_SERVERS 25 answer back to $EXTERNAL_NET
    and in this case the
    "Which IP to Block" should be set to DST or BOTH after putting $SMTP_SERVERS to the white list

    Yeah, you are correct on the rule logic. It was late last night when I posted and I was not thinking clearly ... ☺ .



  • @RedDelPaPa said in Suricata custom rule alerts but won't block:

    @bmeeks 56f3b3bb-4054-4a95-9954-beff10f3aa28-image.png

    This is the screen for the "Which IP to Block" setting. It literally means which IP out of the two contained in the IP header should be blocked (source or destination IP). However, the Pass List still applies; so if either of the IP addresses is in the Pass List then that IP won't be blocked.



  • Just for a test, rewrite the rule and omit the thresholding parameters. I wonder if the threshold logic might could be responsible for my custom blocking plugin not getting a copy of the alert.



  • @bmeeks Ok, here is the new version:

    alert tcp $SMTP_SERVERS 25 -> $EXTERNAL_NET any (msg:"SMTP AUTH FAILED LOGON brute force attempt"; flow:from_server,established; content:"Authentication failed"; offset:54; nocase; classtype:suspicious-login; sid:1000005; rev:3;)

    Again, it alerts, but no blocky.



  • @RedDelPaPa said in Suricata custom rule alerts but won't block:

    @bmeeks Ok, here is the new version:

    alert tcp $SMTP_SERVERS 25 -> $EXTERNAL_NET any (msg:"SMTP AUTH FAILED LOGON brute force attempt"; flow:from_server,established; content:"Authentication failed"; offset:54; nocase; classtype:suspicious-login; sid:1000005; rev:3;)

    Again, it alerts, but no blocky.

    Hmm...this is puzzling. There is another user (from Southeast Asia, I believe) that is having a similar problem with alerts but no blocks. His posts are in the IDS/IPS sub-forum. So far I have been unable to reproduce his issue as the same setup results in blocks in my test lab but not on his production system. I'm wondering if his is load related as his firewall is processing a ton of traffic as a hub for 170 school systems.

    In your case I would not suspect loading as much, but I assume since you protecting a mail server that your firewall is for some type of business or other high user count environment. Is that true?

    Suricata has a chain of processing that packets go through, and an API is provided that allows you to register your own custom plugin into that processing chain. That's how the Legacy Mode blocking module works in Suricata for pfSense. I wrote a custom plugin that registers itself as a "log output module". In this manner, Suricata is supposed to send my plugin a copy of every single alerting packet. My plugin then analyzes the IP addresses in the packet, compares them to the Pass List addresses, and blocks the IPs if they are not in the Pass List.

    The actual alert logging, however, is done by the built-in Suricata alert-fast module. That module is the one creating the alert.log file that the GUI package then reads to display alerts on the ALERTS tab. So that means there could be a theoretical circumstance where an alert was channeled through the alert-fast module and thus logged, but that same alert did not make it to my custom blocking module. At least that's a theory I'm currently mulling around in my head.

    I've been reading through all the Suricata binary's source code trying to figure out what could possibly account for such a weird situation as experienced by the guy in Southeast Asia. Perhaps that situation is also applicable to you in some manner. I may try and get some assistance from the upstream Suricata developers to see if there is something I might be misunderstanding about the packet processing chain.



  • No, my pfsense box is for personal use. Loading is minimal.

    Does my rule block for you on your test box?

    I can make a rule that alerts and blocks if I have it check for content coming from the client instead of being sent by the smtp server.



  • @bmeeks There is also a default rule in the rule set that is almost exactly the same as mine, but I can't use it because it checks for content: Authentication unsuccessful. My server doesn't respond with that message to failed auth. My server uses: "Authentication failed".



  • @RedDelPaPa said in Suricata custom rule alerts but won't block:

    No, my pfsense box is for personal use. Loading is minimal.

    Does my rule block for you on your test box?

    For me to adequately test your rule I would have to simulate the conversation between an SMTP server and client. I don't have the software immediately on hand to accomplish that.

    I can make a rule that alerts and blocks if I have it check for content coming from the client instead of being sent by the smtp server.

    Now this seems to validate my view that your problem is actually not a problem, it's simply a matter of a host being on the Pass List (or covered by a netblock that is within the Pass List). What are the IP addresses for the server and one of these clients? I assume they can't be in the same subnet if pfSense is seeing the traffic. I think you have a potential Pass List issue where your pass list is covering more than you think in terms of IP space. Having an IP on a Pass List will not prevent alerts, but it will prevent subsequent blocks from those alerts.



  • @bmeeks hmailserver is a super simple, free windows mail server that you could use for the test, if it's the mail server you're lacking. I use windows live mail for the mail client.

    I don't think so. The remote IP is the public IP from a client who's computers I'm authorized to work on and maintain. I have remote access to his network. So occasionally I will use one of his workstations to hit back at my network for testing purposes. Pretty certain his public IP is not on any pass lists on my pfsense box.

    My suricata pass list is empty. And his public IP is not listed anywhere in any of my firewall rules. Is there another place I'm not aware of?

    Here's an example of a rule that both alerts and blocks the offending remote IP:
    alert tcp $EXTERNAL_NET any -> any 25 (msg:"SMTP AUTH LOGON brute force attempt"; content:"AUTH LOGIN"; nocase; threshold:type threshold, track by_src, count 3, seconds 60; classtype:suspicious-login; sid:1000002; rev:1;)


  • LAYER 8

    you can check your passlist from console, i don't know where in the gui is this options but anyway it's inside
    /usr/local/etc/suricata/suricata_PIDOFSURICATA_INTERFACE/passlist
    just noticed now that i have ip like 8.8.8.8 inside it that i don't remember i ever put ..
    edit .. ok could be that one time was a DNS configured for pfsense but it was removed long time ago..



  • @RedDelPaPa said in Suricata custom rule alerts but won't block:

    Here's an example of a rule that both alerts and blocks the offending remote IP:
    alert tcp $EXTERNAL_NET any -> any 25 (msg:"SMTP AUTH LOGON brute force attempt"; content:"AUTH LOGIN"; nocase; threshold:type threshold, track by_src, count 3, seconds 60; classtype:suspicious-login; sid:1000002; rev:1;)

    Your rule above that alerts and blocks is basically proving my point about a Pass List. Suricata (and Snort) generates a default Pass List that includes all of the IP subnets defined on the firewall interfaces with the exception of the WAN. For the WAN, the default Pass List only includes the actual WAN IP address. The default Pass List also includes your DNS Server(s), your default gateway and any defined Virtual IPs. You can see the contents of the default Pass List by going to the INTERFACE SETTTINGS tab for the interface you are running Suricata on and then scrolling down to click the View List button beside the Pass List drop-down selector. The content (IP addresses and subnet) of the default Pass List will display in a modal dialog.

    Your SMTP server will most definitely be defined in that default Pass List (by the inclusion of the firewall locally-attached network where it resides). So that presence on the Pass List will prevent the the mail server from being blocked. Also, depending on exactly how the remote client's traffic comes back to the firewall (over a VPN, for example), it might also wind up being included on a Pass List as VPN addresses are also added to the default Pass List.



  • @kiokoman said in Suricata custom rule alerts but won't block:

    you can check your passlist from console, i don't know where in the gui is this options but anyway it's inside
    /usr/local/etc/suricata/suricata_PIDOFSURICATA_INTERFACE/passlist
    just noticed now that i have ip like 8.8.8.8 inside it that i don't remember i ever put ..
    edit .. ok could be that one time was a DNS configured for pfsense but it was removed long time ago..

    The passlist file is only updated when Suricata is stopped and restarted or when a ccnfiguration change is made in the Suricata GUI and saved. If you were to edit the DNS servers listed under the SETUP > GENERAL screen in pfSense, but then did nothing to Suricata's configuration, then your DNS server change would not be reflected until you restarted Suricata.



  • @bmeeks Yes I thought of that one already. I disabled his vpn connection to me before testing. And his public IP is definitely not on any pass lists.

    I'm confused as to why the last rule I posted works, but the first one I posted does not.

    If the last rule works, then clearly his public IP is not on a pass list?



  • @bmeeks just followed your directions and checked the suricata default pass list. His public IP is not on it.



  • @RedDelPaPa said in Suricata custom rule alerts but won't block:

    @bmeeks Yes I thought of that one already. I disabled his vpn connection to me before testing. And his public IP is definitely not on any pass lists.

    I'm confused as to why the last rule I posted works, but the first one I posted does not.

    If the last rule works, then clearly his public IP is not on a pass list?

    Are you sure that previous rule you posted is actually firing properly? For example, I would not expect your mail server to be responding on port 25 necessarily. The client should have initiated the TCP connection by having port 25 as the destination but having a random high port in the message for the SMTP server to reply back on. But to be honest I've never done a packet analysis is an email server's conversations with a client.



  • @bmeeks yes that last one i posted works on LOGIN AUTH being sent from the client to the server.



  • @RedDelPaPa said in Suricata custom rule alerts but won't block:

    @bmeeks yes that last one i posted works on LOGIN AUTH being sent from the client to the server.

    I'm talking about the original rule you posted in your first message. This one:

    alert tcp $SMTP_SERVERS 25 -> $EXTERNAL_NET any (msg:"Custom SMTP AUTH LOGON brute force attempt"; flow:from_server,established; content:"Authentication failed"; offset:54; nocase; threshold:type threshold, track by_dst, count 3, seconds 60; classtype:suspicious-login; sid:1000001; rev:3;)
    

    Have you tried the rule above with "any" for the source port instead of "25"?





  • @RedDelPaPa said in Suricata custom rule alerts but won't block:

    @bmeeks 13ca073b-7d24-4b16-a6cb-c4d8dfaa2df6-image.png

    There is no context to the rule in this image. Is this your custom rule? If so, it is not matching what you originally posted at the start of this thread.

    If the rule above is meant to be what I just suggested a few seconds ago, I meant chaning the port to "any" not the IP address. So like this:

    alert tcp $SMTP_SERVERS any -> $EXTERNAL_NET any (msg:"Custom SMTP AUTH LOGON brute force attempt"; flow:from_server,established; content:"Authentication failed"; offset:54; nocase; threshold:type threshold, track by_dst, count 3, seconds 60; classtype:suspicious-login; sid:1000001; rev:3;)
    

    And one other point, unless you have specifically populated the SMTP_SERVERS variable on the VARIABLES tab, it will default to $HOME_NET which will be the entirety of all the locally-attached firewall subnets (as explained previously for the default pass list).



  • @bmeeks no i have not.. doesn't the email server respond back on the port the client is connected to? I always thought it did.. Keep in mind that rule is the same as the default rule with a slight mod to the content.





  • You need to be sure the relevant variables are properly populated for your servers. In this case, that means your SMTP server. The package just uses defaults for all variables that resolve to be the same as HOME_NET. HOME_NET is the sum total of all the protected networks, so that means every single subnet defined on your firewall interfaces, plus the WAN IP, the default gateway, configured DNS servers, Virtual IPs and VPN addresses.

    If you want a more restrictive setup, then create a firewall alias containing your mail server and then assign that alias to SMTP_SERVERS on the VARIABLES tab, save the change and then restart Suricata on the interface so it will pick up the change.

    You also need to be sure that you are not duplicating a rule SID. You can't have two SIDs the same, and I don't think Suricata will warn you about that in the current version.



  • The fact your bottom rule works tells you that blocking is working fine. You have a rule content issue and/or a pass list content issue. It might be time for you to do a packet capture to see what is actually going back and forth on the wire.

    Blocking is totally unaware of the content of a rule. It simply looks at the IP addresses in the packet, and if the IP address is not on a pass list (or covered by the default pass list), then it is blocked.



  • @bmeeks I don't understand what you mean by populating variables.

    Would being configured wrong cause it to still alert but not block?



  • @RedDelPaPa said in Suricata custom rule alerts but won't block:

    @bmeeks I don't understand what you mean by populating variables.

    Would being configured wrong cause it to still alert but not block?

    No, it won't impact blocking or not blocking. It can impact when the rule triggers, though. The default value for SMTP_SERVERS would be a ton of network space defined on your firewall.

    You appear to have a case where traffic you think should get a block is actually covered in the pass list defined for the interface. If you have not changed the pass list from the value of "default", then I'm telling you that pass list is rather large as it will encompass all the locally-attached networks plus VPNs. It might be that during your testing responses that you think would be your client's public IP might actually be showing up to Suricata as being wrapped with a locally-attached network (the VPN tunnel, for example). That's why you might want to capture the traffic to examine the IP headers.

    The default pass list is more than just individual IP addresses. It includes entire subnets as defined on the pfSense local interfaces.



  • @bmeeks But I told you that the rule that doesn't work alerts, and writes to the log. But it won't block.

    As stated earlier, VPN is down and disabled for testing.

    The suricata log file logs his public IP when it alerts. It just won't block it.



  • @RedDelPaPa said in Suricata custom rule alerts but won't block:

    @bmeeks But I told you that the rule that doesn't work alerts, and writes to the log. But it won't block.

    As stated earlier, VPN is down and disabled for testing.

    The suricata log file logs his public IP when it alerts. It just won't block it.

    Okay. I'm looking at something now in the custom blocking plugin source code that might be at play. If you want to help troubleshoot, do these steps:

    Use the rule that you said is alerting but not blocking, and try it using two different settings for "Which IP to Block" on the INTERFACE SETTINGS tab. Each time you change this parameter, you will need to restart Suricata on the interface so it picks up the change.

    Change that option to block just the DST IP address (change the setting from BOTH to DST). Save the change and restart Suricata. Then test your rule. It should alert and block the remote client (the one with the public IP of your customer).

    If the above test works, that validates what I think I see in the source code of the blocking module.


Log in to reply