I added the following to the cutom.rules:

alert tcp $EXTERNAL_NET any -> any 25 (msg:"SMTP AUTH LOGON brute force attempt"; content:"AUTH LOGIN"; nocase; classtype:suspicious-login; sid:1000001; rev:2;)

alert tcp $SMTP_SERVERS 25 -> $EXTERNAL_NET any (msg:"SMTP AUTH bruteforce attack"; content:"535 Incorrect authentication data."; nocase; classtype:attempted-user; threshold:type threshold, track by_src, count 2, seconds 60; sid:1000500; rev:6;)

I am trying to alleviate the rash of auth failures spammers/bots are causing to our mail server.

Once I added to teh cutom.rules the system validates it and reloads. Than I just sit and wait but I see nothing been blocked.... I look at my mail server logs and I still see: 535 Incorrect authentication data.

Am I doing this wrong?

TIA!

If you are not seeing alerts for your custom rule, then that means it is written incorrectly. By that I don't mean a syntax error, but instead the conditions you specify in order to trigger the rule are not being detected. Here are some suggestions:

Make sure that on the VARIABLES tab you have actually assigned a correct alias or IP address to the $SMTP_SERVERS variable. The best way to do this is create a firewall alias containing the correct IP address or addresses and then make sure that alias is defined for the SMTP_SERVERS variable on the VARIABLES tab for the Snort interface.

You may also need to grab some packet captures to be sure you have the exact strings contained within your rules.

Finally, the advice of @NollipfSense only applies to you if you are running Suricata and using Inline IPS Mode, or only if you are running the Snort 4.0 package on pfSense-2.5 DEVEL and using the new Inline IPS Mode there. If you are using the Snort package on pfSense-2.4.4, then you do not change rule actions because there is no Inline IPS mode available.

I added the following to the cutom.rules:

alert tcp $EXTERNAL_NET any -> any 25 (msg:"SMTP AUTH LOGON brute force attempt"; content:"AUTH LOGIN"; nocase; classtype:suspicious-login; sid:1000001; rev:2;)

alert tcp $SMTP_SERVERS 25 -> $EXTERNAL_NET any (msg:"SMTP AUTH bruteforce attack"; content:"535 Incorrect authentication data."; nocase; classtype:attempted-user; threshold:type threshold, track by_src, count 2, seconds 60; sid:1000500; rev:6;)

I am trying to alleviate the rash of auth failures spammers/bots are causing to our mail server.

Once I added to teh cutom.rules the system validates it and reloads. Than I just sit and wait but I see nothing been blocked.... I look at my mail server logs and I still see: 535 Incorrect authentication data.

Am I doing this wrong?

TIA!

You need to change the rule to drop...for example:

Drop tcp $EXTERNAL_NET any -> any 25 (msg:"SMTP AUTH LOGON brute force attempt"; content:"AUTH LOGIN"; nocase; classtype:suspicious-login; sid:1000001; rev:2;)