Incoming RST-packets detecting to prevent DNS-amplification/Malicious Activity Abuse attacks

[Sergei_Shablovsky]

Hi, IDS/IPS Gurus!

How to detect situation when pfSense appliance receive RST packets came from hosts to which it's never sent any data?

For example if RST packets received:

IP 85.10.207.190.http > my-server-ip-xx-xxx.8389: Flags [R]
IP 85.10.207.190.http > my-server-ip-xx-xxx.8389: Flags [R]
IP 85.10.207.190.http > my-server-ip-xx-xxx.8389: Flags [R]
IP adsl-070-154.sip.pns.bellsouth.net.http > my-server-ip-xx-xxx.8389: Flags [R]
IP adsl-070-154.sip.pns.bellsouth.net.http > my-server-ip-xx-xxx.8389: Flags [R]
IP adsl-070-154.sip.pns.bellsouth.net.http > my-server-ip-xx-xxx.8389: Flags [R]

But have no any previously outgoing packets to this IPs
To take some action: send alerts to admin, etc....

How attacks realize (one client wrote to us this):
Attacker spoof my IP address and DDoS random hosts using my address as source address. (Some of providers block ability to spoofing IPs, some - no. But anyway this is RFC 2827 rules violation.).
So victims generate automatic abuse reports to my hosting providers.
So, my hosting provider can see on abuse log that connections are only in SYN_RECV state (no full TCP-connection established) because they can send only one packet using spoofed IP and can't finish TCP-handshake.
But only few providers have a tech stuff to go deep into the case and just block me.

This types of attacks start from 2017, but most of big issues popup in 2019 and counting up...

[NollipfSense]

@Sergei_Shablovsky Well one guru suggested setting up a floating firewall rule with the quick set checked...that's what I did and that way you save your IPS/IDS from doing any work.

[bmeeks]

@NollipfSense said in Incoming RST-packets detecting to prevent DNS-amplification/Malicious Activity Abuse attacks:

@Sergei_Shablovsky Well one guru suggested setting up a floating firewall rule with the quick set checked...that's what I did and that way you save your IPS/IDS from doing any work.

No you don't. The IDS/IPS sees and handles inbound packets from the WAN before any firewall rules are evaluated (floating or fixed). The path is:

Hardware NIC --> IDS/IPS --> Firewall Rules (when using Inline IPS Mode).

and

Hardware NIC --> Firewall Rules
|
--> IDS/IPS (when using Legacy Mode)

when using Legacy Mode, the libpcap library is used and copies of packets traversing the interface are processed by the IDS/IPS. So in effect, using Legacy Mode, the packets hit the IDS/IPS and the firewall rules in parallel.

[NollipfSense]

@bmeeks said in Incoming RST-packets detecting to prevent DNS-amplification/Malicious Activity Abuse attacks:

The IDS/IPS sees and handles inbound packets from the WAN before any firewall rules are evaluated (floating or fixed).

You know, actually, I had been thinking about that after I had post. You had ensured that I learn that in our many conversations and me mess up!

[Sergei_Shablovsky]

@bmeeks said in Incoming RST-packets detecting to prevent DNS-amplification/Malicious Activity Abuse attacks:

@NollipfSense said in Incoming RST-packets detecting to prevent DNS-amplification/Malicious Activity Abuse attacks:

@Sergei_Shablovsky Well one guru suggested setting up a floating firewall rule with the quick set checked...that's what I did and that way you save your IPS/IDS from doing any work.

No you don't. The IDS/IPS sees and handles inbound packets from the WAN before any firewall rules are evaluated (floating or fixed). The path is:

Hardware NIC --> IDS/IPS --> Firewall Rules (when using Inline IPS Mode).

and

Hardware NIC --> Firewall Rules
|
--> IDS/IPS (when using Legacy Mode)

when using Legacy Mode, the libpcap library is used and copies of packets traversing the interface are processed by the IDS/IPS. So in effect, using Legacy Mode, the packets hit the IDS/IPS and the firewall rules in parallel.

One technical question:

1. how IDS/IPS (Snort or Suricata, for example) impact on traffic flow (limiting throughput bandwidth or speed) in case inline mode;

2. How IDS/IPS (Snort or Suricata, for example) impact on CPU and physical memory using (digits are better than whole words:);

Of course all here about last FerrBSD and last pfSense versions.

[bmeeks]

@Sergei_Shablovsky said in Incoming RST-packets detecting to prevent DNS-amplification/Malicious Activity Abuse attacks:

One technical question:

1. how IDS/IPS (Snort or Suricata, for example) impact on traffic flow (limiting throughput bandwidth or speed) in case inline mode;

2. How IDS/IPS (Snort or Suricata, for example) impact on CPU and physical memory using (digits are better than whole words:);

Of course all here about last FerrBSD and last pfSense versions.

There is no good answer to your first question. It depends on the CPU in the firewall and how many rules are enabled. With an anemic CPU and thousands of enabled rules, it could cut throughput to one-half or even less. With a powerful multicore CPU and plenty of RAM (as in 16 GB or more), it might reduce throughput by only a few percent with the same enabled rules. There is no free lunch, though. When you use an IPS you will get reduced throughput. The amount of throughput reduction is dependent solely on the CPU horsepower and the number of enabled rules.

I have no idea what you are asking me in your second question. The translation to English is creating a confusing question. Try again on question #2 with a different approach and see if I can understand what you are asking.

[jimp]

@Sergei_Shablovsky said in Incoming RST-packets detecting to prevent DNS-amplification/Malicious Activity Abuse attacks:

How to detect situation when pfSense appliance receive RST packets came from hosts to which it's never sent any data?

Those would always be dropped, no matter what. And IDS isn't going to help, as all it could do is block the source host, and that traffic would already be dropped.

The firewall rules on pfSense which pass traffic only really pass TCP with "flags S/SA" which means that it only passes (and creates a state) for packets which have the SYN flag set and ACK flag unset.

No other combination of flags, like RST on its own, would ever pass unless you manually added your own pass rules which allow any combination of TCP flags.

If you are dealing with a flood of such traffic, the only thing you can do is have your upstream provider block it before it reaches you.

Consider: A cup of tea.

[Sergei_Shablovsky]

@bmeeks said in Incoming RST-packets detecting to prevent DNS-amplification/Malicious Activity Abuse attacks:

@Sergei_Shablovsky said in Incoming RST-packets detecting to prevent DNS-amplification/Malicious Activity Abuse attacks:

One technical question:

1. how IDS/IPS (Snort or Suricata, for example) impact on traffic flow (limiting throughput bandwidth or speed) in case inline mode;

2. How IDS/IPS (Snort or Suricata, for example) impact on CPU and physical memory using (digits are better than whole words:);

Of course all here about last FerrBSD and last pfSense versions.

There is no good answer to your first question. It depends on the CPU in the firewall and how many rules are enabled. With an anemic CPU and thousands of enabled rules, it could cut throughput to one-half or even less. With a powerful multicore CPU and plenty of RAM (as in 16 GB or more), it might reduce throughput by only a few percent with the same enabled rules. There is no free lunch, though. When you use an IPS you will get reduced throughput. The amount of throughput reduction is dependent solely on the CPU horsepower and the number of enabled rules.

Thank You for answer!

So the 2-4 of 6core Intel Xeon EX series 6000 + good 10G Intel NIG would be enough. Ok....

I have no idea what you are asking me in your second question. The translation to English is creating a confusing question. Try again on question #2 with a different approach and see if I can understand what you are asking.

Sorry my English.

I mean in Legacy Mode, when IDS/IPS running as separate process. How much this eating from CPU and memory? 5%, 15%, 30%

[Sergei_Shablovsky]

@jimp said in Incoming RST-packets detecting to prevent DNS-amplification/Malicious Activity Abuse attacks:

@Sergei_Shablovsky said in Incoming RST-packets detecting to prevent DNS-amplification/Malicious Activity Abuse attacks:

How to detect situation when pfSense appliance receive RST packets came from hosts to which it's never sent any data?

Those would always be dropped, no matter what. And IDS isn't going to help, as all it could do is block the source host, and that traffic would already be dropped.

The firewall rules on pfSense which pass traffic only really pass TCP with "flags S/SA" which means that it only passes (and creates a state) for packets which have the SYN flag set and ACK flag unset.

No other combination of flags, like RST on its own, would ever pass unless you manually added your own pass rules which allow any combination of TCP flags.

Ok, thank You for perfect answers.

If you are dealing with a flood of such traffic, the only thing you can do is have your upstream provider block it before it reaches you.

In topic I mean not exactly blocking this RST packets but send me some alarm about “You start receiving a RST packets from some hosts, that You never connect before, may be this is DNS-amplification attack”.
Smothering like
If (RST packets come in & the sender host is never be in destination address) then “send alarm to network security admin” :)

Consider: A cup of tea.
:)

[bmeeks]

@Sergei_Shablovsky said in Incoming RST-packets detecting to prevent DNS-amplification/Malicious Activity Abuse attacks:

Sorry my English.

I mean in Legacy Mode, when IDS/IPS running as separate process. How much this eating from CPU and memory? 5%, 15%, 30%

There is no separate process. It's still just the same Suricata binary. The only difference is the means of blocking. My answer is the same as that for your first question. It is totally dependent on the CPU horsepower you have and the number of enabled rules. It's the Suricata packet inspection engine that consumes CPU cycles, not the blocking mechanism. The blocking mechanism is just a very, very tiny infinitesimal portion of the CPU cycles consumed by the IPS.

[jimp]

@Sergei_Shablovsky said in Incoming RST-packets detecting to prevent DNS-amplification/Malicious Activity Abuse attacks:

In topic I mean not exactly blocking this RST packets but send me some alarm about “You start receiving a RST packets from some hosts, that You never connect before, may be this is DNS-amplification attack”.
Smothering like
If (RST packets come in & the sender host is never be in destination address) then “send alarm to network security admin” :)

Even an IDS could not tell you that you received an RST for a host you haven't communicated with. It wouldn't do state tracking to remember that.

You could manually make a block rule to match that -- for example:

• Action: Block
• Interface: WAN
• Protocol: TCP
• Source: Any
• Destination: Any
• Log: Checked
• Advanced Options:
  • TCP Flags:
    • Set: RST
    • Out of: <all checked>

That would block and log packets which only have the RST flag set (and all other flags unset)

After saving the rule, edit it again, and look at the bottom in the Rule Information section and note the tracking ID.

Setup remote syslog to an NMS or log analyzer and in there, setup an alert when you receive a log message with the tracking ID of that rule. The log analysis and alerting are not firewall tasks, but tasks performed by your NMS or log server.

[jimp]

Though in a DDoS situation the logging performed by that rule will likely cause a large increase in CPU usage if it's hit frequently.

A firewall is not a DDoS mitigation device, though, so if you hit that a lot you need to consider other solutions like DDoS mitigation services. Nothing you do on the firewall will really help that. It's too late by the time it has reached you.

[NollipfSense]

@jimp Wouldn't a custom IDS rule help to alert?

[jimp]

I don't think the IDS could tell the difference between a valid and invalid RST packet on its own in this situation. It may be possible, but might not be as simple as it would appear.

[bmeeks]

@jimp is correct. The IDS would have no knowledge of any firewall state, and the inspection engine is sort of ill-suited to track such information on its own.

My question back to the OP would be "why?". The firewall is going to drop any RST packet that it has no pre-existing state for (assuming you have a logically configured firewall: nothing can protect you from security admin incompetence, though).

All this worry about DDoS at the firewall is totally misplaced and screams, to me, a basic misunderstanding of what a DOS or DDoS attack really is. It is, at its heart, totally and completely filling up the Internet connection between you and your provider. That pipe has a finite bandwith be it 1 megabits/sec or 1 terabits/sec. Once that pipe is full, it is game over for your end of the connection. Nothing else can get down the wire no matter what your firewall does. The firewall is helpless (and it really always was when talking about a typical DOS).

As far as how you detect a DOS, well, it's pretty easy. Your Internet connection essentially quits working and you can't talk "out" and nobody can talk "in" and you see on a traffic graph that your pipe is maxed out. At that point your only recourse is upstream at your provider's level or beyond.

[NollipfSense]

@jimp said in Incoming RST-packets detecting to prevent DNS-amplification/Malicious Activity Abuse attacks:

I don't think the IDS could tell the difference between a valid and invalid RST packet on its own in this situation. It may be possible, but might not be as simple as it would appear.

I have seen that IDS/IPS systems that issue forged RST packets; however, this highlights what you said...I don't think the IDS could tell the difference between a valid and invalid RST packet.

[Sergei_Shablovsky]

@bmeeks said in Incoming RST-packets detecting to prevent DNS-amplification/Malicious Activity Abuse attacks:

@jimp is correct. The IDS would have no knowledge of any firewall state, and the inspection engine is sort of ill-suited to track such information on its own.

My question back to the OP would be "why?". The firewall is going to drop any RST packet that it has no pre-existing state for (assuming you have a logically configured firewall: nothing can protect you from security admin incompetence, though).

All this worry about DDoS at the firewall is totally misplaced and screams, to me, a basic misunderstanding of what a DOS or DDoS attack really is. It is, at its heart, totally and completely filling up the Internet connection between you and your provider. That pipe has a finite bandwith be it 1 megabits/sec or 1 terabits/sec. Once that pipe is full, it is game over for your end of the connection. Nothing else can get down the wire no matter what your firewall does. The firewall is helpless (and it really always was when talking about a typical DOS).

As far as how you detect a DOS, well, it's pretty easy. Your Internet connection essentially quits working and you can't talk "out" and nobody can talk "in" and you see on a traffic graph that your pipe is maxed out. At that point your only recourse is upstream at your provider's level or beyond.

Let's to note, that we not talking here about DDoS or RST flooding as well. Please re-read initial message, I describe the situation when attacker using legal mechanism of existed automatic detecting spam systems triggering it to automatically generate abuse message to my client hosing provider, and this abuse messages triggering the hosting provider procedure to block victim client's account.

Now @jimp describe real workaround (that I have in mind also) with log analyser to handpicking and take action by security admin.

My post here on forum - just try to find some more automated solution. But because the attacks like this are due 3-7 days long before host locked by hosting provider, may be constantly parallel task with log analyser with special trigger - are only one solution exist now to against this attack....

[johnpoz]

Why would you trigger your alerting system on a RST? That makes zero sense... If your going to spam yourself over noise - that is on you.. Not the firewall.

[Sergei_Shablovsky]

@johnpoz said in Incoming RST-packets detecting to prevent DNS-amplification/Malicious Activity Abuse attacks:

Why would you trigger your alerting system on a RST? That makes zero sense... If your going to spam yourself over noise - that is on you.. Not the firewall.

Please re-read initial message:

Step 1
attacker sitting everywhere in the world create a DDoS attacks on some hosts using the victim IP's

Step 2 on a DDoSed hosts this triggering the existed in any ISP automatic detecting spam systems that automatically generate abuse message to my client hosing proxy vider

Step 3
this abuse messages triggering the victims hosting provider procedure to block victim client's account.

Are You understand how this exactly work?

And RST packets received by victims server from the hosts that he never initiate connect before - is only one how You able to detect that happened something strange.

[johnpoz]

Please reread your own freaking statements - you just stated this is NOT a ddos..

" that we not talking here about DDoS or RST flooding as well."

Your isp gives 2 shits that there is some noise..

I don't even log such noise - because it serves no purpose.. I only log syn hits to wan, because it can be interesting to what is out there from a noise point of view, of what sort of shit is common out there... Like when that modem thing was happening and seeing traffic on that port, etc.

Again if your going to trigger alerts on noise - that is on you, your ISP sure and the hell is not going to care that RSTs are being sent to you IP..