DDoS pfSense dies on XSYN and OVH scripts.
-
I think we have to look at the kernel route cache and see if that could be tuned somehow.
As stated it seems to choke somewhere when flooded and it doesnt take that big amount of traffic if the attack is done right…
I think you are quite wrong on this point.
-
Since this affects both TCP and UDP even if firewall states are not created, I think it adds credence to the whole routing theory. I would assume this is affects by nearly any type of IP packets from many source IPs.
-
@gonzopancho:
We're investigating synproxy as a defense, but, if you want your firewall to stay up right now, some combination of turning off state for the affected rule(s), or keeping state table creation bounded on the affected rules, is your best bet for now.
More as it occurs.
I was under the impression by other posters that this issue occurs even if the firewall blocks all of the incoming traffic and no states are created. If this is true, then the firewall states getting filled up is just a side effect, but not the main issue.
edit: So even if your state table is "empty" and all of the traffic blocked, the box can still be affected. Correct, incorrect? Maybe just because 100Mb/s of 64byte packets taking the slow path(firewall check) was a bit much?
-
This picture shows it all…
Look at the states. Pretty low....traffic is somewhere around 8-10mbit. 10% of the tested pipe...
Ping is fine...Nothing reported as unusual.
But I cant load any sites through the browser...they just time out. CTRL+F5 doesnt help.
So something is choking and I cant see where...
Edit: INternal sites work fine. Reflection is working no issues and loads quickly. Anything going through the firewall times out.
 -
Correct me if I'm wrong
- Sub 10Mb/s of traffic.
- Under firewall 300 states. New states not being created by attack.
- ICMP seems to be working, but I'm not sure if this includes new ICMP states or only existing states.
- Getting timeouts when attempt to browse the internet
-
Yes. Correct. Unusable both ways.
EDIT: This is what the server sees regarding CPU usage during the attack.
-
I find it interesting that the peaks are roughly the same width, like the system would hit a limit, then do something that used less CPU for a short bit, but one core is always running at 100% during the entire ordeal.
Do you by chance have logging enabled for your default block? I wonder if that could create issues as the log would be spammed. I know I disabled logging on my default block because of network noise.
-
The following assumes that the "script" (C code) that Supermule posted toward the top of the thread fairly represents the discussed attack.
We've found that if you add set timeout tcp.first 5 to pf.conf, then the 'attack' won't render a C2758 attached via 10G interfaces useless. Without same, the C2758 becomes all but wedged in a matter of seconds.
The more effective thing is to turn off keeping state for the affected rule(s), but we understand that might be too much to ask for many people.
Since adding this timeout to the pf.conf by hand won't survive even a rule change (never mind a reboot), I'm going to have people here add code to the 'Advanced' tab (under Firewalling/Rules) to enable same. People who really want the change before we get 2.2.1 released can gitsync the code onto their box.
And now, back to you, Supermule. Chris is complaining that lowprofile isn't responding to repeated requests for more information. All I can say here is that you're in a difficult position if you claim that we're being non-responsive when we're trying to gather more information.
-
I find it interesting that the peaks are roughly the same width, like the system would hit a limit, then do something that used less CPU for a short bit, but one core is always running at 100% during the entire ordeal.
Do you by chance have logging enabled for your default block? I wonder if that could create issues as the log would be spammed. I know I disabled logging on my default block because of network noise.
At least one core is receiving traffic (given that this is ESX(i) who knows what core(s) are being used…). The state table fills, not much occurs, the state table finally starts to timeout states. lather, rinse, repeat.
-
Limiting the creation of states help a great deal but despite low total traffic and almost no states created, the firewall chokes somehow.
I didn't say "limiting" I (effectively) said "eliminating". Big difference.
We dont know why yet and actually dont have a clue where to look.
Assuming that the code you posted is an accurate version of the attack, then yes, yes we do.
Using the syn cookie feature doesnt help.
Who ever said that it did?
So you could be right about the SYN DDoS difference. Thats why we are looking at the FW's response to spoofed packages thats not really traffic…
It's traffic, it's just very short duration traffic. It's a "SYN DDOS" from one box.
If ISPs blocked outbound traffic that didn't originate on their network, (using, for example, Cisco's Unicast Reverse Path Forwarding check) then this would be far less of a problem.
-
This picture shows it all…
The picture shows that you are running 2.1.5. 2.2 will have a much better chance of standing up to the traffic you're seeing (testing?)
-
First of all, nice to see some progress. :)
I've been busy this weekend with family, why i haven't replied.Just to be clear, after more than 50-60 hours of sever testing of these type of attacks, i can conclude that "OVH" ESSYN, SYN ACK, SYN FLOOD all appears to be some kind of SYN attacks. Some are simple SYN ACKs, others are special attacks. These scripts are getting quite normal on the internet, since they are available to take down with no bandwidth. The trend with BIG high volume attacks are no more interesting, since these are much cheaper and much more effective and easy to bypass by ddos'ers.
Anyway, the script which was provided earlier was just 1 out of 100 available on the internet. They are getting better and more efficient , why this area need more attention from pfsense team.
I'm glad that there is a focus on it now, and i am excited to test any "patches" or good advice' to prevent such attacks.
I also testet with pfsense 2.2 which didn't show any progress. Freebsd 10.1 was handling it better, but if you tune pfsense with some parametres and set some limits you almost can prevent 85% of SYN attacks. There were though some special attacks which still took it down (SYN also), i can try find the source code of it and send it, this time to the team directly, instead of Chris. ;)@gonzopancho:
We've found that if you add set timeout tcp.first 5 to pf.conf, then the 'attack' won't render a C2758 attached via 10G interfaces useless. Without same, the C2758 becomes all but wedged in a matter of seconds.
Since adding this timeout to the pf.conf by hand won't survive even a rule change (never mind a reboot), I'm going to have people here add code to the 'Advanced' tab (under Firewalling/Rules) to enable same. People who really want the change before we get 2.2.1 released can gitsync the code onto their box.
That sounds so great! I will test it! Let us know when it is available to gitsync it.
@gonzopancho:
…. Chris is complaining that lowprofile isn't responding to repeated requests for more information. All I can say here is that you're in a difficult position if you claim that we're being non-responsive when we're trying to gather more information.
Not 100% correct, i am though not here to discuss this. Leave it as it is :)
Cheers!
-
According to Supermule, UDP does the same thing, even if the packets are being dropped.
-
According to Supermule, UDP does the same thing, even if the packets are being dropped.
I can't confirm it, it may be a combination of UDP/SYN. I will try it in next few days.
-
What is UDP/SYN? :-)
-
When a UDP packet touches itself, it becomes SYNful and your firewall crashes.
-
There is nothing wrong with a UDP packet touching itself.
Keep your religion out of your network, and you'll be just fine.
-
First of all, nice to see some progress. :)
You'll find you get a better, (less hostile) response if you don't bait me into answering.
Just to be clear, after more than 50-60 hours of sever testing of these type of attacks, i can conclude that "OVH" ESSYN, SYN ACK, SYN FLOOD all appears to be some kind of SYN attacks. Some are simple SYN ACKs, others are special attacks. These scripts are getting quite normal on the internet, since they are available to take down with no bandwidth. The trend with BIG high volume attacks are no more interesting, since these are much cheaper and much more effective and easy to bypass by ddos'ers.
And they've been part of the Internet for a long, long time. They're not even very sophisticated.
Anyway, the script which was provided earlier was just 1 out of 100 available on the internet. They are getting better and more efficient , why this area need more attention from pfsense team.
First, it's not a script, it's C source code that needs fixing to even compile.
Second, I don't care if it's one of 100 available. I don't go looking for this kind of tripe. If I need something like this (for testing), I, or one of the other people here can write one in an afternoon. I won't have the "too cool" Matrix quote on it, but otherwise it will be a much better programming example.
I'm glad that there is a focus on it now, and i am excited to test any "patches" or good advice' to prevent such attacks.
I also testet with pfsense 2.2 which didn't show any progress. Freebsd 10.1 was handling it better,If "FreeBSD 10.1 was handling it better', then your test was faulty, perhaps only in that you didn't duplicate the semantics of the ruleset, but it's not like that part of pfSense 2.2 is significantly different than FreeBSD 10.1.
but if you tune pfsense with some parametres and set some limits you almost can prevent 85% of SYN attacks.
The most telling thing here is that you haven't actually documented any of your "tuning" or "limits".
There were though some special attacks which still took it down (SYN also), i can try find the source code of it and send it, this time to the team directly, instead of Chris. ;)
Sure.
@gonzopancho:
We've found that if you add set timeout tcp.first 5 to pf.conf, then the 'attack' won't render a C2758 attached via 10G interfaces useless. Without same, the C2758 becomes all but wedged in a matter of seconds.
Since adding this timeout to the pf.conf by hand won't survive even a rule change (never mind a reboot), I'm going to have people here add code to the 'Advanced' tab (under Firewalling/Rules) to enable same. People who really want the change before we get 2.2.1 released can gitsync the code onto their box.
That sounds so great! I will test it! Let us know when it is available to gitsync it.
You could test things by hand. I've already given you the line to add to pf.conf (likely /tmp/rules.debug, but you'll have to know how to reload the ruleset from the command line. Since you're already familiar with FreeBSD 10.1, you presumably know how to do this.
@gonzopancho:
…. Chris is complaining that lowprofile isn't responding to repeated requests for more information. All I can say here is that you're in a difficult position if you claim that we're being non-responsive when we're trying to gather more information.
Not 100% correct, i am though not here to discuss this. Leave it as it is :)
Cheers!
-
@gonzopancho:
What is UDP/SYN? :-)
I read it as a question about bandwidth or SYN attack, hence my answer.
I dont even bother to reply your negative posts. We came up with an issue which needed to be enlightened. We did, and you found a "fix". = Good for everyone.
The sourcecode was provided by me anyway (found on the internet), i do know it is used by those scripts etc. No need for "nitpicking" each and every word.
Now relax and go out and enjoy your life, instead of filling this forum with your negative attitude, all the way ;) -
2.2 has the same issues and chokes the same way.
Lowprofile has setup both versions and there was no difference.
@gonzopancho:
This picture shows it all…
The picture shows that you are running 2.1.5. 2.2 will have a much better chance of standing up to the traffic you're seeing (testing?)