DDoS pfSense dies on XSYN and OVH scripts.
-
Yes go on??
Go on? First we seek to understand what is happening, and now we'll go test it, and if there is something we can do about it (it's poorly named, I don't see how it generates a SYN at all, this may be a flaw we can advantage), we will develop a fix and either document it (if it's something that can be done from the GUI or by, i.e. plugging tunables into loader.conf.local) or develop a patch and get it into a release.
-
Does anyone have a copy of whatever the "OVH script" might be?
-
Thats great to hear.
@gonzopancho:
Yes go on??
Go on? First we seek to understand what is happening, and now we'll go test it, and if there is something we can do about it (it's poorly named, I don't see how it generates a SYN at all, this may be a flaw we can advantage), we will develop a fix and either document it (if it's something that can be done from the GUI or by, i.e. plugging tunables into loader.conf.local) or develop a patch and get it into a release.
-
It sounds like there is still some doubt about what this actually was.
After looking at the C code, it looked to me that it is a SSYN(Spoofed SYN) attack. Is this not correct? I know SYN cookies help against "SYN" attacks, but does it primarily only help when the source IP is not spoofed, but coming from a single IP or select few IPs?
I guess my question is, does PFSense only handle a SYN DOS but not a SYN DDOS?
edit: As pointed out, I forgot that this attack still works even if no states are created(SYN blocked), but it is exasperated if states are created.
-
Limiting the creation of states help a great deal but despite low total traffic and almost no states created, the firewall chokes somehow.
We dont know why yet and actually dont have a clue where to look.
Using the syn cookie feature doesnt help.
So you could be right about the SYN DDoS difference. Thats why we are looking at the FW's response to spoofed packages thats not really traffic…
-
We have even added some things under system -> tunables to see if it will combat the flooding, but it doesnt.
We cannot get higher maxsockbuf then the FW crashes constantly. It could be an issue on 10GbE connections since they state that this needs to be increased because of bigger load and throughput…
-
I think we have to look at the kernel route cache and see if that could be tuned somehow.
As stated it seems to choke somewhere when flooded and it doesnt take that big amount of traffic if the attack is done right…
-
hmmm…
Does this affect UDP? If a bunch of traffic from many source IPs and ports in the form of UDP packets hit, would the same issues occur? If not, then what is different about SYN packets...
If this was a routing issue, then I would think it'd affect both protocols equally, unless TCP was somehow integrated into the routing. In an ideal, but not always practical world, layer 3 and 4 would not leak into each other.
-
It can be made to hit UDP as well
21:42:35.519506 4c:00:82:ee:cf:d9 > 00:0c:29:d4:ef:2c, ethertype IPv4 (0x0800), length 67: (tos 0x0, ttl 70, id 28520, offset 0, flags [none], proto UDP (17), length 53)
38.236.77.218.27015 > 80.197.144.99.80: [no cksum] UDP, length 25
21:42:35.519522 4c:00:82:ee:cf:d9 > 00:0c:29:d4:ef:2c, ethertype IPv4 (0x0800), length 62: (tos 0x0, ttl 69, id 45833, offset 0, flags [none], proto UDP (17), length 48)
110.201.38.197.27015 > 80.197.144.99.80: [no cksum] UDP, length 20
21:42:35.519528 4c:00:82:ee:cf:d9 > 00:0c:29:d4:ef:2c, ethertype IPv4 (0x0800), length 62: (tos 0x0, ttl 38, id 61664, offset 0, flags [none], proto UDP (17), length 48)
168.105.75.13.27015 > 80.197.144.99.80: [no cksum] UDP, length 20
21:42:35.519536 4c:00:82:ee:cf:d9 > 00:0c:29:d4:ef:2c, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 77, id 18528, offset 0, flags [none], proto UDP (17), length 37)[modded by gonzopancho. This is what gists are for.]
54.77.185.24.27015 > 80.197.144.99.80: [no cksum] UDP, length 25
21:42:35.532664 4c:00:82:ee:cf:d9 > 00:0c:29:d4:ef:2c, ethertype IPv4 (0x0800), length 62: (tos 0x0, ttl 6, id 2771, offset 0, flags [none], proto UDP (17), length 48)
121.176.97.110.27015 > 80.197.144.99.80: [no cksum] UDP, length 20
21:42:35.532670 4c:00:82:ee:cf -
It sounds like there is still some doubt about what this actually was.
After looking at the C code, it looked to me that it is a SSYN(Spoofed SYN) attack. Is this not correct? I know SYN cookies help against "SYN" attacks, but does it primarily only help when the source IP is not spoofed, but coming from a single IP or select few IPs?
I guess my question is, does PFSense only handle a SYN DOS but not a SYN DDOS?
edit: As pointed out, I forgot that this attack still works even if no states are created(SYN blocked), but it is exasperated if states are created.
As it turns out, linux (the code only runs on linux) creates a new 'connection' as a side-effect. So there are SYNs in each packet.
The question "does PFSense only handle a SYN DOS but not a SYN DDOS" is nonsense.
this is an attack on anything that has a state table. Point in fact, if you turn off "keep state" on the affected rule, pfSense happily runs quite well.
(we're testing on/through 10G interfaces here.)So, assuming you have "keep state" in the affected rule(s), this will ALSO occur on FreeBSD. (So the report that it didn't was false, or didn't properly test the condition..)
We're investigating synproxy as a defense, but, if you want your firewall to stay up right now, some combination of turning off state for the affected rule(s), or keeping state table creation bounded on the affected rules, is your best bet for now.
More as it occurs.
-
I think we have to look at the kernel route cache and see if that could be tuned somehow.
As stated it seems to choke somewhere when flooded and it doesnt take that big amount of traffic if the attack is done right…
I think you are quite wrong on this point.
-
Since this affects both TCP and UDP even if firewall states are not created, I think it adds credence to the whole routing theory. I would assume this is affects by nearly any type of IP packets from many source IPs.
-
@gonzopancho:
We're investigating synproxy as a defense, but, if you want your firewall to stay up right now, some combination of turning off state for the affected rule(s), or keeping state table creation bounded on the affected rules, is your best bet for now.
More as it occurs.
I was under the impression by other posters that this issue occurs even if the firewall blocks all of the incoming traffic and no states are created. If this is true, then the firewall states getting filled up is just a side effect, but not the main issue.
edit: So even if your state table is "empty" and all of the traffic blocked, the box can still be affected. Correct, incorrect? Maybe just because 100Mb/s of 64byte packets taking the slow path(firewall check) was a bit much?
-
This picture shows it all…
Look at the states. Pretty low....traffic is somewhere around 8-10mbit. 10% of the tested pipe...
Ping is fine...Nothing reported as unusual.
But I cant load any sites through the browser...they just time out. CTRL+F5 doesnt help.
So something is choking and I cant see where...
Edit: INternal sites work fine. Reflection is working no issues and loads quickly. Anything going through the firewall times out.
 -
Correct me if I'm wrong
- Sub 10Mb/s of traffic.
- Under firewall 300 states. New states not being created by attack.
- ICMP seems to be working, but I'm not sure if this includes new ICMP states or only existing states.
- Getting timeouts when attempt to browse the internet
-
Yes. Correct. Unusable both ways.
EDIT: This is what the server sees regarding CPU usage during the attack.
-
I find it interesting that the peaks are roughly the same width, like the system would hit a limit, then do something that used less CPU for a short bit, but one core is always running at 100% during the entire ordeal.
Do you by chance have logging enabled for your default block? I wonder if that could create issues as the log would be spammed. I know I disabled logging on my default block because of network noise.
-
The following assumes that the "script" (C code) that Supermule posted toward the top of the thread fairly represents the discussed attack.
We've found that if you add set timeout tcp.first 5 to pf.conf, then the 'attack' won't render a C2758 attached via 10G interfaces useless. Without same, the C2758 becomes all but wedged in a matter of seconds.
The more effective thing is to turn off keeping state for the affected rule(s), but we understand that might be too much to ask for many people.
Since adding this timeout to the pf.conf by hand won't survive even a rule change (never mind a reboot), I'm going to have people here add code to the 'Advanced' tab (under Firewalling/Rules) to enable same. People who really want the change before we get 2.2.1 released can gitsync the code onto their box.
And now, back to you, Supermule. Chris is complaining that lowprofile isn't responding to repeated requests for more information. All I can say here is that you're in a difficult position if you claim that we're being non-responsive when we're trying to gather more information.
-
I find it interesting that the peaks are roughly the same width, like the system would hit a limit, then do something that used less CPU for a short bit, but one core is always running at 100% during the entire ordeal.
Do you by chance have logging enabled for your default block? I wonder if that could create issues as the log would be spammed. I know I disabled logging on my default block because of network noise.
At least one core is receiving traffic (given that this is ESX(i) who knows what core(s) are being used…). The state table fills, not much occurs, the state table finally starts to timeout states. lather, rinse, repeat.
-
Limiting the creation of states help a great deal but despite low total traffic and almost no states created, the firewall chokes somehow.
I didn't say "limiting" I (effectively) said "eliminating". Big difference.
We dont know why yet and actually dont have a clue where to look.
Assuming that the code you posted is an accurate version of the attack, then yes, yes we do.
Using the syn cookie feature doesnt help.
Who ever said that it did?
So you could be right about the SYN DDoS difference. Thats why we are looking at the FW's response to spoofed packages thats not really traffic…
It's traffic, it's just very short duration traffic. It's a "SYN DDOS" from one box.
If ISPs blocked outbound traffic that didn't originate on their network, (using, for example, Cisco's Unicast Reverse Path Forwarding check) then this would be far less of a problem.