DDoS pfSense dies on XSYN and OVH scripts.
-
Did the guy reporting that WTC towers collapsed fly the plane?
Did Charlie Hebdo hire those who almost wiped out the magazine??
Meh.
Because I report it in here, then I am suddenly responsible for any attack that takes either this site or any other site offline?
No, because you report to have used this to attack "this site" (the forum) and "store.negate.com" makes you, by your own admission, someone who used this to attack our infrastructure.
Is that professional to accuse me of doing that or could we somehow get to the bottom of this as fast as possible so we can be ahead of those who wants us of the interwebs??
What? You reported that you took down store.netgate.com and the forum.
I DDoS the shit out of myself and my own sites at the moment since I am trying to combat this with what I have and what I can tweak in the code.
We have gotten a long way but are not there yet…
I don't care what you do to your on infrastructure.
-
Great…
-
I'd have though he was just assuming.
He made a definitive statement. "It takes this forum and store.netgate.com down as well easily."
He did not say, "It would probably take this forum and store.netgate.com down as well easily."
He did not say, "I assume it takes this forum and store.netgate.com down as well easily."
Even if he didn't attack these sites, (and, if you read carefully, I have not accused, only reported back what he said),
He complained about a "lack of response" from the pfsense team, but:Neither he or lowprofile contacted coreteam@
Neither he or low profile contacted security@All I have is:
-
reports that Chris was involved, somehow.
-
knowledge that the forum was taken offline by what appears to be a similar attack on Saturday
-
people running around here reporting that this is broken in pfSense but not in FreeBSD, when, given inspection of the "script" (code) posted, there is no reason to believe that such a statement is true.
-
-
Yes go on??
Go on? First we seek to understand what is happening, and now we'll go test it, and if there is something we can do about it (it's poorly named, I don't see how it generates a SYN at all, this may be a flaw we can advantage), we will develop a fix and either document it (if it's something that can be done from the GUI or by, i.e. plugging tunables into loader.conf.local) or develop a patch and get it into a release.
-
Does anyone have a copy of whatever the "OVH script" might be?
-
Thats great to hear.
@gonzopancho:
Yes go on??
Go on? First we seek to understand what is happening, and now we'll go test it, and if there is something we can do about it (it's poorly named, I don't see how it generates a SYN at all, this may be a flaw we can advantage), we will develop a fix and either document it (if it's something that can be done from the GUI or by, i.e. plugging tunables into loader.conf.local) or develop a patch and get it into a release.
-
It sounds like there is still some doubt about what this actually was.
After looking at the C code, it looked to me that it is a SSYN(Spoofed SYN) attack. Is this not correct? I know SYN cookies help against "SYN" attacks, but does it primarily only help when the source IP is not spoofed, but coming from a single IP or select few IPs?
I guess my question is, does PFSense only handle a SYN DOS but not a SYN DDOS?
edit: As pointed out, I forgot that this attack still works even if no states are created(SYN blocked), but it is exasperated if states are created.
-
Limiting the creation of states help a great deal but despite low total traffic and almost no states created, the firewall chokes somehow.
We dont know why yet and actually dont have a clue where to look.
Using the syn cookie feature doesnt help.
So you could be right about the SYN DDoS difference. Thats why we are looking at the FW's response to spoofed packages thats not really traffic…
-
We have even added some things under system -> tunables to see if it will combat the flooding, but it doesnt.
We cannot get higher maxsockbuf then the FW crashes constantly. It could be an issue on 10GbE connections since they state that this needs to be increased because of bigger load and throughput…
-
I think we have to look at the kernel route cache and see if that could be tuned somehow.
As stated it seems to choke somewhere when flooded and it doesnt take that big amount of traffic if the attack is done right…
-
hmmm…
Does this affect UDP? If a bunch of traffic from many source IPs and ports in the form of UDP packets hit, would the same issues occur? If not, then what is different about SYN packets...
If this was a routing issue, then I would think it'd affect both protocols equally, unless TCP was somehow integrated into the routing. In an ideal, but not always practical world, layer 3 and 4 would not leak into each other.
-
It can be made to hit UDP as well
21:42:35.519506 4c:00:82:ee:cf:d9 > 00:0c:29:d4:ef:2c, ethertype IPv4 (0x0800), length 67: (tos 0x0, ttl 70, id 28520, offset 0, flags [none], proto UDP (17), length 53)
38.236.77.218.27015 > 80.197.144.99.80: [no cksum] UDP, length 25
21:42:35.519522 4c:00:82:ee:cf:d9 > 00:0c:29:d4:ef:2c, ethertype IPv4 (0x0800), length 62: (tos 0x0, ttl 69, id 45833, offset 0, flags [none], proto UDP (17), length 48)
110.201.38.197.27015 > 80.197.144.99.80: [no cksum] UDP, length 20
21:42:35.519528 4c:00:82:ee:cf:d9 > 00:0c:29:d4:ef:2c, ethertype IPv4 (0x0800), length 62: (tos 0x0, ttl 38, id 61664, offset 0, flags [none], proto UDP (17), length 48)
168.105.75.13.27015 > 80.197.144.99.80: [no cksum] UDP, length 20
21:42:35.519536 4c:00:82:ee:cf:d9 > 00:0c:29:d4:ef:2c, ethertype IPv4 (0x0800), length 60: (tos 0x0, ttl 77, id 18528, offset 0, flags [none], proto UDP (17), length 37)[modded by gonzopancho. This is what gists are for.]
54.77.185.24.27015 > 80.197.144.99.80: [no cksum] UDP, length 25
21:42:35.532664 4c:00:82:ee:cf:d9 > 00:0c:29:d4:ef:2c, ethertype IPv4 (0x0800), length 62: (tos 0x0, ttl 6, id 2771, offset 0, flags [none], proto UDP (17), length 48)
121.176.97.110.27015 > 80.197.144.99.80: [no cksum] UDP, length 20
21:42:35.532670 4c:00:82:ee:cf -
It sounds like there is still some doubt about what this actually was.
After looking at the C code, it looked to me that it is a SSYN(Spoofed SYN) attack. Is this not correct? I know SYN cookies help against "SYN" attacks, but does it primarily only help when the source IP is not spoofed, but coming from a single IP or select few IPs?
I guess my question is, does PFSense only handle a SYN DOS but not a SYN DDOS?
edit: As pointed out, I forgot that this attack still works even if no states are created(SYN blocked), but it is exasperated if states are created.
As it turns out, linux (the code only runs on linux) creates a new 'connection' as a side-effect. So there are SYNs in each packet.
The question "does PFSense only handle a SYN DOS but not a SYN DDOS" is nonsense.
this is an attack on anything that has a state table. Point in fact, if you turn off "keep state" on the affected rule, pfSense happily runs quite well.
(we're testing on/through 10G interfaces here.)So, assuming you have "keep state" in the affected rule(s), this will ALSO occur on FreeBSD. (So the report that it didn't was false, or didn't properly test the condition..)
We're investigating synproxy as a defense, but, if you want your firewall to stay up right now, some combination of turning off state for the affected rule(s), or keeping state table creation bounded on the affected rules, is your best bet for now.
More as it occurs.
-
I think we have to look at the kernel route cache and see if that could be tuned somehow.
As stated it seems to choke somewhere when flooded and it doesnt take that big amount of traffic if the attack is done right…
I think you are quite wrong on this point.
-
Since this affects both TCP and UDP even if firewall states are not created, I think it adds credence to the whole routing theory. I would assume this is affects by nearly any type of IP packets from many source IPs.
-
@gonzopancho:
We're investigating synproxy as a defense, but, if you want your firewall to stay up right now, some combination of turning off state for the affected rule(s), or keeping state table creation bounded on the affected rules, is your best bet for now.
More as it occurs.
I was under the impression by other posters that this issue occurs even if the firewall blocks all of the incoming traffic and no states are created. If this is true, then the firewall states getting filled up is just a side effect, but not the main issue.
edit: So even if your state table is "empty" and all of the traffic blocked, the box can still be affected. Correct, incorrect? Maybe just because 100Mb/s of 64byte packets taking the slow path(firewall check) was a bit much?
-
This picture shows it all…
Look at the states. Pretty low....traffic is somewhere around 8-10mbit. 10% of the tested pipe...
Ping is fine...Nothing reported as unusual.
But I cant load any sites through the browser...they just time out. CTRL+F5 doesnt help.
So something is choking and I cant see where...
Edit: INternal sites work fine. Reflection is working no issues and loads quickly. Anything going through the firewall times out.
 -
Correct me if I'm wrong
- Sub 10Mb/s of traffic.
- Under firewall 300 states. New states not being created by attack.
- ICMP seems to be working, but I'm not sure if this includes new ICMP states or only existing states.
- Getting timeouts when attempt to browse the internet
-
Yes. Correct. Unusable both ways.
EDIT: This is what the server sees regarding CPU usage during the attack.
-
I find it interesting that the peaks are roughly the same width, like the system would hit a limit, then do something that used less CPU for a short bit, but one core is always running at 100% during the entire ordeal.
Do you by chance have logging enabled for your default block? I wonder if that could create issues as the log would be spammed. I know I disabled logging on my default block because of network noise.